Error: 11 the configuration database is locked

CDB and CDB API
1

Hello Team,

We are using netconf for our network related entries - Access points and clients to be more specific.
For this we use a huge schema which includes all the configuration for a list of APs.

We have registered few action-points and subscriptions. Find the output of confd --status below.

vsn: 6.0.2
SMP support: no
Using epoll: no
available modules: backplane,netconf,cdb,cli,snmp,webui
running modules: backplane,netconf,cdb,cli,webui
status: started
namespaces: urn:ietf:params:xml:ns:yang:iana-crypt-hash prefix:ianach exported to: all
_ urn:ietf:params:xml:ns:yang:ietf-inet-types prefix:inet exported to: all_
_ urn:ietf:params:xml:ns:yang:ietf-netconf-acm prefix:nacm exported to: all_
_ urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring prefix:ncm exported to: all_
_ urn:ietf:params:xml:ns:yang:ietf-netconf-notifications prefix:ncn exported to: all_
_ urn:ietf:params:xml:ns:yang:ietf-yang-types prefix:yang exported to: all_
_ urn:ietf:params:xml:ns:netmod:notification prefix:nm exported to: netconf_
_ http://tail-f.com/ns/aaa/1.1 prefix:aaa exported to: all_
_ http://tail-f.com/yang/acm prefix:tacm exported to: all_
_ http://tail-f.com/yang/common-monitoring prefix:tfcg exported to: all_
_ http://tail-f.com/yang/confd-monitoring prefix:tfcm exported to: all_
_ http://tail-f.com/yang/netconf-monitoring prefix:tncm exported to: all_
_ http://tail-f.com/ns/webui prefix:webui exported to: all_
_ http://tembosystems.com/ns/datamodels/wireless prefix:wireless exported to: all_

YANG data models: _
_ module: iana-crypt-hash revision: 2014-04-04
_ namespace: urn:ietf:params:xml:ns:yang:iana-crypt-hash_
_ prefix: ianach_
_ exported to: all_
_ module: ietf-inet-types revision: 2013-07-15_
_ namespace: urn:ietf:params:xml:ns:yang:ietf-inet-types_
_ prefix: inet_
_ exported to: all_
_ module: ietf-netconf-acm revision: 2012-02-22_
_ namespace: urn:ietf:params:xml:ns:yang:ietf-netconf-acm_
_ prefix: nacm_
_ exported to: all_
_ module: ietf-netconf-monitoring revision: 2010-10-04_
_ namespace: urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring_
_ prefix: ncm_
_ exported to: all_
_ module: ietf-netconf-notifications revision: 2012-02-06_
_ namespace: urn:ietf:params:xml:ns:yang:ietf-netconf-notifications_
_ prefix: ncn_
_ exported to: all_
_ module: ietf-yang-types revision: 2013-07-15_
_ namespace: urn:ietf:params:xml:ns:yang:ietf-yang-types_
_ prefix: yang_
_ exported to: all_
_ module: netconf_netmod_
_ namespace: urn:ietf:params:xml:ns:netmod:notification_
_ prefix: nm_
_ exported to: netconf_
_ module: tailf-aaa revision: 2015-06-16_
_ namespace: http://tail-f.com/ns/aaa/1.1_
_ prefix: aaa_
_ exported to: all_
_ module: tailf-acm revision: 2013-03-07_
_ namespace: http://tail-f.com/yang/acm_
_ prefix: tacm_
_ exported to: all_
_ module: tailf-common-monitoring revision: 2013-06-14_
_ namespace: http://tail-f.com/yang/common-monitoring_
_ prefix: tfcg_
_ exported to: all_
_ module: tailf-confd-monitoring revision: 2013-06-14_
_ namespace: http://tail-f.com/yang/confd-monitoring_
_ prefix: tfcm_
_ exported to: all_
_ module: tailf-netconf-monitoring revision: 2014-11-13_
_ namespace: http://tail-f.com/yang/netconf-monitoring_
_ prefix: tncm_
_ exported to: all_
_ module: tailf-webui revision: 2013-03-07_
_ namespace: http://tail-f.com/ns/webui_
_ prefix: webui_
_ exported to: all_
_ module: wireless_config_
_ namespace: http://tembosystems.com/ns/datamodels/wireless_
_ prefix: wireless_
_ exported to: all_

user sessions:
_ sessionId=11 2016-10-25 09:01:56 admin@172.17.1.246 netconf/ssh_
_ no locks set_
_ no transactions_
callpoints:
_ id=ap-radio-stats daemonId=0 daemonName=Netconf Proxy_
_ id=ap_status daemonId=0 daemonName=Netconf Proxy_
_ id=discovered_ap_status daemonId=0 daemonName=Netconf Proxy_
_ id=license_counts ** not registered_
_ id=license_file_oper_data ** not registered_
_ **unknown id= daemonId=0 daemonName=Netconf Proxy_

validation points:

actionpoints:
_ id=ap-config-point daemonId=10 daemonName=CDB action_
_ id=ap-global-config-point daemonId=10 daemonName=CDB action_
_ id=ap-image-file-download daemonId=10 daemonName=CDB action_
_ id=ap-image-reset daemonId=10 daemonName=CDB action_
_ id=approve-point daemonId=10 daemonName=CDB action_
_ id=license daemonId=10 daemonName=CDB action_
_ id=license-revoke-point ** not registered_

typepoints:

notification stream replay support:
_ name=NETCONF _
_ name=managed_ap_events _

SNMP inform delivery callbacks:

SNMP notification subscriptions:

authentication callback:
_ not enabled_

authorization callbacks:
_ not enabled_

error formatting callbacks:

_partial running locks: _

_partial candidate locks: _

_partial startup locks: _

cdb:
_ current transaction id: 1477-310728-831843_
_ running:_
_ filename: /home/poornima/wlan/tools/confd/install/var/confd/cdb/A.cdb_
_ disk size: 793.3 kB_
_ ram size: 5.8065 MB_
_ read locks: 0_
_ write lock: unset_
_ operational: disabled_
_ no pending subscription notifications_
_ registered cdb clients:_
_ client name: Netconf Config Proxy 31190/6_
_ type: subscriber_
_ subscriptions:_
_ db=running id=7 priority=0 path=/managed-ap/database_
_ db=running id=6 priority=0 path=/system_
_ client name: Netconf Config Proxy 31190/5_
_ type: inactive_

tts alloc:
_ 5765745 40:16733:246 64:61741:819 72:24:1 120:9:2 176:11:2_

One of our action points writes configuration into the CDB using maapi calls. We would also be doing parallel reads to fetch the entry from the cdb. At some point we get this below error,

DEBUG in use - the configuration database is locked
0015165 | 2016-10-25, 09:17:11.347625 [confd_nc][error] - Failed to add AP to CDB : 02:42:AC:11:02:04
error: 11 the configuration database is locked

Your assistance regarding this issue is very helpful.

Best regards

2

It is likely that you are running into a deadlock situation. Refer to the “Locks” section of the Advanced Topics Chapter in the ConfD User Guide for more information. The section on “Lock impact on user sessions” may provide a possible solution.

3

Yes the user guide section was very helpful. But I need some further clarifications which are lacking in the section.

The operation we are doing is bulk commits using REST. So I presume it is the write lock which is being already taken during validate and one more commit fails with the error. Is this correct?

From the user guide, I understand that the transaction engine manages all transaction locks and there is no API exposing these transactional locks to the north-bound agent. How do I further debug this? Are there any specific logs I can enable?

Best regards,
Poornima.M

4

There are global locks that can be taken, see the UG chapters that Wai pointed you to.
In you case however, it seems like your application have started a CDB session through the CDB API. From the ConfD 6.2 UG Chapter 5.5. A session:

It is important to consider that CDB is locked for writing during a read session using the C API. A session starts with cdb_start_session() and the lock is not released until the cdb_end_session() (or the cdb_close()) call.

If you start a CDB session, and hence take a lock on CDB, then try to do a write transaction through MAAPI or any other northbound interface such as NETCONF, CLI, … the transaction will fail immediately unless you set the commit retry timeout to other than default. See /confdConfig/commitRetryTimeout in confd.conf

Using cdb_start_session2() instead of cdb_start_session() from your application you have some more options to the default lock that is used with cdb_start_session().

5

Thanks a lot cohult.

Yes, I have set this timeout to 120s now and seems to have fixed the problem. But the confd.conf.full explains that this configuration is only for CLI. Since we are doing commits from rest should we use the commitRetry instead?

        Commit timeout in the CLI. This timeout controls for how
        long the commit operation will attempt to complete the
        operation when some other entity is locking the database,
        e.g. some other commit is in progress or some managed
        object is locking the database.

        There is a similiar configuration parameter,
        /confdConfig/commitRetry, which sets a timeout for all
        ConfD transactions, not just for CLI transactions.

Best regards

6

Hi,

That is a documentation bug in confd.conf.full. See the confd.conf man page instead.

The text in confd.conf.full describes /confdConfig/cli/commitRetryTimeout but is placed where /confdConfig/commitRetryTimeout is located.

I don’t know your use case, but setting the /confdConfig/commitRetryTimeout to 120s is in most cases a very long time. If you have a manager with a shorter timeout that might cause issues. So you probably want to take a look at if you can shorten your application read session time. I.e. the time your application spend between cdb_start_session() and cdb_end_session()/cdb_close().