How does locking mechanism work in confd?

[christopher]

Hi guys,

I went through the confd user guide and I still have some questions regarding the locking mechanism. I’ll try to explain what I understood and from that I’ll raise some questions.

When a transaction is in read state (section 7.5), one can fetch the operational data and the configuration data with the help of the maapi API. For operational data, there is no lock involved. For configuration data, a read-lock will be used on the database we reading from it (once the session is started). For instance, if one calls the function maapi_get_int32_elem on a given configuration leaf, that function will start a new session to the database, fetch the configuration data and then close the session.

When the user tries to commit some changes, confs will try to grab a read lock on the database (section 28.2.5) and also lock the transactions lock for that database (section 28.2.2). If it succeeds, from that moment no callbacks to any other transactions towards the same data store will be executed (section 7.7). Now I’m a bit confused about this statement.

• If the statement is true: If no callbacks to any other transactions towards the same data store will be executed once the transaction lock is taken, why do we need to grab any lock on the database (assuming only the data provider has access to the datastore)? I also assume this would be true only for configuration data. It is still possible for a transaction (in read state then) to fetch operational data right?

• If the statement is not true, to me it makes sense to have a read-lock, write-lock mechanism to a given datastore if for instance the transactional lock would not prohibit read accesses to this datastore as long as a write-lock is not taken (i.e as long as the state of this transaction being committed is located before the WRITE state). Actually, this goes with the fact that several read-locks can be taken on the same datastore. In that case, some transactions in read state could keep reading while a transaction (and only one, because of the transaction lock) is in its validate state. It also means that between two maapi_get_int32_elem calls (for instance) there could be a write transaction (a commit) occurring (from other manager). Once the read lock is released from this commit transaction (after validation), the transaction will try to grab a write lock on the database (section 28.2.5). If it succeeds before the second maapi_get_int32_elem gets the read lock then the write state (and all the following states) of the transaction being committed will occur first. This behavior raises another question, how can we code a netwonf get-config in a transnational way? Should take a global lock while reading all the data to make sure not commit will happen in between the readings?

If someone could shed some light on this that would be great!

Thanks in advance for your help,

Christopher.

[per]

@christopher

I went through the confd user guide and I still have some questions regarding the locking mechanism. I’ll try to explain what I understood and from that I’ll raise some questions.

You have some interesting questions, but I believe you could find at least some of the answers by more carefully studying the “ConfD Architecture” section of the chapter “An introduction to ConfD” in the User Guide, in particular the picture titled “ConfD architecture” in that section. Unfortunately, while the picture is fundamental and correct, I don’t think the terminology used is ideal - in particular the “Managed Object API” is something I would rather call the “CDB API” (the blob called “CONFIGURATION DATABASE” is of course CDB) - i.e. this is the API that is documented in the confd_lib_cdb(3) manual page. Also note that there can be other databases, a.k.a. “data providers”, than CDB.

Also, referring to sections of the User Guide by the section numbers alone is problematic, since those are subject to change when new material is added. Specifically, it seems you are looking at the User Guide for ConfD-6.6 or earlier - a version that was released over 2 years ago, and is now “End of Maintenance” (see ConfD EOM & EOS Dates | Tail-f Systems). The information is probably still valid, but at least some of the section numbers are not.

@christopher

When a transaction is in read state (section 7.5), one can fetch the operational data and the configuration data with the help of the maapi API. For operational data, there is no lock involved.

Yes.

@christopher

For configuration data, a read-lock will be used on the database we reading from it (once the session is started).

No, this is not correct, and I’m not sure how you reached that conclusion - I don’t think that anything in the chapter “The external database API” suggests that there is a difference between reading operational and configuration data via MAAPI (or via any of the built-in NB agents). Your “once the session is started” and subsequent text makes me suspect that you are confusing the northbound access to CDB via the “MANAGENT BACKPLANE” with the southbound “direct” access via the CDB API. E.g. cdb_start_session() will indeed take a read lock on the configuration data in CDB - but MAAPI does not use this interface.

@christopher

When the user tries to commit some changes, confs will try to grab a read lock on the database (section 28.2.5) and also lock the transactions lock for that database (section 28.2.2).

Well, perhaps nit-picking, but it would be more accurate to say that ConfD’s transaction engine grabs the transaction lock for the data store, and issues trans_lock() callbacks towards all configuration data providers - CDB and others (if any). CDB, which is an internal data provider, will in turn take a read lock on the CDB configuration data when trans_lock() is invoked,

@christopher

If it succeeds, from that moment no callbacks to any other transactions towards the same data store will be executed (section 7.7). Now I’m a bit confused about this statement.

Yes, this statement is not correct - it should probably say “no write callbacks” rather than just “no callbacks”. Read callbacks may certainly be invoked, both for this transaction (e.g. for validation) and for other transactions, and both for configuration and operational data. As an aside, the entire chapter is (as the name suggested) directed at implementors of external data providers for configuration data - if only CDB is used to store configuration data, you don’t really need to dig into the details.

@christopher

If the statement is not true, to me it makes sense to have a read-lock, write-lock mechanism to a given datastore if for instance the transactional lock would not prohibit read accesses to this datastore as long as a write-lock is not taken (i.e as long as the state of this transaction being committed is located before the WRITE state).

This is already the case - and furthermore read access from other transactions is not prohibited even after the transaction with the transaction lock enters the WRITE state, since such access does not take a read lock.

@christopher

• This behavior raises another question, how can we code a netwonf get-config in a transnational way? Should take a global lock while reading all the data to make sure not commit will happen in between the readings?

Yes.

[christopher]

Dear Per,

Thank you for your answer.

I’m indeed looking at the ConfD-6.6, sorry about that. I’ll move to a more recent user guide for future posts but please allow me still to refer to it in this post.

Regarding my comment about a read-lock that would be taken when the transaction is in READ state I made this assumption by myself based on the comment in section 28.2.5 namely:

“CDB has its own internal locks on the database. The running and startup datastore each has a single write and multiple read locks. It is not possible to grab the write-lock on a datastore while there are active readlocks on it. The locks in CDB exists to make sure that a reader always gets a consistent view of the data (in particular it becomes very confusing if another user is able to delete configuration nodes in between calls to get_next() on YANG list entries).”

I understand that to access the CDB, the core engine uses the Database plugin (also called data provider API right?) API and not the Managed Object API. I also understand that the core engine does not issue a cdb_start_session() (this function is for southbound daemons) but I was assuming that it would call an equivalent function for northbound access that would take a read-lock (at least between get_elem() and all the get_next() that follows), I see now that is not the case. But when reading into the database with get_elem(), get_next() (those callbacks are used in Database plugin API (section 7.6) and stated in the section 28.2.5 as shown hereabove) those functions must issue a read-lock in the database when fetching the data otherwise there could be a race condition if a transaction is writing in the same time. Is that correct? Between two calls (between get_elem() and get_next() for instance) there is no guarantee that the database has not been modified because there is no read-lock maintained between these calls. Is-that correct? If that is correct, the implementation that you show in section 7.6 and 7.7 does not provide any protection against racing condition (add_server could be executed in the prepare callback while a get_next is called). Is-that correct?

The conclusion is that the read-lock mechanism in the database (CDB) is provided only for southbound access (when starting a session) but not for core engine access through the Database Plugin API. Is-that correct? And then in that case, a global lock is the only way for a northbound agent to fetch the data in transactional way if several get_elem(),get_next() are involved.

Thanks a lot for your help.

Best,
Christopher.

[per]

@christopher

The locks in CDB exists to make sure that a reader always gets a consistent view of the data (in particular it becomes very confusing if another user is able to delete configuration nodes in between calls to get_next() on YANG list entries).”

Perhaps not entirely clear in this sentence, but from the section as a whole it should be clear that “a reader” refers to a user of the CDB/MO API. The mention of get_next is a bit confusing since that exact function doesn’t exist in the CDB API, but the typical use of an integer [index] to traverse a list has the same effect.

@christopher

the Database plugin (also called data provider API right?)

Yes.

@christopher

… those functions must issue a read-lock in the database when fetching the data otherwise there could be a race condition if a transaction is writing in the same time. Is that correct?

I’m not quite sure what you’re asking here, but there is no “read-lock” as such taken by those functions, however access to the actual CDB storage (in RAM) is effectively serialized.

I believe the answer is “yes” to all your remaining “Is that correct?” questions.

The claim that a global lock is required when reading from NB could perhaps be modified a bit - it is true that it is needed to absolutely guarantee that what is read is a consistent snapshot under all possible circumstances, but in practice this may not be an issue, since the configuration typically isn’t updated randomly from a number of different clients. In particular automation and orchestration becomes very problematic otherwise, since the management system keeps getting “out of sync” with the devices.