ConfD User Community

Uniqueness fails after upgrade to 7.5.3

Hi, Please help to take a look at this critical issue hit after upgrade to 7.5.3 from 6.7.5.

The yang like below:

augment /exa:config/exa:system {
list ont {
tailf:callpoint config_cp;
tailf:info " ont";
key ont-id;
unique “reg-id”;
leaf ont-id {
type string;
}
leaf reg-id {
description “registration-id”;
type string;
}
}
}

Since “reg-id” is defined as “unique” in the yang model, when configuring it , confd will iterate all the onts’ reg-id to validate whether it’s unique.
But after upgrade to 7.5.3 from 6.7.5, confd incorrectly validate the uniqueness and always rejects the configuration after some configurations later.

E7-2# show running-config ont
ont CXNK1
reg-id someexalme
!
ont CXNK2
reg-id someecefnnc2
!
ont CXNK3
reg-id 2
!
ont CXNK10
!
ont CXNK11
!
E7-2# configure
Enter configuration commands, one per line. End with CNTL/Z.
E7-2(config)# ont 4
E7-2(config-ont-4)# reg-id 2
Aborted: values are not unique: 2
‘ont 4 reg-id’
‘ont CXNK3 reg-id’ =>This uniqueness failure is correct since reg-id 2 is used by CXNK3
Error: failed to apply modifications
E7-2(config-ont-4)# end
E7-2# configure
Enter configuration commands, one per line. End with CNTL/Z.
E7-2(config)# ont 5
E7-2(config-ont-5)# reg-id 5
E7-2(config-ont-5)# ont 6
E7-2(config-ont-6)# reg-id 6
E7-2(config-ont-6)# ont 7
E7-2(config-ont-7)# reg-id 7
E7-2(config-ont-7)# ont 8
E7-2(config-ont-8)# reg-id 8
E7-2(config-ont-8)# ont 9
E7-2(config-ont-9)# reg-id 9
E7-2(config-ont-9)# ont 10
E7-2(config-ont-10)# reg-id 10
Aborted: values are not unique: 10
‘ont 10 reg-id’
‘ont 10 reg-id’ ===>This uniqueness failure is wrong since reg-id 10 is not used.
Error: failed to apply modifications

Take a closer look at the above two uniqueness.
The first one is correct and prompts different ont-id with same reg-id.

E7-2(config)# ont 4
E7-2(config-ont-4)# reg-id 2
Aborted: values are not unique: 2
‘ont 4 reg-id’
‘ont CXNK3 reg-id’ **==>This uniqueness failure is correct since reg-id 2 is used by

But the second one is wrong, and prompts the same ont-id.

E7-2(config-ont-10)# reg-id 10
Aborted: values are not unique: 10
‘ont 10 reg-id’
‘ont 10 reg-id’ ===>This uniqueness failure is wrong since reg-id 10 is not used.
Error: failed to apply modifications

We use EDP and I have double checked there are no changes in our EDP and no changes returned in the responses. I also tried out the CDB with the same above yang and didn’t hit this issue.
Please help to triage this critical issue and instruct us how to address it.
Thanks.

I agree that the error message looks weird; but I do not reproduce it. My guess is that there is something wrong in your external DB implementation - there have been quite a few changes between the two ConfD releases, perhaps some of them have not been correctly reflected in your data provider. I suggest to carefully inspect what callback are invoked on your data provider and what values were returned, especially during the last reg-id 10 command’s transaction.

Let me make a remark not directly related to the problem: the combination of unique and external database is troublesome. To validate uniqueness, ConfD has to go over all ont instances and query the value of reg-id; and using I-style CLI makes it even worse - to add n entries from the command line, ConfD has to do O(n^2) queries. This should be ok if you’ll never have more than 10 or 15 ont entries, but to add 100 entries ConfD needs to invoke tens of thousands of callbacks.

Thanks mvf. Do you mean you can’t reproduce it using EDP or CDB? I also can’t reproduce it using CDB with the same yang model. Could you help to build your own simple EDP using my yang to help to see whether you can reproduce it? I double check the value returned is correct. Please see the below logs:

==cli
E7-2# show running-config ont
ont 3
reg-id 3
!
ont 4
!
ont 5
reg-id 5
!
ont 6
reg-id 6
!
ont 7
reg-id 7
!
ont 8
reg-id 8
!
ont 9
reg-id 9
!
ont 10
!
E7-2# configure
Enter configuration commands, one per line. End with CNTL/Z.
E7-2(config)# ont 11
E7-2(config-ont-11)# reg-id 11
Aborted: values are not unique: 11
‘ont 11 reg-id’
‘ont 11 reg-id’
Error: failed to apply modifications
E7-2(config-ont-11)#

=confd_develoepr.log, the CONFD retrieve all the list instances by get_next() and then retrieval reg-id of each list by get_elem() and then validate error.

27-Mar-2022::17:08:13.094 E7-2 confd[10325]: devel-c new_trans request daemon id: 3 thandle: 42
27-Mar-2022::17:08:13.103 E7-2 confd[10325]: devel-c new_trans succeeded daemon id: 3 session id: 42 worker id: 150
27-Mar-2022::17:08:13.105 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{11}/ont-id
27-Mar-2022::17:08:13.109 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{11}/ont-id
27-Mar-2022::17:08:13.618 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{11}/ont-id
27-Mar-2022::17:08:13.621 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{11}/ont-id
27-Mar-2022::17:08:17.595 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{11}/ont-id
27-Mar-2022::17:08:17.598 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{11}/ont-id
27-Mar-2022::17:08:17.599 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{11}/reg-id
27-Mar-2022::17:08:17.602 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{11}/reg-id
27-Mar-2022::17:08:17.603 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{11}/ont-id
27-Mar-2022::17:08:17.606 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{11}/ont-id
27-Mar-2022::17:08:17.607 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{11}/reg-id
27-Mar-2022::17:08:17.609 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{11}/reg-id
27-Mar-2022::17:08:17.615 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli applying transaction
27-Mar-2022::17:08:17.615 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli entering validate phase
27-Mar-2022::17:08:17.616 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli grabbing transaction lock
27-Mar-2022::17:08:17.616 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli grabbing transaction lock (0.000 s)
27-Mar-2022::17:08:17.617 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli creating rollback file
27-Mar-2022::17:08:17.618 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli creating rollback file (0.000 s)
27-Mar-2022::17:08:17.619 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli run transforms and transaction hooks
27-Mar-2022::17:08:17.625 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli run transforms and transaction hooks (0.007 s)
27-Mar-2022::17:08:17.628 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli pre validate
27-Mar-2022::17:08:17.629 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli pre validate (0.001 s)
27-Mar-2022::17:08:17.629 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli run validation over the changeset
27-Mar-2022::17:08:17.631 E7-2 confd[10325]: devel-c get_next request for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.635 E7-2 confd[10325]: devel-c get_next succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.637 E7-2 confd[10325]: devel-c get_next request for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.642 E7-2 confd[10325]: devel-c get_next succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.643 E7-2 confd[10325]: devel-c get_next request for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.648 E7-2 confd[10325]: devel-c get_next succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.649 E7-2 confd[10325]: devel-c get_next request for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.654 E7-2 confd[10325]: devel-c get_next succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.655 E7-2 confd[10325]: devel-c get_next request for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.660 E7-2 confd[10325]: devel-c get_next succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.661 E7-2 confd[10325]: devel-c get_next request for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.666 E7-2 confd[10325]: devel-c get_next succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.667 E7-2 confd[10325]: devel-c get_next request for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.671 E7-2 confd[10325]: devel-c get_next succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.673 E7-2 confd[10325]: devel-c get_next request for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.677 E7-2 confd[10325]: devel-c get_next succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.678 E7-2 confd[10325]: devel-c get_next request for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.682 E7-2 confd[10325]: devel-c get_next succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.683 E7-2 confd[10325]: devel-c get_next request for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.685 E7-2 confd[10325]: devel-c get_next succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont
27-Mar-2022::17:08:17.687 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{10}/reg-id
27-Mar-2022::17:08:17.690 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{10}/reg-id
27-Mar-2022::17:08:17.698 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{9}/reg-id
27-Mar-2022::17:08:17.702 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{9}/reg-id
27-Mar-2022::17:08:17.704 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{8}/reg-id
27-Mar-2022::17:08:17.709 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{8}/reg-id
27-Mar-2022::17:08:17.710 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{7}/reg-id
27-Mar-2022::17:08:17.714 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{7}/reg-id
27-Mar-2022::17:08:17.715 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{6}/reg-id
27-Mar-2022::17:08:17.718 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{6}/reg-id
27-Mar-2022::17:08:17.720 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{5}/reg-id
27-Mar-2022::17:08:17.723 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{5}/reg-id
27-Mar-2022::17:08:17.724 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{4}/reg-id
27-Mar-2022::17:08:17.727 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{4}/reg-id
27-Mar-2022::17:08:17.729 E7-2 confd[10325]: devel-c get_elem request for callpoint config_cp path /exa:config/system/gpon-base:ont{3}/reg-id
27-Mar-2022::17:08:17.732 E7-2 confd[10325]: devel-c get_elem succeeded for callpoint config_cp path /exa:config/system/gpon-base:ont{3}/reg-id
27-Mar-2022::17:08:17.738 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli run validation over the changeset: error (0.106 s)
27-Mar-2022::17:08:17.745 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli releasing transaction lock
27-Mar-2022::17:08:17.749 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli leaving validate phase (0.134 s)
27-Mar-2022::17:08:17.749 E7-2 confd[10325]: confd progress usid=14 tid=42 datastore=running context=cli applying transaction: error (0.134 s)
27-Mar-2022::17:08:17.756 E7-2 confd[10325]: devel-c close_trans request daemon id: 3 session id: 42
27-Mar-2022::17:08:17.758 E7-2 confd[10325]: devel-c close_trans succeeded daemon id: 3 session id: 42

==lmd.log returns correct ont-id in get_next repsonse and correct reg-id in get_elem repsonse.

03-27-2022 17:08:17.633769 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [3]
03-27-2022 17:08:17.639999 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [4]
03-27-2022 17:08:17.646006 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [5]
03-27-2022 17:08:17.652640 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [6]
03-27-2022 17:08:17.658004 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [7]
03-27-2022 17:08:17.664011 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [8]
03-27-2022 17:08:17.669713 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [9]
03-27-2022 17:08:17.675073 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [10]
03-27-2022 17:08:17.680793 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [11]
03-27-2022 17:08:17.685069 lmd:
…/git/src/lm_confd.c.3688: lm_confd_cfg_get_next_cb: list [ont] is empty, return CONFD_OK


03-27-2022 17:08:17.689381 lmd: …/git/src/lm_confd.c.3434: lm_confd_cfg_get_elem: path not found in cfg db [/config/system/ont{10}/reg-id]
03-27-2022 17:08:17.700565 lmd: …/git/src/lm_confd.c.3330: lm_confd_get_elem_return_value: Return value 9
03-27-2022 17:08:17.708359 lmd: …/git/src/lm_confd.c.3330: lm_confd_get_elem_return_value: Return value 8
03-27-2022 17:08:17.713216 lmd: …/git/src/lm_confd.c.3330: lm_confd_get_elem_return_value: Return value 7
03-27-2022 17:08:17.717818 lmd: …/git/src/lm_confd.c.3330: lm_confd_get_elem_return_value: Return value 6
03-27-2022 17:08:17.722348 lmd: …/git/src/lm_confd.c.3330: lm_confd_get_elem_return_value: Return value 5
03-27-2022 17:08:17.726544 lmd: …/git/src/lm_confd.c.3434: lm_confd_cfg_get_elem: path not found in cfg db [/config/system/ont{4}/reg-id]
03-27-2022 17:08:17.731773 lmd: …/git/src/lm_confd.c.3330: lm_confd_get_elem_return_value: Return value 3

The ont list is large and usually more than thousands in our customer site. We also face the performance in legacy release. But now the basic function is broken . The first few configuration can be created successfully, but at certain point, the next configuration get failed.
Please help us to address it. It causes the basic service failed. we can’t deliver the release to our customer on schedule.
Thanks.

Hi,
I also believe there is something wrong in your external DB implementation.

If you want help from the community debugging that, it would be helpful if you provided your data provider application libconfd trace log with log level set to “CONFD_TRACE” when you for example call confd_init()

Example ConfD-Demos/cdboper_dp.c at 1322bab514354ad5af98006b8dd11b871882ef49 · ConfD-Developer/ConfD-Demos · GitHub

The issue seems to be that ConfD 7.5.3 by default requires your data provider application to sort the data the same way CDB does. You get this error as you are not sorting the ont-id strings in lexicographic order as CDB does. I.e. ont 10 would come first in the list, not last as your application provides it to ConfD.
You can set /confdConfig/sortTransactions to false if you wish to sort the strings the way you currently do it.

From the confd.conf(5) man page:

/confdConfig/sortTransactions (boolean) [true]

This parameter controls how ConfD lists newly created, not yet committed list entries. If this value is
set to 'false', ConfD will list all new entries before listing existing data.
If this value is set to 'true', ConfD will merge new and existing entries, and provide one sorted view
of the data. This behavior works well when CDB is used to store configuration data, but if an external
data provider is used, ConfD does not know the sort order, and can thus not merge the new entries
correctly. If an external data provider is used for configuration data, and the sort order differs from
CDB's sort order, this parameter should be set to 'false'.

@cohult , much appreciated for helping us figure it out.
Yes, Setting /confdConfig/sortTransactions to false does fix the issue.
Thanks a lot.