Hi, Please help to take a look at this critical issue hit after upgrade to 7.5.3 from 6.7.5.
The yang like below:
augment /exa:config/exa:system {
list ont {
tailf:callpoint config_cp;
tailf:info " ont";
key ont-id; unique “reg-id”;
leaf ont-id {
type string;
}
leaf reg-id {
description “registration-id”;
type string;
}
}
}
Since “reg-id” is defined as “unique” in the yang model, when configuring it , confd will iterate all the onts’ reg-id to validate whether it’s unique.
But after upgrade to 7.5.3 from 6.7.5, confd incorrectly validate the uniqueness and always rejects the configuration after some configurations later.
E7-2# show running-config ont
ont CXNK1
reg-id someexalme
!
ont CXNK2
reg-id someecefnnc2
!
ont CXNK3
reg-id 2
!
ont CXNK10
!
ont CXNK11
!
E7-2# configure
Enter configuration commands, one per line. End with CNTL/Z.
E7-2(config)# ont 4
E7-2(config-ont-4)# reg-id 2
Aborted: values are not unique: 2
‘ont 4 reg-id’
‘ont CXNK3 reg-id’ =>This uniqueness failure is correct since reg-id 2 is used by CXNK3
Error: failed to apply modifications
E7-2(config-ont-4)# end
E7-2# configure
Enter configuration commands, one per line. End with CNTL/Z.
E7-2(config)# ont 5
E7-2(config-ont-5)# reg-id 5
E7-2(config-ont-5)# ont 6
E7-2(config-ont-6)# reg-id 6
E7-2(config-ont-6)# ont 7
E7-2(config-ont-7)# reg-id 7
E7-2(config-ont-7)# ont 8
E7-2(config-ont-8)# reg-id 8
E7-2(config-ont-8)# ont 9
E7-2(config-ont-9)# reg-id 9
E7-2(config-ont-9)# ont 10
E7-2(config-ont-10)# reg-id 10
Aborted: values are not unique: 10
‘ont 10 reg-id’
‘ont 10 reg-id’ ===>This uniqueness failure is wrong since reg-id 10 is not used.
Error: failed to apply modifications
Take a closer look at the above two uniqueness.
The first one is correct and prompts different ont-id with same reg-id.
E7-2(config)# ont 4
E7-2(config-ont-4)# reg-id 2
Aborted: values are not unique: 2
‘ont 4 reg-id’
‘ont CXNK3 reg-id’ **==>This uniqueness failure is correct since reg-id 2 is used by
But the second one is wrong, and prompts the same ont-id.
E7-2(config-ont-10)# reg-id 10
Aborted: values are not unique: 10
‘ont 10 reg-id’
‘ont 10 reg-id’ ===>This uniqueness failure is wrong since reg-id 10 is not used.
Error: failed to apply modifications
We use EDP and I have double checked there are no changes in our EDP and no changes returned in the responses. I also tried out the CDB with the same above yang and didn’t hit this issue.
Please help to triage this critical issue and instruct us how to address it.
Thanks.
I agree that the error message looks weird; but I do not reproduce it. My guess is that there is something wrong in your external DB implementation - there have been quite a few changes between the two ConfD releases, perhaps some of them have not been correctly reflected in your data provider. I suggest to carefully inspect what callback are invoked on your data provider and what values were returned, especially during the last reg-id 10 command’s transaction.
Let me make a remark not directly related to the problem: the combination of unique and external database is troublesome. To validate uniqueness, ConfD has to go over all ont instances and query the value of reg-id; and using I-style CLI makes it even worse - to add n entries from the command line, ConfD has to do O(n^2) queries. This should be ok if you’ll never have more than 10 or 15 ont entries, but to add 100 entries ConfD needs to invoke tens of thousands of callbacks.
Thanks mvf. Do you mean you can’t reproduce it using EDP or CDB? I also can’t reproduce it using CDB with the same yang model. Could you help to build your own simple EDP using my yang to help to see whether you can reproduce it? I double check the value returned is correct. Please see the below logs:
==cli
E7-2# show running-config ont
ont 3
reg-id 3
!
ont 4
!
ont 5
reg-id 5
!
ont 6
reg-id 6
!
ont 7
reg-id 7
!
ont 8
reg-id 8
!
ont 9
reg-id 9
!
ont 10
!
E7-2# configure
Enter configuration commands, one per line. End with CNTL/Z.
E7-2(config)# ont 11
E7-2(config-ont-11)# reg-id 11
Aborted: values are not unique: 11
‘ont 11 reg-id’
‘ont 11 reg-id’
Error: failed to apply modifications
E7-2(config-ont-11)#
=confd_develoepr.log, the CONFD retrieve all the list instances by get_next() and then retrieval reg-id of each list by get_elem() and then validate error.
==lmd.log returns correct ont-id in get_next repsonse and correct reg-id in get_elem repsonse.
03-27-2022 17:08:17.633769 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [3]
03-27-2022 17:08:17.639999 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [4]
03-27-2022 17:08:17.646006 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [5]
03-27-2022 17:08:17.652640 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [6]
03-27-2022 17:08:17.658004 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [7]
03-27-2022 17:08:17.664011 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [8]
03-27-2022 17:08:17.669713 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [9]
03-27-2022 17:08:17.675073 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [10]
03-27-2022 17:08:17.680793 lmd: …/git/src/lm_confd.c.3724: lm_confd_cfg_get_next_cb: key[0:ont-id] type [leaf] value_str [11]
03-27-2022 17:08:17.685069 lmd:
…/git/src/lm_confd.c.3688: lm_confd_cfg_get_next_cb: list [ont] is empty, return CONFD_OK
…
03-27-2022 17:08:17.689381 lmd: …/git/src/lm_confd.c.3434: lm_confd_cfg_get_elem: path not found in cfg db [/config/system/ont{10}/reg-id]
03-27-2022 17:08:17.700565 lmd: …/git/src/lm_confd.c.3330: lm_confd_get_elem_return_value: Return value 9
03-27-2022 17:08:17.708359 lmd: …/git/src/lm_confd.c.3330: lm_confd_get_elem_return_value: Return value 8
03-27-2022 17:08:17.713216 lmd: …/git/src/lm_confd.c.3330: lm_confd_get_elem_return_value: Return value 7
03-27-2022 17:08:17.717818 lmd: …/git/src/lm_confd.c.3330: lm_confd_get_elem_return_value: Return value 6
03-27-2022 17:08:17.722348 lmd: …/git/src/lm_confd.c.3330: lm_confd_get_elem_return_value: Return value 5
03-27-2022 17:08:17.726544 lmd: …/git/src/lm_confd.c.3434: lm_confd_cfg_get_elem: path not found in cfg db [/config/system/ont{4}/reg-id]
03-27-2022 17:08:17.731773 lmd: …/git/src/lm_confd.c.3330: lm_confd_get_elem_return_value: Return value 3
The ont list is large and usually more than thousands in our customer site. We also face the performance in legacy release. But now the basic function is broken . The first few configuration can be created successfully, but at certain point, the next configuration get failed.
Please help us to address it. It causes the basic service failed. we can’t deliver the release to our customer on schedule.
Thanks.
Hi,
I also believe there is something wrong in your external DB implementation.
If you want help from the community debugging that, it would be helpful if you provided your data provider application libconfd trace log with log level set to “CONFD_TRACE” when you for example call confd_init()
The issue seems to be that ConfD 7.5.3 by default requires your data provider application to sort the data the same way CDB does. You get this error as you are not sorting the ont-id strings in lexicographic order as CDB does. I.e. ont 10 would come first in the list, not last as your application provides it to ConfD.
You can set /confdConfig/sortTransactions to false if you wish to sort the strings the way you currently do it.
From the confd.conf(5) man page:
/confdConfig/sortTransactions (boolean) [true]
This parameter controls how ConfD lists newly created, not yet committed list entries. If this value is
set to 'false', ConfD will list all new entries before listing existing data.
If this value is set to 'true', ConfD will merge new and existing entries, and provide one sorted view
of the data. This behavior works well when CDB is used to store configuration data, but if an external
data provider is used, ConfD does not know the sort order, and can thus not merge the new entries
correctly. If an external data provider is used for configuration data, and the sort order differs from
CDB's sort order, this parameter should be set to 'false'.