Must statement with count causes a performance issue

Hi all, I hit a performance issue when using must statement with count().
The snippet yang model is as below:

==gpon-interface-cli.yang
augment /exa:config/exa:interface {
			list ont-ethernet{
			    key port;
				
		        leaf port {
			        description "Ethernet port";
		        	type string;
	   	            tailf:info "Ethernet port";
		        }
				leaf role {
			        description "Ethernet role";
		        	type string;
	   	            tailf:info "Ethernet role";
		        }
		        ...
		     }// list ont-ethernet
			 
==gpon-service-cli.yang		 
augment /exa:config/exa:interface/gpon-cli:ont-ethernet {
	list vlan {
			when "../role = 'uni' or ../role = 'enni'" {
				tailf:dependency "../role";
			}
			
			must "count(../dtag-vlan) + count(.) <= 12"  {
					tailf:dependency "../dtag-vlan";
					tailf:dependency ".";
					error-message "interface ont-ethernet service limit exceeded";
				}
			max-elements 12;
			
				
			key "vlan-id";
			leaf vlan-id{
				description "VlanId (range: 1-4094)";
				type uint32;
			}
		}
	list dtag-vlan {
			when "../role = 'uni'" {
				tailf:dependency "../role";
			}
			
			must "count(../vlan) + count(.) <= 12"  {
					tailf:dependency "../vlan";
					tailf:dependency ".";
					error-message "interface ont-ethernet service limit exceeded";
				}
			max-elements 12;
			
				
			key "vlan-id";
			leaf vlan-id{
				description "VlanId (range: 1-4094)";
				type uint32;
			}
		}
}

The cli cmd is :

GPON-8R2-West-Lake(config)# interface ont-ethernet 1/g1
GPON-8R2-West-Lake(config-ont-ethernet-1/g1)# vlan 789

Then the performance issue occurs. CONFD invokes get_next() to iterate all the /config/interface/ont-ethernet to validate the must count statement. What we expect is that the must count statement is only to validate the requested ont-ethernet. Why does CONFD need to iterate all the ont-ethernet?

The snippet confd_developer.log is as below:

<DEBUG> 7-Aug-2020::10:04:28.934 GPON-8R2-West-Lake confd[8571]: confd commit progress db=running usid=13 thandle=34: run dependency-triggered validation...
<DEBUG> 7-Aug-2020::10:04:28.935 GPON-8R2-West-Lake confd[8571]: devel-c get_next request for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.936 GPON-8R2-West-Lake confd[8571]: devel-c get_next succeeded for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.940 GPON-8R2-West-Lake confd[8571]: devel-c get_elem request for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet{1/g1}/role
<DEBUG> 7-Aug-2020::10:04:28.942 GPON-8R2-West-Lake confd[8571]: devel-c get_elem succeeded for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet{1/g1}/role
<DEBUG> 7-Aug-2020::10:04:28.943 GPON-8R2-West-Lake confd[8571]: devel-c get_next request for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet{1/g1}/dtag-vlan
<DEBUG> 7-Aug-2020::10:04:28.945 GPON-8R2-West-Lake confd[8571]: devel-c get_next succeeded for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet{1/g1}/dtag-vlan
<DEBUG> 7-Aug-2020::10:04:28.946 GPON-8R2-West-Lake confd[8571]: devel-c get_next request for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.947 GPON-8R2-West-Lake confd[8571]: devel-c get_next succeeded for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.948 GPON-8R2-West-Lake confd[8571]: devel-c get_next request for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.950 GPON-8R2-West-Lake confd[8571]: devel-c get_next succeeded for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.951 GPON-8R2-West-Lake confd[8571]: devel-c get_next request for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.952 GPON-8R2-West-Lake confd[8571]: devel-c get_next succeeded for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.953 GPON-8R2-West-Lake confd[8571]: devel-c get_next request for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.955 GPON-8R2-West-Lake confd[8571]: devel-c get_next succeeded for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.956 GPON-8R2-West-Lake confd[8571]: devel-c get_next request for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.957 GPON-8R2-West-Lake confd[8571]: devel-c get_next succeeded for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.959 GPON-8R2-West-Lake confd[8571]: devel-c get_next request for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.960 GPON-8R2-West-Lake confd[8571]: devel-c get_next succeeded for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.961 GPON-8R2-West-Lake confd[8571]: devel-c get_next request for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.963 GPON-8R2-West-Lake confd[8571]: devel-c get_next succeeded for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.964 GPON-8R2-West-Lake confd[8571]: devel-c get_next request for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG> 7-Aug-2020::10:04:28.966 GPON-8R2-West-Lake confd[8571]: devel-c get_next succeeded for callpoint imxs_cli_config_cp path /exa:config/interface/ont-ethernet
<DEBUG

is the must count() statement wrong? I tested must “count(…/dtag-vlan) <= 12” replacing must “count(…/dtag-vlan) + count(.) <= 12” , there is no get_next /config/interface/ont-ethernet requests in the confd_developer.log.

Please help to take a look at it. thanks.

The must statement is wrong indeed, though I’m not sure that fixing it would also eliminate the unnecessary iteration over interfaces. But perhaps it is worth trying.

Unless I am mistaken, the expression count(.) evaluates always to 1 - number of “context nodes” is always 1 since there is always exactly one context node. What you might have in mind is count(../vlan) in the list vlan and count(../dtag-vlan) in the list dtag-vlan. Minor improvement may be achieved by observing that only one of the two must statements is necessary and the other one can be safely removed (without changing the meaning).

But there is another problem - the expression is evaluated for every instance of the lists, which is obviously not necessary (even though probably not that harmful since there are only 12 of them at most). To improve that you would have to move the must statement one level higher - which is unfortunately impossible since the higher level is the augment statement and you cannot place must under augment; and I don’t think it can be added through deviations.

But there may be other options:

  • You can drop the must statements and add a validation point to ont-ethernet through an annotation; the validation callback would simply count number of instances in both lists.

  • You can drop the must statements and make sure the lower layers (I understand the configuration is transformed or stored in an external DB) fail the transaction if there are too many instances of the two lists; this and the previous option have the drawback that the data model alone does not indicate the limit on the number of instances.

  • You may merge the two lists into one like this:

  augment "/nt:config/nt:interface/nt:ont-ethernet" {
    list vlan {
      when "../nt:role = 'uni' or ../nt:role = 'enni'" {
        tailf:dependency "../nt:role";
      }
      key "vlan-type vlan-id";
      max-elements "12";
      must "vlan-type = 'vlan' or ../nt:role = 'uni'";

      leaf vlan-type {
        type enumeration {
          enum "vlan";
          enum "dtag-vlan";
        }
      }
      leaf vlan-id {
        description
          "VlanId (range: 1-4094)";
        type uint32;
      }
    }
  }

The semantic is exactly the same and there is only one and much simpler must statement.

Hi @mvf, you are definitely right. We are using the external database provider. We also found the count(.) value always 1. We thought (.) means the current node, so we use count(.) to count the current list node instance. What’s your meaning of ‘context nodes’?
Yes, we can use validation callback in our EDP backend code to implement this semantic check.
But we prefer to address it in the yang model. It’s quite a common use case. There are many places in the yang model using this similar “must count” statement.
I’m interested about your comments of “move the must statement one level higher”, and there is compile error for “must under augment”. So is it achievable to “move the must statement one level higher” by other syntax, such as annotation?
The semantic logic is the sum of vlan list instances and dtag-vlan list instances of the requested ont-ethernet port should be no greater than 12, ie, "must “count(vlan) + count(dtag-vlan) <= 12” ".
It seems like the yang you recommended can’t meet our semantic logic, right?
Please help to take a look at it again.
Thanks.

XPath expressions are evaluated over data trees, and data trees have no list nodes, they have only list instance nodes. So . in your case means the list instance for which the must statement is being verified.

I don’t think it is possible to add the must statement to the parent container, not even using annotations or deviations. That’s why I suggest that you modify your data model. One option is what I described before - the single list is semantically completely equivalent to what you wanted, it allows you to create vlan instances and dtag-vlan instances, up to 12 of all of them. Another option, if you prefer to keep the two lists separated, is to wrap the two lists in one parent container where you can add the must expression.

But then again - when ConfD verifies constraints like must statements, it needs to access the data, and if the data is in an external database, it needs to query the database. If you have many constraints like this, be prepared that you will need to fine-tune the data provider, and possibly to look for performance measures like bulk data access methods.