Hi,
i have a scenario, where my CDB has 1 lakh or more entries, Now when i try to issue a command to display i encounters process hang issue.
I am looking kind of scenario where when i issue a command, show xyz 50, it could retrieve 50 records set, may be when i press enter or tab or spacebar it can go and retrive the next 50. Do we have any key word or option in confd for this ?
Generally, in the CLI when one issues show command, it will display items to fill in screen + --More-- word.
Then CLI waits till user presses SPACEBAR as you suggest. (To prevent this behavior, one needs to use show ... | nomore - then all records are displayed in the CLI.)
Please, can you describe your scenario in more details? What means 1 lakh or more entries? What kind of data are shown (is it configuration or operational data)? Do you get any rows displayed at all?
Do you use any callbacks (e…g for operational data)?
The scenario is i have a CDB with operational data about the users. Now the worst case scenario is having 1million users. Lets assume i have 1 milllion users now.
When i wanted to issue show users, confd will run and get me the list of 1 million users, by doing so the thread is hoaking cpu.
Now instead if there is an option in confd, then i can do show users count, then based on the count confd can run and get data , thereby my process cpu may not hoak. Only when i press spacebar then i collect the second set of count elements.
Currently i am using callbacks to get the data. But i dont want to change my application code and load the code further by doing validation on what data has been send to display and what not, instread i am looking for some way in confd helping me on this .
Thanks for clarification. Still I would like to know some more details:
do you use operational data stored in the CDB or not? (tailf:cdb-oper)
you mention that you use callbacks - do you mean data provider callbacks? (If yes, then your are probably not using tailf:cdb-oper - see prev. queston.)
the behavior you request (collect next set of data) is implemented by ConfD CLI by default (called pagination), so it confuses me why it does not work for you. During pagination, only calls to callbacks to fetch data filling the screen is called. Can you try what happens with command show ... | more (e.g. show users | more).
if you use data provider callbacks, can you find out how many times they are called? (search for get_next, get_elem, …) calls can be traced in the file devel.log (should be found in the current dir of your application) or in your application log (if you have any)
Thanks.
We are using external database. There were 996 vlan services provisioned under a single ethernet interface 1/1/x4, and 1008 match-vlan under vlan. In the beginning, there were only get_elem and get_next callbacks supported, the elapsed time for netconf get-config /interfaces/interface[1/1/x4] which is a common standard interface ethernet retrieving request, was 59s. and then we implemented get_object and get_next_object, but the elapsed time was reduced little to 52s with only 11% improved.
I looked back at the yang model and confd_devel.log closely, and found there are lots of nested-list nodes, and nested-list nodes can’t be retrieved with other nodes in one get_object/get_next_object request, because confd will conplain it like " 11-Apr-2020::11:38:26.143 Jessica-is-using confd[6493]: devel-c bad get_object() return value: /if:interfaces/interface{1/1/x4}/ethernet/vlan: Nested lists or leaf-lists not allowed" .
and confd will trigger get_next_object for nested-list separated like below log:
So, what’s your suggestion to address such lots of nest-list nodes performance? Whether can confd support segment delivery, so the socket between Northbound user with confd will not get timeout? Thanks.
Regarding NETCONF, you can try The Query API (See ConfD User Guide - The NETCONF server), which allows fetching results in chunks.
Regarding performance - you may try to implement get_object and get_next_object (returning multiple elements) also for nested list.
Regarding guidance on the kind of high performance data provider setup you want to target and the callbacks you want to focus on, see the data provider performance demo:
Thank @cohult and @mnovak,
I have implemented get_object and get_next_object, but the test result is the improvement is not obvious and the elapsed time is reduced from 56s to 49s by 12% improved, which is far from what we expected. I’m writing a tail-f ticket to ask for help to discuss how to improvement the performance of get_object and get_next_object or something was used wrongly by me.
Regarding Query API, I read it today but I tried to run the example in the Confd User Guide and got the below error. Can you help to confirm what’s the issue? Does my confd version 6.7 not support it?
===start-query
<rpc message-id="m-51" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> <start-query xmlns="http://tail-f.com/ns/netconf/query"> <foreach>/x/host[enabled = 'true'] </foreach> <select> <label>Host name</label> <expression>name</expression> <result-type>string</result-type> </select> <select> <expression>address</expression> <result-type>string</result-type> </select> <sort-by>name</sort-by> <limit>100</limit> <offset>1</offset> </start-query></rpc>]]>]]>
===error reply
<?xml version="1.0" encoding="UTF-8"?> <rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="m-51"><rpc-error> <error-type>protocol</error-type> <error-tag>operation-not-supported</error-tag> <error-severity>error</error-severity> <error-message xml:lang="en">**Unsupported capability tailf-netconf-query</error-message><error-info><bad-element>start-query**</bad-element> </error-info> </rpc-error> </rpc-reply>]]>]]>
Thanks.
Query API is mentioned in ConfD User Guide of 6.7., so it should be supported.
Did you check it is enabled in confd.conf? (it is not by default)
/confdConfig/netconf/capabilities/query/enabled (boolean) [false]
enabled is either “true” or “false”. If “true”, the http://tail-f.com/ns/netconf/query capability is enabled.
Thank @mnovak, you are right. It’s not enabled in our confd.conf. After it is enabled, Query API is supported now. But the query result is not what I expect. Sounds like the and are not set correctly. But I don’t know how to fill out these two nodes. Please help me. Thanks.
The snippet of yang data model is as below:
==ietf-interfaces.yang
container interfaces {
list interface {
key "name";
leaf name {
type string;
}
leaf description {
type string;
}
leaf type {
type identityref {
base interface-type;
}
mandatory true;
description
"The type of the interface.
}
leaf enabled {
type boolean;
default "true";
}
leaf link-up-down-trap-enable {
if-feature if-mib;
type enumeration {
enum "enabled" {
value 1;
}
enum "disabled" {
value 2;
}
}
} // list interface
} // container interfaces
===ethernet-std.yang
augment /if:interfaces/if:interface {
// Ethernet Interface Config Object List
when "if:type = 'eth-std:ethernetCsmacd'";
container ethernet{
leaf mtu {
type exa:IF_MTU;
description "Ethernet port MTU";
tailf:info "Ethernet port MTU";
default "2000";
}
...
...
...
} // ethernet
}
augment /if:interfaces/if:interface/eth-std:ethernet {
list vlan {
key "vlan-id";
leaf vlan-id{
description "VlanId (range: 1-4094)";
type string;
}
list policy-map {
...
...
...
}
list c-vlan{
...
...
...
}
list match-vlan{
...
...
...
}
...
...
...
}
}
Now there are 996 vlan instances under /interfaces/interface [name=‘1/1/x4’]/ethernet,
and lots of match-vlan, polic-map under /interfaces/interface [name=‘1/1/x4’]/ethernet/vlan,
But when I send Query API , the confd only return the top leaf nodes and then return “no more data”. Please see below quest and response.
==start-query
<rpc message-id="m-51" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
<start-query xmlns="http://tail-f.com/ns/netconf/query">
<foreach>/interfaces/interface[name='1/1/x4']</foreach>
<select><expression>/interfaces/interface[name='1/1/x4']</expression><result-type>inline</result-type></select>
<limit>100</limit>
<offset>1</offset>
</start-query>
</rpc>
]]>]]>
<?xml version="1.0" encoding="UTF-8"?>
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="m-51"><query-handle xmlns="http://tail-f.com/ns/netconf/query">4</query-handle>
</rpc-reply>]]>]]>
==fetch-query-result only gets 3 leaf nodes
> <rpc message-id="m-51" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> > <fetch-query-result xmlns="http://tail-f.com/ns/netconf/query"> <query-handle>4</query-handle> </fetch-query-result></rpc>]]>]]>
> <?xml version="1.0" encoding="UTF-8"?> > <rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="m-51"><query-result xmlns="http://tail-f.com/ns/netconf/query"> > <result> > <select> > <data> > <name xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">1/1/x4</name> > <type xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces" xmlns:eth-std="http://www.calix.com/ns/ethernet-std">eth-std:ethernetCsmacd</type> > <link-up-down-trap-enable xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">disabled</link-up-down-trap-enable></data> > </select> > </result> > </query-result> > </rpc-reply>]]>]]>
==fetch-query-result again and only confd returns no-more-data.
<rpc message-id="m-51" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> <fetch-query-result xmlns="http://tail-f.com/ns/netconf/query"> <query-handle>4</query-handle> </fetch-query-result></rpc>]]>]]> <?xml version="1.0" encoding="UTF-8"?> <rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="m-51"><query-result xmlns="http://tail-f.com/ns/netconf/query"> <result> <no-more-data/> </result> </query-result> </rpc-reply>]]>]]>
Please give me some directions to solve this issue. Much appreciated.
Not sure what is the problem. inline result type should return subtree.
What if you try query directly for vlan
<foreach>/interfaces/interface[name='1/1/x4']/vlan</foreach>
Would it return vlan instances. Maybe devel.log and audit.log can provide some hint.
I meant <start-query>, to see what is returned if you specify directly vlan sublist. My expectation is you should get all vlans for /interfaces/interface[name='1/1/x4']. Steps should be same like you wrote, but foreach would be
<foreach>/interfaces/interface[name='1/1/x4']/vlan</foreach>
While the Query API is very versatile and powerful, I think a lot can be achieved with plain XPath filters; for instance your query can be reformulated like
<nc:rpc xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1">
<nc:get>
<nc:filter type="xpath" select="/interfaces/interface[name='1/1/x4']/ethernet/vlan[position() < 100]"/>
</nc:get>
</nc:rpc>
This means “return me the configuration of first 99 vlan instances of said interface” (< is a XML entity that stands for <, similarly > for >).
Thanks @mnovak and @mvf. Sorry to give the feedback so late. I’m tied with other block issues. Hi @mnovak, whatever the foreach is, it seems like when using result-type inline, confd always triggers the first instance request to our EDP and returns the first instance. Since I’m tied with other block issues, and Query API is not high priority, I leave it now and I’ll investigate it further or submit a Tail-f ticket someday if needed.
Hi @mvf, the filter with limit instance does work for us. but how to get the next 99 vlan? what’s the correct syntax? I used
[position() > 99 < 200]
and it did not work. confd retrieved all the instances from 99th.
Thank you again.