After do in-service upgrade in confd, it seems the application which subscribes can’t receive any new notifications anymore. I try to subscribe the application again but still no notification.
Any idea why ?
At beginning I suspect it is because the notification is disabled. I enabled the netconf trace log in confd after the in-service upgrade completes.
I can see:
Since notification is shown in the response (last line), I think this should be enough to prove that the notification is still enabled, correct ?
Another weird behavior is if I restart confd or restart my service (application which subscribe to the confd notification), both of them will recover the notifications.
Restarting the application which recover the notification make sense to me since the ssh session is established again. But how restarting the confd can recover the notification ? (After restart confd, I didn’t see the ssh be established since I debug in my service)
I believe you need to describe your setup in more detail. E.g. what the stream is, your confd.conf settings etc.
After an in-service upgrade (i.e. you do not restart ConfD), the SSH connection will be left intact and you should not have to restart the NETCONF client that subscribe to NETCONF notifications or ConfD.
A simple way to check is to use the examples.confd/in_service_upgrade/simple example to verify.
Just add the notification capability to the confd.conf in the example:
Use the netconf-console tool to subscribe to for example the built-in NETCONF stream.
$ netconf-console --create-subscription=NETCONF
Build the new verision and perform the in-service upgrade. Note how we get NETCONF config-change and capability-change notifications as we perform the upgrade:
$ make pkg_v2
$ confd_cli -u admin -C
# # upgrade to v2
Initializing upgrade...
>>> System upgrade is starting.
>>> Sessions in configure mode must exit to operational mode.
>>> No configuration changes can be performed until upgrade has completed.
Init OK
>>> System upgrade in progress...
Performing upgrade...
Perform OK
Committing upgrade...
<?xml version="1.0" encoding="UTF-8"?>
<notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0">
<eventTime>2020-02-27T15:32:04.50569+00:00</eventTime>
<netconf-config-change xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-notifications">
<changed-by>
<username>admin</username>
<session-id>0</session-id>
<source-host>127.0.0.1</source-host>
</changed-by>
<datastore>running</datastore>
<edit>
<target xmlns:new="http://tail-f.com/ns/example/new">/must-have</target>
<operation>replace</operation>
</edit>
<edit>
<target xmlns:simple="http://tail-f.com/ns/example/simple">/simple:simple/simple:value2</target>
<operation>replace</operation>
</edit>
<edit>
<target xmlns:simple="http://tail-f.com/ns/example/simple">/simple:simple/simple:color</target>
<operation>replace</operation>
</edit>
</netconf-config-change>
</notification>
<?xml version="1.0" encoding="UTF-8"?>
<notification xmlns="urn:ietf:params:xml:ns:netconf:notification:1.0">
<eventTime>2020-02-27T15:32:04.51586+00:00</eventTime>
<netconf-capability-change xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-notifications">
<changed-by>
<server/>
</changed-by>
<added-capability>http://tail-f.com/ns/example/new?module=new&revision=2018-11-28</added-capability>
<modified-capability>urn:ietf:params:netconf:capability:yang-library:1.1?revision=2019-01-04&content-id=7e7e39d22951ad84362bc216fd6989db</modified-capability>
<modified-capability>urn:ietf:params:netconf:capability:yang-library:1.0?revision=2019-01-04&module-set-id=7e7e39d22951ad84362bc216fd6989db</modified-capability>
</netconf-capability-change>
</notification>
cdb_subscriber got UPGRADE_COMMITED - reloading schemas
TRACE Connected (maapi) to ConfD
TRACE MAAPI_LOAD_ALL_NS
Commit OK
TRACE MAAPI_LOAD_MNS_MAPS
TRACE MAAPI_LOAD_HASH_DB
>>> System upgrade has completed successfully.
Upgrade done. Now let’s commit something to verify that the NETCONF config-change notifications are still being sent by ConfD and received by the netconf-console client.
1.For the stream subscribed, there is no specific stream, just subscribe to all streams.
2.In confd,conf, just enable the notification
<notification>
<enabled>true</enabled>
<!--
Enable this to make the agent handle RPCs while sending
notifications.
-->
<interleave>
<enabled>false</enabled>
</interleave>
</notification>
There is one important point I forgot to mentioned. After the in-service upgrade, I have to close all existed sessions otherwise the service will keep getting the “capabilities changed error” from confd.
See this topic I posted before:
So I do is close all ssh sessions to the confd and reopen, then re subscribe. But even like this, after subscribe again, the application should be able to receive the notification, correct ?
I believe @per referred to that you needed to reconnect the NETCONF sessions. Not the SSH session.
Have you tried to not close the NETCONF session for the notifications?
Can you provide your confd.conf settings for the netconf notification streams you are referring to?
Can you provide your confd.conf settings for the netconf notification streams you are referring to?
In our confd.conf, the configurations I posted above is all we have. We don’t put tag which I believe is used for configure which stream to subscribe ? The stream is optional, correct ? If no configurations, means subscribe to all ?
Have you tried to not close the NETCONF session for the notifications?
No, I have not. But I think desire to have a try. But I think don’t close the notification NEtCONF session will also cause the capabilities error,.
Anyother question is NETCONF session is one layer up of the ssh session ?
No other configuration means that you are only exposing the built-in NETCONF base notifications stream and your application provides no additional streams. See examples.confd/netconf_notifications/confd.conf and look for the <notifications><eventStreams> tags for an example of providing additional streams.
The NETCONF notifications keep being sent to the subscribers as you can see from the example I posted above.
Yes. Here: NETCONF ↔ SSH ↔ TCP ↔ IP ↔ Link Layer (E.g. Ethernet)
What’s the ouput of netconf-console --get -x /ncm:netconf-state/streams and netconf-console --get -x /ncm:netconf-state/sessions before and after the upgrade?
Thanks for detailed information. I am busy in some other stuffs and will come back asap.
One more question, from Confd perspective, after in-service upgrade, is closing all Netconf session mandatory if there are schema changes during in-service upgrade ? If not, how to avoid the capabilities change error ?
You want a new capabilities exchange to happen through a message so that the NETCONF client can. for example, find out what YANG model(s) changed.
The client can then, for example, build a newly-updated representation of the device config-data/oper-data/actions/notifications, etc.
The capabilities are exchanged at the start of the session, so the NETCONF session needs to be restarted if the capabilities change.
I just used one of the examples that come with the ConfD example set, so, yes, there is a schema / YANG model change. It would be a pretty bad example of an in-service upgrade if it didn’t . Again, see examples.confd/in_service_upgrade/simple
It is obvious that the subscriber gone after upgrade, although the application try to subscribe again. It looks like something happend that the subscribe failed after upgrade.
So I think I will firstly to try to restart Netconf session. After restarting the Netconf session, the client needs to subscribe again in order to receive the notification, correct ?
I doubt that there is a NETCONF session to restart. Your client probably disconnected. ConfD didn’t close it if the in-service upgrade was done properly. And yes, if you close a NETCONF session and reconnect, you need to subscribe again.
OK, finally I got a workaround and this issue seems to be more weird.
After in service upgrade, since I closed all Netconf session, I need to reconnect it and subscribe again.
I am using some APIs in JNC package to communicate with confd thru netconf.
the code is like below:
I post the JNC APIs I used above here just in case you are interested.
The workaround is I have to delay for seconds between open the Netconf Session and subscribe in order to subscribe sucessfully. The delay duration may not be constant, sometimes 2 seconds is enough while sometimes needs 5 seconds.
There are two points I need to point out
The delay is NOT needed when do subscribe before any in-service upgrade.
After in service upgrade, if subscribe without delay, the subscribe response from confd shows ‘OK’ which means succeeded. This is verified by JNC APIs NetconfSession.createSubscription().
If failed, there will be exception. I also double check through enable the netconf trace. But the subscription actually failed since there is no subscriber appeared if I check the xpath using netconf-console or in confd_cli.
So in this case, I strongly suspect this is a confd bug. It doesn’t make any sense that subscribe needs seconds level delay after the Netconf session is connected. Even worse, the confd responded with OK which confused the client.
Let me know what you think. If you think I should open a ticket with Tailf support, let me know.
I doubt that the issue is on the ConfD side. From what you wrote earlier, you have a NETCONF session that does more than just subscribing, which is why you had to enable the :interleave NETCONF capability. Seems more likely that there is something in JNC related to that. I would look at that.
Below is how you can run a similar scenario using ConfD and the netconf-console Python script as a client with the examples.confd/in_service_upgrade/simple example. Can you run something similar like the below with your JNC client?:
$ pwd
/Users/tailf/confd-7.3.1/examples.confd/in_service_upgrade/simple
$ sed -i -- 's/\.\/cdb_subscriber/\.\/cdb_subscriber \&/g' Makefile
$ sed -i -- 's/<\/netconf>/<notification><enabled>true<\/enabled><\/notification><\/netconf>/g' confd.conf
$ cat test.sh
#!/bin/bash
make stop clean all start &> /dev/null
ecode=1; while [ $ecode -ne 0 ]; do sleep .5; confd_cmd -o -c "mget /tfcm:confd-state/tfcm:internal/tfcm:cdb/tfcm:client{1}/tfcm:name" > /dev/null; ecode=$?; done;
netconf-console --create-subscription=NETCONF &
netconf-console --get-config -x '/simple'
netconf-console --db=candidate --edit-config=-<<<'<simple xmlns="http://tail-f.com/ns/example/simple"><color>black</color><value1>0</value1><value2>0</value2><value3>0</value3><int xmlns="http://tail-f.com/ns/example/new">0</int></simple>'
netconf-console --commit
netconf-console --get-config -x '/simple'
make pkg_v2
confd_cli -u admin -C << EOF
config
simple color black
commit
exit
upgrade to v2
exit
EOF
netconf-console --get-config -x '/simple'
netconf-console --db=candidate --edit-config=-<<<'<simple xmlns="http://tail-f.com/ns/example/simple"><color>purple</color><value1>5</value1><value2>30000</value2><value3>-5</value3><int xmlns="http://tail-f.com/ns/example/new">-42</int></simple>'
netconf-console --commit
netconf-console --get-config -x '/simple'
I plan to try the example you mentioned. The confd I am using is 6.6.4. But I can’t find the examples.confd directory in the confd installed path.
So where can I find the example you suggest me to run ?
Thanks @cohult. I got the example and took a look, In this example, the yang file which is in-service upgraded is very tiny. I don’t think the notification subscribed works with this example will be enough to prove the problem we saw is not a confd issue.
I think the proper way to go maybe is do the in service upgrade with our fxs using confd API instead of java (maybe C or python). Then do the notification subscribe just after the in service upgrade to see whether it works or not (basic idea is reproduce what we do in our backend service) with another small program.
Do we have python example to do in-service upgrade ? I checked the confd example directory and see there is a python maapi example but it doesn’t have the in-service upgrade example in it.
Another question is in your example, how do you subscribe to the notification after in-service upgrade ?
the in-service upgrade examples are C only examples. Note that the Python API is built on top of the C API. So using the Python API instead of the C API makes no difference.
As you can read in the “test.sh” script, I don’t. The NETCONF session is dedicated to subscribing to NETCONF notifications and hence does not need to reconnect after an in-service upgrade.
Ok, I will modify the upgrade.c to try do in-service uprade with my fxs files.
In my case, as I explained before since the capabilities changed since I have to subscribe to the notification after the in-service is done. So in this case, the easiest way is using netconf-console to sent subscribe request ?
netconf-console is just a Python based NETCONF tool that comes with ConfD.
What you want to focus on is the fact that if you dedicate a NETCONF session to subscribing to the NETCONF notification stream, i.e. no “interleaving” of other NETCONF operations on that session, then you do not have to restart that session after an in-service upgrade.
We discussed this previously in this topic. I mentioned if I don’t restart session after in-service upgrade, I will keep get the capabilities change error. And the below is your response:
You want a new capabilities exchange to happen through a message so that the NETCONF client can. for example, find out what YANG model(s) changed.
The client can then, for example, build a newly-updated representation of the device config-data/oper-data/actions/notifications, etc.
The capabilities are exchanged at the start of the session, so the NETCONF session needs to be restarted if the capabilities change.
You can go back to check our previous discussion about this in this topic to confirm.
What I plan to do is I will modify the upgrade.c in order to do the in-service upgrade with my own fxs, then resubscribe the notification immediately after the in-service upgrade to see whether the subscribe will succeed or not. If I saw the same issue as I saw in my backend service, it should prove this is a confd issue.
. Again, see examples.confd/in_service_upgrade/simple