Can't receive any new notifications from confd after do in-service upgrade

buddha87 · April 13, 2020, 1:54am

Thanks @cohult. I got the example and took a look, In this example, the yang file which is in-service upgraded is very tiny. I don’t think the notification subscribed works with this example will be enough to prove the problem we saw is not a confd issue.

I think the proper way to go maybe is do the in service upgrade with our fxs using confd API instead of java (maybe C or python). Then do the notification subscribe just after the in service upgrade to see whether it works or not (basic idea is reproduce what we do in our backend service) with another small program.

Do we have python example to do in-service upgrade ? I checked the confd example directory and see there is a python maapi example but it doesn’t have the in-service upgrade example in it.

Another question is in your example, how do you subscribe to the notification after in-service upgrade ?

cohult · April 13, 2020, 5:59am

the in-service upgrade examples are C only examples. Note that the Python API is built on top of the C API. So using the Python API instead of the C API makes no difference.

As you can read in the “test.sh” script, I don’t. The NETCONF session is dedicated to subscribing to NETCONF notifications and hence does not need to reconnect after an in-service upgrade.

buddha87 · April 13, 2020, 8:33pm

Ok, I will modify the upgrade.c to try do in-service uprade with my fxs files.

In my case, as I explained before since the capabilities changed since I have to subscribe to the notification after the in-service is done. So in this case, the easiest way is using netconf-console to sent subscribe request ?

cohult · April 14, 2020, 9:18am

netconf-console is just a Python based NETCONF tool that comes with ConfD.

What you want to focus on is the fact that if you dedicate a NETCONF session to subscribing to the NETCONF notification stream, i.e. no “interleaving” of other NETCONF operations on that session, then you do not have to restart that session after an in-service upgrade.

buddha87 · April 15, 2020, 2:17am

Why not have to restart ？

We discussed this previously in this topic. I mentioned if I don’t restart session after in-service upgrade, I will keep get the capabilities change error. And the below is your response:

You want a new capabilities exchange to happen through a message so that the NETCONF client can. for example, find out what YANG model(s) changed.
The client can then, for example, build a newly-updated representation of the device config-data/oper-data/actions/notifications, etc.
The capabilities are exchanged at the start of the session, so the NETCONF session needs to be restarted if the capabilities change.

You can go back to check our previous discussion about this in this topic to confirm.

What I plan to do is I will modify the upgrade.c in order to do the in-service upgrade with my own fxs, then resubscribe the notification immediately after the in-service upgrade to see whether the subscribe will succeed or not. If I saw the same issue as I saw in my backend service, it should prove this is a confd issue.

cohult · April 15, 2020, 3:35pm

Because the NETCONF session for subscribing to the NETCONF notification stream is dedicated to notifications only, and hence does not have to be restarted after a capabilities change due to an in-service upgrade.

buddha87 · April 17, 2020, 8:24pm

Ok, this make senses.

In my case, the client opened several netconf sessions before the in-service upgrade happen. Now what I plan to do is:
After in-service upgrade is completed, close all netconf sessions except the NETCONF session which subscribing to the NETCONF notification stream (so I will leave this NETCONF session open, no touch in this session after in-service upgrade).

Then I check the netconf request which close the netconf session, it will looks like:

<?xml version="1.0" encoding="UTF-8"?> 
<rpc message-id="101" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> 
  <close-session/> 
</rpc>

In this request, there is no field to specify which session to close, then how does confd server to tell which session to close ?
Confd allows multiple netconf sessions opening simultaneously, correct ?

cohult · April 18, 2020, 11:10am

I need to take the above comment back and clarify after revisiting NETCONF over SSH RFC 6242. Closing the NETCONF session will close the SSH session.
https://tools.ietf.org/html/rfc6242#section-5

Exiting NETCONF is accomplished using the operation.
A NETCONF server will process NETCONF messages from the NETCONF
client in the order in which they are received. When the NETCONF
server processes a operation, the NETCONF server
SHALL respond and close the SSH session channel. The NETCONF server
MUST NOT process any NETCONF messages received after the
operation.

No, one NETCONF session per SSH session. A.k.a. “A NETCONF over SSH session”
https://tools.ietf.org/html/rfc6242#section-4

From the NETCONF event notifications RFC 5277:
https://tools.ietf.org/html/rfc5277#section-6

The :interleave capability indicates that the NETCONF peer supports
the ability to interleave other NETCONF operations within a
notification subscription. This means the NETCONF server MUST
receive, process, and respond to NETCONF requests on a session with
an active notification subscription. This capability helps
scalability by reducing the total number of NETCONF sessions required
by a given operator or management application.

So, again, if you don’t want to close the NETCONF over SSH session that is used for notifications after an in-service upgrade, don’t interleave other NETCONF operations on that NETCONF session. Dedicate it to NETCONF notifications.

per · April 18, 2020, 1:14pm

Hm, you’re replying to (but not quoting) a post where you wrote:

However I can’t find the post where I said that - do you have a link? As it stands, it is clearly nonsensical, since (as you point out now) NETCONF sessions and SSH sessions are one-to-one - while you can have any number of NETCONF or SSH sessions in one SSH connection. And I actually tend to avoid talking about “SSH sessions”, since I think it is easily confused with “SSH connections”, and prefer to use “SSH channels” (every SSH session needs a channel - although not all channels are sessions…).

buddha87 · April 20, 2020, 2:14am

@per
This is the topic I mentioned before.

@cohult @per
One Netconf session to one ssh session. This fully makes sense.
The JNC package I am using, is implementing this logic:

github.com

tail-f-systems/JNC/blob/master/jnc/src/com/tailf/jnc/Device.java#L406


}


/**
 * Creates a new named NETCONF session We must not have an existing session
 * with the same name. Thus when we reconnect a Device, we must first call
 * closeSession(sessionName);
 * 
 * @param sub IO subscriber for trace messages.
 * @param sessionName symbolic Name of the session
 */
public void newSession(IOSubscriber sub, String sessionName)
        throws JNCException, IOException {
    final SessionConnData data = getConnData(sessionName);
    // always create the configTree
    newSessionConfigTree(sessionName);


    // Check that we don't already have a session with that name
    if (data != null) {
        int errCode = YangException.BAD_SESSION_NAME;
        throw new YangException(errCode, sessionName);
    }

Now my questions is:
For one ssh connection (or even lower level, one socket connection between confd client and confd server), does it allow multiple netconf sessions living simultaneously?
Or when I open a new netconf session, the previous opened netconf session must be closed before the new netconfd session open ?

Based on my understanding about how socket it works, it is feasible to support multiple session simultaneously but may not deserve to do that.

Now I am really confused by whether one socket connection supporting multiple netconf session.

cohult · April 20, 2020, 7:00am

I’ve read the JNC help text and it is very clear. Just read the help text for these methods in this order:

github.com

tail-f-systems/JNC/blob/master/jnc/src/com/tailf/jnc/Device.java#L343


public SSHSession getSSHSession(String sessionName) {
    final SessionConnData data = getConnData(sessionName);
    return (data == null) ? null : data.sshSession;
}


/**
 * This method finds the {@link DeviceUser} associated to the localUser
 * user name and SSH connects to the device, this method must be called
 * prior to establishing any sessions (channels)
 */
public void connect(String localUser) throws IOException, JNCException {
    connect(localUser, 0);
}


/**
 * This connect() method has an additional timeout paramater. This is not
 * the same thing as the readTimeout. The connectTimeout only applies to
 * the actual connect.
 * 
 * @param localUser The name of a local (for the EMS) user
 */

github.com

tail-f-systems/JNC/blob/master/jnc/src/com/tailf/jnc/Device.java#L393


        trees.add(new SessionTree(sessionName));
    }
}


/**
 * Creates a new named NETCONF session. Each named session corresponds to
 * one SSH channel towards the agent
 * 
 * @param sessionName symbolic Name of the session
 */
public void newSession(String sessionName) throws JNCException,
        IOException, YangException {
    newSession(null, sessionName);
}


/**
 * Creates a new named NETCONF session We must not have an existing session
 * with the same name. Thus when we reconnect a Device, we must first call
 * closeSession(sessionName);
 * 
 * @param sub IO subscriber for trace messages.

github.com

tail-f-systems/JNC/blob/master/jnc/src/com/tailf/jnc/Device.java#L268


}


/**
 * Close the named session associated with this device. Note, this does not
 * close the socket, it closes the SSH channel which is a considerably
 * cheaper operation. The method also clears any accumulated configuration
 * data we have.
 * 
 * @param sessionName symbolic Name of the session
 */
public void closeSession(String sessionName) {
    final SessionConnData data = removeConnData(sessionName);
    if (data.session != null) {
        try {
            // data.session.closeSession();
            data.sshSession.close();
        } catch (final Exception e) {
        }
    }
    clearConfig(sessionName);
}

github.com

tail-f-systems/JNC/blob/master/jnc/src/com/tailf/jnc/Device.java#L284


        } catch (final Exception e) {
        }
    }
    clearConfig(sessionName);
}


/**
 * end all NETCONF sessions and close the SSH socket associated to this
 * device It also clears all accumulated config trees.
 */
public void close() {
    for (int i = 0; i < connSessions.size(); i++) {
        final SessionConnData d = connSessions.get(i);
        try {
            d.session.closeSession();
        } catch (final Exception e) {
        }
    }
    connSessions = new ArrayList<SessionConnData>();
    for (int i = 0; i < trees.size(); i++) {
        final SessionTree t = trees.get(i);

If you still don’t understand what is going on you can get an overview with links to the details here:

per · April 20, 2020, 8:10am

OK, but I don’t find any post in that thread either where I say that “you needed to reconnect the NETCONF sessions. Not the SSH session.”

Yes.

It definitely works from the protocol point of view, but it may not make sense for a very simple client to make use of it.

buddha87 · April 20, 2020, 2:09pm

@cohult
Thanks for the detailed guide.

I did read the comments of these JNC methods when I implemented the design to open/close the netconf session using this JNC device class.

Now what really confused me is I have updated my codes to close the netconf session except the netconf session used for notification. But I am surprised to find even I updated the logic as I mentioned above, the netconf session which subscribe to notification is still closed (I am 100% sure I am using the method device.closeSession(String sessionName), not device.close()).

I suspected there maybe something wrong in JNC methods, but I have take a glace of the inner implementation of closeSession, it seems make sense.

So this is why I asked the question previously:

Then I check the netconf request which close the netconf session, it will looks like:

<?xml version="1.0" encoding="UTF-8"?> 
<rpc message-id="101" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> 
  <close-session/> 
</rpc>

In this request, there is no field to specify which session to close, then how does confd server to tell which session to close ?
Now let’s come back to this question, does the confd rely on the message-id to determine which netconf session to close if there are multiple netconf session living simultaneously?

@cohult @per
Please help me the above question. Thanks.

cohult · April 20, 2020, 3:20pm

One NETCONF session per SSH channel.

If you still don’t understand what is going on you can get an overview with links to the details here:
https://en.wikipedia.org/wiki/Secure_Shell
https://en.wikipedia.org/wiki/NETCONF

buddha87 · April 20, 2020, 3:25pm

The above two links doesn’t help the question I asked about the how the confd determine which netconf session to close if there are multiple netconf session.

In this netconf wikipedia page:

The only thing about close-session is:
<close-session> Request graceful termination of a NETCONF session

This doesn’t help.

cohult · April 20, 2020, 3:28pm

As it says in the JNC code, one NETCONF session per SSH channel.

github.com

tail-f-systems/JNC/blob/master/jnc/src/com/tailf/jnc/Device.java#L388


 * in a list of configTrees, thus we can have named configTrees although
 * there is no existing session towards the device.
 */
public void newSessionConfigTree(String sessionName) {
    if (getTreeData(sessionName) == null) {
        trees.add(new SessionTree(sessionName));
    }
}


/**
 * Creates a new named NETCONF session. Each named session corresponds to
 * one SSH channel towards the agent
 * 
 * @param sessionName symbolic Name of the session
 */
public void newSession(String sessionName) throws JNCException,
        IOException, YangException {
    newSession(null, sessionName);
}


/**

buddha87 · April 20, 2020, 3:31pm

One NETCONF session per SSH channel.

Does one socket connection to confd service support multiple SSH channels simultaneously?
If it does, then I can open multiple ssh channels to the same confd server while each ssh channel carry one netconf session.

Using device.newSession() can open multiple sessions (each one corresponding to one ssh channel) to the same confd server.

I asked this in previous post in this topic, and I got the reply from @per. Please check this post.

I got the answer is Yes

cohult · April 20, 2020, 3:34pm

If you look at the help text for the connect(), newSession(), closeSession(), and close() JNC methods that I pointed to earlier, what do you think?

Great, are we done now?

per · April 20, 2020, 3:50pm

See https://tools.ietf.org/html/rfc6241#section-7.8 and https://tools.ietf.org/html/rfc6241#section-7.9 - i.e. <close-session> closes the session it is issued in (it is used in pretty much all the ConfD examples), <kill-session> closes the session identified by the <session-id> parameter.

buddha87 · April 20, 2020, 3:53pm

Ok.
For the question:
Does one socket connection to confd service support multiple SSH channels simultaneously?
I got the answer.

For the question:
In this request, there is no field to specify which session to close, then how does confd server to tell which session to close ?

No, I don’t get the answer.

I think my question is clear enough but I don’t know why I can’t get the clear and straightforward answer for it.
For this question, I don’t really care one netconf session to one ssh channel.
We have confirmed multiple ssh channels to one confd server is supported. Then it means there will be multiple sessions to one confd server.
Then if I just want to close a specific session, how to tell confd thru netconf request ?

To answer this question, the first thing I need to know is supported or not supported. so as simple as YES or NO.
If it is YES, then the second step is How.

Ok, after I completed this post then I saw the latest reponse from @per.

Thanks for the reference. I think this is the answer I need. Let me check that and see what I should do.

Can't receive any new notifications from confd after do in-service upgrade

Our Blog

Resources

Forum

Contact Us