Hi,
Unless you can show this community exactly how to reproduce that error (I sent that same notification without any issues), we can’t help other than recommend that you double check your code, the ConfD documentation and examples.
Thanks for your help so far.
we are reviewing the entire code and the configuration, compare it to the examples and guidelines to see what is missing/wrong.
I have few questions regarding diff that we found, that might give us a good hint…
- the registration code is as follows:
struct confd_notification_stream_cbs NotificationCallBack;
memset(&NotificationCallBack, 0, sizeof(NotificationCallBack));
NotificationCallBack.fd = m_WorkerSocket;
NotificationCallBack.get_log_times = NULL;
NotificationCallBack.replay = NULL;
strcpy(NotificationCallBack.streamname, “NETCONF”);
NotificationCallBack.cb_opaque = NULL;
if (confd_register_notification_stream(m_DeamonContext, &NotificationCallBack, &m_LiveContext) != CONFD_OK)
{
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::Init confd_register_notification_stream FAILED " );
}
I saw example that the reply wasn’t NULL. as much as I understood from the guideline it is not a callback we need. Can you tell if that can cause such error?
- I see the following prints :
6-Jun-2017::12:08:58.701 22860/b30ffb40/23 SEND op=15 isrel=0 th=-1 {170,‘NETCONF’,undefined,{19,{2017,6,6,12,8,58,701083,0,0}},2,{hxml,[{[1244620176|2103098691],start},{[1244620176|737643097],start},{870658415,#Bin},{372565948,{12,0}},{1399987302,{160,160,160,160}},{[1244620176|737643097],stop},{449538578,{28,0}},{[1244620176|1716487530],start},{1914145912,{34,[1033149831,13885182,[1693656965|555337807]]}},{[1244620176|1716487530],stop},{[1244620176|2103098691],stop}]}}
Does it mean that the message was received OK in the confd code? the path is not printed as text but as numbers. Is it OK?
Thanks again for your guideline.
Inbal
Does it mean that the message was received OK in the confd code?
No, the debug output from libconfd just means that libconfd successfully sent the message. If the message was corrupt when ConfD received it, you may get the “API request” error that you see. Perhaps you should check that using for example wireshark or tcpdump comparing the packets sent by libconfd to the ones received by ConfD.
Looking at your past posts, you seem to have had issues with your socket communication, perhaps that’s something you should look into?
I guess that’s what you suspect too, and therefore you cannot provide an example to enable us to reproduce the issue?
the path is not printed as text but as numbers. Is it OK?
It is printed in binary format, which is what libconfd converts it to, so that’s ok.
Hi,
Is there a way to see print outs from the confD itself?
I am not sure what to suspect… the socket is used for the init registration in addition for the receive callbacks so I guess it is OK, but it worth to double check.
Please see below the init an registration code, and the send notification code, to complete the picture. maybe you can see something that we miss here.
thanks again
Inbal
Init code:
/* Initialize confd daemon context */
confd_init("ConfdAdaptor", stderr , CONFD_PROTO_TRACE);
m_DeamonContext = confd_init_daemon("ConfdAdaptor");
if (m_DeamonContext == NULL)
{
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::Init confd_init_daemon(confdAdaptor) returned NULL " );
printf("ConfDAdaptorManager::Init (confd_init_daemon) F A I L E D ! ! !\n");
return;
}
confd_set_daemon_flags(m_DeamonContext, CONFD_DAEMON_FLAG_STRINGSONLY); //to send and recieve string only
/* Initialize sockets to be used by confd */
struct sockaddr_in socketAddress;
socketAddress.sin_addr.s_addr = inet_addr("127.0.0.1");
socketAddress.sin_family = AF_INET;
socketAddress.sin_port = htons(CONFD_PORT);
m_ControlSocket = socket(PF_INET, SOCK_STREAM, 0);
if (m_ControlSocket < 0)
{
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::Init m_ControlSocket < 0 " );
}
m_WorkerSocket = socket(PF_INET, SOCK_STREAM, 0);
if (m_WorkerSocket < 0)
{
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::Init m_WorkerSocket < 0 " );
}
confd_load_schemas((struct sockaddr*)&socketAddress, sizeof (struct sockaddr_in)); //for debug
if (confd_connect(m_DeamonContext, m_ControlSocket, CONTROL_SOCKET, (struct sockaddr*)&socketAddress, sizeof (struct sockaddr_in)) < 0)
{
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::Init confd_connect(m_ControlSocket) < 0 " );
}
if (confd_connect(m_DeamonContext, m_WorkerSocket, WORKER_SOCKET,(struct sockaddr*)&socketAddress,sizeof (struct sockaddr_in))< 0)
{
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::Init confd_connect(m_WorkerSocket) < 0 " );
}
//registration to Confd, inside:
//confd_register_trans_cb(m_DeamonP, &TransactionCallbacks);
//confd_register_data_cb(a_pDeamon, &data);
m_pConfDAdaptorCallBackClient->RegisterYourself(m_DeamonContext);
m_pConfDAdaptorObjFactory->RegisterList(m_DeamonContext);
//register the stream for notifications to confd
struct confd_notification_stream_cbs NotificationCallBack;
memset(&NotificationCallBack, 0, sizeof(NotificationCallBack));
NotificationCallBack.fd = m_WorkerSocket;
NotificationCallBack.get_log_times = NULL;
NotificationCallBack.replay = NULL;
strcpy(NotificationCallBack.streamname, "NETCONF");
NotificationCallBack.cb_opaque = NULL;
if (confd_register_notification_stream(m_DeamonContext, &NotificationCallBack, &m_LiveContext) != CONFD_OK)
{
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::Init confd_register_notification_stream FAILED " );
}
confd_register_done(m_DeamonContext);
Send notification code:
//get a constant string back
char * path = YangMapperP->GetKeyPath(a_TableId);
confd_tag_value_t values[11];
int Index = 0;
confd_hkeypath_t *p_KeyPath ;
struct sockaddr_in SocketAddress;
struct in_addr an_addr = {0xa0a0a0a0};
//create socket to translate the keypath
int MappiSocket;
SocketAddress.sin_addr.s_addr = inet_addr("127.0.0.1");
SocketAddress.sin_family = AF_INET;
SocketAddress.sin_port = htons(CONFD_PORT);
if ((MappiSocket = socket(PF_INET, SOCK_STREAM, 0)) < 0)
{
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::CommitWasDoneByOther socket creation failed " );
}
if (maapi_connect(MappiSocket, (struct sockaddr *)&SocketAddress, sizeof (SocketAddress)) < 0)
{
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::CommitWasDoneByOther socket connect failed " );
}
if (maapi_xpath2kpath(MappiSocket, path, &p_KeyPath) != CONFD_OK) //translate the keypath
{
DEBUG(netConfTraceP, << "Cannot translate xpath\n");
}
maapi_close(MappiSocket);
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::CommitWasDoneByOther maapi_close ");
//set the values of the notification
CONFD_SET_TAG_XMLBEGIN(&values[Index], ncn_netconf_config_change, ncn__ns);
Index++;
CONFD_SET_TAG_XMLBEGIN(&values[Index], ncn_changed_by, ncn__ns);
Index++;
CONFD_SET_TAG_STR(&values[Index], ncn_username, "Admin");
Index++;
CONFD_SET_TAG_UINT32(&values[Index], ncn_session_id, 0);
Index++;
CONFD_SET_TAG_IPV4(&values[Index], ncn_source_host, an_addr);
Index++;
CONFD_SET_TAG_XMLEND(&values[Index], ncn_changed_by, ncn__ns);
Index++;
CONFD_SET_TAG_ENUM_VALUE(&values[Index], ncn_datastore, ncn_running);
Index++;
CONFD_SET_TAG_XMLBEGIN(&values[Index], ncn_edit, ncn__ns);
Index++;
CONFD_SET_TAG_OBJECTREF(&values[Index], ncn_target, p_KeyPath);
Index++;
CONFD_SET_TAG_XMLEND(&values[Index], ncn_edit, ncn__ns);
Index++;
CONFD_SET_TAG_XMLEND(&values[Index], ncn_netconf_config_change, ncn__ns);
Index++;
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::CommitWasDoneByOther CONFD_SET_TAG done ");
struct confd_datetime EventTime;
struct tm Time;
struct timeval TimeValue;
gettimeofday(&TimeValue, NULL);
gmtime_r(&TimeValue.tv_sec, &Time);
memset(&EventTime, 0, sizeof(EventTime));
EventTime.year = 1900 + Time.tm_year;
EventTime.month = Time.tm_mon + 1;
EventTime.day = Time.tm_mday;
EventTime.sec = Time.tm_sec;
EventTime.micro = TimeValue.tv_usec;
EventTime.timezone = 0;
EventTime.timezone_minutes = 0;
EventTime.hour = Time.tm_hour;
EventTime.min = Time.tm_min;
print_modifications(values, Index, NULL, 0);
int status = confd_notification_send(m_LiveContext, &EventTime, values, Index);
if (status != CONFD_OK)
{
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::CommitWasDoneByOther confd_notification_send was failed\n");
}
else
{
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::CommitWasDoneByOther confd_notification_send for table Id = " << a_TableId);
}
confd_free_hkeypath(p_KeyPath);
}
Hi,
using tcpdump on the CONFD_PORT (4565) we see the following packet dump.
please advice if you understand something new from it.
thanks
Inbal
12:41:43.297143 IP (tos 0x0, ttl 64, id 63495, offset 0, flags [DF], proto TCP (6), length 383)
127.0.0.1.47144 > 127.0.0.1.4565: Flags [P.], cksum 0xff73 (incorrect -> 0x4b2f), seq 22:353, ack 10, win 342, options [nop,nop,TS val 2979493033 ecr 2975759089], length 331
0x0000: 0000 0000 0000 0000 0000 0000 0800 4500 …E.
0x0010: 017f f807 4000 4006 436f 7f00 0001 7f00 …@.@.Co…
0x0020: 0001 b828 11d5 021b 82e1 a49b d211 8018 …(…
0x0030: 0156 ff73 0000 0101 080a b197 74a9 b15e .V.s…t…^
0x0040: 7af1 0000 0147 0000 000f 8368 0661 aa64 z…G…h.a.d
0x0050: 0007 4e45 5443 4f4e 4664 0009 756e 6465 …NETCONFd…unde
0x0060: 6669 6e65 6468 0261 1368 0962 0000 07e1 finedh.a.h.b…
0x0070: 6106 610c 610c 6129 612b 6200 0487 8b61 a.a.a.a)a+b…a
0x0080: 0061 0061 0268 0264 0004 6878 6d6c 6c00 .a.a.h.d…hxmll.
0x0090: 0000 0b68 026c 0000 0001 624a 2f65 9062 …h.l…bJ/e.b
0x00a0: 7d5a bd43 6400 0573 7461 7274 6802 6c00 }Z.Cd…starth.l.
0x00b0: 0000 0162 4a2f 6590 622b f78a 5964 0005 …bJ/e.b+…Yd…
0x00c0: 7374 6172 7468 0262 33e5 316f 6d00 0000 starth.b3.1om…
0x00d0: 0561 646d 696e 6802 6216 34e7 bc68 0261 .adminh.b.4…h.a
0x00e0: 0c61 0068 0262 5372 1c66 6804 61a0 61a0 .a.h.bSr.fh.a.a.
0x00f0: 61a0 61a0 6802 6c00 0000 0162 4a2f 6590 a.a.h.l…bJ/e.
0x0100: 622b f78a 5964 0004 7374 6f70 6802 621a b+…Yd…stoph.b.
0x0110: cb6a 1268 0261 1c61 0068 026c 0000 0001 .j.h.a.a.h.l…
0x0120: 624a 2f65 9062 664f 856a 6400 0573 7461 bJ/e.bfO.jd…sta
0x0130: 7274 6802 6272 178c 7868 0261 226c 0000 rth.br…xh.a"l…
0x0140: 0003 623d 949d 8762 00d3 defe 6c00 0000 …b=…b…l…
0x0150: 0162 64f3 2785 6221 19c8 4f6a 6802 6c00 .bd.’.b!..Ojh.l.
0x0160: 0000 0162 4a2f 6590 6266 4f85 6a64 0004 …bJ/e.bfO.jd…
0x0170: 7374 6f70 6802 6c00 0000 0162 4a2f 6590 stoph.l…bJ/e.
0x0180: 627d 5abd 4364 0004 7374 6f70 6a b}Z.Cd…stopj
Hi,
Is the handling of the callbacks and the notifications through the worker socket done from the same or different threads?
A quick look at your TCP dump seems to indicate that you have two notifications/requests/replies being sent on the same socket simultaneously to ConfD.
Hi,
The implementation is that we send the notification from different thread than the thread that receive and answer the confd callbacks.
However, I tried to send the notification from the callback thread, to be sure it is the only message that being sent at the same time. Still got the internal error message…
- how do you see in the tcp dump I added that 2 message are sent at the same time?
- Is there a way to see what is being received in the CONFD process? Any other idea of how to proceed?
thanks
Inbal
It was just a guess after comparing your tcpdump with one of a NETCONF notification send that was successful. Have you done that comparison yourself?
tcpdump or similar is what you have.
I suggest you use one socket per thread. No sharing of sockets between threads is best practice. Then compare tcpdumps of NETCONF notifications that failed to send and successful sends.
Hi,
when running the example, there is no Internal Error when sending notification.
However, from technical issues of our servers, I can not operate tcpdump.
Is it possible that you will send me a “good” netconf - notification dump so I can compare?
thanks in advance,
Inbal
Hi,
I suggest you run the examples.confd/netconf_notifications example on a machine of your choice, e.g. a laptop, and tcpdump the ConfD IPC port. Port 4565 in that example.
See the README in the example for details.
You can edit the example to send a notification similar to what you do in your application doing something similar to what I did in an earlier post in this thread:
Hi,
we managed to catch a message of the example application.
we compare the message I capture with tcpdump and the message of the sample application - the messages are almost equal.
The only different is on ncn_source_host.
In the message I send it looks like that:
0x0070: 6106 610c 610c 6129 612b 6200 0487 8b61 a.a.a.a)a+b…a
on the sample application it looks like that:
0x0070: 6106 611a 6109 612d 6135 6200 0dd1 4761 a.a.a.a-a5b…Ga
The code looks the same (see below the copy of the create notification code):
struct in_addr an_addr = {0xa0a0a0a0};
can you elaborate the diffrence between these 2 notifications?
can it cause th eInternal error we continue to receive in our application (and not in the sample application)?
thanks
Inbal
confd_tag_value_t values[11];
int Index = 0;
confd_hkeypath_t *p_KeyPath;
struct sockaddr_in SocketAddress;
struct in_addr an_addr = {0xa0a0a0a0};
//create socket to translate the keypath
int MappiSocket;
SocketAddress.sin_addr.s_addr = inet_addr("127.0.0.1");
SocketAddress.sin_family = AF_INET;
SocketAddress.sin_port = htons(CONFD_PORT);
if ((MappiSocket = socket(PF_INET, SOCK_STREAM, 0)) < 0)
{
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::CommitWasDoneByOther socket creation failed " );
}
if (maapi_connect(MappiSocket, (struct sockaddr *)&SocketAddress, sizeof (SocketAddress)) < 0)
{
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::CommitWasDoneByOther socket connect failed " );
}
if (maapi_xpath2kpath(MappiSocket, Path, &p_KeyPath) != CONFD_OK) //translate the keypath
{
DEBUG(netConfTraceP, << "Cannot translate xpath\n");
}
maapi_close(MappiSocket);
DEBUG(netConfTraceP, <<"ConfDAdaptorManager::CommitWasDoneByOther maapi_close ");
//set the values of the notification
CONFD_SET_TAG_XMLBEGIN(&values[Index], ncn_netconf_config_change, ncn__ns);
Index++;
CONFD_SET_TAG_XMLBEGIN(&values[Index], ncn_changed_by, ncn__ns);
Index++;
CONFD_SET_TAG_STR(&values[Index], ncn_username, "admin");
Index++;
CONFD_SET_TAG_UINT32(&values[Index], ncn_session_id, 0);
Index++;
CONFD_SET_TAG_IPV4(&values[Index], ncn_source_host, an_addr);
Index++;
CONFD_SET_TAG_XMLEND(&values[Index], ncn_changed_by, ncn__ns);
Index++;
CONFD_SET_TAG_ENUM_VALUE(&values[Index], ncn_datastore, ncn_running);
Index++;
CONFD_SET_TAG_XMLBEGIN(&values[Index], ncn_edit, ncn__ns);
Index++;
CONFD_SET_TAG_OBJECTREF(&values[Index], ncn_target, p_KeyPath);
Index++;
CONFD_SET_TAG_XMLEND(&values[Index], ncn_edit, ncn__ns);
Index++;
CONFD_SET_TAG_XMLEND(&values[Index], ncn_netconf_config_change, ncn__ns);
Index++;That part of the notification message (around offset 0x0070) is a time stamp, hex 61 ascii ‘a’ is uninitialized memory.
6106 610c 610c 6129 612b 6200 0487 8b61
Jun 12 12 29 43 microsec
OK, the rest of the frame is the same.
The part that contains the data filled by the application is exactly the same.
And still, from the application I receive Internal Error, but when running the sample it doesn’t.
Can we have more detailed info regarding the “internal error”?
Hi,
I did dump to confd according to the instructions in the user guide (confd --debug-dump mydump1).
How can I send to you the output file so maybe you can find some clue why I get “devel-c Internal error on API request” when sending notification.
Thanks,
Debug dump files are something you send to Tail-f support if you have a support account setup with them.
Rather than trying to get help with debug dump files from the user forum, I suggest you enable the error log in confd.conf and print it using for example
$ confd --printlog error.log
When you get a “API request” error from sending your notification, look for something like a “capi_server proto_request” error message.
That should give you a bit more information on what ConfD received from your application that caused the “API request” error.
See ConfD 6.4 UG Chapter 28.12.4. “General Troubleshooting Strategies” under “Debug error log” on how to enable the error log in confd.conf.
See the confd(1) man page under “–printlog” on how to print the error log.
Hi,
When using the error.log I got the following printout:
11-Jul-2017::11:41:53.167 <0.79.0> <0.231.0> capi_server:2180: capi_server proto_request error:badarg for
<<131,104,6,97,170,100,0,7,78,69,84,67,79,78,70,100,0,9,117,110,100,101,102,
105,110,101,100,104,2,97,19,104,9,98,0,0,7,225,97,7,97,11,97,11,97,41,97,
53,98,0,0,98,56,97,0,97,0,97,2,104,2,100,0,4,104,120,109,108,108,0,0,0,11,
104,2,108,0,0,0,1,98,74,47,101,144,98,125,90,189,67,100,0,5,115,116,97,114,
116,104,2,108,0,0,0,1,98,74,47,101,144,98,43,247,138,89,100,0,5,115,116,97,
114,116,104,2,98,51,229,49,111,109,0,0,0,5,97,100,109,105,110,104,2,98,22,
52,231,188,104,2,97,12,97,0,104,2,98,83,114,28,102,104,4,97,160,97,160,97,
160,97,160,104,2,108,0,0,0,1,98,74,47,101,144,98,43,247,138,89,100,0,4,115,
116,111,112,104,2,98,26,203,106,18,104,2,97,28,97,0,104,2,108,0,0,0,1,98,
74,47,101,144,98,102,79,133,106,100,0,5,115,116,97,114,116,104,2,98,114,23,
140,120,104,2,97,34,108,0,0,0,3,98,61,148,157,135,98,0,211,222,254,108,0,0,
0,1,98,100,243,39,133,98,33,25,200,79,106,104,2,108,0,0,0,1,98,74,47,101,
144,98,102,79,133,106,100,0,4,115,116,111,112,104,2,108,0,0,0,1,98,74,47,
101,144,98,125,90,189,67,100,0,4,115,116,111,112,106>>
[{io_lib,format,
["\"~ts\" is not a valid value.",[12,0]],
[{file,"io_lib.erl"},{line,168}]},
{cs_error,add_str,2,[{file,"cs_error.erl"},{line,113}]},
{cs_error,fmt_error,2,[{file,"cs_error.erl"},{line,63}]},
{notif_server,strerror,1,[{file,"notif_server.erl"},{line,357}]},
{capi_server,notif_log,3,[{file,"capi_server.erl"},{line,2294}]},
{capi_server,socket_request,3,[{file,"capi_server.erl"},{line,2177}]},
{capi_server,socket_loop,5,[{file,"capi_server.erl"},{line,2110}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]
Can you let me know what it means? Can I know from it what is the problem? which attribute?
Thanks,
Hi,
Your struct confd_notification_ctx m_LiveContext contain some trash so that the ConfD internal “flags” field in the struct confd_notification_ctx is set to “2”.
I removed a bad suggestion from my previous post after an expert on this forum pointed out to me that the root cause to the flags being set is not “trash” but actually that you set the deamon context flags to CONFD_DAEMON_FLAG_STRINGSONLY. I.e.
confd_set_daemon_flags(dctx, CONFD_DAEMON_FLAG_STRINGSONLY);
And then your application send the notification as ConfD tag_values instead of strings. while the deamon context strings only flag will be reflected in your struct confd_notification_ctx m_LiveContext->flags == 2
Recommended way forward is to remove the call to confd_set_daemon_flags(dctx, CONFD_DAEMON_FLAG_STRINGSONLY) from your notifier application and just continue sending ConfD tag_values instead of strings.
Hi,
We did a short test and it seems to be the problem - no “internal error” when changing the flag from 2 to 0.
However, for configuration and “get” operations we must have the flag “string only”.
Therefore, I think we should have another deamon for notifications. Please advice regarding the following initialization with the additional deamon (for notification only):
confd_init("ConfdAdaptor", stderr , CONFD_TRACE); <-- please advice if needed to do it double, one for each deamon.
//configuration deamon init
m_DeamonConfigContext = confd_init_daemon("ConfdAdaptor");
confd_set_daemon_flags(m_DeamonConfigContext , CONFD_DAEMON_FLAG_STRINGSONLY); <--config deamon only!!!
m_ControlSocket = socket(PF_INET, SOCK_STREAM, 0);
confd_connect(m_DeamonConfigContext , m_ControlSocket, CONTROL_SOCKET, (struct sockaddr*)&socketAddress, sizeof (struct sockaddr_in))
m_WorkerSocket = socket(PF_INET, SOCK_STREAM, 0);
confd_connect(m_DeamonConfigContext , m_WorkerSocket, WORKER_SOCKET,(struct sockaddr*)&socketAddress,sizeof (struct sockaddr_in))
//registration for transaciton callbacks and configuration call backs
confd_register_done(m_DeamonConfigContext );
//notification deamon init
m_DeamonNotifyContext = confd_init_daemon("ConfdNotifications");
confd_register_notification_stream(m_DeamonNotifyContext , &NotificationCallBack, &m_LiveContext) <-- this is done for the notification deamon
confd_register_done(m_DeamonNotifyContext );
Do I miss something?
Hi,
Your struct confd_notification_stream_cbs NotificationCallBack need a worker socket dedicated to sending notifications. That worker socket does not need to be polled. See confd_lib_dp man page “confd_register_notification_stream”