Commit check and validation callpoint in external DB

Dear Support,

we are working with external DB.
I am trying to see how I can implement the commit check with our architecture. I added a validation call point in our yang and implemented the related registrations/calls in the code in order to run it and understand better.
As a result, I have 2 questions regarding:

Q1: I saw that the validation callback is not working in the same way as the configuration callpoint does. If I insert validation point and call point in the same place in the yand model, when I run the commit I receive several callbacks for the callpoint (for each changed leaf below) and only one callback for the validation point (for the root). Is there a way to get callback for each changed leaf under the validation point in a similar way that the callpoint work? beside of course to add validation point for each leaf.

Q2: I see that when I run the commit check, I get the validation callbacks only. but when I run the “commit” without the “check” option, I get callbacks from the configuration transaction and from the validation transaction in parallel. My question here: is there a way to know that these transactions are related to the same configuration? Is there a way in the validation transaction to know that there is also a configuration trasnaction running in parallel? It can be very helpful to our implementation.

thank you in advance
Inbal

Hi,

Q1: See maapi_diff_iterate() in the confd_lib_maapi(3) man page and in examples.confd/validate/c_dependency/more_a_than_b.c

Q2: Yes. You can for example compare the transaction handle. See struct confd_trans_ctx thandle. See also validation_info, t_opaque and v_opaque in that struct (more on those in the confd_lib_dp(3) man page)
You first get a call to your validation point callback. When your validation point application read data from your external database your callpoint callbacks get called.

Hi,

Thanks for your reply,

Regarding Q1 - I will check that in the code and see if there is something missing. I read about it and I am not sure we receive all the needed data, but I will check it.

Regarding Q2 - I am not sure I understand the answer. If CONFD_VALIDATION_FLAG_TEST is set - is it mean that we are in “commit check” operation? Is it turn off in “commit” operation?
I am not sure how , t_opaque and v_opaque can help me, when I get one of the validation callback, to know that right after that “set” and “prepare” will be receive? what is the difference in the confd_trans_ctx of validation transaction when running “commit check” vs “commit”?

thanks again
Inbal

My answer showed you what tools you have since you did not give us any details on what you are trying to do.

This is standard stuff as you likely know.
But if you haven’t looked into RFC 6241 yet, please do so:
https://tools.ietf.org/html/rfc6241#section-8.6 and in particular https://tools.ietf.org/html/rfc6241#section-8.6.5.1

The details of the tools / information existing in the transaction state (struct confd_trans_ctx) is straight forward to use if you know the NETCONF RFC and have read the UG + related man page info.

Dear support,
Hi again.
I have question related to Q1, on maapi_diff_iterate().
I tried to operate it with no luck - after calling the maapi_diff_iterate(), the iter function was not called back. I read about it in the user guide and I am not sure it should work with external DB.
Please advice
Thanks
Inbal

Hi,

Try something like this:

$ diff -u . ~/tailf/confd-6.3.1/examples.confd/intro/6-c_config/
diff -u ./Makefile /Users/tailf/confd-6.3.1/examples.confd/intro/6-c_config/Makefile
--- ./Makefile	2016-11-23 10:43:27.000000000 +0100
+++ /Users/tailf/confd-6.3.1/examples.confd/intro/6-c_config/Makefile	2017-03-19 14:55:05.000000000 +0100
@@ -44,16 +44,21 @@
 SRC	= hosts.c dlist.c
 OBJS	= $(SRC:.c=.o)
 
-all:	hst.fxs hst.h hosts $(CDB_DIR) ssh-keydir
+SRC2	= vp1.c
+OBJS2	= $(SRC2:.c=.o)
+
+all:	hst.fxs hst.h hosts vp1 $(CDB_DIR) ssh-keydir
 	@echo "Build complete"
 
 hosts: $(OBJS)
 	$(CC) -o $@ $(OBJS) $(LIBS)
 
+vp1:	$(OBJS2)
+	$(CC) -o $@ $(OBJS2) $(LIBS)
 
 ######################################################################
 clean:	iclean
-	-rm -rf hst.h hosts RUNNING.ckp 2> /dev/null || true
+	-rm -rf hst.h hosts vp1 RUNNING.ckp 2> /dev/null || true
 
 start:  stop
 	$(CONFD)  -c confd.conf $(CONFD_FLAGS)
@@ -61,7 +66,8 @@
 	### * In another terminal window, run queries
 	###   (try 'make query' for an example)
 	### * In this window, the HOSTS confd daemon now starts:
-	./hosts $(START_FLAGS)
+	./hosts $(START_FLAGS) -q &
+	./vp1
 
 ######################################################################
 stop:
diff -u ./hst.yang /Users/tailf/confd-6.3.1/examples.confd/intro/6-c_config/hst.yang
--- ./hst.yang	2016-11-23 10:43:27.000000000 +0100
+++ /Users/tailf/confd-6.3.1/examples.confd/intro/6-c_config/hst.yang	2017-03-19 14:42:26.000000000 +0100
@@ -16,6 +16,9 @@
     list host {
       key name;
       max-elements 64;
+      tailf:validate vp1 {
+        tailf:dependency '.';
+      }
       leaf name {
         type string;
       }
Only in /Users/tailf/confd-6.3.1/examples.confd/intro/6-c_config/: vp1.c
$ cat /Users//tailf/confd-6.3.1/examples.confd/intro/6-c_config/vp1.c
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <sys/poll.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <stdarg.h>
#include <stdio.h>

#include "confd_lib.h"
#include "confd_dp.h"
#include "confd_maapi.h"

/* include generated ns file */
#include "hst.h"

int debuglevel = CONFD_TRACE;

static int ctlsock;
static int workersock;
static int maapi_socket;
static struct confd_daemon_ctx *dctx;

struct confd_trans_validate_cbs vcb;
struct confd_valpoint_cb valp1;

static void OK(int rval)
{
    if (rval != CONFD_OK) {
        fprintf(stderr, "vp1.c: error not CONFD_OK: %d : %s \n",
                confd_errno, confd_lasterr());
        abort();
    }
}

static int init_validation(struct confd_trans_ctx *tctx)
{
    OK(maapi_attach(maapi_socket, hst__ns, tctx));
    confd_trans_set_fd(tctx, workersock);
    return CONFD_OK;
}

static int stop_validation(struct confd_trans_ctx *tctx)
{
    OK(maapi_detach(maapi_socket, tctx));
    return CONFD_OK;
}

enum maapi_iter_ret iter(confd_hkeypath_t *kp,
                         enum maapi_iter_op op,
                         confd_value_t *oldv,
                         confd_value_t *newv,
                         void *state)
{
    char buf[BUFSIZ];
    char *opstr = "";
    if (op == MOP_CREATED)
        opstr = "create";
    else if (op == MOP_DELETED)
        opstr = "delete";
    else if (op == MOP_MODIFIED)
        opstr = "modif";
    else if (op == MOP_VALUE_SET)
        opstr = "set  ";

    confd_pp_kpath(buf, BUFSIZ, kp);
    fprintf(stderr, "Op= %s Path= %s\n", opstr, buf);
    return ITER_RECURSE;
}

static int traverse_list(struct confd_trans_ctx *tctx,
                        confd_hkeypath_t *keypath,
                        confd_value_t *newval)
{
    char buf[BUFSIZ];
    confd_pp_kpath(buf, BUFSIZ, keypath);
    fprintf(stderr, "traverse_list path %s\n", buf);
    maapi_diff_iterate(maapi_socket, tctx->thandle, iter, 0, NULL);
    return CONFD_OK;
}


static int maapi_sock(int *maapi_sock)
{

    struct sockaddr_in addr;

    addr.sin_addr.s_addr = inet_addr("127.0.0.1");
    addr.sin_family = AF_INET;
    addr.sin_port = htons(4565);

    if ((*maapi_sock = socket(PF_INET, SOCK_STREAM, 0)) < 0 )
        confd_fatal("Failed to open socket\n");

    if (maapi_connect(*maapi_sock, (struct sockaddr*)&addr,
                      sizeof (struct sockaddr_in)) < 0)
        confd_fatal("Failed to confd_connect() to confd \n");

    return CONFD_OK;
}

int main(int argc, char **argv)
{
    int c;

    struct sockaddr_in addr;

    while ((c = getopt(argc, argv, "tdps")) != -1) {
        switch(c) {
        case 't':
            debuglevel = CONFD_TRACE;
            break;
        case 'd':
            debuglevel = CONFD_DEBUG;
            break;
        case 'p':
            debuglevel = CONFD_PROTO_TRACE;
            break;
        case 's':
            debuglevel = CONFD_SILENT;
            break;
        }
    }


    confd_init("MYNAME", stderr, debuglevel);

    if ((dctx = confd_init_daemon("mydaemon")) == NULL)
        confd_fatal("Failed to initialize confd\n");

    if ((ctlsock = socket(PF_INET, SOCK_STREAM, 0)) < 0 )
        confd_fatal("Failed to open ctlsocket\n");

    addr.sin_addr.s_addr = inet_addr("127.0.0.1");
    addr.sin_family = AF_INET;
    addr.sin_port = htons(CONFD_PORT);

    OK(confd_load_schemas((struct sockaddr*)&addr,sizeof (struct sockaddr_in)));

    /* Create the first control socket, all requests to */
    /* create new transactions arrive here */
    if (confd_connect(dctx, ctlsock, CONTROL_SOCKET,
                      (struct sockaddr*)&addr,
                      sizeof (struct sockaddr_in)) < 0) {
        confd_fatal("Failed to confd_connect() to confd \n");
    }

    /* Also establish a workersocket, this is the most simple */
    /* case where we have just one ctlsock and one workersock */

    if ((workersock = socket(PF_INET, SOCK_STREAM, 0)) < 0 )
        confd_fatal("Failed to open workersocket\n");
    if (confd_connect(dctx, workersock, WORKER_SOCKET,(struct sockaddr*)&addr,
                      sizeof (struct sockaddr_in)) < 0)
        confd_fatal("Failed to confd_connect() to confd \n");

    vcb.init = init_validation;
    vcb.stop = stop_validation;
    confd_register_trans_validate_cb(dctx, &vcb);

    valp1.validate = traverse_list;
    strcpy(valp1.valpoint, "vp1");
    OK(confd_register_valpoint_cb(dctx, &valp1));


    OK(confd_register_done(dctx));

    OK(maapi_sock(&maapi_socket));

    while (1) {
        struct pollfd set[2];
        int ret;

        set[0].fd = ctlsock;
        set[0].events = POLLIN;
        set[0].revents = 0;

        set[1].fd = workersock;
        set[1].events = POLLIN;
        set[1].revents = 0;

        if (poll(&set[0], 2, -1) < 0) {
            perror("Poll failed:");
            continue;
        }

        if (set[0].revents & POLLIN) {
            if ((ret = confd_fd_ready(dctx, ctlsock)) == CONFD_EOF) {
                confd_fatal("Control socket closed\n");
            } else if (ret == CONFD_ERR && confd_errno != CONFD_ERR_EXTERNAL) {
                confd_fatal("Error on control socket request: %s (%d): %s\n",
                     confd_strerror(confd_errno), confd_errno, confd_lasterr());
            }
        }
        if (set[1].revents & POLLIN) {
            if ((ret = confd_fd_ready(dctx, workersock)) == CONFD_EOF) {
                confd_fatal("Worker socket closed\n");
            } else if (ret == CONFD_ERR && confd_errno != CONFD_ERR_EXTERNAL) {
                confd_fatal("Error on worker socket request: %s (%d): %s\n",
                     confd_strerror(confd_errno), confd_errno, confd_lasterr());
            }
        }
    }
}

Hi,

Thanks for your reply. My code was something like the code you suggested.
I see the following printout:

TRACE CALL validate init(thandle=8)TRACE MAAPI_ATTACH DEBUG Socket to ConfD is closed
–> CONFD_EOF
–> CONFD_OK
TRACE CALL data validate(thandle=8, /bgp/peer-groups, undefined)DEBUG Socket to ConfD is closed
TRACE MAAPI_DIFF_ITER
–> CONFD_OK
TRACE CALL validation stop(thandle=8)TRACE MAAPI_DETACH DEBUG Socket to ConfD is closed
–> CONFD_EOF
–> CONFD_OK

the Socket is closed everytime, don’t understand why. with gdb I see the following for each maapi socket call we do:

Program received signal SIGPIPE, Broken pipe.
0x44a9ad71 in write () at …/sysdeps/unix/syscall-template.S:81
81 in …/sysdeps/unix/syscall-template.S
(gdb) bt

#0 0x44a9ad71 in write () at …/sysdeps/unix/syscall-template.S:81

#1 0xb7766106 in write_fill () from /usr/lib/libconfd.so

#2 0xb7766193 in confd_write () from /usr/lib/libconfd.so

#3 0xb776623c in op_write_buf () from /usr/lib/libconfd.so

#4 0xb775fadb in maapi_diff_iterate () from /usr/lib/libconfd.so

#5 0x08e83bd8 in ConfDAdaptorBgpPeerGroupMapper::validate (a_pTransaction=0xb530c6d0, a_pKeyPath=0xb3a76040, a_pValue=0xb3a76020)

at ./mappers/bgp/ConfDAdaptorBgpPeerGroupMapper.c++:103

#6 0xb7749663 in ?? () from /usr/lib/libconfd.so

#7 0xb774cf95 in confd_fd_ready () from /usr/lib/libconfd.so

#8 0x08e6fdcd in ConfDAdaptorManager::ConfdAdaptorPollingThreadBody (this=0x9a202a0) at ./adaptor/ConfDAdaptorManager.c++:260

#9 0x08e6fb23 in ConfDAdaptorManager::ConfdAdaptorPolling () at ./adaptor/ConfDAdaptorManager.c++:210

#10 0x0899a3ea in util::RunnableAdapter<void, void>::operator() (this=0x99bdcc0) at export/include/util/RunnableAdapter.h:271

#11 0xb75e7706 in util::thread::_ThreadMain (arg=0x9a0c8c8) at build/kvm/linux/x86/system/util/Thread.c++:461

#12 0x44b55d85 in start_thread (arg=0xb3a81b40) at pthread_create.c:312

#13 0x44aa9a1e in clone () at …/sysdeps/unix/sysv/linux/i386/clone.S:129

can you advise what’s the problem?

thank
Inbal.

Hi Inbal,

What does your developer log (log level trace) say?
The MAAPI socket’s underlying connection seem to EPIPE (SIGPIPE triggered) on send since the connection was closed by the peer with no outstanding data.

Hi,

I found a missing maapi_connect. after adding that I still have issue with confd connection, but different.
I see the following in confd_devel.log now:
22-Mar-2017::15:37:03.611 isim_Host1 confd[10501]: devel-c Control socket request timed out daemon ‘ConfdAdaptor’ id 0
22-Mar-2017::15:37:03.615 isim_Host1 confd[10501]: devel-c validate error {external_timeout, “”} for path /eci-bgp:bgp/peer-groups
r
the traces before the confd connection stuck:
TRACE CALL validate init(thandle=8)TRACE MAAPI_ATTACH --> CONFD_OK
–> CONFD_OK
then after a while :
DEBUG EOF on socket to ConfD
Control socket closed

please see if you can advise according to this info.
thanks in advance
Inbal

this problem was resolved