How to add the support of the find_next_object() callback to the 5-c_stats example?

An updated variant that support returning the next object instead of “-1” and not hard coding the number of objects in each array.
More importantly this variant chunk a max number of objects / list entries / rows for improved performace.

...
static int navals;
static int max_nobjs = 100; // Chunk size hardcoded here to simplify
...
static int find_next_object(struct confd_trans_ctx *tctx,
                            confd_hkeypath_t *kp,
                            enum confd_find_next_type type,
                            confd_value_t *keys, int nkeys)
{
  
  int pos = -1;
  int ipos;

  confd_value_t *v;
  struct confd_next_object *obj;
  int i;
  struct arpdata *dp = tctx->t_opaque;
  int n_list_entries, nobjs;
  struct aentry *ae;
  struct in_addr ip;
  char *iface;
  
  if (need_arp(dp)) {
    if (run_arp(dp) == CONFD_ERR)
      return CONFD_ERR;
  }

  n_list_entries = dp->n_ae;
  ae = dp->arp_entries;
  
  switch (nkeys) {
  case 0:
    if(n_list_entries > 0) {
      pos = 0; /* No keys provided => the first entry will always be "after" */
    }
    break;
  case 1:
    /* one key provided => find the first entry "after"
       - since there can never be a "same" entry, 'type' is not relevant,
       and an entry with same first key is also "after" */
    ip = CONFD_GET_IPV4(&keys[0]);
    ipos = 0;
    for (; ae != NULL; ae = ae->next) {
      if (memcmp(&ae->ip4, &ip, sizeof(struct in_addr)) >= 0) {
	pos = ipos;
	break;
      }
      ipos++;
    }
    break;
  case 2:
    /* both keys provided => find first entry "after" or "same",
       depending on 'type' */
    ip = CONFD_GET_IPV4(&keys[0]);
    iface = (char*)CONFD_GET_BUFPTR(&keys[1]);

    switch (type) {
    case CONFD_FIND_NEXT:
      /* entry must be "after" */
      ipos = 0; 
      for (; ae != NULL; ae = ae->next) {
	if (memcmp(&ae->ip4, &ip, sizeof(struct in_addr)) > 0 ||
            (memcmp(&ae->ip4, &ip, sizeof(struct in_addr)) == 0 &&
             strcmp(iface, ae->iface) > 0)) {
	  pos = ipos;
	  break;
	}
	ipos++;
      }
      break;
    case CONFD_FIND_SAME_OR_NEXT:
      /* entry must be "same" or "after" */
      ipos = 0;
      for (; ae != NULL; ae = ae->next) {
	if (memcmp(&ae->ip4, &ip, sizeof(struct in_addr)) > 0 ||
            (memcmp(&ae->ip4, &ip, sizeof(struct in_addr)) == 0 &&
             strcmp(iface, ae->iface) >= 0)) {
	  pos = ipos;
	  break;
	}
	ipos++;
      }
      break;
    }
    break;
  default:
    confd_trans_seterr(tctx, "invalid number of keys: %d", nkeys);
    return CONFD_ERR;
  }

  if (n_list_entries - pos > max_nobjs) {
    nobjs = max_nobjs + pos;
  } else {
    nobjs = n_list_entries;
  }
  
  v = (confd_value_t *) malloc(sizeof(confd_value_t) * nobjs * navals);
  obj = malloc(sizeof(struct confd_next_object) * (nobjs + 2));

  if (pos != -1) {
    for (i = 0; pos + i < nobjs; i++) {
      obj[i].v = &v[i * navals];

      CONFD_SET_IPV4(&(obj[i].v[0]), ae->ip4);
      CONFD_SET_STR(&(obj[i].v[1]), ae->iface);
      if (ae->hwaddr == NULL) {
	CONFD_SET_NOEXISTS(&(obj[i].v[2]));
      } else {
	CONFD_SET_STR(&(obj[i].v[2]), ae->hwaddr);
      }
      CONFD_SET_BOOL(&(obj[i].v[3]), ae->perm);
      CONFD_SET_BOOL(&(obj[i].v[4]), ae->pub);

      obj[i].n = navals;
      obj[i].next = (long)ae->next;//-1;
      ae = ae->next;
    }
    if (pos + i == n_list_entries)
      obj[i++].v = NULL;
    confd_data_reply_next_object_arrays(tctx, obj, i, 0);
  } else {
    confd_data_reply_next_object_array(tctx, NULL, -1, -1);
  }
  free(v);
  free(obj);
  return CONFD_OK;
}
...
int main(int argc, char *argv[])
{
    struct confd_cs_node *object;
...
    object = confd_cs_node_cd(NULL, "/arpe:arpentries/arpe");
    navals = confd_max_object_size(object);
...

i have a question, if get_next_object and find_nex_object also have return multi-lists in one netconf session, what’s the difference between them, and whether they can co-exist?
And in client side, what api should be used to trigger find_next_object() to be called in confd side? i ever use maapi_find_next & maapi_get_objects to get the lists started from the specified start entry, but find_next_object() failed to be called. Do you have any ideas?

find_next_object() will be called if you have a list with two or more keys that you filter data from.

Under examples.confd/dp/find_next in the ConfD example set, see the README, you have such a YANG list with two keys, “slot” and “port”, and the find_next() callback implemented. Perhaps find_next_object() callback should just be added to that example.

The README show you a few ways to trigger the find_next() callback that would instead trigger the find_next_object() callback had it been implemented instead of or in addition to find_next().

Overview of the YANG model:

$ yanger -f tree ports.yang
module: ports
  +--ro port* [slot port]
     +--ro slot      int8
     +--ro port      int8
     +--ro status    enumeration

NETCONF:

$ netconf-console --get -x '/port[slot&gt;2]/status'

The netconf-console then python script NETCONF client create the following RPC from the above:

<rpc message-id="1" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <get>
    <filter select="port[slot&gt;2]/status" type="xpath"/>
  </get>
</rpc>

And the ConfD NETCONF server reply with the keys and leaf that match the XPath expression:

<rpc-reply message-id="1" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <data>
    <port xmlns="http://tail-f.com/ns/example/ports">
      <slot>4</slot>
      <port>0</port>
      <status>error</status>
    </port>
    <port xmlns="http://tail-f.com/ns/example/ports">
      <slot>4</slot>
      <port>2</port>
      <status>disabled</status>
    </port>
  </data>
</rpc-reply>

Another simple NETCONF example that would work both with subtree and XPath filtering would be to just ask for all entries matching one of the keys, for example slot 4.
Subtree filter:

<rpc message-id="2" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <get>
    <filter type="subtree">
      <port xmlns="http://tail-f.com/ns/example/ports">
        <slot>4</slot>
      </port>
    </filter>
  </get>
</rpc>

XPath filter:

<rpc message-id="3" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <get>
    <filter select="/port[slot=4]" type="xpath"/>
  </get>
</rpc>

MAAPI:

$ confd_cmd -dd -c 'xpath_eval /port[slot=4]'

CLI:
# show port 4 *

i want to retrieve multiple lists from certain start point, can you tell me which api i should use in client and confd server ?

Check out for example the NETCONF RFC section 6.4.7. Multiple Subtrees

Actually there is no requirement for “two or more” keys, find_next()/find_next_object() is quite applicable to lists with a single key. The example has two keys primarily to demonstrate the required logic when “some, but not all, keys” are passed to the callback (case 1: in the example) - this can obviously never happen for a list with a single key.

Ah, yes, thanks @per . I was assuming too much, i.e. that you had registered both get_next_object() and find_next_object().

Also, if @jluo020 is looking for a way to retrieve multiple different lists, not just the entries from one list, or from nested lists using a the MAAPI or CDB API, I believe using cdb_get_values() together with cdb_num_instances() and traverse on instance integers (type C_CDBBEGIN) could be what @jluo020 is looking for. If so, see confd_lib_cdb(3) man page under cdb_get_values() for an example.

@cohult thanks, let me specify the scenario: if we have 20 instances in(one key range: 1…20), the requirement is i can get multiple list(e.g 5 entries) from any start point(e.g key is 10). in client, firstly i use maapi_find_next() to specify the start point(update the maapi_cursor), then use maapi_get_objects() to get 5 entries.
in CONFD server , i implement find_next_object() , i expect that i can receive the start key in the parameters (enum confd_find_next_type type, confd_value_t *keys, int nkeys), but the nkeys is always 0, always traverse the list from the first entry , not the key 10.
Do you have any idea?

Do you implement get_next_object() ?

And since you seem to only have one key, why do you believe you need find_next_object()?

@jluo020, it sounds like what you are doing should work just fine (as I mentioned earlier, find_next() / find_next_object() is useful also for lists with a single key) - in particular, maapi_find_next() maps pretty much directly to those callbacks, and off the top of my head, the only reason I can think of that you would get nkeys 0 in find_next_object() as a result of calling maapi_find_next() is that maapi_find_next() was given n_inkeys as 0.

Are you sure that the find_next_object() invocation is the result of the maapi_find_next() call? If you can run both the maapi application and your data provider with debug level (for confd_init()) CONFD_PROTO_TRACE, the relation between maapi calls and callback invocations should be clear. That trace will also show the exact arguments for the maapi calls and the callback invocations, though the format may be a bit non-obvious.

I tested this use case myself, and a maapi_find_next() will result in one find_next_object() invocation where nkeys = 1.
If followed (before the object cache timeout) by a maapi_get_objects() the data will be taken from the object cache and no data provider callbacks will be invoked.

I.e. a call to maapi_find_next() + maapi_get_objects() will result in one single find_next_objects() invocation.

Thanks for verifying! That’s what I would expect (modulo your object cache timeout caveat, and of course that the reply to the find_next_object() invocation returns enough list entries to satisfy the maapi_get_objects() request). So, @jluo020 is doing “something wrong”, but without further info we can’t really say what.

For completeness, I’ll add my test to the “reference example” pile.

Implemented a simple mapping from the IETF interfaces YANG model to a ifstatus.yang model that implements a subset of the interfaces state.

ifstatus.yang:

module ifstatus {
  namespace "http://tail-f.com/ns/example/if";
  prefix ifs;

  import tailf-common {
    prefix tailf;
  }

  revision 2019-07-22 {
    description "Initial revision.";
  }

  container interfaces-state {
    list interface {
      key name;
      config false;
      tailf:callpoint ifs;
      leaf name {
        type string;
      }
      container statistics {
	leaf in-octets {
	  type uint64;
	  mandatory true;
	}
	leaf in-unicast-pkts {
	  type uint64;
	  mandatory true;
	}
	leaf in-errors {
	  type uint32;
	  mandatory true;
	}
	leaf in-discards {
	  type uint32;
	  mandatory true;
	}
	leaf out-octets {
	  type uint64;
	  mandatory true;
	}
	leaf out-unicast-pkts {
	  type uint64;
	  mandatory true;
	}
	leaf out-errors {
	  type uint32;
	  mandatory true;
	}
	leaf out-discards {
	  type uint32;
	  mandatory true;
	}
      }
    }
  }
}

The ifstatus YANG model has a data provider application that get the data it provides from the IETF Interfaces YANG model data which is stored in the CDB operational datastore.

The data provider registers the get_elem(), get_next(), get_next_object(), and find_next_object() and get the data from the CDB operational datastore using for example cdb_get_values(). For this example you would only need the get_elem() (mandatory) and find_next_object() callbacks registered. Comment out the rest if you dare.

The data provider application:

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/poll.h>

#include <confd_lib.h>
#include <confd_dp.h>
#include <confd_cdb.h>
#include "ietf-interfaces.h"
#include "ifstatus.h"

#define USE_CASE_ASSUME_NOTHING 0
#define USE_CASE_OPTIMAL_FOR_LARGE_LISTS 1
#define USE_CASE_ALL_IN_DEMO 2

/* Our daemon context as a global variable */
static struct confd_daemon_ctx *dctx;
static int ctlsock;
static int workersock;
static int cdbsock;

static int s_init(struct confd_trans_ctx *tctx)
{
  confd_trans_set_fd(tctx, workersock);
  return CONFD_OK;
}

static int get_next(struct confd_trans_ctx *tctx,
                    confd_hkeypath_t *keypath,
                    long next)
{
  confd_value_t v;
  confd_tag_value_t tv[3];
  
  if (next == -1) {  /* first call */
    next = 0;
  }

  if (cdb_num_instances(cdbsock, "/if:interfaces-state/interface") <= next) {
    /* we have reached the end of the list */
    confd_data_reply_next_key(tctx, NULL, -1, -1);
    return CONFD_OK;
  }
  
  /* get the key */
  CONFD_SET_TAG_CDBBEGIN(&tv[0], if_interface, if__ns, next);
  CONFD_SET_TAG_NOEXISTS(&tv[1], if_name);
  CONFD_SET_TAG_XMLEND(&tv[2], if_interface, if__ns);

  if (cdb_get_values(cdbsock, tv, 3, "/if:interfaces-state") != CONFD_OK) {
    /* key not found */
    confd_data_reply_next_key(tctx, NULL, -1, -1);
    return CONFD_OK;
  }

  /* return the key from the list entry and 'next + 1' for next entry */
  CONFD_SET_STR(&v, CONFD_GET_CBUFPTR(CONFD_GET_TAG_VALUE(&tv[1])));
  confd_data_reply_next_key(tctx, &v, 1, next + 1);
  confd_free_value(&v); /* name must be freed since it's a C_BUF */
  return CONFD_OK;
}


static int get_elem(struct confd_trans_ctx *tctx,
                    confd_hkeypath_t *keypath)
{
  confd_value_t v;
  
  if (cdb_get(cdbsock, &v, "%h", keypath) != CONFD_OK) {
    confd_data_reply_not_found(tctx);
  } else {
    confd_data_reply_value(tctx, &v);
  }

  confd_free_value(&v); /* name must be freed since it's a C_BUF, other values is a nop */
  return CONFD_OK;
}

static int num_instances(struct confd_trans_ctx *tctx,
                         confd_hkeypath_t *keypath)
{
  confd_value_t v;
  
  CONFD_SET_INT32(&v, cdb_num_instances(cdbsock, "/if:interfaces-state/interface"));
  confd_data_reply_value(tctx, &v);
  return CONFD_OK;
}

static int ifs_max_obj_tvsize = 11;

static int get_object(struct confd_trans_ctx *tctx,
			   confd_hkeypath_t *keypath)
{
  confd_tag_value_t itv[ifs_max_obj_tvsize+2], tv[ifs_max_obj_tvsize];
  int pos, j;

  pos = cdb_index(cdbsock, "%s{%x}", "/if:interfaces-state/interface", &keypath->v[0][0]);

  if (pos < 0) {
    /* No list entry with a maching key */
    confd_data_reply_not_found(tctx);
    return CONFD_OK;
  }

  j = 0;
  CONFD_SET_TAG_CDBBEGIN(&itv[j], if_interface, if__ns, pos); j++;
  CONFD_SET_TAG_NOEXISTS(&itv[j], if_name); j++;
  CONFD_SET_TAG_XMLBEGIN(&itv[j], if_statistics, if__ns); j++;
  CONFD_SET_TAG_NOEXISTS(&itv[j], if_in_octets); j++;
  CONFD_SET_TAG_NOEXISTS(&itv[j], if_in_unicast_pkts); j++;
  CONFD_SET_TAG_NOEXISTS(&itv[j], if_in_errors); j++;
  CONFD_SET_TAG_NOEXISTS(&itv[j], if_in_discards); j++;
  CONFD_SET_TAG_NOEXISTS(&itv[j], if_out_octets); j++;
  CONFD_SET_TAG_NOEXISTS(&itv[j], if_out_unicast_pkts); j++;
  CONFD_SET_TAG_NOEXISTS(&itv[j], if_out_errors); j++;
  CONFD_SET_TAG_NOEXISTS(&itv[j], if_out_discards); j++;
  CONFD_SET_TAG_XMLEND(&itv[j], if_statistics, if__ns); j++;
  CONFD_SET_TAG_XMLEND(&itv[j], if_interface, if__ns); j++;
  
  if (cdb_get_values(cdbsock, itv, j, "/if:interfaces-state") != CONFD_OK)
    confd_fatal("cdb_get_values() from /if:interfaces-state failed\n");

  j = 0;
  j++; /* get rid of the begin list tag */
  CONFD_SET_TAG_VALUE(&tv[0], itv[j].tag.tag, &itv[j].v); j++;
  CONFD_SET_TAG_VALUE(&tv[1], itv[j].tag.tag, &itv[j].v); j++;
  CONFD_SET_TAG_VALUE(&tv[2], itv[j].tag.tag, &itv[j].v); j++;
  CONFD_SET_TAG_VALUE(&tv[3], itv[j].tag.tag, &itv[j].v); j++;
  CONFD_SET_TAG_VALUE(&tv[4], itv[j].tag.tag, &itv[j].v); j++;
  CONFD_SET_TAG_VALUE(&tv[5], itv[j].tag.tag, &itv[j].v); j++;
  CONFD_SET_TAG_VALUE(&tv[6], itv[j].tag.tag, &itv[j].v); j++;
  CONFD_SET_TAG_VALUE(&tv[7], itv[j].tag.tag, &itv[j].v); j++;
  CONFD_SET_TAG_VALUE(&tv[8], itv[j].tag.tag, &itv[j].v); j++;
  CONFD_SET_TAG_VALUE(&tv[9], itv[j].tag.tag, &itv[j].v); j++;
  CONFD_SET_TAG_VALUE(&tv[10], itv[j].tag.tag, &itv[j].v); j++;
  j++; /* dispose of the end list tag (here only for clarity) */
  
  confd_data_reply_tag_value_array(tctx, &tv[0], ifs_max_obj_tvsize);
  confd_free_value(&tv[0].v); /* name must be freed since it's a C_BUF */
  return CONFD_OK;
}
  
static int max_nobjs = 100;

static int get_next_object(struct confd_trans_ctx *tctx,
			   confd_hkeypath_t *keypath, long next)
{
  int pos, n_list_entries, nobj, i, j;
  confd_tag_value_t *tv, *itv;
  struct confd_tag_next_object *tobj;

  if (next == -1) {  /* first call */
    pos = 0;
  } else {
    pos = next;
  }

  if ((n_list_entries = cdb_num_instances(cdbsock, "/if:interfaces-state/interface")) <= pos) {
    /* we have reached the end of the list */
    confd_data_reply_next_key(tctx, NULL, -1, -1);
    return CONFD_OK;
  }
  
  if (n_list_entries - pos > max_nobjs) {
    nobj = max_nobjs;
  } else {
    nobj = n_list_entries - pos;
  }
  
  //fprintf(stderr, "\nifs_max_obj_tvsize %d n_list_entries %d nobj %d pos %d\n", ifs_max_obj_tvsize, n_list_entries, nobj, pos);
  
  /* get the list entries */
  itv = (confd_tag_value_t *) malloc(sizeof(confd_tag_value_t) * nobj * (ifs_max_obj_tvsize + 2)); /* "+ 2" for the begin and end tags */
  j = 0;
  for (i = 0; i < nobj; i++) {
    CONFD_SET_TAG_CDBBEGIN(&itv[j], if_interface, if__ns, pos+i); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_name); j++;
    CONFD_SET_TAG_XMLBEGIN(&itv[j], if_statistics, if__ns); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_in_octets); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_in_unicast_pkts); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_in_errors); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_in_discards); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_out_octets); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_out_unicast_pkts); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_out_errors); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_out_discards); j++;
    CONFD_SET_TAG_XMLEND(&itv[j], if_statistics, if__ns); j++;
    CONFD_SET_TAG_XMLEND(&itv[j], if_interface, if__ns); j++;
  }

  if (cdb_get_values(cdbsock, itv, nobj * (ifs_max_obj_tvsize + 2), "/if:interfaces-state") != CONFD_OK)
    confd_fatal("cdb_get_values() from /if:interfaces-state failed\n");

  tobj = malloc(sizeof(struct confd_tag_next_object) * (max_nobjs + 1));
  tv = (confd_tag_value_t *) malloc(sizeof(confd_tag_value_t) * max_nobjs * ifs_max_obj_tvsize);
  
  /* create reply */
  j = 0;
  for (i = 0; i < nobj; i++) {
    tobj[i].tv = &tv[i * ifs_max_obj_tvsize];
    j++; /* get rid of the begin list tag */
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[0]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[1]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[2]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[3]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[4]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[5]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[6]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[7]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[8]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[9]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[10]), itv[j].tag.tag, &itv[j].v); j++;
    j++; /* dispose of the end list tag */
    tobj[i].n = ifs_max_obj_tvsize;
    tobj[i].next = (long)pos+i+1;
  }

  if (pos + i == n_list_entries)
    tobj[i++].tv = NULL; /* indicate no more list entries */

  /* reply */
  confd_data_reply_next_object_tag_value_arrays(tctx, tobj, i, 0);
  for (i = 0; pos + i < nobj; i++) {
    confd_free_value(CONFD_GET_TAG_VALUE(&(tobj[i].tv[0]))); /* name must be freed since it's a C_BUF */
  }
  free(itv);
  free(tv);
  free(tobj);
  return CONFD_OK;
}

static int find_next(struct confd_trans_ctx *tctx,
                     confd_hkeypath_t *kp,
                     enum confd_find_next_type type,
                     confd_value_t *keys, int nkeys)
{
  confd_value_t v;
  confd_tag_value_t tv[3];
  int pos = -1;
  
  switch (nkeys) {
  case 0:
    pos = 0; /* first call */
    break;
  case 1:
    switch (type) {
    case CONFD_FIND_SAME_OR_NEXT:
      if((pos = cdb_index(cdbsock, "%s{%x}", "/if:interfaces-state/interface", keys)) < 0) {
	pos = cdb_next_index(cdbsock, "%s{%x}", "/if:interfaces-state/interface", keys);
      }
      break;
    case CONFD_FIND_NEXT:
      pos = cdb_next_index(cdbsock, "%s{%x}", "/if:interfaces-state/interface", keys);
      break;
    default:
      confd_fatal("invalid find next type");
      break;
    }
    
    if (pos < 0) {
      /* key does not exist */
      confd_data_reply_next_key(tctx, NULL, -1, -1);
      return CONFD_OK;
    }  
    break;
  default:
    confd_fatal("invalid number of keys");
    break;
  }

  /* get the key */
  CONFD_SET_TAG_CDBBEGIN(&tv[0], if_interface, if__ns, pos);
  CONFD_SET_TAG_NOEXISTS(&tv[1], if_name);
  CONFD_SET_TAG_XMLEND(&tv[2], if_interface, if__ns);

  if (cdb_get_values(cdbsock, tv, 3, "/if:interfaces-state") != CONFD_OK) {
    /* key not found in the unlikely event that it was deleted after our 
       cdb_index() check */
    confd_data_reply_next_key(tctx, NULL, -1, -1);
    return CONFD_OK;
  }
  
  CONFD_SET_STR(&v, CONFD_GET_CBUFPTR(CONFD_GET_TAG_VALUE(&tv[1])));

  /* reply */
  confd_data_reply_next_key(tctx, &v, 1, pos+1);
  confd_free_value(&v);
  
  return CONFD_OK;
}

static int find_next_object(struct confd_trans_ctx *tctx,
			    confd_hkeypath_t *keypath,
			    enum confd_find_next_type type,
                            confd_value_t *keys, int nkeys)
{
  int pos = 0, n_list_entries, nobj, i, j;
  confd_tag_value_t *tv, *itv;
  struct confd_tag_next_object *tobj;

  switch (nkeys) {
  case 0:
    pos = 0; /* first call */
    break;
  case 1:
    switch (type) {
    case CONFD_FIND_SAME_OR_NEXT:
      if((pos = cdb_index(cdbsock, "%s{%x}", "/if:interfaces-state/interface", keys)) < 0) {
	pos = cdb_next_index(cdbsock, "%s{%x}", "/if:interfaces-state/interface", keys);
      }
      break;
    case CONFD_FIND_NEXT:
      pos = cdb_next_index(cdbsock, "%s{%x}", "/if:interfaces-state/interface", keys);
      break;
    default:
      confd_fatal("invalid find next type");
      break;
    }
    break;
  default:
    confd_fatal("invalid number of keys");
    break;
  }

  if (pos == -1 || (n_list_entries = cdb_num_instances(cdbsock, "/if:interfaces-state/interface")) <= pos) {
    /* we have reached the end of the list */
    confd_data_reply_next_key(tctx, NULL, -1, -1);
    return CONFD_OK;
  }
  
  if (n_list_entries - pos > max_nobjs) {
    nobj = max_nobjs;
  } else {
    nobj = n_list_entries - pos;
  }
  
  //fprintf(stderr, "\nifs_max_obj_tvsize %d n_list_entries %d nobj %d pos %d nkeys %d\n", ifs_max_obj_tvsize, n_list_entries, nobj, pos, nkeys);
  
  /* get the list entries */
  itv = (confd_tag_value_t *) malloc(sizeof(confd_tag_value_t) * nobj * (ifs_max_obj_tvsize + 2)); /* "+ 2" for the begin and end tags */
  j = 0;
  for (i = 0; i < nobj; i++) {
    CONFD_SET_TAG_CDBBEGIN(&itv[j], if_interface, if__ns, pos+i); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_name); j++;
    CONFD_SET_TAG_XMLBEGIN(&itv[j], if_statistics, if__ns); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_in_octets); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_in_unicast_pkts); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_in_errors); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_in_discards); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_out_octets); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_out_unicast_pkts); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_out_errors); j++;
    CONFD_SET_TAG_NOEXISTS(&itv[j], if_out_discards); j++;
    CONFD_SET_TAG_XMLEND(&itv[j], if_statistics, if__ns); j++;
    CONFD_SET_TAG_XMLEND(&itv[j], if_interface, if__ns); j++;
  }

  if (cdb_get_values(cdbsock, itv, nobj * (ifs_max_obj_tvsize + 2), "/if:interfaces-state") != CONFD_OK)
    confd_fatal("cdb_get_values() from /if:interfaces-state failed\n");

  tobj = malloc(sizeof(struct confd_tag_next_object) * (max_nobjs + 1));
  tv = (confd_tag_value_t *) malloc(sizeof(confd_tag_value_t) * max_nobjs * ifs_max_obj_tvsize);
  
  /* create reply */
  j = 0;
  for (i = 0; i < nobj; i++) {
    tobj[i].tv = &tv[i * ifs_max_obj_tvsize];
    j++; /* get rid of the begin list tag */
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[0]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[1]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[2]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[3]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[4]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[5]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[6]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[7]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[8]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[9]), itv[j].tag.tag, &itv[j].v); j++;
    CONFD_SET_TAG_VALUE(&(tobj[i].tv[10]), itv[j].tag.tag, &itv[j].v); j++;
    j++; /* dispose of the end list tag */
    tobj[i].n = ifs_max_obj_tvsize;
    tobj[i].next = (long)pos+i+1;
  }

  if (pos + i >= n_list_entries)
    tobj[i++].tv = NULL; /* indicate no more list entries */

  /* reply */
  confd_data_reply_next_object_tag_value_arrays(tctx, tobj, i, 0);
  for (i = 0; pos + i < nobj; i++) {
    confd_free_value(CONFD_GET_TAG_VALUE(&(tobj[i].tv[0]))); /* name must be freed since it's a C_BUF */
  }
  free(itv);
  free(tv);
  free(tobj);
  return CONFD_OK;
}

int main(int argc, char *argv[])
{
  struct sockaddr_in addr;
  int c, debuglevel = CONFD_DEBUG;
  int use_case_type = USE_CASE_ASSUME_NOTHING;
  struct confd_trans_cbs trans;
  struct confd_data_cbs data;

  while ((c = getopt(argc, argv, "noaxzdpts")) != EOF) {
    switch(c) {
    case 'n':
      use_case_type = USE_CASE_ASSUME_NOTHING;
      break;
    case 'o':
      use_case_type = USE_CASE_OPTIMAL_FOR_LARGE_LISTS;
      break;
    case 'a':
      use_case_type = USE_CASE_ALL_IN_DEMO;
      break;  
    case 'x':
      max_nobjs = atoi(optarg);
      break;
    case 'z':
      ifs_max_obj_tvsize = atoi(optarg);
      break;
    case 'd':
      debuglevel = CONFD_DEBUG;
      break;
    case 'p':
      debuglevel = CONFD_PROTO_TRACE;
      break;
    case 't':
      debuglevel = CONFD_TRACE;
      break;
    case 's':
      debuglevel = CONFD_SILENT;
      break;
    }
  }
  
  /* Initialize confd library */
  confd_init("ifstatus-cdb", stderr, debuglevel);

  addr.sin_addr.s_addr = inet_addr("127.0.0.1");
  addr.sin_family = AF_INET;
  addr.sin_port = htons(CONFD_PORT);

  if (confd_load_schemas((struct sockaddr*)&addr,
			 sizeof (struct sockaddr_in)) != CONFD_OK)
    confd_fatal("Failed to load schemas from confd\n");
  
  if ((dctx = confd_init_daemon("ifstatus-cdb")) == NULL)
    confd_fatal("Failed to initialize daemon\n");

  /* Create and connect the control and worker sockets */
  if ((ctlsock = socket(PF_INET, SOCK_STREAM, 0)) < 0 )
    confd_fatal("Failed to open ctlsocket\n");
  if (confd_connect(dctx, ctlsock, CONTROL_SOCKET, (struct sockaddr*)&addr,
		    sizeof (struct sockaddr_in)) < 0)
    confd_fatal("Failed to confd_connect() to confd \n");

  if ((workersock = socket(PF_INET, SOCK_STREAM, 0)) < 0 )
    confd_fatal("Failed to open workersocket\n");
  if (confd_connect(dctx, workersock, WORKER_SOCKET,(struct sockaddr*)&addr,
		    sizeof (struct sockaddr_in)) < 0)
    confd_fatal("Failed to confd_connect() to confd \n");

  /* Register callbacks */
  memset(&trans, 0, sizeof(trans));
  trans.init = s_init;
  if (confd_register_trans_cb(dctx, &trans) == CONFD_ERR)
    confd_fatal("Failed to register trans cb\n");

  memset(&data, 0, sizeof (struct confd_data_cbs));
  switch (use_case_type) {
  case USE_CASE_OPTIMAL_FOR_LARGE_LISTS:
    /* assuming large lists and not the content of 
       individual leafs are typically requested */
    data.num_instances = num_instances;
    data.get_object = get_object;
    data.find_next = find_next;
    data.find_next_object = find_next_object;
    break;
  case USE_CASE_ALL_IN_DEMO:
    /* going "all in" as a demo */
    data.get_elem = get_elem;
    data.get_object = get_object;
    data.num_instances = num_instances;
    data.get_next = get_next;
    data.get_next_object = get_next_object;
    data.find_next = find_next;
    data.find_next_object = find_next_object;
    break;
  default: /* USE_CASE_ASSUME_NOTHING */
    /* assuming nothing about the use case */
    data.get_elem = get_elem;
    data.num_instances = num_instances;
    data.get_object = get_object;
    data.find_next = find_next;
    data.find_next_object = find_next_object;
    break;
  }
  strcpy(data.callpoint, ifs__callpointid_ifs);
  if (confd_register_data_cb(dctx, &data) == CONFD_ERR)
    confd_fatal("Failed to register data cb\n");

  if (confd_register_done(dctx) != CONFD_OK)
    confd_fatal("Failed to complete registration \n");

  /* Start a CDB session towards the CDB operational datastore */
  if ((cdbsock = socket(PF_INET, SOCK_STREAM, 0)) < 0)
    confd_fatal("Failed to create the CDB socket");
  if (cdb_connect(cdbsock, CDB_DATA_SOCKET, (struct sockaddr *)&addr,
		  sizeof(struct sockaddr_in)) < 0)
    confd_fatal("Failed to connect to ConfD CDB");
  if (cdb_start_session(cdbsock, CDB_OPERATIONAL) != CONFD_OK)
    confd_fatal("Failed to start a CDB session\n");
  if (cdb_set_namespace(cdbsock, if__ns) != CONFD_OK)
    confd_fatal("Failed to set namespace\n");
  
  while(1) {
    struct pollfd set[2];
    int ret;

    set[0].fd = ctlsock;
    set[0].events = POLLIN;
    set[0].revents = 0;

    set[1].fd = workersock;
    set[1].events = POLLIN;
    set[1].revents = 0;

    if (poll(set, sizeof(set)/sizeof(set[0]), -1) < 0) {
      perror("Poll failed:");
      continue;
    }

    /* Check for I/O */
    if (set[0].revents & POLLIN) {
      if ((ret = confd_fd_ready(dctx, ctlsock)) == CONFD_EOF) {
	confd_fatal("Control socket closed\n");
      } else if (ret == CONFD_ERR && confd_errno != CONFD_ERR_EXTERNAL) {
	confd_fatal("Error on control socket request: %s (%d): %s\n",
		    confd_strerror(confd_errno), confd_errno, confd_lasterr());
      }
    }
    if (set[1].revents & POLLIN) {
      if ((ret = confd_fd_ready(dctx, workersock)) == CONFD_EOF) {
	confd_fatal("Worker socket closed\n");
      } else if (ret == CONFD_ERR && confd_errno != CONFD_ERR_EXTERNAL) {
	confd_fatal("Error on worker socket request: %s (%d): %s\n",
		    confd_strerror(confd_errno), confd_errno, confd_lasterr());
      }
    }
  }
}

The MAAPI client that get some list entries from the ifstatus.yang list using maapi_get_next() and maapi_get_objects() that trigger the find_next_object() callback:

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

#include <confd_lib.h>
#include <confd_maapi.h>
#include "ietf-interfaces.h"
#include "ifstatus.h"

static void print_value_array(confd_value_t vs[], int n)
{
    char tmpbuf[BUFSIZ];
    int i;

    for (i=0; i<n; i++) {
      confd_pp_value(tmpbuf, BUFSIZ, &vs[i]);
      printf("%s ", tmpbuf);
    }
}

int main(int argc, char *argv[])
{
  struct sockaddr_in addr;
  struct maapi_cursor mc;
  int c, i, thandle, maapisock, values_per_entry, nobj;
  int debuglevel = CONFD_DEBUG, entries_per_request = 5;
  confd_value_t *v, inkeys[1];
  struct confd_cs_node *object;
  struct confd_ip ip;
  const char *groups[] = { "admin" }, *context = "system";
  char *inkey = "lo1";
  
  while ((c = getopt(argc, argv, "k:e:dpts")) != EOF) {
    switch(c) {
    case 'k':
      inkey = optarg;
      break;
    case 'e':
      entries_per_request = atoi(optarg);
      break;
    case 'd':
      debuglevel = CONFD_DEBUG;
      break;
    case 'p':
      debuglevel = CONFD_PROTO_TRACE;
      break;
    case 't':
      debuglevel = CONFD_TRACE;
      break;
    case 's':
      debuglevel = CONFD_SILENT;
      break;
    }
  }
 
  confd_init("maapi-get-obj", stderr, debuglevel);

  addr.sin_addr.s_addr = inet_addr("127.0.0.1");
  addr.sin_family = AF_INET;
  addr.sin_port = htons(CONFD_PORT);

  if (confd_load_schemas((struct sockaddr*)&addr,
			 sizeof (struct sockaddr_in)) != CONFD_OK)
    confd_fatal("Failed to load schemas from confd\n");

  object = confd_cs_node_cd(NULL, "/ifs:interfaces-state/interface");
  values_per_entry = confd_max_object_size(object);
  
  if ((maapisock = socket(PF_INET, SOCK_STREAM, 0)) < 0)
    confd_fatal("Failed to create the MAAPI socket");
  if (maapi_connect(maapisock, (struct sockaddr*)&addr,
		    sizeof (struct sockaddr_in)) < 0)
    confd_fatal("Failed to confd_connect() to confd \n");

  ip.af = AF_INET;
  inet_pton(AF_INET, "127.0.0.1", &ip.ip.v4);
  
  if ((maapi_start_user_session(maapisock, "admin", context,groups,
				sizeof(groups) / sizeof(*groups),
				&ip,
				CONFD_PROTO_TCP) != CONFD_OK)) {
    confd_fatal("Failed to start user session");
  }

  if ((thandle = maapi_start_trans(maapisock,CONFD_OPERATIONAL,
			      CONFD_READ)) < 0) {
    confd_fatal("Failed to start trans\n");
  }
  
  if(maapi_init_cursor(maapisock, thandle, &mc,
		       "/ifs:interfaces-state/interface") != CONFD_OK)
    confd_fatal("maapi_init_cursor() failed\n");

  CONFD_SET_STR(&inkeys[0], &inkey[0]);
  if (maapi_find_next(&mc, CONFD_FIND_SAME_OR_NEXT, inkeys, 1) != CONFD_OK)
    confd_fatal("maapi_find_next() failed\n");

  if(mc.n == 0) {
    fprintf(stderr, "Key \"%s\" not found\n", inkey);
    exit(0);
  }
  
  v = malloc(sizeof(confd_value_t) * values_per_entry * entries_per_request);
  nobj = entries_per_request;
  if (maapi_get_objects(&mc, v, values_per_entry, &nobj) < 0)
    confd_fatal("maapi_get_objects() failed\n");

  if(nobj == 0) {
    fprintf(stderr, "No entries after \"%s\"\n", inkey);
    exit(0);
  }
  
  for (i = 0; i < nobj; i++) {
    printf("%d: ", i);
    print_value_array(&v[i*values_per_entry], values_per_entry);
    printf("\n");
  }
  exit(0);
}

A simple script to generate some test data:

#!/usr/bin/env python

import sys
import time

def print_config(str, str2):
    print """<config xmlns="http://tail-f.com/ns/config/1.0">
    <interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces" xmlns:ianaift="urn:ietf:params:xml:ns:yang:iana-if-type">
    <interface>
    <name>lo1</name>
    <type>ianaift:softwareLoopback</type>
    <enabled>true</enabled>
    </interface>
    %s
    </interfaces>
    <interfaces-state xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces" xmlns:ianaift="urn:ietf:params:xml:ns:yang:iana-if-type">
    <interface>
    <name>lo1</name>
    <type>ianaift:softwareLoopback</type>
    <admin-status>up</admin-status>
    <oper-status>up</oper-status>
    <if-index>1</if-index>
    <statistics>
    <discontinuity-time>
    2019-07-22T03:00:00+00:00
    </discontinuity-time>
    <in-octets>0</in-octets>
    <in-unicast-pkts>0</in-unicast-pkts>
    <in-broadcast-pkts>0</in-broadcast-pkts>
    <in-multicast-pkts>0</in-multicast-pkts>
    <in-discards>0</in-discards>
    <in-errors>0</in-errors>
    <in-unknown-protos>0</in-unknown-protos>
    <out-octets>0</out-octets>
    <out-unicast-pkts>0</out-unicast-pkts>
    <out-broadcast-pkts>0</out-broadcast-pkts>
    <out-multicast-pkts>0</out-multicast-pkts>
    <out-discards>0</out-discards>
    <out-errors>0</out-errors>
    </statistics>
    </interface>
    %s
    </interfaces-state>
    </config>
    """%(str,str2)

def gen_data(n):
    str = ""
    str2 = ""
    for i in xrange(0,n):
        str += """
        <interface>
        <name>eth%03d</name>
        <type>ianaift:ethernetCsmacd</type>
        <enabled>true</enabled>
        </interface>
        """%(i)
        
        str2 += """
        <interface>
        <name>eth%03d</name>
        <type>ianaift:ethernetCsmacd</type>
        <admin-status>up</admin-status>
        <oper-status>up</oper-status>
        <if-index>%d</if-index>
        <phys-address>00:01:02:03:04:06</phys-address>
        <statistics>
        <discontinuity-time>
        2019-07-22T03:00:00+00:00
        </discontinuity-time>
        <in-octets>%d</in-octets>
        <in-unicast-pkts>%d</in-unicast-pkts>
        <in-broadcast-pkts>%d</in-broadcast-pkts>
        <in-multicast-pkts>%d</in-multicast-pkts>
        <in-discards>%d</in-discards>
        <in-errors>%d</in-errors>
        <in-unknown-protos>%d</in-unknown-protos>
        <out-octets>%d</out-octets>
        <out-unicast-pkts>%d</out-unicast-pkts>
        <out-broadcast-pkts>%d</out-broadcast-pkts>
        <out-multicast-pkts>%d</out-multicast-pkts>
        <out-discards>%d</out-discards>
        <out-errors>%d</out-errors>
        </statistics>
        </interface>
        """%(i,i+2,i,i,i,i,i,i,i,i,i,i,i,i,i)
    print_config(str, str2)

def parse_num(str):
    if str[:2] == '2^':
        return pow(2,parse_num(str[2:]))
    return int(str)

gen_data(parse_num(sys.argv[1]))

After building the YANG models and applications above you can for example generate some test data
$ ./cdbgen.py 1000 &gt; test.xml
Load it into the CDB operational datastore defined by the IETF interfaces YANG model
$ confd_load -O -dd -m -l test.xml
Run the maapi_get_next() maapi_get_object() client application to trigger the find_get_next()
$ ./maapi-get-obj -k eth100 -e 5

And/or run some netconf-console, confd_load, curl, etc. tests to exercise some some other interfaces and API calls.

@cohult @per, thanks for your great support.

I.e. a call to maapi_find_next() + maapi_get_objects() will result in one single find_next_objects() invocation.

this is what i’m seeing. the maapi_get_objects() actually get values from cache.
i’m sure that the nekeys is 1 in maapi_find_next()

i’ll open the trace to debug. then update my findings . thanks

I updated the example data provider application above as there was an issue with the (lack of) handling of the find “type”, and the deallocation of C_BUFs.

For future reference I added the other callbacks() that are relevant together with an “ultimate” reference setup for this particular use-case. See main() where the callbacks are registered with ConfD by the data provider application.

Sorry , i have a question, now i use maapi_find_next to find the next key of one specified point, for example, there are 100 entries, the given start is 5, and then i use maapi_get_objects to get the subsequent 50(so the returning entry should be from 6 to 56),
As we said, actually when calling maapi_find_next , confd will return and cache the data. and maapi_get_objects just get data from cache
My problem is now although i specify the inkeys as 5 and n_inkeys as 1 in maapi_find_next(this part resided in NCS ) , but confd callpoints–find_next_object can’t receive the correct parameter, the nkeys always is 0, so the confd can’t traverse all entries from the specified start point, always from the first one. it’s not my expectation.
Can you help to address it?

(And from the trace in NCS , i can see when maapi_find_next is called, there is no related parameter(e.g. the start point 5) in the netconf.trace, the confd just firstly return all keys after that interaction, then one by one require & reply the entries from 6 to 56.)

Another question, can i update maapi_cursor by my application rather than calling maapi api , just like,
mc.n = 1;
mc.keys[0] = inkeys[0];

It seems it didn’t work, probably i missed something

Your question is really about NCS/NSO, which is considered out of scope in this forum. But generally speaking, there is no equivalent of either find_next or get_next in the NETCONF protocol, thus you can’t expect that those specific operations “carry over” from maapi calls in NCS/NSO (NETCONF client) to DP callbacks in ConfD (NETCONF server). The result of the maapi calls should be correct though.

Definitely not, and surely there is nothing in the documentation that suggests that you can.

Thanks per, it’s a little complicated.