How to implement find_next() so that a large list could be traversed quickly?

Below Yang module is designed for operational data.

container files {
                    config false;
                    list file {
                        key file-number;
                        leaf file-number {
                            description "Number of the file";
                            type uint32;
                        }
                        container file-information {
			    description "Information about the file";
                            leaf file-state {
                                description "State of the file";
                                type base:file-state-read;
                            }
                            leaf filling-time {
                                description "The filling timestamp showing when the file is opened";
                                type yang:date-and-time;
			    }
                        leaf transfer-time {
                               description "the transfer date and time"
                               type yang:date-and-time;
                        }
                        leaf is-stored-file {
                            description "Set if the file is stored on disk.";
                            type empty;
                        }
			    leaf is-compressed-file {
                                description "Set if the file is compressed.";
                                type empty;
                            }
                        leaf file-warning {
                                description "Mutually exclusive warnings set for file
                                type enumeration {
                                enum skipped
                                enum overwritten
                                }
                         }                
			}
		    }
 }

We would like to use confd_cli command “show files file xxx file-information file-state” to display the file xxx’s status. The file list has a maximum of 65535 elements and find_next() is design to traverse the list. However, it takes much time to traverse all the elements in the list. (Note: the files are always on a fixed position like this: the file-number=1 is on the position 0; the file-number=2 is on the position 1; the file-number=N is on the position N-1)

int  find_next_nested (struct confd_trans_ctx* tctx,
                                     confd_hkeypath_t* keypath,
                                     enum confd_find_next_type /*type*/,
                                     confd_value_t* keys,
                                     int nkeys)
{
    confd_value_t v;
    int pos = -1;
    std::vector<FileInfo> current_file_info = findFileInfoList(keypath->v[2][0]);
    switch (nkeys) {
    case 0:
        selected_fileInfo.file_number = UINT_MAX;
        if (current_file_info.size() > 0)
            pos = 0;
        else
            pos = -1;
        break;
    case 1:
        pos = CONFD_GET_UINT32(&keys[0]);
        if (static_cast<int>(current_file_info.size())<=pos)
            pos = -1;
        break;
    default:
        confd_trans_seterr(tctx, "invalid number of keys: %d", nkeys);
        return CONFD_ERR;
    }

    if (pos >= 0) {
        CONFD_SET_UINT32(&v, current_file_info[pos].file_number);
        confd_data_reply_next_key(tctx, &v, 1, (long)(pos + 1));
    } else {
        confd_data_reply_next_key(tctx, NULL, -1, -1);
    }

    return CONFD_OK;
}

I tried to change find_next() to make a “jump” in a list traversal, it seems the result doesn’t satisfy our needs.

xxx@netconf-server- 02:52:56> show files file 1
Possible completions:
  100  - Number (running counter) of the file
  1000 - Number (running counter) of the file
  1100 - Number (running counter) of the file
  1200 - Number (running counter) of the file
  1300 - Number (running counter) of the file
  1400 - Number (running counter) of the file
  1500 - Number (running counter) of the file
  1600 - Number (running counter) of the file
  1700 - Number (running counter) of the file
  1800 - Number (running counter) of the file

Could you please give some suggestion how to change find_next() to show any file’s status quickly? Or could I use “tailf:display-when” to forbidden the input of " file-information file-state" in confd-cli? We are using ConfD 6.2.

I don’t think you can do much more in find_next to improve the performance. Have a look at developer log and at your application log - I expect you will see lots of get_next invocations, likely this is what is taking time, and you can’t improve that by changing your find_next.

Hi, mvf,

Yes, I can see lots of find_next() invoked in application log.

I am trying to implement get_next_object to retrieve the entire list in one invoke. However, I am encountering the error:

**"TRACE CALL data get_next_object(thandle=25, /files/file, -1)DEBUG Invalid confd_vtype value: 33"**
**"DEBUG Expect call to return data, cannot return CONFD_OK when no data is sent thandle=25"**
**"Error on worker socket request: Bad protocol usage or unexpected retval (21): Expect call to return data, cannot return CONFD_OK when no data is sent thandle=25"**

Could you help point out which is wrong in my implementation?

    int get_next_object (struct confd_trans_ctx *tctx,
                                             confd_hkeypath_t *keypath,
                                             long next)
    {

      int pos = -1;

      confd_value_t *v;
      struct confd_next_object *obj;
      int i;

      int n_list_entries;
      FileInfo file_info;
      std::vector<FileInfo> curr_file_list = tctx->t_opaque;
      
      n_list_entries = curr_file_list.size();

      if (next == -1)
      {
        if(n_list_entries > 0) {
          pos = 0; 
        }
      }else if (next < n_list_entries) {
        pos = next;
      } else {
        confd_data_reply_next_object_array(tctx, NULL, -1, -1);
        return CONFD_OK;
      }

    v = (confd_value_t *) malloc(sizeof(confd_value_t) * n_list_entries * 8);
      obj = (struct confd_next_object *)malloc(sizeof(struct confd_next_object) * (n_list_entries + 1));

      if (pos != -1) {
        for (i = 0; pos + i < n_list_entries; i++) {
          file_info = curr_file_list[pos + i];
          obj[i].v = &v[i * 8];

          CONFD_SET_UINT32(&(obj[i].v[0]), file_info.file_number);
          CONFD_SET_XMLTAG(&(obj[i].v[1]), streams_file_information, streams__ns);
          CONFD_SET_ENUM_VALUE(&(obj[i].v[2]), get_enum_file_state(file_info.file_state));
          CONFD_SET_DATETIME(&(obj[i].v[3]), get_date_time(file_info.filling_time));
          CONFD_SET_DATETIME(&(obj[i].v[4]), get_date_time(file_info.transfer_time));
          CONFD_SET_BOOL(&(obj[i].v[5]), file_info.is_stored_file);
          CONFD_SET_BOOL(&(obj[i].v[6]), file_info.is_compressed_file);
          CONFD_SET_ENUM_VALUE(&(obj[i].v[7]), file_info.file_warning);
          obj[i].n = 8;
          obj[i].next = -1;
        }
        if (pos + i == n_list_entries)
          obj[i++].v = NULL;
        confd_data_reply_next_object_arrays(tctx, obj, i, 0);
      } else {
        confd_data_reply_next_object_array(tctx, NULL, -1, -1);
      }

      free(v);
      free(obj);
      return CONFD_OK;
    }

Thanks in advance,
sarahwa

I cannot tell much from this, but your code obviously does not correspond to the data model you pasted in the first post: there are no transfer-time and file-warning leaves in the data model, and the leaf is-compressed-file is empty, but you fill in boolean. It does not look like this is what causes libconfd to complain though. I suggest you debug your code to find why the vtype 33 (this is C_XMLEND) appeared there.

Hi mvf,
Thanks for your info. I have updated the yang module in first post.
Besides, I have implemented exists_optional() for is-stored-file and is-compressed-file.
I am a newbie to confd. Could you tell me how to debug the code? Is it fine to set the debug level to CONFD_PROTO_TRACE?

Thanks,
sarahwa

CONFD_PROTO_TRACE would not be very useful to you. What I meant is a standard debugging process - adding logs, using a debugger, whatever works for you; the fact that your application talks to confd makes no difference here.

I still believe you should not use CONFD_SET_BOOL for an empy-typed leaf in a response to get_next_object - correct usage would be CONFD_SET_XMLTAG if the leaf exists, CONFD_SET_NOEXISTS otherwise.

Hi mvf,

I change the get_next_object() for empty-typed leaf as below and it works well.

  if (file_info.is_stored_file)
  {
    CONFD_SET_XMLTAG(&(obj[i].v[5]), streams_is_stored_file, streams__ns);
  } else {
    CONFD_SET_NOEXISTS(&(obj[i].v[5]));
  }

  if (file_info.is_compressed_file)
  {
    CONFD_SET_XMLTAG(&(obj[i].v[6]), streams_is_compressed_file, streams__ns);
  } else {
    CONFD_SET_NOEXISTS(&(obj[i].v[6]));
  }

Thanks very much,
sarahwa