Most optimal way to read large quantity of statistics from confd with C++ API?

CDB and CDB API
1

Hello,

I have to read large quantity of statistics from confd, something defined like
container counters
{
tailf:info “The container holds the specific statistic of all proccesor threads”;
tailf:cdb-oper;
config false;
list counter
{
tailf:info “Specific statistic”;
config false;
tailf:cdb-oper;
key id;

                            leaf id
                            {
                                            type uint16
                                            {
                                                            range 0..16383;
                                            }
                                            config false;
                                            tailf:cdb-oper;
                                            tailf:info "Counter id";
                            }
                            leaf value
                            {
                                            type uint64;
                                            default 0;
                                            tailf:info "Counter total value";
                                            config false;
                                            tailf:cdb-oper;
                            }
                            leaf version
                            {
                                            type uint16;
                                            default 0;
                                            tailf:info "Counter-version";
                                            config false;
                                            tailf:cdb-oper;
                            }
            }

}

I am using C++ API and the list could contain ~15000 counters or a bit more.
I used the cdb_get_values C++ function and time for retrieving entire list is ~1 second.
Reading 1000 counters takes ~ 70 milliseconds.
Reading 15000 takes ~ 1 sec, so the time is proportional with the number of counters I try to read, so I see that reading bunch of counters is not a solution.

Anyone could suggest an optimal way to read this large quantity of data?

Thanks in advance,
Cris

2

In addition to cdb_get_values( ), there are also cdb_get_object( ) and cdb_get_objects( ) API calls available for bulk read operations.

cdb_get_object( ) reads at most n values from the container or list entry specified by the path, and places them in the values array, which is provided by the caller.

Similar to cdb_get_object(), cdb_get_objects( ) reads multiple entries of a list based on the “instance integer” otherwise given within square brackets in the path - here the path must specify the list without the instance integer. At most n values from each of nobj entries, starting at entry ix, are read and placed in the values array.

Refer to the man pages under confd_lib_cdb for more detailed description of these API calls.

3

Thanks waitai, this helped a bit more.
Reading 1000 counters takes now ~ 40 milliseconds.
Reading 15000 takes ~ 650 miliseconds.
I am using confd3.7.1. Still looking for the optimal way of reading this large quantity of data.

4

Depending on your system performance requirements, CDB may not be the best storage mechanism for working with massive amount of statistics data. You can consider the use of an external database for operational data and continue to use CDB for configuration data.

By the way, ConfD 3.7.1 is almost 5 years old. You may want to consider upgrading to the latest ConfD release and reassess its performance.