Is partial data replication possible?

Hi,

Let me ask dumb question by giving simple example.

There are two separate/independent systems (system1, system2) and both of them use ConfD as their configuration database. System 1 and system 2 are located in different sites so that they are connected by network.

System 1 and system2 should synchronize some of their configuration data stored in ConfD, not all of their configuration data stored in ConfD because their system ID/name/IP address should be kept different.

In this case, I am told that it is not possible to use ConfD’s Master/Slave data replication because ConfD’s Master/Slave data replication always synchronizes all of data which means even system 1’s and system 2’s ID/Name/IP addresses become synchronized.

How do I synchronize some of System1’s configuration data to system2 in real-time?
Anyone can help this problem?

Partial replication is a DIY, aka “do it yourself”. Synchronizing nodes can be very hard, luckily you get help with that from the ConfD transaction manager. You can use this post + application note + blog post as inspiration:

In the above post, all nodes can write to each other active-active style. If you just want to setup an active-standby replication that would simplify things greatly. Just lock the northbound APIs for writing on that standby node for all clients except the active node.

Thank you so much for precious information !!!
I have not read through application note yet, but in your post and response, I would like to ask additionally.

  1. “This can be done for a single leaf, a subtree, or the entire configuration”.
    => Does it mean that any single container can be synchronized to other active ConfD regardless of its depth and size(the number of its child containers, lists, leaves… and everything)?
    => What about multiple leaves or multiple containers? Is it not possible to synchronize multiple leaves and/or multiple containers from one active to another active?

  2. “If you just want to setup an active-standby replication that would simplify things greatly. Just lock the northbound APIs for writing on that standby node for all clients except the active node.”
    => You don’t mean ConfD’s built-in master-slave HA service but you mean active-active HA service of application note you referred. right?
    In active-active HA configuration, you recommend just to lock the northbound APIs for writing on that standby node for all clients except the active node. right?

  1. You are in control since this is a DIY replication. You can replicate whatever you like to the other node(s).
  2. The definition of active-active is that a manager can write to all nodes in the HA cluster, except for when they are locked for writing. In an active-standby setup the manager can only write to the single active node, not to the standby node(s) in the cluster. In an active-active setup there are no standby nodes.

Thanks again for kind response!

BTW, let me ask the other way of data replication by relaxing real-time synch. requirement.

If I use periodic(ex.every one hour) data synchronization from Node 1 to Node 2 instead of real-time manner and for this goal, I want to send every one hour diff of A.cdb and O.cdb from Node 1 to Node 2.
Is there any way to export/extract diff only between A.cdb1(stored at 00:00:00) A.cdb2(stored at 01:00:00) ?

Otherwise, what do you suggest to send ConfD(A.cdb, O.cdb) diff only from Node 1 to Node 2?

Is there any way to export/extract diff only between A.cdb1(stored at 00:00:00) A.cdb2(stored at 01:00:00) ?

You can for example store the content as XML and do the diff. But I believe there is an easier way to do what you want to achieve, see below

Is there any way to export/extract diff only between A.cdb1(stored at 00:00:00) A.cdb2(stored at 01:00:00) ?

See cdb_get_modifications() in the active-active demo code.
Just as in that demo, you can for example take the output of cdb_get_modifications() and store the content in a buffer to a list of buffers.

Every hour you can then for example do a transaction where you, again just as in the active-active demo, use maapi_set_values() to set each of the stored buffers to the standby node in the same transaction (configuration data).

You may prefer to use cdb_set_values() for operational data.

I got your point but I am more interested in classic(?) way of sending “diff file” to mate system since this way doesn’t require netconf client capability for synchronization.

“You can for example store the content as XML and do the diff. But I believe there is an easier way to do what you want to achieve, see below”
=> If I store “diff” in XML file by doing the diff operation and send this XML file to mate system, How does mate system in running can reflect this “diff” into its own CDB"?
Simply reading in this XML file @mate system is the way to reflect configuration diff to mate’s CDB?
Could you elaborate on how mate system can utilize this diff XML files to synchronize configuration data?

Like I wrote before, why do you need to send an XML diff file to the “mate” ConfD system? Why not just send the ConfD tag value array buffer you get from cdb_get_modifications() to the “mate” ConfD system and load it using maapi_set_values()?

But if you for some reason must provide an XML file diff, then if you put the into for example an NETCONF XML file or a RESTCONF YANG patch XML file where elements have operations you can load them to your “mate” ConfD system over one of those interfaces. For more on NETCONF operations, see RFC 6241 - Network Configuration Protocol (NETCONF) or if you prefer RESTCONF YANG patch see RFC 8072 - YANG Patch Media Type