Loading configuration files

[mgr]

We’re using the ‘J’ style CLI. Playing around with the config-mode “load” command and I’m not seeing the behavior I would expect and wondering if there are bugs here…

First off, would be really nice if the config option applied to J style (in fairness it doesn’t mention J style support and it CLEARLY is ignored…I guess the only way to get rid of merge in J style is to do a tag in my clispec, correct?).

Secondly, it seems that “override” and “replace” mode (at least for J style CLI) both do a “merge” and not actually nuke what was there previously. IE, my base-system config defines objects A1, A2, A3, A4. I make and save a config that modifies A1 and A2. Then I make and save a separate config that modifies A3 and A4. Now boot up with a clean (virgin) config. Load saved-config-1, it correctly modifies objects A1 and A2. Now load saved-config-2 and no matter which of “merge”, “override”, or “replace” is used, what I now have is a configuration with A1, A2, A3, and A4 modified. A1 and A2 were not returned to their virgin state (more over, in the saved-config-2, A1 and A2 exist in the virgin config state). The only way I have found (and it’s rather hackish) is to do a rollback to the initial virgin state between loading the first config and loading the second.

That brings up the next point, is there any way from inside the standard J-style CLI to roll back to a virgin state (i.e., empty cdb loaded with just the .xml files in the cdb directory)? I see in the manual that "confd_load -D -m -l cdb/.xml" looks like it will do it from me from linux (and I guess I could wrapper this in a config-mode command myself), just wondering if I’m missing something here…I can do a rollback to the first commit (boot time commit) assuming it hasn’t rolled out of the range of saved commits (unless I want to do infinite storage).

[mgr]

Actually, even the “confd_load -D -m -l cdb/*.xml” isn’t doing the right thing. It deleted some of the objects, but not all leaf nodes that were not part of the original virgin commit.

Specifically, I have

container
list
leaf1
leaf2
leaf3
leaf4

On virgin startup, my .xml files create a bunch of items in the list and pre-configure leaf1 and leaf2 (leaf3 and leaf 4 are optional/default values). Now the user comes along and configures leaf3/leaf4 of several of the list elements. Those changes (to leaf3/leaf4 in all list elements touched) persist through the load commands done above, and likewise persist even after the "confd_load -D -m -l cdb/.xml". This is all on 6.0.

[cohult]

@mgr

Actually, even the “confd_load -D -m -l cdb/*.xml” isn’t doing the right thing. It deleted some of the objects, but not all leaf nodes that were not part of the original virgin commit.

Did you ‘commit’ all the configuration changes you did to the running datastore before you ran the “confd_load -D -m -l cdb/*.xml” command?
From your description it seems like you did not.
Even if you don’t have the candidate datastore enabled, if you in J- or C-style entered configuration mode using ‘terminal’ mode, which is the default mode, you will edit a private copy of the running configuration, no lock is taken.

So your private copy is not affected by the “confd_load -D -m -l cdb/*.xml” command which deleted the existing configuration and loaded your init config to the running datastore.

You can see the diff between your private copy and running cfg from the CLI.
J-style:

% compare running brief

C-style (XR):

# show configuration

A quite useful CLI command that compare your current configuration to the running configuration.

[mgr]

Yes, all changes were committed between steps. I can recreate this.

show configuration # Virgin state shown
configure
load replace saved-1
commit
exit
show configuration # See correct “saved-1” config
config
load replace saved-2
commit
exit
show configuration # See merge of saved-1 and saved-2

Now, drop to a linux shell and do “confd_load -D -m -l cdb/” (my cli is in operational mode, not config, and all commits saved). Now jump back up to my cli I first see:

System message at 2015-09-17 23:21:51…
Commit performed by system via tcp using system.

show configuration # Partial configuration…any items created in saved-1 and saved-2 have been deleted, but any item that existed in the original “virgin config” (i.e., the initial commit/load) are still modified in the config.

[cohult]

What if you do a “confd_load -D -m -l cdb/aaa_init.xml”?
Did all of your configuration get wiped?

[mgr]

Interesting. If I try that, I actually get a fun error:
[root@RFP confd]# confd_load -D -m -l cdb/aaa_init.xml
confd_load: 673: maapi_apply_trans_flags(sock, tid, 0, aflags) failed: external error (19): application error

This is because my validation routine on the objects does not allow a user to delete them and so fails the validation test. Yet if I do:

[root@RFP confd]# confd_load -D -m -l cdb/*.xml

My validation routine does not even get called (at all)…and yet the items in question are not reset back. At this point I am guessing something like the following is happening:

1. load_config starts and begins a transaction
2. The -D gets processed and flags all items for deletion
3. The -l *.xml then causes a number of those list entries to get repopulated with leaf1 & leaf2 fields - this causes these entries to get flagged back to “still exists” and no longer “deleted”…they somehow get relinked to the old leaf3 and leaf4.
4. The transaction commits, and since the list entries are actually unchanged (because they got relinked to their other leaves), my validation routine doesn’t even get called
5. The “merged” config is pushed into operational status

I think the key is that in my .xml file, I don’t populate all the leaves, only a few (the others are optional/default values). My bet is if my .xml directly specified all leaves, I would get what I expect to see, but since not all leaves are specified, we regain that configuration data that should have been purged. At least that’s what it appears to be doing…

[cohult]

@mgr

3. The -l *.xml then causes a number of those list entries to get repopulated with leaf1 & leaf2 fields - this causes these entries to get flagged back to “still exists” and no longer “deleted”…they somehow get relinked to the old leaf3 and leaf4.

Can u show us your YANG implementation of at least this relevant part so that we can try to recreate the scenario you describe?

[cohult]

Looking into this, the issue seems to be the YANG “when” statement.

From the ConfD UG:

The YANG when statement makes its parent data definition statement conditional.This can be problematic in cases where we don’t have control over the order of writing different data nodes. E.g. when loading configuration from a file, the data that will satisfy the when condition may occur after the data that the when applies to, making it impossible to actually write the latter data into the transaction - since the when isn’t satisfied, the data nodes effectively do not exist in the schema.

One workaround is to add the maapi_set_delayed_when() function call as an option in the confd_load tool to optionally be executed before a maapi_delete_all().

From the UG on maapi_set_delayed_when()

int maapi_set_delayed_when(int sock, int thandle, int on);
This function enables (on non-zero) or disables (on == 0) the “delayed when” mode of a transaction. When successful, it returns 1 or 0 as indication of whether “delayed when” was enabled or disabled before the call.
…
When “delayed when” is enabled, it is possible to write to data nodes even though they are conditional on a when that isn’t satisfied.

Here is a diff on what needs to be added to the confd_load tool to enable ‘delayed when’ mode to in your use case be able to delete the leafs that implement a “when” statement:

$ pwd
$CONFD_DIR/src/confd/tools
$ diff -u $CONFD_DIR/src/confd/tools/confd_load.c confd_load.c
--- $CONFD_DIR/src/confd/tools/confd_load.c	2015-06-21 15:03:24.000000000 +0200
+++ confd_load.c	2015-09-18 17:57:42.000000000 +0200
@@ -136,6 +136,7 @@
         "    -N            when saving, do Not include parents to the path\n"
         "    -P  <xpath>   when saving apply this xpath filter\n"
         "    -D            do maapi_delete_all(MAAPI_DEL_ALL) before loading\n"
+        "    -w            do maapi_set_delayed_when(on) before maapi_delete_all()"
         "    -o            when saving include operational data\n"
         "                  when loading ignore operational data\n"
         "    -u  <user>    use this user\n"
@@ -412,6 +413,7 @@
     char *filename = NULL;
     int load_config = 0;
     int delete_config = 0;
+    int delayed_when = 0;
     int attach_init = 0;
     char *username = "system";
     char *context = "system";
@@ -431,7 +433,7 @@
         progname++;
     getcwd(cwd, sizeof(cwd));
 
-    while ((c = getopt(argc, argv, "xnNdjF:CRWSp:P:f:lmrHUaeoMDiu:g:c:tsh?"))
+    while ((c = getopt(argc, argv, "xnNdjF:CRWSp:P:f:lmrHUaeoMDwiu:g:c:tsh?"))
            != -1)
     {
         switch(c) {
@@ -537,6 +539,9 @@
         case 'D':
             delete_config++;
             break;
+        case 'w':
+	    delayed_when++;
+	    break;
         case 'i':
             attach_init++;
             break;
@@ -622,6 +627,9 @@
 
     } else if (load_config) {
         if (delete_config) {
+            if(delayed_when) {
+		maapi_set_delayed_when(sock, tid, 1);
+            }
             OK(maapi_delete_all(sock, tid, MAAPI_DEL_ALL));
         }
         if (path) {

After implementing the above changes to confd_load.c, build the tool with the Makefile.confd_cmd and optionally copy it into the $CONFD_DIR/bin folder.

Now you can run for example “confd_load -D -w -m -l cdb/*.xml” to also delete the leafs with “when” statments