Confd taking a lot of memory at startup

kabhijeet · October 17, 2024, 11:40am

Hi @cohult, Confd taking a lot of memory at startup causing container to get killed due to OOM. We run confd inside a container that has a memory limit of 4gb which is well within our average usage of 2.5gb.

This issue is seen after we updated the OS on worker nodes to RHEL 9.4, before this, worker nodes were running a non RHEL OS. So this combination of worker node OS and confd is causing high memory usage since the same container image runs fine on older worker node with non-RHEL OS.

On checking the top output in container, it appears that confd is taking 36gb of memory, this is much more than the average usage of 2.5gb

 PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND          
313 root      20   0   36.0g  33.6g   0.0  71.7   0:19.55 S      `- /confd/lib/confd/erts/bin/confd.smp -S 4 -K true -MHe true -- -root /confd/lib/confd -progname confd -- -home /root -- -boot confd -noshell -noinput -foreground -+

In /proc/<pid>/smaps I could see the memory region of huge size reserved by confd

7f3b7bfff000-7f437c000000 rw-p 00000000 00:00 0 
Size:           33554436 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:            33554436 kB
Pss:            33554436 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:  33554436 kB
Referenced:     33554436 kB
Anonymous:      33554436 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
THPeligible:    0
ProtectionKey:         0
VmFlags: rd wr mr mw me ac sd

In cat /proc/<pid>/status also memory reserved is too high

Name:	confd.smp
Umask:	0022
State:	S (sleeping)
Tgid:	310
Ngid:	0
Pid:	310
PPid:	186
TracerPid:	0
Uid:	0	0	0	0
Gid:	0	0	0	0
FDSize:	256
Groups:	0 4000 
NStgid:	310
NSpid:	310
NSpgid:	1
NSsid:	1
VmPeak:	37724308 kB
VmSize:	37715316 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	35198624 kB
VmRSS:	35196608 kB
RssAnon:	35189568 kB
RssFile:	    7040 kB
RssShmem:	       0 kB
VmData:	35231196 kB
VmStk:	     156 kB
VmExe:	    3520 kB
VmLib:	    4692 kB
VmPTE:	   69132 kB
VmSwap:	       0 kB
HugetlbPages:	       0 kB
CoreDumping:	0
THP_enabled:	1
untag_mask:	0xffffffffffffffff
Threads:	24
SigQ:	4/191517
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000000001080
SigCgt:	0000000100000206
CapInh:	0000000000000000
CapPrm:	000001ffffffffff
CapEff:	000001ffffffffff
CapBnd:	000001ffffffffff
CapAmb:	0000000000000000
NoNewPrivs:	0
Seccomp:	0
Seccomp_filters:	0
Speculation_Store_Bypass:	thread vulnerable
SpeculationIndirectBranch:	conditional enabled
Cpus_allowed:	ffff
Cpus_allowed_list:	0-15
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	53
nonvoluntary_ctxt_switches:	35

Is there a way to figure out why confd is taking more memory than usual because of change is worker node OS, Is there a way I can configure confd to be agnostic of worker OS changes ?

Regards
Abhijeet

cohult · October 21, 2024, 12:08pm

Hi,

So far, no issues running ConfD with RHEL 9.x have been seen. So, this issue is unlikely a ConfD OS incompatibility issue if you use a recent ConfD version.
If the only other change is in the worker nodes, are those worker nodes providing data to ConfD? If so, I would investigate what data those worker nodes are providing back to ConfD when data is requested by a northbound interface through ConfD.

Regards

kabhijeet · October 21, 2024, 3:40pm

Thanks for your reply @cohult

We are on confd version 8.0.14, The memory reservation is right when confd process is started, we do not have any calls to worker nodes for any data. And it’s not even going to the point where we could query data from northbound interface,

Does confd process depend on kernel version or any of the sysctl parameters ??

One additional information i have is that I ran strace on the confd process and I found that below mmap call is being made that causes this huge memory allocation.

sysinfo({uptime=323553, loads=[92960, 122048, 131776], totalram=50243645440, freeram=39200661504, sharedram=1004027904, bufferram=122945536, totalswap=0, freeswap=0, procs=3807, totalhigh=0, freehigh=0, mem_unit=1}) = 0
mmap(NULL, 34359742464, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7958bbfff000
brk(0x57584c6a6000)                     = 0x57584c6a6000

Can this point to the code where the decision of allocating memory is being done ?

Regards
Abhijeet

cohult · October 21, 2024, 5:17pm

Does confd process depend on kernel version or any of the sysctl parameters ??

No

ConfD is an in-memory database, meaning that at startup, ConfD will load what is in the CDB persisted files (A.cdb, C.cdb, O.cdb) into RAM if they exist.
ConfD will look for XML files to load data into RAM if the CDB persisted files do not exist.
So check the size of the *.cdb files or XML init files that ConfD reads from at startup.
You can dump the content of the *.cdb by using the confd --cdb-debug-dump /path/to/confd-cdb command.
The locations ConfD uses at startup are configured in confd.conf under /confdConfig/cdb/dbDir and /confdConfig/loadPath.

kabhijeet · October 21, 2024, 6:42pm

Size of cdb files are very small and even the loadPath does not contain any file of significant size

root@cni-opm [ /filestore/fed-smf/cfg/fed-smf/confd/var/confd/cdb ]# ls -lrth
total 2.5M
-rw-r--r-- 1 root root 9.5K Oct 16 15:51 nacm_init.xml
-rw-r--r-- 1 root root 2.5K Oct 16 17:52 li-x1-initial.xml
-rw-r--r-- 1 root root 1.9K Oct 16 17:52 li-x2-initial.xml
-rw-r--r-- 1 root root    0 Oct 17 09:58 compact.lock
-rw-r--r-- 1 root root 2.5M Oct 17 09:58 C.cdb
-rw-r--r-- 1 root root 4.8K Oct 18 17:14 nacm_additional_rules.xml
-rw-r--r-- 1 root root 5.5K Oct 18 17:14 A.cdb
-rw-r--r-- 1 root root 1.8K Oct 18 17:14 O.cdb

root@cni-opm [ /filestore/fed-smf/cfg/fed-smf/confd/etc/confd ]# du -sh
1.1M	.

kabhijeet · October 21, 2024, 7:14pm

@cohult In confd code, ulimit -n is being taken as an input to set some environment variables, I figured that in the environment where I do not see this issue, ulimit -n returns 65535, and in the environment (RHEL 9.4 VMs) this returns a huge value 1073741816.

On manually setting the ulimit inside container with ulimit -n 65535, and then running confd manually started confd with lesser memory.

confd code:

max_fds=`ulimit -n`
if [ "${max_fds}" -gt 1024 ]; then
    ERL_MAX_ETS_TABLES=`expr ${max_fds} + 1000`
    export ERL_MAX_ETS_TABLES
fi

We figured that this ulimit was being controlled by containerd service configuration LimitNOFILE=infinity found at /usr/lib/systemd/system/containerd.service. We were able to resolve the issue by commenting this option.

cohult · October 21, 2024, 8:37pm

@kabhijeet, ConfD does nothing unless told to do so, so somewhere in the startup sequence, start-phase 0, 1, or 2, something is done that consumes memory.
If ulimit -n helps, check the connections applications connect to ConfD. Is there an application opening up massive number of connections?
The simplest way to check when ConfD is running is to just run confd --status and look for user sessions.