LSMCD memory issue

[PRO ISP]

Hi,

We're testing your memcached (lsmcd) and got it up and running smoothly. However we are having issues with memory usage. The
/dev/shm/lsmcd folder is not the issue as it's barely 200MBs, but the cached processes simply grows until we have to stop and start lsmcd to start fresh (usually when they get around 30GB+). I am fairly certain that we are removing data from memcached as we should. We used memcached before starting to use lsmcd and had no such issues.

Are you aware of any bugs in regards of this? If not, how can this be debugged? We were hoping to use lsmcd, but if we don't get a solution to this shortly we have to go back to normal memcached.

 

[mistwang]

We will have a new release soon.
It sounds like a memory leak somewhere. if you restart it, keeping the current data in /dev/shm, the memory usage starting low and going up gradually?
We can login to the server to trouble shoot this, if you want.

 

[Hedloff]

I have to stop lsmcd and then remove the content and start lsmcd. If not I will only get a message " Unable to write to cache" if I only try a restart of lsmcd.

 

[PRO ISP]

Hi mistwang,

If I do a restart lsmcd fails and responds with this error:
47 SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY

I have to stop and then start it. I can do that with or without deleting the files and afterwards the memory usage is normal afterwards.

Just to check; What should the ownership/permissions be for these:

• /dev/shm/lsmcd
• /tmp/lsmcd

(we run lsmcd as user/group nobody)

Here is an example output from restart:

# /usr/local/lsmcd/bin/lsmcdctrl restart
+ PROG=/usr/local/lsmcd/bin/lsmcdctrl
+ EXECUTABLE=lsmcd
+ BASE_DIR=/usr/local/lsmcd
+ cd /usr/local/lsmcd
+ '[' '!' -x /usr/local/lsmcd/bin/lsmcd ']'
+ CONF=/usr/local/lsmcd/conf/node.conf
+ DESC=lsmcd
+ '[' '!' -f /usr/local/lsmcd/conf/node.conf ']'
++ grep TmpDir /usr/local/lsmcd/conf/node.conf
+ LINE=TmpDir=/tmp/lsmcd
+ '[' xTmpDir=/tmp/lsmcd '!=' x ']'
++ echo TmpDir=/tmp/lsmcd
++ cut -d = -f2
+ TMPDIR=/tmp/lsmcd
+ PIDFILE=/tmp/lsmcd/lsmcd.pid
+ RUNNING=0
++ uname -s
+ SYS_NAME=Linux
+ '[' xLinux = xFreeBSD ']'
+ '[' xLinux = xDarwin ']'
+ PS_CMD='ps -ef'
+ '[' -f /tmp/lsmcd/lsmcd.pid ']'
++ cat /tmp/lsmcd/lsmcd.pid
+ FPID=26338
+ '[' x26338 '!=' x ']'
++ ps -ef
++ grep lsmcd
++ grep -v grep
++ grep -w 26338
+ PL='root      26338      1  0 Aug14 ?        00:00:01 ./bin/lsmcd -f /usr/local/lsmcd/conf/node.conf
nobody    26340  26338  0 Aug14 ?        00:00:03 lsmcd - replicator
nobody    26341  26338  0 Aug14 ?        00:03:39 lsmcd - cached #01
nobody    26342  26338  0 Aug14 ?        00:05:15 lsmcd - cached #02                            '
+ '[' 'xroot      26338      1  0 Aug14 ?        00:00:01 ./bin/lsmcd -f /usr/local/lsmcd/conf/node.conf
nobody    26340  26338  0 Aug14 ?        00:00:03 lsmcd - replicator
nobody    26341  26338  0 Aug14 ?        00:03:39 lsmcd - cached #01
nobody    26342  26338  0 Aug14 ?        00:05:15 lsmcd - cached #02                            ' '!=' x ']'
+ RUNNING=1
+ PID=26338
+ ret=0
+ case "$1" in
+ restart
+ '[' 1 -eq 1 ']'
+ kill -USR1 26338
+ ret=0
+ '[' 0 -ne 0 ']'
+ exit 0

 

[mistwang]

I have to stop and then start it. I can do that with or without deleting the files and afterwards the memory usage is normal afterwards.

it is a memory leak bug. we may need temp root access to figure it out.

 

[PRO ISP]

Ok. Should I send the required information to bug@litespeedtech.com as usual or in some other preferred way?

 

[mistwang]

sure.