LSWS stops rotating logs for no apparent reason

[andreas]

Sometimes (happened twice in the last months) lsws stops rotating log files for no apparent reason. Example:

-rw-r--r-- 1 www-data www-data 1.2G 2009-01-20 18:20 access.log
-rw-r--r-- 1 www-data www-data  11M 2009-01-12 19:42 access.log.2009_01_12.14
-rw-r--r-- 1 www-data www-data  11M 2009-01-12 20:34 access.log.2009_01_12.15
-rw-r--r-- 1 www-data www-data  11M 2009-01-12 21:29 access.log.2009_01_12.16
-rw-r--r-- 1 www-data www-data  11M 2009-01-12 22:25 access.log.2009_01_12.17
-rw-r--r-- 1 www-data www-data  11M 2009-01-12 23:40 access.log.2009_01_12.18
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 02:46 access.log.2009_01_13
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 08:14 access.log.2009_01_13.01
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 09:30 access.log.2009_01_13.02
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 10:32 access.log.2009_01_13.03
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 11:29 access.log.2009_01_13.04
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 12:22 access.log.2009_01_13.05
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 13:19 access.log.2009_01_13.06
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 14:11 access.log.2009_01_13.07
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 15:01 access.log.2009_01_13.08
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 15:53 access.log.2009_01_13.09
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 16:42 access.log.2009_01_13.10
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 17:35 access.log.2009_01_13.11
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 18:27 access.log.2009_01_13.12
-rw-r--r-- 1 www-data www-data  11M 2009-01-13 19:26 access.log.2009_01_13.13

As you can see the last rotate happened on 2009-01-13. Nothing interesting happened on that day that could have caused it.

lsws version: 3.3.23

config for this vhost:

  <logging>
    <log>
      <useServer>0</useServer>
      <fileName>/var/www/..../shared/log/lsws.log</fileName>
      <logLevel>WARN</logLevel>
      <rollingSize>5M</rollingSize>
    </log>
    <accessLog>
      <useServer>0</useServer>
      <fileName>/var/log/lsws/..../access.log</fileName>
      <pipedLogger></pipedLogger>
      <logFormat></logFormat>
      <logHeaders>7</logHeaders>
      <rollingSize>10M</rollingSize>
      <keepDays>1</keepDays>
      <bytesLog></bytesLog>
      <compressArchive>0</compressArchive>
    </accessLog>
  </logging>

 

[mistwang]

Will a restart fix this?
Have you done any software update on that day?

 

[andreas]

I don't know if a restart would have fixed it, I noticed it when my log partition had run full and I had to kill it and delete the logfile.

I don't think I did any updates on or before that day.

Maybe this is relevant, a "tail -F access.log" is running to do live processing of the logfile. But I can't see how that should affect the rotation.

 

[andreas]

I had the same problem today. The partition was not yet 100% full, but the file was growing and nothing was being rotated. This time I noticed an additional symptom: I could not stop/restart the server the normal way, I had to run "killall -9 lsws".

 

[auser]

I had the same problem today. The partition was not yet 100% full, but the file was growing and nothing was being rotated. This time I noticed an additional symptom: I could not stop/restart the server the normal way, I had to run "killall -9 lsws".

to help identify the issue:
1. lsws version still be 3.3.23 ?
2. this time, is "tail -F access.log" command or script running as before?

 

[andreas]

1. No, it's 4.0.0 this time.
2. Yes.

 

[mistwang]

The parent process of lshttpd running as root is responsible for log rotation, so, if it happen next time, please "strace" that lshttpd process, it should do "stat()" regularly on all log files, and try to do a "rename()" if file size is over the limit. Might be something wrong with that process.

 

[auser]

tested on apache 2.2.11

1.tail -F access.log

keep running there, repeat open access.log every 1 second(default 1 second)

2.in another terminal window:
mv access.log access.log.bak

3.then error message for tail:
tail: cannot open 'access.log' for reading: No such file or directory

but still running there

4. after that, access.log not created by apache even a lot of access.
restart apache, access.log created.

now back to watch litespeed's behavior(testing 4.02). in this regards, litespeed is better than apache:in above step 4, without restart lsws, access.log created and access log still working.

the difference is noticed in step 2:
right after mv, litespeed will create an empty access.log file even no access.
but apache won't create access.log until restart.

so a quick guess, make following 2 system call atomic(if not yet) will be even better ?
<atomic>
mv access.log access.log.1 (==>rotate log)
create access.log
</atomic>

 

[auser]

update: apache not miss logs.

in my previous post, step 4, although access.log not created, but new access still append to access.log.bak

so the mechanism to process log file is different between apache and lsws. apache will append data to log file's file descriptor instead of file name;lsws will append data to file name. when log file name changes, will create a same name log file.The lsws way looks better for log analysis program.