[SOLVED] One of the site suddenly goes offline frequently

[hamityanik]

Hello,
We are using Litespeed Enterprise v5.2.4 on our CentOS 6.9 server. One particular site suddenly goes offline frequently. Website is reachable again when I restarted Litespeed. I checked the logs, there are some logs repeats.

2018-03-03 22:55:28.812417 [WARN] [XX.XX.XX.XX:62093] smProcessReq():  last state: HSPS_HKPT_HTTP_AUTH, current state: HSPS_HANDLER_PROCESSING, ret=-1!
2018-03-03 22:55:28.816708 [WARN] [XX.XX.XX.XX:62092] smProcessReq():  last state: HSPS_HKPT_HTTP_AUTH, current state: HSPS_HANDLER_PROCESSING, ret=-1!
2018-03-03 22:55:29.060744 [WARN] [XX.XX.XX.XX:62094] smProcessReq():  last state: HSPS_HKPT_HTTP_AUTH, current state: HSPS_HANDLER_PROCESSING, ret=-1!
2018-03-03 22:55:29.064484 [WARN] [XX.XX.XX.XX:62095] smProcessReq():  last state: HSPS_HKPT_HTTP_AUTH, current state: HSPS_HANDLER_PROCESSING, ret=-1!

Site is not reachable when this log repeats:

2018-03-04 01:16:58.922001 [NOTICE] [XX.XX.XX.XX:16114] Content len: 0, Request line: 'GET /xyz1.html HTTP/1.1'
2018-03-04 01:16:58.922024 [NOTICE] [XX.XX.XX.XX:16114] Redirect: #1, URL: /index.php
2018-03-04 01:16:59.310905 [NOTICE] [XX.XX.XX.XX:53296] Content len: 0, Request line: 'GET /xyz2.html HTTP/1.1'
2018-03-04 01:16:59.310941 [NOTICE] [XX.XX.XX.XX:53296] Redirect: #1, URL: /index.php
2018-03-04 01:16:59.906645 [NOTICE] [XX.XX.XX.XX:28912] Content len: 0, Request line: 'GET /xyz3.html HTTP/1.1'
2018-03-04 01:16:59.906667 [NOTICE] [XX.XX.XX.XX:28912] Redirect: #1, URL: /index.php
2018-03-04 01:17:00.320637 [NOTICE] [XX.XX.XX.XX:16115] Content len: 0, Request line: 'GET /xyz4.html HTTP/1.1'
2018-03-04 01:17:00.320664 [NOTICE] [XX.XX.XX.XX:16115] Redirect: #1, URL: /index.php

Full log:

Any suggestions? I'll be happy if you help.
Thank you.

 

[NiteWave]

please try:
1. rm -rf /dev/shm/lsws
2. /usr/local/lsws/admin/misc/lsup.sh -f -v 5.2.5
to see if the issue will be gone

 

[hamityanik]

please try:
1. rm -rf /dev/shm/lsws
2. /usr/local/lsws/admin/misc/lsup.sh -f -v 5.2.5
to see if the issue will be gone

I've updated to v5.2.5 with this way. It happened again.

stderr.log file:

2018-03-04 16:27:17.035 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
2018-03-04 16:27:17.636 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
2018-03-04 16:27:17.736 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
2018-03-04 16:27:17.836 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
2018-03-04 16:27:17.936 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
2018-03-04 16:27:18.036 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
2018-03-04 16:27:18.136 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
2018-03-04 16:27:18.236 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
2018-03-04 16:27:18.336 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.

error.log file is same:

2018-03-04 16:27:28.635663 [NOTICE] [XX.XX.XX.XX:52747] Content len: 0, Request line: 'GET /xyz.html HTTP/1.1'
2018-03-04 16:27:28.635687 [NOTICE] [XX.XX.XX.XX:52747] Redirect: #1, URL: /index.php
2018-03-04 16:27:29.698651 [NOTICE] [XX.XX.XX.XX:10048] Content len: 0, Request line: 'GET /xyz1.html HTTP/1.1'
2018-03-04 16:27:29.698680 [NOTICE] [XX.XX.XX.XX:10048] Redirect: #1, URL: /index.php
2018-03-04 16:27:29.698862 [NOTICE] [XX.XX.XX.XX:62349] Content len: 0, Request line: 'GET /xyz2.html HTTP/1.1'
2018-03-04 16:27:29.698877 [NOTICE] [XX.XX.XX.XX:62349] Redirect: #1, URL: /index.php
2018-03-04 16:27:30.031875 [NOTICE] [XX.XX.XX.XX:55369] Content len: 0, Request line: 'GET /xyz3.html HTTP/1.1'
2018-03-04 16:27:30.031902 [NOTICE] [XX.XX.XX.XX:55369] Redirect: #1, URL: /index.php

I'm monitoring the server continuously. Suddenly this is happened:

 

[hamityanik]

Litespeed configuration is attached.

 

[NiteWave]

it is possible the website is being (D)Dos suddenly.
what's your Server->Security->Per Client Throttling ?

 

[hamityanik]

it is possible the website is being (D)Dos suddenly.
what's your Server->Security->Per Client Throttling ?

Here you are:

 

[NiteWave]

please try following(example only, you can adjust it) :
Dynamic Requests/second: 1
Connection Soft Limit:15
Connection Hard Limit:20
to see if it can mitigate the attack effectively(of course need restart lsws after change made)
in Real-Time Stats, check "Anti-DDoS Blocked IP"

 

[hamityanik]

please try following(example only, you can adjust it) :
Dynamic Requests/second: 1
Connection Soft Limit:15
Connection Hard Limit:20
to see if it can mitigate the attack effectively(of course need restart lsws after change made)
in Real-Time Stats, check "Anti-DDoS Blocked IP"

Thank you! This helped too much! Problem gone since these settings.