[Resolved] Continuous Abort requests happens thrice a day at particular time interval for 1 min

[Nandhakumar]

Hi there,

I am having pingdom service enabled for my site

https://www.artifactuprising.com/

I was gone through the continuous abort request made my site offline for 1 mins basis on thrice a day at 7:00:55/8:00:55 and 19:00:55 MST. I don't know how exactly those abort requests happens at particular time interval so checked on my side whether i had any cron at that time to make this happen but i don't have any cron running on my site.

Also I am running load balancing technique with n web server nodes and each node has licensed litespeed installed there with latest lsws software on all nodes (4.2.22-ent) and checked individually on each nodes and each node shows the abort request error on that time downtime period. Please let me know why this happening on particular time am just curios as this bug was resolved in 4.2.13 or something but still persists and makes my business suffer.

Please get me the update on this one ASAP.

 

[NiteWave]

weird and interesting issue.

any special log around 7:00:55 ?

please provide more info or you can provide the access to the server for us to collect.

 

[Nandhakumar]

Please go through the attachments and see abort requests only happens on 14:00 UTC ( 8:00AM MST) until last day yesterday and after one or three minutes, error was gone and getting my site works normal again. This was happened all over the last month and still it's continuing at exact time in abve mentioned thread. Below was the sample pingdom alert.

Subject: DOWN alert: Artifact Uprising - Website Uptime Check
(www.artifactuprising.com) is DOWN

PingdomAlert DOWN:
Artifact Uprising - Website Uptime Check (www.artifactuprising.com) is down
since 03/21/2015 08:00:55AM.(MST)

 

[mistwang]

I think it is a problem with MySQL or memcached if you use it.
It is something accessed by all web server node from PHP.
I have seen that happens to another customer.

 

[Nandhakumar]

I am not using memcached but using rackspace cloud database(MYSQL) for my application.

if it is problem with mysql then how is this happening in every day at particular time? how would i verify this issue belongs to mysql?

 

[mistwang]

if it happen every-day at the same time, it is easy to verify. Just check mysql status with

mysqladmin processlist

you need to specify correct parameter to connect to the cloud database. Maybe that's the time data base being backed up, which could cause DB performance issues.

 

[Nandhakumar]

Yesterday also I was going through the same error at the same time, but when we check with mysql, we didn't see any issues with mysql. fyi, below mentioned time was in UTC timezone (i.e. at 8:00 MST)

2015-03-23 14:00:11.453 [INFO] [10.183.253.52:41227-0#APVH_*:443_artifactuprising.com] Abort request processing by PID:26581, kill: 0, begin time: 10, sent time: 10, req processed: 107
2015-03-23 14:00:21.924 [INFO] [10.183.253.49:24295-0#APVH_*:443_artifactuprising.com] Abort request processing by PID:30377, kill: 0, begin time: 10, sent time: 10, req processed: 0
2015-03-23 14:00:26.537 [INFO] [10.183.253.48:49170-0#APVH_*:443_artifactuprising.com] Abort request processing by PID:29877, kill: 0, begin time: 10, sent time: 10, req processed: 3
2015-03-23 14:00:27.948 [INFO] [10.183.253.50:28574-0#APVH_*:443_artifactuprising.com] Abort request processing by PID:30378, kill: 0, begin time: 10, sent time: 10, req processed: 0

Do you have any other idea why this could happen?

 

[NiteWave]

is it possible to relate to https connection between pingdom and your website ?
your current monitoring URL should be

https://...

how about just change it to

http://...

for 1 or 2 days ?
I know it's just a redirect:

#curl -I www.artifactuprising.com
HTTP/1.1 301 Moved Permanently
Server: LiteSpeed
Date: Tue, 24 Mar 2015 12:33:19 GMT
Keep-Alive: timeout=5, max=100
Location: https://www.artifactuprising.com/
Transfer-Encoding: chunked
Access-Control-Allow-Origin: *
Connection: Keep-Alive
Set-Cookie: X-Mapping-fjhppofk=793F656BB11E146229761037AD68DBD6; path=/

or change the monitoring URL to a small static file to see if any difference.

 

[mistwang]

The abort requests are most likely caused by front-end load balancer closes the connection to web servers.
Maybe the request take too long to process, you need to check the PHP processes running on that server by that time. I think PHP was hanging on something. if by that time, the site is really slow, you should check what the PHP process was doing with strace and lsof.

login to one your backend server, locate a long running lsphp5 process, then do

lsof -p <pid_of_lsphp_process>
strace -tt -T -p <pid_of_los_process>

to find out what happened.

 

[Nandhakumar]

Most likely that this error had happened when mysqldump process going on. so to make sure this i have changed the dump process time then error got reproduced at new time.

 

[mistwang]

mysqldump will lock out all other clients, basically, blocks PHP.