Marcin Krzan
2012-02-19 18:21:06 UTC
Hi,
our service is using two frontend apache/php servers with traffic load
distribution 50/50. A few weeks ago we performed full system upgrade
one of them from:
Server Version: Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny8 with Suhosin-
Patch mod_ssl/2.2.9 OpenSSL/0.9.8g
to
Server Version: Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze3 with
Suhosin-Patch mod_ssl/2.2.16 OpenSSL/0.9.8o
keeping configuration files unmodified.
From that point we noticed increased number of apache processes on
upgraded server and also server load is very high. Server status is
showing:
17:08:21 up 27 days, 1:52, 1 user, load average: 5.38, 3.42, 3.20
CPU Usage: u268.9 s34.7 cu0 cs0 - .0146% CPU load
19.3 requests/sec - 4719 B/second - 244 B/request
69 requests currently being processed, 102 idle workers
___R.C_.R__._W_..._._R.R.__.__..__.R_..R..R._._R.R...._.___...C.
__..._R_R.WC_C..W._.__.__.__R__C_R.C.R_...__.___CCC.RR.R_R_R.R.R
..._.R.R.CC_R__._.____.__.R.C_CR____.R....W__._R_._._W_.R.._.RRR
.C_R_._RRW__R_RC_R__...____.RCC__R_..._____R_.RW_____W_R.____.W.
when at the same time old server doesn't have any problems:
17:08:37 up 143 days, 8:46, 1 user, load average: 1.16, 1.23, 1.31
CPU Usage: u152.14 s41.14 cu0 cs0 - .00156% CPU load
26.3 requests/sec - 10.0 kB/second - 389 B/request
36 requests currently being processed, 0 idle workers
CCCRCWWCWWCCC.WCRCCW.C.WCCC.CCC.CCCCC.CWCW......................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
As you can see there is increased number of reading requests on
upgraded server.
Servers configuration:
Max Requests Per Child: 0 - Keep Alive: off - Max Per Connection: 100
Timeouts Connection: 150 - Keep-Alive: 15
Decreasing server timeout helps, but doesn't solve the problem.
There are a lot of connections like:
Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-5 6204 0/3/860017 R 0.02 1 16 0.0 0.00 277.21 ? ? ..reading..
Configuration of both servers is basically the same. There are same
apache modules enabled. We don't really have idea how to solve this
problem.
Any help would be really appreciated.
Thank you && Best Regards,
Marcin
our service is using two frontend apache/php servers with traffic load
distribution 50/50. A few weeks ago we performed full system upgrade
one of them from:
Server Version: Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny8 with Suhosin-
Patch mod_ssl/2.2.9 OpenSSL/0.9.8g
to
Server Version: Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze3 with
Suhosin-Patch mod_ssl/2.2.16 OpenSSL/0.9.8o
keeping configuration files unmodified.
From that point we noticed increased number of apache processes on
upgraded server and also server load is very high. Server status is
showing:
17:08:21 up 27 days, 1:52, 1 user, load average: 5.38, 3.42, 3.20
CPU Usage: u268.9 s34.7 cu0 cs0 - .0146% CPU load
19.3 requests/sec - 4719 B/second - 244 B/request
69 requests currently being processed, 102 idle workers
___R.C_.R__._W_..._._R.R.__.__..__.R_..R..R._._R.R...._.___...C.
__..._R_R.WC_C..W._.__.__.__R__C_R.C.R_...__.___CCC.RR.R_R_R.R.R
..._.R.R.CC_R__._.____.__.R.C_CR____.R....W__._R_._._W_.R.._.RRR
.C_R_._RRW__R_RC_R__...____.RCC__R_..._____R_.RW_____W_R.____.W.
when at the same time old server doesn't have any problems:
17:08:37 up 143 days, 8:46, 1 user, load average: 1.16, 1.23, 1.31
CPU Usage: u152.14 s41.14 cu0 cs0 - .00156% CPU load
26.3 requests/sec - 10.0 kB/second - 389 B/request
36 requests currently being processed, 0 idle workers
CCCRCWWCWWCCC.WCRCCW.C.WCCC.CCC.CCCCC.CWCW......................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
As you can see there is increased number of reading requests on
upgraded server.
Servers configuration:
Max Requests Per Child: 0 - Keep Alive: off - Max Per Connection: 100
Timeouts Connection: 150 - Keep-Alive: 15
Decreasing server timeout helps, but doesn't solve the problem.
There are a lot of connections like:
Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-5 6204 0/3/860017 R 0.02 1 16 0.0 0.00 277.21 ? ? ..reading..
Configuration of both servers is basically the same. There are same
apache modules enabled. We don't really have idea how to solve this
problem.
Any help would be really appreciated.
Thank you && Best Regards,
Marcin