more on bogged down server

Mon Apr 10 21:05:29 UTC 2006

On Mon, 2006-04-10 at 11:55 -0700, Harold Hallikainen wrote:
> Last week, I was out of town for the week and, of course, my FC4 system
> slowed down to a crawl, and eventually I could not get into it at all. I
> just bought and installed a "Web Power Switch" (under $100 at
> http://www.digital-loggers.com/lpc.html), so I will always (I hope) be
> able to reboot the system from anywhere.
> 
> I also reduced the number of clients (I think that was the term) in
> httpd.conf from 150 to 50.
> 
> Based on Rick's suggestion, I also added this to /etc/sysctl.conf:
> 
> # below lines added 4/8/06 to try to prevent system bog downs due to
> httpd. hh
> net.ipv4.tcp_fin_timeout = 1
> net.ipv4.tcp_max_syn_backlog = 2048
> net.ipv4.tcp_syn_retries = 3
> net.ipv4.tcp_tw_recycle = 1
> net.ipv4.tcp_tw_reuse = 1
> 
> 
> My web server has a lot of large files that will take a while for people
> to download. I suspect Apache is starting another thread for each of
> these, and keeping it open a long time. As more and more requests come in,
> the load just keeps getting bigger.

Which is exactly why http is not a good choice for downloads.  You
should modify the download links to "ftp://" to force FTP downloads.

>  Here's a recent top:
> 
> Cpu(s): 98.3% us,  1.7% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
> Mem:   1027640k total,  1013188k used,    14452k free,     8292k buffers
> Swap:  2031608k total,   244252k used,  1787356k free,   224352k cached
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 19089 apache    16   0 45120  17m 4780 S 17.0  1.8   0:04.59 httpd
> 14416 apache    24   0 60736  31m 4904 R  3.7  3.1  13:02.87 httpd
> 18425 apache    21   0 59872  30m 2996 R  3.7  3.0   2:58.17 httpd
>  8965 apache    25   0 59724  31m 4800 R  3.3  3.2  30:11.74 httpd
> 10263 apache    25   0 60896  32m 4664 R  3.3  3.2  17:42.99 httpd
> 10268 apache    25   0 60620  27m 4512 R  3.3  2.7  17:12.99 httpd
> 13121 apache    25   0 59540  31m 4648 R  3.3  3.2  15:00.67 httpd
> 13585 apache    25   0 51556  12m 4508 R  3.3  1.3  13:09.44 httpd
> 13802 apache    25   0 51364  19m 4504 R  3.3  1.9  13:28.15 httpd
> 14613 apache    25   0 60684  18m 4508 R  3.3  1.8  12:32.11 httpd
> 14682 apache    25   0 51284  10m 2960 R  3.3  1.0  13:36.44 httpd
> 14852 apache    25   0 51332  16m 4992 R  3.3  1.6  10:08.87 httpd
> 14853 apache    25   0 51144  18m 4620 R  3.3  1.9  11:20.88 httpd
> 14935 apache    25   0 51656  18m 5352 R  3.3  1.9  10:52.20 httpd
> 15134 apache    25   0 51360  18m 5364 R  3.3  1.8  10:00.45 httpd
> 15138 apache    21   0 51200  17m 5336 R  3.3  1.8  10:33.49 httpd
> 15504 apache    20   0 60708  20m 4512 R  3.3  2.0  10:24.10 httpd
> 15876 apache    20   0 51028  10m 4528 R  3.3  1.1   8:41.13 httpd
> 15877 apache    23   0 60520  20m 5448 R  3.3  2.1   5:43.60 httpd
> 16331 apache    20   0 60328  32m 5516 R  3.3  3.2   5:19.84 httpd
> 16633 apache    25   0 60420  32m 5408 R  3.3  3.2   5:15.28 httpd
> 17859 apache    25   0 60016  32m 5520 R  3.3  3.3   2:27.50 httpd
> 18089 apache    25   0 60040  32m 5512 R  3.3  3.3   3:17.04 httpd
> 18426 apache    21   0 59588  32m 5292 R  3.3  3.2   0:36.13 httpd
> 18503 apache    25   0 59800  32m 5460 R  3.3  3.3   0:57.90 httpd
> 
> 
> Notice that a lot of those httpd processes have been running quite a while.

Yup.

> My concern now is that while everything seems to be working, I think I'm
> missing some incoming mail. Here's something from /var/log/maillog:
> 
> Apr 10 11:46:48 sujan sendmail[2316]: rejecting connections on daemon MTA:
> load average: 33

You aren't missing mail, but you're not receiving it in a timely manner.
A load average of 33 on a web server is pretty bad unless it's getting
just bloody hammered.  Undoubtedly, HTTP downloads are the main culprit
here, and conversion to FTP would be a big help.

> So... what do I do?

You have to reduce your load somehow.  Ideally, you should create an
anonymous FTP download directory and move all of the downloadable files
to it.  The download directory also is used as the home directory for
the anonymous FTP user (user "ftp").  I actually use a completely
separate filesystem entirely for that.  The filesystem is mounted so
that only root has write access.

Modify your vsftpd.conf file to permit anonymous downloads only
and start up vsftpd.  Make sure you also set the "force chroot for
anonymous users" option.  Then change your links on your web pages to
use "ftp://"-style links pointed at the anonymous download directory
paths for the downloadable files.

FTP is the protocol to use for large file downloads.  HTTP just isn't
efficient for that, as you've now found out (the hard way, I might add).

----------------------------------------------------------------------
- Rick Stevens, Senior Systems Engineer     rstevens at vitalstream.com -
- VitalStream, Inc.                       http://www.vitalstream.com -
-                                                                    -
-                 All generalizations are false.                     -
----------------------------------------------------------------------