AW: Still much more than 350 sockets needed!

[Wes Shull]

On 4/26/06, Andrew Haley <aph at redhat.com> wrote:
> #include <netdb.h>

[snip]

I don't think Andrew's code is showing the real problem here, because
it's not actually trying to send anything over the connections after
they've been opened, to make sure they're still alive.

At first I thought Hendrik was on crack, but I'm able to reproduce
something similar on my FC5 system.  It's not related to the open file
ulimit.

I hacked up the Tcl code at the bottom to open a bunch of sockets,
connect to itself, and pass data back and forth; run it and see what
you get.  On my 2.6.16-1.2080_FC5 system, all but 19 of the
connections die after the client side sends some initial data.  I'm
thinking that some connection tracker (i.e. iptables/netfilter) is
losing track of them and so kicking back a reset.

I don't think my code is bad; on my XP SP2 box at work, it works fine
(there I have to keep numsocks at about 150 to keep Windows from
running out of buffer space, but at least it's not randomly closing
connections)

However, when I unload iptables (made sure no modules were still
loaded) it still does the same thing.  That really doesn't fit my
theory...

Are some of the tunables in /proc/sys/net/ipv4/netfilter/ perhaps
relevant to this?  If my theory somehow is correct, could it be a hash
function falling down with a bunch of connections with a very similar
signature?

I'm running selinux enforcing, tried setenforce 0, didn't seem to help either.


#!/usr/bin/tclsh

set baseport 3000
set numsocks 500 ;# keep <= (`ulimit -n` - 3) / 2

set remote_host 127.0.0.1

set command marco
set reply polo

proc server_accepted {serverport sock address port} {
	puts "server: $serverport accepted"
	fconfigure $sock -buffering line
	fileevent $sock readable [list server_readable $sock $serverport]
}

proc server_readable {sock port} {
	fileevent $sock readable {}

	gets $sock line
	if [eof $sock] {
		puts "server_readable: $port went away"
		close $sock
		return
	}

	puts "server: $port<-$line"
	if {$line eq $::command} {
		fileevent $sock writable [list server_writable $sock $port]
	}
}

proc server_writable {sock port} {
	fileevent $sock writable {}

	if [eof $sock] {
		puts "server_writable: $port went away"
		close $sock
		return
	}

	puts $sock $::reply
	puts "server: $port->$::reply"

	fileevent $sock readable [list server_readable $sock $port]
}

proc client_readable {sock port} {
	fileevent $sock readable {}

	gets $sock line
	if [eof $sock] {
		puts "client_readable: $port went away"
		close $sock
		return
	}

	puts "client: $port->$line"
	if {$line eq $::reply} {
		fileevent $sock writable [list client_writable $sock $port]
	}
}

proc client_writable {sock port} {
	fileevent $sock writable {}

	if [eof $sock] {
		puts "client_writable: $port went away"
		close $sock
		return
	}

	puts $sock $::command
	puts "client: $port<-$::command"

	fileevent $sock readable [list client_readable $sock $port]
}

# main
for {set port $baseport} {$port < $baseport + $numsocks} {incr port} {
	puts "Creating listener on $port"
	socket -server [list server_accepted $port] $port

	puts "Connecting to listener on $remote_host:$port"
	set sock [socket -async $remote_host $port]
	fconfigure $sock -buffering line
	fileevent $sock writable [list client_writable $sock $port]
}

vwait forever