Dropping email on the floor?

[Rick Stevens]

Jeff Kinz wrote:
> On Fri, Apr 15, 2005 at 11:33:35AM -0700, Rick Stevens wrote:
> 
>>Jeff Kinz wrote:
>>
>>>Hi Guys,
>>>I've just recently started seeing large numbers of emails being dropped,
>>>but only from specific sources
>>>
>>>Here is what sendmail verbose mode is showing (two examples):
>>>
>>>26969 >>> 220 redline.kinz.org ESMTP Sendmail 8.11.6/8.11.6; Fri, 15 Apr
>>>2005 13:29:23 -0400
>>>26969 <<< EHLO nl-mail5.internet.com
>>>26969 >>> 250-redline.kinz.org Hello nl-mail5.internet.com [64.62.164.185], pleased to meet you
>>>26969 >>> 250-ENHANCEDSTATUSCODES
>>>26969 >>> 250-8BITMIME
>>>26969 >>> 250-SIZE
>>>26969 >>> 250-DSN
>>>26969 >>> 250-ONEX
>>>26969 >>> 250-ETRN
>>>26969 >>> 250-XUSR
>>>26969 >>> 250-AUTH GSSAPI
>>>26969 >>> 250 HELP
>>>26969 <<< MAIL FROM:<newsletter at nl.internet.com>
>>>26970 >>> 250 2.1.0 <newsletter at nl.internet.com>... Sender ok
>>>26970 <<< [EOF]
>>>26970 >>> 421 4.4.1 redline.kinz.org Lost input channel from nl-mail5.internet.com [64.62.164.185]
>>>26968 >>> 220 redline.kinz.org ESMTP Sendmail 8.11.6/8.11.6; Fri, 15 Apr 2005 13:29:25 -0400
>>>26968 <<< HELO n19a.bulk.scd.yahoo.com
>>>26968 >>> 250 redline.kinz.org Hello n19a.bulk.scd.yahoo.com [66.94.237.48], pleased to meet you
>>>26968 <<< MAIL FROM:<sentto-311578-3615-1113585173-jkinz=kinz.org at returns.groups.yahoo.com>
>>>26971 >>> 250 2.1.0 <sentto-311578-3615-1113585173-jkinz=kinz.org at returns.groups.yahoo.com>...  Sender ok
>>>26971 <<< RSET
>>>26971 >>> 250 2.0.0 Reset state
>>>26968 <<< QUIT
>>>26968 >>> 221 2.0.0 redline.kinz.org closing connection
>>>
>>>
>>>There seem to be two failure modes, one is the "Lost input channel" and
>>>the other is getting a SMTP "RSET" command from the MTA of the sending
>>>side.
>>
>>The first one is a fairly common probe by machines looking for open
>>relays--especially MS Exchange servers.  I'd consider that an attack.
>>The second one looks like a similar attack, but more along the lines of
>>an attempted SMTP DOS attack.  I'm willing to bet that the IP addresses
>>are spoofed as well.
> 
> 
> It seems I forgot some essential information, er, um, these two email
> sources, internet.com and bulk.scd.yahoo.com are legitimate email
> sources trying to send me email for various email lists that I signed up 
> for.  I have been happily receiving email from them for some time but I
> noticed that at some point in the past it was becoming unreliable and
> recently they all started just failing.

Then your ISP has a connectivity issue with those networks.  Complain
to your ISP.  It'd help if you could include traceroutes from you to
the questionable networks.
>>>NOTE: "<<<" seems to indicate messages sent by the external SMTP party and
>>>">>>" seems to indicate responses by my side (the "inside")
>>
>>Yes, "<<<" refers to INCOMING traffic TO your machine, ">>>" refers to
>>OUTGOING traffic FROM your machine (think of the arrows as relative
>>to your system).
>>
>>
>>>NOTE:  Comcast is having DNS server problems, Can that be affecting
>>>this? and if so, why only for internet.com and yahoo groups bulk mail
>>>servers?
>>
>>No, these are not DNS issues (otherwise you'd get only the IP address
>>of the remote machines and not a reverse DNS resolution giving the host
>>names).  The reverse resolution is correct, BTW, but the IPs are
>>probably spoofed.
> 
> 
> It seems the IP's are not spoofed, I did some lookups and correlated my
> tcpdump captures to those ip-domain-name pairs.  The ips and domain
> names are correct as read in from the Internet IP traffic before
> entering the sendmail conversation and they match the info given in the
> sendmail conversation, and - even more disconcerting, the online archives of
> those email lists are providing live traffic info which matches these
> failed deliveries.

Ok.  I've seen those kinds of things with spoofed IPs probing our
network searching for open relay mail servers.  I always assume the
worst when it comes to mail.

> I am dumbfounded.
> 
> Any suggestions on what steps I can take to try to diagnose the
> problems?  Also, is there any command line option for sendmail to report
> its version number (or any other mechanism?)?

Run traceroutes to the mailservers in question and include those in your
discussions with your ISP.  They may have a peering issue.  For example,
one of our upstream providers in Amsterdam decided to stop peering with
Cogent.  Their network crashed because their other peers couldn't handle
the traffic.
----------------------------------------------------------------------
- Rick Stevens, Senior Systems Engineer     rstevens at vitalstream.com -
- VitalStream, Inc.                       http://www.vitalstream.com -
-                                                                    -
----------------------------------------------------------------------