Qmail Support Forum

admin · Site Admin Joined: 21 Jun 2005 Posts: 209

Hi List,

I wrote about this problem a few days ago but now have more information. Any advice is very much appreciated.

I run about 100 servers with identical qmail installs. The servers have been up for years and are all configured the same.

Starting about 48 hours ago, random servers started refusing new SMTP connections. It looked like a SYN_RECV dos attack
but I am now pretty sure that those were all valid hosts trying to connect. However they were getting backlogged when tcpserver
reached the connection limit for qmaill-smtpd (set to 20 concurrent servers).

After a lot of different trouble shooting, I finally turned off smtp for all the servers for a few minutes. I then turned it on again and
everything seemed good. All hosts started to work correctly.

However now about six hours later, the problem starts up again.

Can anyone think of a qmail problem that would take about six hours to develope? Any timeouts set to six hours?

No recent changes had been made to these machines prior to the problem starting - it seems like there must be some external issue that is causing it.
I have removed all RBL checks - I thought perhaps one of the RBL's where timing out.

I've also set tcpserver to create upto 100 current connections.

Right now I see things like this on a server:

mailq 27105 0.0 0.0 1388 280 ? S 02:57 0:00 bin/qmail-queue
qmaild 27132 0.0 0.0 1516 332 ? S 02:57 0:00 /var/qmail/bin/qmail-smtpd 2>&1 > /dev/null
qmailq 27138 0.0 0.0 1388 276 ? S 02:57 0:00 bin/qmail-queue
qmaild 27141 0.0 0.0 1516 332 ? S 02:57 0:00 /var/qmail/bin/qmail-smtpd 2>&1 > /dev/null
qmailq 27147 0.0 0.0 1384 276 ? S 02:57 0:00 bin/qmail-queue
qmaild 27161 0.0 0.0 1512 328 ? S 02:57 0:00 /var/qmail/bin/qmail-smtpd 2>&1 > /dev/null
qmailq 27169 0.0 0.0 1380 280 ? S 02:57 0:00 bin/qmail-queue
qmaild 27187 0.0 0.0 1528 332 ? S 02:57 0:00 /var/qmail/bin/qmail-smtpd 2>&1 > /dev/null
qmailq 27191 0.0 0.0 1380 280 ? S 02:57 0:00 bin/qmail-queue
qmaild 27239 0.0 0.0 1516 332 ? S 02:57 0:00 /var/qmail/bin/qmail-smtpd 2>&1 > /dev/null
qmailq 27240 0.0 0.0 1396 280 ? S 02:57 0:00 bin/qmail-queue
qmaild 27293 0.0 0.0 1540 332 ? S 02:57 0:00 /var/qmail/bin/qmail-smtpd 2>&1 > /dev/null

qmail is started with:

/usr/local/bin/tcpserver -H -R -x /home/vpopmail/etc/tcp.smtp.cdb -c100 -u 451 -g 450 0 smtp /var/qmail/bin/qmail-smtpd 2>&1 > /dev/null
/usr/local/bin/tcpserver -H -R -x /home/vpopmail/etc/tcp.smtp.cdb -c40 -u 451 -g 450 0 2525 /var/qmail/bin/qmail-smtpd 2>&1 > /dev/null

(i run alternate smtp's on 2525)

I've been fighting this for two days now and am getting a bit frustrated.

Can anyone think of any possible causes?

Thanks in advance!

admin · Site Admin Joined: 21 Jun 2005 Posts: 209

> However they were getting backlogged when tcpserver
> reached the connection limit for qmaill-smtpd (set to 20 concurrent
> servers).

It could be something as simple as too many spammers trying to connect.
The first thing i did when our ISP was experiencing similar problems
was install an IP limiter patch for tcpserver available at
http://linux.voyager.hr/ucspi-tcp/

wget
http://linux.voyager.hr/ucspi-tcp/files/tcpserver-limits-2005-09-03.diff
wget http://cr.yp.to/ucspi-tcp/ucspi-tcp-0.88.tar.gz
tar -zxvf ucspi-tcp-0.88.tar.gz
patch -d ucspi-tcp-0.88/ < tcpserver-limits-2005-09-03.diff
cd ucspi-tcp-0.88
make
make setup check

in your /home/vpopmail/etc/tcp.smtp, add:
MAXCONNIP="2",MAXCONNC="5",DIEMSG="421 please try later."

to each line (or, at least, the default :allow line)

that will allow 2 incoming connections per ip, and 5 per /24. others
will temp fail.

I run an array that receives a couple million emails per day, and havent
run into the problems since.

--

Jeremy Kister
http://jeremy.kister.net./

admin · Site Admin Joined: 21 Jun 2005 Posts: 209

What OS are you running?
Have you tried inreasing tcpserver backlog?
For example, try -b 256.

However, in Linux 2.2+ the backlog parameter for listen()
specifies the queue length for completely established sockets waiting to be
accepted, instead of the number of incomplete connection requests.
Sysctl tcp_max_syn_backlog is the thing you might want to tune in this case.

Are there some hosts which are doing particularly many connection
requests? If you run Linux, you could limit them without patching tcpserver:

iptables -A TCPs -p tcp -m state --state NEW -m tcp --dport 25 -m connlimit ! --connlimit-above 2 --connlimit-mask 32 -m hashlimit --hashlimit 2/s --hashlimit-mode srcip,dstip --hashlimit-burst 2 --hashlimit-name tcp25 --hashlimit-htable-expire 60000 -j ACCEPT
iptables -A TCPs -p tcp -m state --state NEW -m tcp --dport 25 -j DROP

Now, there's 2 connections per second limit for each
source IP address/destination IP address combination.
And also max 2 concurrent connections per IP address.
When those limits are reached, packet is dropped and SMTP client
tries sending new packets till it reaches its timeout.

You can view the recently (last 60s) accepted connections with command:

# cat /pr oc/net/ipt_hashlimit/tcp25
40 209.132.176.167:0->80.223.105.208:0 32000 32000 16000
8 209.132.177.30:0->80.223.105.208:0 32000 32000 16000

Also, if you LOG+DROP instead of DROP (the rule above), you can
find out the hosts which are reaching the limits.
For them you might want to consider ACCEPT without limit checks,
or DROP, depending on if you want to communicate with them.
You can also use hashlimit to log one IP address at max once per
minute, for example -- so prevent flooding the logs.

admin · Site Admin Joined: 21 Jun 2005 Posts: 209

Thanks for the advice, I'm running various Linux versions.

The traffic seem to be coming from all over the place though.

At first I thought for sure it was an attack. However I now believe it really is all valid traffic getting
backed up.

It seems like there is something that is causing qmail to hang. However I still do not know what. The problem
just seems to have started out of the blue without any changes to the mailservers.

The oddest thing was what happened last night :

Out of desperation I turned off the RBL list for all the servers. In order to make sure it took effect on all the servers, I did a full
stop on qmail-smtp for all servers. I then did a start on them all. I waiting a moment between the stop and start to make
sure all servers really were stopped (normally I run a script that does a stop/start very quickly)

After the stop and start, the problem totally went away for about six hours. I thought for sure it must have been one of the RBL's timing out or being slow, etc.

However six hours later the problem started up again and continues still.

I'm still trying to decide if it is an attack or not. I know that one cannot be too paranoid but an attack is seeming less and less likely. The attacker would have had to do a bit of work to find all the IP's of the servers (they are spread out among a large IP-space). Not at all impossible but unlikely it would seem. Also, it seems like a true attack would be targetting the webserver. I am really starting to think that all the SYN_RECV are valid hosts wanting to connect but not able to because qmail is hanging on many requests.

I just don't have any idea why qmail would suddenly start hanging.

Thanks again all for any input.

umbria · Joined: 22 Nov 2007 Posts: 1

ahhhhhh I have the sme problems !!!!!

any news ?!?!?

L.