Qmail Support Forum

admin · Site Admin Joined: 21 Jun 2005 Posts: 209

A while back I posted that qmail queue was locking up. I fiddled with some
things and the problem went away. I was never convinced that I actually
fixed the underlying problem, but I haven't had trouble for a few months.
The problem is back again.

qmailctl stat
/service/qmail-send: up (pid 947) 1136 seconds
/service/qmail-send/log: up (pid 948) 1136 seconds
/service/qmail-smtpd: up (pid 1715) 689 seconds
/service/qmail-smtpd/log: up (pid 1722) 689 seconds
/service/qmail-pop3d: up (pid 1721) 689 seconds
/service/qmail-pop3d/log: up (pid 1723) 689 seconds
messages in queue: 835
messages in queue but not yet preprocessed: 563

Everything is up. But the queue won't deliver messages to their inboxes. The
main qmail log stops at the point the queue locks up. The qmail-smptd and
qmail-pop3d logs are logging fine and show no errors. Restarting qmail, and
even rebooting the server does not resolve the trouble. The queue will not
clear. Obviously this is a significant problem since no one can receive
email. I'm at a loss. Any help greatly appreciated.

Thanks,
Roger

admin · Site Admin Joined: 21 Jun 2005 Posts: 209

Can you do an strace on individual qmail-send processes and try and
determine what's going on? Paste output here if necessary...

Ray

admin · Site Admin Joined: 21 Jun 2005 Posts: 209

A little more info. This is in my processlist...
readproctitle service errors: ...unable to link to
/var/log/qmail/@4000000044c8f4a812ae7e5c.u, pausing:

Something wrong with the logs? How do I fix? If I create this file then it
just repeats the message with a different filename.

And, two messages were just delivered, then locked up again.

Roger

admin · Site Admin Joined: 21 Jun 2005 Posts: 209

What do 'df -ik' and 'ls -al /var/log/qmail' say?

admin · Site Admin Joined: 21 Jun 2005 Posts: 209

BTW, this comes from multilog.c of daemontools:
==========
if (link(file,fn) == 0) break;
pause5("unable to link to ",d->dir,"/",fn,", pausing: ");
==========
and means that the link() function failed. 'man 2 link' indicates that
there are a number of reasons for such a failure, including exceeding
LINK_MAX, permission problems, exceeding filesystem space or user's
quota etc

- --
Best regards,
Thanos Massias

admin · Site Admin Joined: 21 Jun 2005 Posts: 209

Yes. It looks like some problem with your filesystem. When the logger
runs out of space or inodes (for example), then it cannot write into
the log files anymore. When it can’t write into the log files: the OS
prevents it from doing so, and so the process stops (in other words,
the process “blocks” while waiting for it’s write command to succeed).
When the logging process stops, it stops reading more log entries out
of the pipe from qmail. When that happens, the pipe buffer fills up,
and qmail can’t write anything more into the pipe:the OS prevents it
from doing so by stopping the process (qmail is “blocked” from
continuing).

It sounds like you need to examine the filesystem where you store your
logs. Maybe you ran out of space/inodes/directory-entries, or
something else along those lines.

~Kyle

admin · Site Admin Joined: 21 Jun 2005 Posts: 209

Hmm... SELinux? Smile

Although you'd think it would have been causing problems
all along...

Ray

admin · Site Admin Joined: 21 Jun 2005 Posts: 209

If you’re using SELinux, I bet that there’s some restriction
that’s preventing maillog from doing what it normally does: are you
allowing hard links? I know SELinux (and grsec and owl) all allow you
to disallow hard links (because it makes permissions-checking more
interesting).

~Kyle