Over the last week, we’ve been having a problem with spam in our shared web service: Something was sending out lots of low-quality, easily-blockable spam, and the bouncebacks were filling up the Postfix queues in our outgoing email cluster.  The way we tracked down the spammer was interesting, so I’m writing it up here in case it’s of interest to anyone else!

Out environment already has measures in place to stop spam: You can’t just make outbound connections to any open relay out there, you have to send all mail through our outgoing email cluster, which runs Postfix.  In addition to some basic client sanity checking, we send all outgoing mail through a spam filter.  It’s reasons like this why we spam-filter all of our outgoing mail.

When the spam-filter rejects that mail, the outgoing email cluster tries to send a bounce back to the envelope sender.  Our web servers don’t receive email from non-localhost IPs, so those emails aren’t going to be delivered:  All of those bounce emails start to pile up in the cluster, and monitoring alarms start to go off.

If a spam flood happens to make it out of the network, once the end-user networks start to report it, that’ll also let us know that something is going on!

Now that we know something is going on, we need to identify who’s sending the flood.  To do that, we need to know how email can be sent.  On our systems, there are three ways to send email out:

1. By making an SMTP (or SMTPS, or SMTP+STARTTLS) connection to the outgoing email cluster.
2. By using the local sendmail command.
3. By using one of the other local commands, such as mail, mailx, Mail, or (on systems running Postfix) postdrop.

Options 1 and 2 are, in my experience, the most likely to happen.  The sendmail command is pretty ubiquitous on UNIX and Linux systems.  Also, if you’ve hacked a web site, you’ll be able to look up what configuration the site is using, so you can figure out what server the site normally uses for outgoing email.

Catching the spammer in the act requires that you have iptables (which you probably do already), along with auditd (which you probably won’t have already-installed).  I’ll cover each use separately.

The iptables Rule

iptables is the Linux built-in firewall (it runs within the Linux kernel), and it looks at both incoming and outgoing traffic, on all interfaces.  Those last two parts are important: If your system is running a local mail server (for example, to handle emails generated by processes like cron), that mail server may be listening on the loopback interface (127.0.0.1, or ::1).  SMTP connections to localhost use the same network stack that SMTP connections to outside systems use, and so the firewall will see that traffic.  What we need is for the firewall to log that traffic.  And not only do we need to log the traffic, we need to log who is sending the traffic.

To get what we need, I use this line:

iptables -A OUTPUT -p tcp -m tcp --dport 25 --syn -j LOG --log-uid --log-level info --log-prefix "Outbound email: "

Let’s break this down a little:

• -A OUTPUT means to apply this rule to outgoing traffic.
• –p tcp means to only look at TCP traffic.
• –m tcp is needed so that iptables will recognize the next two lines.
• dport 25 tells iptables to only match connections to port 25, the SMTP port.  If you want to look at other mail-related ports (like 465), you could do an additional iptables line with the different port number.
• --syn tells iptables to only match the opening of the connection.  iptables looks at packets separately, and we don’t need to log every part of the connection.  It’s enough to just log the opening.
• -j LOG tells iptables that we want to log this packet.  This is kindof weird for iptables, because normally when you see -j it’s followed by a final action like DROP or REJECT.  LOG is not a final action: iptables will log the packet, and then it will continue going through the rules as if nothing has happened.
• --log-uid tells iptables that we want to log the user and group IDs of the process that sent the packet.
• --log-level info tells iptables that we want to use the info logging level.  You could use a different level, or leave this out.
• --log-prefix "Outbound email: " tells iptables to add Outbound email: to the start of each logged line.  The space at the end is intentional; iptables does not separate your prefix from the rest of the log, so you have to!

Once the rule is applied, you’ll start seeing traffic like this:

2016-01-28T15:15:27.812642-08:00 www03 kernel: [2091079.159274] Outbound email: IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=37794 DF PROTO=TCP SPT=46247 DPT=25 WINDOW=32768 RES=0x00 SYN URGP=0 UID=71023 GID=37

The above line is an actual line from the spam flood that triggered all this work.  The spammer was connecting to Postfix via localhost.  Since every user of the shared web service has a separate user ID, we just had to identify user #71023.  If you’re interested in exactly what code is sending mail, you can check out things like ps and lsof.

auditd Rules

auditd is software that hooks in to the Linux kernel’s auditing mechanism.  We need to use it to catch anything that sends mail without using SMTP.

Here’s an example:  Let’s say our spammer is using the sendmail command to send mail through the system’s local Postfix or Sendmail server.  In that case, iptables will log the mail server sending out the mail, using the mail server’s account.  That’s definitely not what we want, so we need to log the actual execution of the commands.  To do that, we use auditd.

The command we need is this:

auditctl -w /usr/sbin/sendmail -p x -k sending_mail

This one is much easier to break down:

• -w /usr/sbin/sendmail tells auditd that we want to watch the file /usr/sbin/sendmail.  If your system places sendmail at a different path, or you want to watch for executions of another command (like mailx), then put that path here instead.
• -p x tells auditd that we only want to watch for someone executing the command.
• -k sending_mail tells auditd that whenever this rule is matched, add sending_mail to the log entry.

Here’s an example of the kindof output you’ll see if you’re using the auditd simplify plugin:

type=EXECVE key=sending_mail auditid=468 time="2016-01-27 13:57:49-0800" hostname="www03.stanford.edu" tty=(none) ppid=28800 pid=28802 exe="/usr/sbin/sendmail" name="/usr/bin/mailq" user=root origuser= cwd="/root" command="mailq"

In this case, the root user is using sendmail to send something.

Use With Care!

To finish up, it’s important to note that both of these methods can cause spikes in the load of your system.  Part of this is because (especially for the iptables rule) the OS has to do some extra rules-checking on every outgoing packet.  Also, all of this additional kernel logging is going  to increase the amount of disk activity (as systems log kernel activity to disk).

To deal with this, you should have these rules ready to go, but you might want to hold off on actually implementing them until you have an actual problem.

On the other hand, if you’ve got a cluster of web servers, it might not be too bad to turn this on for one of them, so you can see how much of a hit your system takes, both during normal operations and during a mail flood!

Regardless, you shouldn’t just take what I’ve provided and blindly apply it to your system.  Take the time to make sure what I wrote here will apply cleanly to your system, and test it first.

Good luck!