Patching QMAIL for mild anti-spam and better logging; and running POP3 and EZMLM ================================================================================= Date last updated: 29.09.2007 This started off as an aide-memoire to myself to remind me of the stages I'd need to follow if I had to rebuild the server sometime. It's turning into a bit of a blog now. Ah well. Plain text will have to do, I really can't be arsed to turn this into HTML! Overall I _strongly_ recommend Qmail. The only reason I've had to apply some tweaks is that one of my accounts (that I can't easily dump) is quite heavily spammed, simply because I need my email address to be visible on certain websites. So my requirements for spam control - whilst not extreme - are more demanding than most. Basic Qmail installation is about as easy as it could be: everything's set up from a handful of tiny little files in /var/qmail/control (or wherever you choose to have it). The other thing to remember is that this software has stood the test of time: no security patches have been required in EIGHT YEARS! You can't say the same for Sendmail - and I wouldn't want to leave a Microsoft internet server unpatched for eight years... FreeBSD is a good OS for Qmail. It's small, quick and simple. I'm up to FreeBSD 6.2 now, and it seems fine. Qmail's POP3 support is good (and can be used over SSL with the 'stunnel' port, optionally with client-side certificates to keep out unauthorised users). The EZMLM-IDX mailing list manager integrates easily enough. The Patches =========== It's nice to block things at the SMTP stage so that no bounce traffic is generated. There are several QMAIL patches for this. I applied them on FreeBSD 6.1 starting from the port in /usr/ports/mail/qmail. Patch #1. First I applied the 'qmail-realrcptto' patch, which blocks SMTP delivery to any email address for which there is no maildir / .qmail-xxx / virtualdomain (etc) entry in existence. This is based on the RCPT TO smtp command. http://code.dogmap.org./qmail/ http://code.dogmap.org./qmail/qmail-1.03-realrcptto-2004.09.14.patch So if someone tries spamming guessthis@mydomainhere.com, my SMTP server won't accept it. Patch #2. Secondly I applied the patch to enhance 'control/badmailfrom'. This lets you specify domains (e.g. your own) which to be banned from SMTP delivery to you. It's based on the MAIL FROM smtp command. http://tomclegg.net/qmail-bmf-wildcard http://tomclegg.net/software/patch-qmail-badmailfrom-wildcard My /var/qmail/control/badmailfrom now contains: @mydomainhere.com @myotherdomainhere.co.uk (etc) because I don't send mail to myself, and these domains attract spam that's often sent with a FROM address in one of these domains, so I'd been seeing some of the bounce messages that Qmail sent. So, anyone claiming to be something@mydomainhere.com can't now talk SMTP to me. This stops me accepting mail from myself, so here's a one-line tweak so that I can talk to myself through 127.0.0.1 (only the 'strcmp' test is new): void smtp_rcpt(arg) char *arg; { if (!seenmail) { err_wantmail(); return; } if (!addrparse(arg)) { err_syntax(); return; } if (flagbarf && (strcmp(remoteip, "127.0.0.1") != 0) ) { err_bmf(); return; } if (relayclient) { --addr.len; if (!stralloc_cats(&addr,relayclient)) die_nomem(); if (!stralloc_0(&addr)) die_nomem(); } Patch #3. Thirdly, I applied the patch to create a 'control/badrcptto' list. This lets you ban specific email addresses at SMTP RCPT TO time. Compare with patch (1.) above, the point of this is to allow you to block specific email addresses e.g. sales@mydomainhere.com even though anything@mydomainhere.com gets through on a .qmail-default rule. http://patch.be/qmail/ http://patch.be/qmail/badrcptto.patch This patch can't be applied fully automatically if you've already applied the first two patches, as a couple of diff context lines are affected by the earlier patches. But it's a lovely short patch, so you can start off by using 'patch', then apply the excluded hunks by hand in vi. Patch #4. More logging... I wanted more stuff in my syslog so that I could see what the hell was going on. There's a lot of scope for improvement but this is a start: http://iain.cx/qmail/patches.html http://iain.cx/qmail/download/qmail-smtpd-pidqplog.patch - log sender of each msg http://iain.cx/qmail/download/qmail-smtpd-verbose.patch - log denied relay attempts (includes the previous patch) -> applied OK on top of the previous 3 patches, with a bit of manual editing. Patch #5. I've made a small patch for qmail-smtpd.c to make it reject a message completely if *any* of the recipient addresses were in 'control/badrcptto', or were rejected due to relaying. So for example, an email addressed to both sales@mydomain.com and a_valid_address@mydomain.com will get binned. This diff was taken after patches 1. to 4. above had already been applied: --- qmail-smtpd.c.old Wed May 17 19:34:02 2006 +++ qmail-smtpd.c Wed May 17 19:44:05 2006 @@ -26,10 +26,12 @@ #include "strerr.h" #define MAXHOPS 100 unsigned int databytes = 0; int timeout = 1200; +int seen_any_badrcpttos = 0; +int seen_any_badrcpthosts = 0; int safewrite(fd,buf,len) int fd; char *buf; int len; { int r; r = timeoutwrite(timeout,fd,buf,len); @@ -144,10 +146,12 @@ if (brtok == -1) die_control(); if (brtok) if (!constmap_init(&mapbrt,brt.s,brt.len,0)) die_nomem(); realrcptto_init(); + seen_any_badrcpttos = 0; + seen_any_badrcpthosts = 0; if (control_readint(&databytes,"control/databytes") == -1) die_control(); x = env_get("DATABYTES"); if (x) { scan_ulong(x,&u); databytes = u; } if (!(databytes + 1)) --databytes; @@ -303,14 +307,15 @@ --addr.len; if (!stralloc_cats(&addr,relayclient)) die_nomem(); if (!stralloc_0(&addr)) die_nomem(); } else - if (!addrallowed()) { err_nogateway(&mailfrom, &addr); return; } + if (!addrallowed()) { err_nogateway(&mailfrom, &addr); seen_any_badrcpthosts = 1; return; } if (!env_get("RELAYCLIENT") && brtcheck()) { strerr_warn4("qmail-smtpd: badrcptto: ",addr.s," at ",remoteip,0); err_brt(); + seen_any_badrcpttos = 1; return; } if (!realrcptto(addr.s)) { out("550 sorry, no mailbox here by that name. (#5.1.1)\r\n"); return; @@ -435,10 +440,12 @@ char *qqx; if (!seenmail) { err_wantmail(); return; } if (!rcptto.len) { err_wantrcpt(); return; } if (realrcptto_deny()) { out("550 sorry, no mailbox here by that name. (#5.1.1)\r\n"); return; } + if (seen_any_badrcpttos) { out("553 sorry, badrcptto seen, so you MUST be spamming. (#5.1.1)\r\n"); return; } + if (seen_any_badrcpthosts) { out("553 sorry, badrcpthost seen, so you MUST be spamming. (#5.1.1)\r\n"); return; } seenmail = 0; if (databytes) bytestooverflow = databytes + 1; if (qmail_open(&qqt) == -1) { err_qqt(); return; } qp = qmail_qp(&qqt); out("354 go ahead\r\n"); Patch #6. I made a small patch for rblsmtpd to make it log SMTP headers from RBL'ed IPs - see below. First let's discuss what RBL is all about, and how you configure Qmail to filter against the RBL in the SMTP dialog... Patch #7. http://qmail.mirrors.pair.com/outgoingip.patch was installed (nice and clean, no clashes, even comes with man page updates). This is a nice simple tweak for servers that have multiple IP addresses. You just create /var/qmail/control/outgoingip with a single line saying which IP needs to be used to send outbound mail. Patch #8. http://www.saout.de/misc/spf/ This is an SPF checking implementation for incoming mail. Fairly clean, just two manual patches that clashed with my modified Qmail sources. This allows you to bin incoming mail that arrives from IP addresses that the (forged) domain owner's DNS records indicate as impossible mail sources. See http://www.openspf.org/ for more info on the SPF initiative. Patch #9. http://www.lewis.org/smtp-delay/ This is a plug-in that runs as its own process in the Qmail pipeline. It blocks email from servers that don't follow the SMTP protocol properly. Apparently this reduces spam too. RBL Black-Holing of Spammers' IPs ================================= I held off doing this for ages - I have reservations about this kind of thing - but the spam levels were getting silly, so I stuck in some SMTP-time filtering in the end. This seems like the best way to do it, but doing it at SMTP-time does mean that you don't have a special email address that always gets through... Of the DNS-based blocking lists, these sounded reasonable (not too aggressive): cbl.abuseat.org (automated spam trap: see http://cbl.abuseat.org/ ) list.dsbl.org (open relays etc: see http://dsbl.org/usage ) dul.dnsdbl.sorbs.net (dynamic IP address list: see http://www.nl.sorbs.net/ ) ... but when using this approach, you _will_ sometimes lose good mail. There is also an aggressive list sbl-xbl.spamhaus.org (see http://dsbl.org/usage) but that might catch too much legit stuff. It's always easy to spot missed spam, but it's very hard to spot missed ham. My /var/service/qmail-smtpd/run file now says: #!/bin/sh QMAILDUID=`id -u qmaild` NOFILESGID=`id -g qmaild` /usr/local/bin/tcpserver -H -R -v -P -l 'mailhost.example.org.uk' -x /etc/tcp.smtp.cdb \ -u $QMAILDUID -g $NOFILESGID 0 smtp \ /usr/local/bin/rblsmtpd -C -b -r cbl.abuseat.org -r list.dsbl.org -r dul.dnsdbl.sorbs.net \ /var/qmail/bin/qmail-smtpd 2>&1 | \ /var/qmail/bin/splogger smtpd Patching RBLSMTPD (patch #6) ============================ OK, here's my little patch for rblsmtpd to make it log SMTP headers from RBL'ed IPs. First, ensure you've built the port /usr/ports/sysutils/ucspi-tcp and not deleted the work directory. Go to: /usr/ports/sysutils/ucspi-tcp/work/ucspi-tcp-0.88 Edit the file rblsmtpd.c and just add just one 12-line function right at the end as per this diff: # diff -U 4 rblsmtpd.c.old rblsmtpd.c --- rblsmtpd.c.old Tue May 16 20:06:03 2006 +++ rblsmtpd.c Tue May 16 19:57:25 2006 @@ -195,4 +195,16 @@ pathexec_run(*argv,argv,envp); strerr_die4sys(111,FATAL,"unable to run ",*argv,": "); } + +int log_smtp_data(char *msg,unsigned int len) +{ + buffer_puts(buffer_2,"rblsmtpd: "); + buffer_puts(buffer_2,ip_env); + buffer_puts(buffer_2," pid "); + buffer_put(buffer_2,strnum,fmt_ulong(strnum,getpid())); + buffer_puts(buffer_2,": "); + buffer_put(buffer_2,msg,len); + buffer_puts(buffer_2,"\n"); + buffer_flush(buffer_2); +} Add two lines to the file commands.c as per this diff: # diff -U 4 commands.c.old command.c --- commands.c.old Tue May 16 20:05:39 2006 +++ commands.c Tue May 16 20:04:13 2006 @@ -4,8 +4,9 @@ #include "case.h" #include "commands.h" static stralloc cmd = {0}; +extern int log_smtp_data(char *,unsigned int); int commands(buffer *ss,struct commands *c) { int i; @@ -23,8 +24,9 @@ if (!stralloc_append(&cmd,&ch)) return -1; } if (cmd.len > 0) if (cmd.s[cmd.len - 1] == '\r') --cmd.len; + log_smtp_data(cmd.s,cmd.len); if (!stralloc_0(&cmd)) return -1; i = str_chr(cmd.s,' '); Now type 'make', then copy the new executable rblsmtpd binary on top of the old one in /usr/local/bin. More on Spam Handling ===================== So far, the CBL list is being almost totally effective in spam blocking, and hasn't logged any false positives. So I'll probably leave it there. Without scripting up some more stuff, you can't tell whether the second and third RBL lists would have blocked the same stuff. I'll leave a background job logging lines containing 'rblsnmpd' from my /var/log/maillog, so that I can get a good idea of its accuracy from watching for email addresses that look valid and therefore may have been false positives. False negatives I'll see in my inbox. Stats for first 24 hours: About 500 emails have been blocked by the combined patches listed above. Only 15 spams got through, so I am well pleased (you can't expect total spam filtering without a lot of false positives). If this level is maintained then using more RBL's would probably be over-zealous. Using SpamAssassin would be another option, but I find Bayesian filtering unconvincing. It needs a lot of training to be effective. I've tried it in the past and given up with it. Maybe there is another tool I could use just to pick off the most obvious of the stuff that the RBL lists don't pick up.... See: http://cr.yp.to/ucspi-tcp/rblsmtpd.html http://www.sdsc.edu/~jeff/spam/Blacklists_Compared.html http://www.chrishardie.com/tech/qmail/qmail-antispam.html http://chris-linfoot.net/d6plinks/CWLT-6KRE5S - efficacy/risk ratings for some RBLs http://www.fehcom.de/qmail/qmail.html - SpamControl for Qmail http://spamthrottle.qmail.ca/ - spam throttling http://openrbl.org/client - web gui to look up an IP on lots of BL zones Other good Qmail background sites: http://www.thedjbway.org http://qmail.softflare.com http://lifewithqmail.org http://www.flounder.net/qmail/qmail-howto.html http://qmail.3va.net/qmailfaq.html http://www.ezmlm.org - for DJB's ezmlm list manager, and the ezmlm-idx extension. Running the POP3 daemon ======================= For Pop3d, I didn't want to have the daemon running as root and consulting the main shadow password file. So I installed checkcdb, which authorizes pop users from a CDB database. http://www.palomine.net/qmail/checkcdb.html http://www.palomine.net/qmail/checkcdb-0.54.tar.gz DBJ's 'cdb-0.75.tar' package - see http://cr.yp.to/cdb.html - from FreeBSD ports. The /var/service/qmail-pop3d/run file now looks like this: # cat /var/service/qmail-pop3d/run #!/bin/sh exec /usr/local/bin/softlimit -m 5000000 \ /usr/local/bin/tcpserver -v -R -H -l 0 0 110 /var/qmail/bin/qmail-popup \ mailhost.example.org.uk /usr/local/bin/checkcdb /var/qmail/bin/qmail-pop3d Maildir 2>&1 The encrypted password text file is /var/qmail/users/poppasswd (hard-coded). To change a password in /var/qmail/users/poppasswd, run 'poppasswd'. To buid the CDB version /var/qmail/users/poppasswd.cdb, run 'newpop'. This would need more work to run as non-root. But you can't hit my POP service from the Internet anyway. Running mailing lists with ezmlm ================================ I went with ezmlm-idx on FreeBSD: cd /usr/ports/mail/ezmlm-idx make install This is a very capable package, with lots of documentation on the web. You just need to take the time to read the docs carefully and test your setup properly. The CGI access to the mailing list archives seemed a bit too complicated, with several mailto: links that would attract spammers. Having taken the trouble to stick some mild Javascript obfuscation of the list addresses, I thought it was worth patching ezmlm-cgi.c to remove the mailto links. The CGI pages also showed a web link to a URL that has now expired and been grabbed by the domain spammers, so I took that out too. Here's the patch for ezmlm-cgi.c . diff -U5 ezmlm-cgi.c.orig ezmlm-cgi.c --- ezmlm-cgi.c.orig Sun May 21 12:10:40 2006 +++ ezmlm-cgi.c Sun May 21 17:25:14 2006 @@ -666,29 +666,29 @@ oputs(">author "); link_msg(infop,ITEM_AUTHOR,DIRECT_NEXT); oputs("->] |\n"); linktoindex(infop,ITEM_DATE); oputs(">[Threads]\n"); - homelink(); - oputs("\ntarget)] = '\0'; - oputs(strnum); - oputs("@"); - oputs(host); - justpress(); - oputs("\">[eMsg]\n"); - oputs("[eThread]\n"); - subfaqlinks(); +// homelink(); +// oputs("\ntarget)] = '\0'; +// oputs(strnum); +// oputs("@"); +// oputs(host); +// justpress(); +// oputs("\">[eMsg]\n"); +// oputs("[eThread]\n"); +// subfaqlinks(); oputs("\n"); } #define SPC_BASE 1 #define SPC_BANNER 2 @@ -715,12 +715,13 @@ oputs("\n\n"); oputs("\n"); oputs("\n"); if (local) { oputs(local); - oputs("@"); - oputs(host); +// oputs("@"); +// oputs(host); + oputs(" list"); oputs(": "); } if (t) oputs(t); if (s) htmlencode_put(s,l); oputs("\n"); @@ -740,25 +741,29 @@ oputs("\n"); } else oputs("\n"); - + homelink(); + oputs("

\n"); } void html_footer(int flagspecial) { - oputs("
"); - oputs(EZ_CRIGHT); - oputs("
"); +// Lose the trailer as it now points to a domain spammer +// oputs("
"); +// oputs(EZ_CRIGHT); +// oputs("
"); if ((flagspecial & SPC_BANNER) && banner && *banner) { oputs("\n"); } + oputs("
\n"); + homelink(); oputs("\n\n"); substdio_flush(&ssout); } /* DATE functions */ @@ -895,12 +900,12 @@ linktoindex(infop,ITEM_INDEX); oputs(">[->>] |\n"); infop->target = tmpmsg; linktoindex(infop,ITEM_DATE); oputs(">[Threads by date]\n"); - subfaqlinks(); - homelink(); + // subfaqlinks(); + // homelink(); oputs("\n"); } int show_index(struct msginfo *infop) { @@ -983,12 +988,12 @@ } if (item != ITEM_INDEX) { linktoindex(infop,ITEM_INDEX); oputs(">[Messages by date]\n"); } - homelink(); - subfaqlinks(); + // homelink(); + // subfaqlinks(); oputs("\n"); } int show_object(struct msginfo *infop,char item) /* shows thread, threads, author */ @@ -1375,29 +1380,30 @@ oputs(""); oputs(""); oputs(constmap_get(&headermap,i + 1)); oputs(":"); decodeHDR(hdr[i].s,hdr[i].len,&line,"",FATAL); - if (i == HDR_SUBJECT - 1 && flagtoplevel) { - oputs(""); - } +// if (i == HDR_SUBJECT - 1 && flagtoplevel) { +// oputs(""); +// } if (flagobscure && i == HDR_FROM - 1) { oputs(" "); decodeHDR(cp,author_name(&cp,line.s,line.len),&decline,"",FATAL); htmlencode_put(decline.s,decline.len); } else { decodeHDR(hdr[i].s,hdr[i].len,&decline,"",FATAL); htmlencode_put(decline.s,decline.len - 1); } if (i == HDR_SUBJECT - 1 && flagtoplevel) - oputs(""); +// oputs(""); + oputs(""); oputs("\n
"); } oputs("\n"); } flaginheader = 0;