PolicyD
Modules
PolicyD has several modules which may or may not apply to the matching policies.
The order the modules are applied in is determined by the priority (high to low) built into each module (ie. RTFS!). Currently (v2.1.0) the order is:
| 90 | AccessControl |
| 80 | CheckHelo |
| 70 | CheckSPF |
| 60 | Greylisting |
| 50 | Quotas |
| 50 | Amavis |
| 10 | Accounting |
AccessControl
AccessControl is a simple match implies verdict.
Unfortunately there is no automatic management of the policy group members so it isn't really suited to having incrementally accumulated sets where those entries might be (expected to be) deleted after a few days.
CheckHelo
CheckHelo performs several tests on the HELO string.
Firstly it must pass RFC 2821 validation. Primarily, the HELO string must either be an literal IP address [a.b.c.d] or must be a primary domain name with an A resource record. That is, it must be an externally resolvable, FQDN -- and not a CNAME.
These validity checks will fail even global messaging corporations so you might need to create an extra policy for (presumably) trusted yet broken senders. Sadly, you'll reject the mail before finding out they've not set their systems up correctly. We all make mistakes, though, so this is a necessary reality.
Note
RFC 5321 (which supersedes RFC 2821) in Section 4.1.4 reiterates that
An SMTP server MAY verify that the domain name argument in the EHLO command actually corresponds to the IP address of the client. However, if the verification fails, the server MUST NOT refuse to accept a message on that basis.
Indeed, CheckHelo does not verify that the sender's IP address matches the HELO string which seems a curious omission in the War on SPAM.
I guess this allows machines behind NAT routers to be allowed to send email -- which are almost certainly the vast majority of machines on the Internet that are causing the problem. My guess is that corporate machines behind NAT routers on private LANs sending to other internal private LANs are the target of this clause and therefore, by accident, the enablers of the Internet-wide SPAM issue.
NB. This relates to IP addresses only. Giving an invalid FQDN is still grounds for dismissal.
It also keeps track of what HELO strings have been used by any given IP address. If it uses too many it is blacklisted.
CheckHelo might reject half the inbound email.
CheckSPF
CheckSPF is a straight-forward verification of the SPF entry for a domain.
It is let down by too many SPF entries being invalid and too many organisation defaulting to ~all which means soft fail i.e. any system sending email from this domain is acceptable.
When it works, it's great!
GreyListing
Greylisting is an incredibly simple trick that is very effective.
If you've never seen the sender of the email before then you politely inform them that you are busy and could they come back later. A real email server will queue the email to be resent in a few minutes time. How many depends on the server, anything from almost instantly from another IP address (bonkers) to a more usual five minutes or so to possibly several hours.
This second time through, we look the system up in our database of previously seen senders, check that it has been a few minutes since we first saw them and then pass the mail through.
That's it, no formal access control, simply whether we've seen them before.
And it works. Most spam systems do not retry emails that were deferred for any reason. This will result in anything from 1/3 to a 2/3 reduction in incoming email.
Why? Why does it work? Why do spam systems not retry emails? Without seeing their infrastructure and other constraints it's impossible to say. The day a spam system requeues emails will be a sad day indeed.
Auto-Blacklisting
Greylisting can also maintain white and blacklists. The blacklists in particular work by looking at the overall sets of sending host/sender address/recipient address tuples.
You can imagine a couple of scenarios:
If the sending host has tried to send a message from A to X and been greylisted, then tried to send a message from B to Y and been greylisted and then tried to send a message from C to Z and been greylisted then we begin to think they might be up to no good. Of course, they don't have to be quite so distinct, so long as the sender/recipient tuple is distinct, so A to X then A to Y then A to Z is also three distinct sender/recipient tuples as is A to X, A to Y then B to X.
If several different hosts are all trying to send mail using the same sender: host1's message is from A to X and host2's message is from A to X (or Y) and host3's message is from A to X (or Y or Z) then that's suspicious.
However, as suspicious as this is (imagine a bonnet trying to flood you with junk mail), PolicyD does not help as it is keyed on the sending host. If the host is different then the results will not be correlated.
A genuine MTA will have backed off all messages destined for us if any have been previously greylisted. With auto-blacklisting we can set a threshold of how many parallel messages have been greylisted from a host and then blacklist the sending host for a period of time.
Note
Nominally, we could be banning the sender but the Greylisting table only accepts a Track value of SenderIP:.... That is, we can only track and therefore auto-blacklist the sending host IP (or subnet).
You might set the number of parallel greylistings to be low (single figures) and the blacklisting period to be high (a week) depending on your mood. A bad mood, no doubt if you're fiddling at this level.
Warning
Big Corp will be connected to the Internet by more than one ISP (for redundancy) and could be sending out via any of its Internet connections. We'll see messages from the same domain (and therefore, possibly, the same sender) from multiple hosts.
You can't know in advance what Big Corp might be up to (changing its ISPs, changing its email server software, changing its mind) so we have to accept that it is possibly legitimate. If annoying.
Of course, you'd like to believe that under those circumstances Big Corp has gotten its act together and will not try to send us email from a different IP address if the first has been asked to defer delivery temporarily. But this is the real world and they'll get it wrong.
PolicyD's mechanism is to update any try-count of an existing host/sender/recipient tuple (in the greylisting_tracking table). Only if that fails (i.e. there is no such tuple) does it look at the number of entries associated with the sending host and if the number passes the AutoBlacklistCount parameter will you get auto-blacklisted.
The side-effect of this is that any honeypot address you might be using will not catch a repeat offender. That's because the same host/sender/recipient tuple, even for a honeypot recipient, will just have the try-count updated repeatedly. You'll not trigger the auto-blacklist code.
You need to have multiple honeypot addresses used by the same host.
Auto-whitelisting
I can't comment on whitelisting as I've not figured it out yet! There aren't enough users on my system but my guess is that if the remote host has been playing by the rules as it sends several emails to your users then you might grant it the benefit of the doubt and whitelist it in advance of some future unseen sender/recipient tuple.
You can then make a decision on whether your users have had cause to mark the emails as SPAM and therefore whether the sender should be granted some trusted sender or known spammer status.
Overall, though, I'm not sure that auto-whitelisting is usable. All greylisting is doing is delaying the first email you receive from a host (for a day until the authentication timer resets). You're not stopping a persistent spammer in any way. So an auto-whitelisting is a free pass (through the greylisting module).
Percentages
For both auto white and blacklisting you can specify a percentage of matches which is applied to the (un)authenticated tuples vs. total number of tuples after you reach the specified number of tuples.
This is an interesting addition, though a bit tricky. Let's consider the case where two senders, a legitimate bulk mailer and a spammer, have a large number of your email addresses in their hands.
For whitelisting, I would propose that you want a very high percentage, close to 100%, maybe. The theory being that if the host has tried to send to any sender/recipient tuples out of turn then they're up to no good so you want to ensure that the MTA is behaving well before auto-whitelisting them. Although there are enough badly behaved legitimate MTAs out there.
For blacklisting, you might have a relatively low percentage, although I have mine quite high (90%). Here we want to punish people not behaving well (spammers) but we are at risk of penalising legitimate but badly behaved MTAs. The punishment period might be quite long (a day, a week) which means you'd never see the mail from the broken MTA.
Tricky!
Persistence
How long do greylisting timeouts interact?
Let's say that you have a GreylistPeriod of four minutes (noting that many popular MTAs have a default minimum retry timeout of five minutes and so the greylisting period should time out before the regular MTA retries).
Let's consider a determined sending MTA who retries an email every minute. The first connection attempt will be deferred and PolicyD will update a FirstSeen record in greylisting_tracking. The second (and third, fourth and (possibly) fifth) attempts to send us an email will all occur before the greylisting period is up. Each of these times we'll increment a Tries counter indicating an attempt to deliver while greylisted.
The next attempt (probably the fifth, possibly the sixth), we'll be beyond the greylisting period and we'll PASS the mail on (to the next policy). We'll update a Count counter indicating an attempt to deliver after the greylisting period whereon we consider the host authenticated.
Cleanup
A host will continue to be greylisting authenticated for GreylistAuthValidity seconds. This might be a day.
Once authenticated, if you send another mail before the GreylistAuthValidity period is over the timer is reset. In other words, once authenticated, if you delivered a mail per day you won't need to be re-authenticated.
On the other hand, if the sending host doesn't send another email before the GreylistAuthValidity period is up then a database record with a non-zero Count will be deleted and the host will have to go through the greylisting process again.
Any hosts with a zero Count (ie. unauthenticated) which have not been seen for GreylistUnAuthValidity seconds will be deleted. Here, rather than a validity it is more of a database cleanup timer. Most spam falls into this category. This timeout might also be a day to avoid accumulating junk.
Quotas
Quotas determines the rolling accumulated number of messages (or message sizes) and compares it to a limit. If you are over the limit before we start the verdict is DEFER, then the details for this email are updated in the database.
Quotas can be tracked by: the sending host (or subnet); or the sender address (or domain); or the recipient address (or domain).
You can see there are two variations on this theme (and you can use both).
You can set a (rolling) daily quota: untrusted senders can send up to 10 messages per day; trusted senders can send 1000 messages per day. This is the usual "email limit" you might think of.
You might want to be careful about allowing "trusted" senders to send effectively unlimited numbers of emails per day, say, as if the user's machine has been compromised then they look like the legitimate sender suddenly sending hundreds of thousands of messages per day. (Even a modest machine can send hundreds of emails per second.)
How many is a reasonable number of mails for your trusted senders to send in a day?
The larger your organisation, the more likely that any bad behaviour by your users will result in your organisation [1] being blacklisted by blacklist servers. Major email providers have very sophisticated and rapid reaction systems to incoming spam. You might be blacklisted before you know there is a problem!
[1] At least your organisation's sending IP address.
You can set a "burst mode" where you restrict the number of emails, say, inbound in a short period.
This has to be big enough to handle a large number of recipients in a single mail but wants to be small enough to prevent a compromised machine from swamping the Internet. You might think that 60 emails in one minute is fine.
Again, if you are an ISP then trusted bulk mailers might be allowed to send several thousand in a minute as the "Daily Deals" emails shower their users with goodness.
Each recipient counts as one for the number of messages calculation. Aside from bulk mailers, that's more likely to affect corporate email. Joe Public email users rarely send to multiple recipients on a regular basis (other than the necessary forwarding of life-fulfilling pictures of cats to everyone they know). On the other hand, in corporations, it's rare to see any email without a dozen or more addresses Cc'd. That means everyone pushing email through the system will requires a proportionally higher quota to avoid tripping any limits. The flip-side here is that in to allow as yet untrusted remote senders to be allowed to reply all we inadvertently allow spammers to be able to send to many people as well.
Tricky.
Verdict
Quotas have a verdict associated with them for when the quota is exceeded. The verdict is transmitted back to the sending host.
It isn't immediately clear what the possible verdicts are nor necessarily what they mean. protocols.pm lists the available verdicts and protocols/*.pm determine how MTAs (eg. Postfix) handle the verdicts.
You should only really be playing with two verdicts:
DEFER meaning "try again later"
This should be the normal verdict for going over quota. It is also the implicit verdict for greylisting.
REJECT meaning, er, reject
You may have no need for this at all but you may have policies for senders you have banned or outright liars (remote hosts sending mail from your own domains -- "impossible, you're barred!").
It is the verdict for failures in CheckHelo and CheckSPF, of course.
You may want to use it in "burst mode" quotas for untrusted senders. If someone you don't expect is sending you an awful lot of email very quickly then a verdict of DEFER is only storing up the damage for later (assuming the sender has a re-queuing MTA or a large botnet). To avoid swamping your systems when this 60 second period is up you may want to REJECT any further emails from this source/to this recipient (however your policy is defined).
There are several other possible verdicts, not limited to:
DISCARD isn't as useful as you'd like. The REJECT verdict is equivalent to slamming the door in the face of the postman without even accepting the mail. DISCARD is accepting the mail then chucking it in the bin after you've politely closed the door.
This is a subtle difference and says to any spammer that you're willing to take emails from them. If you can you should REJECT.
HOLD puts the mail on hold, whatever that means. In Postfix it goes in the Hold queue which requires manual intervention to ameliorate. That sounds like hard work!
Warning
You should not ACCEPT an email. You are only one of any number of policy daemons and unless you absolutely know this email is legitimate then you should keep quiet.
Postfix notes its reactions to the verdicts in access(5).
Accounting
Accounting is quotas for a fixed time window.
Amavis
Amavis is access to the Amavis daemon. You may already use Amavis through another means.
Document Actions