Cron Email is Dead, Long Live Cron Email
30 Jul 2019Many services still rely on email for communicating issues with systems administrators. While there are many fancy new monitoring and metrics systems out there, none of them have the simplicity and wide availability of email.
But what if you have centralized logging and metrics for your server fleet and no longer want to deal with email, spam filters, and SPF records? I built a small tool that replaces outgoing SMTP by quietly logging service-generated email. Click here to skip directly to the tool.
What do servers even use email for?
We’ve all tried to run commands as root with sudo, received the error hans is not in the sudoers file. This incident will be reported, and laughed it off. Behind the scenes, it actually fires off an email with the user and command they attempted to run! If that email goes to a reasonable default address, somebody might act on it.
There are other services out there too. The unattended-upgrades tool will email you whenever it updates a package that requires a system reboot. Cron will happily email you the output of your various cron jobs. Lots of other software you might run also wants to deliver email!
For better or worse, almost everything on a modern (or ancient) Linux distribution relies on the de facto standard of /usr/sbin/sendmail. It provides a relatively mail server-agnostic way of sending email from other programs and the interface hasn’t changed in a long time. The options are arcane at best, and none of the implementations are truly compatible, but it all works well enough. Debian has 11 packages that provide a sendmail shim!
The future is email?
This sounds great! Lots of software will send us useful messages (including warnings and errors) via email, and we have tons of options for sending that email. What’s the problem?
For one, maintaining a modern mail server is a thankless and painful task, a far cry from the halcyon days of the 90s when all you had to worry about was keeping your sendmail patched so it didn’t get hacked. To successfully send mail, you’ll need to set up Sender Policy Framework (SPF) records so that other servers don’t instantly ignore your sending IPs. You might need to set up reverse DNS records for your servers so that other servers don’t assume you’re just a spammer. You need to set up DomainKeys Identified Mail (DKIM) for your servers so that recipients can verify it’s definitely your servers sending the mail. And you need to set up Domain-based Message Authentication, Reporting and Conformance (DMARC) to ensure other systems follow all of your SPF and DKIM settings.
This might be ok if you have one server. But if you have a fleet, setting reverse DNS for every server gets pretty tricky. It’s also really hard if you have a fleet that’s behind a NAT gateway, such as a large set of cloud servers that only have private addresses. Since AWS EC2 servers, for example, come with Amazon-provided IPs and reverse DNS by default, one of the easiest ways to solve the SPF problem is to add a wildcard for every EC2 reverse DNS name to the SPF record. While it means your mail won’t get marked as spam, it also means anybody else who can launch a server in EC2 (i.e. everybody) can impersonate you.
Aha! But what about DKIM! you triumphantly declare. This means you’ll have to distribute the DKIM secrets to every single server you have. Which brings me to the next problem: you’ll also have to configure and run a Mail Transfer Agent (MTA) such as Postfix, Exim, or Sendmail on every single server in your fleet. This is a non-trivial task which requires care and feeding, including the occasional security patch when there’s a new way to turn email into remote code execution. While there exist MTAs with minimal featuresets (like nullmailer), it still speaks SMTP to servers on the internet.
The future is centralized logging and monitoring
For many people, it’s definitely an option to just throw your hands in the air and admit email-based defeat. If you’re operating a fleet of servers, though, somebody might get mad if you turn off their email alerts.
I built a 90% solution to keep the email alerts discoverable while eliminating all of the legacy email baggage. It’s a minimal drop-in replacement for /usr/sbin/sendmail. It sends no email and doesn’t require any mail server setup. It’s setuid so it can log email no matter who on the system calls it. It outputs a JSON object with the calling user, command-line arguments, and the full email body. The focus is on putting the email in a machine-readable log format that somebody (or some system) can handle later. If people want to monitor their alerts, they can use the logging system.
The sendmail-shim utility
It’s a single Go file that you can compile and run. Source (and a README) is available here. This program is honestly pretty small but it does everything you need to handle mail via centralized logging. It emits JSON objects to a single file and is great for the 99.9% of the time when your disk isn’t full. When your disk is full, email failures are probably the least of your concerns. The actual log uploading is handled by whatever log agent you have installed, be it Splunk, Filebeat, or anything else.
I’ve deployed a version of this at my day job and it was wonderful to rid ourselves of creaky MTAs, fleet-wide outbound SMTP, and email that was once discoverable being sent to a distribution list that piled up unactionable garbage as we scaled our fleet. Being able to query email volumes by server and contents made tracking down individual use cases (and silencing unactionable ones) tractable.
It’d be fun to turn this into an actual package to add to the pile of MTAs in Linux distributions. Maybe someday, and pull requests are always welcome!