Gmail drop support for checking other accounts

October 3rd, 2025 by
A photograph of Whitby Abbey graveyard

Another feature killed by Google.

A recent Google support article has quietly announced plans to drop POP3 support from Gmail in January 2026. On the face of it, this is no big deal. For most purposes, POP3 has been pretty much replaced by IMAP anyway, but there’s a more important change buried in the article.

The issue is confused by the fact that Google use the “Gmail” name to refer to two completely different things:

  • The Gmail mobile app that lets you read email on your phone and tablet.
  • The Gmail web interface that lets you read email in a browser on your computer.

The Gmail mobile app lets you connect to multiple different mailboxes, and will continue to do so, just not with POP3. IMAP is much better for this purpose, as it supports mail folders, and properly supports access from multiple different devices, so the removal of POP3 support here is no big deal. If you use the Gmail app to read mail in a mailbox hosted with Mythic Beasts, it’ll continue to work just fine.

The significant change is this one:

  • The option to “Check mail from other accounts” will no longer be available in Gmail on your computer.

“Gmail on your computer” is Google-speak for “the Gmail web interface”. The “Check mail from other accounts” option is a feature that allows you to pull in mail from other mailboxes and drop them straight into your Gmail inbox. It behaves as if you had just forwarded mail from your other address to your Gmail address, but with one crucial difference – it works reliably.

As we discussed in a recent blog article, efforts to make it harder to spoof email have also made it much harder to forward email reliably, and until now, the “check mail from other accounts” feature has been our recommended way to “forward” mail to your Gmail inbox.

Hopefully Google will contact users to tell them about the change, but we are also doing some log analysis and will be contacting customers who appear to be relying on this feature to collect mail from their Mythic Beasts mailboxes.

Customers who are currently using this feature will either need to revert to simply forwarding emails to Gmail — with the associated risk that Gmail may reject some of your legitimate email – or switch to a different webmail platform. And on that last point, we’re working on a refresh of our own webmail platform that we’ll be announcing in the near future.

Funding Open Source DNS

September 24th, 2025 by
xkcd cartoon describing all infrastructure depends on a project maintained by a random person

The DNS Fund exists to help maintain these critical projects. (Image xkcd 2347; CC BY-NC 2.5)

One of our founders, Pete Stevens, has joined the expert advisory panel for the Nominet DNS Fund. Nominet are intending to distribute £370,000 to open source DNS projects.

The fund has a very simple goal, to improve security and sustainability of open source DNS projects. The means is similarly simple, make it as easy and straightforward as possible for open source critical infrastructure components to access needed funds.

We’re excited to be able to help to steer the fund. One thing that attracted us was the implementation goal of ‘Minimum Viable Bureaucracy’; the application form must be short and it must be simple to complete. The fund wants to fund people and organisations that are good at writing and maintaining software, not specialist form fillers. An individual applying has to answer eight questions and a maximum of 2000 words. We enthusiastically encourage answers that are short and to the point (not least because Pete will have to read them!).

We’re also overjoyed that the fund can consider ‘boring’ applications. Maintaining a project or bringing more developers in to ensure long term sustainability of a project are things that are in scope. The fund does not require new features and new code; fixing and improving existing code is fine. The fund will also be able to support projects that are dependencies of DNS projects, as well as those which directly relate to DNS.

The application form is here. Please share with any project that is in scope and would benefit from funding.

Web hosting and Cloudflare

September 3rd, 2025 by
Gas flare, PetroChina Jabung field, Jambi, Indonesia

Not this kind of cloud or flare.

After careful consideration, we have reluctantly taken the decision that the use of Cloudflare will not be supported on our Web and Email Hosting service. First two clarifications:

  • This only applies to our Web and Email Hosting services. It does not, and will not, apply to virtual servers (VPS), dedicated servers or Raspberry Pi servers.
  • We have never formally supported or encouraged the use of Cloudflare with our Web and Email service. Until now, we have generally discouraged it, but we’ve tried to accommodate users who chose to use it.

The problem

As we’ve written about previously, we’ve seen a huge increase in the amount of abusive bot traffic hitting our web servers — much of it badly behaved AI scrapers — and frequently the volume of traffic is such that it overwhelms the server, and makes websites unavailable. At times we’ve seen over 95% of traffic coming from AI scrapers. Our Web and Email Hosting service is what’s often referred to as “shared hosting” meaning that we have websites for many customers on a single server. This is a very cost effective way to provide web hosting, but it does mean that any load issues caused by traffic to one website can affect other sites on the same server.

Our primary tool for dealing with abusive traffic is to identify the source of traffic and block the IPs that it’s coming from.

Cloudflare provides a service that seeks to protect websites by blocking abusive traffic. It does this by operating a “reverse proxy”; rather than web traffic arriving directly at the web server, it is instead sent to Cloudflare’s servers. Cloudflare inspects the traffic, filters out the abusive traffic, and then forwards the legitimate traffic to the actual web server. This has the effect that all traffic arriving at the web server appears to come from Cloudflare’s IP addresses, rather than the actual client IPs.

Some of our customers have chosen to front websites hosted on our shared hosting servers with Cloudflare. The problem is that Cloudflare isn’t perfect; it doesn’t succeed in filtering out all abusive traffic. This is particularly true of the free tier that we tend to see used in conjunction with our Web and Email Hosting service.

Unfortunately, when we see a large volume of abusive traffic arriving via Cloudflare, we are faced with a choice: either we block Cloudflare’s IP addresses, knowing that this will take all websites using Cloudflare completely offline, or we accept the traffic, which potentially has an impact on all websites hosted on that server. With the growth in volume of abusive traffic, we are being forced to make this choice increasingly often.

We have now taken the decision that we will treat Cloudflare’s IPs like any other IPs: if we see abusive traffic from them, we will block them. In the future, we may introduce a permanent block, or redirect traffic to a support page on our site that explains why Cloudflare is not supported on the service in order to avoid customers inadvertently using an unsupported configuration, but we will contact customers who appear to be using Cloudflare prior to taking this step.

FAQs

Obviously these questions are not frequently asked as this is the first announcement of this change, but whatever; here are some questions and answers:

What have you got against Cloudflare?

None of this is a criticism of the service that Cloudflare provides. The same would apply to any reverse proxy service placed in front of our servers that prevents us from seeing the actual source IP, thereby removing our ability to effectively filter traffic ourselves.

We do have concerns about any service that breaks the end-to-end encryption of web traffic, but that’s unrelated to this issue. Cloudflare (and any other such service) relies on terminating the TLS connection from the client on their servers, inspecting the traffic, and then making a new secure connection to the target web server.

Why do you think you’re better at this than Cloudflare?

We’re not, but when a particular attack is affecting our ability to provide our web hosting service to hundreds of customers, we’ve got a much stronger incentive to resolve it quickly.

Cloudflare definitely has some very impressive technology, but we suspect that much of it isn’t provided in the free tier of their service. Upgrading to a paid-for tier, or careful configuration of the free tier, might yield better filtering, but in our case, we’re affected by the most permissive configuration; it only takes one customer to point Cloudflare at our server with minimal filtering and we’ve got a mix of legitimate and abusive traffic arriving from the same IPs and we’re back where we started.

Cloudflare includes the client IP address in an HTTP header – why don’t you filter on that?

Filtering based on the source IP address of a TCP/IP connection can be done very efficiently because you only need to look at the packet header. Filtering based on an HTTP header requires accepting the connection, setting up a TLS connection and decrypting the content, and then parsing the headers, which is a significant overhead, and in extreme cases, the load from doing this is prohibitive.

Can I use Cloudflare on my VPS or dedicated server?

Yes, absolutely. The issues described here that make Cloudflare such a problem for our Web and Email Hosting service don’t apply to VPSs and dedicated servers, so if you want to put Cloudflare — or any other reverse proxy — in front of your server, you are free to do so. Of course, dealing with any abusive traffic that slips through is your responsibility.

Can I use Cloudflare on my managed server?

If you have a managed server with us then we will attempt to resolve any load issues caused by abusive traffic as part of this service. Please do not install Cloudflare (or similar) in front of your managed server without discussing it with us first, as it will hamper our ability to respond to any incidents. A special case is our WordPress Shared tier of managed WordPress hosting. As the name suggests, this is implemented on our shared hosting platform, and so we cannot support the use of Cloudflare with this service.

VMHaus closure & NLNet donation

June 18th, 2025 by
prepayment meter accepting coins

VMHaus implemented Pay just before you Go.

In 2018 Mythic Beasts acquired VMHaus, a small provider of very low cost virtual servers.

As an independent virtual server hosting provider, VMHaus was not financially viable. Post acquisition, we significantly reduced the costs of running VMHaus by using economies of scale from Mythic Beasts. We could recycle retired servers and disks from Mythic Beasts into VMHaus making their hardware effectively free. We provided rack space and transit from Mythic Beasts data centre space at cost price, taking advantage of Mythic Beasts economy of scale and buying power. However, an energy crisis and high inflation post-Covid meant that VMHaus would likely never become financially viable and in 2024 we took the decision to close the company. We gave all VMHaus customers six months notice to migrate to another server, with an offer of discounted hosting in the Mythic Beasts cloud.

VMHaus ran used a pre-payment model; customers had to buy credits in advance, then use up the pool of credits by running virtual servers. If your credits ran out, the servers stopped working and it was your responsibility to refill the meter before this happened. They also had some neat technical features we’ve incorporated into the main Mythic Beasts cloud – per-second billing, cloud-init for customising installs at boot and fully private network segments for each virtual server with IP address portability.

The pre-payment model made the shutdown of VMHaus a bit more complicated as VMHaus held funds that would not be used prior to the shutdown of the company. In order to refund unused credit, we needed customers to tell us where to refund it to, so in order to put a time limit on the wrap-up, we gave customers with positive balances a choice: get a refund, or donate the balance to the NLNet Foundation, defaulting to the latter if we don’t hear from you. The credit balances were typically very small, and in many cases, the accounts had been inactive for a number of years. Donated balances were each rounded up to the nearest dollar, reflecting the fact that this option saved us PayPal payment fees.

The NLNet Foundation funds projects that create and maintain key internet infrastructure – the sort of software that VMHaus and Mythic Beasts rely on.

We’re very pleased to say a large number of customers actively asked us to donate their balance, and combined with the balances from customers who didn’t respond to our many email reminders,we ended up with a final balance of $5240. We rounded this up to €4550 and sent it to NLNet Foundation.

MOSS – the antidote to SaaS

May 23rd, 2025 by
Moss Gametophytes Sporophytes

Not this kind of moss

Software-as-a-Service has transformed the way that we use computers. The ability to access services anywhere, on any device, without having to install, manage or upgrade applications has obvious advantages. You can be up-and-running on GitHub, Slack or Box in a matter of minutes, and very often there’s a free entry-level tier that requires no up-front payment at all.

Even when you move into the paid-for tiers, replacing the cycle of licence fees and upgrade fees with a simple ongoing service fee can also be attractive.

But SaaS comes with a major drawback: lock-in.

You’re at the mercy of a single provider, and your only recourse if you’re not happy with the service is a potentially complex and expensive migration to a completely different platform. It may not be easy, or even possible, to export your data from your old provider.

And those “free” entry-level tiers have to be paid for somehow. As your usage increases, costs can quickly become very expensive. Or the unprofitable cheap and free tiers get withdrawn or restricted as the business needs to move its income stream from investors to actual customers.

Self-hosted open source

Open source alternatives to many popular services exist, and the obvious alternative to SaaS is self-hosting: run your own server and install the software yourself. You’re completely in control, your data is on your servers and there’s no risk of lock-in.

But self-hosting loses many of the advantages of SaaS; you’re now responsible for running the servers, applying security patches, maintaining backups, and monitoring for issues.

Managed Open Source Services

At Mythic Beasts, we offer an alternative: MOSS – Managed Open Source Services. With MOSS, you get the convenience of Software as a Service, but without the lock-in. We manage the service, apply security updates, provide 24/7 monitoring and take regular backups, but your data remains under your control: we’ll happily give you full access to all the data on your server at any time.

By using open source software, you’re not tied to a single provider. If you’re not happy with our service you can migrate to another provider, or even switch to self-hosting, without the cost of having to familiarise users with different software.

And if you decide you do want to change software, open source puts you in a much stronger position for planning a migration.

Our pricing structure is straightforward, and tied to what it costs us to provide the service. With no free, entry-level tiers that need to be subsidised by the higher tiers, prices scale with your usage, rather than leveraging your lock-in.

We’ve been hosting exclusively on open source software for nearly 25 years, and provide Managed Open Source Services for a range of applications, including GitLab, Mattermost, and NextCloud. For more information, please see our Managed Applications page, or contact us to discuss your requirements.

Self-driving robot dragons

April 14th, 2025 by
Little Devils team of five primary school girls holding the duck covered prize winning robot dragon.

Little Devils and their duck covered robot dragon.

Robocon is a competition run by Hills Road Sixth Form College in Cambridge. Teams of secondary school students build autonomous robot dragons which rampage around the area stealing sheep and gems and taking them back to their lair. The more things your robot dragon steals the better. As soon as we heard there were Robots and Dragons we sponsored one of the teams: the Little Devils.

The Little Devils are five primary school girls who entered a competition for secondary students because the rules didn’t forbid it, and somehow everyone forgot to tell them it was too hard for them. Which is just as well: it wasn’t.

The contest

The goal of the contest is to build a robot and program it to complete a set of challenges autonomously (without human input) within a specially designed arena. Each team gets a quarter of the arena as their home area. At the back of the area is their lair. The arena contains sheep and gems (cardboard cubes) and points are awarded for each sheep returned to the team’s home area, with extra points for placing a sheep or gem on top of the lair. This involves picking the cube up and placing it on a 15cm high shelf.

2D barcodes are displayed around the arena, on the lairs, and on sheep and gems.

Each team is supplied with a ‘brain’ as a starting point for building their robot. This is a Raspberry Pi Zero with an attached camera. This comes with OpenCV and some libraries that tell you every time the camera sees a 2D barcode, along with an estimate of how far away the barcode is and what angle it’s at. The brain also has motor controllers, and general purpose IO.

The Robot Dragon battle arena

The battle arena for the Robot Dragons, with machine readable codes on all the sheep (cubes) and gems (coloured cubes).

Collecting sheep and gems is made even trickier by the fact that once you get close to one, the camera can’t focus on it, so the robot has to guess exactly where to stop in order to pick it up. Once picked up, the robot can then search for the marker for its lair and drive towards it to drop the item off. If they taught trigonometry in primary school they could write more sophisticated software that would work out how to travel based on any marker they can see. We’re already looking forward to next year.

This year’s competition took place on 9th and 10th April, and we were pleased to see a variety of robot designs and programming approaches. Some teams tried to round up as many sheep into their space as possible. The Little Devils’ approach to collecting sheep and gems was a vacuum grabber which would try to pick up the cubes and then drive them back to their lair to get the bonus points.

The Robot

Little Devils robot, laden with ducks

Dragon with robot arm, sucker and a lot of ducks.

One of the most important skills in professional programming is the rubber duck technique. This robot has a Redundant Array of Independent Ducks such that programming can continue at full speed even if a duck is lost in the battle. The ducks were a late addition, and resulted in some last-minute code changes; in testing the Dragon wasn’t laden with ducks and moved faster, so some constants in the code had to be tweaked in order to drive the motors harder to cope with the additional weight from the ducks.

The robot has a software-controlled robot arm to pick up the sheep and gems. There’s a vacuum pump that switches on when the robot thinks it has a gem. Once the pump is on, it lifts up the arm high in order to clear the camera’s view. There are two powered wheels; selectively driving one or both allows the robot to turn or travel forwards or backwards. All of the software was written by the team members with the robot manufacturing done under adult supervision.

The result

We’re really pleased to report that the Little Devils took the third place trophy. We are somewhat concerned that this appears to be the plot for an origin story where a set of super villains destroy humanity with their robot army. Perhaps we should have put “you must not take over the world” in the sponsorship agreement.

Abusive AI Web Crawlers: Get Off My Lawn

April 1st, 2025 by

Blissfully unaware that his TV is DDoSing the internet.

As many other folks have reported in the last few weeks, we have also been seeing a huge increase in the amount of traffic from abusive web crawlers.

Automated blocking of abusive traffic has long been a necessary evil. We already block a number of badly behaved SEO and AI crawlers on our shared hosting servers and, on-request, some customer servers. We also have a number of automatic tools to block abusive clients. These are typically attempting to brute force passwords or run web security scanners, and we firewall IP addresses out after a number of suspicious requests. These crawlers and bad-actors can already outnumber real organic traffic but the scale of the recent activity, along with the lengths taken to frustrate automated blocking, are on another level.

This new attack comes from a great many IP addresses, each making a tiny number of requests – often just one – from viable-looking but randomly generated User-Agents. We’ve had some success detecting and blocking these but this has not been without some problems. There have been periods where some of our servers have been struggling under the sheer number of connections they’ve had to deal with and some of the blocks we’ve put in place have impacted some legitimate users, especially those on very old computers. If this is you then we’re sorry you’ve been caught up in this.

To give you some idea of the scale, one of our shared hosting servers has in the last month been averaging over 1.5 million fraudulent requests from 290,000 unique IP addresses per day. These are addresses that we have a very high confidence are not making legitimate requests. We’ve identified 5.1 million unique IP addresses during this period and 3.4 million of those have only made a single request, which has made it very difficult for us to block them proactively.

Chart showing a month of abusive requests and unique IP addresses making them.

From these IPs, we’ve seen 2.4 million unique User-Agents, and again 1.9 million of these have only been seen once. We’d certainly be surprised if there are as many Windows 95 users left as we’ve seen in these logs. Especially with the ability to talk to a modern TLS-enabled website.

The vast majority of these requests are from consumer ISP networks from a wide variety of countries with Brazil being the biggest contributor by far. UK networks only make up around 2% of the attacks we’ve seen but it’s the same pattern – it’s the big consumer ISPs we see the most. We’ve been careful to exclude any IP addresses that also show what looks like legitimate activity here, so it’s possible this is undercounting, especially with the growth of CGNAT. Around 5% of the requests are from IPv6 addresses.

Pie chart showing top sources: BR 37% US 8% IN 6% CN 4% TR 4% SA 3% RU 3% MA 3% GB 2% AR 2% PK 2% and Other 28%

The rumours that this is a botnet mostly made up of compromised Android SetTop Boxes that’s been leased out to an AI crawler that’s trying to avoid being blocked seem likely to us, but we’re not impressed. This has been a significant waste of staff time over the last few weeks as we’ve worked to mitigate the impact on our customers.

If you’re an ISP that wants to know if you’re part of this botnet please check to see if your ASN is in this file then contact support for a copy of our logs. There are over 22,000 distinct ASNs in our data, with more than 200 of those networks based in the UK.

Thanks to bgp.tools for the network and country data.

Email alphabet soup

March 14th, 2025 by

Modern email involves a confusing array of different acronyms. Most of these are attempts to fix the problem of email being fundamentally insecure, with no way to authenticate the sender of an email. The remainder are attempts to fix the new problems created by attempting to fix the first problem.

This blog post tries to provide a concise glossary of these different technologies, and the associated DNS records and URLs needed to make them go.

Cat sniffing letters spelling out email related acronyms

CAT is not yet an email-related acronym

SPF: Sender Policy Framework

Publish a list of servers that are allowed to send mail from your domain.

SPF is published as a DNS TXT record for the domain itself (an apex record), which states which servers are allowed to send mail from your domain. For example:

example.com. IN TXT "v=spf1 include:_spf.mythic-beasts.com ~all"

SPF breaks email forwarding, unless you use SRS. SPF only restricts the “envelope sender”, which is not normally visible to end users.

SRS: Sender Rewriting Scheme

Rewrite sender addresses when forwarding mail in order to avoid failing SPF checks.

SRS is a technique used when forwarding email that replaces the original sender address with an address in your own domain. SRS only affects the envelope sender, which is not normally visible to end users. SRS allows email forwarding to work with SPF.

DKIM: DomainKeys Identified Mail

Digitally sign email envelopes.

Public keys are published in DNS records, allowing recipients to verify that the email is authentic. Public keys are published as TXT records within the _domainkey subdomain. For example:

mythic-beasts-k1._domainkey.example.com. IN TXT "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG...."

The first part of the hostname is a unique identifier for the key. The signature is added to the email as a DKIM-Signature header. The header includes a field (the s field) with the name of the key used to sign the message.

DMARC: Domain-based Message Authentication, Reporting and Conformance

Tell recipients to reject email from your domain if it isn’t DKIM signed and doesn’t pass SPF.

DMARC is a mechanism that allows a domain owner to assert that all email sent will pass either DKIM or SPF validation, and if it doesn’t recipients should reject it. Subtly changes SPF behaviour so that it binds the “From” address (which is the one that users usually see) rather than the envelope sender. Breaks email forwarding even with SRS unless messages are DKIM signed. Example DMARC record:

_dmarc.example.com. IN TXT "v=DMARC1; p=reject; rua=postmaster@example.com"

ARC: Authenticated Received Chain

Enable mailing lists to forward DKIM-signed emails.

ARC allows a system that forwards mail to provide a digitally-signed summary of the results of SPF, DKIM and DMARC validation at the point thta it was received by the forwarding server. This enables the intermediary system to make changes to the message (such as adding a mailing list footer) that will break the original DKIM signature, but still allow the final receiver to verify its integrity. ARC relies on recipients trusting the intermediary. ARC uses the same DNS-published DomainKeys as DKIM.

DANE: DNS-based Authentication of Named Entities

Publish TLS certificates in DNS, and require TLS when connecting to your servers.

DANE is not specific to email, but it can be used to enforce the use of secure TLS connections when mail servers talk to each other. In the absence of DANE, mail servers will generally try to use TLS if possible, but fall back on an insecure connection if it doesn’t work. By publishing a TLSA DNS record, domains can enforce that TLS is used when delivering mail to the servers listed in its MX records. DANE relies on DNSSEC, and also provides an alternative to Certificate Authority-based authentication of TLS certificates.

MTA-STS: Mail Transfer Agent Strict Transport Security

Require TLS when connecting to your servers by publishing a policy at a well known URL.

MTS-STS allows you to require secure TLS connections when mail servers connect to your server by publishing a machine-readable policy at a well-known URL (https://example.com/.well-known/mta-sts.txt). MTA-STS is an HTTPS-based alternative to the TLS-enforcement part of DANE.

Supporting the Open Rights Group

February 25th, 2025 by

Early in 2023, we started providing sponsored hosting to the Open Rights Group (ORG), in order to support their campaign for digital rights in the UK.  Our motivation for sponsoring them is simple: we require security in order to provide our services, and our encryption requirements have only increased since we last mocked the government’s plans to ban it. The current government’s latest efforts on this front underline the ongoing importance of the ORG’s work.

Encrypted data can only be accessed by the holder of the key. This locks out everyone else – including the government. Successive governments keep trying for a magical solution which gives them, and only them, access too, despite the fact that such an approach only serves to undermine the security of the legitimate, everyday uses of encryption. It turns out that when faced with implementing the impossible, providers will simply turn off security entirely, as Apple have just done by disabling Advanced Data Protection for UK and only UK users.

We continue to support the Open Rights Group and their work to try and allow UK users to secure their own data.  Our sponsored services provide public facing servers for the ORG website, the @openrightsgroup@social.openrightsgroup.org fediverse presence and the blocked.org.uk service which tracks which sites have been blocked by major UK ISPs. We also host internal  Nextcloud, Collabora Office, Matrix and email services.

 

Post by @openrightsgroup@social.openrightsgroup.org
View on Mastodon

 

The Death of Email Forwarding

January 29th, 2025 by
Café menu, listing various food options, almost all of which include Spam.

While it’s Spam™ rather than spam, the proportion is about right.
(Image from Eduardo Unda-Sanzana from Antofagasta, Chile, CC BY 2.0 via Wikimedia Commons)
.

A History Lesson

Back in the mists of time, not that long ago, you could happily forward email to another account sending all your email to a single mailbox and replying through your internet provider’s SMTP server with whatever address you wanted as the sender.

Unfortunately, back in the 1970s, someone realised you could send that new fancy ‘electronic mail’ to people you didn’t know. The few hundred users affected were rather negative about it, and it didn’t really catch on. But, within a couple of decades, the ARPANET became the Internet, and unsolicited email started to become annoying.

With the ‘Information Superhighway’ appearing in the mid 90s and the growth of personal computers, the Internet started to grow rapidly, and ‘spamming’ (a term adopted from Usenet) became more common.

Blacklists and whitelists were developed, with lists of known spammer email addresses and IPs which were known to send spam, but the fact is that email addresses could be trivially faked, and there were always other servers to send mail through.

Over the years there have been various different approaches to authenticating email senders, but it’s a continual arms race. Spam remains a problem, as does the plethora of scams and fraud that are enabled by the ease with which email can be forged.

SPF & SRS

In the early 2000s,  ‘Sender Policy Framework’ (SPF) started to gain traction. SPF works by publishing a DNS record that declares which servers are allowed to send mail from your domain.

Unfortunately, SPF had two fatal flaws:

  1. Pretty much by design, it breaks email forwarding. You can’t, for example, use our servers to simply forward mail to your Gmail account, because our servers aren’t allowed to send mail from the original sender’s domain.
  2. SPF restricts the envelope sender, which is the hidden address that bounce messages get sent to. The one that users actually see in their mail client – the “From” address – is still unrestricted!

(1) could be fixed by another Three Letter Acronym: SRS or ‘Sender Rewriting Scheme’. As the name suggests, this replaces the original sender address with a different address in your own domain, and because this sender is usually hidden, the received mail looks no different. We’ve supported SRS forwarding on our servers for a long time, and until recently, it worked well.

(2) was a trickier problem, which we’ll come back to…

DKIM

In the 2010s and with processing becoming cheaper, ‘DomainKeys Identified Mail’ (DKIM) also appeared, which electronically signs the mail so that the remote server can verify the signature. The public keys are published in the sender’s domain’s DNS, so recipients can verify that mail came from the claimed sender.

DMARC

The issue with DKIM is that it’s not universally adopted, so whilst you can check if a signed mail is legitimate, you can’t assume that an unsigned email isn’t. Enter yet another acronym: DMARC (‘Domain-based Message Authentication, Reporting and Conformance’).  DMARC allows a domain-owner to publish yet another DNS record which says “all my mail will either be DKIM signed, or pass SPF, or both, and if it isn’t, bin it!

Adoption of DMARC has been slow, not helped by the fact that publishing a strict DMARC policy in 2014 broke just about every mailing list of the time. Fast-forward ten years, mailing lists have learned to adapt, and the big mail providers are pushing the world towards DMARC. Gmail, probably the largest free email provider, now requires either SPF or DKIM to be enabled on a domain in order for them to accept any mail from it, and if you’re sending bulk email, you need a DMARC record too.

The Effect On Forwarding Mail

While these measures have all helped greatly to combat spam, with a large proportion of it now coming from compromised servers or mailboxes, it’s had some down-sides, the most obvious of which is difficulty in forwarding email.

Lots of our customers like to register a domain with us, set up some email addresses in that domain, and forward them to existing email accounts at other providers, such as Gmail or Yahoo. We also host lots of clubs and other organisations that have role-based addresses (like “treasurer@myclub.org”) that forward to the person currently doing that role, and mail expanders like “committee@myclub.org” that forward to a handful of people.

DMARC alignment

SPF is now widely adopted and, as noted above, you used to be able to work around the issue that it creates for forwarding using SRS.

But, one of the subtle features of DMARC is that it quietly fixes problem (2) with SPF. In order to pass SPF in a way that makes DMARC happy, the hidden sender address that SPF restricts has to be in the same domain as the visible “From” address (this is called “alignment”).

  • If you forward without SRS, the messages that you forward will fail an SPF check.
  • If you forward with SRS, the messages will technically pass SPF, but DMARC will call it a fail because the sender and from addresses don’t match.

In other words, there is no way to pass a DMARC check using SPF when forwarding mail. You have to rely on DKIM, and hope that forwarding mail (with or without SRS) doesn’t break DKIM.

You would hope that senders wouldn’t publish a strict DMARC policy without first ensuring that all of their outbound mail is DKIM-signed, but you may be disappointed, and with mail providers increasingly enforcing DMARC policies, mail forwarding has become less and less reliable.

Yet another acronym – ARC – promises to fix mail forwarding, but it’s not a magic bullet. We’ll discuss that another time.

Of course, forwarding mail works fine if you’re in control of the server you’re forwarding to, and can whitelist the server doing the forwarding, but that’s not common. Most people want to forward to providers like Gmail where they have no such control.

For some platforms there is a workaround; rather than forwarding mail, have the platform collect it from a mailbox using POP3 or IMAP.  For example, Gmail have their “check messages from other accounts” functionality in their mailboxes, which picks up you mail from a mailbox as if it’s been received by the Gmail mailbox directly, and other platforms are adding similar options. [2025-10-02] Google are removing this feature from Gmail.

This works well enough for personal email addresses, but it’s a lot more hassle for things like role-based addresses at emails at clubs and societies, and doesn’t work at all for mail expanders where multiple recipients need to collect the mail.

This is why we can’t have nice things.