Ethics Notice
Prior to publishing this article we engaged with both Google and Microsoft at both the management level and at their request through their vulnerability disclosure processes. The purpose was to enable them to respond and ideally remediate the issues described. Sadly, their engagement was limited, and their conclusion was that this is not a significant vulnerability. We suspect (but cannot confirm) that this conclusion is based on their processes being aligned with technical/code vulnerabilities, while what we are describing is a significant management vulnerability. We disagree with their conclusion, as do other industry experts we have consulted with in both the public and private sectors. We have chosen to publish this because:
a) It represents a management practice, as opposed to a technical vulnerability, that significantly increases risk but on its own is not directly exploitable;
b) Recent and historical cases demonstrate that no organisation (including Google and Microsoft) can claim that their internal processes will prevent a breach, so every opportunity to build defence in depth should be grasped; and
c) Without evidence that Google and Microsoft are going to address the issue, we believe that it is in the public interest for customers understand the risk that these practices present.
Executive Summary
Email is insecure by design, but lot of effort has been put in to address this over the last few decades because of its criticality to all our lives and businesses. The standards that make it harder for criminals to impersonate your business (SPF, DKIM and DMARC) for fraudulent or other malicious purposes are complex and interdependent. Like all aspects of security, they rely on discipline and good governance, and are only as strong as the weakest link. Security is rarely a binary issue, and relies on judgement to balance security, risk and business efficiency. We have found that the approach taken by both Microsoft and Google in their management of the SPF address space appears to lack appropriate discipline and governance, and tilts the balance significantly in favour of their own business efficiency at the expense of their customers’ email security.
This is important, because these standards and controls when applied on a widespread basis build herd immunity like a vaccine. When the two dominant providers of business email services dilute the effect of that vaccine then the whole herd suffers.
Why is email authenticity important?
Email wasn’t designed with security in mind, and without additional controls can be read or spoofed with ease like a postcard. Over time, this has created significant issues since email has become a primary communication method in our personal and professional lives.
Email is one of the most frequent channels used by hackers to compromise individuals and organisations. If we receive an email from someone we trust, we’re more likely to act on it, click on links, or open attachments. Phishing emails that look like they’re from our bank can trick us into entering our login details on fake websites. Emails pretending to be from a CEO or supplier can persuade employees to transfer money to fraudsters. The list of scams goes on.
How has email security improved?
Several standards have been developed over the years to help ensure that an email comes from the organisation it claims to be from. The three essential standards are SPF, DKIM, and DMARC, which work together to improve email security:
Approved Origin: SPF (Sender Policy Framework) was introduced about twenty years ago and standardised in 2014. It helps verify if an email comes from an authorised server. Organisations publish a list of IP addresses permitted to send emails on their behalf. If an email comes from an unlisted IP address, it’s likely to be fraudulent.
Message Signing: DKIM (Domain Keys Identified Mail) was also standardised in 2014, and uses cryptographic signatures to verify that an email hasn’t been altered and is genuinely from the claimed sender. Organisations publish certificates, and recipients’ email systems use these to verify the email’s signature.
Compliance & Assurance: DMARC (Domain-based Message Authentication, Reporting and Compliance) ties SPF and DKIM to the domain the email claims to come from. It enables domain owners to receive reports on email compliance and provides instructions to receiving systems for handling suspicious emails. This standard was introduced over the past decade to address limitations in SPF and DKIM when applied alone.
A non-technical analogy
To provide an analogy, let’s imagine that we are fighting a war (or playing a game of risk) and are operating with allies against a common enemy. We have established a protocol to limit the risk of “blue-on-blue” casualties that could be caused by forces opening fire on friendly troops thinking they were the enemy. This protocol operates on the following basis:
- All troops are issued with a list of locations we know are controlled by allied forces (SPF equivalent).
- Allied uniforms have unique identifiers that enable us to identify an allied soldier (DKIM equivalent).
- We will recognise a soldier as friendly if they are EITHER located in a place controlled by allies OR are wearing an allied uniform (DMARC equivalent).
Security is only as strong as the Weakest Link
The effectiveness of email security depends on the weakest link. DMARC only requires either SPF or DMARC to pass in order to trust an email’s authenticity. If either mechanism is compromised so that a fraudulent email could pass, the other becomes irrelevant. For instance, if an attacker steals a DKIM private key, they can impersonate the sender without detection, irrespective of the email server they send from. Similarly, if an attacker sends an email from an IP address authorised in the SPF record, it enables them to send fraudulent emails without needing valid signatures.
The issue with Microsoft and Google’s SPF implementation
During the recent development of email protection services in our Talosic® platform for small and medium-sized enterprises (SMEs) we uncovered concerning details about Microsoft and Google’s SPF records.
Ideally, an SPF record should list only the IP addresses authorised to send emails. However, due to the vast volume of emails these companies handle, listing every server IP is impractical. It is quite reasonable to use limited ranges of IP addresses rather than an exhaustive list of individual addresses – this is like authorising an entire street to send letters when that street primarily consists of authorised post offices and courier companies. Also, to avoid everyone having to update their records every time there is a change, large providers offer generalised pointers to be included by customers: Microsoft uses spf.protection.outlook.com, and Google uses _spf.google.com.
While this approach is standard practice, the problem lies in the scope of these SPF record pointers. Microsoft’s SPF record has a scope of more than 2 trillion trillion (2×1024 or 2 with 24 zeros after it) IP addresses, and Google’s is greater still at almost 30 million billion trillion (6×1028 or 6 with 28 zeros after it).
Numbers this large are hard to visualise, so we’ll provide some comparisons. The human population is currently around 8 billion (8×109). Our galaxy consists of around 100 billion (1011) stars. The total number of insects on Earth1 is estimated to be a billion billion (1018). It is estimated that there are a trillion trillion (1024) stars in the entire universe2. The number of IP addresses that Microsoft authorises is greater than the total number of stars in the universe, and Google is authorising 30,000 times more than that.
Why does this matter?
This isn’t to suggest that there are rogue email servers within Microsoft’s or Google’s authorised IP space. However, the vast range of these records presents a significant risk that this might happen. If an attacker manages to set up an email server within this space, they could send seemingly legitimate emails from any customer using these services.
Although we have little choice but to have a high level of trust in the security of these companies, this is a risk that we shouldn’t ignore. The larger the space you need to protect and defend, the harder it becomes and the more likely that you will fail. In large organisations decisions taken today may not be clear to another department in a year’s time when they need to carve out some address space for another purpose. The essential concept of ‘Defence in Depth’ recognises that mistakes can happen, and so places independent controls to protect against the failure of one. The approach that Microsoft and Google have taken with their SPF records delivers no benefit beyond the convenience to them of less governance and management responsibility.
Particularly concerning is that much of Microsoft’s approved SPF address space overlaps with Azure Cloud addresses3. This is also true but to a lesser extent at Google, and only in IPv4 ranges, with an overlap between their SPF address space and the ranges available for their Virtual Private Cloud services4 to the tune of more than 128 trillion addresses. This implies the potential risk that external customers implementing a VM might be allocated an IP address that is authorised by SPF to send email coming from any of the customers of Microsoft 365 or Google Workspace.
Returning to the analogy
In the analogy we now have a situation where one of our allies has put the location of their troops as their whole country, because they feel that at some stage in the operation their troops may exist anywhere within their borders. But there are parts of their country that are less secured that others and they only have total control over a number of cities and towns. This means that enemy soldiers who can slip into the less controlled parts of the country can operate with impunity and without challenge despite not wearing a uniform with an allied unique identifier. Because the protocol is executed automatically by our weapon systems, there is no opportunity to apply human judgement and identify there may be a problem.
What should Microsoft, Google and other Cloud Email providers do?
With IPv6, large subnets are common, but this doesn’t mean SPF records should be so expansive. We don’t suggest narrowing SPF records to individual IP addresses, which would cause performance issues. However, the scope should be significantly more tightly controlled, especially given their customer base’s scale and sensitive nature, including large corporations and government departments.
- There should be a target for the proportion of total address space that is not used for authorised email servers; we would argue that this proportion should no exceed 50%.
- There should also be far tighter governance over changes to assets in this address space than other less critical services; and
- There should be similarly tight governance over the content of the SPF records that are provided for their customers to use.
We have singled out Microsoft and Google in this article, because of the scale of the issue in their records, and also because of their dominant position providing cloud email services to businesses around the world. However, the principles apply to all providers of email services, and we have found numerous examples of the same situation happening elsewhere.
One other example with different characteristics is Zendesk, the CRM provider. The number of addresses is many orders of magnitude smaller at around 26,000. But it includes every single address that they own, including thousands that they have allocated to specific customers to manage. For example, more than 4000 of the addresses are allocated to a Russian telecoms company called Сигнал (Signal) based in the City of Zheleznogorsk identified by broadband.signaltv.net. Now Zendesk claim that they have controls that prevent their customers from sending email direct from their leased infrastructure. But if those controls failed for any reason then there would be nothing else to stop these third parties from sending authoritative email imitating any of Zendesks other customers. The SPF record should be authorising Email servers, not an organisation’s entire tech infrastructure.
What do we need to do?
What can we do if Microsoft and Google don’t tighten their SPF records? One option is to rely solely on DKIM, but this could lead to legitimate emails failing to be delivered. Consequently, organisations might relax their DMARC policies to avoid non-delivery of genuine messages, which is not ideal as it would lower the protection against spoof messages as well.
At Digility we are building controls into our Talosic® SaaS platform to address the problem in three ways:
- By analysing DMARC reports, we can identify whether a Client’s DKIM implementation and the nature of their stakeholders is sufficient to rely on DKIM only for mail originating from sources that have very large SPF address spaces;
- Our DMARC report analysis also enables us to identify evidence that a source with a very large SPF address space might have been compromised in the way we have described; and
- We offer to manage our clients’ SPF records dynamically for them and remove a very large SPF address space if the risk of DKIM failure is very low, or if there is evidence that a malicious mail server may be operating from an approved IP address. This can be done while retaining tight DMARC policies (‘quarantine’ or ‘reject’) and therefore maintaining the highest levels of protection.
Summary
Good cyber security is a complex task, and one false step can leave an opening for malicious people to exploit. The vast majority of businesses, 99% of which are small and medium sized enterprises, are unlikely to be equipped with the knowledge, expertise and resources to be able to manage many of the more technical issues themselves. This is why we expect the Big Tech businesses like Microsoft and Google to not only do what is right, but also to lead by setting an example for the rest of the industry.
This is also an example of why at Digility we are developing our Talosic® platform, designed specifically for SMEs so that they can focus on their businesses, confident that we have configured their security for them according to their unique priorities, objectives and appetite for risk.
