For the benefit of our clients, we recently did some research on techniques attackers might use to bypass egress filtering security controls. Here’s what we found.

One of the ways the Equal Experts Security Practice brings value to our clients is by sharing information on topics of interest with them. We recently started researching techniques that attackers might use to bypass egress filtering security controls, as we have a number of clients who approach it differently. We wanted to provide a summary of the risks and trade-offs involved in each approach to help guide their decision-making. In the course of this research, it quickly became clear that domain fronting is currently possible, despite news reports from a year ago claiming that it had been disabled by the major cloud providers.

For those that are relying on either a transparent proxy (for example, Azure Firewall or Aviatrix) or an HTTP CONNECT proxy (Squid being a popular open source product), this can be a major concern. Both of these approaches do not terminate the TLS connection to allow for traffic inspection; instead, they rely on the assumption that the FQDN of the TLS connection and the HTTP host are the same.

Attackers can leverage sites that implement domain fronting to trivially bypass these controls by masquerading potentially malicious activity as authentic, whitelisted traffic.

It was interesting to read the recent blog post by SentinelOne on this topic, where they mention that domain fronting was disabled by both Google and Amazon this time last year. They go on to say that “as of April 2019, domain fronting still works on Microsoft Azure [emphasis added]” – implying that it continues to be blocked by both Google and Amazon. However, our research has demonstrated otherwise – domain fronting currently works on all three major cloud vendors. In fact, we’ve successfully demonstrated to one of our clients how this technique can be used on Google Cloud Storage, AWS CloudFront and Cloudflare. Given these services might change after writing this post, we recorded an asciinema cast demonstrating the technique against both Cloudflare and Google Cloud Storage.

What could go wrong?

As with all security concerns, it’s important to understand what could go wrong. This helps inform the impact and therefore the risk faced to the business. Given domain fronting works on major cloud providers and CDNs, it is very easy for an attacker to sign up and create their own HTTPS service to connect to. This could be as simple as a cloud storage account that can be used for downloading additional hacking tools or uploading stolen data. Another scenario might involve malware in your infrastructure communicating with a command and control server without your knowledge. With these capabilities available to attackers, many possibilities exist for novel forms of abuse.

Am I at risk?

Perhaps you already have some form of egress filtering in place and would like to test whether it blocks attempts to use domain fronting. Fortunately there are a couple of very easy tests to validate this. In their article, SentinelOne described how this can be done with a single curl command and two domains, one permitted, one forbidden, both hosted by the same domain fronting cloud provider. This is a variation on the original research that showed how to perform this using wget:

curl -s -H “Host: forbidden.com” https://permitted.com

Some server environments have been hardened by removing unnecessary tools like curl, but even Alpine Linux, which has been “designed with security in mind”, includes wget by default. However, even without tools like curl or wget at your disposal, many environments still rely on OpenSSL. An equivalent test can be performed using OpenSSL as follows:

openssl s_client -crlf -connect permitted.com:443 -servername permitted.com

GET / HTTP/1.1
Host: forbidden.com

If you’re able to successfully communicate with ‘forbidden.com’ (a domain that is not whitelisted in your egress filter) then you can be quite confident that domain fronting can be exploited from within your network.

It’s the approach, not the product

It is important that our research guides our clients towards making informed decisions about how to reduce their exposure to risk. While it’s easy to identify products that can be trivially abused in this way, the real issue is with the type of proxy used for egress filtering, not with any particular product.

With domain fronting, we are relying on large service providers to implement controls that help protect us from harm. We may not have visibility of how effective these controls are, and as this research shows – controls that are imposed one day can be removed the next. The TLS specification states:

Since it is possible for a client to present a different server_name in the application protocol, application server implementations that rely upon these names being the same MUST check to make sure the client did not present a different name in the application protocol.

By this definition, any service provider that doesn’t “rely upon these names being the same” is under no technical obligation to validate the server_name against the HTTP Host header.

Any organisation concerned by this should consider inspecting egress traffic at the application layer, not simply relying on FQDN whitelists that only observe the TLS connection establishment.

We would be cautious using any product that recommends FQDN whitelisting without traffic inspection, unless the shortcomings (such as the inability to prevent domain fronting) are understood and accepted. It is our experience that vendors often do not clearly articulate the associated risks. There are limited scenarios where egress filtering is important enough to be required, but where the risk of domain fronting can be accepted. It is very tempting to choose a transparent proxy for all the benefits they bring – not least the ease of integration into an existing network. Even a CONNECT proxy suffers the same weakness. That decision may prove to be the right one for you, but the only way to be certain is to understand the trade-offs and make an informed decision, rather than accepting vendor documentation as the complete picture.

As is often the case, the grass is not always greener on the other side. Organisations that do choose to implement TLS inspection should equally be aware of the risks of that approach too! Carnegie Mellon University’s Software Engineering Institute conducted research into this in 2015, finding that “many applications that perform SSL inspection have flaws that put users at increased risk”, which subsequently led to a US-CERT Technical Alert being published to warn of the dangers.

Where to from here?

We are strong proponents of risk-based security. It would be unwise for us to give generic advice without knowing your context, but we can help provide information and guidance that helps you make the right decision based on your context and risk tolerance.

We always recommend that you start with a threat model. It stands to reason that you can’t solve a problem that you haven’t defined, and threat modelling is a very helpful tool to uncover the issues you face. Armed with that knowledge, you can then begin to explore whether this particular issue is of sufficient concern in your context to warrant action.

Want to find out more?

If you’re looking at ways to improve how you tackle the challenges of secure software delivery and would like some help, please get in touch. Or if you’re working in the security industry and think Equal Experts is as cool as it sounds (it is!), we’re always looking for talented consultants to work with.