Launching Secure External Subdomains
The Platform Engineering team at Affirm has received multiple requests for launching subdomains (xyz.affirm.com) which point to external CMS systems. Historically, if we wanted help.affirm.com to point to api.helpdeskprovider.com, we would typically create a CNAME entry in our DNS system with this mapping.
This approach has been effective to date, but there are some inherent security risks that accompany it. To better understand one of these risks, consider the following scenario.
Cookie Integrity
Suppose you run a small boutique, https://www.scarves.com, that manufactures and sells scarves. Business is going well, so you decide to launch a new promotion that allows your loyal customers to take a survey and receive a free scarf in return.
However, you run this business alone and don’t have the resources to build out a survey system/service. Instead, you find another up and coming small company, Promotions Inc., that offers a platform allowing you to set up such promotions for your customers.
You think it makes for a better user experience if you could keep this promotion within a subdomain, i.e. at https://promos.scarves.com, instead of promotions.com/scarves. This way, users would visit https://promos.scarves.com, enjoy the familiarity of your brand, but interact with content from https://promotions.com. Once a user has completed the promotion, they would be returned to your site to complete a checkout for the free scarf.
Now, let’s suppose that you somewhat reasonably use cookies to authenticate a user’s session. The auth_token cookie, depicted below, contains a value uniquely identifying the user’s session.
Name | Value | Domain | Expires | …=========================================================auth_token | 1b0cpdafkl1 | *.scarves.com | …
If your user is logged into https://scarves.com and then visits https://promos.scarves.com, the user’s browser sends this cookie over to https://promotions.com/scarves/. This cookie data can potentially be misused in two scenarios.
- A malicious employee of promotions.com who has access to logs and/or production systems.
- An attacker who has obtained access to promotions.com
An adversary might see this situation as the perfect opportunity to claim lots of free scarves. To achieve their goal, they would do the following:
- Record the user’s auth_token cookie value.
- Use the stolen auth_token to spoof the original user, claiming the free scarf as their own.
- Bask in the glory of their new scarf pile!
While the above scenario might be contrived, the issue with cookie integrity for subdomains is well-known and even acknowledged within the RFC spec for the HTTP cookie [1]. This renders scarves.com vulnerable to security issues in an external service, over which they have no control.
Cookie security was just one of several concerns that we entertained. Other issues that we considered include:
HSTS
We support HSTS and plan on enforcing HTTPS for all of our subdomains. With the includeSubDomains directive set to True, modern browsers will prevent our users from accessing subdomains without HTTPS.
HTTPS Certificate Control
For external domains to support HTTPS, Affirm will need to generate certificates and share them with the external providers. However, allowing external parties to use Affirm’s certificates is problematic.
For an example scenario, let’s suppose that we give the imaginary HelpDeskProvider, Inc. a certificate to present to users that access their service through the help.affirm.com subdomain. Clients accessing help.affirm.com could then verify that their SSL/TLS session is properly verified with Affirm.
An adversary, Alice, might gain access to the private key used in creating the certificate. Possession of the private key would allow Alice to perform man in the middle attacks, as she could then pose as Affirm in the SSL/TLS handshake.
All of this could happen without our knowledge, and upon discovery of the compromise, we would then have to publish a certificate revocation list (CRL) or use the OCSP protocol to revoke the compromised certificate. However, if Alice is in the position to be able to launch a MITM attack, she could still easily bypass the revocation check [2].
It would be better if we could automatically expire certificates after a short period of time and renew them instead. This way, any compromised certificates would be invalidated after expiration.
Key Needs
For our desired solution, it was clear that we would need at least the following:
- A means of sanitizing which cookies are sent to external parties.
- Controls over certificate rotation/renewal.
- Auditing of external subdomain requests.
Our Solution
Our new approach is to use Amazon CloudFront [3], configuring a CloudFront distribution to:
- Proxy connections to external sites.
- Manage and serve a *.affirm.com wildcard certificate to our users.
- Using Lambda@Edge [4], intercept each request.
- Run custom JS code to instruct Cloudfront to select an appropriate backend external server given the Host Header.
- Strip cookies from the request based on a blacklist before sending the request.
- Log all requests to external sites and subdomains in S3.
Takeaways
We have found key advantages to using the above approach — with the common themes of “ease of use” and “less managing complexity”.
Using Lambda@Edge, we can filter which cookies we want to pass to external partners. All access to external subdomains can be logged and presented in an easily accessible format to the corresponding internal business partners. Furthermore, adding a new subdomain is very simple — requiring only a change to the JS code to insert the new mapping. Certificate rotation and renewal can be handled automatically by AWS ACM [5].
“ACM performs Managed Renewal for ACM’s Amazon-Issued Certificates to automatically renew your Amazon-issued SSL/TLS certificates before they expire. To renew a certificate, ACM generates a new public-private key pair.”
Additionally, through CloudFront we can configure a web-application firewall (WAF) to enable more strict security SLAs on certain subdomains, especially if we expect a particular external CMS system to be more vulnerable. With the new approach, our subdomains can support newer technologies like HTTP/2, IPv6, etc. even if the external partners do not support them.
With all the added benefits come additional considerations. At very large traffic volumes (on a monthly basis), we expect that we may run into Lambda@Edge throttling and high costs. Furthermore, if Lambda@Edge and/or CloudFront were to experience an outage, all of our subdomains would go down as well. As a failover strategy, we have set up a Nginx-based proxy which provides similar functionality to cloudfront. During a cloudfront outage, we can update the DNS entries for a subdomain to point to our nginx fleet which proxy the request to subdomains.
Summary
Once we launched this solution, Affirm’s platform was able to launch 6 subdomains in one quarter. This provides the team with increased visibility, control and security over any externally hosted subdomain. The ability to use an AWS managed service reduces the load on my team in terms of maintaining certificate security and a fleet of WAF enabled nginx forward proxy nodes. This allows my team to focus on our core goal of building software & systems that allow the rest of engineering to produce and deploy software that is scalable, reliable, secure, correct, consistent, instrumented and well-tested.
Fun fact: This page is served via our Cloudfront proxy :-)
Learn More
To discover more about Affirm’s Platform team and other engineering teams, please visit:
