On-demand TLS Attacks

Over the last couple of months we dealt with a pretty nasty attack on our network — one that targeted something we love to highlight: our on-demand TLS provisioning feature. It's one of the cooler things we do, powered by Caddy, and as it turns out, one of the more abusable features of our platform. So let's talk about what happened, what we built to deal with it, and what else we put in place to make sure it doesn't happen again.

Quick Recap: What Is On-demand TLS?

If you've read our post on transitioning to on-demand TLS, you already know the basics. But for the uninitiated: a TLS certificate is what gives your browser that little padlock in the address bar. It's the cryptographic proof that the site you're talking to is actually who it claims to be, and that the connection is encrypted. We've written a bit about why TLS matters if you want the full picture.

On-demand TLS is exactly what it sounds like — instead of manually provisioning a certificate for every site ahead of time, the server provisions one automatically the moment a new site is first requested. You point a domain at your server, a request hits your server for the first time, and rather than getting a certificate error, the server quickly (sometimes less than ten seconds) handles the whole issuance process in the background during that first connection. No manual steps or configuration required.

We use this heavily. It powers our customers' custom domains on the CDN, and it's also how we handle our object storage bucket domains under skip2userobjects.com. Every customer bucket gets its own subdomain, and wildcard TLS is how we make sure all of those subdomains are served over HTTPS without us having to manually provision a configuration block and certificate for every single one.

The way we do it, using Caddy and its on-demand TLS feature, is quite elegant. When a browser connects to a hostname it hasn't seen before, Caddy intercepts the TLS handshake, notices there's no certificate yet, reaches out to one of our Certificate Providers, gets a certificate issued on the fly, stores it in our centralized certificate storage, and then completes the handshake. All of this happens during the first connection to a new hostname, taking as little as 10 seconds or as much as 60 seconds. After that, the cert is cached and subsequent connections are instant.

It's a powerful feature. Building deployment previews, static build domains, and per-bucket storage endpoints would be a lot more painful without it.

The Catch

Here's the thing about on-demand TLS: it takes time. Issuing a TLS certificate isn't instantaneous — our CA has to receive the Certificate Signing Request (CSR), validate the domain, sign the cert, and hand it back to us. In practice, this takes anywhere from 10 to 60 seconds depending on the CA's current load, network conditions, and a handful of other factors we don't fully control.

During that window, our servers have to sit and hold the incoming connection open. The TLS handshake is just... pending. The browser is waiting, the connection is alive, and our server is committed to it until either the cert comes back or something in the chain times out. Under normal circumstances, this is fine — first-time connections are rare for any given hostname, and the overhead is worth the convenience. But "normal circumstances" is doing a lot of work in that sentence.

The Attack

Sometime last month, we started seeing something weird in our traffic logs. Then the "something weird" turned into thousands of requests per second hitting wildcard URLs that didn't actually exist. The requested hostnames were garbage — strings of random numbers, nonsense words, expletives, the kind of stuff that makes it pretty obvious pretty quickly that you're not looking at organic traffic.

There was a clear pattern: someone had figured out that hammering wildcard domain endpoints with requests for nonexistent subdomains would force our servers to attempt certificate provisioning for each one. Every single one of those requests would open a connection, trigger an ACME order to our CA, and then just... sit there. For up to 90 seconds. While we waited... With thousands of these piling up per second.

The botnet behind it was not huge, and to date we have blocked over half a million individual IP addresses. This is the part that makes these attacks annoying to deal with at the network level: there's no single source to block, and the per-IP rate of requests can actually look pretty reasonable. It's only when you're looking at the aggregate picture across the entire network that the pattern becomes obvious for what it is.

CA Rate Limiting: The Multiplier Effect

Here, things got worse. As the volume of requests climbed, we started hitting rate limits from our certificate providers. This is a standard protection on the CA's end — they don't want any single client hammering them with thousands of cert issuance requests per minute, and you can't blame them.

But, being rate limited introduced two compounding problems:

Requests hung even longer. When the CA is rate limiting us, provisioning requests don't just fail fast — they queue, retry, try other providers, and stall. What was already a 10-60 second wait could stretch significantly longer. Every server holding open those connections was now holding them even longer, burning more file descriptors, more memory, more worker threads. The resource pressure on our edge servers started to climb.
Legitimate customers couldn't provision certificates. This is the one that really stings. A real customer pointing their domain at Skip2 for the first time — someone who just updated their DNS and is excitedly waiting for their site to come up — would get lumped into the same rate-limited queue as all the garbage requests. Their cert provisioning would fail or hang. From their perspective, we just weren't working at all.

It's a pretty well-designed attack vector, actually. You don't need to saturate bandwidth or crash a server outright. You just need to find something expensive that a server has to do and make it do it a lot. On-demand TLS provisioning is expensive. The attacker found the lever.

What We Did About It

A Network-Wide Analysis Tool

One of the things we've always known about our setup is that per-server analysis has limits. Each edge node knows its own traffic, but it doesn't necessarily know what's happening at us-nyc-1 when it's sitting in nl-ams-1. Our Global Firewall API handles IP blocklists network-wide, but we needed something that could analyze request patterns across the whole fleet and automatically make decisions based on that aggregate view.

So we fast-tracked and shipped an internal tool to do exactly that. It ingests request data from across our entire network, looks for patterns that no single server would have enough context to catch on its own, and feeds that intelligence back into our threat pipeline. IPs that look suspicious get flagged for additional scrutiny. The ones that are clearly bad get pushed to our global blocklist and dropped at Layer 3, before they ever reach our proxies.

This kind of tooling is something we've been moving toward for a while — our Global Firewall API was an early step in this direction, and this new analysis layer builds on top of that foundation. The difference is that instead of just syncing per-server blocklists, we're now doing active pattern detection and feeding it back into our threat intelligence automatically.

Rate Limiting Wildcard Domain Provisioning

The second thing we put in place is simpler but equally important: a rate limiter on the number of wildcard TLS certificates that can be provisioned per root domain within a given time window. This caps how many new cert issuance requests any one domain can generate before we start throttling. It directly limits the blast radius of this class of attack, regardless of how many IPs the attacker is rotating through. Even if they get past our IP-level detection, they now run into a hard ceiling on how much damage they can actually do to our CA relationship and our server resources.

Extra Checks

A significant portion of these requests were for domains that did not exist as subdomains of our object storage domain, skip2userobjects.com. This was set up using on-demand TLS, as all requests take the same route and we don't pre-provision them. Requests for buckets on this domain that don't exist will now be denied a TLS certificate.

The Bigger Lesson

On-demand wildcard TLS is useful technology. The convenience it provides both to us and to our customers is real. But it's a great example of a feature that needs to be provided carefully. The same property that makes it powerful (it'll provision a certificate for any hostname on demand) is exactly what makes it a target.

The shorter TLS certificate lifetimes the industry is moving toward are going to increase provisioning frequency across the board, which means on-demand TLS infrastructure is only going to become more of a target going forward, not less. Building the right defenses around it now is the right call.

Running a CDN means running into problems that most people don't encounter at this scale, and we think there's value in writing about how we work through them — even when (especially when) the story starts with "someone tried to break our stuff."

Check Out What We've Built

Create your account