In 2020, I joined a monthly technical review with engineers from several companies in the same investment portfolio. These meetings were normally useful, orderly, and not known for plot twists.
One team was screen-sharing code related to call recordings when a single word wandered past the screen and changed the agenda:
unsigned
There are words that make infrastructure people sit up a little straighter. Unsigned is one of them.
The application authenticated its users, but an unsigned recording URL raised a sharper question: when the browser left the application and asked storage for the file, what carried that authorization decision with it?
The company is anonymized here. The architecture, unfortunately, requires very little embellishment.
Nothing crashed. No alert fired. The demonstration kept moving. That is what makes a missing boundary so easy to miss: the system can look perfectly healthy while answering requests from people it was never meant to serve.
The smallest useful test
I opened a terminal and tried to list the referenced Amazon S3 bucket anonymously. The reconstructed, sanitized equivalent of that command is:
aws s3 ls s3://<redacted> --no-sign-requestThere was no selected AWS profile, access key, or application session attached to the request. In AWS terms, I had arrived without so much as a name tag. The listing succeeded and returned recording objects.
Subsequent minimal verification established that the public policy allowed at least:
| Permission | What anonymous access permitted |
|---|---|
s3:ListBucket |
Enumeration of recording objects |
s3:GetObject |
Retrieval of call recordings |
s3:PutObject |
Arbitrary uploads and potential storage abuse |
Together, those permissions made the bucket more than publicly readable. ListBucket exposed the map, GetObject exposed the content, and PutObject let an anonymous caller add data or potentially replace a known key. The storage layer had quietly become a public API, just not one anyone had meant to design.
The upload permission also created paths for untrusted-object injection, storage-cost abuse, or replacement of a known object key. I found no evidence that those paths had been used.
That was enough. I stopped at the smallest non-destructive test that established the problem and escalated it immediately. Severity did not require a guided tour of customer recordings.
Five years in one bucket
The bucket had permitted public access throughout its approximately five-year existence. It contained approximately 15 TB of MP3 call recordings encoded at about 64 kbps CBR.
At that bitrate, the bucket held roughly six decades of aggregate audio. The exact total moves depending on whether the remembered storage figure meant decimal terabytes or binary tebibytes; the order of magnitude does not. Low fidelity, high consequence.
The company processed approximately 900,000 credit applications over the same period. That is a measure of business scale, not a count of unique people or recordings. Some applications generated several calls; others may have generated none.
The recordings could contain personal information and answers callers used to authenticate themselves over the telephone. That created a credible path to later impersonation and access to additional consumer-credit information. I found no evidence that anyone had used it, though the missing audit trail would soon become its own problem.
This happened before the company’s broader AWS migration. S3 was its only AWS service, attached to an application that still ran elsewhere. One cloud service is still a cloud estate. It is merely a small estate capable of producing a very large incident.
- Exposure duration
- Approximately 5 years
- Stored recordings
- Approximately 15 TB
- Audio encoding
- 64 kbps CBR MP3
- Anonymous permissions
- List, retrieve, upload
- Historical access logs
- None available
- Controlled access restored
- Approximately 2 hours
Authentication stopped at the boundary
The application successfully authenticated its administrative users. Then it handed the browser a storage URL that carried none of that context.
The application could answer, “Who are you?” S3 effectively answered, “Why are you asking me?” and served the object anyway.
That exposed the missing architectural decision:
How does a system that authenticates a user authorize that user to retrieve protected content from another system?
The legacy implementation avoided that decision by removing access control from the storage layer. The application returned unsigned recording URLs, and the S3 bucket accepted public requests. Anyone who knew or discovered the direct endpoint could skip the application entirely.
The bucket hostname was doing the job of a password. Bucket names make useful labels and terrible credentials.
From the browser’s point of view, the unsigned URL was simply convenient. It required no signature, expiration, or policy that connected the request to the authenticated session. Once the URL existed, the application was optional. The system had two interfaces to the same data, and only one of them knew anything about users.
This was broken access control across a system boundary, coupled with a direct-origin bypass. The front door had a lock; the loading dock had a helpful sign pointing to the recordings.
Fail closed
I escalated immediately, and an authorized operator joined the response. I supplied the diagnosis and containment steps; the operator made the bucket private.
The recording feature stopped working. That was expected. We chose a visible outage over an invisible exposure and told the affected teams what was happening. The feature was about to receive an unscheduled two-hour maintenance window.
Availability matters, but it does not outrank confidentiality simply because an outage is easier to notice. Public access closed first; controlled delivery came next.
Containment was blunt, visible, and reversible. That made it a good first move. Incident response rarely awards extra points for elegance while the unsafe path remains open.
- 01Establish severity
Use the smallest non-destructive check necessary to confirm anonymous access.
- 02Escalate and contain
Guide an authorized operator in making S3 private immediately.
- 03Communicate degradation
Tell affected personnel why the recording feature would remain temporarily unavailable.
- 04Restore controlled access
Bridge application authorization into a private delivery path in approximately two hours.
Two hours to restore trust
The replacement had to carry authorization across two doors: first from the application to the content-delivery layer, then from the content-delivery layer to private storage.
The application issued CloudFront signed cookies to an authenticated administrator’s browser. The browser presented the cookie to CloudFront, which validated whether that viewer could request the recording. CloudFront then used an Origin Access Identity (OAI) when reading the object from private S3.
The repair therefore used two deliberately separate grants:
- Viewer authorization: CloudFront signed cookies established that the browser was allowed to request the restricted content.
- Origin authorization: OAI allowed CloudFront—not the public internet—to retrieve objects from the private S3 bucket.
A pass for the first door did not magically unlock the second. That is a feature, not bureaucratic enthusiasm. A direct request to S3 was denied, the bucket stayed private, and controlled service was restored in approximately two hours.
On every recording request, CloudFront now answered the viewer question before contacting the origin. S3 answered a different question: whether the caller was the trusted CloudFront identity. Closing the public policy removed the bypass; carrying explicit grants across both boundaries completed the design.
This was an authorization bridge, not a harder-to-guess URL. AWS documents the same separation today: applications can issue CloudFront signed cookies to authorized viewers, while origin restrictions prevent viewers from bypassing CloudFront to retrieve content directly from S3. The repair used the OAI model documented for private S3 origins. OAI is now a legacy mechanism, but it is the historically accurate name for the design implemented in 2020.
The question the system could not answer
Containment established what would happen next. It could not reconstruct the previous five years.
S3 server access logging had been disabled, and no other retained telemetry could reliably reconstruct anonymous object access. The policy and its age were visible. Its history was not.
I could prove the exposure. I could not prove whether anyone had accessed it.
Historical unauthorized access could neither be proven nor ruled out. “Unknown” was not a satisfying answer, but it was the only accurate one.
Access control determines who should retrieve data. Auditability tells you who did. Logs are time machines that have to be built before anyone needs to travel back.
Beyond the emergency patch
The immediate repair restored one feature. The incident later helped secure leadership support for repository-integrated dependency and source-code vulnerability scanning, repeatable triage and remediation, formal incident-response procedures, defined escalation paths, and clearer ownership of application security.
That was the more durable repair: turning a surprising word in a code review into a security capability that could find, own, and resolve the next problem without relying on surprise.
For a new design today, I would put preventive guardrails in place before the first object arrives. S3 Block Public Access can constrain public access at bucket, account, and organization levels. Prevention is usually less dramatic than incident response. This is one of its better qualities.
What one unsigned URL taught me
The short version, for anyone whose calendar does not permit six decades of audio:
- Guard every route.Authentication on one interface is irrelevant when another path reaches the same data.
- Carry authorization across boundaries.Identity known by one system does not automatically become permission in another.
- Treat URLs as identifiers.A hostname can tell a browser where to go. It cannot tell the service who may enter.
- Govern the first cloud service.A small cloud footprint can still accumulate enterprise-sized risk.
- Fail closed.A short, communicated outage can be the correct price of restoring confidentiality.
- Create evidence early.Logging cannot reconstruct the years before someone thought to enable it.
- Build capability after the patch.The best incident response makes the next incident less dependent on luck.