We realized that our solution for securing workloads with fs-verity and eBPF (https://bomfather.dev/blog/bpf_get_fsverity_digest/) does not address critical problems, such as workload identity. This is a gap we found, but we haven’t implemented it yet.
What is the underlying problem?
We address runtime security by granting permissions to read, write, socket connect, ptrace, etc. Additionally, we recently added fs-verity, which further improves this by pinning the executable digest and validating it in the Linux kernel. But there are a few threat vectors we haven’t addressed, and we’ll use the Capital One data breach as an example.
Capital One had a genuine ModSecurity WAF running on EC2, without malware or tampered binaries. Yet, SSRF tricked the WAF into calling the metadata service, which returned the instance’s IAM credentials, and used those creds to access S3, bypassing the firewall. which, in turn, resulted in access to 106 million records and an $80M penalty. The Monitoring missed it because it appeared to be normal API calls from an authorized service. https://madsecurity.com/madsecurity-blog/capital-one-data-breach
The AWS Instance Metadata Service (IMDS) verified that the request originated from this host and returned the host’s identity. Its presence on the server equaled possession of the workload’s identity, and this is common practice often perceived as secure. fs-verity has the same shape using a matching digest for authorization. A kernel hook would not have stopped Capital One since the SSRF made the genuine WAF emit the request, so “is the caller the WAF” passes too.
Another critical issue we observed was that “grant” is usually permanent, with no expiration. For example, fs-verity, IMDS, and a socket allow rule all decide once. The check passes once, and we treat it as permanent. The issue with that is, for example, the preloaded process library can fork into something else, keep running long after the original that launched it is gone, and the box can be compromised around it.
SPIFFE
SPIFFE https://spiffe.io/ and SPIRE provide strongly attested, cryptographic identities to workloads across a wide variety of platforms. SPIFFE is a standard for workload identity, and SPIRE is its reference implementation. SPIRE attests workloads, issues SVIDs, rotates them, distributes trust bundles, and stops issuing identities when workloads should no longer be trusted.SVIDs are short-lived, more like OAuth access tokens than permanent machine labels.
How does this help?
fs-verity verifies that the executable content is trusted, and SPIFFE/SPIRE proves that the running workload instance is trusted, and eBPF enforces what that trusted instance may do.
fs-verity -> right binary
SPIFFE -> right workload instance
eBPF -> right runtime behavior
What policy could look like
policies:
- executable: "filepath = /opt/solana/bin/solana-validator"
fsverity_digest: "sha256:..."
require_spiffe: true
allowed_spiffe_ids:
- "spiffe://validator.example/mainnet/validator/val-001"
can_access_dirs:
- "/mnt/ledger:write"
- "/var/solana/identity:read"
SPIFFE identities should be rotated and should expire. Bomfather can treat identity like a short-lived access token: a valid SVID means trusted, an expired SVID means reverify, and an invalid/missing SVID means revoke trust or block sensitive operations. This allows runtime trust to change without restarting the workload.
SPIRE becomes part of the trust chain, and if the SPIRE Agent is compromised, the trust chain weakens. Bomfather can protect SPIRE itself, fs-verity can pin spire-agent, restrict its filesystem access, protect the Workload API socket, restrict network destinations, and block ptrace/tampering. The bootstrap model will first protect SPIRE using path, fs-verity, and eBPF, and then use SPIRE identities to protect other protected executables.
We haven’t covered how the kernel is aware of the SVID, yet we’ll cover it later, when we implement our design.
Why we think this is cool
We have been hard at work on this problem, and we haven’t seen anyone solve it as a whole, as most systems keep these layers separate: file integrity (TBH, most aren’t aware of this), workload identity, and runtime enforcement. We want to combine them so that the policy depends on the cryptographic workload identity, and the SPIFFE identity becomes more than just mTLS metadata.