When building our eBPF agent, we knew it had to keep working even if the server went down. We also needed to make sure it could reliably deliver messages to the server.

The requirements:

1. The agent should operate independently, even when the server is down.
2. On reconnection, the agent should send the queued data to the server.
3. The data should be compressed, and we should utilize batching to minimize round trips and preserve ordering.
4. We didn’t want to slow down our eBPF kernel code when sending data to the server or drop any messages.
5. Provide Prometheus style metrics for monitoring agent health.
6. Use an open, battle tested communication protocol.
7. Prevent all clients from reconnecting at once to avoid the thundering herd problem.
8. Implement exponential backoff for reconnect attempts.
9. The agent may send duplicate messages, with the server responsible for deduplication.

Architecture

We use gRPC with ztsd compression and a bidirectional stream for communication between the agent and the server. Inspired by eBPF, we built a userspace ring buffer to hold data sent from the agent to the server, maintain message order, and avoid blocking during transmission.

The agent sends the API key, hostname, and correlation ID as gRPC metadata, and uses TLS for https:// endpoints.

What does “reliable” mean?

The agent stores events in a userspace ring buffer after reading them from kernel ring buffers. It then batches these events, with batch size configurable, and sends them to the server, tagging each batch with a UUID and sequence number. This way, if the agent sends the same batch more than once, the server can identify duplicates. The server sends back an ACK, so the agent knows which messages made it and which ones it may need to send again.

If the agent restarts, we lose the buffered events, and if there is an extended server outage, the agent can overflow the bounded user space ring buffer. So we designed it to drop the oldest events when the buffer is full.

The agent continues draining kernel event channels into its own ring buffers even as it attempts to reconnect to the server. When we had to, we included delays that grew exponentially from 1 to 30 seconds, and also included jitter to prevent the thundering herd problem https://en.wikipedia.org/wiki/Thundering_herd_problem.

What we learned

We use Cloudflare tunnels for all our public endpoints, but we realized they don’t support gRPC.

So, how do we know whether the gRPC stream is down? This was a surprising issue to us because a batch send failure is not a reliable signal for us to either reestablish the connection or retry sending the stream.

The only way to know whether the server is connected is to have the agent listen on the receive to catch errors to identify the close from the server (https://github.com/grpc/grpc-go/issues/8190#issuecomment-2641471450). The coordination logic for this is hard, but was made much easier with go channels.

We also send periodic keepalive pings, so even an idle connection that quietly dies gets noticed within seconds rather than waiting for the next batch.

In the future, we want to provide offline agents with the capability to sync with the server and support a long time frame with durable, encrypted, and tamper resistant logs.