Our GPU security is really fast, but we don’t expect you to take our word at face value, so we wrote this blog post to explain why.
To create this solution, we worked backward from how ML workloads actually work. In a nutshell, an ML process’s lifecycle looks something like this:
-
The process starts and loads its model weights (from the filesystem or from an external api)
-
The process then accesses a GPU device and loads data into the GPU (model weights, CUDA kernels, etc)
-
Then the inferencing happens
Now, this is a gross simplification, but from a security standpoint, it is a generalization we can take. Steps 1 and 3 are quite easy to secure since we can use standard file restrictions, and, on average, it isn’t hard to optimize these operations for speed.
What is hard to secure while optimizing for speed is the GPU interactions. To understand how we secure step 2, we need to break down the actual interaction between a process and GPUs.
GPUs are external devices exposed to user space processes (like an ML process) as character device files, mostly in the /dev directory, created by the kernel driver (DRM, NVIDIA, AMD, Intel, etc.). So for the ML process to access the GPU, it would have to open one or more of these character devices (/dev/nvidia0, /dev/nvidiactl, etc.).
The important detail in this operation is that the file open occurs only once, at the start, when the process connects to the GPU. After this file is opened, it receives a file descriptor (FD), and all operations between the process and the GPU use that FD.
We secure these GPU interactions at Bomfather by restricting access to these character device files. If, according to our policy, a process is allowed to access the GPU, we also allow it to access the character device file.
What this means for speed is that we only restrict and hook into the initial interaction. So, after this single file open, Bomfather is never triggered, and all communication happens with no overhead. eBPF has barely any overhead, but even a small overhead can multiply drastically if it is a hot path in GPU interactions. But with this solution, our overhead does not scale with the size of the workload. So, no matter the complexity or size of a workload, the overhead is extremely minimal at the boot of the process.
