Post Syndicated from original https://mjg59.dreamwidth.org/63261.html

First up: what I’m covering here is probably not relevant for most people. That’s ok! Different situations have different threat models, and if what I’m talking about here doesn’t feel like you have to worry about it, that’s great! Your life is easier as a result. But I have worked in situations where we had to care about some of the scenarios I’m going to describe here, and the technologies I’m going to talk about here solve a bunch of these problems.

So. You run a typical VM in the cloud. Who has access to that VM? Well, firstly, anyone who has the ability to log into the host machine with administrative capabilities. With enough effort, perhaps also anyone who has physical access to the host machine. But the hypervisor also has the ability to inspect what’s running inside a VM, so anyone with the ability to install a backdoor into the hypervisor could theoretically target you. And who’s to say the cloud platform launched the correct image in the first place? The control plane could have introduced a backdoor into your image and run that instead. Or the javascript running in the web UI that you used to configure the instance could have selected a different image without telling you. Anyone with the ability to get a (cleverly obfuscated) backdoor introduced into quite a lot of code could achieve that. Obviously you’d hope that everyone working for a cloud provider is honest, and you’d also hope that their security policies are good and that all code is well reviewed before being committed. But when you have several thousand people working on various components of a cloud platform, there’s always the potential for something to slip up.

Let’s imagine a large enterprise with a whole bunch of laptops used by developers. If someone has the ability to push a new package to one of those laptops, they’re in a good position to obtain credentials belonging to the user of that laptop. That means anyone with that ability effectively has the ability to obtain arbitrary other privileges – they just need to target someone with the privilege they want. You can largely mitigate this by ensuring that the group of people able to do this is as small as possible, and put technical barriers in place to prevent them from pushing new packages unilaterally.

Now imagine this in the cloud scenario. Anyone able to interfere with the control plane (either directly or by getting code accepted that alters its behaviour) is in a position to obtain credentials belonging to anyone running in that cloud. That’s probably a much larger set of people than have the ability to push stuff to laptops, but they have much the same level of power. You’ll obviously have a whole bunch of processes and policies and oversights to make it difficult for a compromised user to do such a thing, but if you’re a high enough profile target it’s a plausible scenario.

How can we avoid this? The easiest way is to take the people able to interfere with the control plane out of the loop. The hypervisor knows what it booted, and if there’s a mechanism for the VM to pass that information to a user in a trusted way, you’ll be able to detect the control plane handing over the wrong image. This can be achieved using trusted boot. The hypervisor-provided firmware performs a “measurement” (basically a cryptographic hash of some data) of what it’s booting, storing that information in a virtualised TPM. This TPM can later provide a signed copy of the measurements on demand. A remote system can look at these measurements and determine whether the system is trustworthy – if a modified image had been provided, the measurements would be different. As long as the hypervisor is trustworthy, it doesn’t matter whether or not the control plane is – you can detect whether you were given the correct OS image, and you can build your trust on top of that.

(Of course, this depends on you being able to verify the key used to sign those measurements. On real hardware the TPM has a certificate that chains back to the manufacturer and uniquely identifies the TPM. On cloud platforms you typically have to retrieve the public key via the metadata channel, which means you’re trusting the control plane to give you information about the hypervisor in order to verify what the control plane gave to the hypervisor. This is suboptimal, even though realistically the number of moving parts in that part of the control plane is much smaller than the number involved in provisioning the instance in the first place, so an attacker managing to compromise both is less realistic. Still, AWS doesn’t even give you that, which does make it all rather more complicated)

Ok, so we can (largely) decouple our trust in the VM from having to trust the control plane. But we’re still relying on the hypervisor to provide those attestations. What if the hypervisor isn’t trustworthy? This sounds somewhat ridiculous (if you can’t run a trusted app on top of an untrusted OS, how can you run a trusted OS on top of an untrusted hypervisor?), but AMD actually have a solution for that. SEV (“Secure Encrypted Virtualisation”) is a technology where (handwavily) an encryption key is generated when a new VM is created, and the memory belonging to that VM is encrypted with that key. The hypervisor has no access to that encryption key, and any access to memory initiated by the hypervisor will only see the encrypted content. This means that nobody with the ability to tamper with the hypervisor can see what’s going on inside the OS (and also means that nobody with physical access can either, so that’s another threat dealt with).

But how do we know that the hypervisor set this up, and how do we know that the correct image was booted? SEV has support for a “Launch attestation”, a CPU generated signed statement that it booted the current VM with SEV enabled. But it goes further than that! The attestation includes a measurement of what was booted, which means we don’t need to trust the hypervisor at all – the CPU itself will tell us what image we were given. Perfect.

Except, well. There’s a few problems. AWS just doesn’t have any VMs that implement SEV yet (there are bare metal instances that do, but obviously you’re building your own infrastructure to make that work). Google only seem to provide the launch measurement via the logging service – and they only include the parsed out data, not the original measurement. So, we still have to trust (a subset of) the control plane. Azure provides it via a separate attestation service, but again it doesn’t seem to provide the raw attestation and so you’re still trusting the attestation service. For the newest generation of SEV, SEV-SNP, this is less of a big deal because the guest can provide its own attestation. But Google doesn’t offer SEV-SNP hardware yet, and the driver you need for this only shipped in Linux 5.19 and Azure’s SEV Ubuntu images only offer up to 5.15 at the moment, so making use of that means you’re putting your own image together at the moment.

And there’s one other kind of major problem. A normal VM image provides a bootloader and a kernel and a filesystem. That bootloader needs to run on something. That “something” is typically hypervisor-provided “firmware” – for instance, OVMF. This probably has some level of cloud vendor patching, and they probably don’t ship the source for it. You’re just having to trust that the firmware is trustworthy, and we’re talking about trying to avoid placing trust in the cloud provider. Azure has a private beta allowing users to upload images that include their own firmware, meaning that all the code you trust (outside the CPU itself) can be provided by the user, and once that’s GA it ought to be possible to boot Azure VMs without having to trust any Microsoft-provided code.

Well, mostly. As AMD admit, SEV isn’t guaranteed to be resistant to certain microarchitectural attacks. This is still much more restrictive than the status quo where the hypervisor could just read arbitrary content out of the VM whenever it wanted to, but it’s still not ideal. Which, to be fair, is where we are with CPUs in general.

(Thanks to Leonard Cohnen who gave me a bunch of excellent pointers on this stuff while I was digging through it yesterday)

comments