The end of (free tier) ESXi
Merger & Acquisitions - winners and losers. And the winner is…
Free tier? - Isn't that for marketing only? Growth incentives? Like IT trainings? Not according to Broadcom in 2024.
Smells like enshittification?
The term enshittification was coined by the writer Cory Doctorow in November 2022; the American Dialect Society selected it as its 2023 Word of the Year. Doctorow has also used the term platform decay to describe the same concept.
Turns out M&As can create smelly effects. Time for a cleanup. Let’s get rid of ESXi.
Is Linux KVM as good as ESXi?
What’s the cash price for being the best virtualization tech?
KVM and ESXi have in common, that they can be used with management suites to archive horizontal scalability.
With KVM you can use oVirt and others.
ESXi should be used with VSphere.
Both work just fine. There is no cash price for being better.
— But VMware products are more expensive. So why is there a market for these? What are the key differences, and how do they matter?
Use cases and versatility
tl;dr: It seems to be that Linux KVM is on par with VMware’s server virtualization suite. Which is impressive, given the differences in available resources.
Back in the days, servers were built for one purpose. Today they are built for many.
Linux KVM started in 2006, so roughly 18 years ago. ESXi started in 2001, 23 years ago. To be fair: KVM wasn’t a viable alternative until 5–8 years ago. It didn’t release with a comparable spectrum of features. Both technology stacks emerged as general purpose computation systems. — Nothing special about it - by intention.
The key difference is convergence
Some people say it’s the type of hypervisor. But that’s an academic perspective. Type 1, Hybrid Type 1, Type 2. Pure academic discussions. Let’s look at it from a practical perspective.
Setting up a fleet KVM QEMU systems, incl. a convergent management approach, requires in-depth expertise. You need to test the setup thoroughly, and benchmark the related workloads. Doing that is a Linux System Engineering task, and it’s not en vogue, so to say. It’s fashionable to outsource these platform concerns to someone else because there are so many details to consider. Because the server virtualization stack isn’t going to serve one purpose, but many.
Convergence is a big problem with KVM QEMU. You can try to buy it, and you’ll get tested versions from Red Hat. If you believe them to do the right testing for you, that is. Or you can pay a little extra, and get a convergent system stack from VMware, where no testing is needed. Easy decision, in most cases. Because the more systems you have, the more important convergence becomes. And being able to distribute workload consistently.
Enterprise IT readiness
Besides convergence, KVM QEMU (with Libvirt) has a problem with Windows 11 (Q1 2023). And the future of Windows Server operating systems therefore, which traditionally host business workflows, Finance IT, and Corporate IT, like Microsoft Active Directory Services.
No full snapshots for Windows 11 guests (thin clients). This affects every Patch Tuesday.
The KVM hypervisor kernel extension has issues with certain Intel CPU instructions (so-called Model Specific Registers), related to many modern architectures. This affects performance. And leads to Blue Screens. It means there is not much testing going on.
Security-wise, KVM has many blowbacks, especially related to the global setting for nested virtualization that affects all guests. Hyper-V is a must-have for modern Windows environments. But Linux guests don’t require nested virtualization.
I haven’t seen AppArmor or SELinux rules for QEMU, meaning: security isn’t a topic for KVM integration; on Debian. Hardening here will be difficult, but VM guest-escapes are a possibility. Especially with Virtio, where enterprise systems like Windows aren’t the top concern.
oVirt / VSphere
oVirt inherits all the shortcomings. It’s used to scale things, and to orchestrate the fleet of hosts.
Distributing Linux systems is relatively easy, unless they have storage links. You’ll be dealing with vLANs, big storage backends etc. vLANs work fine with Open vSwitch, Linux bridge-interfaces or some network cards. But you need to test this with each other. There is no one size fits all here, because of the diversity of hardware.
Debugging network links with the command-line can take a lot of time
Performance: ESXi versus KVM QEMU
In my tests, ESXi is about 5-10 % faster. Possibly because it uses kernel extensions, and processes data more efficiently. Or because VMware invested money into optimization Libvirt / QEMU do not have.
OpenSSL speed test on KVM QEMU and ESXi - the systems are compareable, but not the same
Summary
KVM QEMU is on par with VMware ESXi if you are willing to put the system and security engineering into it. Security-wise, however, it’s not just 10% behind.
As a security professional, you should be aware that hypervisor escapes from QEMU are relatively common, and that Linux development focuses on security issues like it focuses on any other bug.
Yes, we can replace ESXi. If we are willing to put effort into it.
Because Security lab test
Linux KVM: setup considerations in relation to ESXi
Many tech details, many solutions, benchmarks, and more.
