Understanding QEMU Devices (2018)

https://www.qemu.org/2018/02/09/understanding-qemu-devices/

by sipofwater • 2 years ago

180 58 2 years ago

sipofwater2• 2 years ago

"USB disk as /dev/sda on a not-rooted smartphone using Termux, QEMU, Alpine Linux": https://news.ycombinator.com/item?id=40507319

SunlitCat• 2 years ago

QEMU along with Bochs were my first tries getting into virtualization / emulation way back (maybe around early 2000s? I can't remember!).

Although the emulation / virtualization market already grew larger with more and more options available today, QEMU was (and still is) one of the most awesome projects out there.

rwmj• 2 years ago

A lot of the options (especially the free ones) are either using qemu or using ideas that were developed early (not actually first) for qemu like virtio. There are just a lot more layers on top these days, and not always for the better.

skrtskrt2• 2 years ago

Proxmox is great as a FOSS hypervisor, but their docs for doing pretty much anything advanced are just "here's a qemu command".

koala_man• 2 years ago

Back in the day I ran Bochs in all its 4 Bogomips glory on a university IP address and went to IRC channels where script kiddies were "trading root".

I let them go first, and watched through an instrumented terminal how they clumsily installed a rootkit, then inevitably refused to give anything in return and laughed calling me a noob.

Their laughter was short lived.

I had even spent quite a bit of effort kludging the kernel to report much higher specs than Bochs could deliver, but all that effort was wasted because no one knew how to check.

swozey• 2 years ago

QEMU is used in basically every single hardware vendor today and has been since I've been in virtualization/containerization tech (2010+).

I've only seen Vmware (gsx/esx) at Windows shops for things like big Exchange clusters, etc. Every CDN I've worked at used qemu.

i_am_a_peasant• 2 years ago

There's still a lot of Xen out there

checker6593• 2 years ago

What is the definitive new-comer friendly guide to QEMU? Not just about using it but also understanding its internals (say to add new instructions to a supported ISA etc)?

sipofwater2• 2 years ago

https://www.qemu.org/documentation/

checker6593• 2 years ago

I am looking for some hand-holding with this. The documentation seems more for reference (albeit a bit lack-luster if I may say so).

cpach6• 2 years ago

What are your goals? Are you interested in emulation (i.e. running a VM that uses another architecture than your physical computer) or in virtualization? (i.e. running a VM that uses the same architecture)

checker6593• 2 years ago

Well both. I'm interested in implementing an ISA extension (not sure if you know about CHERI). Also, there are reference implementations for aarch64 and risc-v that I'd like to understand.

hnthrowaway03282• 2 years ago

Just curious, what about simulation? I heard that simulation is more serious than emulation and targets for say pipeline level emulation, but maybe it's just a fancier word?

pm2154• 2 years ago

Unfortuately there is none. QEMU is a large project and doesn't have much formal design or API documentation. On the other hand it's not big enough (compared for instance to the Linux kernel) to have a wider community interested in trying to provide internals documentation for newcomers.

Our general advice is "look at the existing code for the bit you're interested in to see how it works". You can sometimes find descriptions of the overall architecture online in third party blog posts and the like, but if they're more than a few years old then be wary that they might be out of date -- they're likely to be right in general principles and wrong in details, because things change.

For adding new instructions to an existing ISA: the first couple of sections of https://www.qemu.org/docs/master/devel/index-tcg.html are relevant here. Depending on the target it might or might not use decodetree (decodetree is much easier to add a new insn to, but some older targets still do by-hand switch-statement based decoding.) Look at how an existing insn that is similar to what you want to do works.

Implementing CHERI in particular is going to be pretty awful, because the things it does (like 128-bit pointers) break various assumptions QEMU makes. The University of Cambridge forked QEMU to add CHERI support for MIPS and RISC-V and I think also AArch64: https://www.cl.cam.ac.uk/research/security/ctsrd/cheri/cheri... -- but the changes are pretty invasive and also not likely to be very fast. (The fork looks like it's based on 6.0, so three years old now.)

(If anybody is interested in trying to write up some documentation for QEMU's internals (either a general overview/roadmap or something on a particular subsystem), I'd be happy to code-review patches that add something to the "Developer Information" subsection of our manual.)

cpach6• 2 years ago

Out of curiosity, how does one reach you? (Saw no info/contact details in your profile.)

Also, where does QEMU people hang out online? AFAICT the IRC channel is not very active. (Based on few and random visits, so I could be wrong.)

isjamesalive• 2 years ago

QEMU is a large project and doesn't have much formal design or API documentation.

This is bonkers to me considering how it’s used in industry.

candiddevmike• 2 years ago

Start with libvirt, it provides a full GUI around QEMU operations. Run ps to see the underlying QEMU commands it runs. Inspect the XML files to understand how it builds machines.

cpach6• 2 years ago

A coworker came up with a similar idea: We started a VM using Lima, then ran ps to see what args it passed to QEMU. It was enlightening!

freedomben3• 2 years ago

I agree but I think you meant "Virt Manager"? afaik libvirt is an API, not even a CLI and definitely not a full GUI. For a CLI, virsh is your guy.

Neywiny• 2 years ago

I implemented a bit of a STM32 and it was a chore and a half. I've noticed 2 things with the code base: 1. It's C but they really want C++. Qemu wrote their own class system, foreach loops, containers, etc. And because of that, when I tried to use actual cpp, compilation failed due to how many reserved keywords were used in headers and other mess. 2. As noted in other comments, copy paste and modify. It got me where I need to go. But it was a slog. Eventually I had gdb debugging my qemu build, and gdb debugging the program I was running. I could even connect in from the STM32 IDE which was nice

skrtskrt2• 2 years ago

probably just start using Proxmox, as it's a pretty beginner-friendly FOSS hypervisor with extensive docs and forums, and it's largely a wrapper around qemu.

Their docs often include equivalent qemu commands for any UI actions.

For anything the UI can't do yet, they only give the QEMU command

zorked• 2 years ago

qemu is a treasure and reading its source to learn how computers work is very fun.

anonymousDan• 2 years ago

This is such a fantastic description of what is going on underneath the hood - it took me quite a while to understand how qemu works, wish I had seen this before!

_mlbt• 2 years ago

If you’re on a Mac, UTM is an excellent wrapper around Qemu.

https://mac.getutm.app/

adeptima6• 2 years ago

you can choose Apple virtualisation in UTM instead of QEMU too. Apple virtualization is optimized for M1+

cpach6• 2 years ago

QEMU has a Hypervisor[0] backend these days, called “hvf”.

https://wiki.qemu.org/Features/HVF

[0] “Hypervisor” is a “sibling” to the Virtualization framework. IMHO, the naming is incredibly confusing (:

zamadatix• 2 years ago

That's a bit like saying "instead of ext4 you can use an SSD" in that the things involved span multiple layers. When you select the option to use Apple virtualization framework in UTM you're still using QEMU, what you're changing is the backend QEMU is using for the CPU virtualization.

adeptima6• 2 years ago

Warning for humans! If you are trying to run a virtual PC on QEMY let's say on bare metal hosting from hetzner, you will very soon discover - QEMU is a dead slow without actual graphic card or at best you will get all kind of funky missing libs error messages on Ubuntu and other OS in very surprise spots.

Had a very good experience simulating K8s cluster with QEMU aka studing K8s hard way once I figure out how networking actually works between virtual machines and domains can be assigned with external proxy.

yjftsjthsd-h• 2 years ago

Why would qemu care about having a graphics card? Do you mean that whatever system you were running inside qemu expected a GPU and was slow without it?

wmf• 2 years ago

If the OP was running a virtual desktop on a server with no GPU then they probably had to fall back to software rendering which can be slow. This isn't QEMU's fault per se; if you physically don't have a GPU then you don't have a GPU.

TeeMassive• 2 years ago

Had a very good experience simulating K8s cluster with QEMU aka studing K8s hard way once I figure out how networking actually works between virtual machines and domains can be assigned with external proxy.

This is an awesome use of QEMU! I'm both interested in learning K8s and what goes on under the hood at the kernel level because I do cloud connected IoT stuff, so I'll definitely use that!

Is there any kind of "build the kernel from scratch" project for that kind of stuff?

adeptima6• 2 years ago

I was inspired by the article "Kubernetes The Hard Way" ... cant find the original article now

Had to study concepts like "bridged networking with libvirt" https://linuxconfig.org/how-to-use-bridged-networking-with-l...

No way I would recommend my way to study k8s ... it was just a pet project for a greater good.

Now in 2024 it's better to start from projects like minikube or k3s (my preference) on local computer

https://minikube.sigs.k8s.io/docs/ https://docs.k3s.io/

and later use terraform to provision infra

Forget it all and just use managed kubernets from cheaper providers like digital ocean

The result of inflicting pain on myself - I do value the work done by people who provide stable managed kubernets and upgrade it flawlessly.

navaati• 2 years ago

Hi, one probably really wants to use libvirt rather than qemu directly. That way you can create your VMs remotely with a GUI (virt-manager) using a ssh-based libvirt url, or a CLI (virsh) and it will handle all the right parameters for qemu, the required networking setup, etc. Check it out !

adeptima6• 2 years ago

yes, i connect with virt-manager and ssh-based libvirt url.

my main case was to scrape multiple messangers and apps with desktop ui only.

everything must run and render on server.

adeptima6• 2 years ago

as @wmf correctly stated "If the OP was running a virtual desktop on a server with no GPU then they probably had to fall back to software rendering which can be slow. This isn't QEMU's fault per se; if you physically don't have a GPU then you don't have a GPU."

and 99% cheap servers in the wild dont have a GPU or even hardware graphic card

apitman• 2 years ago

Understanding QEMU...

Best of luck

ashconnor2• 2 years ago

Followed by libvirt, VirtIO, KVM, qcow2. If only there was one book or course to pull it all together.

matheusmoreira• 2 years ago

QEMU actually has the ability to glue together a lot of different host formats (raw, qcow2, qed, vhdx, …)
and protocols (file system, block device, NBD, Ceph, gluster, …)

Yeah, and it's awesome. With qcow2 images mounted via nbd, I was able to manipulate ddrescue images without modifying the original. Truly one of the most useful software ever written. I even use it on Android with Termux to use and test x86_64 software.

I wish there was a step by step porting and implementation guide. I tried to port the open source Sensor Watch board to QEMU in order to facillitate software development for it. Didn't get very far. I'm not particularly knowledgeable about electronics but I had all the hardware documentation, I feel like it should have been enough.

israrkhan• 2 years ago

QEMU is a great piece of software. I use it regularly to debug various cross compiled kernel images under gdb without requiring actual ARM hardware (buildroot + custom kernel + gdb).

mgerdts2• 2 years ago

so the QEMU developers have added the virtio-net card (a PCI hardware specification, although no bare-metal hardware exists yet that actually implements it)

Things have changed since 2018. BlueField DPUs provide virtio network [1] and block [2] devices in the host PCI space. These can be passed through to VMs using vfio-pci in the host.

1. https://docs.nvidia.com/networking/display/bluefieldvirtione...

2. https://docs.nvidia.com/networking/display/bluefield2snap380

Disclaimer: I work for NVIDIA, have used BlueField DPUs, but have not used the virtio feature.

freedomben3• 2 years ago

Don't feel obligated to answer if you shouldn't, but how seriously is Nvidia taking Linux nowadays? With the rise of ML is linux being seen as an important support target?

wtallis• 2 years ago

NVIDIA's business these days is dominated by datacenter products that are almost exclusively used with Linux. This has had almost no effect on their approach toward dealing with/integrating with Linux. They did follow in AMD's footsteps and adopted a driver model with an open-source kernel component and closed-source userspace components.

moondev• 2 years ago

It can act as a PCIE root as well is that correct? I assume by connecting it to the input of a PCIE switch with other disks attached downstream, but would love to learn more and what PCIE switches are known to work.

mgerdts2• 2 years ago

The models of BlueField DPUs I’ve worked with have no storage (aside from eMMC to boot the DPU) directly attached. They typically run SPDK, then SPDK block devices are presented to the host as NVMe namespaces or virtio-blk devices. If using NVMe-oF with RDMA, this can provide a zero copy IO path to remote storage.

For NVMe, the devices may be presented to the host as PFs or VFs. I assume but do not know that it would be the same for virtio devices.

Understanding QEMU Devices (2018)

Related Stories