Booting Android OS on a Hypervisor: a paravirtualised bring-up on Xen

Android 15 as a Xen paravirtualised guest with no device tree and no bootloader to write — the kernel fragment, the raw-image trap, and the boot that stopped one HAL short.

The ask was simple to say and not so simple to do: run Android as a guest VM on the Xen hypervisor, in paravirtualised (PV) mode, with a Yocto-based Dom0 underneath. No device tree to author. No bootloader. No VirtIO. Just Xen's own PV interfaces, and whatever we could convince Android to accept.

This is the walkthrough of how that bring-up actually went — the prerequisites, the kernel work, the Android product work, and an honest account of where it stopped.

Having prior experience with the Android boot process is the only reason this did not turn into a long archaeology project. When the console went quiet, I already knew which stage had gone quiet, and that narrowed the search from everything to one thing.

The prerequisites: Yocto and a Dom0 terminal

Before Android enters the picture at all, you need a working Xen host. In our case, that host was built with Yocto.

I did this bring-up on two very different machines, and I would recommend the same order to anyone starting out:

• Raspberry Pi 4 (4GB/8GB) — the cheap, forgiving board where you break things freely.
• Renesas R-Car V4H — the actual automotive-grade target, where things get serious.

Starting on the Pi was a deliberate choice. When your Android guest refuses to boot, you want to be very sure the hypervisor is not the problem. On the Pi, a full rebuild-and-reflash cycle is a few minutes and an SD card swap. On the V4H, it is a much longer story. So I made every mistake on the Pi first, and carried only the working recipe to the V4H.

The Yocto side needs the meta-virtualization layer, which brings in Xen itself along with the xl toolstack. Two lines in local.conf matter:

DISTRO_FEATURES:append = " xen virtualization"IMAGE_INSTALL:append   = " xen-tools kernel-modules e2fsprogs util-linux"

Then a plain bitbake core-image-minimal. What comes out is a Dom0 Linux running on top of the Xen hypervisor. Flash it, boot it, and the first thing to verify is that Xen is genuinely underneath you:

root@raspberrypi4-64:~# xl infohost                   : raspberrypi4-64release                : 6.6.22-yocto-standardmachine                : aarch64nr_cpus                : 4total_memory           : 7860free_memory            : 3512xen_version            : 4.18.0xen_caps               : xen-3.0-aarch64 xen-3.0-armv7l root@raspberrypi4-64:~# xl listName                          ID   Mem VCPUs      State   Time(s)Domain-0                       0  2048     4     r-----      31.4

That Domain-0 line is the checkpoint. If you see it, you have a Dom0 terminal on a real hypervisor, and everything that follows can proceed. If xl info fails, stop right there — no amount of Android work will fix a host that is not virtualising.

One practical note on the console. Almost everything from here on happens over a serial console into Dom0. Get your UART settings and a reliable way to copy image files onto the board sorted out before you start. You will be moving multi-gigabyte images repeatedly, and a slow transfer path will quietly eat your week.

Why paravirtualisation changes everything

The guest runs paravirtualised, and that choice decides the shape of the entire bring-up. PV does not emulate hardware. The guest knows it is a guest. Instead of pretending to talk to a SATA controller, it talks to Xen's block frontend driver (xen-blkfront), which talks over shared memory rings and event channels to a backend in Dom0. Same for networking (xen-netfront), same for the console (hvc0).

We chose PV for two honest reasons:

• Performance and determinism. In an SDV-style workload, you do not want emulation layers sitting between the guest and the disk.
• No device tree of our own to write. This is the one that changes the Android side completely — and it is worth being precise about, because I was loose about it for longer than I should have been. It is not that the guest has no device tree. Xen's toolstack synthesises one and hands it over in memory; that is where Machine model: XENVM-4.18 in the boot log comes from. What disappears is the part you would normally own: there is no board .dts to author for the Android guest, nothing to compile, nothing to append to a boot image. The rest — which block devices exist and where their backends live — the guest learns at runtime from the XenStore and PV bus enumeration.

For someone coming from physical Android boards, this is a genuinely strange feeling. On real hardware, the device tree is where you spend a large chunk of your bring-up effort. Here you never open one, and the kernel discovers its world at runtime instead.

The trade-off is that the guest kernel must have Xen guest support compiled in. A stock GKI kernel will boot on real hardware and go absolutely nowhere under Xen PV, because it has no idea how to find its disks.

Here is the shape of the whole system, worth keeping in mind for the rest:

The kernel: fragment, build config, Bazel target

The Android guest needs a kernel that is still a proper Android GKI kernel, so userspace stays happy, but which also knows about Xen. In the Android kernel tree that means three small files under a new common-modules/xen-virtual-device/ directory: a defconfig fragment, a build config, and a Kleaf BUILD.bazel target.

The fragment is the heart of it, and it is almost entirely one line:

CONFIG_XEN=y    # Enable Xen guest support (to run in a Xen VM)

Turning this on pulls in ARM64 Xen guest support, and with it the frontend drivers — xen-blkfront for block devices and the hvc0 PV console. Without it, the Android kernel boots under Xen and then panics with no root device, because /dev/block/xvda1 does not exist and never will.

Everything else in that fragment is deliberately switched off and written down as an explicit comment: ballooning, xenfs, gntdev, privcmd, the hypervisor sysfs interface, XEN_BACKEND. Those are all things a Dom0 or a management guest needs. Our Android guest is a plain, unprivileged DomU — it consumes PV devices, it never serves them. I wrote the disabled symbols out longhand rather than just omitting them, because when I later opened the generated .config while debugging, I wanted to confirm at a glance that a symbol was off by decision and not off by accident.

The build config merges that fragment on top of the standard GKI defconfig before building, and sets one thing that is not optional here:

PRE_DEFCONFIG_CMDS="... merge_config.sh -m -r \  ${ROOT_DIR}/${KERNEL_DIR}/arch/arm64/configs/gki_defconfig \  ${ROOT_DIR}/common-modules/xen-virtual-device/sdv.aarch64.fragment" BUILD_INITRAMFS=1    # no bootloader will assemble a boot.img for us

There is nothing in this setup to build and parse a boot.img, so we need a standalone ramdisk.img that xl can hand to the guest directly alongside Image.

The same file also points at a modules.blocklist whose entire contents are blocklist vkms.ko. vkms is the virtual KMS driver; under Xen PV with no display backend configured, loading it produced noisy failures during early boot that made the real problems harder to see. Small fix, disproportionate relief.

The Bazel target describes the build to Kleaf. Two arguments in kernel_images() capture the entire architecture:

build_initramfs = True,     # we need ramdisk.imgbuild_boot      = False,    # no boot.img — Xen loads Image directly

We do not want a boot image, because nothing will ever parse one. xl is our bootloader. Build and collect:

$ tools/bazel run //common-modules/xen-virtual-device:xen_virtual_device_aarch64_dist \      -- --dist_dir=out/xen_virtual_device_aarch64/distINFO: Elapsed time: 942.331s, Critical Path: 611.28sINFO: Build completed successfully $ ls out/xen_virtual_device_aarch64/dist/Image  ramdisk.img  System.map  vmlinux

Image and ramdisk.img are the two artifacts we care about.

Before bringing Android into it at all, I wanted to answer one narrow question: does this kernel know it is running under Xen? So I copied just those two files to Dom0 — no Android images, no disk = entries in the config — and booted it.

[    0.000000] Booting Linux on physical CPU 0x0000000000[    0.000000] Xen 4.18 support found[    0.000000] Machine model: XENVM-4.18[    0.612334] xen:grant_table: Grant tables using version 1 layout[    0.688415] xen:events: Using FIFO-based ABI...[    1.204338] VFS: Cannot open root device "(null)" or unknown-block(0,0)[    1.204902] Kernel panic - not syncing: VFS: Unable to mount root fs

Xen 4.18 support found, XENVM-4.18 as the machine model, and the grant table and event channel lines all coming up meant CONFIG_XEN=y had done its job — this was a Xen-aware guest kernel that had successfully talked to the hypervisor. It then panicked for exactly the right reason: there were no disks attached, so there was no root filesystem to find.

That is a small test and it took two minutes, but it split the problem cleanly in half. From here, if the kernel came up and storage still did not work, the fault was in the guest config or the images — not in the kernel build. On a bring-up with this many unknowns, being able to permanently retire one of them is worth the detour.

The Xen guest configuration

This file is the closest thing we have to a bootloader configuration, so it is worth reading closely.

name    = "android"memory  = 4096vcpus   = 2arch    = "aarch64" kernel  = "/home/root/android/Image"ramdisk = "/home/root/android/ramdisk.img" disk = [    'format=raw, vdev=xvda1, access=rw, target=/home/root/android/system.img',    'format=raw, vdev=xvda2, access=rw, target=/home/root/android/vendor.img',    ...] extra = "androidboot.hardware=sdv androidboot.selinux=permissive \androidboot.boot_devices=xvda1,xvda2,xvda3,xvda4 \init=/init rootwait console=hvc0 rw printk.devkmsg=on ..."

The extra line does the job a bootloader's bootargs would normally do, and each part earns its place:

• androidboot.hardware=sdv — this single property is what makes Android look for fstab.sdv, init.sdv.rc, and our ro.hardware.* properties. It is the string that ties the guest config to the Android product.
• console=hvc0 — the Xen PV console. Not ttyAMA0, not ttyS0. On a PV guest there is no UART to speak of. Getting this wrong gives you a VM that boots perfectly and prints absolutely nothing, which is a special kind of frustrating.
• androidboot.boot_devices=xvda1,xvda2,xvda3,xvda4 — tells first-stage init which block devices to wait for and scan.
• init=/init rootwait — go straight to Android's init, and wait for the block devices to actually appear before trying to mount. rootwait saved me from an early race where init ran before blkfront had finished probing.

The extra line also carries three androidboot.vendor.apex.* properties pinning which APEX implementation to use for KeyMint, Gatekeeper and the graphics composer — normally generated by Cuttlefish's own launcher, which never runs in our setup.

Two more small things. on_crash = 'preserve' is worth setting while debugging — it keeps the crashed domain around so you can inspect it, instead of it vanishing before you can read the log. And the four disks are plain raw image files on Dom0's filesystem. No partition table, no GPT, no bootloader — Xen maps each file to a vdev and the guest sees a block device. That is the whole storage story.

The Android side: a new product on Cuttlefish + Trout

Now to userspace. Rather than starting a product from scratch, I created a new product called sdv under device/horizon/sdv, inheriting from two existing Google products:

$(call inherit-product, device/google/cuttlefish/shared/virgl/device_vendor.mk)$(call inherit-product, device/google/trout/aosp_trout_arm64.mk)

Cuttlefish is Google's virtual Android device — it already assumes it is running in a VM, which is exactly our situation. Trout is the automotive (AAOS) reference built on top of Cuttlefish, which gave us the car-specific pieces.

"Why not build from scratch" is a fair question. Cuttlefish's userspace already expects virtual block devices, no real GPU and no real modem. Roughly seventy percent of what we needed came free with that inheritance. Building from bare AOSP would have meant re-solving problems Google had already solved — just with Xen PV in place of crosvm and VirtIO.

The product identity is ordinary. The board config inherits Trout's, sets vsoc_arm64 as the platform, and fixes the image types. One line in there is small and absolutely non-negotiable:

TARGET_USERIMAGES_SPARSE_EXT_DISABLED := true

By default Android produces sparse ext4 images, which is great for fastboot and useless for Xen. Xen's format=raw expects a plain raw filesystem image. With sparse images the guest boots, blkfront finds the devices, and then first-stage init fails to mount /system, because what sits on that block device is not a filesystem it recognises. That cost me a debugging session before the penny dropped.

With no device tree, the fstab is a plain text file, and it is beautifully short:

/dev/block/xvda1 /system ext4 noatime,ro,errors=panic wait,first_stage_mount/dev/block/xvda2 /vendor ext4 noatime,ro,errors=panic wait,first_stage_mount/dev/block/xvda3 /odm    ext4 noatime,ro,errors=panic wait,first_stage_mount/dev/block/xvda4 /data   ext4 rw,nosuid,nodev,noatime,errors=panic wait,check,formattable

Four lines. No logical, no slotselect, no A/B, no dm-verity, no super partition. On production Android devices these entries carry avb_keys, logical and slotselect flags and take real thought. Here the disks are files on the host mapped one-to-one to xvda1..4, and there is no bootloader doing slot selection, because there is no bootloader at all.

Two flags matter. first_stage_mount tells init to mount these partitions in the very first stage, from the ramdisk, before switching root — its absence produces a boot that stops dead right after init starts. errors=panic was a debugging choice: I wanted a loud panic on filesystem corruption rather than a silently degraded mount that would confuse me two hours later.

One practical detail: the same fstab has to be copied to four locations via PRODUCT_COPY_FILES — the ramdisk root, first_stage_ramdisk/, vendor/etc/, and the recovery ramdisk. First-stage and second-stage init each resolve fstab.${ro.hardware} from their own location. Copy it everywhere and stop thinking about it.

Beyond the fstab, three things needed handling and none of them are worth a long section. The inherited HALs expect crosvm-style VirtIO devices we did not have, so the product deliberately falls back to software implementations for Gatekeeper and KeyMint, a stub audio HAL, and ENABLE_EVS_SERVICE := false to keep Trout's automotive camera service from failing against hardware that does not exist. Cuttlefish's shared/device.mk also ships both the nonsecure and cf_remote variants of the KeyMint and Gatekeeper APEXes and picks between them at runtime using bootconfig generated by assemble_cvd — which never runs when xl is your launcher, so both landed on the image and the build hit a duplicate-APEX conflict. A small local patch dropping the nonsecure variants and hard-selecting the remaining implementation cleared it. Finally, a two-line init.sdv.rc stops the serial logging service once boot completes, so the console stops flooding when you want to use it interactively — though as the next section explains, that trigger never actually fired for us.

Building, deploying, and the boot that did not finish

The build is ordinary AOSP from here:

$ lunch sdv-trunk_staging-userdebug$ m

Copy the four raw images plus the kernel and ramdisk to Dom0, and launch:

root@raspberrypi4-64:~# xl create -c /home/root/android/androidguest.cfgParsing config from /home/root/android/androidguest.cfg[    0.000000] Xen 4.18 support found[    0.744118] blkfront: xvda1: barrier or flush: disabled; persistent grants: disabled[    0.751003] blkfront: xvda2: barrier or flush: disabled; persistent grants: disabled[    0.757441] blkfront: xvda3: barrier or flush: disabled; persistent grants: disabled[    0.763890] blkfront: xvda4: barrier or flush: disabled; persistent grants: disabled[    0.912004] Run /init as init process[    0.945513] init: init first stage started![    1.104553] init: [libfs_mgr] __mount(source=/dev/block/xvda1,target=/system,type=ext4)=0[    1.187330] init: [libfs_mgr] __mount(source=/dev/block/xvda2,target=/vendor,type=ext4)=0[    1.401227] init: init second stage started![    1.556902] init: Setting property 'ro.hardware' to 'sdv'

This is the first time the full storage path is visible: four blkfront lines, so the frontend in the guest found the backend in Dom0 and the four raw images showed up as xvda1 through xvda4. Then __mount(...)=0, a switched root, and second-stage init. That chain is exactly what you want to see — it meant the kernel, the guest config, the fstab and the raw image format were all correct, together.

And then this, forever:

[    5.220118] init: Service 'surfaceflinger' (pid 412) exited with status 1[    5.220994] init: Sending signal 9 to service 'surfaceflinger' (pid 412) process group...[    6.221443] init: starting service 'surfaceflinger'...[    7.408820] init: Service 'surfaceflinger' (pid 431) exited with status 1[    7.409662] init: Sending signal 9 to service 'surfaceflinger' (pid 431) process group...[    8.410221] init: starting service 'surfaceflinger'...[    9.601337] init: Service 'surfaceflinger' (pid 448) exited with status 1...

I never saw sys.boot_completed=1. SurfaceFlinger needs a graphics composer HAL to talk to, and in a Xen PV guest there is no display backend at all — no VirtIO GPU, no DRM device, nothing for it to bind to. So it started, failed, got killed, and init restarted it, over and over, indefinitely. Because sys.boot_completed never got set, every service gated behind it stayed down — including the boot animation, the system UI, and my own init.sdv.rc rule that was supposed to stop serial logging at that point. That rule sat there unused, which was a small irony I only noticed later.

This is worth being straight about, because it is easy to write a bring-up article that ends on a clean success line. The honest position is that userspace mounted and ran; the graphics stack did not exist yet. Those are two different milestones, and only the first one was mine to claim at that stage.

So the confirmation had to come from elsewhere: the console and adb. The guest was clearly alive underneath the restart loop, and both interfaces proved it. adb connecting at all means adbd is running, which means second-stage init got far enough to start it. All partitions were mounted read-write-correct with sensible usage figures. ro.hardware resolved to sdv, so the whole androidboot.hardware → fstab → property chain worked end to end. servicemanager, vold and zygote were alive. Android userspace was genuinely up on a Xen PV guest, doing everything except draw.

root@raspberrypi4-64:~# xl listName                          ID   Mem VCPUs      State   Time(s)Domain-0                       0  2048     4     r-----     412.7android                        1  4096     2     -b----     186.3

Yocto Linux and Android 15, side by side on one piece of silicon, with Xen holding the line between them.

What I took away from this

1. 01

   Pick the right success criterion for the stage you are at.

   I went in expecting sys.boot_completed=1 to be the finish line, because on a physical board with a working display stack that is exactly what it is. On a PV guest with no graphics backend, that property is gated behind a HAL that does not exist yet, so waiting for it would have meant waiting forever while a perfectly healthy userspace ran underneath. Learning to read adb shell df, getprop and ps as the proof — instead of one convenient log line — was the single most useful shift in the whole project.

2. 02

   A restart loop is information, not just noise.

   SurfaceFlinger cycling endlessly told me precisely where the gap was. Init was healthy enough to supervise services, the mounts were fine, the property system was fine — one HAL was missing. A boot that fails in a specific, repeatable way is much better news than a boot that hangs silently.

3. 03

   Removing things can be as hard as adding them.

   A large part of this work was deleting: no device tree, no bootloader, no VirtIO, no EVS, no real HALs. Every removal exposed an assumption buried somewhere in the inherited product, and each one had to be found and handled.

4. 04

   Start on the cheap board.

   Doing the Raspberry Pi 4 first and the R-Car V4H second was the best process decision I made. By the time I moved to the V4H, the Android side was a known-good quantity, so every new failure could be attributed to the board — which is exactly the clarity you want on expensive hardware.

5. 05

   The smallest files did the heaviest lifting.

   A one-line modules.blocklist. One TARGET_USERIMAGES_SPARSE_EXT_DISABLED := true. A four-line fstab. None of these look like much in a diff, and each one stood between me and a running system.

The road ahead from here is clear enough: a real graphics path through a PV display frontend so SurfaceFlinger has something to bind to, a proper VHAL, SELinux moved from permissive to enforcing, and verified boot.

But getting Android's userspace up on a PV guest with no device tree of our own and no bootloader is the part that proves the architecture is sound — and that part worked.