Please support the display device model "ramfb". (2023)

Description Laszlo Ersek 21.02.2019 16:14:02 UTC

Libvirt already supports the "virtio-gpu-pci" device for AARCH64 guests (like the "virtio" model), but that's only good enough for Linux guests, not Windows guests. Windows requires a directly accessible framebuffer and virtio-gpu-pci does not provide one. QEMU implements (and supports libvirt) several VGA-like display devices that provide framebuffers. Like standard VGA, virtio-vga (that's virtio-gpu-pci plus VGA), and QXL. The problem with all this is this (AARCH64 only): When using VGA, the framebuffer used by the guest is in a (virtual) PCI MMIO bar. Likewise, the guest maps the framebuffer as non-cacheable. However, on the host side, QEMU (and the host kernel) treats the virtual framebuffer as normal memory. Therefore, the framebuffer is mapped as a cached writeback on the host side. This means that the guest and host have different mappings (different cache ability attributes) to the same memory space. The mapping takes effect (and overrides the guest-side mapping), so both guest and host ultimately access the framebuffer (which resides in normal DRAM) as write-back cacheable. This means that both guest and host first traverse the PCPU data cache(s) and are consistent with each other. This is not the case on aarch64 hosts. Since the guest uses UCmapping, when writing to the framebuffer, it bypasses the PCPU caches and goes straight to DRAM. However, on the host side, QEMU reads the framebuffer through a WB mapping and therefore bypasses the PCPU caches. The display is not coherent and you get display errors (stale data displayed by QEMU from PCPU cache). Behavior similar to x86-64. Until then, however, VGA's "framebuffer-PCI-MMIO-BAR" approach remains useless on aarch64/KVM. For this reason we have removed the QemuVideoDxe driver in edk2/ArmVirtQemufirmware. In QEMU/TCG (i.e. software emulation) this problem is not observable since in this case there is only one mapping (the host-side one). The problem also disappears with virtio-gpu-pci because (a) there is no significant framebuffer, (b) all display data exchanged with virtiotransfers is mapped as normal DRAM (WB) by both guest and host Unfortunately, there is Windows absolutely relies on a framebuffer to boot (although a random access framebuffer is optional according to the UEFI specification). Therefore we cannot use virtio-gpu-pci. (Again, don't confuse virtio-gpu-pci with virtio-vga. The latter is virtio-gpu-pci PLUS traditional VGA. Not good for aarch64.) The problem also goes away with the "ramfb" device model that Gerd wrote for the QEMU. It's a (comparatively) simple platform device that uses DRAM (WB) as a framebuffer, so the mappings on both sides are the same in terms of cacheability. The UEFI driver (QemuRamfbDxe, also by Gerd) exposes the framebuffer for direct access to the Windows bootloader, thus fulfilling the framebuffer requirement while avoiding the incoherency problem on ARM64. Ramfb is not a PCI device, but it doesn't have to be. Windows booted with UEFI is expected to only work with the legacy UEFI framebuffer. Of course, graphics mode changes and such shouldn't work, but the basic display is expected to work. Please introduce the display device model "ramfb" in the XML domain. Thanks!

Commentary 1 Christopher of Dinechin 2019-02-26 16:54:00 UTC

(In response to Laszlo Ersek fromComment #0)> When using VGA, the framebuffer used by the guest is in a > (virtual) PCI MMIO BAR. Likewise, the guest maps the framebuffer as >uncacheable.I'm curious why this issue is specific to VGA memory? Is it because VGA is the only device that maps memory this way in QEMU? (I realize this is rare, not sure if it's unique.) How does QEMU deal with other devices if it can't assume it's seeing valid data for MMIO or UC guest mappings? Mappings known to be supported by real memory? Or alternatively, to otherwise invalidate the cache before accessing the VGA memory. (My concern here is that the ramfb approach wouldn't fix the VGA error, and I think it's likely someone could still misconfigure QEMU.)

Commentary 2 Laszlo Ersek 26.02.2019 17:33:19 UTC

(In response to Christophe de Dinechin deComment #1)> (Answer to Laszlo Ersek byComment #0)>> ​​​​When using VGA, the framebuffer used by the guest is in a>> (virtual) PCI MMIO BAR. As a result, the guest maps the framebuffer as > > non-cacheable. > > I'm curious why this issue is specific to VGA memory? > Is it because VGA is the only device that maps memory this way in QEMU? > (I know this is rare, I'm not sure if it's unique)Typically, a GPA range presented to the guest as writable MMIO is not supported by any host memory. With KVM, there is no memory slot and each write access is intercepted separately. QEMU offers device emulation based on individual write access. VGA is different; It's backed by writable host memory (and a KVM R/W memslot), and QEMU uses something called coalesced MMIO to update the display of that area of ​​host RAM. My understanding is quite lacking in this area, but guest VGA framebuffer captures are not intercepted individually.> How does QEMU deal with other devices if it cannot assume it is seeing valid data > for MMIO mapping or guest UC?For other devices, the MMIO write specifications are propagated to KVM's QEMU based on trap symptoms. QEMU doesn't need to read RAM to get the details.> If the architecture does not guarantee consistency when one mapping is UC and > the other is WB, this does not imply that QEMU should be able to substitute UC > for some mappings known to be backed by real memory become?This was one of the ideas discussed (several times); KVM might be able to provide an ioctl() just for that. Ultimately the idea was discarded. I don't remember why.> Or alternatively, otherwise invalidate the cache before accessing the VGA memory.This was another idea that was eventually rejected. First of all, it would be a privileged operation. (QEMU runs in a restricted manner, so it cannot directly execute the necessary instructions. There may be other obstacles. I don't remember.)> (My concern here is that the ramfb approach wouldn't fix the VGA error, > and I think it's likely someone could still misconfigure QEMU.)ramfb is not intended to fix the VGA bug; ramfb is intended to replace VGA. The ramfb framebuffer is presented to the guest as normal RAM (not MMIO), so the guest is expected to map it with WB attributes. New (and downstream) versions of the "virt" machine type may choose to reject VGA type devices entirely. (However, I'm not sure how this would be implemented in practice.)

Commentary 3 Andrea Bolognani 03.11.2019 09:19:44 UTC

This blog post by Gerd may contain information relevant to the issue at hand.https://www.kraxel.org/blog/2019/02/ramfb-display-in-qemu/

Commentary 4 Gerd Hoffman 09.05.2019 09:44:28 UTC

Support for the vfio ramfb option would also be useful for the vgpu boot screen. Today it can be activated as follows: <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display=' on'> [ ... ] </hostdev> <qemu: command line > <qemu:arg value='-set'/> <qemu:arg value='device.hostdev0.driver=vfio-pci-nohotplug'/> <qemu:arg value='-set'/> <qemu: arg value='device.hostdev0.ramfb=on'/> </qemu:commandline>

Commentary 5 Jonathan Youngsma 09.11.2019 21:36:59 UTC

(In response to Laszlo Ersek fromComment #0)> Libvirt already supports the "virtio-gpu-pci" device for AARCH64 guests > (like the "virtio" model), but that's only good enough for Linux guests, > not for Windows guests. Windows needs a directly accessible framebuffer and virtio-gpu-pci doesn't provide one.So how are we going to test this on Windows/aarch64?>The problem also disappears with virtio-gpu-pci because (a) there is >no significant framebuffer, (b) all display data exchanged with virtio>transfers is mapped as normal DRAM (WB) by both guests and host .> Unfortunately, Windows insists on a framebuffer for booting > (although a framebuffer with direct access is optional according to the > UEFI specification). So we can't use virtio-gpu-pci. > > (Don't confuse virtio-gpu-pci with virtio-vga. The latter is > virtio-gpu-pci PLUS traditional VGA. Not good for aarch64.) > > The problem goes away also with the "ramfb" device model that Gerd > wrote for QEMU. It's a (comparatively) simple platform device that uses DRAM>(WB) as a framebuffer, so again the mappings on both sides match as far as cacheability goes. The UEFI driver (QemuRamfbDxe, also by Gerd) exposes the random access framebuffer > to the Windows bootloader, so that the framebuffer requirement is met while > avoiding the incoherency problem on ARM64. Ramfb isn't a PCI device, but it doesn't have to be either. > UEFI-booted Windows is expected to only work with the UEFI framebuffer > it inherits. Graphics mode changes and such will of course not work, but the basic display is expected.> > Please enter the display device model "ramfb" in the XML domain.> Thank you!So are you suggesting that this ramfb display device is used as the primary display in these situations? In Gerd's blog post he suggests that this is possible (via the ramfb standalone device), but it's not a good idea. Or are you suggesting that ramfb should somehow be used in conjunction with virtio-gpu-pci? In other words, if we add support in libvirt domain xml for ramfb what is the expected configuration we want to support? and how should that be translated into the qemu command? Gerd's suggestion in comment #4 seems to be a slightly different topic.

Commentary 6 Gerd Hoffman 09.12.2019 07:13:42 UTC

> So are you suggesting to use this ramfb display device as the primary display in these situations? In Gerd's blog post he suggests that > ​​it's possible (via the ramfb standalone device), but it's not a good idea.Not very efficient due to lack of dirty tracking. So if you have something else (better), you should use it. The problem is that in aarch64 we don't have anything else. So yes, Ramfb support makes sense.> Or > are you suggesting that ramfb should be used in conjunction with > virtio-gpu-pci in some way?That's another use case (you might want to make this a separate bug).> In other words, if we add support in libvirt domain xml for ramfb what is the expected configuration we want to support? and how should this be translated into the qemu command?(1) independent ramfb: <video> <model type='ramfb'/> Translated to '-device ramfb'(2) vgpu+ramfb <hostdev mode='subsystem' type='mdev' managed='no' model = 'vfio-pci' display='on' ramfb='on'> Translate to '-device vfio-pci-nohotplug,ramfb=on,...'

Commentary 7 Laszlo Ersek 09.12.2019 18:37:28 UTC

Gerd answered me better than I could - thank you. Delete need info.

Commentary 8 Jonathan Youngsma 09.12.2019 21:37:20 UTC

(Answer to Gerd Hoffmann fromComment #6) Thanks Gerd, I thought this was asked, but so far in my experiments I have not been able to successfully use the standalone ramfb device as the primary display device. For example, a boot menu is displayed when the boot process starts. But after booting the operating system, the display is not updated. what am I doing wrong here?

Commentary 9 Gerd Hoffman 13.09.2019 11:17:53 UTC

(In response to Jonathon Jongsma fromComment #8)> (Answer to Gerd Hoffmann fromComment #6)> Thanks Gerd, I thought that was what was asked for, but so far in my > experiments I haven't been able to successfully use the standalone ramfb> device as the primary display device. For example, when the boot process starts, > displays a boot menu. But after booting the operating system, the display is not updated. > What am I doing wrong here?Linux @ x86 bios I think? You need vesafb in the guest (vga = ask on linux command line and choose a vesa mode). VGA text mode (which is used by vgacon) doesn't work because there is no VGA device at all. Alternatively, use something more modern: UEFI. edk2 comes with a ramfb driver and sets up a GOP on the ramfb device. It should work fine with all guests on x86 and arm.

Commentary 11 Jonathan Youngsma 05.05.2020 16:45:55 UTC

Apparently I neglected to update this error after implementing the function. The ramfb display device model became upstream-libvirt starting with version 5.9.0. Also the vgpu boot screen function (seeComment #4) became upstream 5.10.0. rhel-av 8.3 includes version 6.2.0 of libvirt, so this feature should already be implemented.

Commentary 13 Jiri Denmark 05.05.2020 16:52:56 UTC

This feature was implemented in RHEL-AV 8.2.0.

Hi Gerd, when I tried to add the 'ramfb' video device to the guest xml I got the error "unsupported configuration: domain configuration doesn't support the 'ramfb' value of 'video template'". Does the video device 'ramfb' require a different configuration?#virsh edit vm1...<device><video> <model type='ramfb'/></video>...</device>... Same error is also reported when I added the qemu command line in XML: #virsh edit vm1<domain>...<device><video> <model type='ramfb'/></video>...</device > <qemu:commandline> <qemu:arg value='-set'/> <qemu:arg value='device.hostdev0.driver=vfio-pci-nohotplug'/> <qemu:arg value='-set ' / > <qemu:arg value='device.hostdev0.ramfb=on'/> </qemu:commandline></domain>

Commentary 15 Gerd Hoffman 28.05.2020 09:08:31 UTC

Hmm, ramfb has been fixed downstream (commit 67511676246cce57becbd2dcf5abccf08d9ef737 although not mentioned in the commit message). So I think we need a qemu-kvm bug to reverse this.

Test with libvirt-6.0.0-24.module+el8.2.1+6997+c666f621.x86_64 and qemu-kvm-4.2.0-25.module+el8.2.1+6985+9fd9d514.x86_64. Test Steps: 1. Start a guest with standalone 'ramfb' display device: #virsh start test1# virsh dumpxml test1 | grep -A5 video <video> <model type='ramfb' heads='1' primary='yes'/> <alias name='video0'/> </video>Check the line cmd qemu:#ps aux | grep -i qemu-kvm..-device ramfb,id=video0..2.To check the display in the guest OS see:https://bugzilla.redhat.com/show_bug.cgi?id=1841068#c15-https://bugzilla.redhat.com/show_bug.cgi?id=1841068#c18

Move the bug to verified as perCommentary 14- 22. And trace 'vgpu+ramfb'Error 1847791.

Commentary 25 Jeff Nelson 01.08.2021 16:53:46 UTC

Change this TestOnly BZ as CLOSED CURRENTRELEASE. Reopen if the problem is not resolved.