Application uses CPU to render it's own window buffer which is located in system ram or (better) in video memory. (although even this can be hw accelerated if using some vector drawing API's, or e.g. OpenGL)

Then a 3D engine (implemented in compositing manager) uses GPU to transform and render window buffer into a final framebuffer, "pretending" that it's a texture. This is in principle quite similar how 3D accelerated game engines work (much simpler though).

Older way of doing this is to use much more limited but still acceleration-capable 2D portion of the chip which allowed hardware (DMA) blitting of window bitmaps (or tesselated parts of those) into the VRAM framebuffer. Those chips had some other 2D functions but those are now limited compared to capabilities and programmability of current 3D hardware.

As double buffered windows are used in Avalon (and other compositing managers), if app locks front-buffer (requires to draw directly to screen) than compositor can't work and needs to shut down. I'm not sure if it's a policy to remain suspended until problematic app terminates, or SDL/dosbox forgets to unlock front buffer (?).