Virtualbox' old, experimental 3D drivers for Windows XP had to be installed in Windows Safe-Mode (VirtualBox 6.0.24 or earlier, using VBoxVGA device).
That's because they were based on WineD3D of the WINE project.
In Safe-Mode, running the installer for the VM Guest Additions caused Windows XP system files to be replaced by the WINE files.
The reason for this was because Virtualbox had used OpenGL on the host side to 3D render graphics. Using WineD3D helped here.
Running the installer on normal Windows XP mode does only install 2D support (DirectDraw, GDI, Overlay).
The checkbox for 3D support is grayed out, if I remember correctly.
Windows Vista/7 and up is using a different 3D driver set, by the way.
It does not require booting in Safe-Mode for installation.
Edited: I've found some videos that may help.
VirtualBox 6.0.24 or older and Windows XP SP3:
How to go in safe mode and enable the Direct3D in windows xp in VirtualBox
https://www.youtube.com/watch?v=2XoqWDcjy2Q
VirtualBox 7.0.16 or higher and Windows 98SE:
How to enable 3D acceleration in VirtualBox with SoftGPU! (For Windows 98)
https://www.youtube.com/watch?v=yx6_gdnMe_g
"Time, it seems, doesn't flow. For some it's fast, for some it's slow.
In what to one race is no time at all, another race can rise and fall..." - The Minstrel
//My video channel//