The newer AMD drivers for XP tend to get worse the newer you go. The version number you listed is for the HP packaged version of the driver. What's the actual Catalyst version number? If you're happy with the current performance then you probably don't need to worry, but this is worth keeping in mind. What model is the APU? I think those HP's shipped with Terrascale based APUs and Terrascale tends to fair better on older drivers.

This thread has some comparisons on the topic : AMD Catalyst drivers for Windows XP - newer is NOT better

As for the SFF system, a 10w limit on a PCI-e slot seems awfully low, I wouldn't be surprised if you can't exceed it by a little bit. According to the HP documentation, they list several GPUs that the systems could've been configured with and most of those GPUs would blow past that 10w limit easily, assuming they don't just power throttle into oblivion that is. https://support.hp.com/us-en/document/c03522674

Considering the era of games and resolution you're targeting, the integrated graphics on the APU would probably be enough unless the processor has been switched out for one without a iGPU.

Thanks; the info on the catalyst version says 12.8.

The CPU is the A8 so it has the HD 7560 as GPU. It does run games fine but if I can squeeze extra FPS I would not complain 😁

The weird thing is that the GPU on that list you posted seems to be worst than the actual 7560? That is surprising, so I was wondering what is the newest card that would support XP that can fit on that slot, considering that probably up to 18W should be fine... If these cards shipped with the system are actually around 18W TDP (at least this is what the specs for these cards say).
Not sure if there is anything else I should do to improve performances, as I have totally forgot all the dark magic used for XP

The Radeon HD 8570 tends to be a pretty decent and cheap low profile card for XP. Phil'sComputerLab did a video on it some time back. https://youtu.be/04l_DiPaCBE

Sources seem to indicate it has a 50w TDP though. You can usually find it for between $10-$15 USD so it might be cheap enough to take the gamble on. I think there's GDDR3 and GDDR5 versions, but Phil's appeared to be the GDDR3 version and it still performed well.

The HP site lists a GT 630 as being an option and that's also rated at 50w for the HP OEM version on TechPowerUp. If you want to stay on the officially supported list, that's probably the best option since it has a 128 bit memory bus. Try to stick with the HP specific version that has the 128 bit bus if you can, some GT 630s are Fermi based and others use Kepler and may also have a 64 bit bus, so be careful before you buy especially since the Fermi ones will draw more power and have a worse core.