It runs, goes to where it says "installing components" and "Initializing" but just sits there.
After a while I switched the monitor back and I saw a black screen with vertical green lines on the left.

Weird stuff eh?
I tried the June and December 2006 DirectX versions and both do the same thing.

System:
Asus P3V4X
P3 Coppermine 800
512MB SDRAM
GeForce 3 ti 200
Realtek based GBE Network card
160GB Hard drive
Windows 98SE with SP3 updates

Solution:
The problem was setup wasn't finding it's files, rather than tell me it'd rather just sit there twiddling it's thumbs. 😜
The solution was to move the dx install folder to the root of the C drive with a folder name of less than 8 characters.

Both of those worked for me, not sure which one did it though.
The moral is not to try to run it from the desktop anyway.