Windows 3.x era drivers had a bad habit of strewing files all over \WINDOWS and \WINDOWS\SYSTEM with no uninstall utility, so odds are there's no clean way to do this.
For the Control Panel applets, there will be *.CPL files in \WINDOWS\SYSTEM that you need to delete. You should be able to figure out by the time stamps and/or file names which ones belong to the old cards.
Failing that, I'd look through the original driver installer packages. They probably have either compressed single files like *.CP_ or a ZIP file or something like that where you can verify the file names of the stuff you need to delete.
If you ever need to just return to the original VGA drivers, you can use the SETUP util as outlined in the reply above me or do what I do and just edit the display line in \WINDOWS\SYSTEM.INI to say display=VGA.DRV
Oh, and that's another thing: there will most likely be sections in WIN.INI and SYSTEM.INI added by the drivers that you need to remove if you want to totally eradicate them.