I seem to recall some MS articles claimed Win98 couldn't run with more than 512MB and others said it couldn't tolerate more than 1GB and had to be set lower if the system had more than that.
There were two items that could be set in SYSTEM.INI to help, depending on what was causing the problem:
[386Enh]
rem 896MB for Win9x
rem MaxPhysPage=38000
rem 768MB for Win9x
MaxPhysPage=30000
rem 640MB for Win9x
rem MaxPhysPage=28000
rem 512MB for Win9x
rem MaxPhysPage=20000
[vcache]
rem 512-640MB
MinFileCache=524288
rem MaxFileCache=655360
MaxFileCache=524288
Reference Microsoft:
Q181862 - Specifying Amount of RAM Available to Windows Using MaxPhysPage - Win9x
Q181966 - System Configuration Utility Advanced Troubleshooting Settings - Win9x
Q253912 - Out of Memory Error Messages with Large Amounts of RAM Installed - Win9x
Q304943 - Computer May Reboot Continuously with More Than 1.5 GB of RAM - Win9x
The numbers are the Microsoft Knowledge Base Article numbers. I think they don't put the "Q" in front anymore.
Hope this might help.