Windows 9x is, by some accounts, incapable of using more than 512 MB of RAM. The severity of the problem is often questioned and there are reports of people all over the place who somehow get away with running more, but it is not at all surprising that you should be having problems.

If you want, you can boot up your system with Linux (or even Windows 2000) and see for yourself whether you have the same stability problems. Remember that some versions of Linux can be booted from a CD-ROM drive without having to install anything on your hard drive, so you don't need to worry about that.

If you still want to keep 768 MB of RAM in your system for whatever reason, then I suggest limiting the amount of RAM to 512 MB by using HIMEMX in your CONFIG.SYS. Some prefer to use a RAM disk to consume the extra RAM instead.

Here are the details about RAM limits for Windows 9x: https://blogs.msdn.microsoft.com/oldnewthing/20030814-00/
Perhaps the SYSTEM.INI settings mentioned in the comments could help, too?

It isn't uncommon for motherboards to be unstable with more RAM modules installed. Manufacturers are often unable to certify most brands of memory to work with all the slots filled out.

It works with 512MB? Use that. 512MB is plenty for Win98 anyways.

firage wrote:

It isn't uncommon for motherboards to be unstable with more RAM modules installed. Manufacturers are often unable to certify most brands of memory to work with all the slots filled out.

That's good to know, thanks! the Soyo website is long gone and I can't know what brand they would have certified. What would be the better brand to get for such a computer?

Jorpho wrote:

If you want, you can boot up your system with Linux (or even Windows 2000) and see for yourself whether you have the same stability problems. Remember that some versions of Linux can be booted from a CD-ROM drive without having to install anything on your hard drive, so you don't need to worry about that.

MEMTEST86+ is a tool designed to test the memory of a system without relying on an OS. My system fails it, but only when all three slots are populated with 256MB. The version I'm using is from an old Ubuntu 6.06 install CD that I had laying around.

jakethompson1 wrote:

Here are the details about RAM limits for Windows 9x: https://blogs.msdn.microsoft.com/oldnewthing/20030814-00/
Perhaps the SYSTEM.INI settings mentioned in the comments could help, too?

I knew of the Windows 98 bug that prevents it from booting if the memory is above 1GB. I was planning to use a RAMdisk utility to bypass the problem when installing Win98 on a more powerful machine, but I never got around to do it. I was able to install Win98 on said machine after removing the extra RAM to have it down to 1GB exactly though.

dr_st wrote:

It works with 512MB? Use that. 512MB is plenty for Win98 anyways.

That's probably what I'll end up doing, but man the OCD in me won't let go of having an empty slot that could be used for stuff! hnnng...

Even though 3x256MB is overkill, the OS uses all of it for some tasks. The system is completely stable to do most of the common stuff, like using software, playing games, etc. The one thing it fails at is moving a big file or plenty of small files during an install. But the total size of said files aren't even close to 512MB, It hanged when installing Unreal, and when moving a 400MB file from CD to HDD. Windows 98 must be taking advantage of having all that RAM, as stupidly overkill as it is.

jarreboum wrote:

Even though 3x256MB is overkill, the OS uses all of it for some tasks. [...] Windows 98 must be taking advantage of having all that RAM, as stupidly overkill as it is.

I don't see how you can possibly come to that conclusion. Is it not entirely possible that "most of the common stuff, like using software, playing games, etc" is being done within 512 MB of RAM? Why would you expect Windows 98 to work when MEMTEST86 fails?

Some MSKB articles specifically refer to problems with more than 512 MB and starting an MS-DOS session. (Yes, that particular article supposedly relates to Windows 95.)

What I meant that in most common operations in Windows, only a small amount of RAM seemed to be used, whereas during big file transfer the entirety of it was: I had no problem operating with 768MB as long as I didn't try to transfer too many files at once. I did minimal installs of games and played them for example. MEMTEST86+ was also confirming that operating in the first 512MB range, everything was fine, but trying to do anything in the last 256MB gave errors.

Using MaxFileCache would work I believe in limiting the amount of RAM available to Windows. But it's an OS-dependent solution to a systemic error: I would have to find equivalent workarounds for every OS I might be using, to end up with a solution that would be effectively equivalent to physically removing the last RAM stick.

I'm reading about ECC RAM now, which my system seems to support. Maybe having the last stick as ECC or the whole three may help?

for ECC to work all sticks have to be ECC, its an all or nothing deal. Just stick with 2x128 and 1x256 if its that important. Likely the board was never fully tested with 768mb of Ram, at the time that was an ungodly amount, 128-256mb was normal at the turn of the century.

512MB works fine.
Anything above probably requires this patch (it has resolved all issues I had with 1GB of RAM under 98SE):
http://www.msfn.org/board/topic/105373-vcache-fix-attempt/
Just use it, and see if the problem is resolved. It fixes more things than just the one mentioned in the OP.

MrMateczko wrote:

512MB works fine. Anything above probably requires this patch (it has resolved all issues I had with 1GB of RAM under 98SE): htt […]
Show full quote

512MB works fine.
Anything above probably requires this patch (it has resolved all issues I had with 1GB of RAM under 98SE):
http://www.msfn.org/board/topic/105373-vcache-fix-attempt/
Just use it, and see if the problem is resolved. It fixes more things than just the one mentioned in the OP.

But OP says memtest86+ reports the third stick as bad, as well. I doubt this would help. FWIW my PIII ran 98SE with 768MB RAM just fine with no patches.

Oh, must have missed that part 😀
In terms of the patch, when I have 1GB of RAM without the patch, 3DMark 2001SE can't run the benchmarks, and older NVIDIA GPU drivers do not work (4x.xx). With the patch, everything works fine.
I do not know if 768MB causes same issues.

My recommendation is to just stick with 512 MB max 😀 It's tried and tested.

I remember running into this problem a lot. It was not that I wanted to use a full 1 GB of RAM in Windows98SE, but because the system was a multi-boot system with 98SE, XP, W2K, NT4, and sometimes W2K3. A few years back, XP really needed 2 GB of RAM for web browsing, so I didn't want to limit my system to 512 MB. Essentially, you want to do what Jorpho said, except in all my system which are multi-boot like this, I have limited the RAM to 1 GB. I have a text file saved which outlines some steps,

1How to use 2GB memory with Win98SE
2
3Use Japhet's HimemX as the XMS memory manager.
4
5Open the Config.sys file and add the following line to limit the memory available to windows to 1 GB.
6Device=himemx.exe /MAX=1048576
7
8Open System.ini
9[386Enh]
10MaxPagingFileSize=48000
11MaxPhysPage=48000
12
13[vcache]
14MinFileCache=32768
15MaxFileCache=261120
16
17Some people have reported that DOS will recognise 2GB after adding /NUMHANDLES=128 to it's config.sys line, howver Win98SE is capped to 1GB via "MaxPhysPage=3FFFF" in system.ini. 
18
19You can utilise the remaining 1GB of RAM by using a RAMDISK as a SWAP file.  
20
21Allocated the RAMDISK to the top of XMS
22xmsdsk parameter /t
23Then change paging drive to the RAMDISK drive letter using System Properties.

Even with this information, I had to do a lot of trial and error to get 1 GB working well enough. It was very system-dependent. I remember there was another program which took all the guess work out of this, but it costs. I forget the name. Good luck.

I'll try experimenting with smaller size sticks in various order to know exactly what the problem is. But ultimately, I feel like the motherboard just doesn't like having more than 512MB, regardless of what's written on the box. Which is fine, mind you, 512MB is a lot to play with.

I have a system on which I plan to have more than 1GB and both 98 and XP. I will make a RAM disk on the 98 install so that it limits the RAM to 1GB or less, and also put the swap file in that RAM disk. So that I can use all my RAM even if Win98 doesn't like it.

jarreboum wrote:

and also put the swap file in that RAM disk.

ConservativeSwapfileUsage=1 is your friend to completely disable swap file under 98SE. (actually to make it 0 bytes constantly, but it's the same thing)
You can use that RAM disk for other things, like browser cache....or maybe even make 98SE run entirely from the RAM Disk? It'll be like having an SSD, but faster! (is that even possible?)

MrMateczko wrote:

....or maybe even make 98SE run entirely from the RAM Disk?

How would you do that?

I'm running Windows 98 on my retro PC and I'm perfectly happy with 256MB of RAM. Dunno why people are complaining about not being able to have more than 512MB. I ran Windows XP for a few years with only 256MB, then bumped it up to 512MB. THAT PC now has 1GB....but come on, it's XP SP3! You weren't really able to do anything stable unless you HAD 1GB at that point.

Back in 1999-2003, I ran Windows 98 with 128MB of RAM and it never ran out, it never slowed down, etc.

If memtest86 is showing errors with 768MB then settling for 512MB isn't a very convincing solution. It sounds like the board's stability is marginal.
I'd at least do some more thorough testing at 512MB to make sure that it's actually totally stable at that point. Run memtest86 for a longer time in that configuration, letting it get good and hot. Also let it run the Prime95 torture test for a few hours.
If the board allows it, I'd try doing the tests with the FSB overclocked by a few %. If it's a 100FSB setup then clock it at 103 or so. if it doesn't pass sustained stress testing with a tiny overclock then it's not going to be reliable.
If you can, try checking voltages with a multimeter while it's under stress. Check the caps. If voltages are fluctuating with load then the PSU is probably not doing well.

shamino wrote:

If memtest86 is showing errors with 768MB then settling for 512MB isn't a very convincing solution. It sounds like the board's s […]
Show full quote

If memtest86 is showing errors with 768MB then settling for 512MB isn't a very convincing solution. It sounds like the board's stability is marginal.
I'd at least do some more thorough testing at 512MB to make sure that it's actually totally stable at that point. Run memtest86 for a longer time in that configuration, letting it get good and hot. Also let it run the Prime95 torture test for a few hours.
If the board allows it, I'd try doing the tests with the FSB overclocked by a few %. If it's a 100FSB setup then clock it at 103 or so. if it doesn't pass sustained stress testing with a tiny overclock then it's not going to be reliable.
If you can, try checking voltages with a multimeter while it's under stress. Check the caps. If voltages are fluctuating with load then the PSU is probably not doing well.

Some motherboards can handle faster memory timings in the BIOS if less total RAM used.