VOGONS


First post, by Nazo

User metadata
Rank Member
Rank
Member

I was playing a game that did a lot of reading/writing from the harddisk at times and would kind of freeze up while it waited for the reading/writing to finish and a thought struck me. In this modern day, most of us have 1GiB or even more of memory -- even the worst cases often have at least 512MiB. That's enough to fit many games in their entirity into a RAM drive even accounting for the OS and emulation each using a fair bit of that memory.

I tried the idea out with Daggerfall, which was the one that kept freezing up on me, and where it would very noticably jerk when someone added an entry to my map, there was an almost imperceptable hitch and while saving would really jerk and halt like crazy with a long delay between me hitting save and it getting back into the game (complete with the music sounding kind of funky as notes get sustained longer than they were intended to be due to the way MIDI works) after I put it on a ramdrive, it too had an almost completely imperceptable pause, though it was admitedly more noticable than the map entries. Overall the game ran quite noticably smoother with minimal hitching for reading or writing and I felt like it was worth going to all the trouble to make it work like that.

Handling the RAM drive internally in DOSBox would make handling images a lot easier obviously, but what's more, since we're talking about handling images for multiple games, it really makes a difference as there would likely be several images since anyone using this would probably want to use it for several games. Whereas you only need one command, let's say "imgmount e dagger.img -t hdd -ramdisk" if it were done in DOSBox (and even this could be simplified further by tossing it into a batch file,) to handle this for multiple image types in an external program like RamDiskXP, you have to dig through their configurations and change which image file is loaded for every single game (unless you can create a ramdrive so huge that it fits every game that needs it into one image, but I think that's probably unlikely since most of the games that need it are fairly large.) Besides harddrive images for really big games like Daggerfall (some of which can be made to run fully off of harddrive as I'm doing with Daggerfall atm,) I probably don't have to tell you how many games could benefit from mounting a CD-ROM image into memory... Yes, even running the games and CD images from a harddrive isn't always the best way depending on how busy or slow the drive is or perhaps even fragmentation.

The only catch that I can think of is that it really needs to save any changes so savegames don't get lost for the harddrive or floppy images. It struck me that if it could smoothly write in the background, that would be best since it would mean that most crashes would be no more harmful than they normally would be. Even if that were too much to actually implement (I imagine that to keep it from making games get a little jerky as it saves it would be pretty tough to implement,) a save on exit would be effective enough for most games except for the few that like to freeze a system on exit (well, I've seen it in the past, but I admit it doesn't happen much.) Maybe a key that would force a manual save could be useful even? Just as long as there's some way it can at least semi-automatically handle saving so people don't have to worry about forgetting to save manually. Even if it required running some command like say "imgsave" or something, that could just be put into a batch file with the imgmount command itself so the user wouldn't have to worry about forgetting, and isn't so hard to type by hand either.

I'm not sure, but it also struck me that DOSBox would probably handle such a thing a little more efficiently with less overhead as it doesn't have to try to work backwards with an OS, but can actually work at a more low level without much need for trickery. At least the program I was using seemed to use some 17MiB or so of overhead in RAM, which isn't much, but I'm thinking that in some of the worst situations memory could be a little tight for some with really large games (for example, Daggerfall is using well over 500MiB for me after moving all the files from the CD to the harddrive and patching it to the latest version and all) or for people with lower amounts of memory.

It's just a thought anyway, but I think it could really be nice for DOSBox to have such a thing.

Reply 1 of 13, by DosFreak

User metadata
Rank l33t++
Rank
l33t++

You can easily load MS-DOS in DosBox and create RAM drives up to 64mb in size.
The limit is 64mb because DOS games do not require any more than that and the DosBox devs are not going to make it any higher unless there is a need to do so, although they could probably remove that limitation for OS's booted in DosBox.

I don't see a need for storing games in RAM in DosBox....if your HD is really performing that slowly on an ancient DOS games you really have more important things to worry about....

How To Ask Questions The Smart Way
Make your games work offline

Reply 2 of 13, by Nazo

User metadata
Rank Member
Rank
Member
DosFreak wrote:

You can easily load MS-DOS in DosBox and create RAM drives up to 64mb in size.
The limit is 64mb because DOS games do not require any more than that and the DosBox devs are not going to make it any higher unless there is a need to do so, although they could probably remove that limitation for OS's booted in DosBox.

Not really a solution though, now is it? Besides the limitation you already mentioned, you will have to manually copy all of the files over to the new drive, then manually copy them all back. You can maybe set up a batch file to do that, but, it would be very slow, inefficient, and definitely not the best of solutions. Not to mention that you'd have to go to great lengths to trick some games into actually working this way (especially if you're trying to get the CD image into memory, which would probably be a lot more common than the harddrive image to be honest.) Then too, some games don't play nice with things like RAM drives and would have serious problems if they'd even run at all. (For example, I'll bet those with their own memory handling like some of the Wing Commander games would probably really be a problem.) Besides, that 64MB limit is one heck of a limit -- not to mention that once you subtract whatever the RAM drive uses, you have that much less for the game itself to use... Basically no game worth putting on a RAM drive would really benefit.

I don't see a need for storing games in RAM in DosBox....if your HD is really performing that slowly on an ancient DOS games you really have more important things to worry about....

Well, for one thing, I'm running it off of a laptop harddrive. They are improving, but it's still going to be a while yet before any laptops get desktop fast harddrives. However, depending on the size of the game and how it handles its data, it can be inefficient enough to where it's not necessarily even the harddrive at fault. For example, Daggerfall has very nearly 1500 files in its main data folder... What am I supposed to do to my harddrive to make that perform super-fast? Buy a Raptor? If I were rich, I'd buy the hardware to put together the perfect old PC complete with a Yamaha DB50XG and all then play the games on it instead.

Instead of just immediately shooting down the idea as if I were asking for a complete overhaul of DOSBox or something else crazy like that without even a single thought, just consider it for a moment. All I'm talking about is something that could really improve the gameplay of some games with a lot less jerking and such, and I honestly believe that it actually wouldn't even be so hard for the imgmount command to implement such a thing.

Reply 3 of 13, by `Moe`

User metadata
Rank Oldbie
Rank
Oldbie

What you describe is wha a decent operating system does automatically: If there's enough free RAM, keep everything it has already read in memory for later reuse, and prefetch data from files that are being read. This is really the best way to do it, because no one has to code anything.

OTOH, if you had a RAM disk as you describe, and the OS thinks your RAM disk is in it's way, it will happily swap it out to the swap file, thus making your RAM disk a disk disk. Quite pointless, eh?

So what you can reasonably do instead is:
- Check your operating system's settings, make sure this kind of buffering is active and aggressive enough. (Right, your favourite MS operating system allows almost no control over this. System settings contain something, and probably something in the registry.)
- Of course, avoid disk compression and encryption
- Make the OS read your image file in whole before starting DOSBox, copy it to the null device or something. That way, you instruct the OS to buffer as much as it can in RAM.
- Optimize all other disk related settings as you'd do for any game.

Reply 4 of 13, by Nazo

User metadata
Rank Member
Rank
Member

Bah, I was just trying to make a suggest for something that I thought could help improve DOSBox for users. Forget it. I didn't expect such hostility over the concept to the extent that people immediately try to find any thing they can use against it, however much of a stretch it may be. Just for the record, I'll go ahead and address your statements, but I guess I'll just give up on this topic as a lost cause.

`Moe` wrote:

What you describe is wha a decent operating system does automatically: If there's enough free RAM, keep everything it has already read in memory for later reuse, and prefetch data from files that are being read. This is really the best way to do it, because no one has to code anything.

Except that you'll need to change the code behind your operating system's caching, because frankly even linux isn't quite that good at it. It tries to be smart about it, and it's going to end up concentrating first on what it thinks is needed (which will ultimately end up being what is needed for the moment, not all that is needed) and remember that no system in the world has no other applications running -- even DOS tends to have TSRs taking up SOME resources -- so you can bank on it dedicating some of that intelligent buffering to other programs as well.

But, did it occur to you that not everyone using DOSBox is using it in linux? Some of us are stuck with Windows (and I say stuck loosely here, because I'm forced to admit that I rather enjoy getting to use Direct3D -- which my videocard handles a lot beter than OpenGL -- combined with a 2xSaI pixel shader resize algorithm handled by my video hardware so that my CPU can concentrate on keeping those games running smoothly.) Windows does not use all available memory as a buffer. Though again, even if it did it still can't perfectly decide everything to cache for you.

OTOH, if you had a RAM disk as you describe, and the OS thinks your RAM disk is in it's way, it will happily swap it out to the swap file, thus making your RAM disk a disk disk. Quite pointless, eh?

Ah, but part of the beauty of the idea here is that DOSBox's implementation would try to allocate the memory if it could (and if it couldn't it would have to fail anyway) so the OS is going to have to consider it untouchable unless you just run completely out of unallocated memory and it starts swapping even running programs. Anyway, I'm not an idiot, I realize the danger of letting it swap. I have 1GiB of memory and a decently efficient setup, so I had a bit of leeway left over even after loading that huge ram disk for my Daggerfall folder and it's 1,695 files. I kept an eye on the task manager's physical memory listing and made sure that even in the middle of a game at the points where memory usage ran highest I had a bare minimum of some 50MiB of physical memory still unused. But, don't take my word on this for it, consider the fact that I specifically said that it ran almost completley smoothly for me once I did all this, and if, as you said, it started paging that ramdrive, then the game would be so much worse that I think I would probably notice...

Ultimately, you can't count on buffers to do everything for you. When you have a time sensitive application -- eg something actually in the foreground that must stop and wait for a read/write operation to finish -- then eventually you will end up back to the point where buffers fail to handle everything for you and you ultimately fall back onto the speed of the device itself. Really, if this weren't true, do you think they'd be able to so easily sell things like those Raptor drives at such insane prices when it would be, by your system, far more efficient just to have more memory? After all, the chief application of a Raptor drive is for the sort of system where there's going to be a lot of memory to begin with (eg servers and high end systems where the person can afford to splurge on getting more than they should really need.)

- Check your operating system's settings, make sure this kind of buffering is active and aggressive enough. (Right, your favourite MS operating system allows almost no control over this. System settings contain something, and probably something in the registry.)

Actually, there are some settings for this. They are, by default, at their highest, but people can lower them for troubleshooting purposes. I verified quite some time ago that they are still at their highest.

- Of course, avoid disk compression and encryption

I will state for the record that I do avoid these, however, the bottleneck has been proven to be the harddrive, and therefore these are not related as if these were causing a noticable slowdown it would mean that either CPU or memory was acting as a bottleneck.

- Make the OS read your image file in whole before starting DOSBox, copy it to the null device or something. That way, you instruct the OS to buffer as much as it can in RAM.

Well, I posted the response to this one earlier basically. Windows can't really do it, and linux tries too hard to be smart, so still won't do it perfectly. Besides, if nothing else, it's nice to have control over it and ensure that other programs don't start taking buffers that you need for the game.

- Optimize all other disk related settings as you'd do for any game.

You said it yourself. "As you'd do for any game." I've played quite a few, such as Oblivion, where my 1GiB of memory can prove to be not quite enough and harddrive access becomes a must. I've done as much as I reasonably can. But, uhm, I think right now might be a good time to point out that you'd pretty much need a Raptor or that new thing that Seagate is talking about releasing (supposedly running at a full 15,000 RPM) to get the kind of insane seek time that is necessary to handle accessing so many files back and forth. Admitedly I'm playing the game off of a 5400 RPM drive (yeah, it's a Scorpio, so it's 5400 RPM instead of 4200 RPM as you'll find in most other normal pricerange laptop drives) so I can't expect the blazing speeds of even a 7200 desktop drive or a 10,000 RPM Raptor, but I do think that if it's tough enough for a 5400 RPM drive to cause noticable hitches, it may just be something you can chalk up not so much to poor optimizations on the part of the drive as perhaps the fact that when you have some > 1600 files total, it's just plain inefficient, period. But what am I supposed to do about that? Rewrite the game engine myself?

Anyway, I'll just drop it now. Clearly people hate the idea, so just forget it. I mean, seriously to even use buffers as a reason to claim it's automatically a bad idea? Bah, just forget it. If a mod wishes, they can feel free to even close the topic if they choose. Either way, I think I'm going to just give it up for lost and not bother to address further responses. I just meant to suggest what I thought was a good idea for a way to improve playing games through this method and never really meant to start a little fight over why ramdrives are supposedly evil. Hopefully I'll be able to set up a retro gaming system where I won't even need DOSBox in the near future since I'm tired of dealing with games needing so much CPU power anyway (even using timesynched and a number of cycles that my CPU could not possibly hope to handle without that option, it doesn't always work 100% smoothly in some tougher games like Daggerfall.)

Reply 5 of 13, by DosFreak

User metadata
Rank l33t++
Rank
l33t++

Instead of just immediately shooting down the idea as if I were asking for a complete overhaul of DOSBox or something else crazy like that without even a single thought, just consider it for a moment. All I'm talking about is something that could really improve the gameplay of some games with a lot less jerking and such, and I honestly believe that it actually wouldn't even be so hard for the imgmount command to implement such a thing.

Provide evidence that putting DOS games on RAM drives when using Windows 2000+/Linux substantially increases performance and then you may have something.

I've never had this problem on my Dell Latitiude D510 (Probably 5200 RPM at the most) laptops with whatever speed HD's they use. Nor a 120GB 7,200 RPM HD I bought back in 2002 for my desktop machine. Nor the 146GB WD 10,000 RPM HD I just put in my main desktop machine.

Of course I do not play alot of DOS games with tons of files since those games are pretty rare. The fact that a game has tons of files doesn't necessarily mean that the game is using those files all the time. You mentioned Daggerfall. I've played Daggerfall in DosBox a couple of times and I never noticed a situation where increasing HD performance would help out Daggerfall. The limiting factor for Daggerfall in DosBox has been CPU performance.

When I played Daggerfall on a K5 133?? machine back in 1996 I didn't have to worry about CPU or HD performance when I could actually get the game to work due to the extreme bugginess. Heck, I didn't even use Smartdrive since I barely noticed a difference when using it with the games that I played.

Last edited by DosFreak on 2007-03-13, 11:29. Edited 2 times in total.

How To Ask Questions The Smart Way
Make your games work offline

Reply 6 of 13, by DosFreak

User metadata
Rank l33t++
Rank
l33t++
Nazo wrote:

Bah, I was just trying to make a suggest for something that I thought could help improve DOSBox for users. Forget it. I didn't expect such hostility over the concept to the extent that people immediately try to find any thing they can use against it, however much of a stretch it may be. Just for the record, I'll go ahead and address your statements, but I guess I'll just give up on this topic as a lost cause.

If you ask for something then prepare to defend it. We don't implement a feature just because it sounds like a neato whizbang idea. We implement features that people need. If you see this as hostility then that's your problem, it's actually constructive criticism. Deal with it.

How To Ask Questions The Smart Way
Make your games work offline

Reply 7 of 13, by wd

User metadata
Rank DOSBox Author
Rank
DOSBox Author

And try to keep stuff at "daggerfall runs slow thus i want ramdrives in dosbox"
instead of posting epic texts which drip down to the same amount of information.
I personally ignore those completely. Besides that i agree with Moe's
short list, it surely is not the duty of dosbox to fix your caching problems.

Reply 8 of 13, by h-a-l-9000

User metadata
Rank DOSBox Author
Rank
DOSBox Author

DOSBox may want to learn not to lock during longer disk (especially disc) access. On real hardware sound would continue playing while disc spins up because of interrupts. In DOSBox the whole emulation loop is waiting on the I/O-call and thus it seems like dead. I had a patch for that but it caused a few bugs and is currently broken. The idea is to add an additional thread that waits on the I/O while the emulation loop thread is idling around.

1+1=10

Reply 9 of 13, by gulikoza

User metadata
Rank Oldbie
Rank
Oldbie

Hard disks are about 100 times faster then they were in the DOS times and even Windows has useful buffering (try searching for something two times, the second time it will barely scratch the disk surface) so I don't thing disk access is the bottleneck here...

http://www.si-gamer.net/gulikoza

Reply 10 of 13, by h-a-l-9000

User metadata
Rank DOSBox Author
Rank
DOSBox Author

... the second time Tyrian will not corrupt the DOS MCB with that patch 😉
The games usually read things the first time.

It's also about CD-ROM. It takes a few seconds to spin it up. Not to mention Network and floppy access.

Generally game music stutters less (the first time) while loading with that patch.

> Hard disks are about 100 times faster then they were in the DOS times

http://www.cooling-station.net/index.php?page=museum&id=171
http://www.digit-life.com/articles2/disks-2k2 … d1200-2000.html

ST-251:

Track to track 8 ms
Average 40/28 ms
Full stroke 95/70 ms
Average latency 8.33 ms

WD1200:
Latency 4.2 ms
Average search time in reading 8.9 ms
Average search time in recording 10.9 ms
Average track-to-track search time 2.0 ms
Average 'full stroke' search time 21.0 ms

100x? 😀 Though the buffer will probably help a lot.

The virus scanner adds to access times, too.

Reply 11 of 13, by `Moe`

User metadata
Rank Oldbie
Rank
Oldbie

- Check your operating system's settings, make sure this kind of buffering is active and aggressive enough. (Right, your favourite MS operating system allows almost no control over this. System settings contain something, and probably something in the registry.)

Actually, there are some settings for this. They are, by default, at their highest, but people can lower them for troubleshooting purposes. I verified quite some time ago that they are still at their highest.

I am not familiar with Windows, but I'd be surprised if they did it radically different than linux. And that means: there is no "highest" setting per se. Rather, system performance can be optimized by a decent buffers vs. active memory ratio. Linux will swap out memory pages even if enough RAM is free, if it thinks that more disk buffers are better at the moment. The priority of those two usages can be tuned. Windows has something like that as well and at least has a "many buffers" vs. "many active pages" toggle (somewhere in advanced system settings). That's what I meant.

I'd love to see some facts about your claims surrounding the deficiencies of the linux memory manager. If it indeed gets things wrong, I'd like to assist in fixing it. Can you point me to some more technical discussion of the issues you describe?

BTW, don't take a factual discussion personal. I am convinced that the goal you want to achieve can be solved much better using existing mechanism (since they were written for exactly this purpose), so I will defend this opinion. I didn't say "what you say is stupid", nor did I criticise you personally. All I said was that there is a better way (of course, judging from my personal experience and knowledge, thus forming my opinion). Feel free to prove that your experience and knowledge is superior, I won't have any problem with that.

Reply 12 of 13, by computer88

User metadata
Rank Newbie
Rank
Newbie

if you look at a true dos mechine (which i have a few ranging from old 8088's on up to curent day computers) using a ram drive is substancely faster and in do at least true dos 6.22 it actually makes it a formated drive so when u put a file in it it is compleatly in ram and no other app or system process can use that pieace of ram.

but in my curent situation i really need ramdrive support becouse i am running from o flash drive and reunning most games and win 311 is hell becouse of drive acess time if i could load win and the apps into a ramdrive it would elimate the hole problem.