SysEx problems in some Sierra games

Developer's Forum, for discussion of bugs, code, and other developmental aspects of DOSBox.

SysEx problems in some Sierra games

Postby hakonrk » 2006-8-17 @ 19:52

I recently discovered the wonderful DOSBox emulator, and soon learned that it could make use of my old Roland CM-32L module. I hooked it up a few hours ago, and it seems to work great with every game I've tried, except for some (perhaps all) of Sierra's 16-color SCI games.

For example, in Quest for Glory 2: Trial by Fire, instruments and (especially) sound effects are often wrong. I immediately suspected problems with the SysEx commands, and googled around a bit to see if other people were affected by the same problem.

I found no other records of my particular problem, but I found something that confirmed my suspicions: http://www.queststudios.com/quest/qfg2.html. Here, someone has uploaded several of QFG2's MIDI tracks, along with the appropriate SysEx commands. When I load these SysEx files manually with amidi -s <sysex-file>, QFG2 sounds exactly the way it should.

The game database on dosbox.sf.net claims that QFG2 works perfectly on DOSBox 0.63, so either this is a regression in 0.65, or nobody has tested this game with a Roland module.

I'd really appreciate it if someone else with a similar Roland module could confirm that they, too, can reproduce this bug. I've made a copy of QFG2 temporarily available at http://folk.uio.no/hakonrk/tmp/qfg2.zip while we're investigating this problem. Hopefully, I won't get in trouble for this. :-)

Finally, some system information:

DOSBox version: 0.65
OS: Slackware Linux 10.2
Kernel: 2.6.16.27
Compiler: GCC 3.3.6
Sound hardware: Roland CM-32L connected to a SB Live!

Let me know if you need more information.
User avatar
hakonrk
Newbie
 
Posts: 30
Joined: 2004-2-13 @ 09:40

Re: SysEx problems in some Sierra games

Postby Kaminari » 2006-8-17 @ 20:38

I don't have any problem with QFG2 on my CM-500. Actually, no SCI1 should give you any problem with your CM-32L unless (maybe) the Roland driver provided with your game is not the correct one. Or is it a CM-32L+CM-32P you have?

There's a known problem with early SCI0 games (KQ4, PQ2, LSL2) where the Roland driver would play all parts of the MIDI file on a CM-64. This is bad since parts 11-16 contain... the AdLib tracks. Using a later driver (from KQ1SCI, LSL3 or QFG1) usually solves the problem, though I'm not sure why your CM-32L should be affected, especially in QFG2.

Some threads at Quest Studios that might provide you with some info...

Early Sierra games and the CM series
Turning off the CM-32P parts
User avatar
Kaminari
Oldbie
 
Posts: 775
Joined: 2002-12-06 @ 01:54

Re: SysEx problems in some Sierra games

Postby hakonrk » 2006-8-17 @ 20:58

I have a plain CM-32L, so there should be no problems.

I did some reading in the code, and saw that DOSBox uses a different method to send SysEx commands. Unlike amidi, which sends sysex commands directly to the raw device, DOSBox uses snd_seq_ev_set_sysex() followed by send_event(). Could I be experiencing a bug in ALSA? My version of alsa-lib/alsa-utils is 1.0.9, and, as I said before, my kernel is 2.6.16.27.

I'll try different versions of the kernel and the ALSA software, if someone can tell me a combination that works for them.
User avatar
hakonrk
Newbie
 
Posts: 30
Joined: 2004-2-13 @ 09:40

Re: SysEx problems in some Sierra games

Postby hakonrk » 2006-8-17 @ 21:45

Apparently, some SysEx messages get through. I tried disabling SysEx completely by commenting out

Code: Select all
snd_seq_ev_set_sysex(&ev, len, sysex);
send_event(1);


in src/gui/midi_alsa.h's PlaySysex() function, and the resulting sound was different from when the lines were enabled.

I know snd_seq_ev_set_sysex() gets all the sysex commands it should, because I tried dumping it all to a file and inserting it manually with amidi, and that works perfectly. Clearly, the event interface to sysex commands is broken, at least on my system. I'll do some more investigation, and post any new findings here.
User avatar
hakonrk
Newbie
 
Posts: 30
Joined: 2004-2-13 @ 09:40

Re: SysEx problems in some Sierra games

Postby Qbix » 2006-8-18 @ 06:19

keep us informed.
Maybe it has to do with the fact that we use "old" midi code.
Water flows down the stream
How to ask questions the smart way!
User avatar
Qbix
DOSBox Author
 
Posts: 10681
Joined: 2002-11-27 @ 14:50
Location: Fryslan

Re: SysEx problems in some Sierra games

Postby hakonrk » 2006-8-18 @ 08:08

Will do. The next step will require a little more time, though, as I didn't have any experience with ALSA's sequencer interface before yesterday. I'll have to study the API carefully and do some experimentation on my own before I can get any further.
User avatar
hakonrk
Newbie
 
Posts: 30
Joined: 2004-2-13 @ 09:40

Re: SysEx problems in some Sierra games

Postby hakonrk » 2006-8-18 @ 21:02

Status report: I've created a simple version of amidi that uses the event interface to send sysex commands. As a reference, I used pmidi and the alsa-lib sources, so the code should be using ALSA correctly (but my work was sloppy, so I can't be sure :-)). Basically, what my program does is this (my Roland is on port 16:0, so I've hardcoded those values in the following code):

Code: Select all
snd_seq_open(&handle, "hw", SND_SEQ_OPEN_DUPLEX, 0);
snd_seq_create_simple_port(handle, NULL, SND_SEQ_PORT_CAP_WRITE |
                                         SND_SEQ_PORT_CAP_SUBS_WRITE |
                                         SND_SEQ_PORT_CAP_READ,
                                         SND_SEQ_PORT_TYPE_MIDI_GENERIC);
snd_seq_connect_to(handle, 0, 16, 0);

while (get_next_sysex_command(sysex_data, &sysex_len)) {
        snd_seq_event_t ev;
        snd_seq_ev_clear(&ev);
        snd_seq_ev_set_source(&ev, 0);
        snd_seq_ev_set_subs(&ev);
        snd_seq_ev_set_direct(&ev);
        snd_seq_ev_set_sysex(&ev, sysex_len, sysex_data);
        snd_seq_event_output(handle, &ev);
        snd_seq_drain_output(handle);
}


Using this program, I get exactly the same problem as DOSBox: Most of the sysex messages are not received by the MIDI device, and the sound is consequently broken. I could actually show, by timing the process of sending the sysex commands, that something was wrong. Amidi takes over 6 seconds to complete, while my program used 1.5 seconds. This lead me to suspect that the kernel was pushing out sysex commands faster than the Roland could process them, so I inserted a delay of 100 ms between each sysex event, and, voilà, it works! :)

I've already created a trivial patch for my own private use, but it needs some tuning before it goes in the official source. In particular, I need to find a better way to wait for sysex completion, as 100 ms is way too long for most cases (takes forever to start some games), but on the other hand, the required delay may vary, so it doesn't seem like a stable solution to simply choose the shortest delay that seems to work for me.
User avatar
hakonrk
Newbie
 
Posts: 30
Joined: 2004-2-13 @ 09:40

Re: SysEx problems in some Sierra games

Postby Kaminari » 2006-8-19 @ 12:43

That's interesting. Is this a problem specific to the Linux build? Because, AFAICT, the Win build transmits at around 1 kB/s, which was the standard speed provided by most cheap (read: Creative) interfaces.

On my CM-500 + Midi Sport Uno, I can transmit SysEx messages (with SendSX) at around 3 kB/s without any loss. A SysEx of 18 kB takes less than 6 seconds to upload. Your CM-32 should probably suppport this rate as well. Maybe the SB Live port is the bottleneck.
User avatar
Kaminari
Oldbie
 
Posts: 775
Joined: 2002-12-06 @ 01:54

Re: SysEx problems in some Sierra games

Postby hakonrk » 2006-8-19 @ 13:30

Sending at 3 kB/s is no problem for me either - that's the speed used by amidi (and other apps using the rawmidi interface), actually.

The method used for transferring sysex data in DOSBox, however, sends at almost 13 kB/s, which is too fast. I haven't tried the Windows version yet, as I'm almost sure it works there. This is an ALSA specific problem, and so I've posted a message about it on the ALSA developer's mailing list.
User avatar
hakonrk
Newbie
 
Posts: 30
Joined: 2004-2-13 @ 09:40

Re: SysEx problems in some Sierra games

Postby hakonrk » 2006-8-19 @ 13:48

By the way, in case anyone is wondering why I don't do the obvious and use rawmidi to send sysex in DOSBox, it's because it's not possible: ALSA doesn't allow the rawmidi device and the sequencer to be opened simultaneously.

I tried working around it by creating a virtual rawmidi device that routed data to the MPU-401, but the result was exactly the same as when using the event interface: 13 kB/s sysex transfers, and loss of data. Unless I've overlooked something in the ALSA API that can help, I think the only real solution is to patch the appropriate kernel drivers so that the event interface does the same bandwidth throttling as the rawmidi interface.
User avatar
hakonrk
Newbie
 
Posts: 30
Joined: 2004-2-13 @ 09:40

Re: SysEx problems in some Sierra games

Postby Kippesoep » 2006-8-19 @ 14:19

This fragment from the Wikipedia article about the Roland MT-32 may be pertinent:
Buffer overflows

First generation units, having control ROM versions below 2.00, require a 40 millisecond delay between system exclusive messages. Some computer games which were programmed to work with the compatible modules (see above) or later ROM versions that do not require this delay, fail to work with these units, producing incorrect sounds or causing the unit to lock up completely.
My site: Ramblings on mostly tech stuff.
User avatar
Kippesoep
Oldbie
 
Posts: 1015
Joined: 2005-5-04 @ 22:36
Location: Netherlands

Re: SysEx problems in some Sierra games

Postby hakonrk » 2006-8-19 @ 14:44

That's strange - my CM-32L needs more than 40 ms (60 seems to be enough) between messages, but according to the article, it shouldn't need any at all, being a second generation device.
User avatar
hakonrk
Newbie
 
Posts: 30
Joined: 2004-2-13 @ 09:40

Re: SysEx problems in some Sierra games

Postby hakonrk » 2006-8-19 @ 15:08

Tried the Windows version just now: As I suspected, there are no problems here. If the problem lies in the ALSA sequencer kernel drivers, it's strange that no one but me has noticed it yet. Is everyone else using the Windows version?

If there are any Linux users with an equivalent Roland device reading this, I'd appreciate it if you'd let me know if DOSBox works for you.
User avatar
hakonrk
Newbie
 
Posts: 30
Joined: 2004-2-13 @ 09:40

Re: SysEx problems in some Sierra games

Postby hakonrk » 2006-8-20 @ 11:06

After a long and tedious kernel debugging session, I have discovered that this bug is actually caused by an incredibly stupid bug in the ALSA kernel code.

Turns out that the buffer that overflows isn't a hardware buffer in the CM-32L -- it's a software buffer in ALSA that is fixed to a single page of memory (4096 bytes on x86).

This wouldn't be a problem if there were a mechanism that allowed incoming events to wait until there's room in the output buffer, or at least to fail with EAGAIN, allowing the application to retry.

What happens here is that the function in question returns ENOMEM and prints a warning (but only if you enable ALSA debugging, which, of course, I had not) saying that events are lost due to buffer overflow. Unfortunately, the return code is ignored by the calling function, so the error is silently ignored.

At least now I have the problem is pinned down. This no longer has anything to do with DOSBox - the fix will probably be in the form a kernel patch, so I'll continue my monologue on alsa-devel. :)
User avatar
hakonrk
Newbie
 
Posts: 30
Joined: 2004-2-13 @ 09:40

Re: SysEx problems in some Sierra games

Postby wd » 2006-8-20 @ 12:00

Hehe, hope you enjoyed the kernel-level debugging at least a bit :)
Thanks for keeping this up.
wd
DOSBox Author
 
Posts: 10818
Joined: 2003-12-03 @ 21:23

Re: SysEx problems in some Sierra games

Postby hakonrk » 2006-8-20 @ 13:02

No problem, I was dead set on getting this to work in Linux anyway, so I wasn't about to give up. The kernel hacking would've been more enjoyable, though, if I'd setup a virtual machine where I could test new kernels, or if I'd setup up another computer so that I at least didn't have to reboot my development box every 5 minutes. :)

Until I can produce a real fix, I've simply increased the output buffer size to kmalloc's maximum value of 128 KiB. I doubt any game sends more sysex data than this anyway, so it should be sufficient for DOSBox purposes. If anyone else would like to use this kludge, all you need to do is edit sound/core/seq/seq_memory.c in the kernel tree and change static int output_buffer_size = PAGE_SIZE; to use 131072 instead of the PAGE_SIZE constant.

The kludge works for me, so now I'll just sit back and relax while I wait for the ALSA team to comment on my findings. I'm not writing a real fix until I know if they intend to push it upstream.
User avatar
hakonrk
Newbie
 
Posts: 30
Joined: 2004-2-13 @ 09:40

Re: SysEx problems in some Sierra games

Postby hakonrk » 2006-8-21 @ 12:16

Got a reply from one of the ALSA developers:
The sequencer interface is designed to send small messages asynchronously in real time (or at specified scheduled times). Not doing any bandwidth management is part of the current design.
[...]
You said you're trying to fix DOSBox. If it emulates a hardware MIDI interface, it shouldn't accept MIDI data at a faster rate than a real MIDI interface would.

What would be a reasonable maximum bandwidth? Wouldn't that depend on both the interface and the synth? If it's not possible to get status on the output buffer from the kernel, I don't see any fail-safe solution.
User avatar
hakonrk
Newbie
 
Posts: 30
Joined: 2004-2-13 @ 09:40

Re: SysEx problems in some Sierra games

Postby Kippesoep » 2006-8-21 @ 12:28

Not entirely sure where I got this, but I seem to recall that the MIDI spec says that MIDI data is supposed to be transmitted at 31,250 bps.
My site: Ramblings on mostly tech stuff.
User avatar
Kippesoep
Oldbie
 
Posts: 1015
Joined: 2005-5-04 @ 22:36
Location: Netherlands

Re: SysEx problems in some Sierra games

Postby hakonrk » 2006-8-21 @ 13:03

Looks like 24 kbps is the maximum on my CM-32L connected to a SB Live.
User avatar
hakonrk
Newbie
 
Posts: 30
Joined: 2004-2-13 @ 09:40

Re: SysEx problems in some Sierra games

Postby Qbix » 2006-8-21 @ 14:17

we would have to keep track on how long ago the last midi sysex message was send before sending a new one ?
Water flows down the stream
How to ask questions the smart way!
User avatar
Qbix
DOSBox Author
 
Posts: 10681
Joined: 2002-11-27 @ 14:50
Location: Fryslan

Next

Return to DOSBox Development

Who is online

Users browsing this forum: No registered users and 2 guests