Dune "HERAD" Ad Lib Music Hacking

Getting old DOS games working. (DOSBox topics belong in DOSBox areas below, not here).

Re: Dune "HERAD" Ad Lib Music Hacking

Postby binarymaster » 2017-4-09 @ 20:57

So, here is my development branch for HERAD format support in AdPlug:
https://github.com/adplug/adplug/tree/herad-dev

Almost everything works, but there are still some bugs with note transposition macro and strange AGD sounding at different places. Hope you can help out with that, since I'm stuck for a week.

Replayer code is here:
https://github.com/adplug/adplug/blob/h ... rc/herad.h
https://github.com/adplug/adplug/blob/h ... /herad.cpp
by Stas'M
User avatar
binarymaster
Newbie
 
Posts: 55
Joined: 2010-3-07 @ 19:03

Re: Dune "HERAD" Ad Lib Music Hacking

Postby synamax » 2017-4-11 @ 07:25

binarymaster wrote:So, here is my development branch for HERAD format support in AdPlug:
https://github.com/adplug/adplug/tree/herad-dev

Almost everything works, but there are still some bugs with note transposition macro and strange AGD sounding at different places. Hope you can help out with that, since I'm stuck for a week.

Replayer code is here:
https://github.com/adplug/adplug/blob/h ... rc/herad.h
https://github.com/adplug/adplug/blob/h ... /herad.cpp


For MegaRace, the transpose macro is different then Dune and KGB. After messing around with the DOSBox Debugger, I finally figured out how the transpose macro for HERAD 2 works! For MegaRace, the subroutine is found at 3B08:068B or sub_058B in the DFADP.HSQ driver.

Every time the driver grabs the transpose value, it subtracts the value by 31 then compares that value with 60. If the value is greater than 60, the transpose macro value is added to the MIDI pitch value. But if the value is less than 60, the MIDI pitch is overridden and that subtracted value + 18 becomes the new MIDI pitch.

For example the song NewSan uses a transpose value of 0x6B for it's snare instrument.

6B - 31 = 3A
3A < 60


If the value is less than 60, then 18 is added to the subtracted transpose value and that becomes the new MIDI pitch for the instrument.

3A + 18 = 52 = Final MIDI pitch

If the value is more than 60, then the value just modifies the MIDI pitch directly. In this example, we'll use NewSan's hihat instrument, which is the very last instrument in the file. The transpose macro value is 2E and the MIDI pitch is 44.

2E - 31 = FD
FD > 60


Since FD is greater than 60, the original transpose value is added to the MIDI pitch, giving us our final transposed pitch.

44 + 2E = 72 = Final MIDI Pitch

I'm not exactly sure how HERAD v1 handles the transpose value since I haven't reverse engineered Dune or KGB's drivers yet but I'm confident it's much simpler and acts like a adding the value to the MIDI pitch. Hope this helps!

Binarymaster, is it possible for you to build a binary of AdPlug with HERAD beta support? I would very much like to try it out!

Also, just curious, how did you come up with 51968 for the timer?
User avatar
synamax
Newbie
 
Posts: 69
Joined: 2016-9-07 @ 04:57

Re: Dune "HERAD" Ad Lib Music Hacking

Postby binarymaster » 2017-4-11 @ 12:19

synamax wrote:For MegaRace, the transpose macro is different then Dune and KGB. After messing around with the DOSBox Debugger, I finally figured out how the transpose macro for HERAD 2 works! For MegaRace, the subroutine is found at 3B08:068B or sub_058B in the DFADP.HSQ driver.

I'm not exactly sure how HERAD v1 handles the transpose value since I haven't reverse engineered Dune or KGB's drivers yet but I'm confident it's much simpler and acts like a adding the value to the MIDI pitch. Hope this helps!

Commited changes:
https://github.com/adplug/adplug/commit ... e2330b9718

Yep, this solved transposition issue in HERAD v2 songs, many thanks!

But Dune/KGB (version 1) songs still have the issue, looks like it isn't so simple as we would want. 8-O

synamax wrote:Binarymaster, is it possible for you to build a binary of AdPlug with HERAD beta support? I would very much like to try it out!

Here you go, attached. ;)

synamax wrote:Also, just curious, how did you come up with 51968 for the timer?

Well, I recorded some samples in Sound Forge, while playing your NEWPAGA song in AdPlug - one recording for DRO file and another for HERAD v2 file, then I calculated the difference between lengths and the coefficient, then I multiplied the coefficient and old timer division value, and rounded it to nearest beautiful hex number :lol: (51968 is 0xCB00).

Also there was alternative value, from old fmplayer source (forgot where I got it) in fmt_cryo.c, which is equal to 51650.
Attachments
in_adlib.7z
AdPlug Winamp beta 2017-04-11 with HERAD
(188.21 KiB) Downloaded 23 times
by Stas'M
User avatar
binarymaster
Newbie
 
Posts: 55
Joined: 2010-3-07 @ 19:03

Re: Dune "HERAD" Ad Lib Music Hacking

Postby synamax » 2017-4-12 @ 06:45

Wow, thanks so much!! You don't know how long I've been waiting for this. :) This is definitely going to make playback so much easier!

Glad I can help with the transposition issue with HERAD v2! I'll look into what's going on with HERAD 1's transpose macro this weekend.

We're definitely off to a great start, but we still have a lot to figure out. First off, I feel that the timer is slightly off. I can make some test DRO/SDB files so we can get a more accurate timer rate.

Also, I haven't documented how pitch bends work, but I do know they use a table in the header of the driver. The guitar solo in Skyholder sounds weird because we don't know exactly how the pitch bends are handled. I attached an MP3 so you can hear the original track and the one in AdPlug.

Finally, it's possible to play a song in MegaRace without the 0x32 track offset in the header of the file. If I remove it and load the song in AdPlug, the song won't play.

Excellent work so far, binarymaster!!
Attachments
paga_wrong_pitch.zip
Skyholder guitar solo comparison
(1.2 MiB) Downloaded 26 times
User avatar
synamax
Newbie
 
Posts: 69
Joined: 2016-9-07 @ 04:57

Re: Dune "HERAD" Ad Lib Music Hacking

Postby Jepael » 2017-4-12 @ 20:23

Not sure what is really meant with that timer/tempo value, but will the following help with tempo calculation?

The game sets PIT timer divisor to 5957, which amounts to about 200.299 Hz timer interrupt.
My wild guess is the music playing routine is also called from this timer interrupt like on Dune.

Also, I don't know what these typical tempo values are,
but I can't help noticing 256 * 200.299 Hz is about 51276.5 which is pretty close to values you are describing.
Jepael
Oldbie
 
Posts: 1195
Joined: 2005-6-15 @ 19:28
Location: Finland

Re: Dune "HERAD" Ad Lib Music Hacking

Postby synamax » 2017-4-13 @ 06:14

Jepael wrote:Not sure what is really meant with that timer/tempo value, but will the following help with tempo calculation?

The game sets PIT timer divisor to 5957, which amounts to about 200.299 Hz timer interrupt.
My wild guess is the music playing routine is also called from this timer interrupt like on Dune.

Also, I don't know what these typical tempo values are,
but I can't help noticing 256 * 200.299 Hz is about 51276.5 which is pretty close to values you are describing.


That makes more sense. I know that setting the song speed to 1 equates to 500 bpm, because a quarter note at that speed is 120 ms, which means that a measure is 0.48 seconds or 480 ms long. A tick at that speed equals to 5 ms. Of course things get out of sync after a short while so to test the values, I made a HA2 file for MegaRace, a SDB for Dune/KGB, a DRO recording and a WAV recording, so you can test it out to confirm you're getting the correct timing.

The rhythm is 1 whole note, 2 half notes, 4 quarter notes, 8 eighth notes, 16 sixteenth notes, 32 thirty-secondth notes, 96 MIDI tick notes, and finally 4 quarter notes again.
Attachments
speed1.zip
(863.94 KiB) Downloaded 32 times
User avatar
synamax
Newbie
 
Posts: 69
Joined: 2016-9-07 @ 04:57

Re: Dune "HERAD" Ad Lib Music Hacking

Postby Jepael » 2017-4-13 @ 09:08

I don't think things like BPM are any more meaningful since this is not a standard MIDI file we are playing.
Different (DOS) players will anyway use different timer interrupt rates to play standard midi files at the tempo set in the MIDI file. The timer rate can be fixed or even change based on tempo.

But in this case, this is a custom format, where the player timer rate is fixed regardless of tempo set in the file. Another game might use another fixed timer rate.

So it will sound correct when player routine is called 200.299 times per second regardless of song tempo value. Assuming the plugin player code is identical to original code.

I recall having tempo/pitch problems with adplug winamp plugin. I think the pitch was incorrrect unless the OPL emulator playback rate was set to 49716 Hz. And then LAA files were played back too fast, because they were played back with the generic MIDI/LAA/Sierra/??? player that assumed standard tempos, even if LAA playback code is supposed to be called with specific rate (about 473 Hz).

I have not used Winamp since ages so I don't know the current state, but please check if the playback sampling rate affects pitch and tempo.
Jepael
Oldbie
 
Posts: 1195
Joined: 2005-6-15 @ 19:28
Location: Finland

Re: Dune "HERAD" Ad Lib Music Hacking

Postby binarymaster » 2017-4-13 @ 09:42

synamax wrote:I'll look into what's going on with HERAD 1's transpose macro this weekend.

We're definitely off to a great start, but we still have a lot to figure out. First off, I feel that the timer is slightly off. I can make some test DRO/SDB files so we can get a more accurate timer rate.

Ok!

synamax wrote:Also, I haven't documented how pitch bends work, but I do know they use a table in the header of the driver. The guitar solo in Skyholder sounds weird because we don't know exactly how the pitch bends are handled. I attached an MP3 so you can hear the original track and the one in AdPlug.

Yep, this also needs to be fixed.

synamax wrote:Finally, it's possible to play a song in MegaRace without the 0x32 track offset in the header of the file. If I remove it and load the song in AdPlug, the song won't play.

This is just sanity check, because normal SDB files have first track offset equal to 0x32, and AGD is 0x52. If we remove this check, it may lead to false positive detection errors in plugin.

Jepael wrote:So it will sound correct when player routine is called 200.299 times per second regardless of song tempo value. Assuming the plugin player code is identical to original code.

Timer rate in the plugin code is based on the tempo, so it is not identical to original code. The frequency measures ticks per second, so for example, if frequency is equal to 1 Hz, then it will play each MIDI tick one time per second.

Jepael wrote:I think the pitch was incorrrect unless the OPL emulator playback rate was set to 49716 Hz.

I have not used Winamp since ages so I don't know the current state, but please check if the playback sampling rate affects pitch and tempo.

The latest plugin uses output frequency at 49716 Hz by default. Also, the sample rate doesn't affect pitch and tempo, because it uses resampling (as long as I know).
by Stas'M
User avatar
binarymaster
Newbie
 
Posts: 55
Joined: 2010-3-07 @ 19:03

Re: Dune "HERAD" Ad Lib Music Hacking

Postby synamax » 2017-4-13 @ 15:28

The 200.299 timer is finally starting to make sense to me. At HERAD speed setting 1, a tick is 5 ms, so multiply that by 200 and you have a whole second. Taking the song speed and multiply it by 5 equals the duration of a MIDI tick in ms. For example, Newsan is 3.875...3.875 x 5 = 19.375 ms.
User avatar
synamax
Newbie
 
Posts: 69
Joined: 2016-9-07 @ 04:57

Re: Dune "HERAD" Ad Lib Music Hacking

Postby Jepael » 2017-4-14 @ 17:27

binarymaster wrote:Timer rate in the plugin code is based on the tempo, so it is not identical to original code. The frequency measures ticks per second, so for example, if frequency is equal to 1 Hz, then it will play each MIDI tick one time per second.


OK. I am just interested if there is any reason it's played like any other MIDI file, instead of not played like the original game does?

I mean, it might not matter much, but for example DRO files have timing granularity of 1ms ticks, so it basically requires 1000Hz timer that increases time by 1ms, and
I bet there could be some timing issues if it is played differently, like with a 500 Hz timer that increases 2ms at a time, or 100Hz timer increasing time by 10ms every tick, if you see what I am after. Yeah, MIDI is different, and I may be just too fixed on the idea how things should be done exactly as they were done originally :)

binarymaster wrote:The latest plugin uses output frequency at 49716 Hz by default. Also, the sample rate doesn't affect pitch and tempo, because it uses resampling (as long as I know).


OK, that's great. For what I can tell, it might have been the XMMS or Audacious plug-in though.
Speaking of which, I see you might be the release engineer for the next AdPlug so do you have any information what's the best way to test Adplug/Adplay under Linux? Like compile the plugin for Audacious player? I'd like to help testing too, and I've loved AdPlug for well over a decade, and now that's on Github it opened an easy way of contributing as well..
Jepael
Oldbie
 
Posts: 1195
Joined: 2005-6-15 @ 19:28
Location: Finland

Re: Dune "HERAD" Ad Lib Music Hacking

Postby synamax » 2017-4-14 @ 23:03

I would definitely like to keep the HERAD AdPlug as accurate as possible. If it's possible to go with the 200.299 Hz timer, then that would be awesome, but if it's too difficult to implement then we'll skip it and go with the next closest thing. Using the seekbar and having the ablity to fast forward and rewind songs in Winamp is super useful, so I don't want to break that functionality, if it means having to sacrifice accuracy for user-friendly functionality.

So I did some more reverse engineering of the MegaRace driver. Pitch Bends are more complex than I thought...I'll update this post shortly with what I found.

---------------
UPDATE:

So, there's A LOT going on with pitch bends. Looks like both MIDI Pitch Bend events and macro-controlled Pitch Slides are controlled by the same subroutines. I tried to document as much as I could before real life got in the way.

I attached an ASM file that shows what's going on with the pitch bending. To follow along with the DOSBox Debugger, the breakpoint for the pitch bends is at 3B08:07DD. Make sure you have Sound Blaster Pro and No Sounds selected in the setup menu for MegaRace.
Attachments
pitchbend.zip
(2 KiB) Downloaded 28 times
User avatar
synamax
Newbie
 
Posts: 69
Joined: 2016-9-07 @ 04:57

Re: Dune "HERAD" Ad Lib Music Hacking

Postby synamax » 2017-4-15 @ 05:11

I might as well upload the whole DFADP.HSQ assembly file so that you guys gets the whole picture. It's still incomplete but the vast majority has been documented.

I must mention that because of a mistake, the offsets in the file are off by 0x100. For example, when the program calls 3B08:019B, the source code references it as 0x9B. Sorry for that mistake, but fortunately, it's easy to fix.
Attachments
dfadp.zip
MegaRace HERAD driver disassembly
(12.19 KiB) Downloaded 29 times
User avatar
synamax
Newbie
 
Posts: 69
Joined: 2016-9-07 @ 04:57

Re: Dune "HERAD" Ad Lib Music Hacking

Postby binarymaster » 2017-4-23 @ 14:34

Jepael wrote:Speaking of which, I see you might be the release engineer for the next AdPlug so do you have any information what's the best way to test Adplug/Adplay under Linux? Like compile the plugin for Audacious player? I'd like to help testing too, and I've loved AdPlug for well over a decade, and now that's on Github it opened an easy way of contributing as well..

Sorry for late answer, well, actually I'm working on many different projects, and currently I highly involved in one of them :)

Sometimes when I have no VS2015 IDE at hand, I'm using adplay-unix frontend for AdPlug debugging. It compiles well on UNIX-like systems and Windows (with Cygwin).
by Stas'M
User avatar
binarymaster
Newbie
 
Posts: 55
Joined: 2010-3-07 @ 19:03

Re: Dune "HERAD" Ad Lib Music Hacking

Postby binarymaster » 2017-5-09 @ 17:35

Well, now it calculates refresh rate more precisely:
https://github.com/adplug/adplug/commit ... 3e393b64f0

Also tried to make constant refresh rate at 200.299 Hz, but have no idea how to do it correctly.
by Stas'M
User avatar
binarymaster
Newbie
 
Posts: 55
Joined: 2010-3-07 @ 19:03

Re: Dune "HERAD" Ad Lib Music Hacking

Postby Jepael » 2017-5-09 @ 20:02

I did some more research, and wSpeed is just "time increment" units at which rate to handle the midi events, and calling the playback routine makes a "time decrement" of 0x100 units. Or actually, just decrease the high byte, and if result is not negative it skips the midi processing.

So in fact the midi processing rate on average really is 200.299 * 256 / wSpeed which you have calculated correctly, but it must be calculated in time units quantized to 200.299 Hz so it is much simpler to just keep track of fractional time directly in wSpeed units.

Code: Select all
int16_t wTime = 0;
void play(void)
{
    wTime = wTime - 256;
    if (wTime < 0)
    {
        wTime = wTime + wSpeed;
        process_midi_events();
    }
}


So, for different wSpeed values:
0x100 = call each and every time without any skipping (1x tempo)
0x200 = call exactly every other time (0.5x tempo)
0x180 = call two times and skip one, repeating pattern (2/3 tempo)
0x300 = call once, skip two (1/3 tempo)

Basically I'd have the timer to call play() at constant 200.299 rate and let play() handle the tempo/speed calculations like the driver does.
The actual MIDI event processing will decrease the MIDI delta time values by one.

For best results, you can use the following if helpful: one native OPL chip sample is 24 PIT timer ticks, and the play() should be called every 5957 PIT timer ticks. So that's 248.2083 OPL samples between play() calls.

For PIT tick rate, 14318180/12 should be close enough approximation.
Jepael
Oldbie
 
Posts: 1195
Joined: 2005-6-15 @ 19:28
Location: Finland

Re: Dune "HERAD" Ad Lib Music Hacking

Postby binarymaster » 2017-5-09 @ 20:37

Wow, it works pretty well, thanks for the code sample!

Commited: https://github.com/adplug/adplug/commit ... 60a2ae155c

UPD: moved event processing cycle to new procedure:
https://github.com/adplug/adplug/commit ... 8863c488b4
by Stas'M
User avatar
binarymaster
Newbie
 
Posts: 55
Joined: 2010-3-07 @ 19:03

Re: Dune "HERAD" Ad Lib Music Hacking

Postby synamax » 2017-6-06 @ 09:02

Quick update on how pitch bends work in HERAD. It's a bit complicated since pitch bends for both MIDI events and macro slides use the same functions. I can confirm that the start of processing the pitch bend begins by taking the current MIDI pitch and subtracting it by 0x18. Then that value is divided by 0xC.

There's a lot of rotating bits to the right and getting two's compliment. There's also some XLAT instructions where it grabs values from the tables at either 0x83, 0x84, or 0x90 (0x183, 0x184, or 0x190 when loaded in memory). I think 0x83/0x84 is used for the fine tune pitch bends and 0x90 is used for the coarse pitch bends.

I'll post more info later when I get a chance :)
User avatar
synamax
Newbie
 
Posts: 69
Joined: 2016-9-07 @ 04:57

Re: Dune "HERAD" Ad Lib Music Hacking

Postby synamax » 2017-7-04 @ 19:05

Sorry for being inactive for a while, I've been really busy with real life lately.

As you can see, the Pitch Bend subroutine is the most complex task handled by the HERAD driver. Making this flowchart made things a lot easier to understand though.

Image

Because the pitch bend subroutine handles both MIDI Pitch Bends as well as the Pitch Slide Macros, it gets called a lot and there's a lot of instructions that it goes through that seem like they don't do anything at first but they actually do have a purpose.

I did make a small discovery. The Pitch bend tables at 0x83, 0x84 are just for the Pitch Slides. I'm not sure if that applies to the Coarse Pitch Bends as well but it definitely the case when dealing with fine tune pitch bends.

-------------------------------
The subroutine starts by grabbing the MIDI pitch of the note and subtracts 18 from it. We'll use middle C (3C) as an example. All numbers are in hex.

3C - 18 = 24

Then it is divided by C.

24 / C = 3

This is our modified MIDI pitch.

Once that is done, then the subroutine compares the Pitch Slide Range Flag (0x21 in the instrument definition) to zero. If the byte is zero, then the subroutine proceeds with fine pitch bending, but if the byte is one it jumps to the subroutine for coarse pitch bending.

Now, this is where it starts getting complicated, as you can either Bend Up or Bend Down or Coarse Bend Up or Coarse Bend down, but they all start the same and that's by determining where the pitch bend is going. It does this by taking either the Pitch Slide Range (0x24 in the instrument definition) or the MIDI Pitch Bend value and subtracts it by 40. If the number is below 40, it will result in a signed integer. For example: 20 (pitch bend value) - 40 = FFE0. This means we're doing a downwards pitch bend. If we're doing an upward pitch bend with a value of 60 for example, then the result will not be signed and the subroutine proceeds with going upwards.
Last edited by synamax on 2017-8-01 @ 07:54, edited 1 time in total.
User avatar
synamax
Newbie
 
Posts: 69
Joined: 2016-9-07 @ 04:57

Re: Dune "HERAD" Ad Lib Music Hacking

Postby synamax » 2017-7-05 @ 09:39

Figured out how Dune handles transposing! It's actually really, really simple. Instead of a dedicated transpose subroutine like what MegaRace uses, Dune simply adds the transpose byte to the MIDI Pitch and writes the new pitch to memory and the OPL chip.

Here's a twitch video of me working in DOSBox's Debugger figuring this out:

https://www.twitch.tv/videos/156741768

I verified that the instructions are the same in both the demo and the floppy version of the game.
User avatar
synamax
Newbie
 
Posts: 69
Joined: 2016-9-07 @ 04:57

Re: Dune "HERAD" Ad Lib Music Hacking

Postby synamax » 2017-8-01 @ 07:55

I updated the post with the pitch bend flowchart with a new image as the site that originally hosted it changed their image hosting options. We're slowly but surely getting there!
User avatar
synamax
Newbie
 
Posts: 69
Joined: 2016-9-07 @ 04:57

PreviousNext

Return to DOS

Who is online

Users browsing this forum: No registered users and 2 guests