VOGONS

Common searches


Dune "HERAD" Ad Lib Music Hacking

Topic actions

Reply 120 of 167, by binarymaster

User metadata
Rank Newbie
Rank
Newbie
synamax wrote:

I updated the post with the pitch bend flowchart with a new image as the site that originally hosted it changed their image hosting options. We're slowly but surely getting there!

Indeed, thanks!

Important information about writing looped songs in HERAD format.

Yesterday I just figured out that each musical pattern (4 beats, for example) should use exactly 96 MIDI ticks to fit into one measure. Then, you'll need to modify wSpeed value to set correct tempo. After these manipulations it will be a lot easier to set proper loop start and end measures.

Thanks goes to Jepael for his HERAD DOS driver shell, it helped so much!

For now I'm trying to implement looping support in my AdPlug HERAD replaying code. 😀

by Stas'M

Reply 121 of 167, by opl2

User metadata
Rank Newbie
Rank
Newbie

Yes, thanks to you all for all your hard work. I have become very interested in this project, and am working on an Herad SDB -> VGM converter.

I've disassembled all of the drivers (dune/kgb/megarace) and have the bend code working in my (almost perfect?) converter. Here's how the pitch bending part gets done (Ruby code):

def play_note(note, octave, bend_amount = current_bend)
case current_instrument[:macro_slide_type]
when 0 # fine
apply_fine_slide(note, octave, bend_amount)
when 1 # coarse
apply_coarse_slide(note, octave, bend_amount)
end
end


def apply_coarse_slide(note, octave, bend_amount)
if (bend_amount - BEND_CENTER) < 0
apply_coarse_slide_down(note, octave, bend_amount)
else
apply_coarse_slide_up(note, octave, bend_amount)
end
end


def apply_fine_slide(note, octave, bend_amount)
if (bend_amount - BEND_CENTER) < 0
apply_fine_slide_down(note, octave, bend_amount)
else
apply_fine_slide_up(note, octave, bend_amount)
end
end


def apply_fine_slide_down(note, octave, bend_amount)
amount = BEND_CENTER - bend_amount
amount_lo = (amount >> 5)
amount_hi = (amount << 3) & 0xFF
note -= amount_lo

if note < 0
note += 12
octave -= 1
end

if octave < 0
note = 0
octave = 0
end

value = FINE_BEND_LOOKUP[note] # NOTE: in assembly, 0x183 lookup
value *= amount_hi
value >>= 8

write_note_bend(note: note, octave: octave, detune: -value)
end


def apply_fine_slide_up(note, octave, bend_amount)
amount = bend_amount - BEND_CENTER
amount_lo = (amount >> 5)
amount_hi = (amount << 3) & 0xFF
note += amount_lo

if note >= 12
note -= 12
Show last 85 lines
    octave += 1
end

value = FINE_BEND_LOOKUP[note+1] # NOTE: in assembly, 0x184 lookup
value *= amount_hi
value >>= 8

write_note_bend(note: note, octave: octave, detune: value)
end


def apply_coarse_slide_down(note, octave, bend_amount)
amount = BEND_CENTER - bend_amount
note -= amount / 5

if note < 0
note += 12
octave -= 1
end

if octave < 0
note = 0
octave = 0
end

offset = amount % 5
offset += 5 if note >= 6
value = COARSE_BEND_LOOKUP[offset]

write_note_bend(note: note, octave: octave, detune: -value)
end


def apply_coarse_slide_up(note, octave, bend_amount)
amount = bend_amount - BEND_CENTER
note += amount / 5

if note >= 12
note -= 12
octave += 1
end

offset = amount % 5
offset += 5 if note >= 6
value = COARSE_BEND_LOOKUP[offset]

write_note_bend(note: note, octave: octave, detune: value)
end



FINE_BEND_LOOKUP = [
0x13, # ds:0183 start offset for downwards slide
0x15, # ds:0184 start offset for upwards slide
0x15,
0x17,
0x19,
0x1A,
0x1B,
0x1D,
0x1F,
0x21,
0x23,
0x24,
0x25
]


COARSE_BEND_LOOKUP = [
0x00, # ds:0190
0x05,
0x0A,
0x0F,
0x14,
0x00, # ds:0195
0x06,
0x0C,
0x12,
0x18
]


BEND_CENTER = 0x40

Then every song tick, IF there are no other commands to be processed on that channel, it'll do:

  def apply_slide
return if current_slide_duration.zero?

@current_slide_duration -= 1
@current_bend += current_instrument[:macro_slide_amount]

return if (current_note & 0x7F).zero?

note = (current_note - 0x18) & 0xFF
play_note(note % 12, note / 12)
end

Hope this helps!

Reply 122 of 167, by opl2

User metadata
Rank Newbie
Rank
Newbie

I've attached the VGM output of the conversion of all SDB files I have. Most of them sound spot on(?) and I thought I was pretty much done, but then when listening closely to the KGB files I noticed that there's still something off in "Gorbi" (the "V" in Gorbachev), and somewhere near the 30 second mark in "Burokrat" something very strange is happening to some of the "dissonant" notes -- something I don't hear in the original.

I'll keep looking for this bug, but if anyone has an idea just from listening do let me know 😀

Attachments

  • Filename
    herad-vgm-output-20170804.zip
    File size
    357.82 KiB
    Downloads
    93 downloads
    File comment
    VGM files created by converter
    File license
    Fair use/fair dealing exception

Reply 123 of 167, by Jepael

User metadata
Rank Oldbie
Rank
Oldbie

So essentially you have rewritten the playing routine and made it output VGM files?
I ran the original playback driver and made it log the writes into RDOS RAW files, so that leaves no room for errors in playback data or timing when playing it back on real hardware.
Though, while VGM timing is only accurate to 44100Hz timing, it's good enough for cross-platform playback and emulation, plus it supports looping.

opl2 wrote:

I've attached the VGM output of the conversion of all SDB files I have. Most of them sound spot on(?) and I thought I was pretty much done, but then when listening closely to the KGB files I noticed that there's still something off in "Gorbi" (the "V" in Gorbachev), and somewhere near the 30 second mark in "Burokrat" something very strange is happening to some of the "dissonant" notes -- something I don't hear in the original.

Would not that depend on the player or chip emulation and how it is being used?

Also what most people don't seem to realize is that when a real program is sending data to OPL chip, it has to keep a certain delay between writes, while on many playback routines using emulation just send out all register writes with zero delay between them, so it can obviously sound different.

The delay would be such that the rate at which registers can be written is about 38000 per second, for OPL2 chip.
So in theory, to get that approximately right for emulation purposes, it would either require the VGM file to have 1 sample delays between register writes, or the player to emulate this.

Reply 124 of 167, by opl2

User metadata
Rank Newbie
Rank
Newbie
Jepael wrote:

So essentially you have rewritten the playing routine and made it output VGM files?

Yes, with the help of SynaMax's amazing research. I wrote a set of tools that apply multiple transformation passes (song, track pre-parsing and generating of events including which 'tick' they were fired on) to get a better understanding of how HERAD works. Mostly for fun and to learn something, but I have some other things in mind.

Jepael wrote:

I ran the original playback driver and made it log the writes into RDOS RAW files, so that leaves no room for errors in playback data or timing when playing it back on real hardware.
Though, while VGM timing is only accurate to 44100Hz timing, it's good enough for cross-platform playback and emulation, plus it supports looping.

You're absolutely right, though since we know the PIC rate and HERAD's divider we should be able to pretty accurately know at when the commands are being sent to the OPL chip. I'm familiar with the timings of the OPL2 chip itself (I have a project where I have one controlled over USB on a PCB), and you're right, I didn't account for those.

I did think about it but the delays are in the nanosecond ranges, not milliseconds, and I didn't consider that with a larger amount of writes the actual "write time" might spill over into the next millisecond.

I am a little confused though, isn't the resolution of the DOSBOX RAW format not 1ms? vs 1000/441000~=0.023ms resolution of VGM?. The playback rate of VGM can set in the header as well. For preservation purposes it should be just as good as DRO?

Jepael wrote:

Would not that depend on the player or chip emulation and how it is being used?

Yes, but if I play back the "original dosbox recording" from vgmrips.net against the same emulator, there are slight differences, which is what I was getting at. I probably should have clarified that.

Jepael wrote:

Also what most people don't seem to realize is that when a real program is sending data to OPL chip, it has to keep a certain delay between writes, while on many playback routines using emulation just send out all register writes with zero delay between them, so it can obviously sound different.

Indeed, although I do believe this is more of an issue with the OPL2 then it is with the OPL3 chips. My hardware board's firmware takes the timings into account, and no doubt it sounds different. The op-amps/filter stages used (different on each sound card back in the day as well) also greatly influence the outcome, and I don't know if most emulators emulate that (I'm using Nuked, and it doesn't sound like it does).

Besides that, I'm not 100% familiar with what the KGB music SHOULD sound like, so I better go find some actual recordings 😀

Jepael wrote:

The delay would be such that the rate at which registers can be written is about 38000 per second, for OPL2 chip.
So in theory, to get that approximately right for emulation purposes, it would either require the VGM file to have 1 sample delays between register writes, or the player to emulate this.

Ha, of course! Thanks! I'll try this, I'm very curious to see what happens. As for the player emulating it when writing to the emulated chip instead of the real one, I think that's a little beyond the scope of my little project (and currently the timings are dealt with in firmware).

Reply 125 of 167, by Jepael

User metadata
Rank Oldbie
Rank
Oldbie
opl2 wrote:

You're absolutely right, though since we know the PIC rate and HERAD's divider we should be able to pretty accurately know at when the commands are being sent to the OPL chip. I'm familiar with the timings of the OPL2 chip itself (I have a project where I have one controlled over USB on a PCB), and you're right, I didn't account for those.

I did think about it but the delays are in the nanosecond ranges, not milliseconds, and I didn't consider that with a larger amount of writes the actual "write time" might spill over into the next millisecond.

I am a little confused though, isn't the resolution of the DOSBOX RAW format not 1ms? vs 1000/441000~=0.023ms resolution of VGM?. The playback rate of VGM can set in the header as well. For preservation purposes it should be just as good as DRO?

I used RAW as in RDOSRAW, not DosBox DRO.

DRO has indeed 1ms timebase, so for instance player routine being called at 60Hz or 70Hz is not an integer amount of milliseconds so there's always 1ms jittering possible, which would otherwise be not a huge issue, but it might divide a bunch of writes that really happens consecutively in a timer interrupt to happen before and after the 1ms tick, so they are processed at different ticks really.

RAW on the other hand, uses PIT timings, in fact it's ideal of the concept most games play their music, you set the PIT timer divisor and then ticks of interrupts will be the RAW timebase.

Reply 126 of 167, by binarymaster

User metadata
Rank Newbie
Rank
Newbie
opl2 wrote:

I've attached the VGM output of the conversion of all SDB files I have. Most of them sound spot on(?) and I thought I was pretty much done, but then when listening closely to the KGB files I noticed that there's still something off in "Gorbi" (the "V" in Gorbachev), and somewhere near the 30 second mark in "Burokrat" something very strange is happening to some of the "dissonant" notes -- something I don't hear in the original.

I'll keep looking for this bug, but if anyone has an idea just from listening do let me know 😀

Thanks for your work!

Indeed, there are some dissonant notes in short periods. The same bug I have in my AdPlug replaying code, but those notes sound incorrect all the time in "Burokrat", this can be due to improper implementation of pitch bends and/or slides.

Now I'm trying to understand Ruby code... never ever did it. 🤣

Jepael wrote:

Would not that depend on the player or chip emulation and how it is being used?

Nope, it is bug in player (converter) code, I've tested the output with Nuked OPL3.

BTW, my code doesn't have bug with "Gorbachev" pronunciation. 😀

Also I think it would be ideal to have reference data from original drivers in AdPlug REF format - it's very simple plain text format, which stores chip writes and delays. Having such files would allow to test "chip-write-perfect" (like pixel-perfect) output. 😉

See one small file for example:
https://github.com/adplug/adplug/blob/master/test/BEGIN.ref

Typical OPL2 test reference data:

init // start of REF file
bd <- 20 // write 0x20 to register 0xBD
r240.00 // set refresh to 240 Hz and wait for next call

For OPL3 songs it uses setchip commands:

4 <- 6 // write 0x06 to register 0x04 (chip #0)
setchip(1) // switch to high bank
5 <- 1 // write 0x01 to register 0x05 (chip #1)
4 <- 0 // write 0x00 to register 0x04 (chip #1)
setchip(0) // switch to low bank
bd <- 0 // write 0x00 to register 0xBD (chip #0)
8 <- 0 // write 0x00 to register 0x08 (chip #0)
1 <- 20 // write 0x20 to register 0x01 (chip #0)

by Stas'M

Reply 127 of 167, by opl2

User metadata
Rank Newbie
Rank
Newbie

Hi binarymaster,
Thanks for all your help and all work on the AdPlug plugin!

binarymaster wrote:

Indeed, there are some dissonant notes in short periods. The same bug I have in my AdPlug replaying code, but those notes sound incorrect all the time in "Burokrat", this can be due to improper implementation of pitch bends and/or slides.

Yes, but somehow they're more pronounced in my conversion. It's probably a bug I haven't tracked down yet.

binarymaster wrote:

Now I'm trying to understand Ruby code... never ever did it. 🤣

It should read just like "pseudo"-code 😀 It's one of my languages of choice for quickly hammering out ideas and see if they work. Let me know if you have any questions!

binarymaster wrote:
Jepael wrote:

Would not that depend on the player or chip emulation and how it is being used?

Nope, it is bug in player (converter) code, I've tested the output with Nuked OPL3.

BTW, my code doesn't have bug with "Gorbachev" pronunciation. 😀

Very cool, I'll have to study some of the differences. To be honest I haven't built the herad-dev branch yet to listen to it - will do so soon!

binarymaster wrote:

Also I think it would be ideal to have reference data from original drivers in AdPlug REF format - it's very simple plain text format, which stores chip writes and delays. Having such files would allow to test "chip-write-perfect" (like pixel-perfect) output. 😉

Would the RDOS RAW format Jepael mentioned not be exactly this, except in binary?

Reply 128 of 167, by opl2

User metadata
Rank Newbie
Rank
Newbie

In case anyone is interested, here's my (so far) annotated disassembly of the Dune (CD version) driver.

Some notable differences from MegaRace:
- global volume handling
- aftertouch/channel pressure events
- root note offset (like SynaMax describes on the wiki)

Attachments

  • Filename
    dune-annotated.asm.txt
    File size
    42.57 KiB
    Downloads
    90 downloads
    File comment
    DuneCD adlib driver disassembly with notes
    File license
    Fair use/fair dealing exception

Reply 129 of 167, by binarymaster

User metadata
Rank Newbie
Rank
Newbie
opl2 wrote:

Would the RDOS RAW format Jepael mentioned not be exactly this, except in binary?

Yep, it would be good too, since it have proper timing, and can be easily played with many players.

(also I can convert them to REF 😎 )

by Stas'M

Reply 130 of 167, by Jepael

User metadata
Rank Oldbie
Rank
Oldbie

I will consider REF logging directly from the tool but I have no time frame for that. Conversion from RAW is good but text based REF with comments about init part or looping parts might be useful.

And another thing, since Dune supports Adlib Gold so we should support Gold emulation too if nobody else does.
Does anyone of you know how Adlib Gold Surround module processes the audio? Or have Adlib Gold card with or without the Surround Module to verify some things?

I have made a somewhat crude surround module emulation, and while the output is approximately what I would expect with the delays and gains I've tested it, but apparently there are some unknowns in this.

First of all, the surround module takes in mono audio only, so I suspect OPL chip original left and right channels are summed together into mono before reaching Surround Module input.
And since the module is optional, I suspect the Surround Module output is summed back into original left and right channels for output, as I can't see any feature that would switch the FM audio away from direct OPL output to Surround Module output, and that would also remove any stereo panning set at the OPL chip.

So can you give me any pointers about this or people to talk to?
I'd be happy if someone would borrow an Adlib Gold card to me for reverse engineering, but I don't think that's going to happen as it is so rare card.
At least there is one option, if we can play for example Adlib Gold versions of Dune songs into audio with OPL emulation (Nuked maybe?), I can post process with Surround Module emulation and compare with recordings from real card (those should be available on youtube).

(Note: I'm emulating the digital Yamaha delay chip, not the Philips analog chip... yet).

Reply 131 of 167, by binarymaster

User metadata
Rank Newbie
Rank
Newbie

One interesting fact about AdLib Gold music from Dune: there are only two songs (Worm Intro and Wormsuit) that uses features of OPL3 chip - 18 channels, extended wave shapes, and panning.

Other AGD songs are just cloned from SDB and doesn't use these features. They only have different parameters for Surround chip.

There are recordings from VGMPF using original AdLib Gold 1000:
www.vgmpf.com/Wiki/index.php?title=Dune_(DOS)#Recording

Also I attached captured VGM files with modified AGD driver (without Surround chip writes).

Attachments

  • Filename
    AGD_VGM.7z
    File size
    47.47 KiB
    Downloads
    104 downloads
    File license
    Fair use/fair dealing exception

by Stas'M

Reply 132 of 167, by binarymaster

User metadata
Rank Newbie
Rank
Newbie
opl2 wrote:

I've disassembled all of the drivers (dune/kgb/megarace) and have the bend code working in my (almost perfect?) converter.

Hope this helps!

This helps indeed, I just translated the code into C and whoosh... It works pretty good! No more dissonant notes! 😁

The commit:
https://github.com/adplug/adplug/commit/284af … 876082ba62c8756

However this introduced some regressions (tested with Winamp plugin via Nuked OPL3):
1. Strange hiss sound at the beginning of "NewSan"
2. High frequency tone at the end of "Gorbi"

Also, here is the list of knowns issues, that are not solved yet:
1. Splash-cymbal-like sound in the beginning of Gorbi (at 0:23) is missing both in my replayer code and in opl2's converter
2. AGD (my code): First notes of Worm Intro sounds a bit high (maybe the transposition code differs from SDB)
3. AGD / Worm Intro: Drum at the beginning (at 0:22) is missing expression and too quiet (with SDB it's okay)
4. AGD / Worm Intro: OPL3-specific sound effects in the middle (at 0:58) are too loud

by Stas'M

Reply 133 of 167, by opl2

User metadata
Rank Newbie
Rank
Newbie
binarymaster wrote:
This helps indeed, I just translated the code into C and whoosh... It works pretty good! No more dissonant notes! :D […]
Show full quote

This helps indeed, I just translated the code into C and whoosh... It works pretty good! No more dissonant notes! 😁

The commit:
https://github.com/adplug/adplug/commit/284af … 876082ba62c8756

However this introduced some regressions (tested with Winamp plugin via Nuked OPL3):

Awesome, thanks! I'll have a closer look/listen later and see if anything stands out.

It might not fix any of these but I noticed a fix I added later to my code that isn't in the sample code: https://github.com/thatdutchguy/adplug/commit … 0fa441ab42af64e (not sure you want pull requests for your branch)

Reply 134 of 167, by binarymaster

User metadata
Rank Newbie
Rank
Newbie
opl2 wrote:

It might not fix any of these but I noticed a fix I added later to my code that isn't in the sample code: https://github.com/thatdutchguy/adplug/commit … 0fa441ab42af64e (not sure you want pull requests for your branch)

Thanks for the fix! Pull requests are acceptable, I merged it. 😊

When the replayer code will be complete, I'll squash commits from this branch into one single commit, and push into master.

opl2 wrote:

Awesome, thanks! I'll have a closer look/listen later and see if anything stands out.

I attatched the Winamp plugin built from latest source / herad-dev branch. 😉

Attachments

  • Filename
    in_adlib.7z
    File size
    188.44 KiB
    Downloads
    89 downloads
    File license
    Fair use/fair dealing exception

by Stas'M

Reply 135 of 167, by opl2

User metadata
Rank Newbie
Rank
Newbie
binarymaster wrote:

I attatched the Winamp plugin built from latest source / herad-dev branch. 😉

Thanks, but I'm not on windows so I don't think that'll work 🙁. I've been using adplay-unix with SDL to try out the plugins (and to play VGM). I really should look for a front-end for that.

I have worked a bit more on my converter, and I think it's complete, though I've been listening to everything on laptop speakers lately and still need to do a proper headphone session. You can find the main logic at https://gist.github.com/thatdutchguy/2106d686 … a8a73cb7caf6455

I still have a bunch of cleaning up to do and will release the converter (open source) soon, but I'm currently porting the engine to C for a different purpose (more on that soon too 😉).

I attached the latest set of VGM conversions (includes looping) for comparison.

Also, you HAVE an Adlib Gold 1000? That's crazy! I don't think I've ever seen one in the wild. Back in the day I had a Gravis UltraSound MAX + MediaVision Pro-Audio Spectrum 16 but an overseas move made me get rid of all that years ago.

Attachments

  • Filename
    herad-vgm-output-20170815.zip
    File size
    471.81 KiB
    Downloads
    84 downloads
    File comment
    VGM files created by converter
    File license
    Fair use/fair dealing exception

Reply 136 of 167, by Jepael

User metadata
Rank Oldbie
Rank
Oldbie

Can you give some pointers how to compile adplug and adplay-unix under linux? The last time I tried even compiling adplug, I ran into Autoconf issues.

$ autoreconf --install
configure.ac:8: warning: macro 'AM_PROG_LIBTOOL' not found in library

Solving this would motivate me more to contribute, my previous pull request was done blind without trying to compile, and I am not proud of that..

Reply 137 of 167, by opl2

User metadata
Rank Newbie
Rank
Newbie
Jepael wrote:
Can you give some pointers how to compile adplug and adplay-unix under linux? The last time I tried even compiling adplug, I ran […]
Show full quote

Can you give some pointers how to compile adplug and adplay-unix under linux? The last time I tried even compiling adplug, I ran into Autoconf issues.

$ autoreconf --install
configure.ac:8: warning: macro 'AM_PROG_LIBTOOL' not found in library

Solving this would motivate me more to contribute, my previous pull request was done blind without trying to compile, and I am not proud of that..

Hmm, could it be you don't have libtool installed? It's working for me in Ubuntu (16.04), but I have to modify adplay-unix's source to get it to work on OSX. If libtool is indeed not installed, try "autoreconf -fvi" after installing it.

Reply 138 of 167, by Jepael

User metadata
Rank Oldbie
Rank
Oldbie

Thanks, it just required libtool install (Mint 17.3), and poking things randomly, installing compiled libraries to system dirs and running ldconfig to find them, it finally runs.

So what's the preferred workflow, as the libadplug is now installed into system library dir, and I want to develop that, using stock adplay-unix to test?
Compile and reinstall libadplug into system dirs again and again?

Reply 139 of 167, by opl2

User metadata
Rank Newbie
Rank
Newbie
Jepael wrote:

So what's the preferred workflow, as the libadplug is now installed into system library dir, and I want to develop that, using stock adplay-unix to test?
Compile and reinstall libadplug into system dirs again and again?

Ha, I am the wrong person to ask. It is a bit of a hassle, but you don't have to do that, but you'll have to override the prefixes and PKG_CONFIG_PATH.
I really don't know what the best way to go about this is, since I don't do this nearly enough and have only recently dug into adplug a little, but here's what I have been using:

https://gist.github.com/thatdutchguy/1efa4ea8 … f95563214df55a0

You'll end up with a directory structure like:
- adplug/
- adplay-unix/
- build/

Where build/ has a /lib, /bin, etc. completely separate from your system libraries, and you can just run ./build/bin/adplay [options] to test your changes. It's good enough for me 😉