DOSBox ECE (for Windows & Linux)

Developer's Forum, for discussion of bugs, code, and other developmental aspects of DOSBox.

Re: DOSBox ECE (for Windows & Linux)

Postby Qbix » 2019-11-19 @ 16:13

What did you specificly change so that games like BAK are broken ? The cue file doesn't look that bad to me. Why is extra bookkeeping needed about what the game uses ?
Water flows down the stream
How to ask questions the smart way!
User avatar
Qbix
DOSBox Author
 
Posts: 10954
Joined: 2002-11-27 @ 14:50
Location: Fryslan

Re: DOSBox ECE (for Windows & Linux)

Postby krcroft » 2019-11-19 @ 16:17

Pr3tty F1y, interesting history about people wrestling with Castlevania and getting it working through cue manipulation! You knew exactly what was going on with Krondor.
Yes; it's too bad buyers of GoG games aren't given the option to download archival images of the floppies or CDs. I can image a decent percent of users have original hardware and would prefer to install from hard media, instead of transplanting their post-installed game files after clicking through GoG's installer on a modern PC.
User avatar
krcroft
Member
 
Posts: 428
Joined: 2017-4-29 @ 15:07
Location: Ogden's Retreat

Re: DOSBox ECE (for Windows & Linux)

Postby krcroft » 2019-11-19 @ 16:55

Qbix, I only have a mode 1/2048 BIN+CUE, and used bchunk to rip it to individual tracks. The content in those resulting tracks doesn't match with what the game expects to play.

A check for this would activate when the game queries for a track number's MSF offset. It would record the track# as the 'last queried track number'.

Then, when the game requests playback at an MSF offset (and DOSBox subsequently calculates which ogg/wav file that offset lands within), the check would compare this 'actual track' number against the 'last queried track number' and report if they mismatch.

This check would ignore the common scenario where games query each track's MSF time in monotonically increasing or decreasing order; so we would not record the 'last queried track number' on back-to-back queries.

We'd also ignore scenarios where back-to-back playback requests are made without a corresponding query for a track's MSF offset in between.

Probably other false-positive patterns too; but atleast it would catch this particular cue problem in BaK.
User avatar
krcroft
Member
 
Posts: 428
Joined: 2017-4-29 @ 15:07
Location: Ogden's Retreat

Re: DOSBox ECE (for Windows & Linux)

Postby Qbix » 2019-11-19 @ 18:10

I meant what did you change so that the currently working! cue/ogg/mp3 setup of this specific game doesn't work with your patch ?
It is a regression that a working setup stops working after adding your patch.
Water flows down the stream
How to ask questions the smart way!
User avatar
Qbix
DOSBox Author
 
Posts: 10954
Joined: 2002-11-27 @ 14:50
Location: Fryslan

Re: DOSBox ECE (for Windows & Linux)

Postby krcroft » 2019-11-19 @ 22:36

When I rip my BIN to oggs, neither SVN or ECE can mount the resulting CUE. If someone is willing to send me their mountable oggs+cue, I'm happy to investigate!
User avatar
krcroft
Member
 
Posts: 428
Joined: 2017-4-29 @ 15:07
Location: Ogden's Retreat

Re: DOSBox ECE (for Windows & Linux)

Postby collector » 2019-11-20 @ 01:58

Using ECE r4290 it will mount my cue with the tracks ripped to OGGs, but does not play them. Using yesterplay's plain SVN r4290 build it will mount the cue and play the OGGs in game. This is on Win7, not Linux. The cue is the one I wrote for my BaK installer. I will note that the ISO is a dummy one since all of the game files are in the installed directory.

Code: Select all
FILE "CD.ISO" BINARY
  TRACK 01 MODE1/2048
    INDEX 01 00:00:00
FILE "track02.ogg" MP3
  TRACK 02 AUDIO
    INDEX 02 00:00:00
FILE "track03.ogg" MP3
  TRACK 03 AUDIO
    INDEX 03 00:00:00
FILE "track04.ogg" MP3
  TRACK 04 AUDIO
    INDEX 04 00:00:00
FILE "track05.ogg" MP3
  TRACK 05 AUDIO
    INDEX 05 00:00:00
FILE "track06.ogg" MP3
  TRACK 06 AUDIO
    INDEX 06 00:00:00
FILE "track07.ogg" MP3
  TRACK 07 AUDIO
    INDEX 07 00:00:00
FILE "track08.ogg" MP3
  TRACK 08 AUDIO
    INDEX 08 00:00:00
FILE "track09.ogg" MP3
  TRACK 09 AUDIO
    INDEX 09 00:00:00
FILE "track10.ogg" MP3
  TRACK 10 AUDIO
    INDEX 10 00:00:00
FILE "track11.ogg" MP3
  TRACK 11 AUDIO
    INDEX 11 00:00:00
FILE "track12.ogg" MP3
  TRACK 12 AUDIO
    INDEX 12 00:00:00
FILE "track13.ogg" MP3
  TRACK 13 AUDIO
    INDEX 13 00:00:00
FILE "track14.ogg" MP3
  TRACK 14 AUDIO
    INDEX 14 00:00:00
FILE "track15.ogg" MP3
  TRACK 15 AUDIO
    INDEX 15 00:00:00
FILE "track16.ogg" MP3
  TRACK 16 AUDIO
    INDEX 16 00:00:00
FILE "track17.ogg" MP3
  TRACK 17 AUDIO
    INDEX 17 00:00:00
FILE "track18.ogg" MP3
  TRACK 18 AUDIO
    INDEX 18 00:00:00
FILE "track19.ogg" MP3
  TRACK 19 AUDIO
    INDEX 19 00:00:00
FILE "track20.ogg" MP3
  TRACK 20 AUDIO
    INDEX 20 00:00:00
FILE "track21.ogg" MP3
  TRACK 21 AUDIO
    INDEX 21 00:00:00
FILE "track22.ogg" MP3
  TRACK 22 AUDIO
    INDEX 22 00:00:00
FILE "track23.ogg" MP3
  TRACK 23 AUDIO
    INDEX 23 00:00:00
FILE "track24.ogg" MP3
  TRACK 24 AUDIO
    INDEX 24 00:00:00
FILE "track25.ogg" MP3
  TRACK 25 AUDIO
    INDEX 25 00:00:00
FILE "track26.ogg" MP3
  TRACK 26 AUDIO
    INDEX 26 00:00:00
FILE "track27.ogg" MP3
  TRACK 27 AUDIO
    INDEX 27 00:00:00
FILE "track28.ogg" MP3
  TRACK 28 AUDIO
    INDEX 28 00:00:00
FILE "track29.ogg" MP3
  TRACK 29 AUDIO
    INDEX 29 00:00:00
FILE "track30.ogg" MP3
  TRACK 30 AUDIO
    INDEX 30 00:00:00
FILE "track31.ogg" MP3
  TRACK 31 AUDIO
    INDEX 31 00:00:00
FILE "track32.ogg" MP3
  TRACK 32 AUDIO
    INDEX 32 00:00:00
FILE "track33.ogg" MP3
  TRACK 33 AUDIO
    INDEX 33 00:00:00
FILE "track34.ogg" MP3
  TRACK 34 AUDIO
    INDEX 34 00:00:00
FILE "track35.ogg" MP3
  TRACK 35 AUDIO
    INDEX 35 00:00:00
FILE "track36.ogg" MP3
  TRACK 36 AUDIO
    INDEX 36 00:00:00
FILE "track37.ogg" MP3
  TRACK 37 AUDIO
    INDEX 37 00:00:00
FILE "track38.ogg" MP3
  TRACK 38 AUDIO
    INDEX 38 00:00:00
FILE "track39.ogg" MP3
  TRACK 39 AUDIO
    INDEX 39 00:00:00
FILE "track40.ogg" MP3
  TRACK 40 AUDIO
    INDEX 40 00:00:00
FILE "track41.ogg" MP3
  TRACK 41 AUDIO
    INDEX 41 00:00:00
FILE "track42.ogg" MP3
  TRACK 42 AUDIO
    INDEX 42 00:00:00
FILE "track43.ogg" MP3
  TRACK 43 AUDIO
    INDEX 43 00:00:00
FILE "track44.ogg" MP3
  TRACK 44 AUDIO
    INDEX 44 00:00:00
FILE "track45.ogg" MP3
  TRACK 45 AUDIO
    INDEX 45 00:00:00
FILE "track46.ogg" MP3
  TRACK 46 AUDIO
    INDEX 46 00:00:00
FILE "track47.ogg" MP3
  TRACK 47 AUDIO
    INDEX 47 00:00:00
FILE "track48.ogg" MP3
  TRACK 48 AUDIO
    INDEX 48 00:00:00
FILE "track49.ogg" MP3
  TRACK 49 AUDIO
    INDEX 49 00:00:00
FILE "track50.ogg" MP3
  TRACK 50 AUDIO
    INDEX 50 00:00:00
FILE "track51.ogg" MP3
  TRACK 51 AUDIO
    INDEX 51 00:00:00
FILE "track52.ogg" MP3
  TRACK 52 AUDIO
    INDEX 52 00:00:00
FILE "track53.ogg" MP3
  TRACK 53 AUDIO
    INDEX 53 00:00:00
FILE "track54.ogg" MP3
  TRACK 54 AUDIO
    INDEX 54 00:00:00
FILE "track55.ogg" MP3
  TRACK 55 AUDIO
    INDEX 55 00:00:00
FILE "track56.ogg" MP3
  TRACK 56 AUDIO
    INDEX 56 00:00:00
FILE "track57.ogg" MP3
  TRACK 57 AUDIO
    INDEX 57 00:00:00
FILE "track58.ogg" MP3
  TRACK 58 AUDIO
    INDEX 58 00:00:00
FILE "track59.ogg" MP3
  TRACK 59 AUDIO
    INDEX 59 00:00:00
FILE "track60.ogg" MP3
  TRACK 60 AUDIO
    INDEX 60 00:00:00
FILE "track61.ogg" MP3
  TRACK 61 AUDIO
    INDEX 61 00:00:00
FILE "track62.ogg" MP3
  TRACK 62 AUDIO
    INDEX 62 00:00:00
FILE "track63.ogg" MP3
  TRACK 63 AUDIO
    INDEX 63 00:00:00
User avatar
collector
l33t
 
Posts: 4401
Joined: 2003-1-15 @ 10:39

Re: DOSBox ECE (for Windows & Linux)

Postby krcroft » 2019-11-21 @ 03:12

Here's what's going on with Betrayal at Krondor (GoG edition).

Audio track inspection: Please listen to the end of
OGG Track 15 and the start of Track 16.
The end of 15 actually contains a bit more than 2 seconds of "extra" silence, including the first bit of track 16's audio! Track 16 has no "lead up"; it's playing from the get-go.

Game behavior:
  1. The game asks for the minute-second-frame (MSF) for the track number it wants to play, which is Track 16 for the intro sequence.
  2. DOSBox returns Track 16's MSF value of 22:33:22 back to the game, which equates to a sector offset of 101497
  3. The game subtracts 189 sectors (or 2.52 seconds) from this value, which logically becomes the very end of Track 15.
  4. The game asks to play at this deducted location (Track 16 minus 2.52 seconds) for a duration of 44.64 seconds.

Here are the two MSCDEX calls Krondor makes (GetAudioTrackInfo and PlayAudioSector), plus the internal query to GetTrack(); I've slightly adjusted my current debug statements, so yours might look different:
Code: Select all
18:01:04 CDROM: GetAudioTrackInfo(queried MSF for track 16), returned MSF 22:33:22 (sector 101497) with attr=0
18:01:04 CDROM: GetTrack(sector 101308) lands at MSF 22:30:58 in track 15 that starts at 96546 and ends at 101347
18:01:04 CDROM: PlayAudioSector(at sector 101308 for 3348 sectors) inside track 15 that starts at 96546 and ends at 101347


Observations:
Notice that the game wants to play sector 101308, which is Track 15, but Track 15 ends a mere 39 sectors later, at sector 101347. You might ask, what happened to the 189 frames? Where did 150 of them disappear to? This is because DOSBox internally pads the start each track by 2 seconds (or 150 frames) so these "eat up" 150 of them, leaving us 39 frames into the tail-end of Track 15.

So we start to see what's going on.. the game starts playing Track 15 just a split seconds before it ends, however the game believes the track will carry on for 44.6 seconds!

How ECE and SVN differ:
  • ECE treats each file's track as its own sequence, and only will play up-to the end of the file. So if a game tries to a play beyond the track (which can happen in small quantities due to rounding errors or the compressed track being a hair shorter than the binary data), then ECE stops the track playback when no more data can be read from the track.
  • SVN treats each 2352-byte sector as its own independent read: it seeks to the desired sector, reads 2352 bytes, and then repeats. If data isn't available or not "playing", it fills in those 2352-bytes with silence and plays that instead. So we see this behavior allows "reading beyond" a given track; if the compressed track comes up short or the game asks for a bit longer playback then the track actually has, then SVN plays from the start of the next track.

So the immediate fix would be to make the audio patch used by ECE behave similarly. That said, I do have some reservations/questions:

  • First, this is the only CD-DA game I've seen with this behavior.. where it plays the last split second of the prior track and relies on "reading beyond" by almost the entire next track's duration
  • Second, I have to question the RIP -- why are the final ~2.5 seconds of each track actually the start of the subsequent track?
  • If the RIP is perfectly accurate, then these slightly shifted tracks would exist in the physical CD, so perhaps this "subtract 189 frame" behavior was made by the game developers to compensate for the audio CD pressing results. ie: We can rebuild him, we have the technology...the last minute software hack to work around physical issues that would be prohibitive to fix.
  • In the end though, if MSCDEX's specification says that playing-beyond a track's duration is legal and should flow into the subsequent track - then indeed, the patch will need to be updated.
  • A work-around also exists (avoiding the need for ECE to cater to this), which is converting the OGGs into PCM tracks and concatenating them to the data track, and creating a CUE; as we know ECE has no issues with the BIN/CUE release.
  • Sorry for the wall of text.. thanks for making it through, and curious what you think.
User avatar
krcroft
Member
 
Posts: 428
Joined: 2017-4-29 @ 15:07
Location: Ogden's Retreat

Re: DOSBox ECE (for Windows & Linux)

Postby ripsaw8080 » 2019-11-21 @ 04:11

If you look at the believed-to-be-good dumps at re****.org, the 1.02 version has standard pregap (2 seconds = 150 frames) on the first audio track while the (presumably) earlier version has non-standard pregap across most, but not all, audio tracks. It's possible that some versions of the game compensate for what might be a disc mastering issue. Also, some ripping/imaging software mishandles pregap, leading to a 2 second "shift" in audio track start/end -- I've encountered a few images where that seems to have happened.

Anyway, when playing goes past the end of a track it should continue on to the next, and the next, and so on, until the play-length is reached.
User avatar
ripsaw8080
DOSBox Author
 
Posts: 4434
Joined: 2006-4-25 @ 23:24

Re: DOSBox ECE (for Windows & Linux)

Postby krcroft » 2019-11-21 @ 04:37

Thanks ripsaw8080; will update accordingly!
User avatar
krcroft
Member
 
Posts: 428
Joined: 2017-4-29 @ 15:07
Location: Ogden's Retreat

Re: DOSBox ECE (for Windows & Linux)

Postby krcroft » 2019-11-25 @ 02:36

Revision 14 of the audio patch has been posted, along with changelog here: viewtopic.php?f=41&t=62203&p=695839#p695839
This fixes Betrayal at Krondor and Destruction Derby (both when using file-based audio tracks).

The code as-patched has been tested to build under the following:
- Windows - VisualStudio 32-bit target
- Windows - MSYS2, GCC 32-bit and 64-bit targets
- Windows - MSYS2, Clang 32-bit and 64-bit targets
- macOS - Xcode Clang
- macOS - Brew GCC-9
- Ubuntu 16.04 - GCC-5
- Ubuntu 18.04 - GCC-7
- Ubuntu 18.04 - GCC-9
- Ubuntu 18.04 - Clang-8

https://github.com/dreamer/dosbox-stagi ... =326280905

For those with build environments, feel free to test your CDDA titles and report any regressions; thank you!
User avatar
krcroft
Member
 
Posts: 428
Joined: 2017-4-29 @ 15:07
Location: Ogden's Retreat

Re: DOSBox ECE (for Windows & Linux)

Postby Yesterplay80 » 2019-11-25 @ 19:08

krcroft wrote:Revision 14 of the audio patch has been posted, along with changelog here: viewtopic.php?f=41&t=62203&p=695839#p695839

I'll update it as soon as get all the new dependencies sorted out. Opusfile requires opus and libopus, which requires openssl, which requires perl...
My full-featured DOSBox SVN builds (without debugger) for Windows: Vanilla DOSBox and DOSBox ECE (Enhanced Community Edition)
User avatar
Yesterplay80
Member
 
Posts: 459
Joined: 2016-2-23 @ 11:02
Location: Germany

Re: DOSBox ECE (for Windows & Linux)

Postby krcroft » 2019-11-25 @ 20:16

Thanks Yesterplay80.

If you compile opusfile, there's a configure option to disable its web-file-open ability. If that's left in, then yes.. it needs openssl and friends.

The prior patch did this manual build during the make process with the web stuff disabled. Hope that helps.

PS.. when the single header replacement (dr_opus) for Xiph's Opus library is available, I will be using that instead!
User avatar
krcroft
Member
 
Posts: 428
Joined: 2017-4-29 @ 15:07
Location: Ogden's Retreat

Re: DOSBox ECE (for Windows & Linux)

Postby Yesterplay80 » 2019-11-27 @ 08:22

Ah, thank for the hint. However, I can't copmpile opusfile because of missing autoconf files for pkg-config, which isn't available for MSYS/MINGW... Time to get the MSYS2 environment working... So, for the time being, ECE will be stuck on Rev. 13 of your patch until I can compile it under MSYS2.
My full-featured DOSBox SVN builds (without debugger) for Windows: Vanilla DOSBox and DOSBox ECE (Enhanced Community Edition)
User avatar
Yesterplay80
Member
 
Posts: 459
Joined: 2016-2-23 @ 11:02
Location: Germany

Re: DOSBox ECE (for Windows & Linux)

Postby dreamer_ » 2019-11-27 @ 23:19

Yesterplay80 wrote:Ah, thank for the hint. However, I can't copmpile opusfile because of missing autoconf files for pkg-config, which isn't available for MSYS/MINGW... Time to get the MSYS2 environment working... So, for the time being, ECE will be stuck on Rev. 13 of your patch until I can compile it under MSYS2.

Have you considered using Visual Studio 2019 + vcpkg for Windows builds? If yes, but you opted to MSYS/MINGW instead - were there any technical reasons? We have both MSYS2 and VS2019 setup working via CI in dosbox-staging, so feel free to copy whatever you need.

If you want, I can give you push access to my repo, so you could store your patch series (which is super helpful btw, your work is indispensable to the community) as a Git branch (it would make it easier to rebase your series on top of new changes from SVN, if you know your way around git-rebase).
Code: Select all
| ← Ceci n'est pas une pipe
User avatar
dreamer_
Newbie
 
Posts: 77
Joined: 2019-5-17 @ 20:19

Re: DOSBox ECE (for Windows & Linux)

Postby krcroft » 2019-11-28 @ 01:19

We also have this pull-request in progress that will let you build Opusfile and friends statically, and then pass through ./configure without the need for pkg-config.

If you want to give it a try before we merge, you can download the Makefile into your msys environment, `make -j4`, and then set the environment variables as it describes (paths should be absolute, so its location is independent from the DOSBox source directory).

Your resulting DOSBox binary will have all its audio codecs internal to it, but because SDL and other libraries link with even more libraries, the DOSBox binary will superficially appear to have lots of codec dependencies (and compression libraries like lz4, lzma, ... and so on).

ldd /usr/lib/x86_64-linux-gnu/libSDL-1.2.so.0 | grep -i 'ogg\|vorbis\|flac'

Code: Select all
   libFLAC.so.8 => /usr/lib/x86_64-linux-gnu/libFLAC.so.8 (0x00007f05c872b000)
   libogg.so.0 => /usr/lib/x86_64-linux-gnu/libogg.so.0 (0x00007f05c8524000)
   libvorbis.so.0 => /usr/lib/x86_64-linux-gnu/libvorbis.so.0 (0x00007f05c84f7000)
   libvorbisenc.so.2 => /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2 (0x00007f05c844c000)
User avatar
krcroft
Member
 
Posts: 428
Joined: 2017-4-29 @ 15:07
Location: Ogden's Retreat

Re: DOSBox ECE (for Windows & Linux)

Postby Yesterplay80 » 2019-11-28 @ 14:30

dreamer_ wrote:Have you considered using Visual Studio 2019 + vcpkg for Windows builds? If yes, but you opted to MSYS/MINGW instead - were there any technical reasons? We have both MSYS2 and VS2019 setup working via CI in dosbox-staging, so feel free to copy whatever you need.

I tried using it at the beginning but something, I can't even remember what it was, didn't work, so I followed the guide in the official wiki and set everything up using MINGW. For someone like me, without any programming and development skills, it was a good way to start and it was helpful because it also made reproducing the required steps in Linux much easier, because it's so similar. And in the meantime I got used to it and can create a new build with the start of just one shell script per OS.

dreamer_ wrote:If you want, I can give you push access to my repo, so you could store your patch series (which is super helpful btw, your work is indispensable to the community) as a Git branch (it would make it easier to rebase your series on top of new changes from SVN, if you know your way around git-rebase).

TBH: I don't have the slightest idea how to use GIT or any of its tools, I made an account once to put my code there, but I still didn't find the time to get into the whole matter. :cry:
My full-featured DOSBox SVN builds (without debugger) for Windows: Vanilla DOSBox and DOSBox ECE (Enhanced Community Edition)
User avatar
Yesterplay80
Member
 
Posts: 459
Joined: 2016-2-23 @ 11:02
Location: Germany

Re: DOSBox ECE (for Windows & Linux)

Postby KainXVIII » 2019-11-29 @ 07:35

I wonder if this possible - that dosbox vsync will be enabled only when ingame fps is 60 fps or lower, when fps go above 60 (to 70 usual) - vsync will be automatically disabled (ie something like adaptive nvidia sync :cool:)
OR if vsync can be enabled on the fly with some key combination (so you can enable-disable it when you need it, without slowing down menus/music during 70 fps parts)
User avatar
KainXVIII
Member
 
Posts: 340
Joined: 2015-5-20 @ 15:04
Location: Yaroslavl

Re: DOSBox ECE (for Windows & Linux)

Postby Kisai » 2019-11-29 @ 19:12

KainXVIII wrote:I wonder if this possible - that dosbox vsync will be enabled only when ingame fps is 60 fps or lower, when fps go above 60 (to 70 usual) - vsync will be automatically disabled (ie something like adaptive nvidia sync :cool:)
OR if vsync can be enabled on the fly with some key combination (so you can enable-disable it when you need it, without slowing down menus/music during 70 fps parts)


I don't think that's even remotely possible since vsync flags have to be passed to SDL when the window is created.
Kisai
Member
 
Posts: 138
Joined: 2010-5-05 @ 08:04

Re: DOSBox ECE (for Windows & Linux)

Postby KainXVIII » 2019-11-29 @ 21:18

Kisai wrote:
KainXVIII wrote:I wonder if this possible - that dosbox vsync will be enabled only when ingame fps is 60 fps or lower, when fps go above 60 (to 70 usual) - vsync will be automatically disabled (ie something like adaptive nvidia sync :cool:)
OR if vsync can be enabled on the fly with some key combination (so you can enable-disable it when you need it, without slowing down menus/music during 70 fps parts)


I don't think that's even remotely possible since vsync flags have to be passed to SDL when the window is created.

I thought so (but you'll never know..maybe in the future :cool: )
User avatar
KainXVIII
Member
 
Posts: 340
Joined: 2015-5-20 @ 15:04
Location: Yaroslavl

Re: DOSBox ECE (for Windows & Linux)

Postby realnc » 2019-11-30 @ 08:20

KainXVIII wrote:
Kisai wrote:
KainXVIII wrote:I wonder if this possible - that dosbox vsync will be enabled only when ingame fps is 60 fps or lower, when fps go above 60 (to 70 usual) - vsync will be automatically disabled (ie something like adaptive nvidia sync :cool:)
OR if vsync can be enabled on the fly with some key combination (so you can enable-disable it when you need it, without slowing down menus/music during 70 fps parts)


I don't think that's even remotely possible since vsync flags have to be passed to SDL when the window is created.

I thought so (but you'll never know..maybe in the future :cool: )

The DOSBox-SVN core in RetroArch does this.
User avatar
realnc
Member
 
Posts: 460
Joined: 2010-10-13 @ 11:02

PreviousNext

Return to DOSBox Development

Who is online

Users browsing this forum: knowledge [bot] and 1 guest