VOGONS

Common searches


First post, by moog

User metadata
Rank Newbie
Rank
Newbie

Hello,

I'm writing an ebuild for Counter-Strike 1.5 for Gentoo Linux. Turns out, I can accomplish this right now, but I'd like to make the user aware of what's happening. Here's the workflow of how the ebuild works, and what parts I want to improve before publishing it.

1. src_fetch retrieves Counter-Strike 1.5 full installer from a mirror.
2. Because csv15full.exe is a WISE installer file, we need to use an eclass I'm writing that supports unpacking WISE installers' data.
3. src_unpack calls the main function of the eclass to go through all specified WISE installer files.
4. Done.

The eclass' main function itself breaks down into more steps:
-1. Actually, we interject src_fetch from the caller to also download e_wise, a Pascal program for DOS, Windows and OS/2 that does the job just fine... kinda. It gets the correct results, with a DOS executable it isn't really a vendor lock-in for x86, but it's not as nice as let's say 7z or unrar if they could do it.
1. We extract e_wise to ewise directory so we can use it.
2. Create wise_output directory.
3. Declare a counter and initialize at 0.
4. For each WISE installer file:
4.1. Create a symlink to the WISE installer file and call it ${counter}.exe. DOSifying the long names of these installers isn't something I want to do or feel comfortable doing.
4.2. Create the directory called after the WISE installer file, sans the dot exe.
4.3. `push` this directory into the directory stack
4.4. Create a DOSBox config with an autoexec section that basically makes an instance of DOSBox extract ${counter}.exe into ${WISE_installer_dir}
4.5. Run DOSBox with said config. This will take some time, but out WISE installer will eventually be extracted.
4.6. Do some Bash-fu to clean up the mess - re-create directory structures and rename all the files and put them in correct places. e_wise is kind enough to leave this info in 00000000.BAT file.
4.7. Remove "homeless" and useless files, such as 00000000.BAT, 00000000.TXT and .EWI files (e_wise's way of naming dumped files) that do not belong into any directory structure.
4.8. Move the resulting clean output into wise_output directory.
4.9. Increment ${counter}.
5. Report success.

During wise eclass step 4.5, I would like to redirect the standard output and error of e_wise into the standard output and error of the ebuild process that's calling DOSBox. I'm already running it headless with SDL_VIDEODRIVER=dummy, but how can I go the extra mile to make sure users know what is currently happening in the DOSBox process? I'm unfortunately not very good with Pascal, I don't know what version of Pascal is required to build e_wise, so fixing or even adapting it to my use case is really harder than using it as-is.

edit: No, I can't re-upload the decompressed WISE installer. That's illegal. I can only work with what the Counter-Strike team has released in pre-Steam times in the form they have disseminated.

Abit VP6 w/ YT BIOS, 2x Tualatin-S with Connectland northbridge heatsinks, 4x512MB SDRAM PC133 CL2, SanDisk 32GB CF 160/120MB/s R/W, Plextor PX-800A, Voodoo 5500 AGP with passive mod, Voodoo 1, Yamaha YMF724F, D-Link DGE 528T rev B

Reply 1 of 36, by Jorpho

User metadata
Rank l33t++
Rank
l33t++
moog wrote on 2021-02-04, 17:16:

we interject src_fetch from the caller to also download e_wise, a Pascal program for DOS, Windows and OS/2 that does the job just fine... kinda. It gets the correct results, with a DOS executable it isn't really a vendor lock-in for x86, but it's not as nice as let's say 7z or unrar if they could do it.

At the risk of stating the obvious, I'm very surprised that there's no open-source Linux solution for this. Have you tried unar?
https://theunarchiver.com/command-line

Reply 2 of 36, by Ringding

User metadata
Rank Newbie
Rank
Newbie

I’ve recently seen a very similar usage of DOSBox in the open-watcom-v2 build. I did not check what they did exactly, but superficially, it would be mostly the same. Do you really need stdout? Don’t you just need to get out some files? After you have a mount, that would happen automatically anyway.

https://github.com/open-watcom/open-watcom-v2 … mif/wgmlcmd.mif

Reply 3 of 36, by moog

User metadata
Rank Newbie
Rank
Newbie
Jorpho wrote on 2021-02-04, 17:45:
moog wrote on 2021-02-04, 17:16:

we interject src_fetch from the caller to also download e_wise, a Pascal program for DOS, Windows and OS/2 that does the job just fine... kinda. It gets the correct results, with a DOS executable it isn't really a vendor lock-in for x86, but it's not as nice as let's say 7z or unrar if they could do it.

At the risk of stating the obvious, I'm very surprised that there's no open-source Linux solution for this. Have you tried unar?
https://theunarchiver.com/command-line

I wasn't aware of this tool. I tried to unpack it with 7z, unrar and even binwalk - no go. The reason I came up with e_wise in the fist place is because the only Google and DuckDuckGo results I could get for WISE unpackers are UniversalExtractor and UniExtract2 - both of which are written in AutoIt, which already means these are Windows-only solutions. They both require existing tools for handling various formats to be present. They delegate the responsibility for WISE installers to e_wise.. Let's see now about this unar...

19:57 GMT+1: unar is available under app-arch/unar, but requires that you have a Objective-C compiler. I'll let you know how that goes.
20:48 GMT+1:

unar csv15full.exe 
csv15full.exe: Couldn't recognize the archive format.

Well, that was an hour well spent.

Ringding wrote on 2021-02-04, 19:27:

I’ve recently seen a very similar usage of DOSBox in the open-watcom-v2 build. I did not check what they did exactly, but superficially, it would be mostly the same. Do you really need stdout? Don’t you just need to get out some files? After you have a mount, that would happen automatically anyway.

https://github.com/open-watcom/open-watcom-v2 … mif/wgmlcmd.mif

Yes. That is a nice thing to have. Here's what the build prints now:

>>> Creating Manifest for /var/lib/layman/src_prepare-overlay/games-fps/counter-strike-data
* e_wise.arj BLAKE2B SHA512 size ;-) ... [ ok ]
* csv15full.exe BLAKE2B SHA512 size ;-) ... [ ok ]
>>> Unpacking source...
/var/tmp/portage/games-fps/counter-strike-data-1.5/work/ewise /var/tmp/portage/games-fps/counter-strike-data-1.5/work

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=pl_PL.utf8,Utf16=on,HugeFiles=on,64 bits,16 CPUs AMD Ryzen 7 2700 Eight-Core Processor (800F82),ASM,AES-NI)

Scanning the drive for archives:
1 file, 108222 bytes (106 KiB)

Extracting archive: /var/tmp/portage/games-fps/counter-strike-data-1.5/distdir/e_wise.arj
--
Path = /var/tmp/portage/games-fps/counter-strike-data-1.5/distdir/e_wise.arj
Type = Arj
Physical Size = 108222
Name = e_wise.ARJ
Created = 2002-07-01 01:11:24
Modified = 2002-07-01 01:11:24
Host OS = MSDOS

Everything is Ok

Files: 16
Size: 164345
Compressed: 108222
/var/tmp/portage/games-fps/counter-strike-data-1.5/work
/var/tmp/portage/games-fps/counter-strike-data-1.5/work/csv15full /var/tmp/portage/games-fps/counter-strike-data-1.5/work
* Extracting csv15full.exe in DOSBox
DOSBox version 0.74-3
Copyright 2002-2019 DOSBox Team, published under GNU GPL.
---
CONFIG:Loading primary settings from config file /var/tmp/portage/games-fps/counter-strike-data-1.5/work/csv15full.exe.dosbox
Your fullscreen resolution can NOT be determined, it's assumed to be 1024x768.
Please edit the configuration file if this value is wrong.
MIXER:Got different values from SDL: freq 44100, blocksize 512
MIDI:Opened device:none
DOSBox switched to max cycles, because of the setting: cycles=auto. If the game runs too fast try a fixed cycles amount in DOSBox's options.
* Creating directory structures
* Renaming files
* Removing homeless file 00000001.EWI
* Removing homeless file 00000002.EWI
* Removing homeless file 00000003.EWI
* Removing homeless file 00000004.EWI
* Removing homeless file 00000005.EWI
* Removing homeless file 00000008.EWI
* Removing homeless file 00000009.EWI
* Removing homeless file 00000010.EWI
* Removing homeless file 00000011.EWI
* Removing homeless file 00000012.EWI
* Removing homeless file 00001069.EWI
* Removing homeless file 00001070.EWI
* Removing homeless file 00001071.EWI
* Removing homeless file 00001072.EWI
* Removing useless file 00000006.EWI
* Removing useless file 00000000.BAT
* Removing useless file 00000000.TXT
/var/tmp/portage/games-fps/counter-strike-data-1.5/work
>>> Source unpacked in /var/tmp/portage/games-fps/counter-strike-data-1.5/work

Between line 30 and 40 there's a considerable amount of time taken. It would be nice to let us know what's exactly happening. Normally when e_wise is run in DOSBox, it looks like in the attachment.

Attachments

  • e_wise.png
    Filename
    e_wise.png
    File size
    18.92 KiB
    Views
    663 views
    File comment
    e_wise for DOS in action
    File license
    Public domain

Abit VP6 w/ YT BIOS, 2x Tualatin-S with Connectland northbridge heatsinks, 4x512MB SDRAM PC133 CL2, SanDisk 32GB CF 160/120MB/s R/W, Plextor PX-800A, Voodoo 5500 AGP with passive mod, Voodoo 1, Yamaha YMF724F, D-Link DGE 528T rev B

Reply 4 of 36, by DosFreak

User metadata
Rank l33t++
Rank
l33t++

Is there any reason you can't use e_wise_w.exe? It's a 32bit windows app so should run fine in Wine.

DOSBox Compilation Guides
DosBox Feature Request Thread
PC Game Compatibility List
How To Ask Questions The Smart Way
Running DRM games offline

Reply 5 of 36, by moog

User metadata
Rank Newbie
Rank
Newbie
DosFreak wrote on 2021-02-04, 21:15:

Is there any reason you can't use e_wise_w.exe? It's a 32bit windows app so should run fine in Wine.

Yes. It's exactly the reason you mention 😁

First of all, not everyone has x86. Heck, there may be a future where x86 is a thing of the past. What then?

Second, running Windows applications through Wine can be done as part of an ebuild's course of installing software, but it's not easily achieved on non-x86 machines. To get Wine on non-x86 machines, you'd have to emerge crossdev, then manually emerge x86/amd64 Linux toolchain through crossdev, and finally cross-emerge Wine for x86/amd64. Emulating x86 through QEMU x86 userland runtime to run the Windows app is going to cost a lot more than using DOSBox.

Abit VP6 w/ YT BIOS, 2x Tualatin-S with Connectland northbridge heatsinks, 4x512MB SDRAM PC133 CL2, SanDisk 32GB CF 160/120MB/s R/W, Plextor PX-800A, Voodoo 5500 AGP with passive mod, Voodoo 1, Yamaha YMF724F, D-Link DGE 528T rev B

Reply 6 of 36, by DosFreak

User metadata
Rank l33t++
Rank
l33t++

Are we talking about hypotheticals or what you are currently trying to do?
Is your machine not x86?
Are you building a package on a machine you cannot play the game on?
How would the non-x86 machine play the x86 game? It wouldn't be DOSBox.
I know Half-Life uses LFN can't remember if counterstrike does. DOSBox does not support LFN unless you boot a image.

Here's the source, don't know what the license is: https://kannegieser.net/veit/quelle/e_wise_src.arj

DOSBox Compilation Guides
DosBox Feature Request Thread
PC Game Compatibility List
How To Ask Questions The Smart Way
Running DRM games offline

Reply 7 of 36, by moog

User metadata
Rank Newbie
Rank
Newbie
DosFreak wrote on 2021-02-04, 21:28:
Are we talking about hypotheticals or what you are currently trying to do? Is your machine not x86? Are you building a package […]
Show full quote

Are we talking about hypotheticals or what you are currently trying to do?
Is your machine not x86?
Are you building a package on a machine you cannot play the game on?
How would the non-x86 machine play the x86 game? It wouldn't be DOSBox.
I know Half-Life uses LFN can't remember if counterstrike does. DOSBox does not support LFN unless you boot a image.

Here's the source, don't know what the license is: https://kannegieser.net/veit/quelle/e_wise_src.arj

Gentoo Linux supports:

  • amd64
  • x86
  • DEC Alpha
  • ARM
  • HPPA
  • Itanium
  • PowerPC
  • SPARC
  • S390
  • RISCV

As such, it only makes sense to make new packages as architecture-agnostic as possible. Of all these, I own 5, and by the look of things going for Intel and AMD with all the hardware-based 0day, I wouldn't put all my eggs in the x86 basket just to be safe.

Counter-Strike 1.5 can be played with FTEQW engine and complimentary libre FreeCS gamedata - it still needs its original assets though. I can test FreeCS on x86, ARMv6J, Cell, AMD64 and if luck permits, UltraSPARC III. But in order to test on such a wide array of machines, I need open source software that is on of itself architecture agnostic - I can expect that much from FTEQW, but not e_wise. It was written 19 years ago with x86 and DOS-era OSes in mind. Counter-Strike 1.5 at its core is a MOD for Half-Life, so it makes sense to use a compatible free software engine.

I know about the source code. I wrote in the OP "I'm unfortunately not very good with Pascal, I don't know what version of Pascal is required to build e_wise, so fixing or even adapting it to my use case is really harder than using it as-is.". The source code is ridden with independent implementations of various basic functions and there are even bits of x86 assembly stuck in it. LFNs are not going to be an issue with e_wise, because it produces files called 00000000.EWI, 00000001.EWI, ..., 99999999.EWI, and 2 complimentary files - 00000000.BAT and 00000000.TXT. In step 4.6, I have 2 lines of Bash code that re-create the directory structures (easily deduced from 00000000.BAT contents) and properly rename and relocate .EWI files (also easily deduced from 00000000.BAT) - this part is run on Linux, so LFNs are again not an issue.

Abit VP6 w/ YT BIOS, 2x Tualatin-S with Connectland northbridge heatsinks, 4x512MB SDRAM PC133 CL2, SanDisk 32GB CF 160/120MB/s R/W, Plextor PX-800A, Voodoo 5500 AGP with passive mod, Voodoo 1, Yamaha YMF724F, D-Link DGE 528T rev B

Reply 8 of 36, by DosFreak

User metadata
Rank l33t++
Rank
l33t++

So let me get this straight.....
You need:
The original counterstrike files
e_wise
DosBox
Extract the files from the installer inside dosbox to EWI
Rename the EWI files to the real name outside dosbox

I'm assuming you are using a script to do all this to provide to the user and not repacking the content yourself since potentially that is not allowed.
Has anyone bothered to ask Valve if there are any issues with repacking this content and providing a download?
If no response from Valve does the Internet care?
So assuming you can't repackage:
You can't use: "csv15full.exe /x .\extract" in wine since not all platforms have wine? but they want to play a windows game? or maybe they just want to extract for a different system or extract for research purposes?
You are using the DOS version of e_wise since the DOS version of e_wise can run in DOSBox on different operating systems and hardware assuming that DOSBox is in the repo for the distro or can be compiled and used with that hardware and OS. You can't use a linux equivalent since possibly none exists and you can't use the Windows version since possibly you cannot run the Windows version on the same hardware and operating systems as DOSBox can but the assumption is that the original counterstrike files, DOSBox and that the DOS version of e_wise will be runnable and available on the Internet forever.
If it's that important to you then it really needs to be looked at for 7zip and the like to support the compression format which would greatly simplify this.

This kind of reminds me of something similar that came up on the wine bugtracker where a patch for a Windows game does the following:
Uses an Installshield installer to call a dos zip program to update the zip files in the installed game folder.
The easiest solution would be to replace the dos zip program with a win32 one and really Wine should have the ability to identify and do this itself, instead the bug has been open for a 1yr+

If you just want a notification without requiring code changes to DOSBox then you could create a file inside DOSBox after the files have unpacked that the host would check to verify that the process is complete or I guess just checks for the existence of the last file extracted and that it's the correct hash.DOSBox is an emulator so the unpack process will be slow even if you use the fastest settings.

DOSBox Compilation Guides
DosBox Feature Request Thread
PC Game Compatibility List
How To Ask Questions The Smart Way
Running DRM games offline

Reply 9 of 36, by moog

User metadata
Rank Newbie
Rank
Newbie

>I'm assuming you are using a script to do all this to provide to the user and not repacking the content yourself since potentially that is not allowed.
Yes. csv15full.exe was released as a freeware offering by Counter-Strike crew consisting of its developers back in June 12, 2002, before they were acquired by Valve and subsequent release 1.6 was a paid offering on Steam.
>Has anyone bothered to ask Valve if there are any issues with repacking this content and providing a download?
No. It's faster that way. It's months waiting for their legal team to respond (assuming they even will) + work to refactor the script and repackage the assets, versus a ready script I made in one evening that is perfectly legally compliant.
>If no response from Valve does the Internet care?
No. Their legal team however, will care. I find it really coy and disrespectful that in general legal teams won't bother responding to permission requests but will rain hard on you if you do your own thing.

>You can't use: "csv15full.exe /x .\extract" in wine since not all platforms have wine?
They do, it's just non-trivial to accomplish. Wine is essentially an API wrapper for Windows API - as its name states, there is no emulation going on whatsoever. Had it been the case it wouldn't be fast.
As stated before, in order to use Wine on non-x86 platforms, you need:
- QEMU x86/amd64 userland runtime
- x86/amd64 Linux runtime that satisfies runtime dependencies for Wine, which is a lot
- Wine cross-compiled for x86/amd64
With these elements in place, you call QEMU to run Wine with parameters to run a Windows executable. QEMU can either emulate an entire target computer, virtualize a similar computer, or run foreign architecture executables. To do the lattermost thing, QEMU will open the Linux x86 executable of Wine and start resolving its dynamic links and request those libraries - this is why a sufficient Linux runtime is required.

Providing all of this on Gentoo Linux is challenging because normally you can't cross-compile a package. You need to install crossdev (it is a package), and then manually call crossdev to create a cross-compiler toolchain. This toolchain will naturally cross-compile things, but crossdev is nice enough that it also gives you a separate Portage registry, which you can use to cross-compile an entire instance of Gentoo Linux for that architecture. And this is how we come to installing Wine for that instance. Unfortunately, following up on Portage instances created by crossdev is not something that native Portage is aware of, so it's not something I can solve with an ebuild. Or at least, not solve it in a nice way.

>assuming that DOSBox is in the repo for the distro
Yes it is. Has been for as long as I can remember.

>If it's that important to you then it really needs to be looked at for 7zip and the like to support the compression format which would greatly simplify this.
It's not really a matter of the compression format, but the layout of WISE installers in the context of them being obscure archive formats with self-extracting code attached. In the source code of e_wise, we can see that it's checking crc32 a lot and it's using zlib compression. This is further proved by binwalk, which indicates that there are zlib compressed files inside. It just can't get a clear picture and see there are in fact many of them. I tried 7z, unrar, unar, binwalk, UE1, UE2, and they all flopped.

>Uses an Installshield installer to call a dos zip program to update the zip files in the installed game folder.
There's UnShield. It supports some InstallShield installers, but not all. I tried it on GTA2 a year ago and it flopped, but there was something else that worked with it nicely.

>If you just want a notification without requiring code changes to DOSBox then you could create a file inside DOSBox after the files have unpacked that the host would check to verify that the process is complete or I guess just checks for the existence of the last file extracted and that it's the correct hash.
I have considered this, but I would like the updates to occur as soon as they are visible in DOSBox. For that, I would need to:
1. Start DOSBox in the background.
2. Actively check if DOSBox is still running.
3. While it's running, keep an instance of tail -f locked in on the designated output file.
4. Kill tail when DOSBox dies - peacefully or not.
This approach has the problem of doing threads in Bash which is already a pretty bad idea. I could write Python code and use it then, but that requires more time. Also, redirecting the output and error to a file would mean that I cannot distinguish between output and error anymore since it's all in one basket now.

>DOSBox is an emulator so the unpack process will be slow even if you use the fastest settings.
Everyone who uses Gentoo is patient.

Abit VP6 w/ YT BIOS, 2x Tualatin-S with Connectland northbridge heatsinks, 4x512MB SDRAM PC133 CL2, SanDisk 32GB CF 160/120MB/s R/W, Plextor PX-800A, Voodoo 5500 AGP with passive mod, Voodoo 1, Yamaha YMF724F, D-Link DGE 528T rev B

Reply 11 of 36, by Ringding

User metadata
Rank Newbie
Rank
Newbie

Going by the files in the source archive, it seems to have been built with Virtual Pascal. Which I don't know, but I had experience with Turbo Pascal back then. And I speak German, so I can at least easily understand all the messages, comments and identifier names in there.

Reply 12 of 36, by moog

User metadata
Rank Newbie
Rank
Newbie
Ringding wrote on 2021-02-05, 08:37:

Are there free data files available somewhere that I could test e_wise on?

Yes. Counter-Strike 1.5 is free of charge - https://archive.org/download/counter-strike-1 … 5/csv15full.exe

If anyone would like to also take a look, here's what I published:
https://gitlab.com/src_prepare/src_prepare-ov … data-1.5.ebuild
https://gitlab.com/src_prepare/src_prepare-ov … ass/wise.eclass

Abit VP6 w/ YT BIOS, 2x Tualatin-S with Connectland northbridge heatsinks, 4x512MB SDRAM PC133 CL2, SanDisk 32GB CF 160/120MB/s R/W, Plextor PX-800A, Voodoo 5500 AGP with passive mod, Voodoo 1, Yamaha YMF724F, D-Link DGE 528T rev B

Reply 15 of 36, by moog

User metadata
Rank Newbie
Rank
Newbie
latalante wrote on 2021-02-05, 11:36:

I already said why! This file is not part of the original official release and therefore it's a legal liability to host something like this. See http://web.archive.org/web/20031118182924/htt … t/mod_full.html
In case this mirror goes under, there are plenty more.

Abit VP6 w/ YT BIOS, 2x Tualatin-S with Connectland northbridge heatsinks, 4x512MB SDRAM PC133 CL2, SanDisk 32GB CF 160/120MB/s R/W, Plextor PX-800A, Voodoo 5500 AGP with passive mod, Voodoo 1, Yamaha YMF724F, D-Link DGE 528T rev B

Reply 17 of 36, by Ringding

User metadata
Rank Newbie
Rank
Newbie
moog wrote on 2021-02-04, 19:56:

Between line 30 and 40 there's a considerable amount of time taken. It would be nice to let us know what's exactly happening. Normally when e_wise is run in DOSBox, it looks like in the attachment.

Now that I’ve seen it myself, I know what you’re talking about. However, IIRC without ansi.sys, DOS stdout does not have the ability to go back and overwrite what was already written (which is what e_wise does for its progress output). So I would expect it to write directly to video memory, which cannot be forwarded to stdout anyway.

Reply 19 of 36, by moog

User metadata
Rank Newbie
Rank
Newbie
Ringding wrote on 2021-02-05, 14:30:
moog wrote on 2021-02-05, 02:53:

Everyone who uses Gentoo is patient.

😁

This is really really slow, though…

It really depends on your setup. I can understand that compiling LibreOffice on a 2009 laptop, even with SSD and maxed out RAM can take at least 10 hours... but with a current-day Ryzen and 32GB RAM you can do it in 40-50 minutes if you know what you're doing.

Ringding wrote on 2021-02-05, 15:08:

There is also WiseUnpacker, which should be up to the task (not tested yet) and might be portable enough.

This actually looks promising, but needs more work. I've managed to compile this for Mono without issue. It's just that:
1. After extraction, it requires the user press a key. Come on. This is no way for a script utility to conduct itself.
2. I've got a dump full of files and no info about directory structures and filenames so I can even recover from this state:

'MAINDIR\cstrike\maps\de_dust2.txt'
WISE0000
...
WISE042F

Abit VP6 w/ YT BIOS, 2x Tualatin-S with Connectland northbridge heatsinks, 4x512MB SDRAM PC133 CL2, SanDisk 32GB CF 160/120MB/s R/W, Plextor PX-800A, Voodoo 5500 AGP with passive mod, Voodoo 1, Yamaha YMF724F, D-Link DGE 528T rev B