VOGONS

Common searches


First post, by Peter Swinkels

User metadata
Rank Oldbie
Rank
Oldbie

Notice:
I moved everything I felt that was relevant into this thread. The most up to date code can be found here (bottom post). The links in the top post may be ignored. Thank you. 😀

==========================================================================================================================================================================

I am trying to extract image resources from the game and could use some help:

https://encode.su/threads/4330-SAGA_RLE1-deco … 84271#post84271

What I got so far isn't working properly. It's probably some really dumb mistake I might very well catch myself sooner or later, but some help would be welcome anyway.

Last edited by Peter Swinkels on 2024-11-08, 10:03. Edited 2 times in total.

My GitHub:
https://github.com/peterswinkels

Reply 1 of 22, by Peter Swinkels

User metadata
Rank Oldbie
Rank
Oldbie

Here's a post on another forum just in case:
https://www.vbforums.com/showthread.php?90882 … 364#post5661364

My GitHub:
https://github.com/peterswinkels

Reply 2 of 22, by vstrakh

User metadata
Rank Member
Rank
Member

It feels like there are multiple different formats of resources.
Some are clearly compound, with a TOC at the beginning, listing offsets of subresources and some flags/info (e.g. res_0227.res file). And that TOC is definitely not compressed because the records structure is solid.
Some look like they're 320x200 images, but there are cases when such images start with immediately width/height fields (res_1549.res), and in other cases it has some marker in front of it (res_1749.res), while both ways the file size is smaller than the expected 64000, which suggests both were compressed.
Are you sure the ripping was correct, and the same RLE1 is applicable to all files you've shared in res.zip?

Reply 3 of 22, by Peter Swinkels

User metadata
Rank Oldbie
Rank
Oldbie

Indeed not all of them are images and therefore I am focussing on files where if the header data is interpreted as containing the width and height I get result of 320x~136 and I am checking whether the files are large enough to contain a palette. How did you check whether a file contained a TOC? And what did you find out about the record structure? A marker? What do you mean?

Indeed, I expect most alleged image files to be smaller than 64000 bytes because of the SAGA_RLE1 compression. And not all images will be 320x200. In fact the reason I am checking for a height of ~136 instead of 200 is because I expect background images for scenes to be no higher. The game's interface is located below the 136th line.

No other compression algorithm was mentioned so I am assuming RLE1 is the only one used, for images at least.

And the ripping appears to produce legitimate resource files. I tried sagares.py on every *.rsc file I could find and appeared to produce valid resource files. I examined a few files with a binary viewer and it looks legit.

PS:
And to get back to the multiple types of resources, it appears the game assumes a certain resource will be of a given type rather then using any system of type indicators. Which in a worst case scenario would be mean decompiling the executable.

My GitHub:
https://github.com/peterswinkels

Reply 4 of 22, by vstrakh

User metadata
Rank Member
Rank
Member
Peter Swinkels wrote on 2024-11-03, 10:44:

How did you check whether a file contained a TOC? And what did you find out about the record structure? A marker? What do you mean?

You look at the file in hex (preferably have 16 bytes per row, this often simplifies things, as the data structures tend to align on 4/8/16 bytes for performance reasons). You notice some regular patterns. You look in the numbers, try to interpret some as offsets, and then you notice some number in the patter points around the edge where pattern is not observed anymore, etc. You basically look for patterns and discontinuities in the data entropy/randomness. But you still need some experience of building similar stuff to get it recognized by the looks of it.

Peter Swinkels wrote on 2024-11-03, 10:44:

Indeed, I expect most alleged image files to be smaller than 64000 bytes because of the SAGA_RLE1 compression. And not all images will be 320x200. In fact the reason I am checking for a height of ~136 instead of 200 is because I expect background images for scenes to be no higher. The game's interface is located below the 136th line.

The said files I've mentioned has 40h,01h,0c8h,0h as first four bytes. That's two words in Little Endian byte order, translated to decimal as 320 (140h) and 200(0c8h), that's why it's highly suspicious of being a pics of a commonly used resolution.

Reply 5 of 22, by vstrakh

User metadata
Rank Member
Rank
Member

Like see at res_0227.res. The beginning is:

0000000000: 00 00 30 04 FF FF 00 00 │ 0F 00 30 04 FF FF 30 00    0♦    ☼ 0♦  0 
0000000010: 0F 00 5D 05 FF FF 00 00 │ 12 00 8A 06 FF FF 00 00 ☼ ]♣   ↕ è♠
0000000020: 0F 00 E1 07 FF FF 20 00 │ 17 00 15 09 FF FF 20 00 ☼ ß•   ↨ §○
0000000030: 11 00 D5 0A FF FF 00 00 │ 0F 00 32 0C FF FF 30 00 ◄ ╒◙   ☼ 2♀  0
0000000040: 0F 00 5F 0D FF FF 30 00 │ 0F 00 8C 0E FF FF 30 00 ☼ _♪  0 ☼ î♫  0
0000000050: 11 00 B9 0F FF FF 00 00 │ 11 00 0A 11 FF FF 30 00 ◄ ╣☼   ◄ ◙◄  0

See the regularly repeated pattern with the period of 8 bytes?

Now you see some bytes are similar across those 8-byte periods, and some byte pairs form monotonically incremented numbers. Like 30 04, 5D 05, 8A 06. Those a suspicious of being pointers. The lengths would be jumping all over the place, but these are growing, suggesting they're pointing somewhere in the file itself.
And if you look around the offset 430h (the first pair of 30 04, being a word stored in LE order), you will see next:

00000003E0: 11 00 09 A9 09 F0 34 00 │ 17 00 88 AA 0C C0 30 00  ◄ ○⌐○≡4 ↨ ê¬♀└0
00000003F0: 11 00 7B AC 03 30 30 00 │ 17 00 20 AE 90 F9 30 00 ◄ {¼♥00 ↨ «É∙0
0000000400: 11 00 0D B0 9F 09 30 00 │ 10 00 94 B1 9F 08 32 00 ◄ ♪░ƒ○0 ► ö▒ƒ◘2
0000000410: 13 00 ED B2 EC CE 24 00 │ 13 00 76 B4 CF 88 24 00 ‼ φ▓∞╬$ ‼ v┤╧ê$
0000000420: 0F 00 E4 B5 9F 09 40 00 │ 0F 00 11 B7 80 E8 40 00 ☼ Σ╡ƒ○@ ☼ ◄╖ÇΦ@
0000000430: 0E 04 08 08 08 08 0E 0C │ 08 08 08 0C 0C 0C 0C 08 ♫♦◘◘◘◘♫♀◘◘◘♀♀♀♀◘
0000000440: 08 0C 0A 0C 08 08 0C 0C │ 0C 0C 0C 0C 0C 0C 08 08 ◘♀◙♀◘◘♀♀♀♀♀♀♀♀◘◘
0000000450: 0A 08 10 08 08 0C 0C 0C │ 0C 0C 0C 0C 0C 0C 0C 0C ◙◘►◘◘♀♀♀♀♀♀♀♀♀♀♀
0000000460: 0C 08 08 08 06 14 08 08 │ 0C 0C 0C 0C 0C 0C 0C 0C ♀◘◘◘♠¶◘◘♀♀♀♀♀♀♀♀
0000000470: 0C 0C 0C 0C 0C 0C 0C 0C │ 08 08 06 04 18 08 08 04 ♀♀♀♀♀♀♀♀◘◘♠♦↑◘◘♦

See how the data at the offset 0430 stops looking like periodic 8-byte structures, instead there's some uniformly filled something, not unlike a background in the hypothetical sprite/image.

Reply 7 of 22, by Peter Swinkels

User metadata
Rank Oldbie
Rank
Oldbie

@ystrakh:
You mean
D:\Other\RES\RES_1529.RES
D:\Other\RES\RES_1540.RES
D:\Other\RES\RES_1543.RES
D:\Other\RES\RES_1546.RES
D:\Other\RES\RES_1549.RES
D:\Other\RES\RES_1552.RES
D:\Other\RES\RES_1557.RES
D:\Other\RES\RES_1561.RES
D:\Other\RES\RES_1565.RES
D:\Other\RES\RES_1757.RES
D:\Other\RES\RES_1763.RES
D:\Other\RES\RES_1769.RES
D:\Other\RES\RES_1774.RES
D:\Other\RES\RES_1778.RES
D:\Other\RES\RES_1790.RES
D:\Other\RES\RES_1791.RES
D:\Other\RES\RES_1792.RES
D:\Other\RES\RES_1793.RES

Are likely 320 x 200 images?

My GitHub:
https://github.com/peterswinkels

Reply 8 of 22, by Peter Swinkels

User metadata
Rank Oldbie
Rank
Oldbie

@kmeaw: Thank you, thank you! You're a life saver, while the code still doesn't produce proper images the results have improved greatly after fixing the backtracking code! Thanks again! 😀

My GitHub:
https://github.com/peterswinkels

Reply 9 of 22, by vstrakh

User metadata
Rank Member
Rank
Member
Peter Swinkels wrote on 2024-11-04, 09:32:
You mean D:\Other\RES\RES_1529.RES .. Are likely 320 x 200 images? […]
Show full quote

You mean
D:\Other\RES\RES_1529.RES
..
Are likely 320 x 200 images?

Yes, it feels so. Looking at the res_1529.res I'd even suspect the palette is there. See how the data looks different around the offset 308h.
Assuming the first 4 bytes are the header, then you can see something that feels like going in triplets (r/g/b values?). The size of the 256-color palette should be 768 bytes (or 300h in hex). Looking in the file you see that around 308h there's something radically different, it's repeated, suggesting a large area of same color, rle-encoded.

Also look at res_1530.res. This has 320/200 words, but it starts with 44h, and the the statistical distribution of values in the file looks quite different from 1529.res. It might be a picture, but the format is definitely different. It might be something else, related to the image, but either encoded differently, or being a part of some animation scheme.
Maybe it's just the picture, but fully compressed, including the pallette part, unlike res_1529.res which apparently had the palette in plain.

Reply 10 of 22, by Peter Swinkels

User metadata
Rank Oldbie
Rank
Oldbie

RES_1529 decodes to a severely distorted picture of the globe shown at the very beginning when the game starts.

My decoder can't handle RES_1530 at all.

It's probably a really basic screw up on my part but I am completely stumped at the moment.

My GitHub:
https://github.com/peterswinkels

Reply 11 of 22, by kmeaw

User metadata
Rank Member
Rank
Member
Peter Swinkels wrote on 2024-11-04, 20:56:

RES_1529 decodes to a severely distorted picture of the globe shown at the very beginning when the game starts.

My decoder can't handle RES_1530 at all.

It's probably a really basic screw up on my part but I am completely stumped at the moment.

I have just built my version that decodes RES_1530 fine. It is in Ruby but the transformation that you are missing is on the line 86.

Attachments

  • Filename
    ite.zip
    File size
    984 Bytes
    Downloads
    15 downloads
    File comment
    ITE background image resource converter (RES to XPM)
    File license
    GPL-2.0-or-later

Reply 13 of 22, by kmeaw

User metadata
Rank Member
Rank
Member
Peter Swinkels wrote on 2024-11-05, 04:22:

How does your decoder handle the other resources?

It only decodes background images which have height as a multiple of 4, I never tried to make the implementation work with other types.
Here is a test run with large files from your res.zip.
I think your decoder is almost there, you just need that final transformation for Decoded(Pixel) index: idx = (((y >> 2) * width + x) << 2) + (y & 3)

Attachments

  • frame.jpg
    Filename
    frame.jpg
    File size
    339.83 KiB
    Views
    491 views
    File license
    Public domain

Reply 14 of 22, by Peter Swinkels

User metadata
Rank Oldbie
Rank
Oldbie

@kmeaw: you are a real life safer. 😀 I will look at my decoder and your and update it asap. 😀

My GitHub:
https://github.com/peterswinkels

Reply 15 of 22, by Peter Swinkels

User metadata
Rank Oldbie
Rank
Oldbie

@kmeaw: Your Ruby decoder works! You're a life safer!

And I haven't forgotten about fixing my own code. My decoder is making the wrong assumptions about the way the pixels are arranged, correct?

My GitHub:
https://github.com/peterswinkels

Reply 16 of 22, by kmeaw

User metadata
Rank Member
Rank
Member

Yes, your decoder assumes that the pixels are arranged as they do in a linear (chunky) framebuffer. Instead they are arranged as group of vertical columns of 4 pixels.
After a "w" by "h" image is decompressed, it goes like this in (x,y) coordinates: (0,0), (0,1), (0,2), (0,3), (1,0), (1,1), (1,2), (1,3), ..., (w-1, 0), (w-1, 1), (w-1, 2), (w-1, 3), (0, 4), (0, 5), (0,6), (0,7), (1,4), ..., (w-1, h-4), (w-1, h-3), (w-1, h-2), (w-1, h-1).

Reply 17 of 22, by Peter Swinkels

User metadata
Rank Oldbie
Rank
Oldbie

It nearly works - EDIT: there is still some distortion in the output images. I will have a closer look ASAP. Probably yet another basic mistake on my part...

'https://wiki.scummvm.org/index.php?title=SAGA/Datafiles/Background_%26_Interface_Image
'https://wiki.scummvm.org/index.php?title=SAGA/Datafiles/SAGA_RLE1

'This module's imports and settings.
Option Compare Binary
Option Explicit On
Option Infer Off
Option Strict On

Imports System
Imports System.Collections.Generic
Imports System.Convert
Imports System.Drawing
Imports System.IO
Imports System.Linq

'This module contains this program's core procedures.
Public Module CoreModule

'This procedure is started when this program is executed.
Public Sub Main()
For Each Item As String In Directory.GetFiles("D:\Other\SAGA RES DECODER\RES", "*.res")
Try
ConvertToPNG(Item, $"D:\Other\{Path.GetFileName(Item)}.png")
Catch
End Try
Next Item
End Sub

'This procedure attempts to read a resource as an image and convert it to a *.png file.
Private Sub ConvertToPNG(InFile As String, OutFile As String)
Dim Backtrack As New Integer
Dim Bit As New Integer
Dim BitfieldBytes() As Byte = {}
Dim Data As New Byte
Dim Data1 As New Byte
Dim Data2 As New Byte
Dim DataBytes() As Byte = {}
Dim Decoded As New List(Of Byte)
Dim ImageSize As Size = Nothing
Dim MarkByte As New Byte
Dim Palette As New List(Of Color)
Dim Pixel As New Integer
Dim RunAddend As New Integer
Dim RunCount As New Integer

Using ResData As New BinaryReader(New MemoryStream(File.ReadAllBytes(InFile)))
ImageSize = New Size(ResData.ReadInt16, ResData.ReadInt16)
ResData.BaseStream.Seek(&H4%, SeekOrigin.Current)

For PaletteEntry As Integer = 0 To 255
Palette.Add(Color.FromArgb(&HFF%, ResData.ReadByte(), ResData.ReadByte(), ResData.ReadByte()))
Next PaletteEntry

While ResData.BaseStream.Position < ResData.BaseStream.Length
MarkByte = ResData.ReadByte()
Select Case MarkByte And &HC0%
Case &HC0%
RunCount = MarkByte And &H3F%
Decoded.AddRange(ResData.ReadBytes(RunCount))
Show last 55 lines
               Case &H80%
RunCount = (MarkByte And &H3F%) + &H3%
Data = ResData.ReadByte()
Decoded.AddRange(Enumerable.Repeat(Data, RunCount))
Case &H40%
RunCount = ((MarkByte >> &H3%) And &H7%) + &H3%
Backtrack = ResData.ReadByte()
DataBytes = Decoded.GetRange((Decoded.Count - &H1%) - Backtrack, RunCount).ToArray()
Decoded.AddRange(DataBytes)
Case &H0%
Select Case MarkByte And &H30%
Case &H30%
RunCount = (MarkByte And &HF%) + &H1%
Data1 = ResData.ReadByte()
Data2 = ResData.ReadByte()
BitfieldBytes = ResData.ReadBytes(RunCount)
For Each ByteO As Integer In BitfieldBytes
For BitIndex As Integer = &H0% To &H7%
Decoded.Add(If(((ByteO >> BitIndex) And &H1%) = &H0%, Data1, Data2))
Next BitIndex
Next ByteO
Case &H20%
RunCount = ToInt32(MarkByte And &HF%) << &H8%
RunAddend = ResData.ReadByte()
RunCount += RunAddend
Decoded.AddRange(ResData.ReadBytes(RunCount))
Case &H10%
Backtrack = (ToInt32(MarkByte And &HF%) << &H8%)
RunAddend = ResData.ReadByte()
Backtrack += RunAddend
RunCount = ResData.ReadByte()
DataBytes = Decoded.GetRange((Decoded.Count - &H1%) - Backtrack, RunCount).ToArray()
Decoded.AddRange(DataBytes)
End Select
End Select
End While
End Using

DrawPixelsToPNG(Decoded, ImageSize, Palette, OutFile)
End Sub

'This procedure draws the specified pixel data onto a PNG image using the specified palette.
Private Sub DrawPixelsToPNG(Decoded As List(Of Byte), ImageSize As Size, Palette As List(Of Color), OutFile As String)
Dim ImageO As New Bitmap(ImageSize.Width, ImageSize.Height)

For y As Integer = 0 To ImageSize.Height - 1
For x As Integer = 0 To ImageSize.Width - 1
ImageO.SetPixel(x, y, Palette(Decoded((((y >> 2) * ImageSize.Width + x) << 2) + (y And 3))))
Next x
Next y

ImageO.Save(OutFile, Imaging.ImageFormat.Png)
End Sub
End Module

Also, I am going to try to move everything to this one thread and forget about the other forums.

Attachments

  • Filename
    res.zip
    File size
    4.73 MiB
    Downloads
    10 downloads
    File comment
    Example files so people have something to work with when working with the code. :-)
    File license
    Fair use/fair dealing exception

My GitHub:
https://github.com/peterswinkels

Reply 18 of 22, by kmeaw

User metadata
Rank Member
Rank
Member
Peter Swinkels wrote on 2024-11-08, 09:53:

It nearly works - EDIT: there is still some distortion in the output images. I will have a closer look ASAP. Probably yet another basic mistake on my part...

                           For BitIndex As Integer = &H0% To &H7%
Decoded.Add(If(((ByteO >> BitIndex) And &H1%) = &H0%, Data1, Data2))
Next BitIndex

You need to reverse the order of bits - the decoding should start from MSB, not LSB.
There are two ways to do that:
1) invert the loop direction: = &H7% To &H0% Step -1;
2) check MSB instead and shift the other way: ((ByteO << BitIndex) And &H80%) = &H0%

In the lines 40-42 in my version I am using the first approach.

Reply 19 of 22, by Peter Swinkels

User metadata
Rank Oldbie
Rank
Oldbie

@kmeaw:

Thank you! You solution works. Also, thanks to another kind user over at phatcode.net I found I didn't to substract one from the bracktracking values. 😀

'https://wiki.scummvm.org/index.php?title=SAGA/Datafiles/Background_%26_Interface_Image
'https://wiki.scummvm.org/index.php?title=SAGA/Datafiles/SAGA_RLE1

'This module's imports and settings.
Option Compare Binary
Option Explicit On
Option Infer Off
Option Strict On

Imports System
Imports System.Collections.Generic
Imports System.Convert
Imports System.Drawing
Imports System.IO
Imports System.Linq

'This module contains this program's core procedures.
Public Module CoreModule

'This procedure is started when this program is executed.
Public Sub Main()
For Each Item As String In Directory.GetFiles("D:\Other\RES", "*.res")
Try
ConvertToPNG(Item, $"D:\Other\{Path.GetFileName(Item)}.png")
Catch
End Try
Next Item
End Sub

'This procedure attempts to read a resource as an image and convert it to a *.png file.
Private Sub ConvertToPNG(InFile As String, OutFile As String)
Dim Backtrack As New Integer
Dim Bit As New Integer
Dim BitfieldBytes() As Byte = {}
Dim Data As New Byte
Dim Data1 As New Byte
Dim Data2 As New Byte
Dim DataBytes() As Byte = {}
Dim Decoded As New List(Of Byte)
Dim ImageSize As Size = Nothing
Dim MarkByte As New Byte
Dim Palette As New List(Of Color)
Dim Pixel As New Integer
Dim RunAddend As New Integer
Dim RunCount As New Integer

Using ResData As New BinaryReader(New MemoryStream(File.ReadAllBytes(InFile)))
ImageSize = New Size(ResData.ReadInt16, ResData.ReadInt16)
ResData.BaseStream.Seek(&H4%, SeekOrigin.Current)

For PaletteEntry As Integer = 0 To 255
Palette.Add(Color.FromArgb(&HFF%, ResData.ReadByte(), ResData.ReadByte(), ResData.ReadByte()))
Next PaletteEntry

While ResData.BaseStream.Position < ResData.BaseStream.Length
MarkByte = ResData.ReadByte()
Select Case MarkByte And &HC0%
Case &HC0%
RunCount = MarkByte And &H3F%
Decoded.AddRange(ResData.ReadBytes(RunCount))
Show last 55 lines
               Case &H80%
RunCount = (MarkByte And &H3F%) + &H3%
Data = ResData.ReadByte()
Decoded.AddRange(Enumerable.Repeat(Data, RunCount))
Case &H40%
RunCount = ((MarkByte >> &H3%) And &H7%) + &H3%
Backtrack = ResData.ReadByte()
DataBytes = Decoded.GetRange(Decoded.Count - Backtrack, RunCount).ToArray()
Decoded.AddRange(DataBytes)
Case &H0%
Select Case MarkByte And &H30%
Case &H30%
RunCount = (MarkByte And &HF%) + &H1%
Data1 = ResData.ReadByte()
Data2 = ResData.ReadByte()
BitfieldBytes = ResData.ReadBytes(RunCount)
For Each ByteO As Integer In BitfieldBytes
For BitIndex As Integer = &H7% To &H0% Step -&H1%
Decoded.Add(If(((ByteO >> BitIndex) And &H1%) = &H0%, Data1, Data2))
Next BitIndex
Next ByteO
Case &H20%
RunCount = ToInt32(MarkByte And &HF%) << &H8%
RunAddend = ResData.ReadByte()
RunCount += RunAddend
Decoded.AddRange(ResData.ReadBytes(RunCount))
Case &H10%
Backtrack = (ToInt32(MarkByte And &HF%) << &H8%)
RunAddend = ResData.ReadByte()
Backtrack += RunAddend
RunCount = ResData.ReadByte()
DataBytes = Decoded.GetRange(Decoded.Count - Backtrack, RunCount).ToArray()
Decoded.AddRange(DataBytes)
End Select
End Select
End While
End Using

DrawPixelsToPNG(Decoded, ImageSize, Palette, OutFile)
End Sub

'This procedure draws the specified pixel data onto a PNG image using the specified palette.
Private Sub DrawPixelsToPNG(Decoded As List(Of Byte), ImageSize As Size, Palette As List(Of Color), OutFile As String)
Dim ImageO As New Bitmap(ImageSize.Width, ImageSize.Height)

For y As Integer = 0 To ImageSize.Height - 1
For x As Integer = 0 To ImageSize.Width - 1
ImageO.SetPixel(x, y, Palette(Decoded((((y >> 2) * ImageSize.Width + x) << 2) + (y And 3))))
Next x
Next y

ImageO.Save(OutFile, Imaging.ImageFormat.Png)
End Sub
End Module

My GitHub:
https://github.com/peterswinkels