It's interesting but not complicated.
It's a typical RLE compression, but the interesting part is the additional "skip".
You write one of three main modes:
* Dump: a pure stream of 256-color values
* Run: a single 256-color value repeated N times
* Skip: Just skip N pixels on the frame buffer
There are two variants of the above opcodes: short and long.
The Opcodes (1 signed byte) are:
* Dump: just write positive pixel count up to 127. Then write the stream of pixels right after the opcode.
* Run: Write 0. Then an unsigned byte (up to 255) for run length. Then one byte for the color value.
* Skip: Subtract 0x80 from the positive skip length and write it down.
To go for the long variants, first store 0x80 in the opcode. Then write a long opcode as a signed word (2-bytes) as follows:
* Long skip: just write positive skip length (1-32767).
* Stop: just write 0.
* Long dump: Length is between [1-0x4000). Store that value after adding 0x8000 to it, followed by the stream of one-byte color values right after.
* Long run: Length is between [1,0x4000). Store that value after adding 0xC000 to it, then one byte for the color value.
This is a quick write up by looking at the decoder. The RLE stuff is nothing suprising. But the skip instructions mean you do not write anything on the frame buffer. I think this relies on contents from previous frames, so you have temporal compression going on here (cool!). To use this, you will need to compare frames and identify unchanged rows and encode them with skips.
Good luck!
Disclaimer: I might have made a mistake or two in the above "spec" as I just wrote it by looking at the C code. Not tested. Feel free to correct me.
Turbo XT 12MHz, 8-bit VGA, Dual 360K drives
Intel 386 DX-33, Speedstar 24X, SB 1.5, 1x CD
Intel 486 DX2-66, CL5428 VLB, SBPro 2, 2x CD
Intel Pentium 90, Matrox Millenium 2, SB16, 4x CD
HP Z400, Xeon 3.46GHz, YMF-744, Voodoo3, RTX2080Ti