Have you looked at zx7?

Einar Saukas made it as a decompressor for ZX Spectrum. The uncompress routine is very small (69 bytes in assembler, and not too much more while decompressing), but the compression is made offline (using a PC). It is used often to store code, graphics and text and decompress it on-the-fly...

Pros:
- Very small decompressor.
- Very small memory usage.
- It can give you some ideas.

Cons:
- Compression is made on a PC (do you really need to compress data on your 8-bit system?).
- It is LZ77/LZSS, as you wished.
- Not the best compression ratio, but still very efficient. Other works from Einar Saukar are more efficient, and can help you too.

I have ZX0 and based some of my techniques on it, and right now, one of them is doing better than its main competitor. I don't need to do it on an 8-bit system. In fact, I plan to implement it on 32-bit Windows and ANSI C. I'm just working on it for 8 bits for now. My problem now is that right now, it's very slow. Hash tables helped but not enough, and the table requires well over 8k of memory and produced very poor compression ratio. That's why I'm asking.

I looked at ZX7 and didn't find any source code. 🙁 I may come back soon to ask for more ideas but am not ready for them right now.

This is a little premature, but how can I implement BPE efficiently? I'm currently using a LZ77 variation to do BPE's job but think the original's tokenization design should be better. I don't want to go through the file many times and keep tabs of every possible token, though. A better approach doesn't have to produce 100% best results but require little RAM and CPU time. As I said, I want to work on BWT soon. I should still have code left over from a previous experiment with it, but it was very inefficient.