Replace the tools/lzcomp compressor?

[Rangi42]

I'd like to discuss replacing tools/lzcomp (our vendored copy of ax6's repo) with lzss.c (after editing it to use our tools/common.h utilities).

Pros:

• lzss.c is shorter (320 lines of code, versus 995 dense lines across 13 files).
• lzss.c is easier to read as a self-documenting explanation of how LZ compression happens, and what the flags in gfx/lz.mk would be varying.
• lzss.c would be easier to edit if someone wants to change how the whole algorithm behaves (i.e. changes that affect the home/decompress.asm decompressor too). (For example, LZ: add pack16 literals and extended opcodes Rangi42/polishedcrystal#1343.)

Cons:

• lzcomp will automatically give users better compression just by removing the gfx/lz.mk flags and letting it brute-force pick the best heuristic method.
  • Caveat: lzcomp is not actually capable of the best compression; "dpcomp" does better or at least as well for every file I've tested it on.
• Users might be used to running ./tools/lzcomp, so this could be disruptive.

I'd support us providing a dpcomp-only program in a separate repo, so users who want better compression can download it and replace lzss. That way the "best" compressor could be developed independently of pokecrystal, pokegold, pokegold-spaceworld, and other projects like pokesilver. It could even add opt-in support for other LZ-related formats, like these variants or the experimental new opcodes linked above. Meanwhile the compressor bundled with pokecrystal would be just the essentials.

[Rangi42]

lzss.c does not currently support decompressing LZ files, so that would have to be implemented. Its "verbose" output mode could also be removed.

[vulcandth]

It is certainly easier to read than the multiple folder w/ files. I support this.

[mid-kid]

I think doing the best possible thing once users remove the scaffolding holding matching in place makes the most sense. If dpcomp indeed generates better results, then that should be integrated with the new lzss.c, and we should use the dpcomp fork even if we keep using lzcomp.
Otherwise, I'm ambivalent towards using lzcomp vs lzss.c, but most of those extra LoC in lzcomp are extra features, such as decompression and visualizations of the byte stream, so it's not exactly an apples to apples comparison. The dpcomp fork reduces its size by ~500 lines by removing the other compression algorithms [EDIT: we actually need those compression methods for matching...]

[Rangi42]

Here's the source for just dpcomp.c. Possibly it could be combined with a more lzss-like approach for matching and optimal compression in one program.