PC engine compression schemes

[malducci]

NightWolve, is there a common compression scheme used for gfx in PCE CD games? Most CDs I've looked at don't have any gfx compression, but a few do. Gate of thunder is one of them that I'm trying to grahpic hack.

 I figured since about 99.9% of the graphics in Hucards appear to have some sort of graphic compression, that it might be part of an Hudson library and possible used with some CDs title.

[NightWolve]

I couldn't tell ya. I could tell you that Falcom's text compression was custom work as far as we know. Anyhow, dealing with compression on PCE CD was never my specialty. It was Neill Corlett that uncovered the text decompression, and Dave Shadoff reversed it for me so I could compress the text blocks back into the data track. As such, I didn't have to deal with it. Dave's lurking around in the forum it so happens, so he'd be one of the few left that might be of help. I assume you already have some technical knowhow in programming though, cause honestly, if not, you're barking up the wrong tree with this.

[Dave Shadoff]

Honestly, very few titles have graphic compression, but some do.

The graphics can be some pretty weird formats in order to go into VRAM though, such as 1-bit and so on.

There are a few titles with run-length encoding and simple repeat-this-sequence-from-a-trailing-buffer, but they are all a bit different from each other.

But it might help if I knew what you were trying to accomplish.

[malducci]

it might help if I knew what you were trying to accomplish.

 For now I just want to decompress the graphics, alter them, and recompress them. I've figured some games might store certain tiles and sprites as 1-bit, 2-bit, and  3-bit as a one type of compression - depending on the number of colors used. I know byte alignment can be off from one tile set to the next.

 I'm actually using Tmod2 with your PCE tile/sprite plugin  

Anyways reconstructing/analyzing a save state instead of using a debugger - isn't exactly my idea of fun   Here's hoping Mednafen team add a debugger

I assume you already have some technical knowhow in programming though, cause honestly, if not, you're barking up the wrong tree with this.

 I've programmed a few demos for the PCE and SGX hardware in assembly, so I'm at the point were I feel pretty confident - though my experience with compression algorithms is zero.

 I actually had this crazy idea: running a hucard or cd game on a real SGX (in SGX mode) - with added/alter coded for when pausing the game, you could bring up your own (debug) code and use the second VDC to display the content of  memory like a hex editor 

[Dave Shadoff]

Is the name of your project's game a secret ?
I could at least tell you if I did any preliminary investigation on it...

[NightWolve]

That would be Gate of Thunder.

[malducci]

Yeah, Gate of Thunder.

[NightWolve]

I've programmed a few demos for the PCE and SGX hardware in assembly, so I'm at the point were I feel pretty confident - though my experience with compression algorithms is zero.

See, I'm in the same boat really. But here's the kicker, with PC games, what I do is track down the decompression code in the interactive disassembler software that I'm using, copy out all the Intel Assembly for it (That means I gotta find all the functions it calls and track every other one down that they call, recursively, etc.) then paste all of it into an Assembly module for a VC++ project where I reuse it. Once I understand how to interface to it (how it accepts the input buffer, the output buffer, etc.), I can have the rest of my code in C++ and call upon it when needed. This is how I built the NSIS plugins in my Ys I, Ys II patches that extract all the image files from the Falcom data libraries when you run them. The same technique is what I used to decompress the script for ED6. Ys Felghana & Ys VI actually used the standard zlib, and I got the info and code from elsewhere, so I didn't have to use my crude method.

Anyhow, the point is, with the tools I have for Windows, I can harness my strengths, and bypass my weaknesses. I don't have to know how their decompression code works. I just need to a) find all the assembly for it, and b) discern how to interface with the top-level function, that is, discern how its parameters work, etc. The only issue then is if I needed to compress the files back, but I've taken care of that by intercepting the functions that use compressed images/text, etc. post decompression, and just overwriting the decompressed results with my uncompressed replacement files. So I still can workaround that issue in a rather clever fashion if I do say so myself. Yes, I will say so in fact! Quite clever!

Now, with PC-Engine, I don't have this kind of power. I must invoke the power of Dave in order to reverse the decompression algorithm. Plus, there just aren't the tools for the platform to get very far unless your experience is at Dave's or Neill Corlett's level.

[malducci]

but I've taken care of that by intercepting the functions that use compressed images/text, etc. post decompression, and just overwriting the decompressed results with my uncompressed replacement files.

So after you have decompressed the data/image/text, you replace the old compression function and interface a new function that reads from the uncompressed files - using the same calling/handling parameters as the original? That's pretty cool... and clever

Now, with PC-Engine, I don't have this kind of power. I must invoke the power of Dave in order to reverse the decompression algorithm.  Plus, there just aren't the tools for the platform to get very far unless your experience is at Dave's or Neill Corlett's level.

Yeah, not having the ability to use a debugger for PCE, means a lot more work than normally required and ofcourse tools wouldl have to be made as well.



The way I figure it, I need to do one of the following:


1) Write some test code and figure out where everything is placed in the emu's save state file. Use the CPU' PC located in the file to provided a starting point for disassembling from the save state file.

2) Take random save states - rinse and repeat until everthing lines up and all code/functions have the correct starting/beginning address alignment.

3) Then after straining my eyes for hours/days/weeks crossing my fingers that I actually recognise which code is the decompression code (all because I can't do a triggered save state to get the actual point I looking for originally). 

4) Finish the dynamic dissassembler I have written. I need to added dummy read/write, skip instruction, and follow branch/jump instruction - for stepping through the code.

or

 Try to add a basic debugger for Mednafen since the source is available. But the idea of jumping into someone elses code (even with all the comments you can shake a stick at) and trying to make heads or tails of what is what, gives me a headache just thinking about it. On top of that, it's written in C++ which I've written in, but [inothing at or near a professional level/understanding.

[NightWolve]

So after you have decompressed the data/image/text, you replace the old compression function and interface a new function that reads from the uncompressed files - using the same calling/handling parameters as the original? That's pretty cool... and clever

Not quite. I let the game code proceed normally, that is, it goes into its library files, finds the original compressed image/data file(s), whatever, it decompresses to some buffer, and then continues along its merry way. The decompression buffer is what I target. My hook is at some safe point where I can both detect the file that I want replaced was just processed, and then overwrite the contents of this decompression buffer with the external files that I edited. So the hook monitors the files being decompressed, and if it finds that one exists externally in a subfolder where I've placed it in, it then reads that into the decompression buffer, then safely returns control back to the program. If done correctly, the game code won't know the difference, and voila!

Now it should make sense. In this way, I never have to rebuild the libraries with recompressed edited replacements nor do I have to learn how to reverse the decompression algorithm. I've been using this for everything. Ys I & II Complete, and for the upcoming Ys Felghana and Ys VI patches. Though for the latter two, I have the zlib DLL so I could recompress, but it is much faster and easier to just leave the image files uncompressed in a subfolder and read them into the decompression buffer at the right point. Works like a charm every time, every game.