My Analysis of ED6's Hacking Potential. I've green-lighted it!

[NightWolve]

Well, in these past two days, as I was about to finish the ReadMe for my TurboRip utility, I got sidetracked cause I hate doing ReadMe writeups, and I invested some time into the ED6 project I had decided to work on as you're well aware. I've applied the knowledge gained thus far and attached my JIT (Just-In-Time) string replacement mechanism to the executable, the results of which are below. This is the same hacking strategy that is employed in the upcoming Felghana and Ys VI patches. Now, I was thinking of describing it to you next, but perhaps it's better I don't since if that information works its way around to developers, they might work to specifically defeat such tactics in the future and we don't want them to take away our fun of course. Plus, you probably don't care as much about the cool details that go into this. Anyway, here are some progress shots:




I had some concerns initially, but those have been alleviated. One of the first things you need to examine when deciding if you wanna hack a game or not, is finding out what kind of English ASCII font support there is. Now, in the case of Felghana and Ys VI, both of those use the Windows API CreateFontA to set up a font for the text. This allows me the hacker to very easily support English as well as externally load my own fonts in TTF form. The same was true for Ys II Complete but only for the Diary feature. If you notice with that patch, in your YS2_WIN or YS2_CMP folder, there is a "MagicMedieval.ttf" file in there. That's a regular true type font called Magic Medieval which I use for the Diary replacement font, the one place where I can do it. The rest of the game uses its own custom font tiles, and I never bothered to do anything with that, since the font looked good enough and it would take who knows how long to make changes to it. It was variable-width too, so there was really very little reasons to have messed with something that was already set up and working.

Anyway, the reason I mentioned all that was to give you some background before I mentioned why I was concerned about this game. This game, unlike Felghana and Ys VI, went back to using its own custom font tiles like Ys II Complete. So, that means I have to hope they already built-in English font support that is well-behaved. Otherwise, I would have to do a lot of work to figure out their custom print routines and try to do it myself. But anyhow, as you can see from the progress shots, they already built-in decent English font support, so thankfully, I no longer have to worry about this. If this wasn't the case, I'd likely drop the project since such a task is not especially something I want to have to deal with so much. I've got Felghana and Ys VI in my hat, once those go out of course, and with my past work, I think I've well established my credentials and respect in the fan translation scene.

OK, so now the next step from here is dumping the script for the translators. I'm still searching for it and it's down to a few files, but unfortunately they're very big files, so it's not fun handling them in a hex editor to conduct the search. So far, I just extracted most of the strings from the executable, and that is what's on demonstration with the screenshots.

Well, these translators that wanna work on this better be serious and not back out cause I've already put in some time investment here. So yeah, that's news on this front in case you were, you know, interested... We have a green light for yet another Falcom Fan translation project after all.

[rocket]

Nice work night. didnt have a chance to read all of it but will tonight

[Arjak]

Hey, alright!

You are a TRUE Falcom fan to go through all of this...

[Jest]

It's kinda funny how I never really thought about the problems that font and actual support for English might come up during the translation. Not being very computer savvy, I always just think that you dump the script and then plug in the replacement stuff without having to worry about "support."

I don't know, I really do respect the kind of time and effort that people put forth into this, especially given all my ignorance of how this is all done. I suppose that I should be helping out in the form of monetary donations. So, you can be expecting some from me fairly soon, probably once I get my taxes back.

Also, I was wondering if you'd be doing both chapters of ED6 or just the SC? I want to know if I should pick them both up or just read the script for the first and start up with the second.

Peace Out
Jest

[NightWolve]

Also, I was wondering if you'd be doing both chapters of ED6 or just the SC? I want to know if I should pick them both up or just read the script for the first and start up with the second.

Heh. I wouldn't be adjusting/coordinating your plans around this. We're talking one/two years or more before there's anything to show for it. This is only dealing with the first one and I have not considered SC at all so let's not get carried away here. I make no promises.

[rocket]

glad to hear the font issues worked out ok.  good luck with the translators. do you find the programming style more similar to ys II complete  vs the style used in ys VI and oif?

[NightWolve]

Alright, I had a little trouble with intercepting the main dialogue box function, but I seem to have all that in working order now. That took time away from conducting the search for the script unfortunately. Anyway, here's how the font looks like with that:




It could use some slight x,y coordinate adjusting within the box for the first line, I'd say. Or actually, not really. Just the line height needs to be reduced then. But yeah, the print function is being intercepted at the right earliest point so that whatever code auto-sizes the width and height of the message box gets to process it. If not done correctly at the right point, you'd have the English being printed, but the message box sizing being based on the original Japanese string's computed dimensions. You know, they've got their routines to process a string and compute how much width and height will be needed for it, etc. That's all done for ya. No control codes or anything to have to specify dimensions for each and every box which is nice of course.

[Miles]

I like how the townspeople talk like Darth Vader in this game. :3

(I also like how Nightwolfy had to tell me what that was from) ;_;

[NightWolve]

I found your lack of knowledge very disturbing. Yes... yes, disturbed I was...

[rocket]

You go Yodawolve!  LOL

[NightWolve]

Alright, I've extracted what I believe is the main script archive file. Alllll too easy.

So, the packed script archive is "ED6_DT01.dat" and it has 650 files in it (many are dupes though). The unpacking algorithm that I copied out of the executable (after a few hours of searching today) and reused in VC++ to make a command-line unpacker is about 2,488 lines of raw x86 assembly code. Don't know if it's zlib (which is used for Ys VI & Felghana), some other known compression algorithm, or Falcom's custom work, don't really care since it's of no consequence to me. (Ha ha ha, he he he. I'm just smokin' tonight, what can I say.) The reason being given the string replacement strategy that I've engineered and mastered, I will never have to rebuild hundreds of script files, and in this case, need to have the decompression/unpacking algorithm reversed to compress them back after rebuilding. Out with the obselete, in with the new methods. If I hadn't come up this technique, I would have to rebuild about 2,600 separate script files both for Felghana and Ys VI. Take a guess what a nightmare that would be to manage.

Anyway, 'tis done. No cheap XOR-based encryption, just compression was used. So now I'll work a bit on transferring out the strings into my script editing database, then I'll have the script stats to report: a total string count and the total bytes in size. Can't say I'm too confident with the new translators though, but we'll see how it goes once I can give them this to work on. All in good time.

EDIT #1:

Alright, more good news! I finished creating an UnPacker awesomely enough. Everything and I mean everything is now unpacked from the DAT files. Here's some info on what I know so far:

ED6_DT01.DAT = 487 Script files.

ED6_DT08.DAT = 988 WAVE files for sound effects, etc.

ED6_DT15.DAT = 1,930 DDS Image files, the mother lode! These are in the image format Falcom starting using since the Ys VI engine, and continued with Felghana. Some pretty stuff, but I don't see everything like for the sprites.

Now these might interest you:

ED6_DT17.DAT = This is the ending movie. Rename it to an AVI and you can watch it in Media Player. Don't watch it if you don't want to spoil getting to this ending on your own, obviously.

ED6_DT18.DAT = This movie is a surprise about the future release of the Second Chapter. This too is shown when you finish the game. Same thing, rename it to an AVI for viewing. Oh, and remember to rename them back so as not to break the game when it looks for them.

Interesting tidbit: Both of these movies were made with the help of http://www.tmpgenc.net/ apparently. I wondered why they were disqualified as library DAT files when my UnPacker tried to process them, so then I looked at them in a hex editor and noticed a tag with TMPGEnc close to the start of the file. So then I just renamed them to AVI to see what would happen and voila, they worked! Yay me.

Anyway, the rest of the DAT files output program related files and I don't know nor will have to care what they do or what they're for, etc. We have what we need to proceed. Magical.

[MP83]

Excellent work, I must say. I envy your hacking skills.

[X-Calibar]

This is one beautiful green light~
Awesome lol

I wonder how many or if any: image, movie, sound files need to be edited as well...
[just wondering at this point, but most likely will offer my services when things settle down around here]

Oh and you need a catchy theme song / splash screen intro ~ 
(Maybe just during install?  )

[NightWolve]

Well, I have some more interesting news. I let Gillian have access to my ED6UnPack program and he went and tried it for the new SC... and it works there too! Falcom didn't change its library file system nor the compression employed, so that's nice. This doesn't mean I'm ever gonna work on it, so don't start asking me about it, though... These non-Psycho DeuceBag translators are untested and so when and if we ever finish something, ED6UnPack will be released along with the Patch, so if I swear off ever working on SC, others will have a chance to do something with it then.

[NightWolve]

Oh my oh my... Another update: The good news is, I've extracted most of the script... The bad news is, I've extracted most of the script... How can this be both good and bad news you ask? Well, it's quite, quite, how shall we put it... humungous... Yes, that's a good word for it... 1,946,608 bytes to be exact, or 1,900 KB, and that's 35,130 unique CRC32s for each string. Every string gets a CRC32 computed when I extract it, so I use that to know how many unique strings I have. So yeah, we're talking 35,130 unique strings here! This is unheard of!!! It's 52,201 strings if you count duplicates, so you'll need to do more reading, but be saved of some translating!! A script that's 1,900 KB in size is literally unheard of! Anybody, anybody, wanting to translate all of this would have to be out of their friggin' minds! Out of their minds! LOLOLOL.

To give you an idea, here are the stats I recorded in another post:

Well, I can measure that for you in two forms: String count and total byte size.

Ys I Complete has 1,804 unique strings and is 98,126 bytes in size.
Ys II Complete has 5,327 unique strings and is 205,606 bytes in size.
Ys VI has 4,688 unique strings and is 213,771 bytes in size.
Ys Felghana has 4,356 unique strings and is 204,318 bytes in size.

This script is 9 times as big as Felghana's... The biggest script I've ever heard about during the SNES romhacking days was like 400 KB, for maybe one of those Dragon Quest or Star Ocean games. 1,900 KB is literally off the fucking charts!!!! Crazzzzzzzzzzzzy I say! C-c-c-c-c-c-crazzzzzzzzzzzzzy!!!!

Here, I attached a screenshot. Check out the record count at the bottom left. I'll leave this one for clicking on/off since it's big. Just click the thumbmail below:

[Freakman]

That's because you've only tackled text-light RPG's before. The game I'm currently translating is 2,32mbs big. Taking off duplicate lines, it must be about as big as ED6
And I've already translated more than half of it in ~6 months on my own.
With 3 or 4 translators and enough work, it should be easy (erm) enough.

By the way, I wouldn't mind getting a copy of that extracting program

[NightWolve]

That's because you've only tackled text-light RPG's before.

I've only completed, what, four projects over the years, with two still on the way. I wasn't merely considering my experience with my few projects. That's a limited scope, plus yeah, we know the Ys games are pretty quick so the script is smaller than say a traditional RPG (not an action one). What I was conveying is that I've never heard of a script this big throughout the translation scene by others, in other words.

The game I'm currently translating is 2,32mbs big. Taking off duplicate lines, it must be about as big as ED6
And I've already translated more than half of it in ~6 months on my own.
With 3 or 4 translators and enough work, it should be easy (erm) enough.

Well, I know how much trouble it is editing script and getting images ready for my projects, and they're a fraction of the size this is, so I'm not buying the "easy enough." It'll depend on how much confidence I'll have in reading the results and seeing how well they're being naturalized. I don't like having to do that myself, but we'll see. If you're claiming you're dedicated, I welcome it. Anyway, we're gonna need to get everyone in a meeting when I've prepared the script for editing and talk this over. Btw, English is your second language, correct?

By the way, I wouldn't mind getting a copy of that extracting program

Trade secrets, I'm sure you understand. Only trusted associates such as Gillian are allowed access in such cases. You and floof girl will only need access to this editor when I'm ready, anyway.

Well, back to work. I'm currently in the process of figuring out control codes (byte values usually less than 0x20) and planning a more perfect extraction. I've redone it a few times already since there are control codes that can get in the way of correctly determining the true starting point of the string and even the ending point, because some control codes take 0x00 as a parameter, and so in that case, 0x00 must not be considered the null terminator. It's gotta be done right or I'll not have the correct CRC32 computed afterwards to intercept a string for English replacement during runtime.

[Freakman]

Btw, English is your second language, correct?

Unfortunately it is
But I do have some confidence in my ability.
Oh well anyway, I'm not even the main translator in this (although I'm wondering where the others are ^^; )

As for the "easy enough", that's just a way to put it.
I've put in roughly 500-600 hours (if not more) to translate half of the game I'm busy with at the moment.
No matter how you look at it that's pretty hefty.


About your custom editor : I'm seeing a "find and replace", but I'm kind of worried I don't see a way to display more than one line of text at the same time (which is important for context, switching back and forth would get annoying fast). Will we have another way to access the script ?

[NightWolve]

Yes, I will also provide a HTML view, so you could copy the address from the editor, do a Find on the HTML page while open in a browser, and see what strings are there before and after, etc. We'll also have a Dialogue Dumper compiled in the special intercepting DLL, so as you play the game, it will write the script lines exactly as they appear onscreen. Someone has to beat the game, talk to everyone, etc. to get the full dump that way. That too is written to a HTML file and provides better context if the raw script dump is not exactly in the order that it appears in the game, which is usually the case. The Dialogue Dumper needs to be run when everything is in English too, so then you can see how everything plays out and catch more context issues. Psycho DeuceBag and I only did it in English for Felghana after he translated it, and that worked well enough to clean things up. If someone can do it in Japanese too, that's even better. We might be able to get some Ys nerd to at least do that for us, but I never hold out hope for these things.

Oh, right, the HTML files look something like this:

https://www.ysutopia.net/downloads/ysc/YsIIScriptHTMLData.rar

http://home.earthlink.net/~nlivaditis/xakiii/download/XakIII%20Jap-Eng%20Script.cab

[Freakman]

In Y's case, getting all possible dialogue isn't that hard... but doing that in ED6 definitely would take a mad man.

[Dave Shadoff]

As a quick guide, let's say that a novel has about 500 words per page.
...And each word is about 5 characters... plus a space... so about 3KB per page.

That's about the same as a 650-page novel !

[MarkMack]

Wow, this is excellent news!  You're amazingly dedicated, Nightwolve, I must say. 

I've spent more than my fair share of time lurking this community, and I'd really like to help if possible.  I can read Japanese (although I'm a bit out of practice).  I could also help clean up the grammar, spelling, and sentence structure of the English dialogue, if and when it becomes available.

Obviously getting through that massive mound of Japanese text will be the true test of this project's viability.  I'm here to help any way I can.

[NightWolve]

Thanks, yeah, you can wait for when I make the HTMLData rar like for the other projects and help by proofreading that.

https://www.ysutopia.net/downloads/ysc/YsIIScriptHTMLData.rar

That. I put those up for a reason to let everyone that wanted to proofread and find errors.

Anyway, does anybody know a cheat for this ED6 game? I found text that appears to be for a cheat DEBUG menu to let you access points in the game. I'll wanna know how to access this in case there are other print functions to hook.

[MarkMack]

I've never heard of any means of accessing a debug menu in the game.  I know in Ys: Oath in Felghana you open a debug menu by completing the nightmare boss rush mode.  But after you finish ED6, all you get is access to the 3 movies.

[Gillian Seed]

I've never heard about this debug menu too
 

[DaMarsMan]

Wow this is awesome news. Looks like everything is going smoothly and I wish you the best of luck. Speaking of big script you can't match mine... After extracted, Dragon Quest 5 for PS2's script is over 3MB. 

[NightWolve]

Ah yeah, I figured the largest script size I had heard of from the SNES glory hacking days when that scene was at its height was from Dragon Quest back then too. Not surprising its much worse now on a PS2. Who are you gonna get to translate 3 MB if at all?? LOL

[omgfloofy]

I'm also expecting Sora no Kiseki SC to have a much larger script, being that the game allows you to change characters out and the scenes /do/ change according to who you have in your party, so I've seen.  So there are a large number of variables on scenes according to the party members.

...which reminds me. I should check out the final battle stuff with other chars. But! That's for later. XD;;

I guess, though, I'd like to get together with some of the people who've also been interested in translating. I've had a couple of people helping me out- due to the fact that, while I can understand a fair amount of Japanese, my ability to word what I've read into English is kinda... shakey.

However, it's a pleasure to meet everyone else on here. Even if I'm a lurker. That's kind of my specialty.

[ViperD]

Debug mode does nothing that I know of.

[DragonmasterX]

A script that's 1,900 KB in size is literally unheard of!

Isn't a script>=1,900 KB normal for visual novels?

The biggest script I've ever heard about during the SNES romhacking days was like 400 KB, for maybe one of those Dragon Quest or Star Ocean games.

One game I assisted with, Der Langrisser, was said to be 1MB.

[NightWolve]

A script that's 1,900 KB in size is literally unheard of!

Isn't a script>=1,900 KB normal for visual novels?

You're asking me? So this game is classified as a "visual novel" and apparently such visual novels have script sizes in the starting range of 1,900 KB? There is still more script in it I have to dig out, but I've not dealt with nor heard of them getting this big. I always figured a long quality RPG still hovered around a range of like 400 KB - 600 KB, but admittedly I'm going more by SNES romhacking history than from PC games that I know of. Anyway, I had not heard it classified as a "visual novel" going into this. I know what the first game was like and it is surprising that it's gotten this big nonetheless.

One game I assisted with, Der Langrisser, was said to be 1MB.

Uh OK. Well, what I heard of is what I heard of from that time period. My issue now is do I really wanna work on something this big.

---

Anyway, lemme hoist some quick updates before I shut the PC down probably for another week. Well, right before I do my tax return. OK, unfortunately, I haven't touched the script extraction in a week. I took a break from the PC and got back to my bathroom remodeling project. I had a problem while working on it last Sunday, March the 26th I believe. I can properly extract about 46,000 strings perfectly, which is like 90% of combined script files. The last 10% is the problem because I cannot accurately determine the starting the point of each string. So in order to have peace of mind and know for sure I've read the string at its true starting point, I'm gonna have to figure out how the script file works, that is, how is it indexed. I had a hard time with that, and I dunno when I'll continue the work. I was hoping to finish it quick, hand off the script to floofy, and then be able to handle what I need to with my offline concerns, but it's not gonna work out that way.

So yeah, if you don't see me online for long stretches of time, this is why. I could extract the rest of the strings as is, but the uncertainty bothers me and I'll pay for it later when it comes time to get the English replacements to show up. So, I gotta hold off until I can work on it some more. Well, tune in for the next update then. See yas.

EDIT: Oh, one last thing, I have figured out all the control codes at least. They affect things like text color, clearing the text box entirely of text (like dos 'cls' command), delaying the pausing, and scaling the font two sizes up or down, although that is quite ugly to use and I can't imagine needing to use it. Here some samples of how I format a string after extraction:


ここで言うオーブメントは、<\n>
<color>[04]身体能力を高め、魔法を使えるようにする<\n>
<color>[00]《戦術オーブメント》ってやつのことだ。<\p><\0>

……と、まあこんな具合だよ。<\p>
<cls>
<color>[04]料理は食べて覚えるのが基本<color>[00]さ。<\n>
見たことのない料理は<\n>
取り敢えず食べてみた方がいいよ。<\p><\0>

#000F#1Pルゥイって子の話を聞いて<\n>
どうしても気になっちゃって……<\p>
<cls>
念のため調べてみたいから<\n>
入口の鍵を貸してくれないかな？<\p><\0>

#730F予行演習は終わったのに<\n>
まだ稽古をやってるとはな。<\p>
<cls>
２人とも精が出るじゃないか。<\p><\0>


The #000F at the start of some strings is actually needed. That will indicate who the character is, but it's really for fetching the various expressions of every character that appear with a dialogue box. If the dialogue box has a facial expression of the character, it's cause there's some code like #730F at the start of the string which fetches a particular expression of the character like happy, sad, excited, whatever they made appropiate for the scene. One other tidbit is the <color>[00]. That's a special situation where if byte 0x07 is found within a string, and the next byte is 0x00, that is taken as a parameter to request the font color be white (changed back to white in this case), and so therefore 0x00 should not be taken as the null terminator. I had wrongly extracted many strings before later learning this.

[Seldane]

I think there's some confusion here. ED6 is not a visual novel at all. I believe he was simply comparing the script sizes between ED6 and a visual novel.

I believe this game is a visual novel: Nadia: The Secret of Blue Water.

[NightWolve]

I think there's some confusion here. ED6 is not a visual novel at all. I believe he was simply comparing the script sizes between ED6 and a visual novel.

To me it seemed like he was implying ED6 was one, and yeah, I never heard it characterized that way at all prior to this. But if it was just about comparing, I still never saw script size figures floating around for such games anyhow. Whether or not >= 1,900 KB is normal for those, that's not a size figure I've heard of in my experience nor know anyone who else who has.

Oh, your link is incomplete. You have to copy the url right off the search results to carry the gameid parameter over in the link when it comes to pcecp. Here goes:

http://www.pcecp.com/index.php?mode=catalog&action=info&gameid=476

[MarkMack]

Regardless of how many bigger scripts exist, this one is not trivial by any means.  It's a massive undertaking to say the least, and I have no idea why people are making such a big deal out of a minor statement Nightwolve made.

Anyway...thanks for the info Nightwolve.  I'm really looking forward to seeing the script and helping in any way I can.  Keep us updated on any new developments whenever you get around to it.

[Gillian Seed]

Games like Tsukihime, Fate Stay night are famous visual novels. They are very popular games in japan, and they often feature erotic or porn pics.
I'm not sure about this but I think that ED6SC has a bigger script than the FC

[Zarcon]

For reference, Tsukihime's script is about 3.65MB and Fate/Stay Night's is about 5.85MB in size.

Anyway, I hope this means I have something to look forward to...in like, a few years. XD
Loved your Y's I & II Complete translations and they brought me deeper into the world of Falcom and for that you'll always have my thanks.

[NightWolve]

Fate/Stay Night's is about 5.85MB in size.

Yikes! OK, now there's a real nightmare!

[omgfloofy]

I'm not sure about this but I think that ED6SC has a bigger script than the FC

You *think*? XD;;; Is there any sarcasm in that, because I'm pretty sure SC's script is going to be bigger. XD;; SC has a lot of variables for possible scenes.

[Ruri]

I want to know if you need help with translating Sora No Kiseki. I've already played both games till the end and have experience with translations (mainly translating Super Robot Taisen games for the AGTP group). My english isn't so good, but I love the game and am willing to translate it to the end.

[NightWolve]

Hi Ruri. Thanks, I'll keep that in mind. The thing is this is now way on hold. When I get back to this after I release the Felghana & Ys VI patch, along with finishing some offline work, I'll try to contact all the translators I know of that want to work on this. The thing is I don't know if you'll be around by then, whenever "then" is. It's not gonna be this year, that's for sure. omgfloofy said she'll probably be around a year from now when I checked with her, but I can't say for sure if it'll even be longer. Things just got hectic and I couldn't package the script accurately back when I wanted. If I can't do it accurately, I won't do it at all.