Ripping music (and other data) from Koumajou Densetsu .dat files by Atorasu at 8:17 PM EDT on May 30, 2020
I want to rip the music files from Koumajou Densetsu and its sequel for the purpose of music mods in other games. The hex of bgm.dat indicates that each song is an .ogg file, and includes what may be a pointer to that song's location. Additionally, define.dat seems to contain a bgmlist.txt, which could contain each song's loop points. I was told that the bgm.dat file has a --keep-going argument.
KD1: there is definitely a list in the first 0x350 bytes. More precicely: # Struct "DPMX" [header size] [file count] [0] /for each file: {finename}:0x10 [FFFFFFFF] [ ??? ] [file start (relative, after header)] [file size] / # End struct On quick inspection (0.5 hrs, so not really quick!), they seem rotated, i do not know what N-rot (rot-N) is used, have not revised extensively, but each file seem to have its own rot-N. I do not think it is XORed; if it is, then key must be really large.
KD2: Appareance of first bytes suggest that unlike the KD1's .dat, there is no filelist, maybe just a container string, then the data streams one after another. The encryption, if XOR, must be large enough (at least 1K chars), and there is few repetition so i do not think it is rotN-ed. Just my guess in this one.
Interesting findings, though I wish I knew how to help more. Do the files from KD2 at least have evident beginning points? If for whatever reason it might be helpful to have other files such as the .exe for either game, I would be able to provide a full download. There isn't a whole lot outside of the "data" folder however.
Hello; yes of course it's very possible that hints for deobfuscation of KD2's music be in the other files, even .dll's or .exe's - it will require time to investigate. Jap developers are very structured about data resources - i'm 99.999999% sure all music is within those bgm.dat (as well as having dedicated files for static graphics, dynamic-vector ones, sound fx, even for menu messages and dialogs).
Edited: files on KD2 have similar bytes at start, so they at most share the same starting bytes (fake container probably). Progress on KD1.
You're making good progress, the files don't play but extraction is still a good start. Keep it up, working at your own pace of course, this isn't urgent. The MediaFire folder is updated in case you find that the outside files might be necessary.
KD2: XOR key: EF C1 20 01 DC F7 BB 72 FA CB F2 01 It's most frequent string in all of the .dat files (prominently in se_sys.dat). As I expected, a fake container 0x18 bytes size. .ogg files extracted with vgm-t, KD2's ogg. Key works on all the .dat files.
KD1: I upload ADDed (256-X, X:most frequent byte= $0x8) and XORed (with same $0x8) .ogg's that still do not play because they are not entirely decrypted/deobfuscated. Hopefully you or someone else have a good eye to figure some pattern in these - just use any (normal,unencrypted) .ogg as reference for byte analysis/comparison. Value $0x8 (i.e. value at offset 0x8 of each file) contains the most repeated byte which is used for the bit operations. Values between brackets in the filenames are the "???" in my first post detailing the struct.
I made memory dumps of both games. I'm not sure whether or not there would be unencrypted audio, but in both dumps I found loop points (as samples). In the second game's dump, I see the loop points close to their song titles, but in the first game's dump I only see the points for b_alice.ogg. Additionally, I see song titles surrounded by seemingly garbage, but probably useful data. They are valid loop points at 44100Hz, but that was all I could see in 2 memory dumps of the first game. KD1's files may need to be fully decrypted for music and loops, but the presence of the song titles may somehow help. Also if it helps, this is what I found in the dump of the first:
It seems that bgm_19.ogg (Last Phantasm) from KD2 didn't fully extract, the end loop point from the memory dump goes past the end of the song, and the song lacks a fadeout that the rest had. Would you mind telling me how you used vgm-t for the .dat or re-extracting it?
The memory dump for KD1 would be useful hoping that some .ogg chunks would be present there for byte-comparing with the obfuscated .ogg, as a possible way to guessing more about the encryption. For KD2, you can deobfuscate the bgm.dat with the 12-bytes key I provided; I did not find filenames for the .ogg files, how did you find those? Use wxhexeditor or a better alternative to XOR the original bgm.dat with the key, then use vgm-t on the resulting file to extract the music. vgm-toolbox has automatic OGG extraction in Misc. Tools > Extraction Tools > Streams > Xiph.Org OGG extractor. On track #19, it's possible that some chunk was left out of extraction, it might have been some 2KB at most iirc, enough for a data loop point/chunk maybe (i just let vgm-t to do the extraction).