Tetrisphere USF set question by Lunar at 9:43 AM EDT on June 9, 2005
Something me and a friend noticed while discussing USFs in general.
I was led to believe that the *.miniusf files were simple files that contained tag information, and the command to send to the *.usflib. With that in mind, they should be of a very small filesize, and for the most part they have been.
I just noticed yesterday that the Tetrisphere *.miniusfs actually go up to as large as 448kb! Why is this? I'm wondering how this particular set functions internally, since the *.usflib for it is only 620kb. I'm getting the impression this set does indeed contain its own samples or song instruction data within each *.miniusf. But if that was the case, would that not make them *.usf?
Sorry if I'm being hopelessly naive or this has been explained previously. Seems pretty interesting though.
There was at least one set that used compressed data for its audio files. When the ripper ripped that set, they actually grabbed the code after it was decompressed, but before it was initialized. Or something like that. If I recall correctly, the data they pulled was actually from the RAM, rather than the ROM (like most USFs are). As such, the files each contain their own information, as it is unique to that specific song, since it is in the decompressed form. If it had been pulled straight from the ROM, then it might have been the small files you're used to.
As for being *.USF instead of *.miniusf, that is not correct, if my understanding is correct. From what I understand, all the *.miniusfs use the same player program, which is contained in the *.usflib. However, they each have their own sequencer and soundbank data. The reason the *.usflib is 620kb (if I recall) is because of a small fluke in the ripping process. It's certainly not a problem, it's just that at the time everything was put together, it was easiest to do it this way. Apperently, when creating a USFLIB file, it takes all of the data that is shared among all the files, and puts it into one file. However, because of the way it compares that data, which ever file was compared last is almost wholly included into the USFLIB file. Because of that, you will find that "Tetrisphere - Spooky Baby.miniusf" is a very small file. My guess is that it was the last file to be compared for inclusion into the USFLIB, so most of its data is in the USFLIB file, rather that in its own file.
HCS, please correct me where/if I'm wrong. This is all from memory from quite a few months ago. I'm pretty sure the general idea is more or less correct, but I'm alse quite certain that there's plenty of room for correction and clarification. Please feel free to add/subtract what is necessary.
Hopefully, I helped answer your question, Lunar, at least a little. Mouser X over and out.
Everything sounds right to me. I'm not entirely sure why the miniUSFs are so huge, but your guess sounds most plausible. miniUSF/USFlib sets obtain their efficiency primarily from the fact that all the tracks refer to the same ROM, where the most data (the samples) reside. When the miniUSF file is larger than a few hundred bytes there's something different in RAM between the tracks, as the ROM by definition is unchangeable. I make an effort to create the save state for my USFs before song-specific things start to happen, because as soon as you start loading data out of ROM to different places in RAM size can start to get out of hand. Your analysis of the "Spooky Baby" phenomenon is correct, using the typical USFlib generation process the completed USFlib will contain exactly the data needed for the last track analyzed, as well as data not accessed by that track but needed by other ones. Thus the USFlib should be a valid track in itself, and the miniUSF for that track need only contain a reference to it and tag data. It doesn't always work out perfectly, though.
Actually, everything I said was from memory of a session in the PSF_Rippers chatroom. Zoopd was discussing his method of ripping one of his sets. I think what I remembered was actually in referance to the New Tetris set, which also contains large *.miniusf files. I'm somewhat surprised you don't remember the conversation, (or at least you don't seem to) because I could have sworn you were there. I remember someone making a comment on the size of the files. Zoopd expalined why the files were so large. He couldn't find the code to decompress the files (if I recall), but he could find the player code. Either that, or it took to long to decompress the files (there was a noticeable pause as each track tried to load, or something). So, to bypass the load times, he ripped it after the music was already loaded.
It was one of these 2 problems, but I don't remember which one. Perhaps one applies to one set, and the other applies to the other set. Meaning, perhaps he couldn't find the decompressor in say, New Tetris, so he ripped the music after it was loaded, but before it was played. Whereas with Tetrisphere there might have been a load time problem so to bypass it he used the same method he did for New Tetris (or something like that, perhaps). As for my analysis of the "Spooky Baby" file, that is also from a memory of mine from the chatroom. Again, you were having a conversation with zoopd. He asked you a similar question, of why some files were 400kb (or whatever "big" size they were), and there was always one that was really really small. You answered him by telling him how the USFLIB "compiler" works. He asked if it would be possible to correct that little fluke, and make the USFLIB contain only the data it was supposed to, and not the song data of the small file. As I recall, you said it was possible, but would require a lot more work than it was worth (recoding software, completely changing your approuch of gathering and assemblying the info for the USFLIB file and such). After brief discussion, he agreed, as I recall. After all, the method you do use works, and has no ill side effects at all, other than that the data isn't quite where it should be (since it should be in the file it belongs to). Since that's not a big deal, there's no reason to change it, especially now, after all the sets that have been ripped.
Gag, grammatically, that paragraph should be split up... Anyway, glad I could be of service. If only I could figure out how to rip USFs now...
That reminds me. The reason I never got past the questions stage is because I was told I would need to find a certain command in the decompiled ROM. I finally found that command, but when I tried to create a breakpoint there with the debugger N64 emulator you gave me, I couldn't run the ROM because it didn't recognize the controller plugin. It always said "No controller in slot 1" or something like that, so no major code was ever run. Because of that, I could never find where in the code the needed threads might be when I tried to decompile a savestate (so that I could then alter those threads and try to isolate the sound software). I seem to recall asking if anyone had any ideas as to what I could do, but no one had any (that I can remember).
Anyway, thanks again for your great help, and USF rips! Mouser X over and out.
I'm not sure what "debugger N64 emulator" you mean, I have quite a few different sources floating around, but the one that goes by "Project 64 USF" has most of the controller and graphics support taken out (though that might not be it, since you said the game displayed an error message). My method for doing breakpoints is adding them right in to the interpreter CPU core, like:
if (PROGRAM_COUNTER==address_of_interest) DisplayError("Hi there, ra=%08x",GPR[31].UW[0]);
And are you using IDA for your analysis? If so you can spot all the threads easily by looking for calls to osCreateThread, a2 is the address of the function. At the end of the body of the thread is generally an unconditional branch back to the top of a main loop, typically I replace this with a NOP to kill the thread (as it nicely runs all the teardown functions that are unreachable under normal execution).
I've been using IDA Pro (from Caithsith2) to decompile the ROM. And yes, that command I was looking for was called osCreateThread (I remembered after I had already posted). I'm also aware that I was needing to look in the a2 address. The problem is, I can't get the emulator to pause at the osCreateThread command so that I can see what info is in the a2 piece.
Sadly, I'm not nearly advanced as you in this, so I don't know what you mean when you say you add "them right in to the interpreter CPU core" As for killing a thread, I knew how to do that, but I can't kill a thread if I can't find it. Since I'm trying to alter a savestate (I've been told that's the best method, since you don't have to actually alter the ROM to do it, and therefore you don't have to change the CRC count everytime, or handle as much code either), I need to know where those threads are started. I find that out by tracking down osCreateThread. I track down osCreateThread by creating a breakpoint at that location, so that I can read the data stored in the a2 register, which will tell me the address of the thread. However, the debugger I have errors out before it even gets that far, so creating a breakpoint does nothing for me. If there was some other way, or something else I need to do, or if I should try another program, I'd be happy to give it another effort.
It would seem that the debugger I'm trying to use is from v.1.4 of PJ64. I just loaded it though, and it crashes without question. I think either my computer settings have changed (which they have, drastically, at that) or I played with the emulator's settings enough that it messed it up. I should probably look into that more, when I have the time (I should be heading for bed right now as it is...).
Anyway, thanks for the help. Hopefully I can start this up again, and actually show something for the effort this time around... Mouser X over and out.
Most programs tend to be very simpleminded with setting the parameters for functions. When you've located osCreateThread select that name (anywhere, really) and press x. That will load a list of cross references to osCreateThread that IDA knows of. Select one of them, that'll take you to a JAL ocCreateThread command. Look up a few commands and you should see an li a2, 0x80blahblah instruction. If you press control-o while 0x80blahblah is selected it will intreperet the immediate value as an offset into the current segment (RAM, which is what you want) and you'll be able to simply press return to go to that address (or press P to let IDA know that this is a function that you want analyzed).
Do you have Visual C? If so, I think that's you best choice for adding your own debug functions into Project 64 v1.4. Locate ExecuteInterpreterOpCode in Interpreter CPU.c, add the line I mentioned above at the top of that function, and compile. Tada, you've built in a breakpoint, which you could use to trigger the saving of a save state:
if (PROGRAM_COUNTER==address_of_interest) { sprintf(SaveAsFileName,"mrsave"); Machine_SaveState(); }
or writing something to memory
if (PROGRAM_COUNTER==address_of_interest) { ((DWORD*)N64MEM)[(0xmraddress & 0x3fffffff)/4]=0x0; }
or a plentitude of other things. There's a trick to compiling the "Pj64 Build 52" sources, you have to remove the line "#include "Reverse Code.h" from Debugger.h, and there may be something else I don't remember.
Oh, and if you don't have MSVC6 that can be remedied...