@furrybob - doing AHX stuff I noticed some AHX in the wild (Amagami) also have keyword encryption (type 08 in the header).
The algo seems a lot simpler, but enough to throw off decoders. It justs encrypts a few bits (6?) near the MPEG side info every new frame.
You wouldn't happen to know how this works? It's mostly used for voices but would be easy add to vgmstream (for science) once we know the algorithm. I think you have access to ahxencd.exe but I can upload samples and stuff.
Okay, here's how the encryption works. For the setup, you take your keystring and run the same process on it as you do with ADX to get 3 prime numbers. Assign these primes to the indexes 1, 2, and 3, and assign the number 0 to index 0.
For example, with the keystring "test": 0 - 0 1 - 0x5ced 2 - 0x4efb 3 - 0x5381
Now take the bitstream you want to encode, up to 18-bits, and read the first 2 bits. These bits correspond to the index of the number you'll use, as described above. We'll call this the key.
Take the next 2 bits from the bitstream, and take the two low bits from the key (i.e. bits 1 and 0). Xor these 2 values together, and put the resulting 2 bits in the output bitstream.
Again, take the next 2 bits from the bitstream, and now take bits 3 and 2 from the key. Xor them together and output the result.
Continue until you've encrypted all the bits you want to.
Awesome, thanks a lot! Worked like a charm. In Amagami the key is shared with ADX too ("mituba").
For AHX, the bitstream/index seems to start 103b into each frame (presumably some Layer II field), and somehow always uses index 2. AHX "for DC" encrypt the full 16b, while normal AHX 6b only (size of whatever field?).
Not exactly a mass-demanded feature, but I love little fun things like this.