@Cloudschatze

thanks for another idea about the potential encoding/packing/compression they are using. in the attachment here:

Re: About Roland Virtual Sound Canvas 3

in "ramdump.bin" (after "CM-64/32L.ME" data sequence), there are few unpacked samples.

I am working on finding the data that correspond to them in a "packed" form. if I succeed that maybe will shed more light on the exact encoding/packing/compression that is used.

progressed more thanks to VSC 2.x - I guess it's always good idea not to use the latest (and thus most advanced) version of anything you're trying to reverse-engineer, like I was using VSC 3.23 until now.

So, with VSC 3.23 when the Sound Bank is decrypted/unpacked to the RAM it's scattered all over the place in 4KB (1 page) chunks and you cannot reconstruct the whole bank (in any simple and fast way). However, with VSC 2.x more often than not (3 times out of 5 restarts of the system) the Sound bank is decrypted/unpacked to the RAM in one continuous memory location, i.e. you can dump the entire bank decrypted/unpacked by Roland from the RAM.

that's quite a nice "feature" of VSC 2.x, because in like 5 minutes for doing few reboots, you can dump the whole bank decrypted and unpacked for you from the RAM memory.

I now listen to it after converted the sounds with 'sox' and pianos, drums, steelpans, etc are very clearly identifiable by my ears. However, the more interesting part - whatever encoding Roland is using is not giving any compression, for example "block2" with size of 1,278,975 bytes from the Sound Bank file is decrypted/unpacked to exactly same 1,278,975 bytes in memory. that's quite unexpected. So, it's not using ADPCM compression or mu-law as @Cloudschatze was suspecting.

Anyway, my next step is to try pinpoint where the first sound decrypted/unpacked in memory (which sounds like piano note to me) is in the encrypted/packed data stream. That I hope thanks to the above aforementioned "feature" of VSC 2.x won't be that hard just with generating some "fake" block2 data that I can investigate how are placed in the RAM memory.

"block2" is like it's not packed, but 2 layers encrypted, 1st layer "vscdecv3.c" can remove and the 2nd layer is some weird, I don't know what to call it "substitution" or "shuffling" (maybe it's known crypto algo, but i don't know it) - I still cannot understand the math behind it - when I replaced 16-bytes in the block with fake ones I cannot see any obvious relation, but when i replaced 256-bytes with fake ones - the relation is very clear - see the attached screenshot. So, it's like every 1247th bytes was changed in the decrypted stream, but as I've just mentioned I still cannot understand the math behind that change, even I have the feeling I am very close. If someone here is crypto expert and can identify the algo, let me know. It has to be something basic, because I see no any use of "key", just the bytes are shuffled, but in such weird way, that I cannot understand (yet), even I clearly see that every 1247th byte is changed.

hmm, when i visualized what is going on, i got the relation, but partially - it's diagonal and every next byte on the diagonal, that's why when I replaced only 16 bytes with fake one there was no any relation, because you need to have matrix with at least 16 bytes diagonal and that means 16 lines or 256 - exactly why the magic happens when I replaced 256 bytes with fake one. I am definitely getting closer....

[EDIT] searching for "encryption diagonal matrix" gave link to "Modifying des algorithm by using diagonal matrix based" - I don't know maybe I am reinventing the wheel, but I don't think Roland use DES - I mean I see no key, i.e. no byte change, just the bytes are shuffled in this weird and very hard to understand way.

I manages to track down the installer for the 3.20 retail version and it is a godsend...

appiah4 wrote on 2020-09-26, 19:42:

I manages to track down the installer for the 3.20 retail version and it is a godsend...

3.20 you can update to the latest 3.23, the update is still alive on Roland website:

https://static.roland.com/jp/media/exe/VSC323U.exe

back to the 2nd block and its 2nd layer of encryption (really very complicated byte shuffling??!!)...

Guys, I really don't know what is that - if I make the matrix with 16 elements in a row, then it acts on "column" instead the "diagonal" from the picture attached 2 post above. The problem is that while on "column 0" (see the attached screenshot to this post) it acts on all elements, on "column 3" and "column 9" it acts only on some elements. it makes no sense to me!

also on some of the elements it adds or subtracts "8", below are details what it does on "column 3" (in full details to understand what exactly happens) and for column0 and columd9 (i put just a short table, you can read it, based on column3 detailed explanation).

Anyone recognizes this? Is it just shuffling or some well-know (and complicated) cryptography algorithm?

1(0x8d - 0x08) = 0x85 --> value of "0x85" is assigned to address  (0x6EDE4 + 4DF*0)
2(0x1c       ) = 0x1c --> value of "0x1c" is assigned to address  (0x6EDE4 + 4DF*1)
3(0x17 + 0x08) = 0x1f --> value of "0x1f" is assigned to address  (0x6EDE4 + 4DF*2)
4(0xf0       ) = 0xf0 --> value of "0xf0" is assigned to address  (0x6EDE4 + 4DF*3)
5(0xe0 + 0x08) = 0xe8 --> value of "0xe8" is assigned to address  (0x6EDE4 + 4DF*4)

1column 9:
2e8 - 8  12875
30c      12d54
4fb - 8  13233
5fe      13712
606 + 8  13BF1
7d0      140D0
841 + 8  145AF
90d      14A8E
1010 + 8  14F6D
11db      1544C
1201 + 8  1592B

1ac - 8  1*0x4de + 0
2dc      2*1246 + 1
336 + 8  3*1246 + 2
4fc      4*1246 + 3
5e7 + 8  5*1246 + 4
6f1      6*1246 + 5
720 + 8  7*1246 + 6
830      8*1246 + 7
913 + 8  and so on
10d5
11db - 8
12ea
1320 + 8
1414
15de - 8

OK, made some very small progress:

1. the "+/-8" drift from the original value from my previous post is solved - my mistake - I made mistake in 1st layer encryption of the "fake" block2 that I was feeding to Roland software. So, the "byte shuffling" Roland are using for 2nd layer encryption doesn't change the value of the input data

2. I generated several small "fake" block2 - with size 256 bytes, 384bytes, 512bytes. So, now the math problem to be solved can be redefined: Find function that on the given input generates the given output. And write it in program language of your choice. the attached files names indicate what input what output should generate.

256bytes input - no change
384, etc bytes input - see the picture and for more see the files in attached "func_input_output.zip"

I am over my head with the 2nd layer of encryption, i.e. the byte shuffling they are doing.

Please, check the attachment, to see the output of special input matrix I made. It has to do something with odd and even index of the first 2 line and moving them at the end, at least for first 16x16 bytes, but last 16 bytes are scrambled very hard. Also, I believe it's kind of recursive in nature, because "bigger outside" data (first 16x16 bytes) are less scrambled than "smaller inside" one (last 16 bytes).

Anyone recognizes the algo or has any ideas?!

A little closer... see the attached input / output pairs... starting to see a logic... it's like it's divided in 8x8 (64 bytes) matrix, then the remainder (8 in first case, 4 in the second case) is used as number of byte (with odd indexes) to get and put at the back.

Come guys, any genius that can write the algo?

OK, I think I kind of start to getting the algo myself:

1
2* iteration 0, move first 2 odd indexes to the back:
3
4E0 E2 E4 E5   E6 E7 E1 E3
5
6* iteration 1, move 2 "new" odd indexes of the first half to the back, do the same with the other half:
7
8E0 E4 E2 E5   E6 E1 E7 E3
9
10* iteration 2, move odd indexes of the "new" first half to the back, do the same with the other "new" half:
11                    E1 E6 E3 E7
12
13END result: E0 E4 E2 E5  E1 E6 E3 E7
14

Does that make sense to you as correct?

unfortunately, no, the above is not (entirely) correct - on last step i made wrong swap 🙁

So, the unknown "func" do the following on the indexes:

1                     {0} ->                      {0}
2                  {0, 1} ->                   {0, 1}
3               {0, 1, 2} ->                {0, 2, 1}
4            {0, 1, 2, 3} ->             {0, 2, 1, 3}
5
6         {0, 1, 2, 3, 4} ->          {1, 3, 0, 4, 2}
7      {0, 1, 2, 3, 4, 5} ->       {2, 5, 1, 3, 0, 4}
8   {0, 1, 2, 3, 4, 5, 6} ->    {5, 1, 6, 3, 0, 4, 2}
9{0, 1, 2, 3, 4, 5, 6, 7} -> {0, 4, 2, 5, 1, 6, 3, 7}
10

until 4 elements, there was nice recursive relation:

{func(x0, x1,...xn)} = {func(x0,...,xn-1), xn}

but too bad it doesn't hold for n>=4. Even it's in front of my eyes, I just cannot see the relation, but I guess it has to be recursive...

Another approach, because we can get (dump) the unencrypted Sound Bank from the RAM memory, but after modification we cannot re-encrypt it, is just to patch the software when load the soundbank to not decrypt it.

I already kind of simulated this and that it will work - patching the header of unencrypted bank offset 0x2a to 0x28 (which is the address of 2nd block start) to be the same value as offset 0x22 to 0x20 (which is the file size) - that way tell the software the 2nd block begins after the end of the file. Indeed the unencrypted soundbank was properly loaded by the software (inspected RAM dump as well to confirm this). Of course, there is no sound that way, because "offset 0x2a to 0x28" also tells the software the location of the WAV data.

So, the above approach will 100% work if the software is patched to not do 2nd layer decryption of the 2nd block and then we can just use unencypted banks.

I actually, like such approach very much and much better - we acquire the unencrypted bank from ram dump, then patch the software to not decrypt on-load the bank and from there on use unencrypted banks - even easy to use when you make changes to the bank. Unfortunately, I also failed miserably to find where exactly the software calls for decrypt of the soundbank on-load to disable it.

Good progress being made, and extremely interesting. Thanks for posting all the detail. Looking forward to seeing how it continues.

Shreddoc wrote on 2020-09-28, 22:11:

Good progress being made, and extremely interesting. Thanks for posting all the detail. Looking forward to seeing how it continues.

I think to continue with understanding the structure of decrypted bank (after all at least at this point we can get one fully decrypted one from a ram memory dump), even write tool to extract instrument sound from such bank or replace it sound. that way some day if someone reverse-engineer real hardware SC55 roms dumps (i already failed on that as well) and extract the sound from there we can make VSC sound bank with the real sounds. of course, that still needs either way to reencrypt the bank or be able to do this:

Re: About Roland Virtual Sound Canvas 3

there are a lot more mysteries to be solved than just the 2nd block 2nd layer of encryption. So, as you can see on the screenshot here:

Re: About Roland Virtual Sound Canvas 3

SC55 Sound bank of VSC 2.x has "226 Tones + 9 Drumsets". That coincides exactly with what Microsoft GM.DLS file has - i am attaching picture to show that.

It makes perfect sense, because as you probably know MS licensed the sounds from Roland, but at the end made soft-synth lacking effects like reverb and chorus, making it sound bad compared to other soft-synth.

That's very good clue for when reverse-engineer the whole format of SC55 VSC 2.x Sound Bank - there has to be 1:1 map for "tones", "drumsets" and even for "sound samples" to Microsoft GM.DLS and that last one has very well documented format, in fact whole 77 pages of so called "Downloadable Sounds Format, Version 1.1b" anb available as PDF.

So far so good and that's why as a fast test I made small tool to just extract available "tone name" from SC55 VSC 2.x Sound Bank descrypted with my "vscdec" tool (note here that removing the 1st layer encryption, which we know how to do, is enough to get the names of the available tones), but another surprise:

I am able to find only 217 tones (216 on my picture as I count from 0) - see the attached picture. That's another unexpected thing - maybe I missed something obvious - anyway as next step I will need to compare my list and the list I got from MS GM.DLS to see which 9 "tones" I am missing and why, i.e. where they are to be found in SC55 VSC 2.x Sound Bank.

ok, the above is solved: in MS GM.DLS some of what Roland in VSC called "tones" are defined 2 times only with different "Midi bank":

1Piano 1 	(bank 0/8)
2Piano 2 	(bank 0/8)
3Piano 3 	(bank 0/8)
4Honky-tonk 	(bank 0/8)
5Harpsicord 	(bank 0/16)
6Vibraphone 	(bank 0/8)
7Marimba 	(bank 0/8)
8Orchetra

still missing 1 "tones" though. However, for the time being that is good enough explanation why I got several less tones when parse the information in VSC SC55 2.x sound bank.

That means in VSC format 1 entry can define the same "tone" for at least 2 Midi banks, while in DLS format that requires 2 entries.

So, the fact MS GM.DLS is based on Roland sounds really helps for the purpose of reverse-engendering the VSC Sound bank format - that's the key fact to be used.

So, the VSC soundbank file format itself seems is not a problem, because with no effort at all, already got "bird's-eye view" of the format:

1header (0x54 byts in size)
2GS Instruments definitions (I counted 322, each definition is 0xd8 bytes in size, anyone familiar with GS - does that number sound right?)
3GM Instruments definitions (each definition is 0x3c bytes in size, I counted 217, several less than expected - see the above post for answer why. that confuses me a little, because Microsoft call it "instruments" in their MS GM.GLS, but Roland called that "tones" in the GUI of their soft-synth. So, maybe, it's tones and the above are all instruments, no matter of GM/GS compatibility, not sure)
4Drumset definitions (together with their WAV/PCM data - those are the only WAVE Samples protected only with 1st layer encryption)
5next is what i called in my previous posts "block2" - it's heavily protected and it contains all of the actual WAV/PCM data

next, is to figure out each byte of those chunks of 0x3c and 0xd3 mean, because currently i know only the 1st 12 bytes - that's the name and if it's shorter then it;s padding. Most crucial information those other bytes should contain is which range of "block2" WAV/PCM data are part of that instrument/tone definition.

a little more progress, "block2" data format found, i.e. all the WAV/PCM sounds:

they are 12-bit resolution exactly like real Hardware SC-55 (not exact sound, but exact bit resolution) and stored as 2 samples packed in 3bytes. So, you need to unpack them back to 16bit in order to play. It's like they are compressed, because 32bits (4 bytes) unpacked are stored as 24bits (3 bytes) packed.

another explanation why MS GM.DLS and their soft-synth sound is less quality. it's not just Microsoft lack reverb and chorus effects, but it seems to me their Roland sounds are 8-bit up-sampled to 16-bit. So, Roland licensed then less quality (less resolution sounds). That means when we are/I am able to extract each separate sound from "block2" then even better quality custom GM.DLS for the MS soft-synth could be build from those 12-bit resolution sounds.

Last, but not least, Kudos to the guy who reversed-engineered Yamaha TG100 Sound bank - "vampirefrog", because Roland stores their 12-bit samples, exactly as Yamaha in TG100 and that gave me the idea to check if by any chance "block2" is made of such "12-bit" samples.

So, bottom line - really the Sound bank format is not a problem - for less than 1 hour spent in total today, I found so much - after all "block2" alone is 75% of the file and now it's format is clear. That means the main problem remains 2nd layer encryption of the block2 or how this to be achieved:

Re: About Roland Virtual Sound Canvas 3

then making Custom Sound Banks will be possible - best is SC55 ROM be reversed-engineered as well and real SC55 Sound bank be created that way.