I have attached what I think could count as some sort of template for a description of PC protection methods, together with how I understand your kind of "info file" could look like for these schemes. Basically, the first "layout" line would just list the default number of cylinders, sides, sectors per track and sector size. Additional lines would specify different settings for specific tracks or sectors, or particular layouts. I have only described two schemes as examples.

We might be having a terminology issue. One of the main things I was trying to avoid was to create a list of copy protection templates. Things like Copy Lock, OSI-1, etc... Those usually use a combination of techniques (though there are those that only use one.) For example, Sierra's (don't know it's name) uses an intentional error, sector overlap, non-standard track size, and non-standard sector sizes all at the same time. What I am trying to create is a list of those techniques, and a way to encode a header or info file so that the techniques can be specified and simulated. The easies way, to my mind, though it will create a larger file, is to just create information for every track & sector. For a smaller file, you would just create the information for the tracks/sectors that are different. Then, when DOSBox reads a track/sector that is part of a particular technique, it will no how to simulate the read of that track/sector.

One reason for this approach is that it would be easier (IMHO) in the long run just to simulate the techniques. There are only so many of them, so there is less to write code for. It would leave the work of imaging the games and building the information to the end user (who actually, in theory, owns the game.) The copy protection templates themselves are technically under copyright, and building something geared directly towards the template could be considered an infringement. While I don't know how any of them turned out, I do know that some of the parameter builders have been sued over this issue in the past. If you build something to simulate the techniques themselves, instead of the templates, you don't face that particular issue.

So far, I have been unable to find anything beyond the 10 I listed in my last update. So that's only 10 things that have to be programmed, aside from the routine that detects, reads, & interprets the info in the first place. Then, regardless of which of the dozens(?) of templates a game uses, it will only be using 1 or more of the techniques, and DOSBox will just handle the ones it needs to.

But how would this be different from or easier to implement than just adding support for the TransCopy (or PCE .PSI) format?

NewRisingSun wrote:

But how would this be different from just adding support for the TransCopy (or PCE .PSI) format?

I don't know about the PSI format, but with TD0 the data is compressed (if only slightly with normal compression.) For one, if a sector doesn't contain anything, it just notes it and doesn't create the actual data. When writing to the file, if data is stored in one of those, DOSBox would have to basically re-write the entire file first. Second, TD0 files are proprietary to Sydex (or the current copyright holder.) While other emulators may go ahead and use them (the holder is not likely to say anything if they aren't making money off it,) DOSBox has maintained fairly strong efforts to avoid that sort of thing. Finally, the code for handling raw IMG files is already in place, what I'm proposing would just add to that code, no need to write an entirely new format, just add something to tweak things a bit.

One copy protection technique (or template, not sure where this would fall,) that I came across on another system, but is perfectly functional on the PC, was to just try to format the disk (or a portion thereof.) If it was the original, that would fail, but if it was a copy, it would wipe itself. Then you would get an Anti-Piracy method and the system would re-boot. I considered it rather funny really. Easy to defeat, just write protect the disk after making your copy 😀 If any for the PC do exist, though, it would always fail in DOSBox, can't mark IMG files as read only. If they are, DOSBox refuses to mount them 🙁. Though I haven't tried it with the boot command. Not important, really, unless games with that method actually exist.

.PSI format is open source, and TransCopy is by defunct Central Point Software.

FeedingDragon wrote:

When writing to the file, if data is stored in one of those, DOSBox would have to basically re-write the entire file first.

That's hardly a problem these days. PCE seems to manage it quite nicely.

FeedingDragon wrote:

Second, TD0 files are proprietary to Sydex (or the current copyright holder.)

You cannot hold a copyright on a format. You can hold patents associated with that format. Any patents associated with TeleDisk or TransCopy will have expired by now.

FeedingDragon wrote:

Finally, the code for handling raw IMG files is already in place, what I'm proposing would just add to that code, no need to write an entirely new format, just add something to tweak things a bit.

The current raw image file handling code is so simple precisely because it does not specifically account for any special cases. Adding support for an info file that provides metadata to the raw image file would already amount to rewriting the whole thing, which would have to be undertaken, while existing code for TeleDisk, PSI and TransCopy support could simply be reused from another GPL-licensed emulator.

Ok, let me put it another way 😀 The authors have repeatedly resisted using any of the other formats. I'm also trying to avoid asking them to do something they've already said "no" to.

Also, file formats are rarely if ever patented. They are usually placed under trade secret protection instead. A trade secret, as long as the holder doesn't release the information themselves, is protected indefinitely. As a matter of fact, I've never heard of a file format actually being patented, which requires releasing the information to the patent office, which makes it freely available eventually. The only information for TD0, as far as I can find, was reverse engineered, which doesn't negate trade secret protection. I just grouped it under copyright for simplicity - maybe I shouldn't have done that. That being said, it "may" not be covered under trade secret either, but the fact that the format has never been officially documented suggests that it probably was.

As for the raw IMG file being simple because it doesn't account for special cases, the code can't account for special cases because the raw IMG format is completely incapable of holding such information. It is a simple raw dump of the data, which is why it cannot include variable sizes, error information, etc... Yes, new code will need to written, but lets not "replace" anything, lets just add things. So, the choice becomes, lets write a new routine to handle a different file format, or lets use what is already there to read the data, and just write code that would alter how we read/represent that data.

Normally, and this is just a guess, when the IMG file is opened DOSBox determines the layout (how many sectors in a track, how many bytes in a sector, etc..) and when a read to a specific track/sector comes up it just multiplies by the appropriate amount, and starts reading from that position in the file. In my suggested plan, when dos box runs the routine to open the IMG file, the first thing it would do is check for the info file. If it doesn't exist, it just works normally (does exactly what it does now.) Otherwise, while it still opens the IMG file normally, it would call a new routine that will handle the protection simulation. When it goes to seek a new location in the file to read a specific track/sector, instead of just multiplying things, it will either use the info file to do more complex calculations, or just read the offset from the info file instead. It will still use the exact same code to actually read the data from the IMG file. I could really speak more on this if I understood exactly how DOSBox does this.... Yet another thing I need to research.

FeedingDragon wrote:

The authors have repeatedly resisted using any of the other formats. I'm also trying to avoid asking them to do something they've already said "no" to.

I have never seen an official statement to that effect. I suppose as long as they don't have to write it, they would accept the contribution of code supporting one additional image format for the purpose of running games in their unaltered form. The current situation requiring some games to be cracked could be seen as an encouragement of piracy, especially given that those very games are listed as "supported" on the DOSBox main page.

FeedingDragon wrote:

The only information for TD0, as far as I can find, was reverse engineered

Forget TeleDisk. As I mentioned before, PSI is completely open-source. TransCopy is reverse-engineered, but the company that made it no longer exists, and Symantec probably does not even know what the hell you're talking about if you asked them, so they won't modify it. And of course, supporting a file format using reverse-engineered information is perfectly legal in and of itself.

FeedingDragon wrote:

So, the choice becomes, lets write a new routine to handle a different file format, or lets use what is already there to read the data, and just write code that would alter how we read/represent that data.

No, that's not the choice. The choice is to

1. write completely new code to process a complex info file and integrate it into the existing very few lines handling raw image data, AND carefully create info files for every single existing copy-protected game out there --- either by hand or by writing a completely new utility for that;
2. keep the raw image data handler as it is and add the PSI and/or TransCopy handler from PCE to be used whenever the user mounts such images. These handlers already are coded and only need to be modified for DOSBox' file and calling convention. No completely new code would need to be written. This would allow the use of the many already-existing TransCopy and TeleDisk/CopyIIPC/AnaDisk images (the latter converted to PSI with a simple command-line tool that comes with PCE) without further modification.

Well, if PSI can handle all the protection techniques, and is covered under the same open source license as DOSBox (they said "no" to something or another because it used a different open source license than they did,) then I would be all for it 😀 From what I understand, there is already a utility to convert a Kryoflux's raw data streams into PSI format (would have to search for it though.) That would relieve me of the duty of writing a converter myself. I have a Kryoflux I've been digging to find (without much luck so far.) I was planning to write a TD0 converter myself, but hadn't started yet because I have no idea what the final image format would be. Though, a TD0, IMD, whatever other formats are used for protected software, converter may still be needed. I don't know anything about PSI myself.

The need to "crack" things has bothered me. First, because I don't like doing it in the first place. Second, because the official DOSBox stance has been, "If it's cracked, we don't support it." Which sort of contradicts many of the "supported" tags on games in the official support list. Though, in the interest of full disclosure, there are quite a few that have officially un-protected version available. Usually on compilation CDs and such. There are also, and I personally find this quite funny, quite a few games that when "updated" with an official patch (going from v1.0 to 1.2 for example,) also removes said copy protection. Can't remember any specific cases, off hand, but I've come across them in the past..... I know one was a CD game, that when you downloaded and installed the "official" bug fixing patch, also removed all the CD checks.

Regardless of whatever route, if any, the developers (or some enterprising third party who is a better programmer than I,) take, I still think it's a good idea to try and compile a list of the techniques involved (for simulation purposes.) I haven't been able to discover any more, but I find it a little hard to believe that there are only 10. Anyone know of any that I am missing?

FeedingDragon wrote:

Also, file formats are rarely if ever patented. They are usually placed under trade secret protection instead. A trade secret, as long as the holder doesn't release the information themselves, is protected indefinitely. As a matter of fact, I've never heard of a file format actually being patented, which requires releasing the information to the patent office, which makes it freely available eventually. The only information for TD0, as far as I can find, was reverse engineered, which doesn't negate trade secret protection. I just grouped it under copyright for simplicity - maybe I shouldn't have done that. That being said, it "may" not be covered under trade secret either, but the fact that the format has never been officially documented suggests that it probably was.

This is wholly incorrect. A patent allows the holder a 20-year right to exploit the invention covered by the patent, and reverse engineering a patent does not negate the right of the patent holder to enforce his patent. A trade secret is protected so long as it remains a secret. Reverse engineering, assuming it is not done by illegitimate means, breaks the secrecy and hence the protection. Otherwise no one would file for a patent or or be able to compete with the secret holder.

Great Hierophant wrote:

This is wholly incorrect. A patent allows the holder a 20-year right to exploit the invention covered by the patent, and reverse engineering a patent does not negate the right of the patent holder to enforce his patent. A trade secret is protected so long as it remains a secret. Reverse engineering, assuming it is not done by illegitimate means, breaks the secrecy and hence the protection. Otherwise no one would file for a patent or or be able to compete with the secret holder.

That isn't what I was taught. The main reason for filing a patent or copyright instead of a trade secret is so you can publish. Either charging someone else to manufacture what you designed, or actually publishing a book. So, if I were to design an "Ultimate Memory Machine" (just made it up, no idea what it could possibly be,) I would have the following options:

Trade Secret: I could manufacture and sell the device for all the profits. Woohoo, I'm going to be rich! Even if you were to take an "Ultimate Memory Machine" apart and figure out how to make one yourself, it's still protected under trade secrets because I didn't tell you how to do it or give you permission. This protection lasts until I decide otherwise. Refer to the Unified Trade Secrets Act. As long as I don't release the information myself (or allow an authorized entity to do so,) it is protected. Gaining the information illegally (via theft or reverse engineering,) does not constitute me releasing that information.

Patent: Uh oh, I don't have the ability to make it myself. I built a prototype and it's awesome! But if I want to make money with it, I have to make them and sell them 🙁 So, I patent it, then advertise & demonstrate it to companies that "do" have to ability to produce it. Companies like it (it is an awesome device after all,) so they offer me payments to license them to make it. Woohoo, I'm going to be not quite as rich, but still rich!!! This lasts (under current laws,) 20 years.

Trade secret laws are a lot more binding, longer lasting, than any of the other options. The problem is, you actually have to officially maintain the secret. It's entirely possible for an author to protect a book under trade secret. However, that would be totally ridiculous, since they could not let anyone read the book without losing that protection. Same thing with a trade mark. What use is a trade mark if you cannot show it to anyone. Companies have been known to first put a new product under trade secret for the first rush of release. Then, after a calculated period of time, patent it and stop actually making it themselves. Instead collecting licensing fees on it, while they work on their next product.

The PCE and MESS methods seem reasonable to mirror. Once we have a good database with the different schemes and availability, then the path ahead should be clearer, too.

As an aside, I've been wondering why newer games seem less innovative, and now I know the reason, they must have all their ideas filed under trade secrets; never to be released! 😀

FeedingDragon wrote:

That isn't what I was taught. The main reason for filing a patent or copyright instead of a trade secret is so you can publish. […]
Show full quote

Great Hierophant wrote:

This is wholly incorrect. A patent allows the holder a 20-year right to exploit the invention covered by the patent, and reverse engineering a patent does not negate the right of the patent holder to enforce his patent. A trade secret is protected so long as it remains a secret. Reverse engineering, assuming it is not done by illegitimate means, breaks the secrecy and hence the protection. Otherwise no one would file for a patent or or be able to compete with the secret holder.

That isn't what I was taught. The main reason for filing a patent or copyright instead of a trade secret is so you can publish. Either charging someone else to manufacture what you designed, or actually publishing a book. So, if I were to design an "Ultimate Memory Machine" (just made it up, no idea what it could possibly be,) I would have the following options:

Trade Secret: I could manufacture and sell the device for all the profits. Woohoo, I'm going to be rich! Even if you were to take an "Ultimate Memory Machine" apart and figure out how to make one yourself, it's still protected under trade secrets because I didn't tell you how to do it or give you permission. This protection lasts until I decide otherwise. Refer to the Unified Trade Secrets Act. As long as I don't release the information myself (or allow an authorized entity to do so,) it is protected. Gaining the information illegally (via theft or reverse engineering,) does not constitute me releasing that information.

Patent: Uh oh, I don't have the ability to make it myself. I built a prototype and it's awesome! But if I want to make money with it, I have to make them and sell them 🙁 So, I patent it, then advertise & demonstrate it to companies that "do" have to ability to produce it. Companies like it (it is an awesome device after all,) so they offer me payments to license them to make it. Woohoo, I'm going to be not quite as rich, but still rich!!! This lasts (under current laws,) 20 years.

Trade secret laws are a lot more binding, longer lasting, than any of the other options. The problem is, you actually have to officially maintain the secret. It's entirely possible for an author to protect a book under trade secret. However, that would be totally ridiculous, since they could not let anyone read the book without losing that protection. Same thing with a trade mark. What use is a trade mark if you cannot show it to anyone. Companies have been known to first put a new product under trade secret for the first rush of release. Then, after a calculated period of time, patent it and stop actually making it themselves. Instead collecting licensing fees on it, while they work on their next product.

You can publish a product and keep trade secret protection, consider the recipe for Coca Cola, the classic Trade Secret. If you stole the formula from the company (insanely difficult to be sure), that would be industrial espionage and Coke could sue you. Anyone can buy it and try to reverse engineer it in a lab, but if they discover a formula that can be implemented cheaply, then Coke's share prices will fall. Reverse engineering is legitimate unless the law says otherwise.

Reverse engineering an "obsolete" file format, no longer protected by a patent, is no different than how Compaq, Phoenix and the rest cloned IBM's PC BIOS. Even if IBM hadn't published the code, a trade secret would not have protected the BIOS from the clone makers. Similarly, if I bought your "Ultimate Memory Machine", reversed engineered it and figured out what it did, I could market my own brand of the product without your permission. Otherwise, no one would ever need to file for a patent. If you licensed the trade secret, the licensing terms would always require the licensee to keep the secret. If you read the Comment to Section 1 of the Uniform Trade Secrets Act, you will see Reverse Engineering under that portion of the text identifying "Proper means" of discovering a trade secret.

The Cocoa Cola reference doesn't apply, they are producing the product, and selling the product, without publishing the recipe. It's the recipe that is a trade secret, and not the product itself.

As for reverse engineering being considered a legal means of acquisition, are you basing that off the Wiki? That's the first place I found that referenced it (though I did find it in other places as well.) First, reverse engineering is never listed in the actual code itself. The only places it is located is in commentaries of the code. Second, in the places I found it, I found it in 2 forms. The first just listing "reverse engineering" as acceptable, the other listing "reverse engineering, unless prohibited" as acceptable. Reverse engineering of a trade secret is a risky proposition. In these cases, it's left up to a court to decide (on a case by case basis,) whether reverse engineering of a trade secret is or is not legal. On digging out my text book (took a while, I'm just glad I had it here,) and re-reading the code in detail, I did, however, come across a stumbling block that you didn't bring up. A trade secret does lose its protection if it is no longer profitable. The only issue with that is, that it is still up to a court to decide, and is not arbitrary.

If you want something more specific to reverse engineering protections, why is there not an exact duplicate of Cocoa Cola available? Or Pepsi for that matter. They are still the top sellers of cola drinks, and yet they are still singular (for over 20 years, so patent doesn't apply.) A detailed chemical analysis, and some trial and error maybe, could give any company that wants to try everything they need to produce an identical product. Yet it hasn't been done. Maybe it would cost the same to produce, maybe they could find a way to make it cheaper (outsourcing production maybe.) But still, nobody has done it, even if to just give themselves an "in" to a market that is mostly exclusive. To remain with your Cocoa Cola reference 😀

The key here is that, yes, Grand Island Cola (made up company, apologies to you if you really exist,) could analyze and reproduce exact replica's of Pepsi & Coke, then release them as their own products (they obtained the recipes through acceptable means, right?) But they would then be immediately taken to court over the issue by one or both of the other companies (Coke & Pepsi.) They would spend huge amounts of money in court defending their "legal" acquisition. Coke & Pepsi have a financial stake to protect the recipes, and even if they lose it's still worth the attempt (they get to paint Grand Island Cola as a thief in public opinion.) If they win, the maintain their exclusive product. Meanwhile, Grand Island Cola is facing bad publicity, the possibility of losing in court and facing punitive damages, just to maybe gain "equal" footing.

In my class notes, it was commented by the instructor, that the key to trade secret protection is taking steps to maintain the secret. Putting a seal on a piece of hardware denying the end user permission to open it up is all that is necessary. Covering a key screw, putting a sticker across a seam, etc... These were specifically mentioned. Also, just putting a line in the manual stating that opening up the product isn't allowed, was sufficient, but you then had to ensure that sales of a used product always included the manual. Extrapolating for software, putting a notice in the software somewhere, even if it isn't easily seen (tiny print,) just seen at least once (during installation maybe,) is sufficient. The steps didn't have to be effective, they just had to be present.

The point is moot though. If the DOSBox authors want to take the safe course, that is their prerogative. If they want to take a chance, knowing that there are 2 avenues of defense and are not likely to face the issue anyways, that is again their choice. My main question is, as long as they can come up with their own format, or use one that is completely open source, why take any chances at all? They are under no obligation to make things extra easy for us. There are also plenty of end users more than willing to create converters for favored formats. The only thing I'm going to ask for is "something", "anything" to let me play my games again 😀

Great Hierophant wrote:

is no different than how Compaq, Phoenix and the rest cloned IBM's PC BIOS. Even if IBM hadn't published the code, a trade secret would not have protected the BIOS from the clone makers.

This I actually learned in taking IT classes at TSTC. The original 8088, it's structure, BIOS, etc.., when released by IBM was done so without any protections. An internal memo shown in the class read (paraphrased, don't have a copy of the actual memo any more,) "The home computer will never produce a viable market, but there is some interest, lets put something together just to showcase for the more reliable business market." Literally everything about the original 8088 IMB PC was free for the taking. From the ISA port, to the BIOS, to the memory structure, everything. Everything that followed on the PC market comes down from that. It's why the expansions for the PC were so much cheaper to obtain (someone making a sound card or graphics card didn't have to buy a license to the ISA interface.) IBM later tried making a proprietary interface so they could start taking advantage of the market that grew (that they didn't believe would,) but the interface failed miserably.

Well, I haven't been able to find any more protection methods. So, for now, I'm going to assume that the 10 I listed are all of them. I've also gotten my Kryofulx up and running 😀 It will produce IMG files of copy protected disks without complaint (Ultima 1-5,) but nothing I have will open them 🙁 My next project is the examine the IMG files to see if I can figure out how they are being structured. I'm not sure if I'm up to that task or not. I have not been able to find the page I referenced earlier concerning track skew, even repeating searches on the expansion cards that I had looked into earlier (Deluxe Option Board, Catweasel, etc...) Maybe it was for a different system entirely (though I can't imagine what I could have been searching for to find information for a system I have no interest in.)

When it comes down to it, the exact format used for disk images isn't really important. If the authors want to use one already in existence, or create one of their own, it doesn't matter. All I'm asking for is that I can image my disks that I cannot (for one reason or another, usually speed,) play on my vintage system, so that I can play them in DOSBox. Right now, I have RAW stream files archived for my PC disks (as well as Amiga, Apple II, C64, & Atari,) that I currently have access too (have a lot that are in storage, and I cannot get to right now.) There are tools already written to convert those to several formats. Some I understand, some I don't. As it stands now, the only way I can play these is to "crack" them. Something I am loathe to do, and which the authors have stated in the past isn't supported 🙁

Perhaps google for how others have handled each game and its format, such as use of PCE (?) or MESS. It may be interesting to catalog which emulators already handle the existing methods. Once you have this, given the emulator has available source code, then that should help document the structure of these exceptional floppy disks.

truth5678 wrote:

PCE (?)

http://www.hampa.ch/pce/

FeedingDragon wrote:

It will produce IMG files of copy protected disks without complaint (Ultima 1-5,) but nothing I have will open them

PCE comes with the pfi, pri and psi command-line utilities that can open Kryoflux stream files and convert them to other formats including TransCopy and TeleDisk.

Not having much luck with PCE. I haven't tested the tools for conversion, not much point in making images of my Kryoflux dumps if I can't actually use them. DOSBox is so easy, I really wish there was some way to get the copy protected games I have running in it. I now have my available collection (many still in storage,) archived, and can re-master them for play on a real machine, I just don't want to have 6 machines just to play everything.

Let us know if you get anywhere. I just ordered my KryoFlux and may be facing the same issues. At least I have a more modern Win32 decrypter for the CPC games. Not sure if I have any other types of disk based CP games, but I will probably be starting to hunt down non-PC games now that I will be able read the disks.

In all honesty, I haven't been doing a lot of searching on this subject for a while. I've been concentrating on finishing up my PIII build and archiving the disks I currently have access to. Turns out, I had almost all of my Commodore 64, Amiga, Atari, & Apple II disks with me, but only a very small percentage of my PC disks. Sort of ironic as I only have my PCs with me, and my other computers are in storage 🙁

I've actually, in my attempt to test out my archival dumps of my disks, been doing a lot of searches for converters for other systems. The Kryoflux DTC software actually has built in converters that work just fine (in most cases,) with Commodore 64 & Amiga disks. It's the Atari & Apple II disks that are proving the hardest to verify. Had to find a 3rd party converter for Atari 8-Bit to convert those disks to a format that maintained the copy protection for testing in emulation. But all the images work just fine. The Apple II is proving problematical, as I can't find a converter for any of the formats that will work with copy protected games. All the games I have for the Apple II include copy protection, so I can't test those.

The PC disks, are the easiest to test the dumps with, just re-master the image onto a blank floppy and see if it loads on my PIII system (it has 5.25" & 3.5" drives.) Can't play most of them there (too fast, and in some games no CGA-Composite card.) But I can see if the game actually works. I guess, if I wanted to put up with incorrect graphics & loss of audio, I could use MoSlo and play them, but I prefer to bring back the original feel as much as possible.

I've been doing more reading up on how to directly access the floppy drives, and from what I've been reading, the list I have is pretty much complete. Since DOSBox would be more accurately described as a simulator instead of an emulator, I'm not sure how effective incorporating another image format would actually be. From what I've been reading, the copy protection coverage of true emulators (PCE etc...) is done by making the HW emulation more accurate. Since DOSBox actually just simulates the HW, something different would probably be called for. So I still think my header idea would probably be easier to implement in this case.

Looking at the code, it seems to just be set up for standard parameters only. Directly reading to/from the image file based on fixed geometry values. I cannot see a way (with my limited ability,) to alter the code to handle the more complex nature of copy protected disks. So totally new routines will need to be written. Everyone seems to prefer turning simulation into emulation for disk access. Simulation would probably be easier to implement, but would require the image file to be more interpretative, and more complex to build. Emulation would be a lot more complex to code, but the image file would be easier to make more standardized (raw MFM flux data could actually be used - from what I've read on some of the emulators.)

Sorry if I'm rambling a bit, it's 4am now, and I'm only up because my meds aren't working as well as normal 🙁 I'll keep looking at the code when I can, but I'm admittedly out of my depths a bit.