Hello my friend Wouter, how are you?
First of all, PDI was created to be used exclusivelly in a MSX computer with it's FDCs and limitations.
For example, the READ TRACK COMMAND doesn't work properly in the FDC's of all MSX computers.
The main goal of it is its simplicity and the facility to be used in DSK-PRO.
I don''t want to create something very complex because nowadays we don't use disks anymore so no more disk locks will be created,
So the main ideia is to preserve the MSX disks that we already have. And I can assure you that PDI is good enough for that.
Answering your questions:
* How are disks with more than 40/80 tracks represented? Just append the extra tracks at the end? If so, what is the meaning of the byte at offset 1?
All PDI files goes to the track 41 or 81. It depends on the format chosen to create the file:
1 - 40 tracks single side (PDI file has 42 tracks) [FC]
2 - 40 tracks double side (PDI file has 84 tracks) [FD]
3 - 80 tracks single side (PDI file has 82 tracks] [F8]
4 - 80 tracks double side (PDI file has 164 tracks) [F9]
So all PDI files has extra tracks because in these tracks may have some locks.
So the byte of offset 1 indicates that. Just to be clear: the byte of offset 0 and offset 1 repeats over and over again just for security purposes, but this informations is the same for the entire disk.
* What track layout does the PDI format assume? (Standard IBM format?) What is the length of a full raw track? (Always exactly 6250 bytes?) At offset 3 there are 5 bytes for "formatting gap bytes", are these the filler bytes for the 5 gaps in the standard IBM format? What about the length of those gaps?
Yes, it assumes the IBM standart format. (http://1drv.ms/1MUs0g9)
And yes, the 5 bytes are those with 4Eh values.
And as the READ TRACK COMMAND does not work properly in MSX FDCs, it is not necessary to have the length of them,
* In the PDI spec I only see room to store 1 pair of CRC bytes, I assume this is the CRC for the sector data? Does this mean the CRC of the sector header is not stored?
There is a mistake in the draft. The CRCs are for the header of each sector, not for the data. Actually there is no way to read the CRCs for the data. Just using the READ TRACK COMMAND but as I told you, it does not work properly .
* Per track there is a data buffer of 5120 bytes, does that mean PDI can store at most 10 sectors of 512 bytes per track? What if my protected disk stores more data?
Yes, it means exactly that. The maximum lenght is 10 sectores x 512 bytes. Well, I don't know any with more than that, but as I told you, the main goal of the format is for preservation purposes. Just for the protected disks that already exist.
And it is impossible to copy or replicate all possible disk locks. I can create at least 10 locks that anyone in the world can copy using a MSX FDC. Even in PCs there is no disk copier to do that.
I hope it helps you.
Regards,
Marcos Daniel Blanco
First of all, PDI was created to be used exclusively in a MSX computer with it's FDCs and limitations.
For example, the READ TRACK COMMAND doesn't work properly in the FDC's of all MSX computers.
The main goal of it is its simplicity and the facility to be used in DSK-PRO.
I don't want to create something very complex because nowadays we don't use disks anymore so no more disk locks will be created,
So the main idea is to preserve the MSX disks that we already have. And I can assure you that PDI is good enough for that.
Ok, you created PDI to be simple to use in an MSX program. I looked at it from a use-inside-an-emulator point of view (because in an earlier post you suggested exactly that). Maybe I'm a bit biased (because I implemented DMK support in openMSX), but now that I know more details about the PDI format, I'd still argue that for both use cases DMK is better suited than PDI. Let me try to explain:
* The biggest problem I have with PDI is that it can't represent all _existing_ copy protections. I mentioned the 'Sunrise' example in my previous post. Sunrise released many high quality disk games, so not being able to preserve those is a big limitation IMHO. To the best of my knowledge (correct me if I'm wrong) DMK can represent all existing copy-protections (and many (all??) non-existing ones).
* You mention a few times that the READ TRACK command doesn't work properly in all MSX FDCs. I can confirm that, it doesn't always work _reliably_ (a subtle difference from not working at all). Because of this, in the READ-DMK tool, I had to retry the command a few times and combine the obtained results with some heuristics until I do get a reliable representation of the full raw track (see comments in the READ-DMK source code for a lot more details). _BUT_ the MSX programs don't need to execute READ TRACK commands to be able to detect non-standard track layouts. The Sunrise copy protection is one example of an existing copy protection that relies on the track layout (it overlaps sectors to be able to fit everything in one track). Another example could be to measure the time it takes between reading two specific sectors (this depends on the rotational delay of the disk, and thus the position of the sectors in the track).
* This point is mostly personal preference, but I'd say that DMK is actually a simpler format than PDI (and at the same time more complete). I case you haven't yet looked it up, the full spec of DMK is basically this: you have a buffer with the full raw track data, plus the positions of all sector headers within that buffer. That's it. From this information it's very easy to extract the same information as is located in a PDI file, e.g. the track number, side number, sector number, sector size are directly located in the sector headers (and DMK gives you the locations of those headers). From there it's also very easy to locate the CRC of the header (present in PDI), the data-mark (present in hidden form in PDI: the error field), the actual sector data (of course also present in PDI, but only up to 5120 bytes) and the CRC of the sector data (not present in PDI).
I do admit that creating a faithful DMK image from a disk on a MSX might be more difficult than creating a PDI image. On the other hand, if you limit yourself to the disk images that can be represented by PDI (so possibly not a faithful copy of the disk) then representing the data in DMK format is no more difficult than representing it in PDI.
And especially from an emulator point of view, DMK is by far simpler than PDI, because DMK is very close to the raw data in the track and thus very easy to emulate the FDC at that level.
That all said, I'm still looking forward to the release of DSK-PRO. Probably that tool is _a_lot_ more user friendly than READ-DMK. My hope is that someone (you, anyone) steps up, takes the READ-DMK code, improves it, and transforms it into a more usable tool.
Answering your questions:
...
I hope it helps you.
Thanks, it did.
Hello Wouter,
Thanks fot your good explanation!
There are two important things to say:
1 - The PDI format is in its first version and copy almost every protected disk here in Brasil that is the Country with more different disk locks by a mile
2 - The PDI format is by a mile the more apropriated format for DSK-PRO because it was made to it AND I have some limitations in free memory space and other things.
3 - I can create a new version of PDI that copies the sunrise disks. I would be glad to work in this new project.
Just send me the disks that I assure you I can do it.
And I can create a conversion tool between PDI and DMK.
What do you think about it?
I checked the protection of Ys 2 recently, using a Supercard Pro interface, and there are 2 protections on this game:
A missing sector 5 on side 1 (no problem with this one):
250Kbps MFM, 9 sectors, 512 bytes/sector, gap3=84:
0.1 1 2 3 4 5 6 7 8 9
1.1 1 2 3 4 5 6 7 8 9
2.1 1 2 3 4 5 6 7 8 9
250Kbps MFM, 8 sectors, 512 bytes/sector, gap3=84:
3.1 1 2 3 4[g756] 6 7 8 9
250Kbps MFM, 9 sectors, 512 bytes/sector, gap3=84:
4.1 1 2 3 4 5 6 7 8 9
5.1 1 2 3 4 5 6 7 8 9
6.1 1 2 3 4 5 6 7 8 9
...
2 duplicated sectors on side 0:
250Kbps MFM, 11 sectors, 512 bytes/sector, gap3=8:
72.0 5 4 1 2[r] 3[r] 9 8 7 6 2[r] 3[r]
250Kbps MFM, 9 sectors, 512 bytes/sector, gap3=84:
73.0 1 2 3 4 5 6 7 8 9
74.0 1 2 3 4 5 6 7 8 9
75.0 1 2 3 4 5 6 7 8 9As you can see, track 72.0 contains 11 sectors (with 2 duplicated sectors), using a reduced gap, so the Sunrise disks are not the only one to use that kind of protection.
I'm keeping images of my original disks in .SCP format, which is around 12MBytes per disk ! Once compressed it shrinks to +/- 1MBytes. The big advantage of this format is that it's a real physical copy, but it's of course totally unusable for now on emulators... (and it's PC only).
Actually it's not the same protection of sunrise disks.
They use the sector ovelaping technique.
The Ys disk has one difficult lock that is the 2 sectors with same values.
Just using the READ TRACK COMMAND to discover that.
But both are good locks.
Hi Wouter,
I do admit that creating a faithful DMK image from a disk on a MSX might be more difficult than creating a PDI image. On the other hand, if you limit yourself to the disk images that can be represented by PDI (so possibly not a faithful copy of the disk) then representing the data in DMK format is no more difficult than representing it in PDI.
And another thing. The PDI image was created for DSK-PRO 9.0 that has to dump and copy disks in a MSX disk drive. So, here we are not talking about Images, we are talking about direct access to the FDC in a real MSX.
And the best file format for DSK-PRO is PDI. It was created for its needs.
But of course, I can improve DSK-PRO in the near future to give it the ability to copy Sunrise Protections. I just need the disks. If someone send them to me I can do the magic happens.
Unfortunately I am terrible to take someone's codes to make it better. But I can assure you that DSK-PRO 9.0 is your dream come true.
The internal core was totally rewritten from zero. Now when creating DSK files or copying normal disks it does it in the speed of light, I mean, it reads and writes all sectors in the same rotation of the disk. And even the Toshiba chipsets from Panasonics's FDCs are supported, The speed is amazing. At least 1000% faster than DSK-PRO 8.51 (Last Version).
Using Megaflashrom to store the disk image, it creates an image from a real disk [80 tracks Double Face] in 88 seconds (In a Panasonic WX computer with toshiba chipset). In computers with Western Digital or Fujitsu chipsets it is even faster. (Sony, National, etc). Do I need to say more?
So, PDI was created exclusively for DSK-PRO without the purpose to store all kinds of locks, but most of them.
And it still has the ability to be upgraded to a new version with the ability to store protections like Sunrise's and others.
But it works together with DSK-PRO. When I receive Sunrises disks from someone, I can create a new version of DSK-PRO that supports these new disk protections. And it's internal file format (PDI) will be upgraded as well together.
Did you get the spirit of PDI file format?
Regards
Marcos Daniel
Something I had in mind when thinking about the PDI format: if supporting Sunrise (or general "overlapping sectors") requires a deep refactoring of the format (something like a buffer of data and pointers from sectors into it... just saying bullshit here, I have no idea how you plan to do that), it might be better to wait before making the format offical ?
Of course it would work with a "version" byte or word at the beginning of the PDI file, but people wanting to support PDI format will have to add code for the 2 versions of the format (the normal one, and the "overlapping sectors" more complex format).
If the "overlapping format" supports all disks (including the normal ones), it might be better to (officially) release only this one ? Having a format that evolves too much with versions (and is not forward compatible) will probably be a no-go for emulator developers. I think the spirit of the PDI file format should include that "generic" aspect ? What do you think ?
I think we do not need to care about it. DSK-PRO 9.0 is almost done and will be released this week probably.
So, we can not wait anymore. For now, this is the version of the PDI file format.
And you are talking about a file format, and I am talking about the ability to read that from a real disk with DSK-PRO. For now, DSK-PRO can read some types of protected disks, so this version of the PDI handles that.
When I receive the disks from someone, If I receive them, I can start working to create the new version of DSK-PRO and it's own file format (PDI).
Remember, without the disks, I can not do anything. Anyone can send them to me?
Regards
Marcos Daniel
Do you own any Microcabin game? It also uses 11 overlapping sectors...
Unfortunately I don't have any...