Building a valid GDI from scratch (Help wanted)

How to Build a valid GDI from scratch

I recently purchased a GDEmu board, and one of the things I was hoping to do is experiment around with existing games by building custom GDI images. Unfortunately, after a few days messing around I've gotten stuck. There appears to be some things about a valid GD-ROM that either aren't documented, or I can't find information about them anywhere. I've gotten to the point where I can build a GDI that GD-ROM explorer believes is valid, and the DC shows the disc art on the music player, but when you try to start it you're asked to set the system clock twice then returned to the BIOS menu. Update: Solved. See below.

First of all, here's something that I haven't seen documented. Marcus Comstedt's page on IP.BIN files doesn't have anything about the TOC section. This section is just a duplicate of the track layout in the high density section. It goes from 0x100 to 0x288, and allows up to 97 tracks to be defined. (The first two tracks are part of the Low density area, thus making up the rest of the 99 track limit)
At 0x100 is the TOC header, the ASCII "TOC1". This is followed by 97 four byte entires for the remaining tracks on the disc. Each track consists of a 3 byte, little endian LBA value. This is the start LBA of the corresponding track, with 150 added to each track for some reason. So the first track in the high density area will always be 45150, and so on. The 4th byte represents the track type. 0x41 is a data track, 0x01 is an audio track. Unused tracks are filled with 0xFF.
Fairly simple, no?

So as part of my experiments, I first attempted to start modding an existing game, but when I ran into repeated problems, I decided to start over with something small and simple. The goal: rebuild the SA2 Trial demo disc from scratch and make it be bootable. It's less than 100MB and only one data track, so it seemed like a good place to start.

Steps:
1. Dump my copy of SA2: The Trial to GDI with httpd-ack. Interestingly, this disc is region free according to the IP.BIN. I didn't realize that going into this. (Does anyone know of other Dreamcast discs that were region free?)
2. Confirm the GDI boots, by testing on hardware and Makaron and NullDC emulators. [OK]

Since my intention is to use mkisofs to make the track image, which outputs raw data, first let's make sure I can do that.

3. Convert track03.bin from the game dump into raw 2048 sectors. (I used CD Image converter, to convert from 2352 Raw Mode 1 to 2048 sectors)
4. Update the .gdi file, changing the sector size for track 3 from 2532 to 2048.
5. Confirm that the original .gdi (now with 2048 byte sectors) still will boot. It does. [OK]

At this point, I know I won't need to convert my image back to 2532 sectors, (which CD Image converter can do) so that will make things a bit easier later on.

6. Extract the existing data and IP.BIN from the dump made in step 1. I saved these to a folder called data.
7. Rebuild our track03 image using mkisofs:
mkisofs.exe -C 0,45000 -V VOLUME_NAME -G IP.BIN -l -o data.iso data

We now have a new track 3, called data.iso, with the original IP.BIN as the boot sector.

8. Modify the .gdi to point to our new track 3, data.iso.
9. Pad the new ISO back to the original size by adding 00's to the end. This will allow GD-ROM Explorer to open it again. [OK]

At the initial time I posted this, I got stuck, because the resulting image would not boot. However, after watching the disc reads in a debugger, I was able to solve the final piece of the puzzle. It would always read the first sector of the TOC and stop. (It's only looking for 1ST_READ.BIN, it doesn't care about other files at this point)

10. The DC is going to read the TOC, and as soon as it finds 1ST_READ.BIN, it's going to look at the location of it. If the LBA of the file is wrong, it won't be loaded. The end of 1ST_READ.BIN must always be 150 sectors before the end of the image. (There's that 150 offset again that we saw in IP.BIN) In other words, if your 1ST_READ.BIN is 1,297,060 bytes in size like the SA2 demo, that means takes up 634 sectors. (Ceiling of 1297060 / 2048.) A complete DC disc is 549150 sectors, so 549150 - 150 - 634 = 548366. Your 1ST_READ.BIN in this case must be located at LBA 548366.

I can't really provide easy instructions for this step at the moment. In my case, I copy/pasted it there in a hex editor. Then I manually updated the location in the TOC to point at the file so it was in the same place as the original disc.

11. Test the new image. The disc will now boot.

Thus, a summary of the requirements:
1. Data must be built with the LBA starting at 45000. If the game contains CD audio, you'll need to follow steps similar to a DATA/DATA MIL-CD, where you create multiple track sessions. This is probably going to require a custom tool because it will get complicated.
2. IP.BIN must be injected to the boot sector at 45000, just like an MIL-CD would be.
3. Image must be always be a normal sized GD-ROM, due to requirement 4.
4. 1ST_READ.BIN must end exactly 150 sectors before the end of the disc.

I hope this information was useful to someone else. I haven't decided yet if I'm going to write some new tools for this process, or just take care of the 1ST_READ.BIN moving process by hand.

---
As a final note, yes, I could easily accomplish what I need to do by making a CDI image. But nobody is going to learn anything new if I do that. (And I suspect for some later titles that using GDI for modifications will avoid some nasty protection issues in the future.)

 

Nice explanations, thank you.
I will check later whether gdrom explorer really needs to check the file size. Maybe we can skip the padding operation.
About the 1st_read bin file, why not patching the boot sector so that it fetches it from another LBA? Then you would just need to use a sortfile with the 1st_read.bin file being placed at the first position. Makes sense?

 

Awesome post, thanks for all the info!

 

I'm not sure if that's even possible. According to Marcus Comstedt's documentation on it, IP.BIN is not responsible for loading 1ST_READ.BIN, the boot sector code runs after the BIOS has already loaded it to memory. Thus I think the location is a hard requirement. The 1ST_READ.BIN data is already loaded before the SEGA screen appears. Feel free to correct me if I'm mistaken, but from what I've seen it can't be moved.

On a related note, I found a... I guess not really a bug... in your GD-ROM explorer as a result of this. I produced a valid disc that boots on my DC, but your tool can't read the High Density area of it. One of the discs I wanted to rebuild has CDDA, and I was lazy, so I basically did this:

1. Built my data image, to find out how big it would be.
2. Did the math and calculated what the LBA needed to be in order to fit all of the data into the final track, with extra room for me to copy/paste 1ST_READ.BIN to the end. (The sort order file exported by your tool doesn't seem to be properly applied for some reason. It makes mkisofs take longer to generate the ISO, but 1ST_READ.BIN, which had the lowest priority, ended up in the middle instead of the end for some reason.)
3. Re-built the ISO for the final data track using the LBA I calculated in the previous step.
4. Built the ISO again for track03 with the 0,45000 LBA
5. Chopped track03 from the previous step down to a few MB.
6. Copied the TOC from the final track over the top of the track03 TOC. In other words, it's going to read the root directory information from track 3 and end up seeking to the same information in the final track instead because I blatantly copied over the LBAs. None of the retail discs I've seen do this, but there was nothing preventing it from working since the only thing the game cares about is the LBA and not the track itself. (This might come in handy in the future if someone wants to use it for copy protection)
7. Manually calculated new locations for the CD audio tracks. Updated the GDI and IP.BIN files to point to these new locations. Of course, since I had already built the image, I needed to update IP.BIN inside of track03.
8. Moved 1ST_READ.BIN to the LBA it needs to be at, and updated the TOC in both track03 and the final track.

With that, I had a bootable disc with CDDA, with a lot of manual steps involved.

During the process, something else I observed was that for games with CDDA, track03 seems to only contain the TOC followed by padding space. The data itself is always stored on the last track. (Which has no TOC of its own, it just starts with a file right away. My manually created GDI violated that.)

It should be pretty easy for me to throw together a tool to expand an ISO built with mkisofs to the correct size and move 1ST_READ.BIN, but to handle discs with CDDA is probably going to require building a brand new tool to create ISO images. I haven't decided yet if I'm going to keep hacking them together by hand, or actually create a tool to do it. Right now I only plan to put together a few discs, so it's quicker for me to do it by hand.

 

If you wanna recreate a 5+ tracks game I'd suggest this:
- make a single track03 that extends to the end of the disc with zeropadding for lower lba.
(Put a dummy file called 0.0 with the right size and add -hide 0.0 in the mkisofs command.)

- Cut that big track in three chunks.
One chunk from LBA of first file until the end.
One chunk from LBA 45000 until LBA of first file minus size of audio data (in sectors)
Discard remaining chunk that should be full of zeroes.

You might need to update the IP.BIN TOC for the audio tracks.
You need to take into account the gaps between different kind of tracks. I don't remember by heart, but I believe it's 150 for data->audio (& vice versa) and 75 between audio tracks, though I'm not sure.

 

The gap between data and audio appears to be 150 sectors. The one disc I checked so far has no gaps between audio tracks, which is probably universally true since CD audio usually doesn't have gaps. (For CD's where they separate a long continuous recording into tracks)

I found some code yesterday for building ISO9660 images, and spent most of the evening butchering it up. (Removed Joilet, moved the TOC before the files since this code put it after, added LBA stuff, push file data to the end of the disc, etc.) I've got code now that builds a working track03.bin without any manual intervention (You provide IP.BIN and a data folder, it does the rest) Just need to finish up adding CDDA support and then I'll clean it up and release the tool along with source code.

This will save me a bunch of time, since I need to make some additional changes to the image with CDDA that I put together on Friday.

 

So will this allow converting a CDI homebrew to GDI image, so they can be used with UGC?

 

No. In order to do that, much more have to be done. (undone actually)

This will allow for mods.

 

This should be considered alpha quality, use at your own risk, etc:
http://projects.sappharad.com/tools/GDIbuilder_alpha.7z (Note: This is an old version, you should probably see the new thread for the finished tool)
This should not be considered a good release at this time. I thought I promised a build today but I actually didn't. Either way, this can be used to rebuild both data only and GD-ROMS with CDDA.

Provide a folder with the data files, IP.BIN to embed, any CDDA tracks you want to include (make sure you order them in the order you want them) and hit build.
It'll output track03.bin, and if CDDA exists, whatever trackXX.bin will be. Then it will display a dialog with the lines you need to put in your GDI file for tracks 3 and up.

To do list:
- Implement the progress bar
- Make the volume metadata configurable. Disc name is hard coded to DREAMCAST right now.
- Make the file sort order configurable with a sort file. (Does anyone actually want/need this?) 1ST_READ.BIN must always be the last file, so that one is not negotiable.
- Add an option to generate 2352 sector tracks instead of 2048. (This is a low priority, since CD Image Convertor can convert the output from this tool for you)
- Port to OS X.
- Post source code on GitHub.
- Make new thread when first "official" release is done. (Since I can't adjust the topic and it'll be good to promote this when finished)

As -=FamilyGuy=- stated above, he's absolutely correct. This is not useful for homebrew, only for modifying files in actual games. Hopefully we solve the problem of fan translations requiring a GDI as input and suddenly spitting out a completely different format as output. (Source code will probably help with that too in case anyone wants to use it in another application)

Does anyone else have a need for this tool? Any suggestions missing from my TODO list above?

Edit: Tool completed. See this thread:
GDIBuilder - Tool for building GDI data tracks