Wrote a tool for GDEmu users,for converting redump gdi format to general gdi format

Source code download:

https://github.com/SONIC3D/gdi-utils

Written in TypeScript and compiled JavaScript target code running in node.js(6.x or later) environment.
Yet no GUI version, only command-line version.​

Why another tool for gdi:
During my using of GDEmu I found redump released games with split data track(Patter III type of High Density Area in Sega Docs) would cause boot failed or other problem on GDEmu. And GDROM-Explorer(1.6.3) will pop-up the error message indicate there is overlapping problem in these dumps.
Some of them have Tosec equivalent release for the same game title as alternative.But for the other redump exclusive dumps, I have to find my way to make them be working with GDEmu or working with GDROM-Explorer for further content extraction.

For example:

• Fushigi no Dungeon - Furai no Shiren Gaiden - Jokenji Asuka Kenzan! (Japan)
• ChuChu Rocket! (Europe) (En,Ja,Fr,De,Es)

Though I'm also a member in redump.org, I have no experience in Dreamcast GD-ROM dumping and the data structure of it.
But after digging up with the sectors data of exist release, I found the problem is caused by the embedded Pause/PreGap Data and failed to easily identify if a dump is in this gap embedded format without reading the actual content of the track(this should originally be finished at gdi cue parsing time. Parsing the track content to archive this is not a proper way for any ODE implementation).

Check the "docs" directory in my project source for detailed notes and example or looking at the #10 post of this thread.

I tried to manually fix that last year and got succeeded. So I decided to write a tool. But real life is too busy to allow me finish that.
Finally I decide to make it done 2 weeks ago and that's it. I personally tested its current version with 2 games and they seems all works now.
I'm not sure if it suitable for all cases. So if anyone interested in test it, just get it from the above link, take a look at the ReadMe to compile and have a play.

* Code is written in a rush. Feel free to report any bug. I will fix severe bug when I'm free.
Thanks.

 

Haven't tried this tool since I don't have any of their broken images on hand, but if there's ever a scenario where this fails you can always just extract the image data and rebuild it with GDIBuilder. I've used that approach in the past to solve the same problem.

 

actual source of problems - GDI format is not designed to contain offset-corrected tracks. as well as sad fact Redump guys think it is.

 

It's actually not offset correction info.It's the embedded pregap data or so-called "pregap and pause data" in GD-ROM format specification docs ("Gdfm_k214e" and "GDfm_s132e").

I have no contact with the GD-ROM dumpers in redump.org. But I guess they want to complementing the track data and make the tracks data continuous in sector. It's a bit like multi file cue/bin format to CD-ROM image released by redump.

For cue/bin file format.You can use single bin file with timecode as index in cue for identify the tracks.And more of that,cue sheet syntax allows two types of pregap description.The first way is using 'PREGAP' keyword,that means the actual pregap data does not exits.It's just imply there is a such gap with specific time in length.And the second way is using "INDEX 00" and "INDEX 01" to mark the pregap area in the provided file.This is exactly what redump.org release do in the CD dump release.And although these release are in the form of multi track files, these files can be binary concatenated directly into a single bin file and the rest works of modifying the cue file to adopt this single bin file can be very easy.The track file is continuous in binary data.

So back to the gdi released by redump.org. I found it's very similiar. But it seems wrong CUE or TOC data representing format has been chosen. As ".gdi" is a very simple format, there is no way to add additional "INDEX ??" annotation in it like what ".cue" does.So when loading ".gdi" files, ODE like GDEmu may have to guess if it's ok to directly treat the track file content and it's LBA as user data directly or pregap data integrated.Without parsing the data content of track, it cannot be determined. But data content reading should not be happened technically in TOC parsing phase, just like GD-ROM drive loading GD-ROM disc, there shouldn't be any reason to read the actually track content at the TOC reading phase.
Additional reference can be found in this thread:
http://dcemulation.org/dumpcast/viewtopic.php?t=537

And GDI release in redump.org format has additional changes to the ".gdi" file.Makes parsing this file a little bit harder than general gdi format. Track id leading white spaces for indentation(in a file that usually not read by human) and quoted long file name with space makes parameters cannot be simply delimited by white spaces. Though it won't be big problem, I still think a new file extension name should be used in such format evolvement, personally.

 

http://dknute.livejournal.com/42778.html#comments

Read the comments also...

 

Yes, I have read it 3 years ago, before I purchase GDEmu. And that's the initial reason why I decided to write a tool to do conversion. Help users skip the knowledge of GD-ROM logical structure and the proper way of dumping and data structure of storing. Just focus on how to convert data from a format that not easy to adopt to a format that already known compatible.

Beside the idea is not good for storing pause data and pregap data in track file of an exist dump format. I'm agree to the redump format for removing the last 75 sectors of data in the last audio track of Patter III type of high density area, which the general gdi format choose to keep it. By referencing the docs (in attachment) I think those data are not the actual audio data belongs to the last audio track as it is in Pattern II type of HD area. But padding with these data of course will not affect ODE like GDEmu or emulators.

And by using this converter, I found that there are games improperly dumped for audio tracks in TOSEC set like "King of Fighters, The - Dream Match 1999 v1.004 (1999)(SNK)(NTSC)(JP)(M4)[!]", the audio data offset seems wrong. And the redump one ("King of Fighters, The - Dream Match 1999 (Japan) (En,Ja,Es,Pt)") seems good. What I mean "offset" is definitely not that "pause data" or "gap data". So there is still valuable for this tool to help check dumps via convert and compare exist data from redump dump source.

 

the flaw is so small also how many games use the 3rd pattern? and how did you find that 1 sec on the dump? can the gdrom read that 1sec ?

 

1. Download the Dreamcast gdi cue pack from redump.org website.
2. Extract the pack to any folder and use EmEditor to do the Find in Files with "4 2352", you would get a long list of result.
3. In the result list of step 2, search with regexp "\n(.+)45000 4 2352(.+)\n" and replace with regexp "\n", all track 3 results are filtered out.
4. In the result list of step 3, search with regexp "\n(.+)(\s+)0 4 2352(.+)\n" and replace with regexp "\n", manually take care of the 1st line and press the Replace All button for multiple times until there is no changes left to the file for this replacement changes the cursor position may cause result to be skipped. Now all track 1 results are filtered out.
   
5. In the result list of step 4, search with regexp "\.gdi\((.+)\n" and replace with regexp "\n", filtered unnecessary text with only disc title left.
6. Now you get all the disc name list with patter III type of HD area, so there are 356 disc images in redump releases.
7. For easy result, just download and check the file in attachment.The "ChuChu Rocket! (Europe) (En,Ja,Fr,De,Es)" mentioned in your referenced Deunan post is also included in the list.

I don't have full Tosec or Trurip .gdi files for checking. I think it would be more than 356 as Tosec and Trurip dumps are more than redump's.

Just hex tools, calculator apps, official documents and patient are enough to find it.

And I think it can be read via GD-ROM driver on dreamcast can read this 75 sectors region as it should be valid "audio data" as it's claimed on official docs on page 9(the second image attachment of my last post) and the httpd-ack do save the result data. I just think it not a proper way of saving this chunk of data to the last audio track though it will not affect anything in program running or audio track playing.Just a 1 second longer track than it's designed to own(should be owned by the next data track but for general gdi format, this chunk of data should be ignored as it's not the "user data" of this track).

 

nice thanks for the list
but about the 1sec i believe it is separated from last audio track i mean even if the dreamcast can read it as audio it will be as track 20 with 1 sec, I dont see the point creating 1sec longer audio track. Anyway as we agree this 1 sec will not hurt anyone , But i am not qualified to answer this properly

 

By care fully investigate a little more, I found that all httpd-ack stores the 150 sectors (2 seconds) of zero filled data of Index 00 area of the *NEXT* track to the tail of current track since track 04 to track N-2(N here is the last track index).And for track N-1, it stores only 75 sectors (1 second) Index 00 data of the next track as I said in above posts.

I have made a detailed excel table to explain that and it seems the problem of "ChuChu Rocket! (Europe) (En,Ja,Fr,De,Es)" can be explained now. In a short,I think the redump version has no "overlapping" problem.But the dealing ways of tracks since track 04 to the last track are different from that to the general gdi format.

Check the attachment if any one has interests on it. Or check out the git project of this project, I have this excel file stored in the "docs" directory of this project. The blue region and yellow region represent the actual data that the gdi track files stores over the disc layout view. White areas between blue are data that not stored in general gdi format. Redump dumped those data and stored it.So the whole track should be plus 150 sectors to the proper FAD,too.Thus,there would not be any overlapping.

 

Thanks a whole lot for this

 

@SONIC3D could you please give me a step by step guide on how to use this tool to convert the Redump GDI dumps to be able to be used on GDEMU? I tried to figure it out but I have had no luck

 

I will add a tutorial video here soon.

 

Look forward to a tutorial

 

many thanks for this !! I gave it a shot on Street fighter Zero 3 and it worked with GDEmu

 

Could you please explain how to use the tool?

 

hello,

everything is written and self explanatory on the github...

Usage

1. Install Node.js runtime environment for your OS(https://nodejs.org, 6.x LTS version is enough).
2. Open terminal or console with node.js envrionment initialized and do the following steps.
3. Update NPM(sudo npm update -g npm).
4. Install Typescript compiler in your OS(npm install -g typescript).
5. Open terminal and git clone this project(git clone https://github.com/SONIC3D/gdi-utils.git).
6. Change working directory to project root(cd gdi-utils).
7. Run npm install to install all required components for compiling.
8. Run npm run build to build this project.
9. Run node bin/bin-nodejs/index.js to see the help
10. Run node bin/bin-nodejs/index.js <your_gdiimage_dir>/input.gdi 0 <output_dir> to do conversion.

 

As I have no disk space to record any video tutorial, I coded a GUI version to help user with no development experience.

Download pre-built Win32/OSX binary here:
https://mega.nz/#F!uhs2yJqY!rkezPQGj2E7j28VtuWt2RA

How to use:
I think the UI is self-explained.But please note that the conversion progress is using blocking I/O and I don't have time to refine the code.So after you clicked the "Start" button, the program turns into no response state.Please wait the disk I/O patiently for around 30 secs(SSD may save your time.).If you see the log shows up, then the conversion finished.

P.S.
The input GDI should be in Redump GDI format, or the result output data may not be correct.

Source code:
https://github.com/SONIC3D/gdi-utils-gui
How to compile:

1. Checkout the repo.
2. Install Node.js with npm.
3. Run `npm install -g typescript` to install TypeScript compiler(May require root privilege on Mac OSX).
4. Run `npm install` to intall required library for this project.
5. Run `npm run build` to build the whole project.
6. Run `npm start` to run the built code.
7. It's actually powered by electron framework.So if you decide to compile and pack it yourself,please check electron doc.

Future development:
This is a pure format conversion tool for in-house fun. And it seems good and stable in doing its job. So I won't add any feature that not directly related to GDI format conversion purpose. But if anyone found bug in my program. Please tell me. (Game title name is required and the conversion log is another good source to trace the bug). I'd like to fix it ASAP.



@Night-Dweller

 

Great，thank you

 

I'll submit 2 that don't work .