Getting a 64DD unit for emulation research

I am now regretting skipping introduction to C and C++. Baka!

 

BUMP!

Reviving this old thread, as I've finally made progress with reading 64DD disks using an Arduino Due.

The process is quite slow for various coding reasons atm, so it takes about 2 hours to backup a whole disk.

I also just found that copying fails on my Mario Artist: Paint Studio disk around LBA 2442, and the drive gives a BIT_AM_FAIL error.
I'm not sure if this is a genuine error, copy protection, or just a load of blank tracks at the end of the disk yet.

But, I don't think the last sectors of this disk is too important anyway, since the ROM part is only around 30MB.

I'm currently using Tera Term to talk to the Arduino and kick off the transfer ("Programming" USB port).
The disk data is then transferred via the "Native" USB port.

atm, I'm using PuTTY to capture the data to a file on the PC. It's still not ideal, but it's one of the few programs which works OK with the bigger captures.
(It does also add "PuTTY log 9-7-14 blah blah" to the start of the file, but this is easily removed with a Hex Editor.)

So, the file for Paint Studio is only around 30MB atm, as it's missing some of the upper RAM blocks.
I think it will be plenty to try to emulate the 64DD now, as the upper "RAM" blocks are likely only for saving anyway.

The only way to force it to read the blocks after a BIT_AM_FAIL error is to do a full hardware reset by toggling the Reset_N line.
I've added this to the code now, so it should continue writing to the file on the PC regardless (it will simply keep transferring the previous contents of the block buffer).

Hopefully this time the System area blocks are intact now too. They certainly look like it.

Pretty sure I can' link directly to the image file here, so we'll have to figure that out.

OzOnE.

 

Good to see work is progressing! Is there some data in the save area that is worth preserving? Isn't it only the 'SRAM' for the disk?

 

If you attach a SD card to the Arduino and use this library you can change "Serial.write(...)" to "myFile.write(...)" and it will write the rom to the SD instead of the serial port.
It's a bit faster, at least when dumping SNES carts.

 

Matters. The F-Zero X disk dump had some default items in its writable area which could be read by Mario Artist. It also had a 'deleted file' entry.
[edit]Technically those could be restored in the case of a disk format so the contents themselves would be present in the read-only segment.

More importantly, there happened to be some overwrite at the end of the disk segment from the last time they had it connected to Randnet. Basically, it's the server request strings and the last viewed image--not sure if it was uploaded or downloaded though. Anyway, it's the only look we've ever had at how they managed their network access.

Better safe than sorry, in other words.

BTW, excellent progress oZoNe!
[edit]That wouldn't be a demagnetized spot if it keeps happening there, right?

 

Yep, it's essentially just the save areas. Hopefully the game will still boot from the captured ROM areas when trying to emulate the drive (or run under MESS etc.)

We should be able to just skip the blocks which give the BIT_AM_ERROR code. I think "AM" probably means "Address Mark"?
That makes me think they are either intentional bad blocks / tracks, or they don't bother formatting them if the game doesn't need all the extra RAM space.

I was thinking of doing that at first, but the serial connection actually seems pretty fast because it's using the "Native" port, so the baud rate is effectively ignored on both sides.
When transferring larger chunks of data via USB, the LED only flashes for a tiny amount of time compared to the time it takes to grab the data from the 64DD.

I also commented out the SerialUSB stuff temporarily, and the bottleneck still seems to be the reading of the data from the drive (due to the evil Arduino digitalWrite / digitalRead routines).
I might try an SD card too at some point, as I know it's easy enough to hook up, and there are plenty of libraries for it (I don't even mind if it just dumps the data on the card without using FAT32. )

Well worth using an Arduino Due btw, as it should be capable of MUCH faster transfer speeds once the code is sorted. They are also quite cheap now.

With it's proper 32-bit ARM chip, native USB, more Flash / RAM, more IO pins, low price, and C code, and Arduino library support it seem the perfect choice for ripping.


@Zoinkity - There certainly doesn't seem to be a way to "force" the drive to read those few tracks which give BIT_AM_ERROR, it simply can't find the valid servo info, and just stops altogether until a hardware reset?
You can hear the drive trying to find the address marker for a few seconds too, then it just retracts the heads and stops reacting to further read commands (until /Reset).

It's best if I just let everyone test out the code and see if we can work through these issues - I'll definitely be posting the Arduino code as-is later tonight.

I'm confident that it's reading the data OK now, but I don't see the "64dd-multi" text that I was seeing on the real N64 transfers?
I think what I was seeing was actually a write TO the disk for a test block?...

(this was an old capture from the FPGA version)...

Think this is a "test" DISK ID, not the real one? (does another read from tracks 7,6,0, then)...
seek_count == 20
asic_data == 0x02660000 Track 614 / LBA 1229 ??
asic_command == 0x00010001 (ASIC_RD_SEEK)
asic_bm_ctl == 0xC25A0000 (START_BM, BM_MODE, XFER_BLKS bits set).
0x5A (90) is the start sector. 85 USER sectors + 4 C2 sectors + 1 Gap sector. First sector of 2nd block on the Track in this case.

asic_seq_ctl == 0x40000000 "Micro PC Enable" bit set.

asic_host_secbyte == 0x00AF0000 <- SecSize (176) minus 1. (bytes to write to the sector buffer)

asic_sec_byte == 0x59EE0000 0x59 (89) = Valid number of sectors in a logical block(excluding GAP)<<16
ie. 85 USER sectors + 4 C2 sectors.

0xEE (238) = ( SecSize (232) + C1 bytes-per-sector (7) ) minus 1. (total bytes to read from disk for each sector)




asic_bm_status == ? (TODO)
363464642D4D756C7469303130310001
54455354000000000000000000000000
00000000087B9CCD00120000B20045F8
00000000000000000000000000000000
0000000000000000000700080009FFFF
000B000C000DFFFF000F0010FFFF0012
001300140015FFFF001700180019001A
FFFF001C001D001EFFFF00200021 <- Got bored (more WORDs to do!) OzOnE.


In ASCII, this first part is...


64dd-Multi0101..TEST............
.....{........E.................
................................
........................... .!


64dd-Multi, then
FS-Type 01, Ver 01,
Attrib 0x00, <- Writing seems to be enabled, so I think this block is for test only.
Disk Type 0x01,
Vol name "TEST "
Format date 0x087B9CCD,
BC 0x12 <- Number of blocks used for ROM file management area.
DC 0x00 <- Destination code, 0x00 == Japan. (0x01 would be US).
Reserved 0xB20045F800 (five bytes?)


Actually, I'm now wondering if I've been looking at the correct LBA for the Disk ID, because the info from Kammedo and others said there are 25 System blocks, which would mean LBA 0 to 24.
It could also be that the "LBA 0" value actually starts from block 26 from the point of view of the normal software?

But, the code from the SDK says...
#define SYSTEM_LBAS 24 /* Number of blocks in the system area */

...which would mean LBA 0 to 23.

(Although, the LBA number is kind of arbitrary, since you only send the Track number to the drive, plus the sector offset within the track.)
I'll see if I can get the Paint Studio image to read with the offsets in the expected places under the SDK DDDump util.

Going out for a meal now, but will be putting together the code etc. for later tonight.

OzOnE

 

"64dd-multi" is a string embedded in a multifilesystem (MFS) header. It would be written in the disk image, not transmitted. It's isn't necessary for a game to use MFS--Dezaemon doesn't, for instance--but really, really sure the Mario Artist titles did to be interlinked like this.

There are two structs, one for ROM segments and one for RAM (implemented a union), and you should see the string at least once. Even if the ROM segment wasn't wrapped in one and they didn't initialize RAM, we know that it could read from an MFS directory, therefore has the MFS library at least in part, and this does test for the string using a static sample.
[edit]
Should specify: the Artist titles could load bonus content from other disks including non-Artist titles, such as F-Zero X. These were kept in an MFS directory, so they minimally require read-only support. Being 1st party though the game likely is 100% MFS.

If you're worried about the state of the code, the beginning of the ROM segment (LBA 0 or LBA 24 or wherever the heck the system region/header stops) should be bootcode, akin to what's at a normal cartridge run address. Typically you can stick a generic cartridge bootstrap to this and "run" it straight up to where the emulator locks for lack of disk access. If it isn't ASM there's something funny going on.

 

Hi, Zoinkity,

I've just spent some hours trying to make ripping faster by doing direct port manipulation.
Due to the crazy Arduino pin ordering, it ended up like this...

// Clear the bus bits first!!
REG_PIOA_CODR = 0x180080;
REG_PIOC_CODR = 0xC03FE;
REG_PIOD_CODR = 0x600;

REG_PIOA_SODR = (data & 0x3000) << 7; // Data bits 13:12 shifted up to A20:A19.
REG_PIOA_SODR = (data & 0x2) << 6; // Data bit 1 shifted up to A7.


REG_PIOC_SODR = (data & 0xFF8) >> 2; // Data bits 11:3 shifted down to C9:C1.
REG_PIOC_SODR = (data & 0x4000) << 5; // Data bit 14 shifted up to C19.
REG_PIOC_SODR = (data & 0x8000) << 3; // Data bit 15 shifted up to C18.

REG_PIOD_SODR = (data & 0x4) << 8; // Data bit 2 shift up to D10.
REG_PIOD_SODR = (data & 0x1) << 9; // Data bit 0 shift up to D9.


That all seems to work OK, but it's still about the same speed as before.

I then saw with the FPGA hooked up that the drive is triggering an interrupt when the Arduino is reading the sector data.
This happens around Word 64, so maybe the BIT_OVER_RUN error I'm seeing is simply because I'm trying to read more that 64 Words (128 Bytes) in one burst?

This was mentioned in the info Kammedo posted years ago, and I suppose it makes sense.
I haven't yet hooked up to the real N64 again to see what it's doing, but I'll do that tomorrow.

So, I could release the code as-is tonight, but I'd rather see if I can fix the above issue first, as it could / should make it many times faster (say 20 minutes, instead of 2 hours).

The data dumps so far still look good, but I can't find 64dd-multi, or even just "64dd" in the files?
The Arduino code now also does byteswapping, so the text looks correct etc. This is mainly because the SerialUSB.write function sends in little-endian.

I'll have to look into the MFS stuff again. It's been a very long while since I messed with that.
Just had a quick look though, and it does seem to confirm that their normal way of numbering the blocks is starting LBA 0 at the first ROM block.

So, the system areas aren't usually "seen" by the user side code. Also somewhat confirmed by this (I'm using MAX_P_LBA as my last LBA anyway, so it should read the whole lot)...

#define BLK_P_TRK 2 /* Number of logical blocks in a track */
#define SYSTEM_LBAS 24 /* Number of blocks in the system area */


/*------------------------------*/
/* system data block address */
/*------------------------------*/
#ifdef KAIHATU /* blue development disk */
#define SYS_DATA_LBA1 2 /* System LBA data recorded */
#define SYS_DATA_LBA2 3 /* System LBA data recorded (spare) */
#define SYS_DATA_LBA3 10 /* System LBA data recorded (spare) */
#define SYS_DATA_LBA4 11 /* System LBA data recorded (spare) */
#define SYS_UNCORR_LBA 12 /* LBA to check for false C2 data (copy protection) */
#else /* gray retail disk */
#define SYS_DATA_LBA1 0 /* System LBA data recorded */
#define SYS_DATA_LBA2 1 /* System LBA data recorded (spare) */
#define SYS_DATA_LBA3 8 /* System LBA data recorded (spare) */
#define SYS_DATA_LBA4 9 /* System LBA data recorded (spare) */
#define SYS_UNCORR_LBA 12 /* LBA to check for false C2 data (copy protection) */
#endif


#define RD_BYTES_PER_DRV 0x03d78f40UL


#define MAX_P_LBA 0x10dbUL // 4315
#define MAX_L_LBA (MAX_P_LBA - SYSTEM_LBAS)


Found some nice bits of text in the dumps though...

"pagepalette 2...CREATED PALETTE POTS....Picture pages %d %d.....Created colour sprite...Setting current colour..colours are %d %d %d....displaying %d icons.....Creating bg.....Destroying PaintSelected sprite %d..>>>>>>>>>>>>>>>>>>>>>>>>>Created testurerightsleect 4...DESTROYED PALETTE POTS..Preserve next button....icon has sprite entry %s- FIX IT!...BALLS...pot allready has sprite index %d....pot allready has sprite index %d....pot allready has sprite index %d....setting rightclick..go away other player!...Removed erase frame by player %d....test 6..set all icon sprites again..Erased frame by player %d...test 7....---- MUSIC SELECT -----...WANTING TUNE %d %s..songs %d %d %d %d...Music Select 1..Music Select 2..Music Select 3..Music Select 4..Music Select oFF....Destroyed select page!..Loading page %d of font set %d..-- PAINT FONT --....Create new select set...flip or rotate..1 CHAR BUFFER...Theta at %d.....Selected a font Character...1 CHAR BUFFER...Selected page is %d.....current page is %d..Selected page is %d.....current page is %d..Requested change screen mode....------ Used undo to retrieve 320x240 picture ------.....Already in this screen mode. Exiting....Selica canvas size change

Turn power off and connect controller with 64GB pak and pocketcamera....GB Error returned from osGbpakInit()....GB Error returned from osGbpakReadId()..GB Error returned from osGbpakCheckConnector()..GB Error returned from osGbpakPower()...Connect the 64 GB pak and Pocket Camera to controller %d....Invalid accessory. Please connect GB pak with the pocket camera"


Everything is using the faster IO code now, so the time it's taking to read each sector / block is due to the way I'm handling the 64DD rather than the Arduino being a bottleneck.

From what I can see on the FPGA capture, it's reading a whole sector in around 188us, or roughly 16ms per block (sounds about right?).
I'm just obviously not reading the blocks in the proper continuous fashion that the N64 normally does, or that trying to read more than 64 Words at once is causing an error?


Do you think I should just release the code as-is, then we can sort through the last few issues?
It will of course have a disclaimer on it to say that it may not be 100% reliable, and there WILL be some bugs to work through.

I just found this, and remembered that Barc0de helped with a fair amount of 64DD stuff before, so add him to the list...
http://64dd.net/modules/newbb/viewtopic.php?viewmode=flat&type=&topic_id=284&forum=1


OzOnE

 

I wonder how I did help you though? I didn't do much except probably being interested on 64DD and all kinds of random stuff.

Anyway, nifty news. I'd still love to have an emulator.

 

@LuigiBlood - I'm sure I spoke to you for quite a while about the Satellaview on IRC many months ago?

I've also seen your posts on various sites about the more technical side of the 64DD, so even if you didn't talk to me directly, it's all good motivation.

I'll just post the code later tonight now (it's 6am here, and I've been awake all night on a Google Hangout session. lol)
It should basically work for ripping the important areas of the disks, but is still very slow.

It should be a fairly quick fix to get it to skip any bad blocks too, I've just been busy making the IO code faster instead.

I don't want to just post the code on it's own - I'll make a quick guide too with a few photos explaining how to hook up the Arduino, and which parts are needed.

I do intend to start work on the FPGA 64DD emulator again at some point, but I have a lot of other projects I'd rather make progress on first (N64 related ).

More news later. Need sleep now.

OzOnE

 

it's interesting to know how your current 64dd-emu works at all, maybe you can show common scheme - how/where it attaches to N64, signals of interface...
and if you show src code, maybe we can help you.

unfortunately, i have not 64dd, only n64 w/ couple carts. is it enough for such project, or i need have all stuff like 64dd, cassete and cart ?

 

Hi, cybdyn, how's it going?

OK, I've taken a few photos of the setup, and here comes the (first) release...


Firstly, the standard disclaimer: This code is still very much a Beta version!
I'm not an expert coder in any sense of the word, I'm self-taught in electronics / computers, so there WILL be some coding mistakes and ugly code.

Neither I, nor anyone on Assemblergames is responsible if you break your 64DD, disks, N64, PC, melt your brain, or burn your house down by the use of this info or code!


Might as well link to the photos instead of posting, as they are quite big.

I'm using the expansion connector de-soldered from an old N64 board for the connection to the Arduino Due.
This is a pain to desolder without the right tools / station, but it was easier for

Connection from the 12V PSU. It's rated at only 0.5A (500mA), but seems to be enough. Ideally it should be 800mA to 1 Amp.
(I did originally use heat-shrink on the +12V wire, but I broke the pin and had to swap it. lol)...
http://i59.tinypic.com/256elv8.jpg

N64 cart connector plugged on top...
http://i60.tinypic.com/ekpxk2.jpg

Connections to an old N64 cart PCB.
It has to have it's ROM chip removed.
The 64DD / Arduino doesn't use the CIC chip / EEPROM anyway, but might be a good idea to remove them just in case...
http://i58.tinypic.com/smpttf.jpg

Closer look at the "front" of the cart PCB.
It needs a thin wire soldered to the very top of the pad 8 on the PCB for the "Write_n" signal (thin enough so it still fits into the N64 cart slot connector...

The wire with the blue heat-shrink is just a straight connection to pad 20 on the PCB for the Cold_Reset_n signal...

The Red and Green wires are for 3V3 power from the Arduino to power the logic in the 64DD.
The regulator on the Ardy should be able to handle the power requirements fine.
Most cart PCBs have an electrolytic cap across the 3V3 rail, so it's easiest to solder the wires there...
http://i59.tinypic.com/1ze9x04.jpg

On the "back" of the cart PCB, another wire comes through the hole, and connects to the very top of pad 44 for the Int1_n signal (Interrupt).
There is also a pull-up resistor between the Int1_n pin and 3V3 (under the yellow heat-shrink on the front of the board.
The N64 has a 47K pull-up resistor internally, but I'm using 4K7, 'cos I'm lazy. lol...
http://i60.tinypic.com/ip77h5.jpg

Cart PCB plugged into the connectors / 64DD...
http://i61.tinypic.com/2wprspj.jpg

Close-up of the Arduino Due. The 3V3 wires MUST go to the 3V3 and Ground pins of course.
If you accidentally connect it to 5V or anything else, it will likely fry your 64DD!
http://i62.tinypic.com/16h3out.jpg

Overview of the whole setup. The ribbon cable is an old 34-pin floppy drive cable.
It's connected to the Arduino using two rows of SIL pin headers, and it's connected so it "skips" the two 5V pins at the top-right of the Arduino (above "digital" pins 22 and 23)...
http://i62.tinypic.com/1e8j13.jpg

VERY handy pinout map for Due...
http://www.robgray.com/temp/Due-pinout-WEB.png

(I'll post the pinout mapping between the Arduino and cart slot / cart ROM in the next post.)

And last but not least, here's the Arduino Due source code...

As far as I can tell, the dumped data looks good, but we ideally need to confirm that somehow.
It's also still not skipping bad sectors, and it's slow at ripping atm, but we should be able to sort through those issues now...

https://mega.co.nz/#!Kshz0SyR!XqQBW5zZo-5nsvc5jzg_sQpkNojVMfx-oAKmuCQI5rE

(if the link above doesn't work, try a manual copy 'n' paste into the URL bar.)

OzOnE.

 

OK, here are the pinouts for the N64 Cart / Expansion slot connector, and the Cart Mask ROM, and how the Arduino Due should connect to either...



The Arduino should NEVER be connected to or "through" an N64, only the 64DD!

The idea is to power the 64DD directly using 3V3 from the Arduino's own regulator.

The 12V supply then comes direct from a decent wall wart PSU to the 64DD to power the spindle motor / head sled.
The 12V PSU should be a proper switched-mode power supply so it's fully regulated (preferably 800mA to 1.5 Amps).

Remember to triple-check the polarity of your 12V PSU wires using a Multimeter before connecting to the 64DD...

+12V should connect to pin 13 or 38 (or both) of the cart / expansion slot, and Ground / "negative" should connect to pin 22,23,47, or 48.

Same triple-checks apply to the 3V3 and Ground wires from the Arduino to the 64DD.
Be sure to insulate all power connections with heatshrink or tape too (not like I've done it. lol)

My Arduino Due is powered via the "Programming" USB port, and that works OK.
If anyone has any weird issues with the drive, it may be worth powering the Arduino from a wall-wart too.

You could even use the N64 PSU to power the 64DD directly, but that would need some wires or pins hooked up to the correct pins etc.
(again, NEVER connect the Arduino nor 12V power to the N64 console's cart / expansion slot in any way!)

The main reason I re-used an old cart PCB was because it allows me to plug into the N64 with the 64DD connected, then use the FPGA to capture the real transfers between the two.
But for ripping, you ONLY connect the Arduino directly to the 64DD.

(I have an extra plug on the data cable, so I can hook up the FPGA board in parallel to the Arduino too.)


You could just connect the Arduino directly to the pins on the 64DD without using the Cart board + cart / expansion connectors (and those relevant pinouts),
but I haven't found a much easier way to do it tbh.

Important to also remember that the chips on the cart PCBs normally face backwards, so my Arduino is actually connected to the "underside" of the cart PCB.
The pinouts on the Excel sheet shown above for the Mask ROM are viewed from the top / "rear" side of the cart PCB too, so you have to be careful how you solder everything.

If you do use a cart PCB, be sure to check that the pinouts of the Mask ROM on your cart PCB connect to the correct cart slot pinouts too - some carts may differ!

Some of the signals like Write_n and Int1_n are obviously not normally connected to the Mask ROM chip on 99% of game carts, so they have to be soldered to the tops of the cart edge pads.

The Arduino Due does support internal pull-ups on most IO pins, but that didn't work reliably for me.
So, I simply added one externally between Int1_n and 3V3. The resistor value should be from around 4K7 to 47K.


It sounds a lot more complex than it really is tbh.
Just take it slowly, check all power wires for correct polarity / voltage, insulate everything etc.

The floppy drive cable should be kept quite short (say 15cm max from end-to-end).

I just chopped off the plugs at one end, and kept the standard 34-pin IDC plug on the other end.
To easily connect the plug end to the Arduino Due, I just used two 17-pin rows of SIL pin headers, like these...

http://www.ebay.co.uk/itm/40-Pin-St...plies_ET&var=550461408051&hash=item3a9206f39d

It's best if you can find the longer type though, so you know they're making proper contact.

I then just stripped about 5mm off the wire ends of the cable, and soldered them to the Cart PCB.

The "Arduino" pin numbers on the Excel sheet of course relate to the "digital" pin numbering on the Arduino itself, NOT the 34-pin floppy cable!
You should ideally connect the two "GND" pins on the Arduino IO socket to one of the Grounds on the Cart PCB too.

If you use an "IDE" cable for the connections, be sure to get the old type with 40 wires, NOT the 80-wire type.
The 80-wire "UDMA" type have some of the pins connected to Ground inside the connectors. That could be very bad (tm).


You will need....

*A 64DD plus disk(s).
*An Arduino Due board (please buy the official version, they're not very expensive).
*A PC / laptop with an OS of your choice (Windows XP/7/8 preferred - I won't include Vista. lol).
*Two micro-USB cables to connect the Arduino to the PC.
*Standard 34-pin floppy cable (doesn't need to be very long. Or, just make your own, or use an old 40-pin IDE cable instead).
*12V switched-mode fully regulated PSU, rated at 800mA minimum (I'm using only 500mA, and it works OK, but should really be 800mA minimum).
*Old N64 Cart PCB (with chips removed)
*Good soldering iron, solder, wire stripper / side-cutters, steady hands, patience.
*Tera Term and / or RealTerm, or a good terminal program of your choosing that can capture the raw data.
*Latest Arduino IDE software.


OK, I think that covers most of it for now.

If anyone has any rare-ish disks (especially Blue Dev disks) that need ripping once we've sorted out the code,
I might consider sending out some pre-soldered cables to hook up the Arduino to your 64DD.

Again, it's up to the owner if they choose to release the disk images publicly, but the idea is to get the data from as many disks as we can preserved before the dust bunnies eat them.

OzOnE.

 

Again, I just want to repeat that there is still some work to be done on the software, but it does seem to be mostly working atm (albeit VERY slowly - ~2 Hours for a full disk!).

If it finds an unreadable block on the disk, it will just stop reading altogether and spit out the last contents of the block buffer over and over.

I'm not sure if the bad blocks near the middle of my Mario Artist:aint Studio disk are intentional (copy protection), or if they are genuine?
We should be able to tweak the code to get it to skip any bad blocks and just read the next good blocks though.

I could probably have sorted that part out, but I felt like if I didn't release the code as-is now, it might never happen.


The way the code works is quite simple...

You just hook up both USB cables to the PC, then open the relevant COM port for the "Programming" port (baud rate set to 115200 baud)...

I use Tera Term for the programming port, as it's nice and simple to use, and easier on the eyes.
You can tell which port number is the correct one by only plugging in one cable at a time, then checking in Device Manager (or whichever new COM port then shows up in Tera Term's Setup menu.)

You should always connect the software to the programming COM port first, as it will then reset the Arduino and re-enumerates the Native USB port (disappears then reappears).
Tera Term should then show something like this...



Once connected to the programming port, I then open Realterm and choose the Native serial / COM port.
I use Realterm for that, as it's one of the few programs which captures the data reliably to a file.

The Native port is only used for the actual raw block data from the 64DD disk while doing a full disk dump, or for any individual block(s) you want to test.

To try a full disk dump, I just use the "Capture" tab on Realterm, choose a file name (should really end in .bin or .64dd or whatever, but doesn't matter), then hit the "Start Overwrite" button.

Then just go back to the Tera Term window, and hit 'D' to start the full disk dump.

(Make sure to not do any manual block "Read" commands in Tera Term before doing a full disk dump, or you will have rogue blocks at the start of your new image file!)


I'm sure somebody will come along and make a nice all-in-one GUI for the DD Ripper in time.
That will make things a lot more convenient.

You can have a play with the code on the Arduino without connecting it to a 64DD, but it will probably freeze or do weird stuff.


btw, if you do a block Read ('R') followed by showing part of the Buffer contents ('B'), it's kind of screwed up because of the crazy way the Arduino's serial.print() function omits the leading zeros on the Hex numbers!

This will be fixed in time too, and we should have a nicer way to view the sector / block data eventually.


There are other things that still need to be sorted - like the fact that I'm reading more than 64 Words at a time, and I think that's why the drive gives the BIT_AM_ERROR for every block?

You can see on the old CrazyNation site that the N64 normally reads from carts in 64-Word bursts. The 64DD probably only has a 64-Word buffer on the ASIC side, so the same should apply?....
http://web.archive.org/web/20111129093804/http://www.crazynation.org/N64/n64_cart_info.htm

In fact, I was seeing on the FPGA captures that the 64DD was triggering an extra interrupt after around 33-34 Words, so we'll have to look more closely into that.

btw, I haven't tried writing back to a disk at all yet, as it's not a priority atm, and will likely be a big can 'o' worms. lol

OK, that's it for now.

Please let me know if any of you intend to try this project, or have any other questions / suggestions / code fixes etc.

It's been a very long road to get to this stage.

It could probably have been done years ago (and we know it was done by others) by using the N64 itself, but I'm not experienced with the N64 coding stuff.

Hopefully nobody will bother trying to make any money out of ripping disks or whatever, as this will be for the whole community now.
It's not as if the 64DD had a huge library of titles anyway. lol


Regards,
OzOnE.

 

@OzOnE Wow you're awesome but now were "that" close to having 64DD emulation since its almost possible to dump roms

 

Thanks.

Yep, hopefully once we get more disks ripped it will lead to proper HW / SW emulation.

I'm a tiny bit worried that I haven't found the "64dd-multi" text in the image file yet, but almost everything else seems in order (apart from the slow speed, and bad block handling).

I don't think the Arduino will be quite fast enough to handle the drive emulation directly.
Probably best to go back to the FPGA design, since we can stream off of SD card, and the low-level code is already in place.

It would probably be easier just to finish the DD emulation under MESS or something?

I read on 64dd.net or somewhere that Project 64 might have basic support for 64DD images, but I'm a tad doubtful about that.

There's probably plenty enough info in the Arduino source for people to start work on a hardware emu too.
It can even ignore all the MSEQ data stuff, it just needs to transfer the requested sectors / blocks from SD card or USB.

OzOnE.

 

Excellent work! Hopefully I'll be able to make one someday. You'll definately want to get in touch with willis82 about getting him a cable.

 

I'll have to look into getting a few cables made for people.

The main problem is having to sacrifice an N64 for the expansion connector, as it's quite a weird spacing and I'm not sure what else would fit the drive.
It's kind of like the middle part of a very large Centronics connector. Hmm, maybe there's a PCB edge connector that would fit the wide spacing too?


To explain what I mean about "grabbing more than 64 Words of data at once"...



You can see that the 64DD triggers an extra interrupt (dotted line in the centre) after I've read 64 Words (128 Bytes) of sector data with the Arduino.
I need to check whether it's giving an error at that point, or if the "Data Request" bit is still set.

Then, use a global variable to store how many Words have been transferred, so we can continue reading the data only when the drive is ready.

I'm not sure what this means for the data I've captured so far, but I think it's certainly something we can sort out.

We still need to try to "characterize" the 64DD and it's quirks. I'll get and do some more captures with the real N64 later today.

I also found last night that the Arduino got stuck while ripping, so I probably made some of the read / write routines a bit too fast.
I just uncommented the delaymicroseconds() stuff, and it seems to be working again.

I might need to buy myself another disk to test soon - they should be fairly hardy, but I've read from this Paint Studio disk quite a few times now. lol

Oh btw - does anyone know where I could upload the Paint Studio image I have so far, without breaking the forum rules?

I've found that each Block of data has a handy marker at the beginning. eg...

0079 for the first block.
0179 for the second block, and so on...

Strangely, when I was using PuTTY to capture the data, the first Word of each block was byteswapped? ie. 7900, 7901, 7902 etc.
This is probably a quirk of either PuTTY itself, or the serialUSB.print() function on the Arduino (my vote is on PuTTY, since the rest of the data seems normal).
I'll check that against what the N64 is reading too, just to make sure it's correct.

From reading Kammedo's info again last night, it does look like they just add 24 from the requested LBA number, so the game doesn't normally "see" the system blocks directly.

That's probably also why the LBA number in DDDump util doesn't match up with the expected data (it assumes the image file only starts at LBA 24.)

I've also just noticed that I'm not taking into account the "alternate" tracks - these are 4 extra tracks at the end of each zone that are likely used for when a normal track has read errors at the factory (or when a RAM track gets corrupted, so the software can use an alternate track).

We might need to subtract those extra tracks from the final image file, or just skip them when doing emulation.

It should also be possible to rip the entire disk including the C2 error correction sectors, and maybe the C1 bytes as well.

I'm still working out various things, and looking through the image data.
More updates soon.

OzOnE

 

Right - first silly bug in the code...

Around line 538, this line...

SerialUSB.write( (uint8_t *)buffer, BlockSize/2 ); // Fast write a whole block via USB.

Should be changed to this...

SerialUSB.write( (uint8_t *)buffer, BlockSize ); // Fast write a whole block via USB.


I was dividing the BlockSize by 2 because I switched from sending one Word at a time to sending the whole buffer.
I forgot that the SerialUSB.print function needs the size value in Bytes instead of Words. lol

(I didn't notice on Realterm because it shows it's "Char Count" as doubled anyway. I'll just check the size of a single block of data in future.)


I'm assuming that casting the value as a unit8_t should convert the short values in the buffer to chars properly?


I'm just transferring the whole disk again, so I can work with a full copy.

It still gets stuck on some bad sectors between LBA 2442 and LBA 2479.
So, I just cancel the transfer at that point then edit the code to start from LBA 2480 instead (it then dumped the rest of the disk, right to the end).

I don't think there's much useful data after that point anyway (on this disk at least), but I can then stitch the two files together to give a full image.
For the missing blocks, I can just repeat the previous blocks until we sort out the code to do that for us.

I did some quick captures from the N64 as well, and it looks like the game does check for some intentional bad sectors / blocks.
It should eventually be possible to dump all of the C1 and C2 data anyway, so we can emulate everything.

OzOnE

 

Having some weird issues now with data corruption once I try transferring while blocks at once (using the above code fix)?

(I updated the link in the first post #52 with that fix as well).

Also, it randomly freezes after about 150-400 blocks, and it's definitely to do with the SerialUSB function, or the PC / driver / buffer / Realterm?

When it does freeze (stops sending data) and I close Realterm, the drive tries to read another block.
So, I know it's something to do with the USB transfer.

I've been reading about similar issues with SerialUSB and how you ideally need to transfer in multiples of 64KB, but not sure how to fix the problem.
It seemed to work OK when sending half the data, but it's still only 19.7KB per block maximum, which isn't a huge amount.

Maybe we could send the buffer back in two chunks, 'cos I think the Arduino itself is doing some buffering which is slowing it down / overflowing?

I'm assuming that when I declare the sector / block buffer array in the code, it's using 20,000 shorts (Words), and not chars (Bytes)?...

unsigned short buffer[20000];

I know this is 40KB, and is quite a lot for the poor Arduino's 96KB SRAM, so that may be a problem too?
Using the Arduino libraries probably isn't helping either - I'm sure the native USB is capable of much faster speeds.


Oh, actually, it still does it on mine even when changing back to the original code, so something has gone a bit wrong.
It might be my 12V PSU starting to die, or a ground loop, or the 3V3 reg on the Arduino not having quite enough current to run the 64DD logic?

I'm going to switch to using the N64 PSU, and just desolder the plug connector from the old broken N64 mobo.
That will mean the N64 PSU powering the 64DD directly, and the Arduino only used for controlling it.

OzOnE