New SMAP driver + HDLDump v0.93 prototype

EDIT 2013/02/19: Fixed bugs and cleaned up code.
EDIT 2012/12/24: Fixed bugs and split components apart (For use in other homebrew projects).
EDIT 2012/12/08: Added network manager and isolated the SMAP driver into it's own module.
EDIT 2012/12/04: I've fixed some problems in the EE RPC client in the source code.
EDIT 2012/11/30: Adjusted thread priorities to give transmission priority.
EDIT 2012/11/29: Made another attempt to fix the stability issue.
EDIT 2012/11/25: I've fixed the problem with frame transmissions causing a deadlock when the TX FIFO runs out of space. Please test and report problems.

I've also adjusted the settings for the LWIP EE-side stack, so performance is a bit better.

Unfortunately, the changes I made to improve the stability of the SMAP driver itself has caused performance to dip a little (~0.2MB/s).

Again, many thanks to Silverbull for suggesting that disabling the device interrupts is a valid method of ensuring mutual exclusion.

I think that there is little one can do to improve the performance of the EE-side stacks. :/
The IOP is really just too slow, and the DMA channels are too simple in design to work around the limitations of the LWIP protocol stack (Those alignment problems).

I've also included my most recently updated pseudo code file of the Sony SMAP v2.25 module for educational purposes.

----

As mentioned before (For quite a long time actually), I was working on a new SMAP driver that was based on the last version of the Sony SMAP driver (v2.25).

I've completed it, more or less... and so here it is.

The driver is coupled with some form of a HDLoader game server for testing purposes (LWIP stacks use the OPL HDLoader server, while PKTDRV stacks will use their own servers).

There are three builds in the package, but only one is useful for installing games.

The included README file contains more information about this project and the specifics of it, so please spend some time reading it.

The testing platform that uses LWIP v1.4.1 requires the EE library instead of the usual IOP port, so please compile and install the port of LWIP v1.4.1 for the EE.
If it's stable enough too, I'll make a binary release in the future.

As mentioned in the README file, the focus of this new driver would be (Compared to the current SMAP PS2SDK SMAP driver):

1. Fully working and functional auto-negotiation.
2. Superior network peripheral compatibility.
3. Superior performance, with the introduction of full DMA support.
4. Superior reliability.

Please provide stability, compatibility and performance reports with your network equipment and Playstation 2 consoles.
Please mention the model of your Playstation 2 network model, the model of your Playstation 2, how you are connecting your Playstation 2 to your computer, and the model numbers of all network peripherals (e.g. switches, routers and hubs) that your console will go through to communicate with your PC.

As with a lot of my projects: USE THE SOFTWARE AT YOUR OWN RISK. Since the software writes data to your hard drives, there is always a risk of data loss in the event of data corruption.

Downloads/Links
Project home page: http://ichiba.geocities.jp/ysai187/PS2/smap.htm

***
For those interested to know, here are the NICs/network equipment that gave auto-negotiation issues with the current homebrew SMAP driver, when used with my SCPH-10190 PCMCIA network adaptor:

1. Realtek 8101E Fast Ethernet Controller (Lenovo Ideapad Z460 laptop)
2. D-Link DIR-655 (4-port Gigabit network router)
3. Linksys EZXS55W (10/100 5-port switch)

 

Hi, sp193!

So, finally you made it.
This is the the maximum optimized driver you talked about for a while, right?


Rgds.

 

Yes, indeed.

Currently, there is a design fault with it: it's transmission capability is really poor now as there is a design fault. If Tx FIFO space runs out, it may freeze up if the response being sent was sent from the interrupt handling thread (Which handles incoming packets too).

The thread will be unable to mark freed space in the Tx FIFO as available, and the driver will enter a deadlocked state.

 

New version released!

I've fixed the problem with frame transmissions causing a deadlock when the TX FIFO runs out of space. Please test and report problems.

I've also adjusted the settings for the LWIP EE-side stack, so performance is a bit better.

Unfortunately, the changes I made to improve the stability of the SMAP driver itself has caused performance to dip a little (~0.2MB/s).

Again, many thanks to Silverbull for suggesting that disabling the device interrupts is a valid method of ensuring mutual exclusion.

I think that there is little one can do to improve the performance of the EE-side stacks. :/
The IOP is really just too slow, and the DMA channels are too simple in design to work around the limitations of the LWIP protocol stack (Those alignment problems).

I've also included my most recently updated pseudo code file of the Sony SMAP v2.25 module for educational purposes.

 

ok just to double check this is a early but stable build of your network game installer right?
or is this just an improved method of getting an already made .iso onto the hard drive via network

 

Hi!

Could you please tell me where find the fixed version?



Thanks a lot!

 

New version released!

Note: Performance doesn't dip as it does with the same fix in the previous version.

Changelog:

1. Cleaned up thread creation code in the SMAP driver.
2. Converted the values passed to DelayThread() from hexadecimal numbers into decimal numbers for clearer representations.
3. Removed duplicated declarations of I/O functions from main.h and xfer.h.
4. Modified the EE-side to have all the IRX modules embedded within the main ELF.
5. Modified the IOP-only testing platform to use a separate thread to service TXEND interrupts.

As usual, the files can be found in the first post of the thread.

 

Hi and thanks for your reply!

Sorry, but it's for me the first time different mediafire files got the same link.
Strange...


Rgds.

 

Just clearing up the project - closure.

New version: Adjusted thread priorities to give transmission priority. The stack sizes have been halved as well, to save IOP memory.

I've decided that I won't try to port the driver back to support existing homebrew software like OPL and SMS. I tried to, but ran into the old stability problems that I had last year... so I won't try doing that again.

Everyone else is welcome to try though. I've uploaded my work files

I've redirected downloads to a page on my web site to make things easier for me to maintain.

 

Hi!

As always thank you for your work.

So as i understand your words there'll be no backwards compatibility as stated above for i.e OPL.
Wouldn't it be nice if you support the OPL forum with your great work and have a big among of testers at he same time?
:friendly_wink:


Rgds.

 

Yes, it'll be great... but I won't take up that task. I have spent enough time on this project, and I want to do something else... instead of wasting more time battling with something difficult to fix, especially without a working debugger.

It's not that there is no "backward compatibility". There isn't "backward compatibility" as this driver is a totally different one, designed to run with different network protocol stacks.

I'm saying that while it can be ported to work with the IOP-side LWIP protocol stacks (Which I've completed and uploaded), it's not stable and I won't continue attempting to fix those problems that I can't fix.

Plus by fixing that for OPL, I won't gain anything as I don't and can't use the USB device and SMB support of OPL.

 

I like that stance of staying away from the IOP. That makes more sense than trying to fix broken and poorly designed homebrewn software. Make something new which is better than past stuff !

 

Agreed!

EDIT: And it's also because the IOP is too slow to allow TCP connections to operate at full speed, if the network protocol stack was to be implemented on the IOP side.

Well, I'm quite at a lost here at what to do next. To make network-enabled homebrew software, we need a fully working network protocol stack that supports at least TCP (UDP shouldn't be used for data that needs integrity). The problem is... my EE-side port of LWIP doesn't work right.

Other than the stability issue, there is one more issue that has been bugging me for a really long time: I can't port the timeout mechanism from the old IOP port of LWIP over to the EE. D:

Somehow, using iReleaseWaitThread() seems to cause something to freeze up. I can't tell whether it's a problem with the sys_arch of my LWIP port, a bug in the PS2SDK, or a bug in the PS2's kernel.

Alright, so maybe it isn't really an issue here since I am using an alternative method of supporting timeouts, but I don't like to leave things unanswered as it could be a sign of further stability issues.

Help with getting a nice protocol stack that fits the PS2's (DMA controller) architecture will be great, from anyone.

 

I'm just here to say thank you for your effort and time you invested in this project. Sounds really interesting.

 

Making the "half assed" PS2 homebrewn SDK "full assed" is a big project. Someone is working diligently to improve the toolchain, including the compiler/linker to give programmers more freedom. I won't mention names because it could generate pressure and discomfort for the said person. If he feels it's ok to chime in it's up to him.

Don't understand me wrong, the homebrewn PS2 SDK is truly a feat. Making a huge blob of proprietary code into something free was really a awesome job. But there's a lot of underlying quirks on the homebrewn SDK which are specific to it and some even make it difficult to achieve certain objectives...

 

Excellent work, thank you for sharing! I didn't have the time to try it out yet, but I'll make sure to catch up (just returned from a business trip, and still kind of sleepy).

Weren't some system routines being "enhanced" by libkernl from the official SDK? If I remember correctly, with certain SDK versions, you simply cannot invoke some of the kernel routines directly; you always invoke routines from the library, which then perform some stuff prior to calling the real ones.

ExecPS2 and LoadExecPS2 are examples for this. I think the kernel does not reset the TLB to its default state when these are invoked. So if your program fiddles with the TLB, and you invoke the kernel routines afterward, you can actually hand over the EE in a state that makes the next program behave erroneously :stupid:. To fix that, the versions in the official SDK contain TLB setup code that is automatically invoked before the kernel implementation of those functions.

There is also some stuff related to alarms and thread scheduling/wakeup. The library creates a special thread that seems to be dedicated to this, but I never bothered to check what it really does, or why. You may want to throw libkernl.a into your disassembler of choice and have a look at its initialization code.

For exactly the reasons you mention I may regret writing this, but anyway...

Binutil updates:

• Port of EE/IOP binutils to version 2.22. No DVP support yet, sorry (but you should be able to continue using the old dvp-as).
• Re-added -membedded-pic to ee-as
• Added support for Sony-style ".ilb" files in iop-ld. Your IOP modules no longer need those "imports.lst" or related magic; just #include the header, add the right .ilb to the list of libraries, and the linker generates the necessary stubs automatically.
• Fixed opcode definitions for some VU0 macro mode instructions (VRSQRT anyone?)

GCC updates:

• New assembler constraints for VU0 vector and integer registers. For example, this allows to add pseudo-intrinsics for macro mode instructions, and the compiler then makes sure to hand in data in the right registers. And I really love that inline VFTOI4... :loyal:
• Fixed (at least some of the) 128 bit arithmetic on EE. If you don't know what that means: try to do some non-trivial stuff, like shifting a non-constant value into a u128, and watch the generated code of the old ee-gcc :dejection:.

Known bugs:

• Some customized linker scripts are just plain incompatible. These worked for the old (2.14-based) ld, but produce bad executables with the 2.22-based ones, or none at all.
• Generation of debug symbols. I spent almost three days fixing those stupid STABS-in-ECOFF encapsulation in ee-as and ee-ld, and somehow got to a "stable" version... Which means the linker does no longer crash for my test runs, and the symbols don't look that wrong.
• Line number information is randomly broken. I did not figure out why it happens, but for some of my larger programs, it seems to somehow mix STABS and others, making the Sony debugger not recognize STABS line numbers. So no source-level debugging.

Apart from that, the modified toolchain generally seems to work. I moved over completely, and even recompiled the homebrew PS2SDK, and Kermit+ODEM still work.

Indeed, the homebrew SDK is really nice. But one still needs to know where the tripwires are, although that can be said about almost all non-trivial tools... :topsy_turvy:

One of the most useful (and, at the same time, irritating) "feature" is that the homebrewn RPC client libraries do not perform version checks. Most of the official SDK libraries make sure that the correct counterpart runs on the IOP, and simply abort with a "version mismatch" error if they find discrepancies. The homebrew libraries just perform the call... if you are lucky (or made sure the correct IOPRP is loaded), it simply works. Otherwise, you will probably get the most bizarre IOP crashes you have ever seen. Debugging those on the TOOL was already hard enough, and I prefer not to think about on how to do the same on a retail machine.

There are also no asynchronous RPC calls, as the "mode" argument to sceSifCallRpc is simply ignored. This is not a problem for regular use, but if will hurt if you try stunts like "reset IOP from IOPRP in EE RAM".

 

More updates

I've fixed some problems in the EE RPC client in the source code.

I'm uploading the source code without updated binaries as the only people who will probably be interested in this are other developers.

I'm currently working on a network manager so that the same SMAP driver can be loaded and run with a network protocol stack that runs on either the EE or IOP sides, without needing to be recompiled.

Hopefully, not too much performance will be lost.

For regular users, nothing has been changed. I just wanted to upload my work so that other developers can play with working code.

The next version will take a while to be completed, depending on how successful I end up being, my available free time and my level of motivation...

---

Oh? Well, take your time.

My work is probably still in the development stage - no libraries are built yet, although I've started structuring the SMAP driver and support RPC libraries into a form suitable for building into libraries.

Yes, that is true.

But I'm not using the Sony PS2SDK, but the homebrew PS2SDK instead (Which seems to have been designed to work with the kernel and libraries in the boot ROM). :S

(Surely the odds of the kernel crashing because of a change in design semantics is not great here...?)

I think that possible that ReleaseWaitThread() on the EE kernel doesn't function like ReleaseWaitThread() on the IOP does... for the same reasons you mentioned, or because it wasn't designed to (The Sony documentation doesn't explain what it actually does in the EE kernel).

I seriously welcome these changes.

Especially on how we can link to functions of other IOP modules like the Sony developers do. The bugfix for the EE's TI mode is very important too!

I've been trying to install your stuff, and I can say that it works if your locale is set to use codepage 437 (And not something else like Japanese/Codepage 932). GCC seems to screw up during stage 2 when my Windows installation's locale was set to Japanese (I don't remember which file it was, but somehow it ends up trying to build some non-existent file that has a malformed filename).

EDIT: He's not trying to make commits for GCC, but Binutils: http://psx-scene.com/forums/f167/how-you-guys-doing-108468/#post1021479

I think that Megaman from PSX-scene has been trying to get the GCC steering committee to accept his R5900 patches to GCC, but he has run into some weird licensing issues.

But I didn't really read what his recent posts were about, so he might have been attempting something else altogether. :X

It's up to you in the end, but I feel that the both of you should collaborate together to get the entire toolchain updated and brought up to speed. Compiling GCC 3.2.2 in some environments is getting difficult... (Recent versions of MinGW can't build GCC 3.2.2 properly)

How about SifRpcGetOtherData()? :highly_amused:
It's broken internally, because the semaphore is initialized incorrectly (Initial count = 1 instead of 0).

Agreed! :S

Yup.
This needs to be fixed...

I need to go now. I'm tired again, and my brain doesn't want me to stay up any longer... :subdued:

 

More updates

The network manager is almost completed.

I wasted half of today and a few hours last night struggling with my code again, thinking that it's a problem with my code (Strangely, the RPC client on the IOP won't connect to the server on the EE).

Apparently, I've hit yet another bug in the PS2SDK, which is probably another one of those phantom bugs I found when working on PS2ESDL in year 2009. :mad-new:

ee/kernel/sifrpc.c::SifSetRpcQueue

I should learn to trust myself more, and to not assume that the PS2SDK is bug-free (Which apparently, is actually still loaded with plenty of bugs, especially in the EE libraries).

So yay, one more mystery solved.

Thankfully, it was not a runtime error that occurs while the SMAP setup is running actively, or that would have been impossible for me to use PCSX2 for debugging. D:

For those interested, it's this line in SifRpcGetOtherData() that is breaking SifRpcGetOtherData():

It should be 0 instead of 1.

 

Itching to see if that SIFRPC fix affects Open PS2 Loader ...

 

New version released!

Changelog:

1. Added NETMAN - The Playstation 2 network manager.
2. Modified all stacks and the SMAP driver to use NETMAN as a common interface.
3. Killed chain/bulk DMA transfers from the SMAP. There goes the bonus ~0.3MB/s boost.
4. The SMAP driver has now been consolidated into only one version, as the network protocol stacks have been decoupled from the SMAP driver.
5. The PKTDRV network stack that was previously integrated into the SMAP driver has been split off into a module (pktdrv.irx).

Refer to the first post for downloads/links.

---

Depending on what you are hoping to see, you may be disappointed.

This fix only fixes crashes that occur if more than one SIF RPC server is registered on the EE side (Which I don't think that OPL does at all).

Since probably nearly no homebrew software use these buggy SIF RPC library functions, this bug was never reported earlier.