(CBT) HDLoader game installer

It's worse. I never had IGR problems with games like Bully, MGS2 and MGS3, Black with OPL but now I do using the GameInstaller. What happens now is I just get a black screen. Sometimes it works, sometimes it doesn't. Truly random.

 

That is strange, but it probably means that there are still problems with the way IGR works and they are getting exposed after changes are made to how OPL starts up.

But just to be sure that I'm understanding your observations properly:
1. The normal version of OPL is able to invoke IGR properly for your games.
2. When the game is installed by HDLGameInstaller and you boot it directly from the HDDOSD, the embedded copy of OPL has problems with IGR.
3. When the normal version of OPL boots the game installed in (2), IGR works properly.

 

That's exactly it.

 

HDLGameInstaller v0.806 OBT released!

Changelog for v0.806:

1. Updated ATAD module (PS2SDK update).
2. Fixed installation of games that are 4GB or larger in size, on disks that are larger than 250GB.
3. Added a partition slice limit to HDLFS, as I forgot that the HDLoader game format uses a 32-bit value to record the size of each slice (4GB partition parts are the largest it can support).
4. Removed old, broken code from HDLFS which automatically adjusted the size of the game as data is written. To begin with, it doesn't do anything because the size of the game is recorded during formatting and can't be adjusted afterwards. It was originally there to make its write function behave like how the the standard write function would behave when data is written to beyond the end of the file.
5. Fixed the installation of the 2nd layer (of DVD9 games) from HDLGManClient. Please reinstall your DVD9 games that were installed via HDLGManClient!
6. Tidied up the server functions.
7. Fixed the logic of the connection state checking functions of HDLGManClient (Should now report client-server version mismatches properly).
8. Modified HDLGManClient to report connection losses instead of stating that the operation failed (Makes things less confusing for the user, as some connection-related problems resulted in error messages that suggested a hardware problem).

Therefore, please reinstall your affected game if your game meets at least one of the following criteria:

1. Your console's HDD is larger than 250GB, and the game is larger than or equal to 4GB in size.
2. Your game is a dual-layer game (DVD9).

Downloads/links and for more information:
HDLGameInstaller project homepage: http://ichiba.geocities.jp/ysai187/PS2/HDLGameInstaller.htm

***

Hmm... then this should be considered a problem with OPL instead, since we're now doing a 1:1 copy of the EE core and IOP stuff. As long as it functions normally with the normal version of OPL, it means that the installed game is fine.

Thanks for reporting this issue.

 

Slightly offtopic - I've noticed problems with some of the newer builds of OPL (both plain OPL as well as OPL+GSM) with IGR, in games where IGR worked previously. I'd have to do some testing to record the behavior with specific builds, but using OPL rev650 and newer (I think, I'd have to double-check with the builds I have on my PS2's HDD) as well as OPL rev655+GSM 0.38, I have gotten BSOD when using IGR on some games where I had no problems. Also, with the aforementioned build of OPL+GSM, I have had issues with GSM activating properly, instead of switching to the desired vmode I would get a BSOD, when trying either 480p or 1080i (the two primary modes I use). Once reverting back to OPL rev644 and either the same revision with GSM 0.37 or OPL rev646+GSM 0.37 (I'd have to check my MC to see which one I am currently using), the problems go away.

This may be relevant to jsnepo's problem, depending on which OPL (or OPL+GSM) build they're using as a standalone ELF, versus the specific build of mini-OPL used with the installer.

Also a quick question about the installer itself - I'm very interested in a long-term goal of moving away from my current FMCB-based setup to an HDDOSD-based one, and I'm just curious if there are any future plans to provide the option of VMC with installed games when launched from HDDOSD? I rely on them for all of my installed games when using standalone OPL, and am very happy with the stability (I no longer boot with my MC in slot 2, for example, as I have had no issues with corruption in a very long time, and should I encounter some corruption I have the ability to reinstall FMCB even without a working install, thanks to my Swap Magic disc). I understand that VMCs may be implemented in a somewhat "hacky" manner, if I've read what you've written on the topic correctly, but I find the VMC capability to be invaluable to the increased enjoyment I get from my current discless gaming setup, and especially since I currently only have one MC (my 16MB Max Memory card with FMCB installed is on loan to a friend), with the homebrew I have on my card I have little space free and so would not be able to revert back to physical MCs with mini-OPL. I would be happy with a "generic" naming system that would simply use the ELF name as a base, as standard OPL does by default when creating a new VMC. In the situations where games can look for other games' saves to unlock certain things, I would have no problems with manually copying such saves onto the VMC using uLE so that the games can read it. Either that, or when installing the game, the installer could ask for input as to which named VMC the game should use, or even if VMCs should be supported at all, so that games that do exhibit issues with VMCs could be booted without VMC support automatically, as one can do with standalone OPL. Out of curiousity, would you be willing to go further in depth to explain why you (and by you, I of course mean you and anyone else working on the codebase) chose not to support VMCs, other than because it's "hacky"? If not, no big deal, just curious. At least this isn't the Wii scene, where the similar functionality with Gamecube games (something they call NMM or "No More Memory (Card)") is extremely hacky and not that compatible with games, where with OPL, compatibility is more often than not.

 

Ah, alright. In that case, a discussion on this matter should take place because that's the only way of finding out.

If you or somebody else can come up with a nice UI for configuring the VMC settings (Toggling between enabled/disabled states for each game, as well as managing the savedata itself), and the VMC saves can be saved properly into the __common partition like Sony says games should do... then go ahead.

I won't be adding such a feature to the installer because I've already gotten burned by this project and the other projects I worked on. While it seemed like I only worked on those projects, I had to struggle with the weird requirements of the PS2 (Unlike the PC, it has some requirements for memory alignment etc), and the bugs and missing features in the PS2SDK... (mainly the network driver, which was a pain because debugging was entirely done via trial-and-error) hence why the huge number of commits from me recently, since I got write permission in March. :/

I think that the VMC project in HDProject originally just replaced MCMAN, which is what I would prefer doing.

However, OPL seems to be hooking entirely onto SIO2MAN instead... probably just so that it'll have a more generic system that works with the two memory card systems which Sony devised (MCMAN, and the newer module which had the filesystem support part of it moved onto the EE). This means that instead of just redirecting where the files are written to with MCMAN, the entire card is emulated instead. (Of course, when dealing with that newer MC device module, you'll have to do this anyway).

The saves in the __common partition are like the savedata in your memory card (contains icons + the save itself), and are not images of the memory card. If the savedata is stored as a memory card image, then you have to know where the save is stored (as in, the folder it is in, because you'll be loading that image for the VMC system and not the game) and you have to create the icon for that savedata folder on your own (The game's icon files will not be visible to the OSD, from inside the memory card image).

With the full version of OPL, that isn't a problem because the saves are not stored in the __common partition for the OSD to use (It doesn't have to care about storing them in the "right" way for the OSD). OPL also seems to record the VMC settings for each game somewhere... which this GUI-less version of OPL doesn't support because it doesn't have a UI (Hence it can't be used to manage those saves), and it doesn't have the code for loading those parameters anyway.

Finding the location of the save probably isn't an issue if the game's ID is used. However, creating and managing the saves is the real challenge.

As for the files that the OSD needs:
1. An icon file. Like I mentioned above, the game's icon can't be used directly because it's within the MC image. If a generic icon is used, all saves will look the same.
2. An icon parameter file (icon.sys). This is going to be the main challenge here as you won't know what to enter for the save data title (maximum 32 characters, in SJIS encoding... not ASCII). The HDLoader game title cannot be used directly because it can be up to 160 characters long (and in UTF-8 now too). Unless you can come up with a system which can intelligently decide on how to shorten long titles in the right way (and to deal with characters that can't be represented in SJIS), the user may have to be troubled to manually rename each save he/she has (and to do that, you need to provide a UI for that).

This project is open-source. Anyone who manages to pull that off (As in, coming up with a fully functional system that has those VMC management problems taken care off) will have my full support. :encouragement:

Alternatively, the saves could be dumped into a folder that either isn't visible by the OSD (Like what OPL does now) or a generic saves folder under __common. But we don't like that idea because the original plan was to come up with a system that still fits the Sony-designed system as much as possible. A UI for managing the saves is still required though.

 

Sounds like adding some features that original game cannot support from scrach. For example miniopl just EMULATE virtual dvd-image, but not use the sony specs as it is done, for example ins METAL SAGA

If we really was trying to fit sony specs - we have to create PFS partition than we have to copy all data from disk into that folder and the most dificult thing we have to redirect all files from DVD into the HDD. for now it is not impossible - adding atad, pfs, hdd support into commercial games isnt so easy and universal for any game We just use dvd image emulation.

The same can be said about Memory cards. Much more easier (and universal) is adding its emulated prototypes neither learn each game how to read from HDD. So maybe really add such a feature into miniopl (for example as unused Mode7)?

But anyway if someone is interested how looks like game that is installed fully "officially by Sony" - just download Metal Saga, extract atad.irx, patch it with atad patcher 0.0.3 (or just replace with homebrew ps2atad.irx if you are using more than 137Gb disks) and place it back into image. Next run this image but not from hdd (usb or network opl, or directly from burned disk). And install it via "Install on HDD".

If someone wanna see how looks like save data - you can launch HDD OSD (Browser 2.00), open memory card, select any save and push "Copy" and select "Copy to HDD". Then open ulaunchelf and see what is done.

BTW Metal Saga copies all data from disc except of main elf and 2 irx files

Please check requirements at TRC_HDD_Supplement-1.2-English.pdf

 

If one replaced MCMAN instead of hooking onto SIO2MAN, it becomes possible to redirect I/O to the __common partition instead. Of course, you'll have to provide the APA and PFS filesystem drivers too... but it allows the game to obtain nearly native support for saving onto the HDD.

I don't remember how far HDProject got with that system though.

This won't be doable for newer games that don't use MCMAN though, so a memory card image will have to be used to store the data instead for those games.

Thank you for adding more information on this topic!

 

I would at least be interested in seeing an MCMAN replacement, which would be better than nothing. MC emulation would suffice for games that would require it (unless someone figured out a way to redirect mc2_s1.irx and mc2_d.irx to __common). I can tolerate using uLE to manage files on VMC images, as I'm used to doing that already, but it would be handy to have OSD-manageable saves for games which would work with an MCMAN replacement. I don't have the skills to make that happen but I would put myself forth as a tester if someone else did.

Would it be possible for the installer to detect which module is present on the game disc/image and install either a version of Diskload with MCMAN replacement, or alternately either a non-VMC Diskload, or one with MC emulation? I do know that the loader is replicated inside of each HDL partition, so it could also be different for each game as well.

 

Not to put a crimp on this but why the need for VMC you can copy to HD from MC with the HDDOSD and copy back only a limited few game may really need this I know truly MC free boot and play is the only reason I see

 

Basically, yeah, that's it. I'd much rather not have to wear out the write cycles on my PMC by copying saves back and forth. I do agree with SP193, however, than an MCMAN-style replacement that dumps the files into __common instead of to an MC image would be more elegant (and possibly faster than current VMC support for compatible games?). Plus, I take my PS2 various places and would be very happy if I didn't have to worry about taking my MC with me, one less things to lose.

Edit: SP193, just noticed something minor with the most recent installer (the 130804 release). When connected to the client, the server labels it as "Remote administation mode" instead of "Remote administration mode". Just figured you might want to add that r in there before your next release

 

For those who wants to keep their games bootable from HDDOSD, and still want to use VMC, you can do that (I tried it just yet), as an alternative solution :
take hdl_dumx from AKUHAK, use "normal" OPL (OPL as an ELF file), renammed as boot.elf and placed in hdl_dumx folder, then install your game. Compatibility modes & VMCs remains available, and can be changed via OPL GUI.

Little inconvenient for this solution is that when booting your game, you will see all your games installed on your HDD. A mutilated OPL (from all useless stuff for HDD : USB support, network one....) and restricted only to the partition where it is installed seems to be an easy solution - from my normal user point of view, Im not a dev [please be nice so]

 

That seems a little pointless to me since you see the full game list, no need to install full OPL in the game partition. Better idea for that is to make a partition and use hdl_dumx modify_header to put OPL on it's own partition and make that partition HDDOSD bootable (like I have done on my experimental HDD with OPL+GSM). Still need the +OPL partition with public binaries of OPL, although I'm thinking there's not any reason those with the ability to compile OPL couldn't change that in the source to point to their application partition instead and use it for VMC/CFG/ART storage.

Still, I like the elegance of actually booting games directly from the HDDOSD. For now, if I want to do that, I simply use my PMC to store saves and move it off later, as RandQalan suggested. GSM works with the HDDOSD too, but seemingly only in 480p (I've never gotten the system to boot up, load GSM, switch to 1080i, then actually load the HDDOSD, it still ends up being browser 1.20 with no HDD access). Don't really care about 720p support, don't know about VGA support as I don't have a SoG VGA cable.

 

Yes, that could be another option - tho I don't like too much wasting at least 128 Mb only to install a single ELF. Also, in your solution, your game partition is not bootable, only OPL+GSM one.

OPL being what it is now, no solution is perfect, I'm aware of it.

 

Correct, although of course you can have both - Diskload in the game partition and OPL+GSM on it's own. I'm going to try out changing the partition name in a custom build to see if one can store everything there instead of on +OPL, to use that "useless space" might be useful.

 

That's the idea! But unfortunately, implementing that is another story. :/

I mean, I can't be rewritting the entire PlayStation 2 world, can I? I'll have to stop somewhere.

Ah, noted. Thanks for reporting that!

 

HDLGameInstaller v0.808 OBT released!

Changelog for v0.808

1. Updated to work properly with the modern PS2SDK.
2. Ported the fixes to the PS2SDK over.
3. Corrected old mistakes that led to an insufficiently aligned buffer being used for I/O operations with the HDD.
4. Increased the size of the buffer that contains the path of the savedata icon, which is used during installations.
5. Changed the icon selector's colour to light red and increased its opacity.
6. Added memset() statements to initialized uninitialized values. This gives a cleaner installation.
7. Fixed the problem with garbage appearing within the OSD title line 1 and 2 fields, whenever the icon data is missing.
8. Corrected logic errors within the icon loading code.
9. Greatly increased the number of buffers used by APA, which greatly improves performance when the list of games is generated.
10. Enhanced data receiving and transmission code, to totally eliminate the a possibility of errors occuring because the packet header is partially transferred (regardless of whether it's currently even possible or not).
11. Corrected a mistake within HDLFS: offset 4 is unused, while offset 6 of the HDLoader game partition structure is probably a version field and 0x1337 should not be set there. Thanks to l_Oliveira!
12. Increased the server stack size, since it was overflowing.
13. Updated DISKLOAD to Open PS2 Loader commit 025a6bb.

Downloads/links:
HDLGameInstaller project homepage: - HDLGameInstaller support page -

 

SP193, Do you think that maybe we can add more then 16 char. to OSD title someday. Maybe just in HDLGManClient and not on the PS2 itself.
I know why you set it to only 16.. but I've see legit games use from then 16 on just line 1, like Final Fantasy X Internattional's Instal Data. Also if using AKuHAK's Reworked HDL_Dump add more.
Its not that 16 is not good and all as its great, its just some names are a little bigger and having the full name looks nicer.

 

The problem is, what is the correct limit for English letters? The HDD support documentation from SONY stated that it's 16 characters per line.
Depending on the character set (Let's say, Japanese), exceeding 16 characters per line might cause the HDD OSD to deem the installation as corrupted data. Titles written in pure ASCII do not seem to have this low limit, but we don't even know for sure that the restrictions are the same across all HDD OSD versions.

And it's not a limitation that I can resolve by adjusting HDLGManClient. It's a limitation with the SONY software (HDD OSD).

 

The OSD limit is because the line size limits are set in bytes, not characters. In ASCII you have 1 character per byte but in Japanese you have either always 2 bytes per character (Shift-JIS) or 1 to 4 bytes per char (UTF8) so the binary size of the text really depends on the encoding. HDDOSD seems to require UTF8 on HDD contents and that doesn't help out at all with writing text.

Because the HDD OSD has this "hidden" backwards compatibility with ASCII text we can get away with really long lines sometimes but you can see it get out of center, which means that's certainly not intended to be used.