I've been trying to reverse engineera DC .lib file but I'm not having much success. I assumed they were elf, but read-elf said they have the wrong magic bytes and every disassembler I try doesn't seem to understand the format or distinguish code from data. Has anyone done this before? Have any tips?
Just link the lib and compile a test app then reverse engineer that? If the headers aren't there, shouldn't be difficult to reconstruct them. Also, is it a static or dynamic library?
My guess would be they're 'ar' (not 'tar'!) archives, containing object code. Just like the '.a' files mentioned above, actually... Everyone - even including Microsoft, for whatever reason (legacy, probably) - seems to use that for their statically linked libraries.
I thought for a second you had it @rso with the 'ar' thing, but apparently it's not an ar archive, or a tar. I've tried a COFF viewer on it and that didn't help, so maybe it's OMF? Looking at the hex output there are definite sections with apparent padding. Each section seems to start 0xE0, 0xE2, 0xE4 etc. I'm not sure what format it is
Windows uses lib.exe for creating libraries, and dumpbin for viewing the object files within the library, just like Unix systems use ar for creating and viewing libraries. Dumpbin should be used for extraction, not ar.
I was in a hurry earlier, but I have time to elaborate now. Windows and Unix both archive object files in very similar ways - i.e. .lib files in Windows are (almost) identical to Unix .a libraries. There are very slight differences in the the archive formats but I don't know if they are enough to make ar not work on a Windows library. I decided to take a peek at some DC SDK libraries, and I think you may be examining a file that is not an archive at all (at least not a standard one). If it's an archive (either Windows or Unix), it should begin with the magic value !<arch>, and when I look at files from the Windows CE Toolkit, they do indeed have that header, but it looks like lib files from Katana do not, so if it is a Katana lib file you're looking at, then neither ar nor dumpbin should work. I've never done any DC programming, but a quick look at some of the docs suggests that CodeWarrior is used with Katana, so you may want to investigate the CodeWarrior documentation to see how it creates libraries (perhaps they use a modified version of the standard archive format?) and if they provide a tool to extract object files from them. Edit: Just created a test library with CodeWarrior and it does create a proper archive file. Not sure why the Katana libraries look so strange...but as a starting point, you should check out its documentation to see if it has its own format, or if it compresses the standard format in some way. However, it looks like there is a GNU folder within the Katana lib folder that contains standard versions of the lib files that readelf/ar/bindump can examine.
Can you say which library file you're trying to reverse engineer? I've only used the GNU objdump tool that comes with the SDK to take a look at how some of the functions work.
The Dreamcast libraries were compiled for little-endian Hitachi SH-4. If you are trying to disassemble them with an x86/64 disassembler you will get garbage. I don't know about the WinCE .lib files, but static libraries for all the supported toolchains were provided with the Katana SDK. Using the SDK supplied GNU tools (Utl/Dev/Gnu/Bin/) against the static libraries in the 'Lib/Gnu' folder dumps the library exports fine. IDA Pro also recognizes and disassembles them correctly.
