This forum seems to have been pretty quiet in the last while, so perhaps this might bring some excitement in :-).
As part of my ongoing struggles with porting a reasonably recent GCC (and other GNU stuff), I discovered that one of the largest sources of issues on NS and OS are the ancient system libraries. Even things as simple as outdated printf() formatting strings make many tools fail, such as Perl for example. Consequently, I have been trying to figure out how to refresh some of the functions within the NeXT libraries without breaking the existing system.
Unfortunately it looks like there is no help forthcoming from Apple in the form of the original source code, so we are likely going to be on our own. I spent hours searching through the old usenet posts trying to determine how to patch or to tweak the NS3.3 shlibs, only to discover that no one other than NeXT themselves were ever able to build shared libraries. And then I finally had a bit of an eureka moment when I came across old references to a "mysterious"
mkshlib tool. As it turns out the source code to mkshlib has been released by Apple in the Darwin cctools package, and it is almost unchanged since the dark ages.
With a few minor tweaks I was able to almost completely compile cctools-384.1 on OS i386 ('dyld' does not build of course, and 'ar' chokes as well). I then studied the mkshlib manpage and continued searching the net for other references and examples. I ultimately discovered that the mkshlib tool seems to be related to Sys V Unix and there is reasonable amount of information available on the SCO site at
http://osr507doc.sco.com/en/tools/ShLib.html.Armed with all that info I set off to test the concept. I created a replacement object file for the frexp() function on my NS m68k, using some assembly code I found a while ago. I then created the required library spec file, moved everything to the OS i386 machine and after a few final tweaks I was staring at two output files: libsysnew_s.A.shlib and libsysnew_s.a. I quickly moved the files into /lib and /usr/shlib on my NS m68k and built a quick frexp() test program with the "factory" libraries. I then used 'ar' to remove the NeXT version of frexp() from /lib/libsys_s.a and rebuilt the test program against /lib/libsys_s.a and /lib/libsysnew_s.a.
And it worked just fine!!! Inspecting the executable with otool showed that it was pulling objects from the NeXT libsys shared library as well as from my newly built shared library.
Because of how the NeXT shlib scheme works, removing an object reference from the .a "host" library of a "target" shlib will have no impact on the existing executables and will only impact newly created & linked executables. This mechanism therefore appears to allow substitution of new versions of existing functions without the need to change their names with #define of #pragma. I am really thrilled that this might provide a reasonably decent solution to the system library refresh dilemma.
One of the challenges with this approach is a need for a centralized control of the memory space in order to prevent different shared libraries from overlapping. I already mapped out all the existing NS 3.3 shlib memory space, however, so that I know where new shlibs can be slotted in.
Next I will do a bit more work to build cctools natively on NS. This could also potentially open the doors to building a cross-compiler for the m68k and perhaps a BSD 4.4 based 'ar' with support for long object names...
...now time to get some sleep.
Well done!
This is exciting. Would I dare say the seeds for an OPENSTEP 4.3?
Quote from: "t-rexky"
One of the challenges with this approach is a need for a centralized control of the memory space in order to prevent different shared libraries from overlapping. I already mapped out all the existing NS 3.3 shlib memory space, however, so that I know where new shlibs can be slotted in.
Details soon please.
Thanks for all the hard work,
/pitz
Excellent work!
I have no idea what most of this post means, but it sounds very impressive.
Correct me if I'm wrong, but what you're saying is that you've made progress getting a modern cctools to build. If so, excellent work
Thanks for the encouragement guys. Here is a bit of an update.
I spent a bit more time tinkering on the slab over the weekend and was able to partially build cctools-358 (from OS X 10.0.4) on NeXTstep 3.3 m68k. It took quite a bit of elbow grease since there have been many system level changes (libraries, compilers, etc.). I had to first build cc-471.7 from prerelease OPENSTEP 4.0 as it supports the __private_extern__ compiler keyword that is (kind of) required to build cctools with minimal changes. I first tried to rebuild the 3.3 compiler cc-437.2.6 with added support for __private_extern__ but as it turns out this endeavour was not as easy as I had hoped and the resultant compiler is flaky. In addition, the cctools build many of the libraries and code with the -dynamic compiler option that does not work on the NS33 compiler.
To make a long story short, I was able to get cc-471.7 running on NS33 and with it I was able to build majority of cctools-358. I now have the following cctools binaries running natively on NS33:
./as/a68_dir/as
./file/file
./gprof/gprof
./ld/ld_dir/ld
./misc/nm
./misc/lipo
./misc/size
./misc/strings
./misc/libtool
./misc/checksyms
./misc/indr
./misc/strip
./misc/nmedit
./misc/segedit
./misc/kern_tool
./misc/cmpdylib
./misc/dylib_pcsampler
./misc/pagestuff
./misc/redo_prebinding
./misc/seg_addr_table
./misc/check_dylib
./misc/seg_hack
./misc/atom
./mkshlib/mkshlib
./mkshlib/cmpshlib
./mkshlib/hack_libgcc
./otool/otool
./profileServer/dylibprof
./profileServer/profileServer
I was particularly interested in the linker, so I immediately substituted it for the NeXT linker and so far everything seems to be working well! I am having some issues getting 'ar' to build because it uses a lot of BSD4.4 specific functions and features, requiring a lot of hacking. I am at the point where I now need to build a complete replacement for sprintf() because 'ar' breaks when using the NeXT library version on NS33 and OS42...
However, all of the above means that building a cross compiler is now definitely a much closer possibility since the assembler and linker can be build from sources. Of course, there have been many changes to the linker from NS33 to OS X 10.0.4 so I am shocked that it seems to work and I am expecting it to croak really soon :-).
@gregwtmtno: The shlib stuff is NeXTstep specific as on OPENSTEP shlibs were pretty much replaced by dylibs and bundles, akin to OS X. The advantage of being able to build shlibs for NS is that a large library including fixes and enhancements to the existing NS system library functions can be built as a shared library as opposed to a static library. With a static library, every running executable will load into memory its own copy of the library functions (such as sprintf() for example), using up precious RAM. If sprintf() resides in a shlib, on the other hand, every executable in memory will only use one and the same instance of it - it resides in RAM only once.
Brilliant work!
I had a bit of a revelation. A few quick tweaks of the printf formatting strings and I now appear to have 'ar' working without the need for custom printf functions. And yes, it does support long file (object) names without the 15 character limit of NS and OS 'ar' which should make life much easier with newish gcc library builds.
Now I have to do a bit of testing of all this new stuff.
I now have a quad fat package put together of cctools-435.4 from OS X 10.2.8 that will be submitted to the file archive here as soon as Nitro resets my access. No point in cross-posting so I will provide more info in my gcc-3.2.3 thread...