[csw-maintainers] Building Mozilla NSS (it aborts)

[Maciej (Matchek) Blizinski]

I've found the place where it aborts[1], by running shlibsign under dbx:

(dbx) unOS5.8_DBG.OBJ/lib/libsoftokn3.so"                                     <
Running: shlibsign -v -i
/home/maciej/src/opencsw/pkg/nss/trunk/work/build-isa-sparcv8/nss-3.12.4-with-nspr-4.8/mozilla/security/nss/cmd/shlibsign/../../../dist/SunOS5.8_DBG.OBJ/lib/libsoftokn3.so
(process id 1696)
Reading libsoftokn3.so
Reading libnssutil3.so
Reading libsqlite3.so.0
Reading libbsm.so.1
moduleSpec configdir='' certPrefix='' keyPrefix='' secmod=''
flags=noCertDB, noModDB
Reading libfreebl_32fpu_3.so
Reading libkstat.so.1
t at 1 (l at 1) signal ABRT (Abort) in (unknown) at 0xff3db7b0
0xff3db7b0:     bcc,pt   %icc,0xff3db7c4        ! 0xff3db7c4
Current function is PR_NewLock_stub
  429       abort();
(dbx) where
current thread: t at 1
  [1] 0xff3db7b0(0xffbfe220, 0xa3, 0x1, 0x6, 0x0, 0x6), at 0xff3db7b0
  [2] 0xff3d7ff4(0x1, 0x6, 0x0, 0xfefbc000, 0xff166000, 0x0), at 0xff3d7ff4
  [3] 0xff3db710(0x1, 0x6, 0x0, 0xfefbc000, 0xff166000, 0x0), at 0xff3db710
  [4] raise(0x6, 0x0, 0x0, 0xffffffff, 0xfefc03cc, 0x29010), at 0xfef4bd1c
  [5] abort(0xfefbc000, 0xff1b3e18, 0xff1b2984, 0x1494, 0x1e90c,
0x1400), at 0xfef35a34
=>[6] PR_NewLock_stub(), line 429 in "stubs.c"
  [7] rng_init(), line 391 in "drbg.c"
  [8] PR_CallOnce(0x1494, 0xfec3ac08, 0x1400, 0x27c24, 0xfecbdcfc,
0xff1b2984), at 0xff18adcc
  [9] PR_CallOnce_stub(once = 0xfecbdcfc, func = 0xfec3ac08 =
&`libfreebl_32fpu_3.so`drbg.c`rng_init()), line 463 in "stubs.c"
  [10] RNG_RNGInit(), line 469 in "drbg.c"
  [11] RNG_RNGInit(), line 834 in "loader.c"
  [12] nsc_CommonInitialize(pReserved = 0xffbfe8f4, isFIPS = 0), line
2582 in "pkcs11.c"
  [13] NSC_Initialize(pReserved = 0xffbfe8f4), line 2710 in "pkcs11.c"
  [14] softokn_Init(pFunctionList = 0xfee59224, configDir = (nil),
dbPrefix = (nil)), line 474 in "shlibsign.c"
  [15] main(argc = 4, argv = 0xffbff314), line 802 in "shlibsign.c"
(dbx)

>From reading the comments at the top of the stubs.c file it looks like
it's a file which defines function stubs which are to be later
overriden when loading shared libraries (util and nspr).  I've trussed
the execution of shlibsign and as far as I can tell, it finds each and
every shared object it's looking for.  Also, ldd shows that it finds
all the libraries:

 maciej at build8st [build8st]:~/src/opencsw/pkg/nss/trunk > ldd
/home/maciej/src/opencsw/pkg/nss/trunk/work/build-isa-sparcv8/nss-3.12.4-with-nspr-4.8/mozilla/security/nss/cmd/shlibsign/SunOS5.8_DBG.OBJ/shlibsign
        /usr/lib/secure/s8_preload.so.1
        libplc4.so =>    /opt/csw/lib/nspr//libplc4.so
        libplds4.so =>   /opt/csw/lib/nspr//libplds4.so
        libnspr4.so =>   /opt/csw/lib/nspr//libnspr4.so
        libthread.so.1 =>        /usr/lib/libthread.so.1
        libnsl.so.1 =>   /usr/lib/libnsl.so.1
        libsocket.so.1 =>        /usr/lib/libsocket.so.1
        librt.so.1 =>    /usr/lib/librt.so.1
        libdl.so.1 =>    /usr/lib/libdl.so.1
        libc.so.1 =>     /usr/lib/libc.so.1
        libpthread.so.1 =>       /usr/lib/libpthread.so.1
        libmp.so.2 =>    /usr/lib/libmp.so.2
        libaio.so.1 =>   /usr/lib/libaio.so.1
        /opt/csw/lib/nspr/cpu/sparcv8plus/libnspr_flt4.so
        /usr/platform/SUNW,SPARC-Enterprise-T5220/lib/libc_psr.so.1

I'm really stuck now.  Right now, it looks like I have to see how
shared libraries are loaded and how function names get overriden.  Hic
sunt dracones.

How do I debug the order in which shared libraries are loaded and why
aren't NSPR stubs overriden with the actual functions?  Do you have
any hints?

[1] http://mxr.mozilla.org/security/source/security/nss/lib/freebl/stubs.c#424
[2] http://mxr.mozilla.org/security/source/security/nss/lib/freebl/stubs.c#38