[csw-users] The great chkpkg mystery on Solaris 8 Sparc

Fri Jul 25 18:35:29 CEST 2008

Pardon me while I follow up on the chkpkg problem that occured on the
new build server now called ra.  A restore of the binary from an older
machine fixed the issue because a Sun patch has broken the binary.

Here is what happened, more or less in order.

The old server had  failed internal fibre controllers and I had to
attach external SCSI drives with an old three disk array JBOD ( Netra
D130 ) and that seemed to work but was slow. It seemed reasonable to
me to replace the  mess with a single 1U rack solution which had twice
as much memory and internal Ultra320 disks that would be  many times
faster. Did I mention redundant?

I installed Solaris 8 HW4 ( Solaris 8 2/04 s28s_hw4wos_05a SPARC ) on
the new server and then take a snapshot of its patch configuration.
You can see the cronological list of  all patchs required for a fresh
install of s8hw4 here :

http://www.blastwave.org/dclarke/patch/Solaris8_s28s_hw4wos_05a_SPARC/patch_report.txt

The third column is the age of the patch in days. This list was
generated on 7th of July 2008.

I then apply patches in a few  passes. The basic recommended patchesgo
in first and then various others. Eventually one must install a JDK
and then with that in place you can install Sun Studio 11.  Don't get
me started on the horrors of the Studio installer which is written in
Java.

Since yesterday there have been two more patches released :

112169-07 RS-   2 SunOS 5.8: platmod patch for Volume System H/W Series Products
121017-16 ---   2 Sun Studio 11: Patch for Sun C++ 5.8 compiler

I don't care about the 112169-07 but I'll get 121017-16.

THE POINT HERE is that I trust patches and I seem to be wrong in where
I place that trust.  The chkpkg binary on the old Sparcbuild server
was working fine but the new server fails to execute pkgchk with the
desired results. I checked and the binaries on the eold server and the
new server are not the same. They are only different because of a
patch.

So let me quickly look at the current binary that is in the new server
and compare
it to the binary that was on old ra :

# ls -lap /usr/sbin/pkgchk
-r-xr-xr-x   1 root     sys       169188 Apr  8 13:02 /usr/sbin/pkgchk
# ls -lap usr/sbin/pkgchk
-r-xr-xr-x   1 root     sys       169188 Aug 10  2006 usr/sbin/pkgchk
# /opt/csw/bin/openssl md5 /usr/sbin/pkgchk
MD5(/usr/sbin/pkgchk)= 4701004e52def0a05e8a0d3f2d3120ca
# /opt/csw/bin/openssl md5 usr/sbin/pkgchk
MD5(usr/sbin/pkgchk)= 7bc0627b61049a98b639a9aacab406e8

Looks like the file is the same size but the date is different and the
MD5 hash reveals that the two binares are just not the same.

Let me replace the binary thus :

# cp -p /usr/sbin/pkgchk /usr/sbin/pkgchk.backup
# cp -p usr/sbin/pkgchk /usr/sbin/pkgchk
# /opt/csw/bin/openssl md5 /usr/sbin/pkgchk
MD5(/usr/sbin/pkgchk)= 7bc0627b61049a98b639a9aacab406e8

So now you have pkgchk from the backup.

This has been tested and found to work.

Clearly the chkpkg binary has been broken by a patch and yet somehow
it was released to the production running Solaris 8 public. I checked
Sunsolve and there are reams of bugs filed against chkpkg which all
break the process for internal SUNW packages and I can confirm that
external organizations ( Blastwave.org ) suffer the  same fate.

Please see the list ( a truncated list ) of bugids attached below.

I am, quite frankly, disgusted at the  loss of trust that has occured
for me personally as I was always under the impression that a
production Solais machine can be trusted after you patch it and that a
patch which breaks a binary would be caught, fixed and dealt with
swiftly. This not the case here and the dates on the bugids prove
that.

I will dig like mad and find the offending patch id number.

Any questions?

Dennis Clarke
Exhausted Sysadmin

------------------------ truncated bug id list

These are all bugs filed against chkpkg internally at Sun for SUNW
packages all failing :

 SUNWtfhlf doesn't pass package policy check  |
bug 6471630
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471630-1 -
Mar 5, 2007

 SUNWsesscj doesn't pass package policy check  |
bug 6471626
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471626-1 -
Mar 5, 2007

 SUNWtfhlc doesn't pass package policy check  |
bug 6471629
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471629-1 -
Mar 5, 2007

 SUNWse6130uh doesn't pass package policy check  |
bug 6471614
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471614-1 -
Mar 5, 2007

 SUNWse6130uj doesn't pass package policy check  |
bug 6471617
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471617-1 -
Mar 5, 2007

 SUNWtfhlk doesn't pass package policy check  |
bug 6471633
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471633-1 -
Mar 5, 2007

 SUNWse6130uk doesn't pass package policy check  |
bug 6471618
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471618-1 -
Mar 5, 2007

 SUNWtfhlh doesn't pass package policy check  |
bug 6471631
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471631-1 -
Mar 5, 2007

 SUNWse6130uf doesn't pass package policy check  |
bug 6471613
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471613-1 -
Mar 5, 2007

 SUNWsefms-dpi-array-6130 doesn't pass package policy check  |
bug 6471621
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471621-1 -
Feb 28, 2007

 SUNWsefms-dpi-array-6140 doesn't pass package policy check  |
bug 6471622
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471622-1 -
Jan 19, 2007

 SUNWstksm doesn't pass package policy check  |
bug 6471628
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471628-1 -
Jan 19, 2007

 SUNWsefms-dpi-array-flx380 doesn't pass package policy check  |
bug 6471625
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471625-1 -
Jan 19, 2007

 SUNWse6130ui doesn't pass package policy check  |
bug 6471616
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471616-1 -
Jan 19, 2007

 SUNWsefms-dpi-array-6540 doesn't pass package policy check  |
bug 6471624
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6471624-1 -
Jan 19, 2007