[csw-devel] SF.net SVN: gar:[11879] csw/mgar/gar/v2
wahwah at users.sourceforge.net
wahwah at users.sourceforge.net
Fri Dec 10 17:48:59 CET 2010
Revision: 11879
http://gar.svn.sourceforge.net/gar/?rev=11879&view=rev
Author: wahwah
Date: 2010-12-10 16:48:58 +0000 (Fri, 10 Dec 2010)
Log Message:
-----------
checkpkg: File collision detection
This revision introduces file collision detection in checkpkg.
It also changes the database usage scenario. Instead of a private
sqlite database for every user, there's one shared mysql database for
everyone.
Each packages is checked in the context of the whole catalog. Packages
installed in the system have no influence on the checking process.
The automatic database update after each package installation or removal
is now gone. There is a distinction between CSW packages and other
packages.
It's still possible to run a private sqlite database, but the
performance is currently very poor.
This commit is a squashed version of multiple commits done in git.
Happy package checking!
Maciej
Squashed commit of the following:
commit 2d96a59e519fa01b1739560d958c8510735bd250
Merge: a5a5338 134cf22
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Fri Dec 10 17:16:09 2010 +0100
Merge branch 'local-db-mgmt' of file:///home/maciej/public_html/opencsw
Conflicts:
gar/v2/lib/python/opencsw.py
commit 134cf22ebc317c578cd6e98b4dc8cff6a7f2ce6e
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Fri Dec 10 13:06:43 2010 +0000
checkpkg: Automatic database management support
It's running very, very slowly. The re-population procedure can easily
take one hour.
commit 962e61622c3061744412159f1f9ff9696f7968ec
Merge: ee5fb1f 0b6dde6
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Fri Dec 10 09:42:20 2010 +0000
Merge branch 'collisions' into col-buildfarm
There was a lot of conflicts. This is what you get if you try to use
the HTTP transport for git repositories, which exports old data. You
end up working against a really old code and getting a lot of code
conflicts afterwards.
Conflicts:
gar/v2/lib/python/checkpkg.py
gar/v2/lib/python/checkpkg2.py
gar/v2/lib/python/checkpkg_lib.py
gar/v2/lib/python/checkpkg_lib_test.py
gar/v2/lib/python/checkpkg_test.py
gar/v2/lib/python/common_constants.py
gar/v2/lib/python/configuration.py
gar/v2/lib/python/database.py
gar/v2/lib/python/ldd_emul.py
gar/v2/lib/python/models.py
gar/v2/lib/python/package_checks.py
gar/v2/lib/python/package_stats.py
gar/v2/lib/python/pkgdb.py
gar/v2/lib/python/system_pkgmap.py
gar/v2/lib/python/system_pkgmap_test.py
gar/v2/tests/run_tests.py
commit 0b6dde6512739689bd6446418134e9b72d061c7f
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Dec 7 08:37:47 2010 +0000
checkpkg: Added support for sqlite connections
If the configuration file is not present, it's created automatically and
populated with sane defaults: using sqlite in ~/.checkpkg.
commit 2a6908a04f6a1de0b1e911b6e04070c6dfb9cd28
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Dec 7 08:36:31 2010 +0000
checkpkg: Using SunOS9.9 format in the help msg
Also removed commented out option which is no longer needed.
commit a5ff4289810b956d320829cd8df3d028d72dafba
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 23 11:53:54 2010 +0100
mGAR v2: checkpkg, moved db connection stuff to configuration.py, started working on checkpkg runs.
commit 03ca394fcfefcbec1d4a8658215a0b5f9ae3b573
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 10:19:35 2010 +0000
mGAR v2: checkpkg, removed unused imports from checkpkg.py.
commit a0c20b64e9b63dccea2244da3b421ec3d0e5be20
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 10:10:25 2010 +0000
mGAR v2: checkpkg, created checkpkg_lib.py with the intention to empty checkpkg.py.
commit b0b71585b1a86d7c5ba96887087f4fe1ba86081a
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 08:08:53 2010 +0000
mGAR v2: checkpkg, implemented GetInstalledPackages in the Catalog class.
commit 3e9a26cd53b88c64d2c35f1d31e210c0484d219c
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 07:06:37 2010 +0000
mGAR v2: checkpkg: splitting off LddEmulator and SystemPkgmap to separate files.
commit 0ff2904a44fd07bc2b5bfddd951474e6a1b0a4fa
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Mon Nov 22 23:23:25 2010 +0000
mGAR v2: checkpkg, checking package architecture before inserting into a catalog.
commit d1a920d836d9d996efa9c62179b5787f6ba0aa32
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Mon Nov 22 22:00:18 2010 +0000
mGAR v2: checkpkg, a registering an existing package works now.
commit 3940ac5ebddb6749a30a7452c7dc6dad438cd90a
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Mon Nov 22 21:50:10 2010 +0000
mGAR v2: checkpkg, removing the super() call in SystemPkgmapMixin
commit d251e74fba8effc7a6252cbc5366fba7b5174054
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Mon Nov 22 21:06:26 2010 +0100
mGAR v2: Removing the database classmethods, and making sure that we're returning files that belong to packages that are in catalogs (although the check is slightly redundant).
commit cad7a0eb75bfc76b85646e7c9bb91a27632c06c0
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Mon Nov 22 11:14:44 2010 +0100
mGAR v2: checkpkg, more db refactoring, tests are passing, pkgdb needs to be fixed.
commit 46da2df03e5a281a4b5ee6d8919601c690781d15
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 18:25:00 2010 +0000
mGAR v2: checkpkg, removing a package from a catalog works now.
commit c6663f06904d9f15d7c5f0f6b9cac747abd8e6fd
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 15:46:49 2010 +0000
mGAR v2: checkpkg, not allowing two packages with the same pkgname in one catalog.
commit ce90fc50d3f956c695e3478a60f4d63e335c093c
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 14:46:37 2010 +0000
mGAR v2: checkpkg, checking that putting one package two times in a catalog raises an error.
commit d57371da016d94b976e57ec0cf06c94c02bb643d
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 14:34:33 2010 +0000
mGAR v2: checkpkg, small unit tests cleanup.
commit 1d86b191635a574f50b37e52b6c0b9e1b02610ac
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 14:26:27 2010 +0000
mGAR v2: checkpkg, querying the database for files works.
commit ab0e6e80b920065f9bad5097a9697fbff9480963
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 12:13:25 2010 +0000
mGAR v2: using #!/usr/bin/env for checkpkg_test.py
commit 00d0eba622308a78f0a536d04aa87d2952d7f144
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Nov 21 13:08:32 2010 +0100
mGAR v2: checkpkg, start of a test for retrieving pkgnames for a basename. The netzwerk lags today, moving development home.
commit a0775af72c164a548411c5808f0dafaac571c27d
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Nov 21 12:44:48 2010 +0100
mGAR v2: checkpkg, registering a package with a catalog.
commit cdfe61d4cf9e04c787accddbd8dfe56df696116c
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sat Nov 20 19:00:28 2010 +0100
mGAR v2: checkpkg, a test for inserting stats into the db.
commit dd7e5f88dbb1dc35c9883ba90a32d9a3f33de7b5
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sat Nov 20 18:10:58 2010 +0100
mGAR v2: checkpkg, started a database related unit tests.
commit ca23f8080f0ee979a01765dfc21619ae8b525c8e
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sat Nov 20 11:07:37 2010 +0100
mGAR v2: checkpkg, removing the remaining korn shell code.
commit 1e007cd71ef772ffa181f85829aecdf71400a160
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Thu Nov 18 12:32:33 2010 +0100
mGAR v2: checkpkg, insignificant change to models.py
commit ee5fb1fa6d5a16b845107c4e853f7a90af4d0690
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Mon Dec 6 18:17:54 2010 +0100
mGAR v2: Checking packages against 'current'
Defining CATALOG_RELEASE and setting it to 'current' by default. This
means that packages will be checked against the current branch by
default.
The current branch is meant to be kept in sync with the on-disk
(on-mirrors) catalogs, while 'experimental', 'unstable', 'testing' and
'stable' are to be updated with a different set of packages.
commit 88df8e297f99a7904e77c54af87180177d7ca956
Merge: 49d51c7 5ad7504
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Dec 8 10:36:53 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit 49d51c75c7c28023bc805a2f5579e695a48a3963
Merge: 489957d 1d38d9d
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Dec 8 10:36:32 2010 +0100
Merge branch 'master' of file:///home/maciej/public_html/opencsw into collisions
commit 1d38d9d9130ebeff1cdac2108ea02c4312624dc0
Merge: 332c204 28ccacb
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Dec 8 09:50:28 2010 +0100
Merge branch 'master' of file:///home/maciej/public_html/opencsw
commit 5ad7504c053039b9b07a81ca16b36f1c3a7011d2
Merge: c5306dc 489957d
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Wed Dec 8 08:50:20 2010 +0000
Merge branch 'collisions' of ssh://login.opencsw.org/home/maciej/public_html/opencsw into collisions
Conflicts:
gar/v2/lib/python/package_checks.py
commit 28ccacbece5b6479a5b73b29f71c176d930bb17d
Merge: 918abba daaa503
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Dec 8 09:08:34 2010 +0100
Merge branch 'master' of file:///home/maciej/public_html/opencsw
commit c5306dcfc5cb0eda325a74d892576f184aca020d
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Wed Dec 8 07:54:45 2010 +0000
checkpkg: A fix for PackageStats unit test
There was a change to use frozensets, unit tests needed updates.
I also added a check for mtime value; the mtime field is currently null
in the database, and it needs debugging.
commit 489957d59ba14a1d5d89247930c29921e907e5ec
Merge: 6ab061d daaa503
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Dec 8 00:42:20 2010 +0100
Merge branch 'master' of file:///home/maciej/public_html/opencsw into collisions
commit daaa50376aed243c189668c8fc0101b7b91bfda1
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Dec 7 22:51:00 2010 +0000
checkpkg: A message for an init file class check
Added a message and a GAR line suggestion for the
init-file-missing-cswinitsmf-class error tag.
commit 6ab061db2c6ae4fe5d6fadb5f69a0b585d2e3c2e
Merge: 5d004ef 08da91c
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Dec 7 14:25:44 2010 +0100
Merge branch 'master' into collisions
commit 5d004ef738ef2c25b0770541808faf2405018bdf
Merge: 9d343d0 a8f0add
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Mon Dec 6 17:48:23 2010 +0100
Merge branch 'master' of file:///home/maciej/public_html/opencsw into collisions
commit 76c35400eadab8a8ee4d5d999360e1a26d5c74f4
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Dec 5 23:39:11 2010 +0000
checkpkg: Profiling support in the main executable
commit 8d51eb10aa5b74b1daabe9585ef7bc8ef8508eef
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Dec 5 23:38:33 2010 +0000
checkpkg: Adding a missing CATALOGNAME_CSWfoo line
When suggesting shared library split-offs.
commit 9d343d09fa4b0f33bb2c3ff0ea0bfb0dee9780a1
Merge: 6f9cf13 e686424
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Mon Dec 6 00:24:06 2010 +0100
checkpkg: Using os.normpath for path sanitization
Conflicts:
gar/v2/lib/python/checkpkg.py
commit 6f9cf137c05655c1ba4d3a17adb969087379be6b
Merge: e744306 ad653d0
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Mon Dec 6 00:11:24 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit e744306709404652a04fa435af79e04fe9b2ed16
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Dec 5 19:27:46 2010 +0100
checkpkg: File collisions check is working
There is warning displayed if colliding files are found, it can be fixed
but I'm leaving it for later.
commit bd13f29160f98f3fda8d60c2b7d3e3168b5c9e40
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Dec 5 16:50:52 2010 +0100
GAR v2: Calling checkpkg with new required options
commit 573a0f0c4161286f29b467e62d2dc2b8b03cb7b3
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Dec 5 14:30:10 2010 +0000
checkpkg: A better message for the libmagic bug
Displays helpful information to the user - says it's a known problem and
suggests restarting the process.
commit a6eabd39f6fb5c2ff46a5f1752603b9d6e57b1f2
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Dec 5 15:24:12 2010 +0100
pkgdb: One command to import one catalog release
Imports all OS releases and all catalogs.
commit 3c7d0c3612692ef9fd064decdc1c0d63ade68025
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Dec 5 14:23:45 2010 +0100
pkgdb: Started a unit test file.
commit 919fe07a4d79f6c6dae8eb8d147f55d64c8025c6
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Dec 5 13:22:43 2010 +0000
pkgdb: Adding models_test.py
commit f6e389d289056a85cd5b396992b008633efa7eee
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Dec 5 13:09:13 2010 +0100
pkgdb: A few fixes for the import catalog feature
Tested live on the buildfarm, works. It could use some refactoring, and
perhaps one more convenience function, that imports all architectures
and all OS releases.
commit f40b848d1de2e6444e53a07fb5711aad67b79d3a
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Dec 5 10:27:41 2010 +0000
pkgdb: Syncing a single catalog from disk
This convenience function allows to import a catalog state from the
disk, using a catalog file, to the database. No matter what were the
previous srv4-catalog assignments, it brings the catalog in the db to
the state that is on disk.
commit 8332605c49a5994493efb7114884973e7371a8d0
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 16:42:13 2010 +0000
pkgdb: Added 'pkg search' command.
commit ecc931ea3b17f96b45d2d076e683d5660a1f3b9a
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 16:27:47 2010 +0000
checkpkg: Unit test for ToGarFormat()
commit 514e69806b4afb9ad90b92f4f95b156b2a65c290
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 16:02:32 2010 +0000
checkpkg: The checkpkg tool passes smoke test
It can be fed file names or md5_sums. Runs and displays results.
commit 3e81a9d6f748ded49d1840e34e23719cc55f6669
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 15:46:43 2010 +0000
checkpkg: Fixed a bug in system files import
When linking a file with a package, all files previously associated with
the package would be removed. Changed this so that only the first
occurence cleans the files previously associated with the package.
commit 51fcdc8d2c399b0ca7cd42eb694956382d726548
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 13:06:20 2010 +0000
checkpkg: A unit test for CheckpkgManager2.Run()
Verifies that error tags are actually inserted into the database.
commit 37495c9ebdb9bcf9176195dd86dd7799f16e0b98
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 12:46:53 2010 +0000
checkpkg: Using common_contants to check catrels
It's better than a hardcoded value.
commit 9dad9b1d2d8405c8fb4edd61d26ab2f79a57488c
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 07:51:35 2010 +0000
mGAR v2: checkpkg, cleaning up CheckpkgManager2
Added a method to clean up checkpkg results. Some work towards unit
testing the main Run() methods.
commit 09220b8c10e5e9e9283974aace90ec7f224189e6
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Thu Dec 2 22:56:22 2010 +0000
mGAR v2: pkgdb, making initdb idempotent
Not that it was a problem, but it's nicer that way.
commit 4580244a7eeb2cb25b139dd548e5b5038898dd70
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 30 21:05:43 2010 +0000
mGAR v2: checkpkg, skipping over missing md5
If a md5 sum was supposed to be added to the catalog, but is not in the
database, print a warning and continue.
commit aa061769c925e7e46bebd7435d492a6163549c69
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 30 08:32:14 2010 +0100
mGAR v2: checkpkg, calling chkpkg by full path
Some hosts don't have /usr/sbin in the $PATH.
commit e0879aca39279f017ecc787f05fa402f91a93ffe
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 30 00:39:28 2010 +0100
mGAR v2: checkpkg, models cleanup
Also, a bugfix for the pkginfo and contents streams, which may be lists,
and sometimes can't be close()d.
commit 918fc003772f4e3442c58e0aa7536d35b081263c
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Mon Nov 29 23:36:27 2010 +0000
mGAR v2: checkpkg, storing srv4 file size
Also, a unit test for _ColectStats, which had to mock quite a lot of
calls to other classes functions. Could use some cleanup.
commit a83b90174cdfba999013743b2ec32841c163cb35
Author: Maciej Blizi?\197?\132ski <blizinski at google.com>
Date: Mon Nov 29 10:54:55 2010 +0000
mGAR v2: checkpkg, a bugfix for package_stats.py
Sadly, this bit of code is not unit tested.
commit 904acfc689602ec1da8eb7179c5d2bb00f39a91c
Author: Maciej Blizi?\197?\132ski <blizinski at google.com>
Date: Mon Nov 29 10:49:00 2010 +0000
mGAR v2: checkpkg, not shelling out to isalist
Sheling out to 'isalist' to get the list of ISAs imposed a restriction,
that sparc packages could not be indexed on i386 boxes and vice-versa.
This change hardcodes the isalists and allows to index packages
independently of the architecture.
Data structures remain to hold the isalist as part of the pickled data
in the database, so each package has its own list of ISAs.
commit cb83c7332d6b3221c6545c93969c7d02afb5e12c
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 28 22:33:06 2010 +0000
mGAR v2: checkpkg, using highest pickling format
Pickling seems to be fairly memory-intensive with the data sets we have.
Setting the pickling format to the highest format in the hope it'll be
more efficient.
commit cfdc66adb3e2ed4650d929963a589b7c0d640591
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Nov 28 19:46:11 2010 +0100
mGAR v2: checkpkg, install/contents support
Accepting a bad line in install/contents.
There is a line that does not follow the spec.
E-mail thread:
http://lists.opencsw.org/pipermail/maintainers/2010-November/013277.html
commit eea5db79dcb25c64030a2784e2842a0eb243e2c6
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 28 16:37:52 2010 +0000
mGAR v2: checkpkg, importing system files into db
Imports files into the database, providing idempotence.
- MuteProgressBar helps with batch mode
commit 3a1aa1d654aba0c09e76e04c4238925d7583dbba
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Nov 27 13:08:18 2010 +0000
mGAR v2: pkgdb, implemented proper parsing and pickling of /var/sadm/install/contents.
commit 4c342e8a64f4320f7ed33b2c56c42c568cc345af
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Fri Nov 26 09:39:04 2010 +0100
mGAR v2: checkpkg, adding 'overriden' to the error tag class.
commit b4729052a533c5a58c16e24558f7213e8ceef493
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Thu Nov 25 08:40:46 2010 +0000
mGAR v2: checkpkg, added a frontend to del-from-cat.
commit bde27cae59374cb0b6f0eb995ed3ebbebbdb3bdb
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Thu Nov 25 01:37:58 2010 +0100
mGAR v2: checkpkg, pkgdb: Logging which packages are added to a catalog.
commit a2edf0e211b1e41819a654c0240e55174554a1d4
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Thu Nov 25 00:37:33 2010 +0000
mGAR v2: Moved heavy pickled data to a separate table, from Srv4FileStats to Srv4FileStatsBlob.
commit 1ae31596999fb57d924b151a4d77f6236eb68bea
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Thu Nov 25 00:09:05 2010 +0000
mGAR v2: checkpkg, warning on filename vs pkginfo architecture declaration mismatch. Setting logging level to 'error' in unit tests.
commit a38d6128029c84ee982703081d26e70a11e09ff7
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Thu Nov 25 00:06:10 2010 +0000
mGAR v2: checkpkg, allowing to add a i386 package to a sparc catalog if the filename declares 'all'.
commit 5f303d7d3c5a74fafe54810af509d59667d35625
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 24 18:08:04 2010 +0100
mGAR v2: Added filename_arch to the srv4_file_stats table, so I'll be able to quickly access both pkginfo and filename arch declarations.
commit ce765b5856a28ae33b7bec68fc3a0022c7abfaa1
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 24 13:49:49 2010 +0100
mGAR v2: checkpkg, more verbose logging when importing packages.
commit f92bd6603b742f36dce6952fbfbeeeb1cc73b7b3
Author: Maciej Blizi?\197?\132ski <blizinski at google.com>
Date: Wed Nov 24 12:49:11 2010 +0000
mGAR v2: checkpkg, not throwing an error when adding a package to a catalog and the package is already there.
commit 392addca6d76372f966a1a7895b173a82cf41e2f
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Wed Nov 24 09:56:58 2010 +0000
mGAR v2: checkpkg, comments in the models.py file.
commit dffec9394053b8fc09a836d00b4f0f2e496d51e4
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Wed Nov 24 08:57:22 2010 +0000
mGAR v2: checkpkg, ImportPkg(): if a package was previously imported, don't replace.
commit a1add3419a14cc4e2920aab4f295e9021e3df774
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Wed Nov 24 08:53:49 2010 +0000
mGAR v2: checkpkg, removing previously existing files in ImportPkg().
commit 1a6c25a6a1bcf9a3728aaed97ddfa105d0c9f605
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Wed Nov 24 08:33:05 2010 +0000
mGAR v2: checkpkg, support for latin1 encoded file names.
commit 2d8c448677155eff0f68ba164e9af944424fe7e0
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 24 08:58:10 2010 +0100
mGAR v2: checkpkg, a test case for the latin1 encoded file name problem.
commit bb65d859b0debfde01bec49a0a0677a0ea56187c
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 24 00:22:29 2010 +0100
mGAR v2: checkpkg, adding missing operator module import to checkpkg_lib, checkpkg just started working.
commit f8a94116407d9b621d480d99d08f6d33f0e40753
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 24 00:07:31 2010 +0100
mGAR v2: checkpkg, memoization of GetPkgByPath().
commit f257045629b66164ad99e158d3dde6dbe0e533a8
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 24 00:02:07 2010 +0100
mGAR v2: checkpkg, implemented new GetPkgByPath() in the Catalog class.
commit a67d494fb01e7b68f7c025a32a8aad2b8bb15024
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 23 21:53:00 2010 +0100
mGAR v2: checkpkg, throwing an error if an unregistered package is given for inclusion in a catalog.
commit 8c24f724a7b1e726016701741f5bccb591005d18
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 23 19:43:02 2010 +0100
mGAR v2: checkpkg, importing to the database can import some packages, but it fails with sqlobject.dberrors.OperationalError: Got a packet bigger than 'max_allowed_packet' bytes
commit b4d2689808a41d2ba4c6fc52dfb2cf146f0cbbac
Author: Maciej Blizi?\197?\132ski <blizinski at google.com>
Date: Tue Nov 23 14:24:32 2010 +0000
mGAR v2: Added sqlobject to the list of dependencies on Debian/Ubuntu.
commit 059b2a187252e32753651888284c1ad79ea5c4a5
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 10:55:21 2010 +0000
mGAR v2: checkpkg, renamed 'bar' to 'pbar' to satisfy pylint.
commit 330688c955f950ca2d5936f052997d9e32dba166
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 23 11:53:54 2010 +0100
mGAR v2: checkpkg, moved db connection stuff to configuration.py, started working on checkpkg runs.
commit deb3592857b917ec45507d0df21fc7a17f5c27de
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 10:19:35 2010 +0000
mGAR v2: checkpkg, removed unused imports from checkpkg.py.
commit 8e3d9c1629b099df5eb8c532ec7852e6bdc47cf3
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 10:10:25 2010 +0000
mGAR v2: checkpkg, created checkpkg_lib.py with the intention to empty checkpkg.py.
commit 603df4ed5da34ca73653ac0c78dc1d14300128ab
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 08:08:53 2010 +0000
mGAR v2: checkpkg, implemented GetInstalledPackages in the Catalog class.
commit 6fb79e23175b64a17bf17737d704e5b55e402a04
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 07:06:37 2010 +0000
mGAR v2: checkpkg: splitting off LddEmulator and SystemPkgmap to separate files.
commit 6841f6378d3aecf0087914a41610fadc4ea2cde0
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Mon Nov 22 23:23:25 2010 +0000
mGAR v2: checkpkg, checking package architecture before inserting into a catalog.
commit 36f71f6fd6d61fb9152b115c14ce8d90a6acde41
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Mon Nov 22 22:00:18 2010 +0000
mGAR v2: checkpkg, a registering an existing package works now.
commit 4b4ae3496ce42583d74498f76a14be17bc7c7de9
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Mon Nov 22 21:50:10 2010 +0000
mGAR v2: checkpkg, removing the super() call in SystemPkgmapMixin
commit c029546b72d8050ca9295b7ae392c029c963fc8f
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Mon Nov 22 21:06:26 2010 +0100
mGAR v2: Removing the database classmethods, and making sure that we're returning files that belong to packages that are in catalogs (although the check is slightly redundant).
commit 80fe1f38463aeb84aaae4afac204c3c570e272a4
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Mon Nov 22 11:14:44 2010 +0100
mGAR v2: checkpkg, more db refactoring, tests are passing, pkgdb needs to be fixed.
commit 6de1d3a315bb4d17395f43fd795736abf97e20af
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 18:25:00 2010 +0000
mGAR v2: checkpkg, removing a package from a catalog works now.
commit bd311acc53e551279f3d7dffb0e2250b08c78d85
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 15:46:49 2010 +0000
mGAR v2: checkpkg, not allowing two packages with the same pkgname in one catalog.
commit 543f01b737c1ae948b9bda6b58045c72500844ae
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 14:46:37 2010 +0000
mGAR v2: checkpkg, checking that putting one package two times in a catalog raises an error.
commit af4b68bb853318179b43db599728f836bfd6bba7
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 14:34:33 2010 +0000
mGAR v2: checkpkg, small unit tests cleanup.
commit 5a1e29b064fe6cf28a1ca1ef07de55ad08b62d0c
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 14:26:27 2010 +0000
mGAR v2: checkpkg, querying the database for files works.
commit a0995e0faa9d28d9a3c30f67bd8c4d4d7e819285
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 12:13:25 2010 +0000
mGAR v2: using #!/usr/bin/env for checkpkg_test.py
commit 3c0a9f725bb70f4d744368876321715f8a8f01c1
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Nov 21 13:08:32 2010 +0100
mGAR v2: checkpkg, start of a test for retrieving pkgnames for a basename. The netzwerk lags today, moving development home.
commit f22c753eefcd7a2874d625ef6751817be4628980
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Nov 21 12:44:48 2010 +0100
mGAR v2: checkpkg, registering a package with a catalog.
commit 445fd5d8d78349f1828d65c8e6e145324ae4b793
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sat Nov 20 19:00:28 2010 +0100
mGAR v2: checkpkg, a test for inserting stats into the db.
commit f320ca6f8e31bf24aeed1d754025e72b3631e32b
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sat Nov 20 18:10:58 2010 +0100
mGAR v2: checkpkg, started a database related unit tests.
commit dd322d77647b6894ee426d8e1209d46eac0a1e4e
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sat Nov 20 11:07:37 2010 +0100
mGAR v2: checkpkg, removing the remaining korn shell code.
commit 5f306157431ea10be7273617bbd75117ac612cc8
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Thu Nov 18 12:32:33 2010 +0100
mGAR v2: checkpkg, insignificant change to models.py
commit ad653d03fc409356d477c24bc2515c206b571fe9
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Dec 5 19:27:46 2010 +0100
checkpkg: File collisions check is working
There is warning displayed if colliding files are found, it can be fixed
but I'm leaving it for later.
commit 3d98f9deda4e6e7e9a329c2b3116322e97aa72d4
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Dec 5 16:50:52 2010 +0100
GAR v2: Calling checkpkg with new required options
commit d87ff582fd2b78403085dc7fd754aa0b88f4e62b
Merge: 7db825e 01d1b93
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Dec 5 15:31:17 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit 01d1b93716caa4dcd94618df5c7b3656f63ccb71
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Dec 5 14:30:10 2010 +0000
checkpkg: A better message for the libmagic bug
Displays helpful information to the user - says it's a known problem and
suggests restarting the process.
commit 7db825ee329114da5043d6fd7c5cb1eb8afeb10f
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Dec 5 15:24:12 2010 +0100
pkgdb: One command to import one catalog release
Imports all OS releases and all catalogs.
commit 47e96015d125d12b4422fa9bc63d1843e5dec7a3
Merge: c307773 9bae110
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Dec 5 14:25:30 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit c30777357c6674505c7fcdcbea58a602c5c9a339
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Dec 5 14:23:45 2010 +0100
pkgdb: Started a unit test file.
commit 9bae11075d3a0b066007551aa1cfb1e6e02e180b
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Dec 5 13:22:43 2010 +0000
pkgdb: Adding models_test.py
commit 643cc52677853847f4177a1e1785a438e27262a2
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Dec 5 13:09:13 2010 +0100
pkgdb: A few fixes for the import catalog feature
Tested live on the buildfarm, works. It could use some refactoring, and
perhaps one more convenience function, that imports all architectures
and all OS releases.
commit 1691c5cc4709ddb94ea0b0eedad9be37e482d0dd
Merge: e3dddeb ab6e10d
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Dec 5 11:28:58 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit ab6e10daf883b60e0fea1fab5c4e600399f48615
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Dec 5 10:27:41 2010 +0000
pkgdb: Syncing a single catalog from disk
This convenience function allows to import a catalog state from the
disk, using a catalog file, to the database. No matter what were the
previous srv4-catalog assignments, it brings the catalog in the db to
the state that is on disk.
commit e3dddebb782c403b7cd43bbc09d3e539bde5909d
Merge: 7e32e6f a67e05d
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sat Dec 4 17:42:21 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit a67e05dd0f31ac0e6564fa0a18612424e5aa7f86
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 16:42:13 2010 +0000
pkgdb: Added 'pkg search' command.
commit ff28462a8817e82115a70c006d9f1fc9a1c841f8
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 16:27:47 2010 +0000
checkpkg: Unit test for ToGarFormat()
commit 7e32e6f2e99f1cd1d0c0116db9df8c9e2e4256f1
Merge: 17530b3 e87ea55
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sat Dec 4 17:03:45 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit e87ea55be20509703b3127b74026dc5b90f4c749
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 16:02:32 2010 +0000
checkpkg: The checkpkg tool passes smoke test
It can be fed file names or md5_sums. Runs and displays results.
commit 17530b318738cac9973bb3fcf2aa4ea83dcef6a8
Merge: 1fb6151 2abee78
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sat Dec 4 16:48:22 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit 2abee78f910810da4938c973bddd7c79b981346b
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 15:46:43 2010 +0000
checkpkg: Fixed a bug in system files import
When linking a file with a package, all files previously associated with
the package would be removed. Changed this so that only the first
occurence cleans the files previously associated with the package.
commit 18fcb6c3584786eaa13c195714efc83f539a1606
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 13:06:20 2010 +0000
checkpkg: A unit test for CheckpkgManager2.Run()
Verifies that error tags are actually inserted into the database.
commit 72f82113376254704d5d19631908e5872c060ccc
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 12:46:53 2010 +0000
checkpkg: Using common_contants to check catrels
It's better than a hardcoded value.
commit 7d0becb86c3384e9cd13ce8a58ea3437970b67db
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Dec 4 07:51:35 2010 +0000
mGAR v2: checkpkg, cleaning up CheckpkgManager2
Added a method to clean up checkpkg results. Some work towards unit
testing the main Run() methods.
commit a48e5a3e81c01f03658cdf3a0bda22ea3b1e097d
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Thu Dec 2 22:56:22 2010 +0000
mGAR v2: pkgdb, making initdb idempotent
Not that it was a problem, but it's nicer that way.
commit 1fb61518a1e1ece53dc1e4cae4270476b2dada3f
Merge: 48c88c9 034d650
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 30 22:06:22 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit 034d65058012c36c04f35373cd2198801fcd562c
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 30 21:05:43 2010 +0000
mGAR v2: checkpkg, skipping over missing md5
If a md5 sum was supposed to be added to the catalog, but is not in the
database, print a warning and continue.
commit 48c88c9981f23198ba3dfe2170ba2efaac5fb0da
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 30 08:32:14 2010 +0100
mGAR v2: checkpkg, calling chkpkg by full path
Some hosts don't have /usr/sbin in the $PATH.
commit 21d2c1b406e4675b3582c24eb9b4f12451083cd9
Merge: 5a13ae1 ec98fd4
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 30 00:40:42 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit 5a13ae19ddfaf641e2b2fc06d0448bee23622399
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 30 00:39:28 2010 +0100
mGAR v2: checkpkg, models cleanup
Also, a bugfix for the pkginfo and contents streams, which may be lists,
and sometimes can't be close()d.
commit ec98fd428ce63d3468c2560c88ef72a8c4bea5d3
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Mon Nov 29 23:36:27 2010 +0000
mGAR v2: checkpkg, storing srv4 file size
Also, a unit test for _ColectStats, which had to mock quite a lot of
calls to other classes functions. Could use some cleanup.
commit 143dea83827609a82ffd5a25f6ccb630e07448d6
Author: Maciej Blizi?\197?\132ski <blizinski at google.com>
Date: Mon Nov 29 10:54:55 2010 +0000
mGAR v2: checkpkg, a bugfix for package_stats.py
Sadly, this bit of code is not unit tested.
commit 00ab8a575a700fa6bf36b35ec888923ddfbc8ce1
Author: Maciej Blizi?\197?\132ski <blizinski at google.com>
Date: Mon Nov 29 10:49:00 2010 +0000
mGAR v2: checkpkg, not shelling out to isalist
Sheling out to 'isalist' to get the list of ISAs imposed a restriction,
that sparc packages could not be indexed on i386 boxes and vice-versa.
This change hardcodes the isalists and allows to index packages
independently of the architecture.
Data structures remain to hold the isalist as part of the pickled data
in the database, so each package has its own list of ISAs.
commit df4bd110d11465de23ccffe3822a32f2c3e85b8e
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 28 22:33:06 2010 +0000
mGAR v2: checkpkg, using highest pickling format
Pickling seems to be fairly memory-intensive with the data sets we have.
Setting the pickling format to the highest format in the hope it'll be
more efficient.
commit 62f0c6801aa2ce243e6955694a1d492f51cfefbb
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Nov 28 19:46:11 2010 +0100
mGAR v2: checkpkg, install/contents support
Accepting a bad line in install/contents.
There is a line that does not follow the spec.
E-mail thread:
http://lists.opencsw.org/pipermail/maintainers/2010-November/013277.html
commit 86eb898780ae8c639056de3164419364d000b6f0
Merge: dada99f 200303b
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Nov 28 17:39:29 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit 200303b2e127ca16a7de911102c62476dd98667f
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 28 16:37:52 2010 +0000
mGAR v2: checkpkg, importing system files into db
Imports files into the database, providing idempotence.
- MuteProgressBar helps with batch mode
commit dada99f84dda60934a9f49c6cc69be690fb43643
Merge: af4e86e 8e4ba00
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sat Nov 27 14:10:25 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit 8e4ba00b5d805a0ec6224993b9a3f47fb9dcce85
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sat Nov 27 13:08:18 2010 +0000
mGAR v2: pkgdb, implemented proper parsing and pickling of /var/sadm/install/contents.
commit af4e86e1e4ca051e68f0c487de6f60663a455f48
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Fri Nov 26 09:39:04 2010 +0100
mGAR v2: checkpkg, adding 'overriden' to the error tag class.
commit d296a3ac141fb7e11bc4b4b1be32552a64937c75
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Thu Nov 25 08:40:46 2010 +0000
mGAR v2: checkpkg, added a frontend to del-from-cat.
commit f34cff6c3892a5fd82da606ce26ca257b05f7209
Merge: 8e960bd d296a3a
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Thu Nov 25 09:40:46 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit 8e960bd354bc78d63754349fc5ec6cbfc1bc9d19
Merge: b29f6e0 33c16ec
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Thu Nov 25 01:38:04 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit b29f6e00dae17bed9a96a44bf2cc305d8b130b72
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Thu Nov 25 01:37:58 2010 +0100
mGAR v2: checkpkg, pkgdb: Logging which packages are added to a catalog.
commit 33c16ecba7e58ff5f528781e3d607f32f49a8826
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Thu Nov 25 00:37:33 2010 +0000
mGAR v2: Moved heavy pickled data to a separate table, from Srv4FileStats to Srv4FileStatsBlob.
commit 0df28fc119a0788152e712bc223cf23cd5c32f14
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Thu Nov 25 00:09:05 2010 +0000
mGAR v2: checkpkg, warning on filename vs pkginfo architecture declaration mismatch. Setting logging level to 'error' in unit tests.
commit e0ec4eb5b7430bc718f8923d75abc1bf09c48cff
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Thu Nov 25 00:06:10 2010 +0000
mGAR v2: checkpkg, allowing to add a i386 package to a sparc catalog if the filename declares 'all'.
commit 02d2403525d8e6bf19915fd1c19fd2d68445fede
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 24 18:08:04 2010 +0100
mGAR v2: Added filename_arch to the srv4_file_stats table, so I'll be able to quickly access both pkginfo and filename arch declarations.
commit 0c8cd3ec8e65a6d9cd3bf610a684ad5c41118d0f
Merge: 4d2b4e9 f94d332
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 24 13:49:57 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit 4d2b4e9ff9c33e6ac6b5622dae12fb73800d600c
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 24 13:49:49 2010 +0100
mGAR v2: checkpkg, more verbose logging when importing packages.
commit f94d332a7796e318b2e3f45849585dc1ec489094
Author: Maciej Blizi?\197?\132ski <blizinski at google.com>
Date: Wed Nov 24 12:49:11 2010 +0000
mGAR v2: checkpkg, not throwing an error when adding a package to a catalog and the package is already there.
commit 35d53ee05d143bd5bfb6b31e909ed0a08fa9c931
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Wed Nov 24 09:56:58 2010 +0000
mGAR v2: checkpkg, comments in the models.py file.
commit 1f77c9453c2c6c5a78871664089aa26bd5a84018
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Wed Nov 24 08:57:22 2010 +0000
mGAR v2: checkpkg, ImportPkg(): if a package was previously imported, don't replace.
commit 32fcf3009a281200b16b80a8bab0ef6cdf6d91e7
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Wed Nov 24 08:53:49 2010 +0000
mGAR v2: checkpkg, removing previously existing files in ImportPkg().
commit 9528a322a6fa26d749688a6685c50abe0a2e1415
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Wed Nov 24 08:33:05 2010 +0000
mGAR v2: checkpkg, support for latin1 encoded file names.
commit ec58dbb63a70413159ba64ffb8d289745b683420
Merge: 49f4aec 587bd3d
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Wed Nov 24 07:59:30 2010 +0000
Merge branch 'collisions' of ssh://login.opencsw.org/home/maciej/public_html/opencsw into collisions
commit 587bd3d3f7103ee631ccf31a59eaba92fb4d96a8
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 24 08:58:10 2010 +0100
mGAR v2: checkpkg, a test case for the latin1 encoded file name problem.
commit 3eb9d385731b98a31fe342e3e52cbe1ef18a72fc
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 24 00:22:29 2010 +0100
mGAR v2: checkpkg, adding missing operator module import to checkpkg_lib, checkpkg just started working.
commit 6749f47bba082f9d492d335fe24286982d18d3fb
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 24 00:07:31 2010 +0100
mGAR v2: checkpkg, memoization of GetPkgByPath().
commit 68beafc7fe8f69a87081a5bf5fc26285fe23f550
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 24 00:02:07 2010 +0100
mGAR v2: checkpkg, implemented new GetPkgByPath() in the Catalog class.
commit 49f4aecb95c1be8ce6e55a312026f55f60a4fbe1
Merge: 86a8abd 4c75fa9
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 22:29:33 2010 +0000
Merge branch 'collisions' of ssh://login.opencsw.org/home/maciej/public_html/opencsw into collisions
commit 4c75fa9b3b6b06b0bee69f64874f5e1487efba03
Merge: 937c34e 0204d2c
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 23 21:57:14 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit 937c34eb38b42097796cd00b8141634dae0c33af
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 23 21:53:00 2010 +0100
mGAR v2: checkpkg, throwing an error if an unregistered package is given for inclusion in a catalog.
commit 20b9a6b1cb04991a836054bad89676d80a4734b4
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 23 19:43:02 2010 +0100
mGAR v2: checkpkg, importing to the database can import some packages, but it fails with sqlobject.dberrors.OperationalError: Got a packet bigger than 'max_allowed_packet' bytes
commit 0204d2c1f6c2fadc88e1ddec31572cdd59ede8c7
Author: Maciej Blizi?\197?\132ski <blizinski at google.com>
Date: Tue Nov 23 14:24:32 2010 +0000
mGAR v2: Added sqlobject to the list of dependencies on Debian/Ubuntu.
commit 86a8abd52ecbc636d2add023718c65f6a7240090
Merge: d77ebb3 ac9d118
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 10:55:26 2010 +0000
Merge branch 'collisions' of ssh://login.opencsw.org/home/maciej/public_html/opencsw into collisions
commit d77ebb3886c1b3b74ed9cf3ffe707a2b4758f6a4
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 10:55:21 2010 +0000
mGAR v2: checkpkg, renamed 'bar' to 'pbar' to satisfy pylint.
commit ac9d118bd750ffe7d5a824a27dc0d0cdb2333eeb
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Tue Nov 23 11:53:54 2010 +0100
mGAR v2: checkpkg, moved db connection stuff to configuration.py, started working on checkpkg runs.
commit 4356b072d1546314e8cb9c3bd15174f507da44ea
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 10:19:35 2010 +0000
mGAR v2: checkpkg, removed unused imports from checkpkg.py.
commit a45c5fee794935dea3854defef99bb2b61bc473b
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 10:10:25 2010 +0000
mGAR v2: checkpkg, created checkpkg_lib.py with the intention to empty checkpkg.py.
commit 9931638ee6ed87bf1f8c2694dc36ae2b74693a6b
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 08:08:53 2010 +0000
mGAR v2: checkpkg, implemented GetInstalledPackages in the Catalog class.
commit 9aa73511ba7e67574085ba623d3eae0419a88b18
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Tue Nov 23 07:06:37 2010 +0000
mGAR v2: checkpkg: splitting off LddEmulator and SystemPkgmap to separate files.
commit aa650b652a9d4c24c8ab3e8c6f63a9d0a9afa380
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Mon Nov 22 23:23:25 2010 +0000
mGAR v2: checkpkg, checking package architecture before inserting into a catalog.
commit ef28624d237bc2949f491bdc2e89913c60da83ed
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Mon Nov 22 22:00:18 2010 +0000
mGAR v2: checkpkg, a registering an existing package works now.
commit 913547c974bde31d97c91f400877a7ca5b767d4e
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Mon Nov 22 21:50:10 2010 +0000
mGAR v2: checkpkg, removing the super() call in SystemPkgmapMixin
commit 9655c817cdb56ef27de103b2bfd68428087717f2
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Mon Nov 22 21:06:26 2010 +0100
mGAR v2: Removing the database classmethods, and making sure that we're returning files that belong to packages that are in catalogs (although the check is slightly redundant).
commit 25dbd71a94f7e7db4ccab1d820ba305b37ce0066
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Mon Nov 22 11:14:44 2010 +0100
mGAR v2: checkpkg, more db refactoring, tests are passing, pkgdb needs to be fixed.
commit 5c44f40f21b6155216eaff5923cc97d23c05be14
Merge: 426f6c8 4e03ab2
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Nov 21 19:36:48 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit 4e03ab231ebad07bfc58c9cfc016d180ef4d865e
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 18:25:00 2010 +0000
mGAR v2: checkpkg, removing a package from a catalog works now.
commit 1fcb2db5e243015001252ee96a11b045d52f4320
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 15:46:49 2010 +0000
mGAR v2: checkpkg, not allowing two packages with the same pkgname in one catalog.
commit 426f6c862262092c19dc422400b8d98a92606b23
Merge: 90098b5 1767ae5
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Nov 21 16:23:30 2010 +0100
Merge branch 'collisions' of file:///home/maciej/public_html/opencsw into collisions
commit 90098b5fac1e4c4ddbe1356d4172c9c346864289
Merge: 5d940e5 f3ae454
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Nov 21 16:20:47 2010 +0100
Merge branch 'master' of file:///home/maciej/public_html/opencsw into collisions
commit 0d9de2311b68856db552ecc452aaa0c12c00c900
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 14:46:37 2010 +0000
mGAR v2: checkpkg, checking that putting one package two times in a catalog raises an error.
commit 1767ae5d3e6e6c8ea37ad91ed5b05651b148f929
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 14:34:33 2010 +0000
mGAR v2: checkpkg, small unit tests cleanup.
commit 787874a57a0c3568f7991750b958df63689ad533
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 14:26:27 2010 +0000
mGAR v2: checkpkg, querying the database for files works.
commit 728670ac4ecc7dccc8fe5d33e0eff5edcdae3019
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Sun Nov 21 12:13:25 2010 +0000
mGAR v2: using #!/usr/bin/env for checkpkg_test.py
commit 5d940e5a98876f51a0e095d80af42075371466b0
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Nov 21 13:08:32 2010 +0100
mGAR v2: checkpkg, start of a test for retrieving pkgnames for a basename. The netzwerk lags today, moving development home.
commit 38c8c3a6db74d373859abef28ac1de400ac99bef
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sun Nov 21 12:44:48 2010 +0100
mGAR v2: checkpkg, registering a package with a catalog.
commit 77de13f7cdcd6af28d38735bd64518217c61b6e7
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sat Nov 20 19:00:28 2010 +0100
mGAR v2: checkpkg, a test for inserting stats into the db.
commit 1adc64e53ac87fd88f57b707c8e4b4b68b7487b7
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sat Nov 20 18:10:58 2010 +0100
mGAR v2: checkpkg, started a database related unit tests.
commit 73c3fb73e0989bd8cf1fee46ebe5ac96e17ca208
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Sat Nov 20 11:07:37 2010 +0100
mGAR v2: checkpkg, removing the remaining korn shell code.
commit 24acc21103eb336d82bd5d7198144bea633bb550
Merge: 3e34c07 46dd5e8
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Fri Nov 19 00:54:55 2010 +0100
Merge branch 'master' into collisions
commit 3e34c07a5187e9babcd85b75d4397b18d646d7a5
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Thu Nov 18 12:32:33 2010 +0100
mGAR v2: checkpkg, insignificant change to models.py
commit 286195b81d1507f8094676efbb0e209be431e543
Merge: 4b8a652 953d348
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 17 23:47:29 2010 +0100
Merge branch 'master' into collisions
commit 953d348e07b9fa5b58b00c2af67c97946dee38ca
Author: Maciej Blizinski <maciej at opencsw.org>
Date: Wed Nov 17 23:40:35 2010 +0100
mGAR v2: pkgdb, adding soname to the binaries_dump_info section.
commit de9fcb7042b1466499c3d56aceed1df1de9b9e79
Author: Maciej Blizi?\197?\132ski <maciej at opencsw.org>
Date: Wed Nov 3 08:43:14 2010 +0000
Revert "mGAR v2: checkpkg, README, the list of dependencies to run the unit tests on Ubuntu."
This reverts commit 46b7b6bd95a737c1f65a8e1d60bf07e1cd3301de.
commit 9b9d1a7d46cbefed0316f08939b41a0b3775880c
Author: Maciej Blizinski <maciej at quince.home.blizinski.pl>
Date: Wed Nov 3 08:40:03 2010 +0000
mGAR v2: checkpkg, README, the list of dependencies to run the unit tests on Ubuntu.
Modified Paths:
--------------
csw/mgar/gar/v2/bin/checkpkg
csw/mgar/gar/v2/bin/mkpackage
csw/mgar/gar/v2/gar.pkg.mk
csw/mgar/gar/v2/lib/python/README
csw/mgar/gar/v2/lib/python/catalog.py
csw/mgar/gar/v2/lib/python/checkpkg.py
csw/mgar/gar/v2/lib/python/checkpkg_test.py
csw/mgar/gar/v2/lib/python/configuration.py
csw/mgar/gar/v2/lib/python/database.py
csw/mgar/gar/v2/lib/python/dependency_checks.py
csw/mgar/gar/v2/lib/python/dependency_checks_test.py
csw/mgar/gar/v2/lib/python/inspective_package.py
csw/mgar/gar/v2/lib/python/models.py
csw/mgar/gar/v2/lib/python/opencsw.py
csw/mgar/gar/v2/lib/python/package.py
csw/mgar/gar/v2/lib/python/package_checks.py
csw/mgar/gar/v2/lib/python/package_checks_test.py
csw/mgar/gar/v2/lib/python/package_stats.py
csw/mgar/gar/v2/lib/python/package_stats_test.py
csw/mgar/gar/v2/lib/python/pkgdb.py
csw/mgar/gar/v2/lib/python/sharedlib_utils.py
csw/mgar/gar/v2/lib/python/sharedlib_utils_test.py
csw/mgar/gar/v2/lib/python/tag.py
csw/mgar/gar/v2/lib/python/tag_test.py
csw/mgar/gar/v2/lib/python/testdata/neon_stats.py
csw/mgar/gar/v2/lib/python/testdata/tree_stats.py
csw/mgar/gar/v2/tests/run_tests.py
Added Paths:
-----------
csw/mgar/gar/v2/lib/python/checkpkg2.py
csw/mgar/gar/v2/lib/python/checkpkg_lib.py
csw/mgar/gar/v2/lib/python/checkpkg_lib_test.py
csw/mgar/gar/v2/lib/python/common_constants.py
csw/mgar/gar/v2/lib/python/ldd_emul.py
csw/mgar/gar/v2/lib/python/ldd_emul_test.py
csw/mgar/gar/v2/lib/python/models_test.py
csw/mgar/gar/v2/lib/python/mute_progressbar.py
csw/mgar/gar/v2/lib/python/pkgdb_test.py
csw/mgar/gar/v2/lib/python/shell.py
csw/mgar/gar/v2/lib/python/system_pkgmap.py
csw/mgar/gar/v2/lib/python/system_pkgmap_test.py
csw/mgar/gar/v2/lib/python/test_base.py
Removed Paths:
-------------
csw/mgar/gar/v2/bin/analyze_module_results.py
csw/mgar/gar/v2/bin/checkpkg_collect_stats.py
csw/mgar/gar/v2/bin/checkpkg_run_modules.py
Property Changed:
----------------
csw/mgar/gar/v2/bin/checkpkg
csw/mgar/gar/v2/lib/python/package_stats_test.py
Deleted: csw/mgar/gar/v2/bin/analyze_module_results.py
===================================================================
--- csw/mgar/gar/v2/bin/analyze_module_results.py 2010-12-10 13:48:16 UTC (rev 11878)
+++ csw/mgar/gar/v2/bin/analyze_module_results.py 2010-12-10 16:48:58 UTC (rev 11879)
@@ -1,73 +0,0 @@
-#!/opt/csw/bin/python2.6
-# $Id$
-
-import itertools
-import operator
-import optparse
-import os
-import pprint
-import progressbar
-import sys
-import textwrap
-
-# The following bit of code sets the correct path to Python libraries
-# distributed with GAR.
-path_list = [os.path.dirname(__file__),
- "..", "lib", "python"]
-sys.path.append(os.path.join(*path_list))
-import checkpkg
-import overrides
-import package_stats
-
-BEFORE_OVERRIDES = """If any of the reported errors were false positives, you
-can override them pasting the lines below to the GAR recipe."""
-
-AFTER_OVERRIDES = """Please note that checkpkg isn't suggesting you should
-simply add these overrides do the Makefile. It only informs what the overrides
-could look like. You need to understand what are the reported issues about and
-use your best judgement to decide whether to fix the underlying problems or
-override them. For more information, scroll up and read the detailed
-messages."""
-
-UNAPPLIED_OVERRIDES = """WARNING: Some overrides did not match any errors.
-They can be removed, as they don't take any effect anyway. If you're getting
-errors at the same time, maybe you didn't specify the overrides correctly."""
-
-def main():
- parser = optparse.OptionParser()
- parser.add_option("-c", "--catalog_file", dest="catalog",
- help="Optional catalog file")
- parser.add_option("-q", "--quiet", dest="quiet",
- default=False, action="store_true",
- help=("Display less messages"))
- options, args = parser.parse_args()
- filenames = args
-
- # This might be bottleneck. Perhaps a list of md5 sums can be given to this
- # script instead.
-
- # It might be a good idea to store the error tags in the database and
- # eliminate the need to access the directory with the error tag files.
-
- pkgstats = package_stats.StatsListFromCatalog(filenames, options.catalog)
- overrides_list = [pkg.GetSavedOverrides() for pkg in pkgstats]
- override_list = reduce(operator.add, overrides_list)
- error_tags = reduce(operator.add, [stat.GetSavedErrorTags() for stat in pkgstats])
- (tags_after_overrides,
- unapplied_overrides) = overrides.ApplyOverrides(error_tags, override_list)
- if not options.quiet:
- if tags_after_overrides:
- print textwrap.fill(BEFORE_OVERRIDES, 80)
- for checkpkg_tag in tags_after_overrides:
- print checkpkg_tag.ToGarSyntax()
- print textwrap.fill(AFTER_OVERRIDES, 80)
- if unapplied_overrides:
- print textwrap.fill(UNAPPLIED_OVERRIDES, 80)
- for override in unapplied_overrides:
- print "* Unused %s" % override
- exit_code = bool(tags_after_overrides)
- sys.exit(exit_code)
-
-
-if __name__ == '__main__':
- main()
Modified: csw/mgar/gar/v2/bin/checkpkg
===================================================================
--- csw/mgar/gar/v2/bin/checkpkg 2010-12-10 13:48:16 UTC (rev 11878)
+++ csw/mgar/gar/v2/bin/checkpkg 2010-12-10 16:48:58 UTC (rev 11879)
@@ -1,322 +1 @@
-#!/bin/ksh -p
-#
-# $Id$
-#
-# checkpkg 1.51
-#
-# diff to 1.46a
-# - check multiple package files
-# - checkpkg.d plugin support
-# - getopts support for command line options
-# - colors
-# - modular architecture + unit tests
-# - reliable shared library checking
-#
-# This script examines a package that has been put together
-# for submittal to the CSW archive at opencsw.org
-#
-# It examines it for compliance with the packaging standards at
-# http://www.opencsw.org/standards/
-# It DOES NOT CATCH EVERYTHING. However, the package will be
-# tested with this script before acceptance, so you may as well
-# save yourself some time, and run the script yourself!
-#
-# Be sure to occasionally do a "pkg-get update cswutils" so that
-# you know you are tracking the most current version.
-#
-# TODO:
-# - add message handlig to the CheckInterface class.
-#
-
-PATH=$PATH:/usr/sbin
-readonly NAME_MAX_LENGTH=${NAME_MAX_LENGTH:-20}
-
-command_basename=`basename $0`
-command_basedir="${0%/${command_basename}}"
-libshdir="${command_basedir}/../lib/sh"
-readonly command_basename command_basedir libshdir
-. "${libshdir}/libcheckpkg.sh"
-
-LOCAL_ARCH=`uname -p`
-CHECKPKG_TMPDIR=${CHECKPKG_TMPDIR:-/var/tmp}
-readonly CHECKPKG_TMPDIR
-
-# Colors only when running interactively
-if [[ -t 1 ]]; then
- GREEN="\\033[0;32;40m"
- RED="\\033[1;31;40m"
- BOLD="\\033[1m"
- COLOR_RESET="\\033[00m"
-else
- GREEN=""
- RED=""
- BOLD=""
- COLOR_RESET=""
-fi
-readonly GREEN RED BOLD COLOR_RESET
-
-readonly selfpath="$0"
-readonly selfargs="$@"
-
-cleanup(){
- if [[ -d "$EXTRACTDIR" ]] ; then
- rm -rf $EXTRACTDIR
- fi
- cleantmparchives
-}
-
-cleantmparchives() {
- for TMPARCHIVE in $tmparchives; do
- if [[ "$TMPARCHIVE" != "" ]]; then
- [ -f "$TMPARCHIVE" ] && rm $TMPARCHIVE
- fi
- done
-}
-
-cleanupset(){
- if [ "`echo $SETINF*`" != "$SETINF*" ]; then
- rm $SETINF*
- fi
-}
-
-# Print error message, and quit program.
-errmsg(){
- print ERROR: $* >/dev/fd/2
- cleanup
- cleanupset
- print "To run checkpkg in the debug mode, add the '-d' flag, for example:"
- # selfargs can be very, very long. Find a way to truncate it.
- # print "${selfpath} -d ${selfargs}"
- print "After you modify any overrides, you need to do gmake remerge repackage"
- print "or gmake platforms-remerge platforms-repackage."
- exit 1
-}
-
-debugmsg() {
- if [[ "${DEBUG}" != "" ]]; then
- print "DEBUG: $*" > /dev/fd/2
- fi
-}
-
-# TODO: Options to add:
-# - Use an pre-cached (from a catalog file?) list of md5 sums
-# - Don't use the data from /var/sadm/install/contents
-display_help=0
-SKIP_STATS_COLLECTION=0
-MD5_SUMS_CATALOG_FILE=""
-INSTALL_CONTENTS_FILES="/var/sadm/install/contents"
-ANALYZE=1
-PROFILE=0
-QUIET=0
-
-while getopts hsdNM:o:c:Apq opt; do
- case "${opt}" in
- c)
- INSTALL_CONTENTS_FILES="${INSTALL_CONTENTS_FILES} ${OPTARG}"
- ;;
- d)
- DEBUG=1
- ;;
- h)
- display_help=1
- ;;
- N)
- SKIP_STATS_COLLECTION=1
- ;;
- M)
- MD5_SUMS_CATALOG_FILE="${OPTARG}"
- ;;
- A)
- ANALYZE=0
- ;;
- p)
- PROFILE=1
- ;;
- q) QUIET=1
- ;;
- *)
- echo "Unknown option '${opt}'"
- ;;
- esac
-done
-shift $(( $OPTIND -1 ))
-
-readonly INSTALL_CONTENTS_FILES
-readonly MD5_SUMS_CATALOG_FILE
-readonly SKIP_STATS_COLLECTION
-readonly ANALYZE
-readonly PROFILE
-readonly QUIET
-
-if [[ "${display_help}" -eq 1 ]] ; then
- print 'Usage: checkpkg [options] pkg1 [pkg2 ....]'
- print 'Options:'
- print ' -c <file> use an additional install/contents file'
- print ' -d display debug messages'
- print ' -N skip statistics collection'
- print ' -M <file> use package md5sums from a catalog file'
- print ' -A Do not analyze the results.'
- print ' -p Enable profiling'
- print ' -q Display less messages'
- print ''
- print 'Error tags are saved to the sqlite database.'
- exit 0
-fi
-
-# a unique filename for the list of package deps and libs we see in a 'set'
-SETINF=$CHECKPKG_TMPDIR/checkpkg.$$.`date +%Y%m%d%H%M%S`
-SETLIBS=$SETINF.libs
-SETDEPS=$SETINF.deps
-pkgnames=""
-tmparchives=""
-
-EXTRACTDIR=$CHECKPKG_TMPDIR/dissect.$$
-
-if [ -d $EXTRACTDIR ] ; then
- errmsg ERROR: $EXTRACTDIR already exists
-fi
-
-for f in "$@"
-do
-
- if [[ ! -f $f ]] ; then
- errmsg ERROR: $f does not exist
- fi
-
-
-[ -d ${EXTRACTDIR} ] || mkdir ${EXTRACTDIR}
-
-########################################
-# Check for some common errors
-#########################################
-
-# TODO: To be ported.
-#
-# # find all executables and dynamic libs,and list their filenames.
-# if [[ "$basedir" != "" ]] ; then
-# print
-# if [[ -f $EXTRACTDIR/elflist ]] ; then
-# print "Checking relocation ability..."
-# xargs strings < $EXTRACTDIR/elflist| grep /opt/csw
-# if [[ $? -eq 0 ]] ; then
-# errmsg package build as relocatable, but binaries have hardcoded /opt/csw paths in them
-# else
-# print trivial check passed
-# fi
-# else
-# echo No relocation check done for non-binary relocatable package.
-# fi
-# fi
-
-tmparchives="$tmparchives $TMPARCHIVE"
-done
-
-# Plugin section. This is here for support for other programming languages
-# than Python. As of 2010-03-16 there are no checks in there. If this keeps
-# empty, if no checks in other languages get written, it could be removed.
-#
-# Plugins should live in checkpkg.d subdirectory in the same directory in which
-# checkpkg is. Each plugin file name should be an executable and begin with
-# "checkpkg-".
-
-test_suite_ok=1
-checkpkg_module_dir="${command_basedir}/../lib/checkpkg.d"
-checkpkg_module_tag="checkpkg-"
-checkpkg_stats_basedir="${HOME}/.checkpkg/stats"
-
-# Cleaning up old *.pyc files which can cause grief. This is because of the
-# move of Python libraries.
-for pyc_file in ${checkpkg_module_dir}/opencsw.pyc \
- ${checkpkg_module_dir}/checkpkg.pyc; do
- if [ -f "${pyc_file}" ]; then
- echo "Removing old pyc file: '${pyc_file}'"
- rm "${pyc_file}"
- fi
-done
-
-if [[ "${DEBUG}" != "" ]]; then
- extra_options="--debug"
-fi
-if [[ "${PROFILE}" -eq 1 ]]; then
- extra_options="${extra_options} --profile"
-fi
-if [[ "${QUIET}" -eq 1 ]]; then
- quiet_options="--quiet"
-else
- quiet_options=""
-fi
-
-if [[ -n "${MD5_SUMS_CATALOG_FILE}" ]]; then
- catalog_options="--catalog=${MD5_SUMS_CATALOG_FILE}"
-else
- catalog_options=""
-fi
-
-# /var/sadm/install/contents cache update
-# TODO: Either remove this section or stop the stats collection phase from
-# updating the cache.
-${command_basedir}/update_contents_cache.py ${extra_options}
-if [[ $? -ne 0 ]]; then
- errmsg "Updating the contents cache has failed."
-fi
-if [[ "${SKIP_STATS_COLLECTION}" -eq 0 ]]; then
- # Collects package stats to be analyzed later
- ${command_basedir}/checkpkg_collect_stats.py \
- ${catalog_options} \
- ${extra_options} \
- "$@"
- if [[ "$?" -ne 0 ]]; then
- errmsg "Stats collection phase has failed."
- fi
-fi
-
-# TODO: A performance problem. The following line means that the md5sums are
-# calculated once more.
-if [ "${MD5_SUMS_CATALOG_FILE}" ]; then
- debugmsg "Reading md5sums from ${MD5_SUMS_CATALOG_FILE}"
- md5sums=`cat "${MD5_SUMS_CATALOG_FILE}" \
- | awk '{print $5}' \
- | ggrep -E '[0-9abcdef]{32}'`
-else
- debugmsg "Calculating md5 sums of all the package files."
- md5sums=`gmd5sum "$@" | awk '{print $1}'`
-fi
-debugmsg "All md5 sums: ${md5sums}"
-
-# Running the checks.
-${command_basedir}/checkpkg_run_modules.py \
- ${extra_options} \
- -b "${checkpkg_stats_basedir}" \
- ${quiet_options} \
- ${md5sums}
-if [[ "$?" -ne 0 ]]; then
- print "There was a problem analyzing package stats."
- test_suite_ok=0
-fi
-
-if [[ ${test_suite_ok} -ne 1 ]]; then
- errmsg "One or more tests have finished with an error."
-fi
-
-if [[ "${ANALYZE}" -eq 1 ]]; then
-# Collecting errors and applying the overrides.
-# This has to use the original files.
- ${command_basedir}/analyze_module_results.py \
- ${catalog_options} \
- ${quiet_options} \
- "$@"
- if [[ "$?" -ne 0 ]]; then
- errmsg "${RED}Checkpkg has reported errors.${COLOR_RESET}"
- else
- print "${GREEN}Checkpkg reports no errors.${COLOR_RESET}"
- fi
-else
- echo "Skipping result analysis."
-fi
-
-print ""
-
-# Cleaning up after all packages
-cleanup
-
-cleanupset
+link ../lib/python/checkpkg2.py
\ No newline at end of file
Property changes on: csw/mgar/gar/v2/bin/checkpkg
___________________________________________________________________
Deleted: svn:executable
- *
Added: svn:special
+ *
Deleted: csw/mgar/gar/v2/bin/checkpkg_collect_stats.py
===================================================================
--- csw/mgar/gar/v2/bin/checkpkg_collect_stats.py 2010-12-10 13:48:16 UTC (rev 11878)
+++ csw/mgar/gar/v2/bin/checkpkg_collect_stats.py 2010-12-10 16:48:58 UTC (rev 11879)
@@ -1,69 +0,0 @@
-#!/opt/csw/bin/python2.6
-#
-# $Id$
-#
-# Collects statistics about a package and saves to a directory, for later use
-# by checkpkg modules.
-
-import itertools
-import logging
-import optparse
-import os
-import os.path
-import subprocess
-import sys
-import progressbar
-
-# The following bit of code sets the correct path to Python libraries
-# distributed with GAR.
-path_list = [os.path.dirname(__file__),
- "..", "lib", "python"]
-sys.path.append(os.path.join(*path_list))
-import checkpkg
-import opencsw
-import package_stats
-
-def main():
- parser = optparse.OptionParser()
- parser.add_option("-d", "--debug", dest="debug",
- default=False, action="store_true",
- help="Turn on debugging messages")
- parser.add_option("-c", "--catalog", dest="catalog",
- help="Catalog file")
- parser.add_option("-p", "--profile", dest="profile",
- default=False, action="store_true",
- help="A disabled option")
- options, args = parser.parse_args()
- if options.debug:
- logging.basicConfig(level=logging.DEBUG)
- else:
- logging.basicConfig(level=logging.INFO)
- logging.debug("Collecting statistics about given package files.")
- args_display = args
- if len(args_display) > 5:
- args_display = args_display[:5] + ["...more..."]
- file_list = args
- logging.debug("Processing: %s, please be patient", args_display)
- stats_list = package_stats.StatsListFromCatalog(
- file_list, options.catalog, options.debug)
- # Reversing the item order in the list, so that the pop() method can be used
- # to get packages, and the order of processing still matches the one in the
- # catalog file.
- stats_list.reverse()
- total_packages = len(stats_list)
- counter = itertools.count(1)
- logging.info("Juicing the srv4 package stream files...")
- bar = progressbar.ProgressBar()
- bar.maxval = total_packages
- bar.start()
- while stats_list:
- # This way objects will get garbage collected as soon as they are removed
- # from the list by pop(). The destructor (__del__()) of the srv4 class
- # removes the temporary directory from the disk. This allows to process
- # the whole catalog.
- stats_list.pop().CollectStats()
- bar.update(counter.next())
- bar.finish()
-
-if __name__ == '__main__':
- main()
Deleted: csw/mgar/gar/v2/bin/checkpkg_run_modules.py
===================================================================
--- csw/mgar/gar/v2/bin/checkpkg_run_modules.py 2010-12-10 13:48:16 UTC (rev 11878)
+++ csw/mgar/gar/v2/bin/checkpkg_run_modules.py 2010-12-10 16:48:58 UTC (rev 11879)
@@ -1,56 +0,0 @@
-#!/opt/csw/bin/python2.6
-# $Id$
-
-"""This script runs all the checks written in Python."""
-
-import datetime
-import logging
-import os
-import os.path
-import sys
-import re
-import cProfile
-
-CHECKPKG_MODULE_NAME = "Second checkpkg API version"
-
-# The following bit of code sets the correct path to Python libraries
-# distributed with GAR.
-path_list = [os.path.dirname(__file__),
- "..", "lib", "python"]
-sys.path.append(os.path.join(*path_list))
-import checkpkg
-import opencsw
-
-
-def main():
- options, args = checkpkg.GetOptions()
- if options.debug:
- logging.basicConfig(level=logging.DEBUG)
- else:
- logging.basicConfig(level=logging.INFO)
- md5sums = args
- # CheckpkgManager2 class abstracts away things such as the collection of
- # results.
- check_manager = checkpkg.CheckpkgManager2(CHECKPKG_MODULE_NAME,
- options.stats_basedir,
- md5sums,
- options.debug)
- # Running the checks, reporting and exiting.
- exit_code, screen_report, tags_report = check_manager.Run()
- screen_report = unicode(screen_report)
- if not options.quiet and screen_report:
- sys.stdout.write(screen_report)
- else:
- logging.debug("No screen report.")
- sys.exit(exit_code)
-
-
-if __name__ == '__main__':
- if "--profile" in sys.argv:
- t_str = datetime.datetime.now().strftime("%Y-%m-%d-%H-%M")
- home = os.environ["HOME"]
- cprof_file_name = os.path.join(
- home, ".checkpkg", "run-modules-%s.cprof" % t_str)
- cProfile.run("main()", sort=1, filename=cprof_file_name)
- else:
- main()
Modified: csw/mgar/gar/v2/bin/mkpackage
===================================================================
--- csw/mgar/gar/v2/bin/mkpackage 2010-12-10 13:48:16 UTC (rev 11878)
+++ csw/mgar/gar/v2/bin/mkpackage 2010-12-10 16:48:58 UTC (rev 11879)
@@ -25,6 +25,9 @@
# Tool Version/Revision Information
$TOOLVERSION = "1.4";
($REVISION) = q/$Revision$/ =~ /(\d+)/;
+# This shows a warning:
+# "Use of uninitialized value $REVISION in sprintf at
+# /home/maciej/src/opencsw/pkg/nspr/trunk/gar/bin/mkpackage line 31."
$VERSION = sprintf '%s (r%d)', $TOOLVERSION, $REVISION;
# Discover network support
Modified: csw/mgar/gar/v2/gar.pkg.mk
===================================================================
--- csw/mgar/gar/v2/gar.pkg.mk 2010-12-10 13:48:16 UTC (rev 11878)
+++ csw/mgar/gar/v2/gar.pkg.mk 2010-12-10 16:48:58 UTC (rev 11879)
@@ -52,6 +52,7 @@
GARPKG_v1 = CSWgar-v1
GARPKG_v2 = CSWgar-v2
RUNTIME_DEP_PKGS_$(SRCPACKAGE) ?= $(or $(GARPKG_$(GARSYSTEMVERSION)),$(error GAR version $(GARSYSTEMVERSION) unknown))
+CATALOG_RELEASE ?= current
_PKG_SPECS = $(filter-out $(NOPACKAGE),$(SPKG_SPECS))
@@ -842,7 +843,11 @@
# pkgcheck - check if the package is compliant
#
pkgcheck: $(foreach SPEC,$(_PKG_SPECS),package-$(SPEC))
- $(_DBG)( LC_ALL=C $(GARBIN)/checkpkg $(foreach SPEC,$(_PKG_SPECS),$(SPKG_EXPORT)/`$(call _PKG_ENV,$(SPEC)) mkpackage --tmpdir $(SPKG_TMPDIR) -qs $(WORKDIR)/$(SPEC).gspec -D pkgfile`.gz ) || exit 2;)
+ $(_DBG)( LC_ALL=C $(GARBIN)/checkpkg \
+ --architecture "$(GARCH)" \
+ --os-releases "$(SPKG_OSNAME)" \
+ --catalog-release "$(CATALOG_RELEASE)" \
+ $(foreach SPEC,$(_PKG_SPECS),$(SPKG_EXPORT)/`$(call _PKG_ENV,$(SPEC)) mkpackage --tmpdir $(SPKG_TMPDIR) -qs $(WORKDIR)/$(SPEC).gspec -D pkgfile`.gz ) || exit 2;)
@$(MAKECOOKIE)
pkgcheck-p:
Modified: csw/mgar/gar/v2/lib/python/README
===================================================================
--- csw/mgar/gar/v2/lib/python/README 2010-12-10 13:48:16 UTC (rev 11878)
+++ csw/mgar/gar/v2/lib/python/README 2010-12-10 16:48:58 UTC (rev 11879)
@@ -21,9 +21,11 @@
tests as means to run various bits of code. Here's the dependency list
for Ubuntu.
- python-cheetah
- python-hachoir-parser
- python-magic
- python-mox
- python-progressbar
+sudo aptitude install \
+ python-cheetah \
+ python-hachoir-parser \
+ python-magic \
+ python-mox \
+ python-progressbar \
+ python-sqlobject \
python-yaml
Modified: csw/mgar/gar/v2/lib/python/catalog.py
===================================================================
--- csw/mgar/gar/v2/lib/python/catalog.py 2010-12-10 13:48:16 UTC (rev 11878)
+++ csw/mgar/gar/v2/lib/python/catalog.py 2010-12-10 16:48:58 UTC (rev 11879)
@@ -123,6 +123,7 @@
def _GetCatalogData(self, fd):
catalog_data = []
for line in fd:
+ if line.startswith("#"): continue
try:
parsed = self._ParseCatalogLine(line)
catalog_data.append(parsed)
Modified: csw/mgar/gar/v2/lib/python/checkpkg.py
===================================================================
--- csw/mgar/gar/v2/lib/python/checkpkg.py 2010-12-10 13:48:16 UTC (rev 11878)
+++ csw/mgar/gar/v2/lib/python/checkpkg.py 2010-12-10 16:48:58 UTC (rev 11879)
@@ -3,44 +3,22 @@
# This is the checkpkg library, common for all checkpkg tests written in
# Python.
-import copy
-import cPickle
-import errno
import itertools
import logging
-import operator
import optparse
-import os
import os.path
import re
import pprint
import progressbar
-import socket
-import sqlite3
import sqlobject
-import time
-from sqlobject import sqlbuilder
import subprocess
-import textwrap
-from Cheetah import Template
import database
-import package
import inspective_package
-import package_checks
-import package_stats
import models as m
-import configuration as c
-import tag
+import common_constants
-DEBUG_BREAK_PKGMAP_AFTER = False
-SYSTEM_PKGMAP = "/var/sadm/install/contents"
-NEEDED_SONAMES = "needed sonames"
-RUNPATH = "runpath"
-SONAME = "soname"
-CONFIG_MTIME = "mtime"
-CONFIG_DB_SCHEMA = "db_schema_version"
DO_NOT_REPORT_SURPLUS = set([u"CSWcommon", u"CSWcswclassutils", u"CSWisaexec"])
DO_NOT_REPORT_MISSING = set([])
DO_NOT_REPORT_MISSING_RE = [r"\*?SUNW.*"]
@@ -48,17 +26,11 @@
PSTAMP_RE = r"(?P<username>\w+)@(?P<hostname>[\w\.-]+)-(?P<timestamp>\d+)"
DESCRIPTION_RE = r"^([\S]+) - (.*)$"
BAD_CONTENT_REGEXES = (
- # Slightly obfuscating these by using the default concatenation of
- # strings.
+ # Slightly obfuscating these by using concatenation of strings.
r'/export' r'/medusa',
r'/opt' r'/build',
)
-SYSTEM_SYMLINKS = (
- ("/opt/csw/bdb4", ["/opt/csw/bdb42"]),
- ("/64", ["/amd64", "/sparcv9"]),
- ("/opt/csw/lib/i386", ["/opt/csw/lib"]),
-)
INSTALL_CONTENTS_AVG_LINE_LENGTH = 102.09710677919261
SYS_DEFAULT_RUNPATH = [
"/usr/lib/$ISALIST",
@@ -67,8 +39,7 @@
"/lib",
]
-CONTENT_PKG_RE = r"^\*?(CSW|SUNW)[0-9a-zA-Z\-]?[0-9a-z\-]+$"
-MD5_RE = r"^[0123456789abcdef]{32}$"
+MD5_RE = re.compile(r"^[0123456789abcdef]{32}$")
REPORT_TMPL = u"""#if $missing_deps or $surplus_deps or $orphan_sonames
Dependency issues of $pkgname:
@@ -90,57 +61,6 @@
#end if
"""
-SCREEN_ERROR_REPORT_TMPL = u"""#if $errors
-#if $debug
-ERROR: One or more errors have been found by $name.
-#end if
-#for $pkgname in $errors
-$pkgname:
-#for $error in $errors[$pkgname]
-#if $debug
- $repr($error)
-#elif $error.msg
-$textwrap.fill($error.msg, 78, initial_indent="# ", subsequent_indent="# ")
-# -> $repr($error)
-
-#end if
-#end for
-#end for
-#else
-#if $debug
-OK: $repr($name) module found no problems.
-#end if
-#end if
-#if $messages
-#for $msg in $messages
-$textwrap.fill($msg, 78, initial_indent=" * ", subsequent_indent=" ")
-#end for
-#end if
-#if $gar_lines
-
-# Checkpkg suggests adding the following lines to the GAR recipe:
-# This is a summary; see above for details.
-#for $line in $gar_lines
-$line
-#end for
-#end if
-"""
-
-# http://www.cheetahtemplate.org/docs/users_guide_html_multipage/language.directives.closures.html
-TAG_REPORT_TMPL = u"""#if $errors
-# Tags reported by $name module
-#for $pkgname in $errors
-#for $tag in $errors[$pkgname]
-#if $tag.msg
-$textwrap.fill($tag.msg, 70, initial_indent="# ", subsequent_indent="# ")
-#end if
-$pkgname: ${tag.tag_name}#if $tag.tag_info# $tag.tag_info#end if#
-#end for
-#end for
-#end if
-"""
-
-
class Error(Exception):
pass
@@ -157,11 +77,12 @@
pass
+class SetupError(Error):
+ pass
+
+
def GetOptions():
parser = optparse.OptionParser()
- parser.add_option("-b", "--stats-basedir", dest="stats_basedir",
- help=("The base directory with package statistics "
- "in yaml format, e.g. ~/.checkpkg/stats"))
parser.add_option("-d", "--debug", dest="debug",
default=False, action="store_true",
help="Turn on debugging messages")
@@ -172,8 +93,6 @@
default=False, action="store_true",
help=("Print less messages"))
(options, args) = parser.parse_args()
- if not options.stats_basedir:
- raise ConfigurationError("ERROR: the -b option is missing.")
# Using set() to make the arguments unique.
return options, set(args)
@@ -195,859 +114,9 @@
return m.group("username") if m else None
-class SystemPkgmap(database.DatabaseClient):
- """A class to hold and manipulate the /var/sadm/install/contents file."""
-
- STOP_PKGS = ["SUNWbcp", "SUNWowbcp", "SUNWucb"]
-
- def __init__(self, system_pkgmap_files=None, debug=False):
- """There is no need to re-parse it each time.
-
- Read it slowly the first time and cache it for later."""
- super(SystemPkgmap, self).__init__(debug=debug)
- self.cache = {}
- self.pkgs_by_path_cache = {}
- self.file_mtime = None
- self.cache_mtime = None
- self.initialized = False
- if not system_pkgmap_files:
- self.system_pkgmap_files = [SYSTEM_PKGMAP]
- else:
- self.system_pkgmap_files = system_pkgmap_files
- self.csw_pkg_re = re.compile(CONTENT_PKG_RE)
- self.digits_re = re.compile(r"^[0-9]+$")
-
- def _LazyInitializeDatabase(self):
- if not self.initialized:
- self.InitializeDatabase()
-
- def InitializeRawDb(self):
- """It's necessary for low level operations."""
- if True:
- logging.debug("Connecting to sqlite")
- self.sqlite_conn = sqlite3.connect(self.GetDatabasePath())
-
- def InitializeDatabase(self):
- """Established the connection to the database.
-
- TODO: Refactor this class to first create CswFile with no primary key and
- no indexes.
- """
- need_to_create_tables = False
- db_path = self.GetDatabasePath()
- checkpkg_dir = os.path.join(os.environ["HOME"], self.CHECKPKG_DIR)
- if not os.path.exists(db_path):
- logging.info("Building the cache database %s.", self.system_pkgmap_files)
- logging.info("The cache will be kept in %s.", db_path)
- if not os.path.exists(checkpkg_dir):
- logging.debug("Creating %s", checkpkg_dir)
- os.mkdir(checkpkg_dir)
- need_to_create_tables = True
- self.InitializeRawDb()
- self.InitializeSqlobject()
- if not self.IsDatabaseGoodSchema():
- logging.info("Old database schema detected.")
- self.PurgeDatabase(drop_tables=True)
- need_to_create_tables = True
- if need_to_create_tables:
- self.CreateTables()
- self.PerformInitialDataImport()
- if not self.IsDatabaseUpToDate():
- logging.debug("Rebuilding the package cache, can take a few minutes.")
- self.ClearTablesForUpdates()
- self.RefreshDatabase()
- self.initialized = True
-
- def RefreshDatabase(self):
- for pkgmap_path in self.system_pkgmap_files:
- self._ProcessSystemPkgmap(pkgmap_path)
- self.PopulatePackagesTable()
- self.SetDatabaseMtime()
-
- def PerformInitialDataImport(self):
- """Imports data into the database.
-
- Original bit of code from checkpkg:
- egrep -v 'SUNWbcp|SUNWowbcp|SUNWucb' /var/sadm/install/contents |
- fgrep -f $EXTRACTDIR/liblist >$EXTRACTDIR/shortcatalog
- """
- for pkgmap_path in self.system_pkgmap_files:
- self._ProcessSystemPkgmap(pkgmap_path)
- self.SetDatabaseSchemaVersion()
- self.PopulatePackagesTable()
- self.SetDatabaseMtime()
-
- def _ProcessSystemPkgmap(self, pkgmap_path):
- """Update the database using data from pkgmap.
-
- The strategy to only update the necessary bits:
- - for each new row
- - look it up in the db
- - if doesn't exist, create it
- - if exists, check the
- TODO: continue this description
- """
- INSERT_SQL = """
- INSERT INTO csw_file (basename, path, line)
- VALUES (?, ?, ?);
- """
- sqlite_cursor = self.sqlite_conn.cursor()
- break_after = DEBUG_BREAK_PKGMAP_AFTER
- contents_length = os.stat(pkgmap_path).st_size
- if break_after:
- estimated_lines = break_after
- else:
- estimated_lines = contents_length / INSTALL_CONTENTS_AVG_LINE_LENGTH
- # The progressbar library doesn't like handling larger numbers
- # It displays up to 99% if we feed it a maxval in the range of hundreds of
- # thousands.
- progressbar_divisor = int(estimated_lines / 1000)
- if progressbar_divisor < 1:
- progressbar_divisor = 1
- update_period = 1L
- # To help delete old records
- system_pkgmap_fd = open(pkgmap_path, "r")
- stop_re = re.compile("(%s)" % "|".join(self.STOP_PKGS))
- # Creating a data structure:
- # soname - {<path1>: <line1>, <path2>: <line2>, ...}
- logging.debug("Building database cache db of the %s file",
- pkgmap_path)
- logging.info("Processing %s, it can take a few minutes", pkgmap_path)
- count = itertools.count()
- bar = progressbar.ProgressBar()
- bar.maxval = estimated_lines / progressbar_divisor
- bar.start()
- # I tried dropping the csw_file_basename_idx index to speed up operation,
- # but after I measured the times, it turned out that it doesn't make any
- # difference to the total runnng time.
- # logging.info("Dropping csw_file_basename_idx")
- # sqlite_cursor.execute("DROP INDEX csw_file_basename_idx;")
- for line in system_pkgmap_fd:
- i = count.next()
- if not i % update_period and (i / progressbar_divisor) <= bar.maxval:
- bar.update(i / progressbar_divisor)
- if stop_re.search(line):
- continue
- if line.startswith("#"):
- continue
- fields = re.split(c.WS_RE, line)
- pkgmap_entry_path = fields[0].split("=")[0]
- pkgmap_entry_dir, pkgmap_entry_base_name = os.path.split(pkgmap_entry_path)
- # The following SQLObject-driven inserts are 60 times slower than the raw
- # sqlite API.
- # pkgmap_entry = m.CswFile(basename=pkgmap_entry_base_name,
- # path=pkgmap_entry_dir, line=line.strip())
- # This page has some hints:
- # http://www.mail-archive.com/sqlobject-discuss@lists.sourceforge.net/msg04641.html
- # "These are simple straightforward INSERTs without any additional
- # high-level burden - no SELECT, no caching, nothing. Fire and forget."
- # sql = self.sqo_conn.sqlrepr(
- # sqlobject.sqlbuilder.Insert(m.CswFile.sqlmeta.table, values=record))
- # self.sqo_conn.query(sql)
- # ...unfortunately, it isn't any faster in practice.
- # The fastest way is:
- sqlite_cursor.execute(INSERT_SQL, [pkgmap_entry_base_name,
- pkgmap_entry_dir,
- line.strip()])
- if break_after and i > break_after:
- logging.warning("Breaking after %s for debugging purposes.", break_after)
- break
- bar.finish()
- self.sqlite_conn.commit()
- logging.debug("All lines of %s were processed.", pkgmap_path)
-
- def _ParsePkginfoLine(self, line):
- fields = re.split(c.WS_RE, line)
- pkgname = fields[1]
- pkg_desc = u" ".join(fields[2:])
- return pkgname, pkg_desc
-
- def PopulatePackagesTable(self):
- logging.info("Updating the packages table")
- args = ["pkginfo"]
- pkginfo_proc = subprocess.Popen(args, stdout=subprocess.PIPE)
- stdout, stderr = pkginfo_proc.communicate()
- ret = pkginfo_proc.wait()
- lines = stdout.splitlines()
- bar = progressbar.ProgressBar()
- bar.maxval = len(lines)
- bar.start()
- count = itertools.count()
- INSERT_SQL = """
- INSERT INTO pkginst (pkgname, pkg_desc)
- VALUES (?, ?);
- """
- # If self.GetInstalledPackages calls out to the initialization,
- # the result is an infinite recursion.
- installed_pkgs = self.GetInstalledPackages(initialize=False)
- for line in stdout.splitlines():
- pkgname, pkg_desc = self._ParsePkginfoLine(line)
- if pkgname not in installed_pkgs:
- # This is slow:
- # pkg = m.Pkginst(pkgname=pkgname, pkg_desc=pkg_desc)
- # This is much faster:
- self.sqlite_conn.execute(INSERT_SQL, [pkgname, pkg_desc])
- i = count.next()
- bar.update(i)
- # Need to commit, otherwise subsequent SQLObject calls will fail.
- self.sqlite_conn.commit()
- bar.finish()
-
- def SetDatabaseMtime(self):
- mtime = self.GetFileMtime()
- res = m.CswConfig.select(m.CswConfig.q.option_key==CONFIG_MTIME)
- if res.count() == 0:
- logging.debug("Inserting the mtime (%s) into the database.", mtime)
- config_record = m.CswConfig(option_key=CONFIG_MTIME, float_value=mtime)
- else:
- logging.debug("Updating the mtime (%s) in the database.", mtime)
- res.getOne().float_value = mtime
-
- def SetDatabaseSchemaVersion(self):
- try:
- config_option = m.CswConfig.select(
- m.CswConfig.q.option_key==CONFIG_DB_SCHEMA).getOne()
- config_option.int_value = database.DB_SCHEMA_VERSION
- except sqlobject.main.SQLObjectNotFound, e:
- version = m.CswConfig(option_key=CONFIG_DB_SCHEMA,
- int_value=database.DB_SCHEMA_VERSION)
-
- def GetPkgmapLineByBasename(self, filename):
- """Returns pkgmap lines by basename:
- {
- path1: line1,
- path2: line2,
- }
- """
- if filename in self.cache:
- return self.cache[filename]
- self._LazyInitializeDatabase()
- res = m.CswFile.select(m.CswFile.q.basename==filename)
- lines = {}
- for obj in res:
- lines[obj.path] = obj.line
- if len(lines) == 0:
- logging.debug("Cache doesn't contain filename %s", filename)
- self.cache[filename] = lines
- return lines
-
- def _InferPackagesFromPkgmapLine(self, line):
- """Given a pkgmap line, return all packages it contains."""
- line = line.strip()
- parts = re.split(c.WS_RE, line)
- pkgs = []
- if parts[1] == 'd':
- parts = parts[6:]
- while parts:
- part = parts.pop()
- if self.digits_re.match(part):
- break
- elif "none" == part:
- break
- pkgs.append(part)
- # Make the packages appear in the same order as in the install/contents
- # file.
- pkgs.reverse()
- return pkgs
-
- def GetPathsAndPkgnamesByBasename(self, filename):
- """Returns paths and packages by basename.
-
- e.g.
- {"/opt/csw/lib": ["CSWfoo", "CSWbar"],
- "/opt/csw/1/lib": ["CSWfoomore"]}
- """
- lines = self.GetPkgmapLineByBasename(filename)
- pkgs = {}
- # Infer packages
- for file_path in lines:
- pkgs[file_path] = self._InferPackagesFromPkgmapLine(lines[file_path])
- # self.error_mgr_mock.GetPathsAndPkgnamesByBasename('libc.so.1').AndReturn({
- # "/usr/lib": (u"SUNWcsl",)})
- logging.debug("self.error_mgr_mock.GetPathsAndPkgnamesByBasename(%s).AndReturn(%s)",
- repr(filename), pprint.pformat(pkgs))
- return pkgs
-
- def GetPkgByPath(self, full_path):
- if full_path not in self.pkgs_by_path_cache:
- self._LazyInitializeDatabase()
- path, basename = os.path.split(full_path)
- try:
- obj = m.CswFile.select(
- sqlobject.AND(
- m.CswFile.q.path==path,
- m.CswFile.q.basename==basename)).getOne()
- self.pkgs_by_path_cache[full_path] = self._InferPackagesFromPkgmapLine(
- obj.line)
- except sqlobject.main.SQLObjectNotFound, e:
- logging.debug("Couldn't find in the db: %s/%s", path, basename)
- logging.debug(e)
- self.pkgs_by_path_cache[full_path] = []
- logging.debug("self.error_mgr_mock.GetPkgByPath(%s).AndReturn(%s)",
- repr(full_path), pprint.pformat(self.pkgs_by_path_cache[full_path]))
- return self.pkgs_by_path_cache[full_path]
-
- def GetDatabaseMtime(self):
- if not self.cache_mtime:
- res = m.CswConfig.select(m.CswConfig.q.option_key==CONFIG_MTIME)
- if res.count() == 1:
- self.cache_mtime = res.getOne().float_value
- elif res.count() < 1:
- self.cache_mtime = 1
- logging.debug("GetDatabaseMtime() --> %s", self.cache_mtime)
- return self.cache_mtime
-
- def GetFileMtime(self):
- if not self.file_mtime:
- stat_data = os.stat(SYSTEM_PKGMAP)
- self.file_mtime = stat_data.st_mtime
- return self.file_mtime
-
- def GetDatabaseSchemaVersion(self):
- schema_on_disk = 1L
- if not m.CswConfig.tableExists():
- return schema_on_disk;
- res = m.CswConfig.select(m.CswConfig.q.option_key == CONFIG_DB_SCHEMA)
- if res.count() < 1:
- logging.debug("No db schema value found, assuming %s.",
- schema_on_disk)
- elif res.count() == 1:
- schema_on_disk = res.getOne().int_value
- return schema_on_disk
-
- def IsDatabaseUpToDate(self):
- f_mtime_epoch = self.GetFileMtime()
- d_mtime_epoch = self.GetDatabaseMtime()
- f_mtime = time.gmtime(int(f_mtime_epoch))
- d_mtime = time.gmtime(int(d_mtime_epoch))
- logging.debug("IsDatabaseUpToDate: f_mtime %s, d_time: %s", f_mtime, d_mtime)
- # Rounding up to integer seconds. There is a race condition:
- # pkgadd finishes at 100.1
- # checkpkg reads /var/sadm/install/contents at 100.2
- # new pkgadd runs and finishes at 100.3
- # subsequent checkpkg runs won't pick up the last change.
- # I don't expect pkgadd to run under 1s.
- fresh = f_mtime <= d_mtime
- good_version = self.GetDatabaseSchemaVersion() >= database.DB_SCHEMA_VERSION
- logging.debug("IsDatabaseUpToDate: good_version=%s, fresh=%s",
- repr(good_version), repr(fresh))
- return fresh and good_version
-
- def ClearTablesForUpdates(self):
- for table in self.TABLES_THAT_NEED_UPDATES:
- table.clearTable()
-
- def PurgeDatabase(self, drop_tables=False):
- if drop_tables:
- for table in self.TABLES:
- if table.tableExists():
- table.dropTable()
- else:
- logging.debug("Truncating all tables")
- for table in self.TABLES:
- table.clearTable()
-
- def GetInstalledPackages(self, initialize=True):
- """Returns a dictionary of all installed packages."""
- if initialize:
- self._LazyInitializeDatabase()
- res = m.Pkginst.select()
- return dict([[str(x.pkgname), str(x.pkg_desc)] for x in res])
-
-
-class LddEmulator(object):
- """A class to emulate ldd(1)
-
- Used primarily to resolve SONAMEs and detect package dependencies.
- """
- def __init__(self):
- self.runpath_expand_cache = {}
- self.runpath_origin_expand_cache = {}
- self.symlink_expand_cache = {}
- self.symlink64_cache = {}
- self.runpath_sanitize_cache = {}
-
- def ExpandRunpath(self, runpath, isalist, binary_path):
- """Expands a signle runpath element.
-
- Args:
- runpath: e.g. "/opt/csw/lib/$ISALIST"
- isalist: isalist elements
- binary_path: Necessary to expand $ORIGIN
- """
- key = (runpath, tuple(isalist))
- if key not in self.runpath_expand_cache:
- origin_present = False
- # Emulating $ISALIST and $ORIGIN expansion
- if '$ORIGIN' in runpath:
- origin_present = True
- if origin_present:
- key_o = (runpath, tuple(isalist), binary_path)
- if key_o in self.runpath_origin_expand_cache:
- return self.runpath_origin_expand_cache[key_o]
- else:
- if not binary_path.startswith("/"):
- binary_path = "/" + binary_path
- runpath = runpath.replace('$ORIGIN', binary_path)
- if '$ISALIST' in runpath:
- expanded_list = [runpath.replace('/$ISALIST', '')]
- expanded_list += [runpath.replace('$ISALIST', isa) for isa in isalist]
- else:
- expanded_list = [runpath]
- expanded_list = [os.path.abspath(p) for p in expanded_list]
- if not origin_present:
- self.runpath_expand_cache[key] = expanded_list
- else:
- self.runpath_origin_expand_cache[key_o] = expanded_list
- return self.runpath_origin_expand_cache[key_o]
- return self.runpath_expand_cache[key]
-
- def ExpandSymlink(self, symlink, target, input_path):
- key = (symlink, target, input_path)
- if key not in self.symlink_expand_cache:
- symlink_re = re.compile(r"%s(/|$)" % symlink)
- if re.search(symlink_re, input_path):
- result = input_path.replace(symlink, target)
- else:
- result = input_path
- self.symlink_expand_cache[key] = result
- return self.symlink_expand_cache[key]
-
- def Emulate64BitSymlinks(self, runpath_list):
- """Need to emulate the 64 -> amd64, 64 -> sparcv9 symlink
-
- Since we don't know the architecture, we are adding both amd64 and
- sparcv9. It should be safe - there are other checks that make sure
- that right architectures are in the right directories.
- """
- key = tuple(runpath_list)
- if key not in self.symlink64_cache:
- symlinked_list = []
- for runpath in runpath_list:
- for symlink, expansion_list in SYSTEM_SYMLINKS:
- for target in expansion_list:
- expanded = self.ExpandSymlink(symlink, target, runpath)
- if expanded not in symlinked_list:
- symlinked_list.append(expanded)
- self.symlink64_cache[key] = symlinked_list
- return self.symlink64_cache[key]
-
- def SanitizeRunpath(self, runpath):
- if runpath not in self.runpath_sanitize_cache:
- self.runpath_sanitize_cache[runpath] = os.path.normpath(runpath)
- return self.runpath_sanitize_cache[runpath]
-
-
- def ResolveSoname(self, runpath_list, soname, isalist,
- path_list, binary_path):
- """Emulates ldd behavior, minimal implementation.
-
- runpath: e.g. ["/opt/csw/lib/$ISALIST", "/usr/lib"]
- soname: e.g. "libfoo.so.1"
- isalist: e.g. ["sparcv9", "sparcv8"]
- path_list: A list of paths where the soname is present, e.g.
- ["/opt/csw/lib", "/opt/csw/lib/sparcv9"]
-
- The function returns the one path.
- """
- # Emulating the install time symlinks, for instance, if the prototype contains
- # /opt/csw/lib/i386/foo.so.0 and /opt/csw/lib/i386 is a symlink to ".",
- # the shared library ends up in /opt/csw/lib/foo.so.0 and should be
- # findable even when RPATH does not contain $ISALIST.
- original_paths_by_expanded_paths = {}
- for p in path_list:
- expanded_p_list = self.Emulate64BitSymlinks([p])
- # We can't just expand and return; we need to return one of the paths given
- # in the path_list.
- for expanded_p in expanded_p_list:
- original_paths_by_expanded_paths[expanded_p] = p
- logging.debug(
- "%s: looking for %s in %s",
- soname, runpath_list, original_paths_by_expanded_paths.keys())
- for runpath_expanded in runpath_list:
- if runpath_expanded in original_paths_by_expanded_paths:
- # logging.debug("Found %s",
- # original_paths_by_expanded_paths[runpath_expanded])
- return original_paths_by_expanded_paths[runpath_expanded]
-
-
-def ParseDumpOutput(dump_output):
- binary_data = {RUNPATH: [],
- NEEDED_SONAMES: []}
- runpath = []
- rpath = []
- for line in dump_output.splitlines():
- fields = re.split(c.WS_RE, line)
- if len(fields) < 3:
- continue
- if fields[1] == "NEEDED":
- binary_data[NEEDED_SONAMES].append(fields[2])
- elif fields[1] == "RUNPATH":
- runpath.extend(fields[2].split(":"))
- elif fields[1] == "RPATH":
- rpath.extend(fields[2].split(":"))
- elif fields[1] == "SONAME":
- binary_data[SONAME] = fields[2]
- if runpath:
- binary_data[RUNPATH].extend(runpath)
- elif rpath:
- binary_data[RUNPATH].extend(rpath)
-
- # Converting runpath to a tuple, which is a hashable data type and can act as
- # a key in a dict.
- binary_data[RUNPATH] = tuple(binary_data[RUNPATH])
- # the NEEDED list must not be modified, converting to a tuple.
- binary_data[NEEDED_SONAMES] = tuple(binary_data[NEEDED_SONAMES])
- binary_data["RUNPATH RPATH the same"] = (runpath == rpath)
- binary_data["RPATH set"] = bool(rpath)
- binary_data["RUNPATH set"] = bool(runpath)
- return binary_data
-
-
-class CheckpkgManagerBase(object):
- """Common functions between the older and newer calling functions."""
-
- def __init__(self, name, stats_basedir, md5sum_list, debug=False):
- self.debug = debug
- self.name = name
- self.md5sum_list = md5sum_list
- self.stats_basedir = stats_basedir
- self.errors = []
- self.individual_checks = []
- self.set_checks = []
- self.packages = []
-
- def GetPackageStatsList(self):
- return [package_stats.PackageStats(None, self.stats_basedir, x)
- for x in self.md5sum_list]
-
- def FormatReports(self, errors, messages, gar_lines):
- namespace = {
- "name": self.name,
- "errors": errors,
- "debug": self.debug,
- "textwrap": textwrap,
- "messages": messages,
- "gar_lines": gar_lines,
- }
- screen_t = Template.Template(SCREEN_ERROR_REPORT_TMPL, searchList=[namespace])
- tags_report_t = Template.Template(TAG_REPORT_TMPL, searchList=[namespace])
- return screen_t, tags_report_t
-
- def SetErrorsToDict(self, set_errors, a_dict):
- # These were generated by a set, but are likely to be bound to specific
- # packages. We'll try to preserve the package assignments.
- errors = copy.copy(a_dict)
- for tag in set_errors:
- if tag.pkgname:
- if not tag.pkgname in errors:
- errors[tag.pkgname] = []
- errors[tag.pkgname].append(tag)
- else:
- if "package-set" not in errors:
- errors["package-set"] = []
- errors["package-set"].append(tag)
- return errors
-
- def GetOptimizedAllStats(self, stats_obj_list):
- logging.info("Unwrapping candies...")
- pkgs_data = []
- counter = itertools.count()
- length = len(stats_obj_list)
- bar = progressbar.ProgressBar()
- bar.maxval = length
- bar.start()
- for stats_obj in stats_obj_list:
- # pkg_data = {}
- # This bit is tightly tied to the data structures returned by
- # PackageStats.
- #
- # Python strings are already implementing the flyweight pattern. What's
- # left is lists and dictionaries.
- i = counter.next()
- # logging.debug("Loading stats for %s (%s/%s)",
- # stats_obj.md5sum, i, length)
- raw_pkg_data = stats_obj.GetAllStats()
- pkg_data = raw_pkg_data
- pkgs_data.append(pkg_data)
- bar.update(i)
- bar.finish()
- return pkgs_data
-
- def Run(self):
- """Runs all the checks
-
- Returns a tuple of an exit code and a report.
- """
- packages_data = self.GetPackageStatsList()
- db_stat_objs_by_pkgname = {}
- obj_id_list = []
- for pkg in packages_data:
- db_obj = pkg.GetDbObject()
- db_stat_objs_by_pkgname[db_obj.pkginst.pkgname] = db_obj
- obj_id_list.append(db_obj.id)
- logging.debug("Deleting old %s errors from the database.",
- db_obj.pkginst.pkgname)
- conn = sqlobject.sqlhub.processConnection
- # It's the maximum number of ORs in a SQL statement.
- # Slicing the long list up into s-sized segments. 1000 is too much.
- obj_id_lists = SliceList(obj_id_list, 900)
- for obj_id_list in obj_id_lists:
- # WARNING: This is raw SQL, potentially breaking during a transition to
- # another db. It's here for efficiency.
- sql = ("DELETE FROM checkpkg_error_tag WHERE %s;"
- % " OR ".join("srv4_file_id = %s" % x for x in obj_id_list))
- conn.query(sql)
- # Need to construct the predicate by hand. Otherwise:
- # File "/opt/csw/lib/python/site-packages/sqlobject/sqlbuilder.py",
- # line 829, in OR
- # return SQLOp("OR", op1, OR(*ops))
- # RuntimeError: maximum recursion depth exceeded while calling a Python object
- #
- # The following also tries to use recursion and fails.
- # delete_predicate = sqlobject.OR(False)
- # for pred in delete_predicate_list:
- # delete_predicate = sqlobject.OR(delete_predicate, pred)
- # conn.query(
- # conn.sqlrepr(sqlbuilder.Delete(m.CheckpkgErrorTag.sqlmeta.table,
- # delete_predicate
- # )))
- # res = m.CheckpkgErrorTag.select(m.CheckpkgErrorTag.q.srv4_file==db_obj)
- # for obj in res:
- # obj.destroySelf()
- errors, messages, gar_lines = self.GetAllTags(packages_data)
- no_errors = len(errors) + 1
- bar = progressbar.ProgressBar()
- bar.maxval = no_errors
- count = itertools.count(1)
- logging.info("Stuffing the candies under the pillow...")
- bar.start()
- for pkgname, es in errors.iteritems():
- logging.debug("Saving %s errors to the database.", pkgname)
- for e in es:
- db_error = m.CheckpkgErrorTag(srv4_file=db_stat_objs_by_pkgname[e.pkgname],
- pkgname=e.pkgname,
- tag_name=e.tag_name,
- tag_info=e.tag_info,
- msg=e.msg)
- bar.update(count.next())
- bar.finish()
- flat_error_list = reduce(operator.add, errors.values(), [])
- screen_report, tags_report = self.FormatReports(errors, messages, gar_lines)
- exit_code = 0
- return (exit_code, screen_report, tags_report)
-
-
-class CheckInterfaceBase(object):
- """Proxies interaction with checking functions.
-
- It wraps access to the /var/sadm/install/contents cache.
- """
-
- def __init__(self, system_pkgmap=None, lines_dict=None):
- self.system_pkgmap = system_pkgmap
- if not self.system_pkgmap:
- self.system_pkgmap = SystemPkgmap()
- self.common_paths = {}
- if lines_dict:
- self.lines_dict = lines_dict
- else:
- self.lines_dict = {}
-
- def GetPathsAndPkgnamesByBasename(self, basename):
- """Proxies calls to self.system_pkgmap."""
- return self.system_pkgmap.GetPathsAndPkgnamesByBasename(basename)
-
- def GetPkgByPath(self, path):
- """Proxies calls to self.system_pkgmap."""
- return self.system_pkgmap.GetPkgByPath(path)
-
- def GetInstalledPackages(self, initialize=True):
- return self.system_pkgmap.GetInstalledPackages(initialize)
-
- def _GetPathsForArch(self, arch):
- if not arch in self.lines_dict:
- file_name = os.path.join(
- os.path.dirname(__file__), "..", "..", "etc", "commondirs-%s" % arch)
- logging.debug("opening %s", file_name)
- f = open(file_name, "r")
- self.lines_dict[arch] = f.read().splitlines()
- f.close()
- return self.lines_dict[arch]
-
- def GetCommonPaths(self, arch):
- """Returns a list of paths for architecture, from gar/etc/commondirs*."""
- # TODO: If this was cached, it could save a significant amount of time.
- if arch not in ('i386', 'sparc', 'all'):
- logging.warn("Wrong arch: %s", repr(arch))
- return []
- if arch == 'all':
- archs = ('i386', 'sparc')
- else:
- archs = [arch]
- lines = []
- for arch in archs:
- lines.extend(self._GetPathsForArch(arch))
- return lines
-
-
-class IndividualCheckInterface(CheckInterfaceBase):
- """To be passed to the checking functions.
-
- Wraps the creation of tag.CheckpkgTag objects.
- """
-
- def __init__(self, pkgname, system_pkgmap=None):
- super(IndividualCheckInterface, self).__init__(system_pkgmap)
- self.pkgname = pkgname
- self.errors = []
-
- def ReportError(self, tag_name, tag_info=None, msg=None):
- logging.debug("self.error_mgr_mock.ReportError(%s, %s, %s)",
- repr(tag_name), repr(tag_info), repr(msg))
- checkpkg_tag = tag.CheckpkgTag(self.pkgname, tag_name, tag_info, msg=msg)
- self.errors.append(checkpkg_tag)
-
-
-class SetCheckInterface(CheckInterfaceBase):
- """To be passed to set checking functions."""
-
- def __init__(self, system_pkgmap=None):
- super(SetCheckInterface, self).__init__(system_pkgmap)
- self.errors = []
-
- def ReportError(self, pkgname, tag_name, tag_info=None, msg=None):
- logging.debug("self.error_mgr_mock.ReportError(%s, %s, %s, %s)",
- repr(pkgname),
- repr(tag_name), repr(tag_info), repr(msg))
- checkpkg_tag = tag.CheckpkgTag(pkgname, tag_name, tag_info, msg=msg)
- self.errors.append(checkpkg_tag)
-
-
-class CheckpkgMessenger(object):
- """Class responsible for passing messages from checks to the user."""
- def __init__(self):
- self.messages = []
- self.one_time_messages = {}
- self.gar_lines = []
-
- def Message(self, m):
- logging.debug("self.messenger.Message(%s)", repr(m))
- self.messages.append(m)
-
- def OneTimeMessage(self, key, m):
- logging.debug("self.messenger.OneTimeMessage(%s, %s)", repr(key), repr(m))
- if key not in self.one_time_messages:
- self.one_time_messages[key] = m
-
- def SuggestGarLine(self, m):
- logging.debug("self.messenger.SuggestGarLine(%s)", repr(m))
- self.gar_lines.append(m)
-
-
-class CheckpkgManager2(CheckpkgManagerBase):
- """The second incarnation of the checkpkg manager.
-
- Implements the API to be used by checking functions.
-
- Its purpose is to reduce the amount of boilerplate code and allow for easier
- unit test writing.
- """
- def _RegisterIndividualCheck(self, function):
- self.individual_checks.append(function)
-
- def _RegisterSetCheck(self, function):
- self.set_checks.append(function)
-
- def _AutoregisterChecks(self):
- """Autodetects all defined checks."""
- logging.debug("CheckpkgManager2._AutoregisterChecks()")
- checkpkg_module = package_checks
- members = dir(checkpkg_module)
- for member_name in members:
- logging.debug("Examining module member: %s", repr(member_name))
- member = getattr(checkpkg_module, member_name)
- if callable(member):
- if member_name.startswith("Check"):
- logging.debug("Registering individual check %s", repr(member_name))
- self._RegisterIndividualCheck(member)
- elif member_name.startswith("SetCheck"):
- logging.debug("Registering set check %s", repr(member_name))
- self._RegisterSetCheck(member)
-
- def GetAllTags(self, stats_obj_list):
- errors = {}
- pkgmap = SystemPkgmap()
- logging.debug("Loading all package statistics.")
- pkgs_data = self.GetOptimizedAllStats(stats_obj_list)
- logging.debug("All package statistics loaded.")
- messenger = CheckpkgMessenger()
- # Individual checks
- count = itertools.count()
- bar = progressbar.ProgressBar()
- bar.maxval = len(pkgs_data) * len(self.individual_checks)
- logging.info("Tasting candies one by one...")
- bar.start()
- for pkg_data in pkgs_data:
- pkgname = pkg_data["basic_stats"]["pkgname"]
- check_interface = IndividualCheckInterface(pkgname, pkgmap)
- for function in self.individual_checks:
- logger = logging.getLogger("%s-%s" % (pkgname, function.__name__))
- logger.debug("Calling %s", function.__name__)
- function(pkg_data, check_interface, logger=logger, messenger=messenger)
- if check_interface.errors:
- errors[pkgname] = check_interface.errors
- bar.update(count.next())
- bar.finish()
- # Set checks
- logging.info("Tasting them all at once...")
- for function in self.set_checks:
- logger = logging.getLogger(function.__name__)
- check_interface = SetCheckInterface(pkgmap)
- logger.debug("Calling %s", function.__name__)
- function(pkgs_data, check_interface, logger=logger, messenger=messenger)
- if check_interface.errors:
- errors = self.SetErrorsToDict(check_interface.errors, errors)
- messages = messenger.messages + messenger.one_time_messages.values()
- return errors, messages, messenger.gar_lines
-
- def Run(self):
- self._AutoregisterChecks()
- return super(CheckpkgManager2, self).Run()
-
-
-def GetIsalist():
- args = ["isalist"]
- isalist_proc = subprocess.Popen(args, stdout=subprocess.PIPE)
- stdout, stderr = isalist_proc.communicate()
- ret = isalist_proc.wait()
- if ret:
- logging.error("Calling isalist has failed.")
- isalist = re.split(r"\s+", stdout.strip())
- return tuple(isalist)
-
-
-def ErrorTagsFromFile(file_name):
- fd = open(file_name)
- error_tags = []
- for line in fd:
- if line.startswith("#"):
- continue
- pkgname, tag_name, tag_info = tag.ParseTagLine(line)
- error_tags.append(tag.CheckpkgTag(pkgname, tag_name, tag_info))
- return error_tags
-
-
-def SliceList(l, size):
- """Trasforms a list into a list of lists."""
- idxes = xrange(0, len(l), size)
- sliced = [l[i:i+size] for i in idxes]
- return sliced
-
def IsMd5(s):
# For optimization, move the compilation elsewhere.
- md5_re = re.compile(MD5_RE)
- return md5_re.match(s)
+ return MD5_RE.match(s)
def GetPackageStatsByFilenamesOrMd5s(args, debug=False):
filenames = []
@@ -1059,15 +128,15 @@
filenames.append(arg)
srv4_pkgs = [inspective_package.InspectiveCswSrv4File(x) for x in filenames]
pkgstat_objs = []
- bar = progressbar.ProgressBar()
- bar.maxval = len(md5s) + len(srv4_pkgs)
- bar.start()
+ pbar = progressbar.ProgressBar()
+ pbar.maxval = len(md5s) + len(srv4_pkgs)
+ pbar.start()
counter = itertools.count()
for pkg in srv4_pkgs:
pkgstat_objs.append(package_stats.PackageStats(pkg, debug=debug))
- bar.update(counter.next())
+ pbar.update(counter.next())
for md5 in md5s:
pkgstat_objs.append(package_stats.PackageStats(None, md5sum=md5, debug=debug))
- bar.update(counter.next())
- bar.finish()
+ pbar.update(counter.next())
+ pbar.finish()
return pkgstat_objs
Added: csw/mgar/gar/v2/lib/python/checkpkg2.py
===================================================================
--- csw/mgar/gar/v2/lib/python/checkpkg2.py (rev 0)
+++ csw/mgar/gar/v2/lib/python/checkpkg2.py 2010-12-10 16:48:58 UTC (rev 11879)
@@ -0,0 +1,178 @@
+#!/usr/bin/env python2.6
+#
+# checkpkg
+#
+
+import logging
+import operator
+import optparse
+import os
+import sys
+import textwrap
+import configuration
+import datetime
+import database
+
+import package_stats
+import checkpkg
+import checkpkg_lib
+import overrides
+import models
+import sqlobject
+
+USAGE = """%prog [ options ] pkg1 [ pkg2 [ ... ] ]"""
+CHECKPKG_MODULE_NAME = "The main checking module."
+BEFORE_OVERRIDES = """If any of the reported errors were false positives, you
+can override them pasting the lines below to the GAR recipe."""
+
+AFTER_OVERRIDES = """Please note that checkpkg isn't suggesting you should
+simply add these overrides do the Makefile. It only informs what the overrides
+could look like. You need to understand what are the reported issues about and
+use your best judgement to decide whether to fix the underlying problems or
+override them. For more information, scroll up and read the detailed
+messages."""
+
+UNAPPLIED_OVERRIDES = """WARNING: Some overrides did not match any errors.
+They can be removed, as they don't take any effect anyway. If you're getting
+errors at the same time, maybe you didn't specify the overrides correctly."""
+
+
+class Error(Exception):
+ """Generic error."""
+
+
+class UsageError(Error):
+ """Problem with usage, e.g. command line options."""
+
+
+def main():
+ configuration.SetUpSqlobjectConnection()
+ parser = optparse.OptionParser(USAGE)
+ parser.add_option("-d", "--debug",
+ dest="debug",
+ action="store_true",
+ default=False,
+ help="Switch on debugging messages")
+ parser.add_option("-q", "--quiet",
+ dest="quiet",
+ action="store_true",
+ default=False,
+ help="Display less messages")
+ parser.add_option("--catalog-release",
+ dest="catrel",
+ default="unstable",
+ help="A catalog release: experimental, unstable, testing, stable.")
+ parser.add_option("-r", "--os-releases",
+ dest="osrel_commas",
+ help=("Comma separated list of ['SunOS5.9', 'SunOS5.10'], "
+ "e.g. 'SunOS5.9,SunOS5.10'."))
+ parser.add_option("-a", "--architecture",
+ dest="arch",
+ help="Architecture: i386, sparc.")
+ parser.add_option("--profile", dest="profile",
+ default=False, action="store_true",
+ help="Enable profiling (a developer option).")
+ options, args = parser.parse_args()
+ assert len(args), "The list of files or md5 sums must be not empty."
+ logging_level = logging.INFO
+ if options.quiet:
+ logging_level = logging.WARNING
+ elif options.debug:
+ # If both flags are set, debug wins.
+ logging_level = logging.DEBUG
+ logging.basicConfig(level=logging_level)
+ logging.debug("Starting.")
+
+ dm = database.DatabaseManager()
+ dm.AutoManage()
+
+
+ err_msg_list = []
+ if not options.osrel_commas:
+ err_msg_list.append("Please specify --os-releases.")
+ if not options.arch:
+ err_msg_list.append("Please specify --architecture.")
+ if err_msg_list:
+ raise UsageError(" ".join(err_msg_list))
+
+ stats_list = []
+ collector = package_stats.StatsCollector(
+ logger=logging,
+ debug=options.debug)
+ # We need to separate files and md5 sums.
+ md5_sums, file_list = [], []
+ for arg in args:
+ if checkpkg.MD5_RE.match(arg):
+ md5_sums.append(arg)
+ else:
+ file_list.append(arg)
+ if file_list:
+ stats_list = collector.CollectStatsFromFiles(file_list, None)
+ # We need the md5 sums of these files
+ md5_sums.extend([x["basic_stats"]["md5_sum"] for x in stats_list])
+ assert md5_sums, "The list of md5 sums must not be empty."
+ logging.debug("md5_sums: %s", md5_sums)
+ osrel_list = options.osrel_commas.split(",")
+ logging.debug("Reading packages data from the database.")
+ # This part might need improvements in order to handle a whole
+ # catalog. On the other hand, if we already have the whole catalog in
+ # the database, we can do it altogether differently.
+ # Transforming the result to a list in order to force object
+ # retrieval.
+ sqo_pkgs = list(models.Srv4FileStats.select(
+ sqlobject.IN(models.Srv4FileStats.q.md5_sum, md5_sums)))
+ tags_for_all_osrels = []
+ sqo_arch = models.Architecture.selectBy(name=options.arch).getOne()
+ sqo_catrel = models.CatalogRelease.selectBy(name=options.catrel).getOne()
+ for osrel in osrel_list:
+ sqo_osrel = models.OsRelease.selectBy(short_name=osrel).getOne()
+ check_manager = checkpkg_lib.CheckpkgManager2(
+ CHECKPKG_MODULE_NAME,
+ sqo_pkgs,
+ osrel,
+ options.arch,
+ options.catrel,
+ debug=options.debug,
+ show_progress=(not options.quiet))
+ # Running the checks, reporting and exiting.
+ exit_code, screen_report, tags_report = check_manager.Run()
+ screen_report = unicode(screen_report)
+ if not options.quiet and screen_report:
+ # TODO: Write this to screen only after overrides are applied.
+ sys.stdout.write(screen_report)
+ else:
+ logging.debug("No screen report.")
+
+ overrides_list = [list(pkg.GetOverridesResult()) for pkg in sqo_pkgs]
+ override_list = reduce(operator.add, overrides_list)
+ args = (sqo_osrel, sqo_arch, sqo_catrel)
+ tag_lists = [list(pkg.GetErrorTagsResult(*args)) for pkg in sqo_pkgs]
+ error_tags = reduce(operator.add, tag_lists)
+ # TODO: Set the 'overriden' tag for specific error tags
+ (tags_after_overrides,
+ unapplied_overrides) = overrides.ApplyOverrides(error_tags, override_list)
+ tags_for_all_osrels.extend(tags_after_overrides)
+ if not options.quiet:
+ if tags_after_overrides:
+ print textwrap.fill(BEFORE_OVERRIDES, 80)
+ for checkpkg_tag in tags_after_overrides:
+ print checkpkg_tag.ToGarSyntax()
+ print textwrap.fill(AFTER_OVERRIDES, 80)
+ if unapplied_overrides:
+ print textwrap.fill(UNAPPLIED_OVERRIDES, 80)
+ for override in unapplied_overrides:
+ print "* Unused %s" % override
+ exit_code = bool(tags_for_all_osrels)
+ sys.exit(exit_code)
+
+
+if __name__ == '__main__':
+ if "--profile" in sys.argv:
+ import cProfile
+ t_str = datetime.datetime.now().strftime("%Y-%m-%d-%H-%M")
+ home = os.environ["HOME"]
+ cprof_file_name = os.path.join(
+ home, ".checkpkg", "run-modules-%s.cprof" % t_str)
+ cProfile.run("main()", sort=1, filename=cprof_file_name)
+ else:
+ main()
Property changes on: csw/mgar/gar/v2/lib/python/checkpkg2.py
___________________________________________________________________
Added: svn:executable
+ *
Added: csw/mgar/gar/v2/lib/python/checkpkg_lib.py
===================================================================
--- csw/mgar/gar/v2/lib/python/checkpkg_lib.py (rev 0)
+++ csw/mgar/gar/v2/lib/python/checkpkg_lib.py 2010-12-10 16:48:58 UTC (rev 11879)
@@ -0,0 +1,600 @@
+# A collection of checkpkg-specific classes.
+#
+# This file is supposed to drain the checkpkg.py file until is becomes
+# empty and goes away.
+
+import copy
+from Cheetah import Template
+import logging
+import package_stats
+import package_checks
+import sqlobject
+import itertools
+import progressbar
+import database
+import models as m
+import textwrap
+import os.path
+import tag
+import pprint
+import operator
+import common_constants
+import sharedlib_utils
+import mute_progressbar
+import cPickle
+
+
+class Error(Exception):
+ pass
+
+
+class CatalogDatabaseError(Error):
+ pass
+
+
+SCREEN_ERROR_REPORT_TMPL = u"""#if $errors
+#if $debug
+ERROR: One or more errors have been found by $name.
+#end if
+#for $pkgname in $errors
+$pkgname:
+#for $error in $errors[$pkgname]
+#if $debug
+ $repr($error)
+#elif $error.msg
+$textwrap.fill($error.msg, 78, initial_indent="# ", subsequent_indent="# ")
+# -> $repr($error)
+
+#end if
+#end for
+#end for
+#else
+#if $debug
+OK: $repr($name) module found no problems.
+#end if
+#end if
+#if $messages
+#for $msg in $messages
+$textwrap.fill($msg, 78, initial_indent=" * ", subsequent_indent=" ")
+#end for
+#end if
+#if $gar_lines
+
+# Checkpkg suggests adding the following lines to the GAR recipe:
+# This is a summary; see above for details.
+#for $line in $gar_lines
+$line
+#end for
+#end if
+"""
+
+# http://www.cheetahtemplate.org/docs/users_guide_html_multipage/language.directives.closures.html
+TAG_REPORT_TMPL = u"""#if $errors
+# Tags reported by $name module
+#for $pkgname in $errors
+#for $tag in $errors[$pkgname]
+#if $tag.msg
+$textwrap.fill($tag.msg, 70, initial_indent="# ", subsequent_indent="# ")
+#end if
+$pkgname: ${tag.tag_name}#if $tag.tag_info# $tag.tag_info#end if#
+#end for
+#end for
+#end if
+"""
+
+
+class SqlobjectHelperMixin(object):
+
+ def GetSqlobjectTriad(self, osrel, arch, catrel):
+ logging.debug("GetSqlobjectTriad(%s, %s, %s)", osrel, arch, catrel)
+ sqo_arch = m.Architecture.select(
+ m.Architecture.q.name==arch).getOne()
+ sqo_osrel = m.OsRelease.select(
+ m.OsRelease.q.short_name==osrel).getOne()
+ sqo_catrel = m.CatalogRelease.select(
+ m.CatalogRelease.q.name==catrel).getOne()
+ return sqo_osrel, sqo_arch, sqo_catrel
+
+
+class CheckpkgManagerBase(SqlobjectHelperMixin):
+ """Common functions between the older and newer calling functions."""
+
+ def __init__(self, name, sqo_pkgs_list, osrel, arch, catrel, debug=False,
+ show_progress=False):
+ self.debug = debug
+ self.name = name
+ self.sqo_pkgs_list = sqo_pkgs_list
+ self.errors = []
+ self.individual_checks = []
+ self.set_checks = []
+ self.packages = []
+ self.osrel = osrel
+ self.arch = arch
+ self.catrel = catrel
+ self.show_progress = show_progress
+
+ def GetProgressBar(self):
+ if self.show_progress:
+ return progressbar.ProgressBar()
+ else:
+ return mute_progressbar.MuteProgressBar()
+
+ def GetSqlobjectTriad(self):
+ return super(CheckpkgManagerBase, self).GetSqlobjectTriad(
+ self.osrel, self.arch, self.catrel)
+
+ def GetPackageStatsList(self):
+ raise RuntimeError("Please don't use this function as it violates "
+ "the Law of Demeter.")
+
+ def FormatReports(self, errors, messages, gar_lines):
+ namespace = {
+ "name": self.name,
+ "errors": errors,
+ "debug": self.debug,
+ "textwrap": textwrap,
+ "messages": messages,
+ "gar_lines": gar_lines,
+ }
+ screen_t = Template.Template(SCREEN_ERROR_REPORT_TMPL, searchList=[namespace])
+ tags_report_t = Template.Template(TAG_REPORT_TMPL, searchList=[namespace])
+ return screen_t, tags_report_t
+
+ def SetErrorsToDict(self, set_errors, a_dict):
+ # These were generated by a set, but are likely to be bound to specific
+ # packages. We'll try to preserve the package assignments.
+ errors = copy.copy(a_dict)
+ for tag in set_errors:
+ if tag.pkgname:
+ if not tag.pkgname in errors:
+ errors[tag.pkgname] = []
+ errors[tag.pkgname].append(tag)
+ else:
+ if "package-set" not in errors:
+ errors["package-set"] = []
+ errors["package-set"].append(tag)
+ return errors
+
+ def GetOptimizedAllStats(self, stats_obj_list):
+ logging.info("Unwrapping candies...")
+ pkgs_data = []
+ counter = itertools.count()
+ length = len(stats_obj_list)
+ pbar = self.GetProgressBar()
+ pbar.maxval = length
+ pbar.start()
+ for stats_obj in stats_obj_list:
+ # This bit is tightly tied to the data structures returned by
+ # PackageStats.
+ #
+ # Python strings are already implementing the flyweight pattern. What's
+ # left is lists and dictionaries.
+ i = counter.next()
+ raw_pkg_data = cPickle.loads(stats_obj.data_obj.pickle)
+ pkg_data = raw_pkg_data
+ pkgs_data.append(pkg_data)
+ pbar.update(i)
+ pbar.finish()
+ return pkgs_data
+
+ def Run(self):
+ """Runs all the checks
+
+ Returns a tuple of an exit code and a report.
+ """
+ # packages_data = self.GetPackageStatsList()
+ assert self.sqo_pkgs_list, "The list of packages must not be empty."
+ db_stat_objs_by_pkgname = {}
+ for pkg in self.sqo_pkgs_list:
+ db_stat_objs_by_pkgname[pkg.pkginst.pkgname] = pkg
+ logging.debug("Deleting old errors from the database.")
+ for pkgname, db_obj in db_stat_objs_by_pkgname.iteritems():
+ sqo_os_rel, sqo_arch, sqo_catrel = self.GetSqlobjectTriad()
+ db_obj.RemoveCheckpkgResults(
+ sqo_os_rel, sqo_arch, sqo_catrel)
+ errors, messages, gar_lines = self.GetAllTags(self.sqo_pkgs_list)
+ no_errors = len(errors) + 1
+ pbar = self.GetProgressBar()
+ pbar.maxval = no_errors
+ count = itertools.count(1)
+ logging.info("Stuffing the candies under the pillow...")
+ pbar.start()
+ for pkgname, es in errors.iteritems():
+ logging.debug("Saving errors of %s to the database.", pkgname)
+ for e in es:
+ if e.pkgname not in db_stat_objs_by_pkgname:
+ logging.warning("Not saving an error for %s.", e.pkgname)
+ continue
+ db_error = m.CheckpkgErrorTag(srv4_file=db_stat_objs_by_pkgname[e.pkgname],
+ pkgname=e.pkgname,
+ tag_name=e.tag_name,
+ tag_info=e.tag_info,
+ msg=e.msg,
+ os_rel=sqo_os_rel,
+ catrel=sqo_catrel,
+ arch=sqo_arch)
+ pbar.update(count.next())
+ pbar.finish()
+ flat_error_list = reduce(operator.add, errors.values(), [])
+ screen_report, tags_report = self.FormatReports(errors, messages, gar_lines)
+ exit_code = 0
+ return (exit_code, screen_report, tags_report)
+
+
+class CheckInterfaceBase(object):
+ """Provides an interface for checking functions.
+
+ It wraps access to the catalog database.
+ """
+
+ def __init__(self, osrel, arch, catrel, catalog=None, lines_dict=None):
+ self.osrel = osrel
+ self.arch = arch
+ self.catrel = catrel
+ self.catalog = catalog
+ if not self.catalog:
+ self.catalog = Catalog()
+ self.common_paths = {}
+ if lines_dict:
+ self.lines_dict = lines_dict
+ else:
+ self.lines_dict = {}
+
+ def GetPathsAndPkgnamesByBasename(self, basename):
+ """Proxies calls to class member."""
+ return self.catalog.GetPathsAndPkgnamesByBasename(
+ basename, self.osrel, self.arch, self.catrel)
+
+ def GetPkgByPath(self, file_path):
+ """Proxies calls to self.system_pkgmap."""
+ return self.catalog.GetPkgByPath(
+ file_path, self.osrel, self.arch, self.catrel)
+
+ def GetInstalledPackages(self):
+ return self.catalog.GetInstalledPackages(
+ self.osrel, self.arch, self.catrel)
+
+ def _GetPathsForArch(self, arch):
+ if not arch in self.lines_dict:
+ file_name = os.path.join(
+ os.path.dirname(__file__), "..", "..", "etc", "commondirs-%s" % arch)
+ logging.debug("opening %s", file_name)
+ f = open(file_name, "r")
+ self.lines_dict[arch] = f.read().splitlines()
+ f.close()
+ return self.lines_dict[arch]
+
+ def GetCommonPaths(self, arch):
+ """Returns a list of paths for architecture, from gar/etc/commondirs*."""
+ # TODO: If this was cached, it could save a significant amount of time.
+ if arch not in ('i386', 'sparc', 'all'):
+ logging.warn("Wrong arch: %s", repr(arch))
+ return []
+ if arch == 'all':
+ archs = ('i386', 'sparc')
+ else:
+ archs = [arch]
+ lines = []
+ for arch in archs:
+ lines.extend(self._GetPathsForArch(arch))
+ return lines
+
+
+class IndividualCheckInterface(CheckInterfaceBase):
+ """To be passed to the checking functions.
+
+ Wraps the creation of tag.CheckpkgTag objects.
+ """
+
+ def __init__(self, pkgname, osrel, arch, catrel, catalog=None):
+ super(IndividualCheckInterface, self).__init__(osrel, arch, catrel, catalog)
+ self.pkgname = pkgname
+ self.errors = []
+
+ def ReportError(self, tag_name, tag_info=None, msg=None):
+ logging.debug("self.error_mgr_mock.ReportError(%s, %s, %s)",
+ repr(tag_name), repr(tag_info), repr(msg))
+ checkpkg_tag = tag.CheckpkgTag(self.pkgname, tag_name, tag_info, msg=msg)
+ self.errors.append(checkpkg_tag)
+
+
+class SetCheckInterface(CheckInterfaceBase):
+ """To be passed to set checking functions."""
+
+ def __init__(self, osrel, arch, catrel, catalog=None):
+ super(SetCheckInterface, self).__init__(osrel, arch, catrel, catalog)
+ self.errors = []
+
+ def ReportError(self, pkgname, tag_name, tag_info=None, msg=None):
+ logging.debug("self.error_mgr_mock.ReportError(%s, %s, %s, %s)",
+ repr(pkgname),
+ repr(tag_name), repr(tag_info), repr(msg))
+ checkpkg_tag = tag.CheckpkgTag(pkgname, tag_name, tag_info, msg=msg)
+ self.errors.append(checkpkg_tag)
+
+
+class CheckpkgMessenger(object):
+ """Class responsible for passing messages from checks to the user."""
+ def __init__(self):
+ self.messages = []
+ self.one_time_messages = {}
+ self.gar_lines = []
+
+ def Message(self, m):
+ logging.debug("self.messenger.Message(%s)", repr(m))
+ self.messages.append(m)
+
+ def OneTimeMessage(self, key, m):
+ logging.debug("self.messenger.OneTimeMessage(%s, %s)", repr(key), repr(m))
+ if key not in self.one_time_messages:
+ self.one_time_messages[key] = m
+
+ def SuggestGarLine(self, m):
+ logging.debug("self.messenger.SuggestGarLine(%s)", repr(m))
+ self.gar_lines.append(m)
+
+
+class CheckpkgManager2(CheckpkgManagerBase):
+ """The second incarnation of the checkpkg manager.
+
+ Implements the API to be used by checking functions.
+
+ Its purpose is to reduce the amount of boilerplate code and allow for easier
+ unit test writing.
+ """
+ def _RegisterIndividualCheck(self, function):
+ self.individual_checks.append(function)
+
+ def _RegisterSetCheck(self, function):
+ self.set_checks.append(function)
+
+ def _AutoregisterChecks(self):
+ """Autodetects all defined checks."""
+ logging.debug("CheckpkgManager2._AutoregisterChecks()")
+ checkpkg_module = package_checks
+ members = dir(checkpkg_module)
+ for member_name in members:
+ logging.debug("Examining module member: %s", repr(member_name))
+ member = getattr(checkpkg_module, member_name)
+ if callable(member):
+ if member_name.startswith("Check"):
+ logging.debug("Registering individual check %s", repr(member_name))
+ self._RegisterIndividualCheck(member)
+ elif member_name.startswith("SetCheck"):
+ logging.debug("Registering set check %s", repr(member_name))
+ self._RegisterSetCheck(member)
+
+ def GetAllTags(self, stats_obj_list):
+ errors = {}
+ catalog = Catalog()
+ logging.debug("Loading all package statistics.")
+ pkgs_data = self.GetOptimizedAllStats(stats_obj_list)
+ logging.debug("All package statistics loaded.")
+ messenger = CheckpkgMessenger()
+ # Individual checks
+ count = itertools.count()
+ pbar = self.GetProgressBar()
+ pbar.maxval = len(pkgs_data) * len(self.individual_checks)
+ logging.info("Tasting candies one by one...")
+ pbar.start()
+ for pkg_data in pkgs_data:
+ pkgname = pkg_data["basic_stats"]["pkgname"]
+ check_interface = IndividualCheckInterface(
+ pkgname, self.osrel, self.arch, self.catrel, catalog)
+ for function in self.individual_checks:
+ logger = logging.getLogger("%s-%s" % (pkgname, function.__name__))
+ logger.debug("Calling %s", function.__name__)
+ function(pkg_data, check_interface, logger=logger, messenger=messenger)
+ if check_interface.errors:
+ errors[pkgname] = check_interface.errors
+ pbar.update(count.next())
+ pbar.finish()
+ # Set checks
+ logging.info("Tasting them all at once...")
+ for function in self.set_checks:
+ logger = logging.getLogger(function.__name__)
+ check_interface = SetCheckInterface(
+ self.osrel, self.arch, self.catrel, catalog)
+ logger.debug("Calling %s", function.__name__)
+ function(pkgs_data, check_interface, logger=logger, messenger=messenger)
+ if check_interface.errors:
+ errors = self.SetErrorsToDict(check_interface.errors, errors)
+ messages = messenger.messages + messenger.one_time_messages.values()
+ return errors, messages, messenger.gar_lines
+
+ def Run(self):
+ self._AutoregisterChecks()
+ return super(CheckpkgManager2, self).Run()
+
+
+def SliceList(l, size):
+ """Trasforms a list into a list of lists."""
+ idxes = xrange(0, len(l), size)
+ sliced = [l[i:i+size] for i in idxes]
+ return sliced
+
+
+class CatalogMixin(SqlobjectHelperMixin):
+ """Responsible for functionality related to catalog operations.
+
+ These include:
+ - getting a list of all packages
+ - getting a list of packages that contain certain files
+ - getting a list of packages that contain files of certain names
+ """
+
+ def __init__(self):
+ super(CatalogMixin, self).__init__()
+ self.pkgs_by_path_cache = {}
+
+ def GetInstalledPackages(self, osrel, arch, catrel):
+ sqo_osrel, sqo_arch, sqo_catrel = self.GetSqlobjectTriad(
+ osrel, arch, catrel)
+ res = m.Srv4FileInCatalog.select(
+ sqlobject.AND(
+ m.Srv4FileInCatalog.q.osrel==sqo_osrel,
+ m.Srv4FileInCatalog.q.arch==sqo_arch,
+ m.Srv4FileInCatalog.q.catrel==sqo_catrel))
+ pkgs = []
+ for srv4_in_cat in res:
+ pkgs.append(srv4_in_cat.srv4file.pkginst.pkgname)
+ return pkgs
+
+ def GetPathsAndPkgnamesByBasename(self, basename, osrel, arch, catrel):
+ """Retrieves pkginst names of packages that have certain files.
+
+ Since it needs to match a specific catalog, a table join is required:
+ - CswFile (basename)
+ - related Srv4FileStats
+ - related Srv4FileInCatalog
+
+ Args:
+ basename: u'libfoo.so.1'
+ osrel: u'5.9'
+ arch: 'sparc', 'x86'
+ catrel: 'stable'
+
+ Returns:
+ {"/opt/csw/lib": ["CSWfoo", "CSWbar"],
+ "/opt/csw/1/lib": ["CSWfoomore"]}
+ """
+ pkgs = {}
+ sqo_osrel, sqo_arch, sqo_catrel = self.GetSqlobjectTriad(
+ osrel, arch, catrel)
+
+ # Looks like this join is hard to do that way.
+ # res = m.Srv4FileInCatalog.select(
+ # sqlobject.AND(
+ # m.Srv4FileInCatalog.q.osrel==sqo_osrel,
+ # m.Srv4FileInCatalog.q.arch==sqo_arch,
+ # m.Srv4FileInCatalog.q.catrel==sqo_catrel)).
+ # throughTo.srv4file.thoughTo.files
+
+ # We'll implement it on the application level. First, we'll get all
+ # the files that match the basename, and then filter them based on
+ # catalog properties.
+ res = m.CswFile.select(m.CswFile.q.basename==basename)
+ file_list = []
+ for f in res:
+ # Check whether osrel, arch and catrel are matching.
+ for cat in f.srv4_file.in_catalogs:
+ if (f.srv4_file.registered
+ and cat.osrel == sqo_osrel
+ and cat.arch == sqo_arch
+ and cat.catrel == sqo_catrel):
+ file_list.append(f)
+ for obj in file_list:
+ pkgs.setdefault(obj.path, [])
+ pkgs[obj.path].append(obj.pkginst.pkgname)
+ logging.debug("self.error_mgr_mock.GetPathsAndPkgnamesByBasename(%s)"
+ ".AndReturn(%s)", repr(basename), pprint.pformat(pkgs))
+ return pkgs
+
+ def GetPkgByPath(self, full_file_path, osrel, arch, catrel):
+ """Returns a list of packages."""
+ # Memoization for performance
+ key = (full_file_path, osrel, arch, catrel)
+ if key not in self.pkgs_by_path_cache:
+ pkgs = []
+ file_path, basename = os.path.split(full_file_path)
+ sqo_osrel, sqo_arch, sqo_catrel = self.GetSqlobjectTriad(
+ osrel, arch, catrel)
+ res = m.CswFile.select(
+ sqlobject.AND(
+ m.CswFile.q.path==file_path,
+ m.CswFile.q.basename==basename))
+ for sqo_file in res:
+ # Making sure that we're taking packages only from the right catalog.
+ for cat in sqo_file.srv4_file.in_catalogs:
+ if (sqo_file.srv4_file.registered
+ and cat.osrel == sqo_osrel
+ and cat.arch == sqo_arch
+ and cat.catrel == sqo_catrel):
+ pkgs.append(sqo_file.srv4_file.pkginst.pkgname)
+ self.pkgs_by_path_cache[key] = frozenset(pkgs)
+ return self.pkgs_by_path_cache[key]
+
+ def CommonArchByString(self, s):
+ return sharedlib_utils.ArchByString(s)
+
+ def Srv4MatchesCatalog(self, sqo_srv4, sqo_arch):
+ cat_arch = self.CommonArchByString(sqo_arch.name)
+ pkg_arch = self.CommonArchByString(sqo_srv4.arch.name)
+ ans = (cat_arch == pkg_arch) or (pkg_arch == common_constants.ARCH_ALL)
+ if not ans:
+ logging.debug("Srv4MatchesCatalog(): mismatch: %s / %s and %s / %s",
+ cat_arch, pkg_arch, pkg_arch, common_constants.ARCH_ALL)
+ # Some packages have the architecture in the file saying 'all', but pkginfo
+ # says 'i386'.
+ filename_arch = self.CommonArchByString(sqo_srv4.filename_arch.name)
+ if filename_arch == common_constants.ARCH_ALL:
+ ans = True
+ if filename_arch != pkg_arch:
+ logging.warning(
+ "Package %s declares %s in pkginfo and %s in the filename.",
+ sqo_srv4, repr(pkg_arch), repr(filename_arch))
+ return ans
+
+ def AddSrv4ToCatalog(self, sqo_srv4, osrel, arch, catrel):
+ """Registers a srv4 file in a catalog."""
+ logging.debug("AddSrv4ToCatalog(%s, %s, %s, %s)",
+ sqo_srv4, osrel, arch, catrel)
+ sqo_osrel, sqo_arch, sqo_catrel = self.GetSqlobjectTriad(
+ osrel, arch, catrel)
+ if not self.Srv4MatchesCatalog(sqo_srv4, sqo_arch):
+ raise CatalogDatabaseError(
+ "Specified package does not match the catalog. "
+ "Package: %s, catalog: %s %s %s"
+ % (sqo_srv4, osrel, arch, catrel))
+ if not sqo_srv4.registered:
+ raise CatalogDatabaseError(
+ "Package %s (%s) is not registered for releases."
+ % (sqo_srv4.basename, sqo_srv4.md5_sum))
+ # Checking for presence of a different srv4 with the same pkginst in the
+ # same catalog
+ pkginst = sqo_srv4.pkginst
+ res = m.Srv4FileStats.select(
+ m.Srv4FileStats.q.pkginst==pkginst).throughTo.in_catalogs.filter(
+ sqlobject.AND(
+ m.Srv4FileInCatalog.q.osrel==sqo_osrel,
+ m.Srv4FileInCatalog.q.arch==sqo_arch,
+ m.Srv4FileInCatalog.q.catrel==sqo_catrel,
+ m.Srv4FileInCatalog.q.srv4file!=sqo_srv4))
+ if len(list(res)):
+ raise CatalogDatabaseError(
+ "There already is a package with that pkgname: %s" % pkginst)
+ # Checking for presence of the same srv4 already in the catalog.
+ res = m.Srv4FileInCatalog.select(
+ sqlobject.AND(
+ m.Srv4FileInCatalog.q.osrel==sqo_osrel,
+ m.Srv4FileInCatalog.q.arch==sqo_arch,
+ m.Srv4FileInCatalog.q.catrel==sqo_catrel,
+ m.Srv4FileInCatalog.q.srv4file==sqo_srv4))
+ if len(list(res)):
+ logging.debug("%s is already part of %s %s %s",
+ sqo_srv4, osrel, arch, catrel)
+ # Our srv4 is already part of that catalog.
+ return
+ obj = m.Srv4FileInCatalog(
+ arch=sqo_arch,
+ osrel=sqo_osrel,
+ catrel=sqo_catrel,
+ srv4file=sqo_srv4)
+
+ def RemoveSrv4(self, sqo_srv4, osrel, arch, catrel):
+ sqo_osrel, sqo_arch, sqo_catrel = self.GetSqlobjectTriad(
+ osrel, arch, catrel)
+ sqo_srv4_in_cat = m.Srv4FileInCatalog.select(
+ sqlobject.AND(
+ m.Srv4FileInCatalog.q.arch==sqo_arch,
+ m.Srv4FileInCatalog.q.osrel==sqo_osrel,
+ m.Srv4FileInCatalog.q.catrel==sqo_catrel,
+ m.Srv4FileInCatalog.q.srv4file==sqo_srv4)).getOne()
+ sqo_srv4_in_cat.registered = False
+ # TODO(maciej): Remove all files belonging to that one
+ for cat_file in sqo_srv4_in_cat.srv4file.files:
+ cat_file.destroySelf()
+
+
+class Catalog(CatalogMixin):
+ pass
Added: csw/mgar/gar/v2/lib/python/checkpkg_lib_test.py
===================================================================
--- csw/mgar/gar/v2/lib/python/checkpkg_lib_test.py (rev 0)
+++ csw/mgar/gar/v2/lib/python/checkpkg_lib_test.py 2010-12-10 16:48:58 UTC (rev 11879)
@@ -0,0 +1,74 @@
+#!/usr/bin/env python2.6
+
+import copy
+import unittest
+import checkpkg_lib
+import tag
+import package_stats
+import database
+import sqlobject
+import models
+import package_stats
+import inspective_package
+import mox
+import test_base
+
+from testdata.neon_stats import pkgstats as neon_stats
+
+
+class CheckpkgManager2UnitTest(unittest.TestCase):
+
+ def setUp(self):
+ super(CheckpkgManager2UnitTest, self).setUp()
+ self.mox = mox.Mox()
+
+ def testSingleTag(self):
+ m = checkpkg_lib.CheckpkgManager2(
+ "testname", [], "5.9", "sparc", "unstable")
+ tags = {
+ "CSWfoo": [
+ tag.CheckpkgTag("CSWfoo", "foo-tag", "foo-info"),
+ ],
+ }
+ screen_report, tags_report = m.FormatReports(tags, [], [])
+ expected = u'# Tags reported by testname module\nCSWfoo: foo-tag foo-info\n'
+ self.assertEqual(expected, unicode(tags_report))
+
+ def testThreeTags(self):
+ m = checkpkg_lib.CheckpkgManager2(
+ "testname", [], "5.9", "sparc", "unstable")
+ tags = {
+ "CSWfoo": [
+ tag.CheckpkgTag("CSWfoo", "foo-tag", "foo-info"),
+ tag.CheckpkgTag("CSWfoo", "bar-tag", "bar-info"),
+ tag.CheckpkgTag("CSWfoo", "baz-tag"),
+ ],
+ }
+ screen_report, tags_report = m.FormatReports(tags, [], [])
+ expected = (u'# Tags reported by testname module\n'
+ u'CSWfoo: foo-tag foo-info\n'
+ u'CSWfoo: bar-tag bar-info\n'
+ u'CSWfoo: baz-tag\n')
+ self.assertEqual(expected, unicode(tags_report))
+
+
+class CheckpkgManager2DatabaseIntegrationTest(
+ test_base.SqlObjectTestMixin, unittest.TestCase):
+
+ def setUp(self):
+ super(CheckpkgManager2DatabaseIntegrationTest, self).setUp()
+ self.mox = mox.Mox()
+
+ def testInsertNeon(self):
+ self.dbc.InitialDataImport()
+ sqo_pkg = package_stats.PackageStats.SaveStats(neon_stats[0], True)
+ cm = checkpkg_lib.CheckpkgManager2(
+ "testname", [sqo_pkg], "SunOS5.9", "sparc", "unstable",
+ show_progress=False)
+ cm.Run()
+ # Verifying that there are some reported error tags.
+ self.assertTrue(list(models.CheckpkgErrorTag.select()))
+
+
+if __name__ == '__main__':
+ unittest.main()
Property changes on: csw/mgar/gar/v2/lib/python/checkpkg_lib_test.py
___________________________________________________________________
Added: svn:executable
+ *
Modified: csw/mgar/gar/v2/lib/python/checkpkg_test.py
===================================================================
--- csw/mgar/gar/v2/lib/python/checkpkg_test.py 2010-12-10 13:48:16 UTC (rev 11878)
+++ csw/mgar/gar/v2/lib/python/checkpkg_test.py 2010-12-10 16:48:58 UTC (rev 11879)
@@ -1,16 +1,24 @@
-#!/opt/csw/bin/python2.6
+#!/usr/bin/env python2.6
# $Id$
+import copy
import re
import unittest
import mox
import difflib
import checkpkg
+import checkpkg_lib
+import database
+import models as m
import tag
-import testdata.dump_output_1 as dump_1
-import testdata.dump_output_2 as dump_2
-import testdata.dump_output_3 as dump_3
+import package_stats
+import sqlite3
+import sqlobject
+import test_base
+from testdata.tree_stats import pkgstats as tree_stats
+from testdata.neon_stats import pkgstats as neon_stats
+
"""A set of unit tests for the library checking code.
A bunch of lines to test in the interactive Python shell.
@@ -23,250 +31,6 @@
SELECT * FROM systempkgmap WHERE basename = 'libncursesw.so.5';
"""
-class GetLinesBySonameUnitTest(unittest.TestCase):
-
- def setUp(self):
- self.pkgmap_mocker = mox.Mox()
- self.e = checkpkg.LddEmulator()
-
- def testExpandRunpath_1(self):
- isalist = ["foo", "bar"]
- runpath = "/opt/csw/lib/$ISALIST"
- expected = ["/opt/csw/lib", "/opt/csw/lib/foo", "/opt/csw/lib/bar"]
- bin_path = "opt/csw/lib"
- self.assertEquals(expected, self.e.ExpandRunpath(runpath, isalist, bin_path))
-
- def testExpandRunpath_2(self):
- isalist = ["foo", "bar"]
- runpath = "/opt/csw/mysql5/lib/$ISALIST/mysql"
- expected = ["/opt/csw/mysql5/lib/mysql",
- "/opt/csw/mysql5/lib/foo/mysql",
- "/opt/csw/mysql5/lib/bar/mysql"]
- bin_path = "opt/csw/lib"
- self.assertEquals(expected, self.e.ExpandRunpath(runpath, isalist, bin_path))
-
- def testExpandRunpath_OriginSimple(self):
- isalist = ()
- runpath = "$ORIGIN"
- expected = ["/opt/csw/lib"]
- bin_path = "opt/csw/lib"
- self.assertEquals(expected, self.e.ExpandRunpath(runpath, isalist, bin_path))
-
- def testExpandRunpath_OriginDots(self):
- isalist = ()
- runpath = "$ORIGIN/.."
- expected = ["/opt/csw/lib"]
- bin_path = "opt/csw/lib/subdir"
- self.assertEquals(expected, self.e.ExpandRunpath(runpath, isalist, bin_path))
-
- def testExpandRunpath_Caching(self):
- """Make sure that the cache doesn't mess it up.
-
- Two invocations, where the only difference is the binary path.
- """
- isalist = ()
- runpath = "/opt/csw/lib/foo"
- expected = ["/opt/csw/lib/foo"]
- bin_path = "opt/csw/lib"
- self.assertEquals(expected, self.e.ExpandRunpath(runpath, isalist, bin_path))
- expected = ["/opt/csw/lib/foo"]
- bin_path = "/opt/csw/lib/foo"
- self.assertEquals(expected, self.e.ExpandRunpath(runpath, isalist, bin_path))
-
- def testExpandRunpath_OriginCaching(self):
- """Make sure that the cache doesn't mess it up.
-
- Two invocations, where the only difference is the binary path.
- """
- isalist = ()
- runpath = "$ORIGIN"
- expected = ["/opt/csw/lib"]
- bin_path = "opt/csw/lib"
- self.assertEquals(expected,
- self.e.ExpandRunpath(runpath, isalist, bin_path))
- expected = ["/opt/csw/foo/lib"]
- bin_path = "/opt/csw/foo/lib"
- self.assertEquals(expected,
- self.e.ExpandRunpath(runpath, isalist, bin_path))
-
- def testExpandRunpath_OnlyIsalist(self):
- """Make sure that the cache doesn't mess it up.
-
- Two invocations, where the only difference is the binary path.
- """
- isalist = ("bar",)
- runpath = "/opt/csw/lib/$ISALIST"
- expected = ["/opt/csw/lib", "/opt/csw/lib/bar"]
- bin_path = "opt/csw/lib"
- self.assertEquals(expected, self.e.ExpandRunpath(runpath, isalist, bin_path))
-
- def testEmulate64BitSymlinks_1(self):
- runpath_list = ["/opt/csw/mysql5/lib/foo/mysql/64"]
- expected = "/opt/csw/mysql5/lib/foo/mysql/amd64"
- self.assertTrue(expected in self.e.Emulate64BitSymlinks(runpath_list))
-
- def testEmulate64BitSymlinks_2(self):
- runpath_list = ["/opt/csw/mysql5/lib/64/mysql/foo"]
- expected = "/opt/csw/mysql5/lib/amd64/mysql/foo"
- result = self.e.Emulate64BitSymlinks(runpath_list)
- self.assertTrue(expected in result, "%s not in %s" % (expected, result))
-
- def testEmulate64BitSymlinks_3(self):
- runpath_list = ["/opt/csw/mysql5/lib/64/mysql/foo"]
- expected = "/opt/csw/mysql5/lib/sparcv9/mysql/foo"
- result = self.e.Emulate64BitSymlinks(runpath_list)
- self.assertTrue(expected in result, "%s not in %s" % (expected, result))
-
- def testEmulate64BitSymlinks_4(self):
- """No repeated paths because of symlink expansion"""
- runpath_list = ["/opt/csw/lib"]
- expected = "/opt/csw/lib"
- result = self.e.Emulate64BitSymlinks(runpath_list)
- self.assertEquals(1, len(result), "len(%s) != %s" % (result, 1))
-
- def testEmulateSymlinks_3(self):
- runpath_list = ["/opt/csw/bdb4"]
- expected = "/opt/csw/bdb42"
- result = self.e.Emulate64BitSymlinks(runpath_list)
- self.assertTrue(expected in result, "%s not in %s" % (expected, result))
-
- def testEmulateSymlinks_4(self):
- runpath_list = ["/opt/csw/bdb42"]
- expected = "/opt/csw/bdb42"
- not_expected = "/opt/csw/bdb422"
- result = self.e.Emulate64BitSymlinks(runpath_list)
- self.assertTrue(expected in result,
- "%s not in %s" % (expected, result))
- self.assertFalse(not_expected in result,
- "%s is in %s" % (not_expected, result))
-
- def testEmulateSymlinks_5(self):
- """Install time symlink expansion."""
- runpath_list = ["/opt/csw/lib/i386"]
- expected = "/opt/csw/lib"
- result = self.e.Emulate64BitSymlinks(runpath_list)
- self.assertTrue(expected in result, "%s not in %s" % (expected, result))
-
- def testEmulateSymlinks_6(self):
- """ExpandSymlink for /opt/csw/lib/i386."""
- runpath_list = ["/opt/csw/lib/i386"]
- expected = "/opt/csw/lib"
- not_expected = "/opt/csw/lib/i386"
- result = self.e.ExpandSymlink("/opt/csw/lib/i386",
- "/opt/csw/lib",
- "/opt/csw/lib/i386")
- self.assertTrue(expected in result, "%s not in %s" % (expected, result))
- self.assertFalse(not_expected in result,
- "%s is in %s" % (not_expected, result))
-
- def testSanitizeRunpath_1(self):
- self.assertEqual("/opt/csw/lib",
- self.e.SanitizeRunpath("/opt/csw/lib/"))
-
- def testSanitizeRunpath_2(self):
- self.assertEqual("/opt/csw/lib",
- self.e.SanitizeRunpath("/opt//csw////lib/"))
-
-
-
-class ParseDumpOutputUnitTest(unittest.TestCase):
-
- def test_1(self):
- expected = {
- 'RPATH set': True,
- 'RUNPATH RPATH the same': True,
- 'RUNPATH set': True,
- 'needed sonames': ('librt.so.1',
- 'libresolv.so.2',
- 'libc.so.1',
- 'libgen.so.1',
- 'libsocket.so.1',
- 'libnsl.so.1',
- 'libm.so.1',
- 'libz.so.1'),
- 'runpath': ('/opt/csw/lib/$ISALIST',
- '/opt/csw/lib',
- '/opt/csw/mysql5/lib/$ISALIST',
- '/opt/csw/mysql5/lib',
- '/opt/csw/mysql5/lib/$ISALIST/mysql'),
- 'soname': 'libmysqlclient.so.15',
- }
- self.assertEqual(expected,
- checkpkg.ParseDumpOutput(dump_1.DATA_DUMP_OUTPUT))
-
- def testEmpty(self):
- expected_runpath = ()
- self.assertEqual(
- expected_runpath,
- checkpkg.ParseDumpOutput(dump_2.DATA_DUMP_OUTPUT)["runpath"])
-
- def testRpathOnly(self):
- expected = {
- 'RPATH set': True,
- 'RUNPATH RPATH the same': False,
- 'RUNPATH set': False,
- 'needed sonames': ('librt.so.1',
- 'libresolv.so.2',
- 'libc.so.1',
- 'libgen.so.1',
- 'libsocket.so.1',
- 'libnsl.so.1',
- 'libm.so.1',
- 'libz.so.1'),
- 'runpath': ('/opt/csw/lib/$ISALIST',
- '/opt/csw/lib',
- '/opt/csw/mysql5/lib/$ISALIST',
- '/opt/csw/mysql5/lib',
- '/opt/csw/mysql5/lib/$ISALIST/mysql'),
- 'soname': 'libmysqlclient.so.15',
- }
- self.assertEqual(
- expected,
- checkpkg.ParseDumpOutput(dump_3.DATA_DUMP_OUTPUT))
-
-
-class SystemPkgmapUnitTest(unittest.TestCase):
-
- def testParsePkginfoLine(self):
- line = ('application CSWcswclassutils '
- 'cswclassutils - CSW class action utilities')
- expected = ('CSWcswclassutils',
- 'cswclassutils - CSW class action utilities')
- spkgmap = checkpkg.SystemPkgmap()
- self.assertEqual(expected, spkgmap._ParsePkginfoLine(line))
-
- def test_InferPackagesFromPkgmapLine(self):
- line = ("/opt/csw/sbin d none 0755 root bin CSWfping CSWbonobo2 "
- "CSWkrb5libdev CSWsasl CSWschilybase CSWschilyutils CSWstar "
- "CSWcommon CSWcacertificates CSWfacter")
- expected = ["CSWfping", "CSWbonobo2", "CSWkrb5libdev", "CSWsasl",
- "CSWschilybase", "CSWschilyutils", "CSWstar", "CSWcommon",
- "CSWcacertificates", "CSWfacter"]
- spkgmap = checkpkg.SystemPkgmap()
- self.assertEqual(expected, spkgmap._InferPackagesFromPkgmapLine(line))
-
- def test_InferPackagesFromPkgmapLine_2(self):
- line = ("/usr/lib/sparcv9/libpthread.so.1 f none 0755 root bin 41296 28258 "
- "1018129099 SUNWcslx")
- expected = ["SUNWcslx"]
- spkgmap = checkpkg.SystemPkgmap()
- self.assertEqual(expected, spkgmap._InferPackagesFromPkgmapLine(line))
-
- def test_InferPackagesFromPkgmapLine_3(self):
- line = ("/usr/lib/libCrun.so.1 f none 0755 root bin 63588 "
- "6287 1256043984 SUNWlibC")
- expected = ["SUNWlibC"]
- spkgmap = checkpkg.SystemPkgmap()
- self.assertEqual(expected, spkgmap._InferPackagesFromPkgmapLine(line))
-
- def test_InferPackagesFromPkgmapLine_4(self):
- line = ("/opt/csw/apache2/lib/libapr-1.so.0=libapr-1.so.0.3.8 s none "
- "CSWapache2rt")
- expected = ["CSWapache2rt"]
- spkgmap = checkpkg.SystemPkgmap()
- self.assertEqual(expected, spkgmap._InferPackagesFromPkgmapLine(line))
-
-
class ExtractorsUnitTest(unittest.TestCase):
def testExtractDescriptionFromGoodData(self):
@@ -296,68 +60,67 @@
self.assertEqual(expected, re.match(checkpkg.PSTAMP_RE, pstamp).groupdict())
-class CheckpkgManager2UnitTest(unittest.TestCase):
-
- def test_1(self):
- m = checkpkg.CheckpkgManager2("testname", "/tmp", ["CSWfoo"])
- tags = {
- "CSWfoo": [
- tag.CheckpkgTag("CSWfoo", "foo-tag", "foo-info"),
- ],
- }
- screen_report, tags_report = m.FormatReports(tags, [], [])
- expected = u'# Tags reported by testname module\nCSWfoo: foo-tag foo-info\n'
- self.assertEqual(expected, unicode(tags_report))
-
- def test_2(self):
- m = checkpkg.CheckpkgManager2("testname", "/tmp", ["CSWfoo"])
- tags = {
- "CSWfoo": [
- tag.CheckpkgTag("CSWfoo", "foo-tag", "foo-info"),
- tag.CheckpkgTag("CSWfoo", "bar-tag", "bar-info"),
- tag.CheckpkgTag("CSWfoo", "baz-tag"),
- ],
- }
- screen_report, tags_report = m.FormatReports(tags, [], [])
- expected = (u'# Tags reported by testname module\n'
- u'CSWfoo: foo-tag foo-info\n'
- u'CSWfoo: bar-tag bar-info\n'
- u'CSWfoo: baz-tag\n')
- self.assertEqual(expected, unicode(tags_report))
-
-
class SliceListUnitTest(unittest.TestCase):
def testOne(self):
l = [1, 2, 3, 4, 5]
s = 1
expected = [[1], [2], [3], [4], [5]]
- self.assertTrue(expected, checkpkg.SliceList(l, s))
+ self.assertTrue(expected, checkpkg_lib.SliceList(l, s))
def testTwo(self):
l = [1, 2, 3, 4, 5]
s = 2
expected = [[1, 2], [3, 4], [5]]
- self.assertTrue(expected, checkpkg.SliceList(l, s))
+ self.assertTrue(expected, checkpkg_lib.SliceList(l, s))
-class LddEmulartorUnitTest(unittest.TestCase):
+class SqliteUnitTest(unittest.TestCase):
+ "Makes sure that we can lose state between tests."
+
def setUp(self):
- self.pkgmap_mocker = mox.Mox()
- self.e = checkpkg.LddEmulator()
+ self.conn = sqlite3.connect(":memory:")
+ self.c = self.conn.cursor()
- def testResolveSoname_1(self):
- # runpath_list, soname, isalist, path_list, binary_path
- runpath_list = ["/opt/csw/bdb47/lib", "/opt/csw/lib"]
- soname = "foo.so.1"
- path_list = ["/opt/csw/lib", "/opt/csw/bdb47/lib", "/usr/lib"]
- binary_path = "unused"
- isalist = ["amd64"]
- result = self.e.ResolveSoname(runpath_list, soname, isalist,
- path_list, binary_path)
- self.assertEqual("/opt/csw/bdb47/lib", result)
+ def tearDown(self):
+ self.conn = None
+ def testCannotCreateTwoTables(self):
+ self.c.execute("CREATE TABLE foo (INT bar);")
+ self.assertRaises(
+ sqlite3.OperationalError,
+ self.c.execute, "CREATE TABLE foo (INT bar);")
+ def testOne(self):
+ self.c.execute("CREATE TABLE foo (INT bar);")
+
+ def testTwo(self):
+ self.c.execute("CREATE TABLE foo (INT bar);")
+
+
+class SqlobjectUnitTest(test_base.SqlObjectTestMixin, unittest.TestCase):
+
+ "Makes sure that we can lose state between methods."
+
+ class TestModel(sqlobject.SQLObject):
+ name = sqlobject.UnicodeCol(length=255, unique=True, notNone=True)
+
+ # This does not work. Why?
+ # def testCannotCreateTwoTables(self):
+ # self.TestModel.createTable()
+ # self.assertRaises(
+ # sqlite3.OperationalError,
+ # self.TestModel.createTable)
+
+ def testOne(self):
+ self.TestModel.createTable()
+
+ def testTwo(self):
+ self.TestModel.createTable()
+
+
if __name__ == '__main__':
+ # import logging
+ # logging.basicConfig(level=logging.DEBUG)
unittest.main()
@@ Diff output truncated at 100000 characters. @@
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
More information about the devel
mailing list