[csw-devel] psuedo sketch of web and mantis automation

[Ben Walton]

Excerpts from Maciej Bliziński's message of Sun Aug 21 03:27:08 -0400 2011:

> > As far as the catalog diff engine is concerned, both of these are
> > the same.  They'll appear as the drop of one package and the
> > addition of a new one.  The special case of _stub packages is
> > handled in the pseudo code.  The general case would require doing
> > a reverse index of package names.  The _stub handling rename
> > detection could be extended to:
> >
> > * Build a package to catalog map, based on previous catalog, of
> > every  package we're going to alter.
> 
> What's the definition of a package that can be altered?

This is just an object that we thing needs database changes to reflect
current state.  The results of the catalog diff will determine this.

> > * Do the same for the new catalog.
> > * Iterate through new packages and see if the package name exists in
> >  the old catalog.
> > * If we 'hit' in the index, add [p, q] to the rename list and remove
> >  them from the new and remove lists.
> 
> I think that at this point, we know the area in which we can look
> for corner cases.  It'll be best to talk in code: if you have a unit
> test with test cases, we can define the test cases and see what does
> the diff engine output.

Yes, that's the way to approach it.

> > We'd still want to filter stubs to a separate list first as those
> > packages shouldn't be displayed in the web db or mantis.
> 
> I'm thinking that mantis should not contain the packages (whatever
> the word 'package' means), but something that Dagobert calls
> bundles. They are roughly equivalent to upstream projects.

This is ideally what should happen.  This could be grouped based on
the SVN url that we embed.  It's a massive change from what we have
now though.  If we want to go this route, we should simply mark the
current stuff as legacy and start fresh with a) a normalized and sane
db for the web and b) a bug tracker that is better able to meet our
current (and hopefully future) needs.

> >> 3. Only the pkgname changes (CSWfoodev/foo_dev → CSWfoo-dev/foo_dev)
> >
> > This should be handled by an update operation.  As things are keyed on
> > the catalog name, a change (based on version comparison) will trigger
> > a refresh of file list, maintainer name, etc.  Mantis cares not about
> > package name so that won't be a problem.
> >
> > Reasonable?
> 
> I expected that you'll say that this scenario is invalid /
> forbidden.  If the pkgname changes, you need to provide a _stub for
> the catalogname.  We need some kind of detection of invalid
> scenarios, too.

Yes, the high level code that I've written can detect several things.
I'm starting to think that instead of an asynchronous job running
independently that we should instead place this in serial right after
generating a new catalog, but before mirror push.

> Let's continue by reviewing test cases, how does that sound?

Sounds good.  I need to refactor my code such that it's a testable
module.  The catalog diff part is already, but the actionable items is
not presently.  I'll work on that tonight.  The code is now capable of
spitting out a list of high level action items though.  I still need
to get a test environment running, but the logic and unit testing of
the high level part can proceed without that.

Given the catalog in current (5.9/sparc) and the catalog in
future/unstable (5.9/sparc), the following is the list that my code
generates:

Rename py_libxslt/CSWpylibxslt -> py_libxslt/CSWpy-libxslt
Rename berkeleydb3_doc/CSWbdb3doc -> berkeleydb3_doc/CSWbdb3-doc
Rename libserf_devel/CSWlibserf-devel -> libserf_dev/CSWlibserf-dev
Rename pcre_devel/CSWpcre-devel -> libpcre_dev/CSWlibpcre-dev
Rename zlib/CSWzlib -> libz1/CSWlibz1
Rename pm_x11protocol/CSWpmx11protocol -> pm_x11_protocol/CSWpm-x11-protocol
Rename libxslt_devel/CSWlibxsltdevel -> libxslt_dev/CSWlibxslt-dev
Rename libdbusdev/CSWlibdbusdev -> libdbus_dev/CSWlibdbus-dev
Rename apr/CSWapr -> libapr1_0/CSWlibapr1-0
Rename libgnugetopt/CSWlibgnugetopt -> libgnugetopt0/CSWlibgnugetopt0
Rename id3lib/CSWid3lib -> libid3_utils/CSWlibid3-utils
Rename libdnet/CSWlibdnet -> libdnet1/CSWlibdnet1
Rename screen/CSWscrn -> screen/CSWscreen
Rename lzma/CSWlzma -> xz/CSWxz
Rename eventlog/CSWeventlog -> eventlog_dev/CSWeventlog-dev
Rename pm_version/CSWpmversion -> pm_version/CSWpm-version
Rename cadaver/CSWcadav -> cadaver/CSWcadaver
Rename apr_util/CSWapr-util -> libaprutil1_ldap/CSWlibaprutil1-ldap
Rename pcre_rt/CSWpcrert -> libpcreposix0/CSWlibpcreposix0
Rename libiconv/CSWiconv -> libiconv2/CSWlibiconv2
Rename libevent_devel/CSWlibevent-devel -> libevent_dev/CSWlibevent-dev
Rename sqlite3rt_stub/CSWsqlite3rt -> libsqlite3_0/CSWlibsqlite3-0
Rename libfbopenssl/CSWlibfbopenssl -> libfbopenssl0/CSWlibfbopenssl0
Rename glew/CSWglew -> libglew_dev/CSWlibglew-dev
Rename python_devel/CSWpython-devel -> python_dev/CSWpython-dev
Rename libdbus/CSWlibdbus -> libdbus1_3/CSWlibdbus1-3
Rename libffi/CSWlibffi -> libffi5/CSWlibffi5
Rename sqlite3devel_stub/CSWsqlite3devel -> libsqlite3_dev/CSWlibsqlite3-dev
Rename libdatrie/CSWlibdatrie -> libdatrie1/CSWlibdatrie1
Register py_mysql/CSWpy-mysql
Register idle_27/CSWidle27
Register libexpat0/CSWlibexpat0
Register libexpat1/CSWlibexpat1
Register libz_dev/CSWlibz-dev
Register pm_test_object/CSWpm-test-object
Register libaprutil1_dbd_freetds/CSWlibaprutil1-dbd-freetds
Register pm_ppix_regexp/CSWpm-ppix-regexp
Register python_27_tk/CSWpython27-tk
Register python_27_dev/CSWpython27-dev
Register berkeleydb3_dev/CSWbdb3-dev
Register python_31_devel/CSWpython31-devel
Register libdatrie_dev/CSWlibdatrie-dev
Register opencsw_policy/CSWopencsw-policy
Register libslp1/CSWlibslp1
Register libbz2_dev/CSWlibbz2-dev
Register libpython3_1_1_0/CSWlibpython3-1-1-0
Register rb18_activerecord_3_0_7/CSWrb18-activerecord-3-0-7
Register libbz2_1_0/CSWlibbz2-1-0
Register py_xapian/CSWpy-xapian
Register libglew1_6/CSWlibglew1-6
Register rb18_mysql_2_8_1/CSWrb18-mysql-2-8-1
Register rtmpdump/CSWrtmpdump
Register libexpat_dev/CSWlibexpat-dev
Register rb18_i18n_0_5_0/CSWrb18-i18n-0-5-0
Register pm_string_format/CSWpm-string-format
Register libmp4v2_dev/CSWlibmp4v2-dev
Register libpython2_7_1_0/CSWlibpython2-7-1-0
Register libiconv_dev/CSWlibiconv-dev
Register rb18_activemodel_3_0_7/CSWrb18-activemodel-3-0-7
Register libwsutil0/CSWlibwsutil0
Register py_virtualenv/CSWpy-virtualenv
Register libxslt1/CSWlibxslt1
Register xmms_faad2/CSWxmms-faad2
Register libapr_dev/CSWlibapr-dev
Register py_docutils/CSWpy-docutils
Register py_jinja2/CSWpy-jinja2
Register pm_ppix_utilities/CSWpm-ppix-utilities
Register pm_ppi/CSWpm-ppi
Register libaprutil_dev/CSWlibaprutil-dev
Register libfaad2/CSWlibfaad2
Register libmp4v2_0/CSWlibmp4v2-0
Register libmp4v2_utils/CSWlibmp4v2-utils
Register openslp_devel/CSWopenslp-devel
Register libwireshark0/CSWlibwireshark0
Register rb_xapian/CSWrb-xapian
Register proj_data/CSWproj-data
Register libmpc2/CSWlibmpc2
Register libmpfr_dev/CSWlibmpfr-dev
Register libfbopenssl_dev/CSWlibfbopenssl-dev
Register libidn/CSWlibidn
Register python_27/CSWpython27
Register libmysqlclient_r15/CSWlibmysqlclient-r15
Register libcharset1/CSWlibcharset1
Register libffi_dev/CSWlibffi-dev
Register libclam6_dev/CSWlibclam6-dev
Register libexslt0/CSWlibexslt0
Register pinentry_gtk2/CSWpinentry-gtk2
Register pcp/CSWpcp
Register rb18_ratom_0_6_7/CSWrb18-ratom-0-6-7
Register libaprutil1_dbd_sqlite3/CSWlibaprutil1-dbd-sqlite3
Register libdnet_dev/CSWlibdnet-dev
Register py_dbutils/CSWpy-dbutils
Register pm_netaddr_ip/CSWpm-netaddr-ip
Register libevtlog0/CSWlibevtlog0
Register idle_31/CSWidle31
Register pm_test_subcalls/CSWpm-test-subcalls
Register libserf1_0/CSWlibserf1-0
Register xapian/CSWxapian
Register rb18_activesupport_3_0_7/CSWrb18-activesupport-3-0-7
Register py_openssl/CSWpy-openssl
Register libaprutil1_dbd_odbc/CSWlibaprutil1-dbd-odbc
Register libproj0/CSWlibproj0
Register libtre5/CSWlibtre5
Register libgmpxx4/CSWlibgmpxx4
Register libmpfr4/CSWlibmpfr4
Register libgmp_dev/CSWlibgmp-dev
Register py_pygments/CSWpy-pygments
Register libxapian22/CSWlibxapian22
Register libiconv_utils/CSWlibiconv-utils
Register libgmp10/CSWlibgmp10
Register rb18_builder_3_0_0/CSWrb18-builder-3-0-0
Register libmpc_dev/CSWlibmpc-dev
Register py_subvertpy/CSWpy-subvertpy
Register libid3_3_8_3/CSWlibid3-3-8-3
Register libwiretap0/CSWlibwiretap0
Register libfaad_dev/CSWlibfaad-dev
Register libmysqlclient15/CSWlibmysqlclient15
Register libxapian_dev/CSWlibxapian-dev
Register pm_email_address/CSWpm-email-address
Register py_bridge/CSWpy-bridge
Register python_31_tk/CSWpython31-tk
Register tcllib/CSWtcllib
Register rb18_json_1_5_3/CSWrb18-json-1-5-3
Register librtmp_dev/CSWlibrtmp-dev
Register librtmp0/CSWlibrtmp0
Register pm_perl_ostype/CSWpm-perl-ostype
Register proj_dev/CSWproj-dev
Register libaprutil1_0/CSWlibaprutil1-0
Register python_31/CSWpython31
Register libdatrie_utils/CSWlibdatrie-utils
Register pm_perl_critic/CSWpm-perl-critic
Register cas_etcshells/CSWcas-etcshells
Register py_sphinx/CSWpy-sphinx
Register libgnugetopt_dev/CSWlibgnugetopt-dev
Register tre_dev/CSWtre-dev
Register libaprutil1_dbm_db/CSWlibaprutil1-dbm-db
Register pm_pod_spell/CSWpm-pod-spell
Register pm_dbi2/CSWpmdbi
Register guilt/CSWguilt
Register libid3_dev/CSWlibid3-dev
Register rb18_libxml_1_1_4/CSWrb18-libxml-1-1-4
Update easytag/CSWeasytag
Update enscript/CSWenscript
Update asciidoc/CSWasciidoc
Update lftp/CSWlftp
Update xorriso/CSWxorriso
Update expat/CSWexpat
Update libmagic_data/CSWlibmagic-data
Update libclam6/CSWlibclam6
Update git_completion/CSWgit-completion
Update xjobs/CSWxjobs
Update unrar/CSWunrar
Update mysql5/CSWmysql5
Update memconf/CSWmemconf
Update py_nose/CSWpy-nose
Update gawk/CSWgawk
Update dhcp/CSWdhcp
Update cswclassutils/CSWcswclassutils
Update imapproxy/CSWimapproxy
Update proj/CSWproj
Update ruby_augeas/CSWrubyaugeas
Update mercurial/CSWmercurial
Update py_tz/CSWpy-tz
Update py_libmagic/CSWpy-libmagic
Update geoip/CSWgeoip
Update openssh_client/CSWosshclient
Update gxmessage/CSWgxmessage
Update lsof/CSWlsof
Update gfile/CSWgfile
Update sqlite3/CSWsqlite3
Update git_cvs/CSWgit-cvs
Update python_test/CSWpython-test
Update libgd_dev/CSWlibgd-dev
Update py_xcbproto/CSWpyxcbproto
Update clusterssh/CSWclusterssh
Update pinentry/CSWpinentry
Update libflac8/CSWlibflac8
Update clamav/CSWclamav
Update bind_utils/CSWbindutils
Update libgd2/CSWlibgd2
Update python/CSWpython
Update wgetpaste/CSWwgetpaste
Update mysql5devel/CSWmysql5devel
Update idle/CSWidle
Update mailscanner/CSWmailscanner
Update dcc/CSWdcc
Update elinks/CSWelinks
Update bash/CSWbash
Update zsh/CSWzsh
Update bar/CSWbar
Update gd/CSWgd
Update vsftpd/CSWvsftpd
Update git_svn/CSWgit-svn
Update netcat/CSWnetcat
Update py_hachoir_parser/CSWpy-hachoir-parser
Update rlwrap/CSWrlwrap
Update bind_chroot/CSWbindchroot
Update xmlstarlet/CSWxmlstarlet
Update gcal/CSWgcal
Update libpcrecpp0/CSWlibpcrecpp0
Update pca/CSWpca
Update libmagic_dev/CSWlibmagic-dev
Update liblzma5/CSWliblzma5
Update bind_dev/CSWbind-dev
Update mysql5client/CSWmysql5client
Update mysql5rt/CSWmysql5rt
Update liblzma_dev/CSWliblzma-dev
Update git_doc/CSWgit-doc
Update tnef/CSWtnef
Update libflac_dev/CSWlibflac-dev
Update openssl_rt/CSWosslrt
Update ttcp/CSWttcp
Update geoip_dev/CSWgeoip-dev
Update lighttpd/CSWlighttpd
Update flac/CSWflac
Update py_graph/CSWpy-graph
Update cas_initsmf/CSWcas-initsmf
Update git_dev/CSWgit-dev
Update python_tk/CSWpython-tk
Update libpython2_6_1_0/CSWlibpython2-6-1-0
Update pymysql/CSWpymysql
Update berkeleydb3/CSWbdb3
Update pm_netaddrip/CSWpmnetaddrip
Update xmms_flac/CSWxmms-flac
Update libmagic1/CSWlibmagic1
Update clamavsigs/CSWclamavsigs
Update openssl_devel/CSWossldevel
Update namebench/CSWnamebench
Update tcsh/CSWtcsh
Update bzip2/CSWbzip2
Update facter/CSWfacter
Update pbzip2/CSWpbzip2
Update libbind/CSWlibbind
Update miltergreylist/CSWmiltergreylist
Update gitk/CSWgitk
Update libgmp/CSWlibgmp
Update git_emacs/CSWgit-emacs
Update libevent2_0_5/CSWlibevent2-0-5
Update nicstat/CSWnicstat
Update dhcp_dev/CSWdhcp-dev
Update bind/CSWbind
Update tmux/CSWtmux
Update py_sqlobject/CSWpy-sqlobject
Update faad2/CSWfaad2
Update git/CSWgit
Update libxslt/CSWlibxslt
Update dbus/CSWdbus
Update openssl_utils/CSWosslutils
Update libmpfr/CSWlibmpfr
Update openssh/CSWossh
Update openssl/CSWossl
Update cmake/CSWcmake
Update jpegoptim/CSWjpegoptim
Update spamassassin/CSWspamassassin
Update mgar/CSWmgar
Update nginx/CSWnginx
Update git_gui/CSWgit-gui
Update xcbproto/CSWxcbproto
Update pydocutils/CSWpydocutils
Update pm_html_parser/CSWpm-html-parser
Update cas_migrateconf/CSWcas-migrateconf
Update phpmyadmin/CSWphpmyadmin
Update pcre/CSWpcre
Update pm_htmlparser/CSWpmhtmlparser
Update libpcre0/CSWlibpcre0
Update unzoo/CSWunzoo
Removing drupal/CSWdrupal
Removing libclam6_devel/CSWlibclam6-devel
Removing libsmi_mibs_stub/CSWlibsmimibs
Removing mediawiki/CSWmediawiki
Removing pm_dbi_stub/CSWpmdbi
Removing buildbot/CSWbuildbot
Removing zope/CSWzope
Removing libidn_stub/CSWlibidn
Removing py_atomixlib/CSWpy-atomixlib
Removing mysql5bench/CSWmysql5bench
Removing squirrelmail/CSWsquirrelmail

Thanks
-Ben
--
Ben Walton
Systems Programmer - CHASS
University of Toronto
C:416.407.5610 | W:416.978.4302