[csw-maintainers] web visibility of packages

[Maciej Bliziński]

2011/8/6 Ben Walton <bwalton at opencsw.org>:
> I'm now looking at the web code to display package info.  Unless we're
> going to modify the way things work so that we can pull primary info
> from the catalog database backing csw-upload-pkg, we need to complete
> the mantis integration at the same time as the web package database.
> The wordpress code uses both databases.
>
> Maciej: Would you like to pool resources on this?  This _could_ be
> done as another restful interface that is called by the backend of
> csw-upload-pkg.

I'd suggest one modification: Instead of making it part of pkgdb (the
backend of csw-upload-pkg), it could be a separate program which would
reach out to both pkgdb and mantis for information, and that could be
run asynchronously to package uploads.  This way, each component would
be itself simpler, which is a point for maintainability.

> For each package in the set of things uploaded, the
> backend would make a request of the form:
>
> /register/pkgname/?maintainer=...&...
> /deregister/pkgname
>
> A re-registration (eg: version bump) would force in fresh
> information.  This would accommodate takeovers as well.
>
> Separating this logic to a separate unit would have the advantage that
> when we're ready to switch BTS we have a nice modular place to do it.
>
> Additionally, this interface should manage both the mantis and the
> other database as a single unit.  This would make things a
> one-stop-shop for the csw-upload-pkg backend and make it easier to
> fully merge these databases in the future.
>
> Am I missing useful commands to the restful interface?  It might make
> sense to have some sort of generalized query too, but I see that as
> less important than stuffing info into the database.

There is currently no separate documentation of the RESTful interface.
 However, the code itself is (I think) not that hard to read.

http://sourceforge.net/apps/trac/gar/browser/csw/mgar/gar/v2/lib/web

If you look at pkgdb_web,py, you'll see the list of URLs at line 16,
which already gives you an idea what kind of queries can you currently
make.  Each URL maps to a class, and if you look up the class, you can
see what HTTP methods are implemented.  Usually it's only GET, because
the public pkgdb interface is read-only.  All the code that allows
writing is in releases_web.py, which is available only on the internal
network (see example-apache.conf).

Looking at pkgdb_web.py, you can see this line:

r'/rest/srv4/([0-9a-f]{32})/', 'RestSrv4Detail',

This means that you can call
http://buildfarm.opencsw.org/pkgdb/rest/srv4/<md5-sum>/ and retrieve
package details. For example:

maciej at login [login]:~ > gmd5sum <package>.pkg.gz
688d3a5b038d172a36a4cba30272441f  <package>.pkg.gz

How that we can identify the package by its md5 sum, we can query
pkgdb. The last echo is to inject a newline character.

maciej at login [login]:~ > curl -s
http://buildfarm.opencsw.org/pkgdb/rest/srv4/688d3a5b038d172a36a4cba30272441f/
; echo
{"maintainer_full_name": null, "version_string":
"1.8.7p334,REV=2011.03.24", "basename":
"ruby18-1.8.7p334,REV=2011.03.24-SunOS5.9-sparc-CSW.pkg.gz",
"maintainer_email": "bwalton at opencsw.org", "mtime": "2011-04-01
17:42:37", "file_basename":
"ruby18-1.8.7p334,REV=2011.03.24-SunOS5.9-sparc-CSW.pkg.gz", "arch":
"sparc", "osrel": "SunOS5.9", "size": 1365872, "md5_sum":
"688d3a5b038d172a36a4cba30272441f", "pkgname": "CSWruby18", "rev":
"2011.03.24", "filename_arch": "sparc", "version":
"1.8.7p334,REV=2011.03.24", "catalogname": "ruby18"}

A similar example which parses JSON and pretty-prints it:

maciej at login [login]:~ > python -c "import json, urllib2, pprint;
pprint.pprint(json.load(urllib2.urlopen('http://buildfarm.opencsw.org/pkgdb/rest/srv4/688d3a5b038d172a36a4cba30272441f/')))"
{u'arch': u'sparc',
 u'basename': u'ruby18-1.8.7p334,REV=2011.03.24-SunOS5.9-sparc-CSW.pkg.gz',
 u'catalogname': u'ruby18',
 u'file_basename': u'ruby18-1.8.7p334,REV=2011.03.24-SunOS5.9-sparc-CSW.pkg.gz',
 u'filename_arch': u'sparc',
 u'maintainer_email': u'bwalton at opencsw.org',
 u'maintainer_full_name': None,
 u'md5_sum': u'688d3a5b038d172a36a4cba30272441f',
 u'mtime': u'2011-04-01 17:42:37',
 u'osrel': u'SunOS5.9',
 u'pkgname': u'CSWruby18',
 u'rev': u'2011.03.24',
 u'size': 1365872,
 u'version': u'1.8.7p334,REV=2011.03.24',
 u'version_string': u'1.8.7p334,REV=2011.03.24'}

We can add more fields to the JSON structure, or add more URLs.  This
RESTful interface has been written by my pretty much solo, and has not
been reviewed.  I only implemented as much as I needed to get
csw-upload-pkg to work.  I'll be happy to receive comments on how to
develop it further.

Maciej