[csw-maintainers] Full package statistics via REST

[Maciej (Matchek) Bliziński]

During the Summercamp, there was a discussion about plugins for
checkpkg.  People wanted to write package checks, but couldn't or
didn't want to write in Python.  I suggested, as an intermediate step,
getting package statistics over REST and writing checks in the
language of choice.

I've just committed r15980, which make the web interface export full
package statistics via REST.

What does it mean?

It means that if you make a GET request at the REST package details
address, you will get a JSON-serialized data structure, which
represents everything that the buildfarm database knows about the
package.

You can see the JSON data printed to screen by calling this address:

curl -s 'http://buildfarm.opencsw.org/pkgdb/rest/srv4/3a483f192e2e9d788e922895ef1261e9/'

Here's an interactive Python example:

>>> import urllib2
>>> import json
>>> url = "http://buildfarm.opencsw.org/pkgdb/rest/srv4/3a483f192e2e9d788e922895ef1261e9/"
>>> p = json.load(urllib2.urlopen(url))
>>> p["pkg_stats"]["binaries_dump_info"][0]
{u'base_name': u'9menu', u'RUNPATH RPATH the same': True, u'runpath':
[], u'RPATH set': False, u'needed sonames': [u'libXext.so.0',
u'libX11.so.4', u'libsocket.so.1', u'libnsl.so.1', u'libc.so.1'],
u'path': u'opt/csw/opt/csw/bin/9menu', u'RUNPATH set': False}

An example in Perl, which gets the data and pretty prints them to screen:

use strict;
use JSON;
use LWP::Simple qw(get);
use Data::Dumper;
my $url = "http://buildfarm.opencsw.org/pkgdb/rest/srv4/3a483f192e2e9d788e922895ef1261e9/";
my $json_repr = get $url;
my $pkg_data = decode_json($json_repr);
print Dumper($pkg_data);

I encourage enthusiasts of other languages to post their examples.

Once you fetch and deserialize that data, you know about the package
everything that checkpkg knows.  You can write your own checking
function and run it across any packages.  If you want to fetch a list
of packages from a specific catalog, call an URL like this one...

http://buildfarm.opencsw.org/pkgdb/rest/catalogs/unstable/sparc/SunOS5.9/

...and deserialize JSON.  It takes a while to get that data, so be patient.

The REST interface is generally slow at the moment.  I haven't spent
any time on optimization.  I decided I'll spend time on performance
when there's a need for it.  So far, the consumers of the data are
mostly automation, cronjobs, etc.

If you have any questions, feel free to ask.  Enjoy writing your
checks and analyzing the package data!

Maciej

http://sourceforge.net/apps/trac/gar/changeset/15980