[csw-maintainers] Catalog checks via REST

[Maciej (Matchek) Bliziński]

Hello maintainers,

This is a follow-up to the recent incident with a dependency loop in
the package catalog.

During the last camp, we've discussed writing own package checks in
different languages than Python. To recap, interfacing with other
languages is possible but requires additional work. As an intermediate
step, I offered access to the package data via REST, so that people
can write own checks in any languages they want. If/when there is
enough traction for checks in other languages, we can go back to the
idea of creating bindings to other languages.

The same idea can be applied to catalog checks.

I've written a proof-of-concept script in Perl, which pulls catalog
information and pretty-prints it. To check a catalog, the script would
do some kind of inspection of the data structure and draw some
conclusions (e.g. there's a dependency loop).

The script is available from our code repository:
http://sourceforge.net/apps/trac/gar/browser/csw/mgar/gar/v2/lib/web/examples/catalog_data.pl

One of the concerns were "I don't know REST". The answer is that
there's nothing to learn. Here's the script body:

use LWP::Simple;
use JSON;
use Data::Dumper;

my $url = 'http://buildfarm.opencsw.org/pkgdb/rest/catalogs/unstable/sparc/SunOS5.9/';
my $json_string = get $url;
die "Couldn't get $url" unless defined $json_string;
my $catalog_data = decode_json $json_string;
print Dumper($catalog_data);

The current situation with the catalogs is that we don't have good
diagnostics of catalogs. If something goes wrong, the new catalog is
not pushed, but we have no feedback about what was wrong, until
Dagobert pulls out a cron email from the depths of his email inbox.

Would anybody be so kind and write a script which polls the catalogs
and check them for problems? The script above can serve as a starting
point (assuming you want to write in Perl, but you can do the same in
any other language that has a JSON library).

Maciej