[csw-maintainers] Catalog checks via REST

[Maciej (Matchek) Bliziński]

2011/12/29 Dagobert Michelsen <dam at opencsw.org>:
> I adjusted my code to retrieve "depends" and "i_depends" for each package and
> I think this is not practical: getting all the data takes some hours
> whereas parsing the catalog file takes only a second. I really like the
> REST approach, but at the current performance the checks are just not
> feasible.

That's true. I've been planning to extend the mysql table structure to
accommodate the dependency (and more) information. I kept on being
distracted by things such as GCC and now MySQL and OpenSSL (dynamic
prototypes). To get the schema change done, I'll have to pause the
work on packages and get back to tool development.

Meanwhile, did you do any measurements, how long would it take to
retrieve a whole catalog? Is it a day, or a week, or a month?

Another observation is that one query runs on one core only. The
buildfarm has many cores, so you can gain performance by making
multiple calls in parallel, which will be processed in parallel. This
can probably shorten the required time five- to tenfold.

Maciej