[csw-maintainers] Faster catalog generation

[Maciej Bliziński]

On Tue, May 21, 2013 at 12:29:13PM +0200, Peter FELECAN wrote:
> "Maciej (Matchek) Bliziński" <maciej at opencsw.org> writes:
> 
> > I spent some time yesterday working on the buildfarm[1], and managed
> > to remove one bottleneck and cut down the catalog generation time from
> > ~80 minutes to ~32 minutes. The new (and current) bottleneck seems to
> > be what I described in [2]: mass unlinking and relinking of files.
> >
> > There's also a new HTTP endpoint:
> > /rest/catalogs/([^/]+)/(sparc|i386)/(SunOS[^/]+)/for-generation/
> >
> > (Prefix: http://buildfarm.opencsw.org/pkgdb/)
> >
> > It's quite quick to return and returns all information necessary to
> > generate a catalog.
> >
> > Maciej
> >
> > [1] https://sourceforge.net/apps/trac/gar/changeset/21086
> > [2] http://lists.opencsw.org/pipermail/maintainers/2013-May/018016.html
> 
> Does this invalidate the project that you proposed in [2] ?

No, quite the opposite! When I proposed [2], the unlinking was not yet
the bottleneck, because there was another bottleneck that was way worse:
unnecessary data transfers and deserialization.  Now that I've optimized
it away, the mass unlinking and relinking is the current standing
bottleneck.

Maciej