[csw-maintainers] TeXLive packaging take almost 3 days

[Peter FELECAN]

Dagobert Michelsen <dam at opencsw.org> writes:

> Hi Peter,
>
> Am 19.11.2012 um 18:48 schrieb Peter FELECAN <pfelecan at opencsw.org>:
>> Dagobert Michelsen <dam at opencsw.org> writes:
>>> The numbers are… impressive. I have never optimized pathfilter as it usually
>>> runs a second for average packages, so it probably leaves much room. Please use
>>>  DEBUG_PACKAGING=1 mgar repackage
>>> to get the actual pathfilter invocation and grab the file it is applied to so I can
>>> rerun and profile it. I can then happily try to speed it up.
>> 
>> The script containing the invocation of the 2 utilities, as extracted
>> from the packaging log, can be found on the build farm at
>> ~pfelecan/opencsw/texlive/trunk/4o; beware that the utilities are a
>> little bit hacked in my instance of .buildsys/v2…
>
> This is now roughly 40 times faster by precompiling the regexps:
>   http://sourceforge.net/apps/trac/gar/changeset/19753
>
> There is probably more to gain, but this may help on the first stage.
> cswproto depends on pkgproto which already takes some time and uses
> precompiled regexps, but it is only called once during packaging
> where pathfilter is called for every package.
>
> Please drop me a note how it goes, I will also take more looks on the code.

Thank you.

I confirm a 50 times optimization for pathfilter.

As for cswproto, still a hog, here is some additional profiling data:

Total Elapsed Time = 1184.726 Seconds
  User+System Time = 1060.576 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 98.1   1040. 1040.6 112180   0.0093 0.0093  main::is_common
 0.09   0.980  0.980 112164   0.0000 0.0000  main::exclude
 0.01   0.069  0.236      7   0.0099 0.0337  main::BEGIN

As you can see the time is spent is determining if the components are
part of the common package. I understood that in the future we get rid
of that but in the mean time we need to find a quicker way to determine
that relation. With the current optimization we probably can package
TeXLive in 1.5 days which is still too much...
-- 
Peter