[csw-maintainers] Assessing catalog quality

[Philip Brown]

On Sat, Jun 25, 2011 at 5:04 PM, Jonathan Craig <jcraig at opencsw.org> wrote:
> 2011/6/25 Maciej Bliziński <maciej at opencsw.org>
>>
>> Here's my question to maintainers: What metrics would you apply to the
>> catalog for the purpose of catalog quality evaluation?  Which aspects
>> of packages, or interactions between packages would you take into
>> consideration?  Do you have any already working code that could be
>> used for this purpose?
>
> Here are some thoughts, but I'm not knowledgeable enough of the
> internals of package publishing to know if they are determinable.
>
> Positive measures could include:
>   New package submission - Count of packages that didn't previously exist
>   Updated packages           - Count of packages that increment their
> version number
>
> Negative measure could include:
>   Patched packages           - Count of packages submitted that didn't
> increment their version number
>

[...]

This seems to be numerical "quality" based.  Which, while a good
thing, is not all I was concerned with.
What you wrote, is in some ways more "quantity/freshness" based.
In addition to the things you wrote, I was also thinking of overall
consistency between packages. In ways that *cannot* be measured by
simple pre-release scripts.
Many things can be. But not everything.
Random things like... i dunno.. naming of rc files, and placement, in
complex cases. there's other random small stuff like that.

To me, its the difference between, "hey lets just automate the BSD
ports tree to autocompile stuff for solaris instead", vs
"Hey lets look at the BSD ports tree, *package by package*, and ensure
that BSD-isms are replaced with more Solaris standard stuff".

Yes it's possible to write a robo-compiler mechanism.  But it yields
"higher quality" for a Solaris user, to have the packages individually
considered, *on top of* the initial grunt work done by robo-porting.


Yes, metrics are good!  But not as an end in and of itself.  To take
the biggest programmer metric failure:

How do you measure the "quality" of a programmer's work?
At one point, IBM said, "We'll measure it based on KLOC".
Epic Fail.

After 40 years, there is still no uniform "metric" for that sort of
thing. Nor is it possible to uniformly create one.
Some kinds of quality are simply not possible to reduce to "metrics".