[csw-maintainers] Assessing catalog quality

[Philip Brown]

On Tue, Jun 28, 2011 at 1:33 PM, Peter FELECAN <pfelecan at opencsw.org> wrote:
> Philip Brown <phil at bolthole.com> writes:
>
>> How do you measure the "quality" of a programmer's work?
>> At one point, IBM said, "We'll measure it based on KLOC".
>> Epic Fail.
>>
>> After 40 years, there is still no uniform "metric" for that sort of
>> thing. Nor is it possible to uniformly create one.
>> Some kinds of quality are simply not possible to reduce to "metrics".
>
> LOC is the beginning of effort measure. Not really a failure except when
> used as a silver bullet. (...)

yes and no. it can *sometimes* be useful as a relative measure. but
the problem lies in the management axiom of (make sure you reward what
you are really interested in).
When "more KLOC=good", programmers start to needlessly bloat programs.
  (the extra irony being that in reality, usually a program is good
specifically because it has fewer lines)
That type of scenario is true failure of applying metrics to a problem.

Probably many others here have also experienced the unwanted side
effects of other metrics gone bad. For example;
When metrics management people decide "bugs closed quickly=good", then
support people start "accidentally" closing bug reports that have been
open long enough to make them start looking bad.
And, curiously, there is no way to re-open the original bug, you must
create a whole new ticket. Resetting the time factor. What an amazing
coincidence

Similarly, when companies decide "few bugs filed=good", and motivate
staff based on low total bug count... and then the staff then proceeds
to make it almost impossible for people to actually open a bug.

Attempting to "increase quality" based on blind application of
metrics, can sometimes lead to places not originally intended. Even
when there is not deliberate subversion of the metrics like the above
examples... sometimes they subconciously skew people towards unwanted
behaviour.  So, metrics need to be applied cautiously.