trying to improve mandoc Solaris support

[Ingo Schwarze via buildfarm]

Hi Joerg,

Joerg Schilling wrote on Thu, Mar 19, 2015 at 12:00:15PM +0100:

> I did recently run a test with the mandoc program on Solaris
> and tried to check the results.  In general, the tbl support
> is extremely bad.  Many tables are unreadable

It is true that tbl(7) support is still not as good as mdoc(7)
support.  That is mostly due to the fact that BSD manuals rarely
use tables, but it can certainly be improved.  In fact, i implemented
some improvements recently, for example horizontal centering of
tables as a whole.

> or disappear from the output.

That is one of the things i recently improved, so it should be better
in mandoc-1.13.3 than in 1.13.2.  I don't claim that all of such
issues are gone, though.

> In general, the mandoc _format_ (introduced with BSD-4.4-lite)

I assume you are talking about the mdoc(7) format designed and
implemented by Cynthia Livingston - as opposed to the traditional
Version 7 AT&T UNIX man(7) format.

> is not a problem on Solaris if you use the enhanced man program
> I created some years ago and that is published together with
> Schillix-ON, see:
> 
> https://sourceforge.net/p/schillix-on/schillix-on/ci/default/tree/usr/src/cmd/man/
> 
> What I did was to take the mandoc troff macros from BSD-4.4-lite (those
> that still work with original troff) and added a few minor fixes (e.g.
> a Y2000 fix).

I wasn't aware of that and have not checked your work in detail,
but i have a rough idea of the quality of Cynthia's original macro
implementation because it is still in use in Heirloom roff; Carsten
Kunze is working on improvements right now.

These macros are usable for historical manuals, but less so for modern
manuals.  For example, they still have the nine argument limit, and
some macros are not parsed and/or not callable that are generally
assumed to be in modern manuals.  Basically, this old version of
the macros lacks all the improvements that Werner Lemberg (GNU) and
Ruslan Ermilov (FreeBSD) did for groff, in particular the rewrite
for groff-1.17, which many modern manuals now depend on.  Note that
Werner was kind enough to not change the license from BSD to GPL
on the occasion of his rewrite, but the macros need groff features
that are not available in older roff implementations.

> What I don't understand is why the mandoc program was written at all.

There are a number of reasons:

 1. Kristaps wasn't satisfied with groff's HTML output quality,
    and due to the basic architecture of groff, that's almost
    impossible to improve.
 2. OpenBSD wanted to remove groff from the bases system, both
    for licensing reasons, to get rid of a major chunk of code
    written in C++ (which is generally considered an inappropriate
    language for the OpenBSD base system), and because groff is
    relatively hard to maintain properly.
 3. While groff is a rather fast typesetting system, it is a rather
    slow documentation formatter (though still *much*, much better
    than DocBook-based toolchains which are positively ridiculously
    slow in addition to producing man(7) output of abysmal quality;
    never use DocBook for anything!).  Typically, mandoc is by a
    factor of 3 to 20 faster than groff, depending on the source
    language and manual size.  That allowed switching from
    installing preformatted manuals to installing manual sources
    and formatting on the fly, even on older, slower architectures.
    That also sped up the system build, which is considered an
    important point in OpenBSD's developer-centric culture.
 4. mandoc has support for semantic searching in apropos(1),
    see my conference presentations available from mdocml.bsd.lv
    for details.

Anyway, these arguments convinced FreeBSD, OpenBSD, NetBSD, Minix 3
and illumos to use mandoc as the system default formatter.

> Compared to most software that has been recently developped,
> it looks nice,

Thanks.  :)

> but if I remember correctly, the sourcecode for "mandoc" is 2-3x more
> that the sourcecode for "man", "troff", "tbl", "soelim", "col" ...
> all together

It is still much smaller than groff, even though it does support
a subset of roff(7) that is now and then used in manuals (including
full support for user-defined macros, conditional and numerical
expressions).

> and the current state seems to be that only 80-90% of the Solaris
> man pages work with the mandoc program. 

The sounds like Solaris manuals tend to use some features that are
less commonly used elsewhere - now that i have an account on your
build cluster, i can have a look what exactly that is.

To put this into perspective, all OpenBSD manuals work with mandoc,
and i'm not aware of any current complaints with respect to FreeBSD,
NetBSD, or illumos base system manuals.  The OpenBSD ports tree
contains about 9000 third-party packages; more than 3000 of those
contain manuals; less than 300 still use groff for formatting, and
the vast majority of those 300 are DocBook generated manuals.  There
are plans to improve mandoc to deal with even DocBook insanity, but
that will still need a bit of work.

> The former problem that groff is under GPL is no longer a problem
> since the original AT&T troff has been made OpenSource under a free
> license together with OpenSolaris in June 2005.

CDDL-licensed software is considered free software by the OSI,
but the OpenBSD project regards the CDDL as a non-free license,
and CDDL-licensed code cannot be used in the OpenBSD base system,
but only in the OpenBSD ports tree.  The CDDL has at least two
defects that disqualifies it for free software:

 1. It has strings attached with respect to patent law (6.2).
 2. It is viral (3.2).

Even if Heirloom roff were free software, i doubt the code
quality would be sufficient for inclusion into OpenBSD.
It probably would have been good enough in 1994, but i doubt
it is still good enough by today's standards.  Cynthia's original
mdoc(7) macros - even though i highly admire has work, it was
excellent for the time, and the language itself is still the best
documentation language available IMHO - certainly aren't good
enough by today's standards.

> Can you enlighten me please?

Sure, i hope the above helps.  Note that we may well disagree
on parts of it, but you asked for the motivation for mandoc,
and these are the main points motivating it.

I hope you don't consider these informations too off-topic here,
but i didn't want to let the questions go unanswered.

Yours,
  Ingo