issue with pkgdb

[Ben Walton]

Hi Folks,

I was attempting to upload new git packages this morning and was
getting 503's from the csw-upload-pkg script. Visiting the url
http://buildfarm.opencsw.org/pkgdb/ yielded the same.

Digging into the backend, it looks like there is a wsgi issue that has
resulting in apache not properly respawing the backend processes:

[Sat May 31 04:24:26 2014] [info] [client 213.178.77.176] mod_wsgi
(pid=12178, process='buildfarm.opencs
w.org', application='buildfarm.opencsw.org|/releases'): Loading WSGI
script '/home/web/bin/gar/lib/web/r
eleases_web.py'.
[Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Maximum
requests reached 'buildfarm.opencsw.org'
.
[Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Shutdown
requested 'buildfarm.opencsw.org'.
[Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Stopping
process 'buildfarm.opencsw.org'.
[Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Destroying interpreters.
[Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Destroy
interpreter 'buildfarm.opencsw.org|/rele
ases'.
[Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Cleanup interpreter ''.
[Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Terminating Python.
[Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12178): Python has shutdown.
[Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12207): Attach interpreter ''.
[Sat May 31 04:24:52 2014] [info] mod_wsgi (pid=12207): Create
interpreter 'buildfarm.opencsw.org|/relea
ses'.
[Sat May 31 04:24:52 2014] [info] [client 213.178.77.176] mod_wsgi
(pid=12207, process='buildfarm.opencs
w.org', application='buildfarm.opencsw.org|/releases'): Loading WSGI
script '/home/web/bin/gar/lib/web/r
eleases_web.py'.
[Sat May 31 04:25:00 2014] [info] mod_wsgi (pid=12207): Create
interpreter 'buildfarm.opencsw.org|/pkgdb
'.
[Sat May 31 04:25:00 2014] [info] [client 213.178.77.178] mod_wsgi
(pid=12207, process='buildfarm.opencs
w.org', application='buildfarm.opencsw.org|/pkgdb'): Loading WSGI
script '/home/web/bin/gar/lib/web/pkgd
b_web.py'.
[Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Maximum
requests reached 'buildfarm.opencsw.org'
.
[Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Shutdown
requested 'buildfarm.opencsw.org'.
[Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Stopping
process 'buildfarm.opencsw.org'.
[Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Destroying interpreters.
[Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Destroy
interpreter 'buildfarm.opencsw.org|/rele
ases'.
[Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Destroy
interpreter 'buildfarm.opencsw.org|/pkgd
b'.
[Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Cleanup interpreter ''.
[Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Terminating Python.
[Sat May 31 04:25:21 2014] [info] mod_wsgi (pid=12207): Python has shutdown.

[..snip lines from other stuff...]

[Sat May 31 04:30:21 2014] [error] [client 213.178.77.176] Script
timed out before returning headers: re
leases_web.py
[Sat May 31 04:35:00 2014] [error] [client 213.178.77.178] Script
timed out before returning headers: pk
gdb_web.py
[Sat May 31 04:35:24 2014] [error] [client 213.178.77.176] Script
timed out before returning headers: pk
gdb_web.py
[Sat May 31 04:37:02 2014] [error] [client 213.178.77.176] Script
timed out before returning headers: pk
gdb_web.py
[Sat May 31 04:40:00 2014] [error] [client 213.178.77.178] Script
timed out before returning headers: pk
gdb_web.py

This is now resulting in errors logged for each connection attempt:

[Sat May 31 08:00:00 2014] [error] [client 213.178.77.178]
(146)Connection refused: mod_wsgi (pid=11958): Connection attempt #1
to WSGI daemon process 'buildfarm.opencsw.org' on
'/var/run/wsgi.20108.0.1.sock' failed, sleeping before retrying again.
[Sat May 31 08:00:00 2014] [error] [client 213.178.77.178]
(146)Connection refused: mod_wsgi (pid=11958): Connection attempt #2
to WSGI daemon process 'buildfarm.opencsw.org' on
'/var/run/wsgi.20108.0.1.sock' failed, sleeping before retrying again.
[Sat May 31 08:00:00 2014] [error] [client 213.178.77.178] Script
timed out before returning headers: pkgdb_web.py

I just kicked apache and that seems to have brought things back to
life. I'm not currently planning to dig any further for a root cause
here, but am noting the event in case it becomes a pattern in the
future - we'll have more data to work with.

Thanks
-Ben