Modular X 101 : Debugging modular X

[Mike A. Harris]

Did you know, that all of the modular X packages now come with
full debuginfo packages, including the X server and driver modules?

The X server uses dlopen now also, so it can be debugged using
the stock gdb that comes with Fedora.

In order to debug the running X server, install the debuginfo
packages for it, and any drivers you are loading.  You then
need to have 2 computers[1] networked together, or using a serial
port or some other remote access method.

Edit the xorg.conf file, and enable the option:

     Option "NoTrapSignals"

Then, enable coredumps on the system, if you have not done so
already, by using "ulimit -c unlimited" in a script dropped into
/etc/profile.d/enable-coredumps.sh  or whatever.  Be sure to run
the script or log out and back in.

As the X server is a setuid root process, and the kernel will not
allow coredumps of SUID processes by default, you have 2 choices.

You can tweak the kernel to permit SUID coredumps by doing:

     echo 1 > /proc/sys/kernel/core_setuid_ok

or you can run the X server as root.  If you enable the kernel to
allow SUID coredumps as per above, just be sure to disable it
afterwards by echoing '0' to the same file, to avoid potential
security issues.

Now you are ready to break things and debug them.  I prefer to
start the X server with "startx" all the time, but particularly
when debugging.  I recommend doing the same, unless the problem
you are debugging only occurs when using gdm/kdm/xdm, as it tends
to be much simpler to debug with startx.  YMMV however.

If the server crashes, you can gdb the coredump now.  Assuming
you installed debuginfo packages for the server (make sure the
debuginfo package is the same version-release as the server that
is installed), you should get useful backtraces, etc.

If you want to debug the running server, single step it, etc.,
you need the second computer[1] mentioned above.  Do all the same
steps as above, but this time, ssh in from the remote computer
(or use minicom on serial port or whatever), and get the PID
of the X server with 'pidof X'.  Then attach gdb to the X server:

     gdb --pid=$(pidof X)

Now go to town!  Find the bug, install the X server/driver sources,
fix the bug, generate a patch with gendiff, and attach it to a bug
report.  It's really that simple!!!  ;oP

This informative message has been brought to you by the Red Hat
X Development team, and Frank's Red Hot Chile & Lime Hot Sauce.

Have fun!


[1] The reason that 2 computers are required, is that if you start
the X server, open an xterm and run gdb on the running X server,
gdb will stop the X server, and try to print to stdout.  Since
stdout is your xterm, and xterm is displaying to the X server which
is now no longer responding because gdb stopped it, you are screwed.
It's kindof like trying to stand on a carpet, and then pull it out
from under yourself.  You can't really do it.  This is why you
need[2] 2 computers.

[2] Ok, it is technically possible to do it with one computer too,
but it is complex enough to not really bother trying to do, so
don't even waste time thinking of it.  If you don't have 2
computers, go dumpster diving at 3am in the garbage dumpster of
a computer store.  They throw out good stuff I'm told, and you
can have a "X server debugging terminal" in no time, like everyone
else on the block.  Alternatively, surf ebay.com for a bargain, or
hang out at a computer recycling shop.


Note: Please feel free to forward this message on to other people
or mailing lists, or to copy or paraphrase into wiki pages or other
helpful resources if you think other formats or people would
benefit from this info.