Comment 8 for bug 199418

This is an upstream patch proposal to do just this:
http://studio.imagemagick.org/pipermail/magick-developers/2009-October/003188.html

The rationale
is that this makes it much easier for distributions to provide packages
that don't depend on X.

This doesn't seem to be very hard to achieve. Attached is a proof of
concept patch that moves the code...

http://studio.imagemagick.org/pipermail/magick-developers/2009-October/003191.html
We added this suggestion to our list of things to do but it may be some time
before we get to it.
For now try ./configure --without-x