It looks like it just uses an existing X server as a way to create an OpenGL context, not as an X server per se. Perhaps it would be possible to port it to some non-X method of producing an OpenGL context, such as EGL.

Yes that's the idea

So its a really hard way to run an X server on top of an X server?

Because this glosses over the really complex parts. Still, if they are written in Lisp, this would comprise a complete self-sufficient X server (how many lines of code this would take though?).