thos: IIRC, a compositing manager together with the XComposite extension is in fact required for having a semi-transparent (thus arbitrarily-shaped) window.
There is another X extension available allowing arbitrarily shaped windows without any semi-transparency (each pixel is either displayed or omitted). It uses a bitmap, i.e. you explicitly set which pixels to display and which to “carve out”. It probably takes care of that by sending expose events to the underlying windows, and overlaying the results afterwards. This would still mean on-screen drawing, though, compared to off-screen XComposite drawing.
On a sidenote: I think the problem with XComposite is that even if the library is loaded, there is no guarantee that the underlying graphics hardware supports it, and that the user gets semi-transparent output.
For instance, some people demanded semi-transparent drag icons for Nautilus, but it turned out that finding out whether compositing is actually done is nontrivial. I made a proposal on the xdg freedesktop.org mailing list for finding out whether a compositing manager is actually running, but nobody has proposed a freedesktop spec yet.