I have mixed feelings about X. On the one hand, it's a very impressive bit of kit. It offers a unified protocol that separates the program drawing the window from the hardware that displays the window on the screen. This allows the program and the hardware to be completely separate, possibly even running on separate machines.
On the other hand, that very separation is X's greatest hindrance. It imposes a fairly large cost, not just in terms of performance, but also in terms of complexity. You have to deal with a network protocol just to display things on the screen. No other windowing system imposes this level of cost on the developer, and it can be argued that the reason we never got a Linux desktop was because of X. Simply put, Apple showed us what could be done with a Unix system if you abandon X and set allow your GUI API to talk directly to the hardware. A large part of Apple's success can be tied directly to their decision to forego X11.
I really don't think this has anything to do with it. Yes, there are complexities with X, but nobody writes serious applications targeting X. You target a toolkit. And the toolkits have effectively abstracted X away for many years. So well, in fact, that both GTK and QT can run fine with other backends (including a web browser for GTK, as well as DirectFB and Wayland).
No normal application developer "have to deal with a network protocol". Even if you write code directly against Xlib that is largely transparent to the developer.
Performance was an issue until XRender and DRI arrived around 2000. It has not been an issue for years.
Incidentally, one of the reasons Wayland is becoming so attractive is exactly because so little of what X was traditionally is actually being used any more in most setups, since XRender and DRI combined means most apps does all or the vast majority of rendering client side, and the server is largely left as a resource manager and compositor and not much more.
If Apple has shown something, is what putting millions of dollars and more than a thousand full-time engineers working on a Unix OS with custom hardware looks like. Most Linux users don't care about the X server nor find it underperformant. Hell, not even GUI developers have to touch it.
I would argue that Linux isn't mainstream because it has nothing like Apple's budget, specially on design and marketing. Add to that those hardware vendors that are actively hostile to support from other operating systems, and software developers that don't care, and you have it. Still, Linux is doing fine, for what resources it has.
Linux doesn't have Apple's budget, true, but neither did Apple, when they started the OSX project. Remember, when Jobs came back on board and made the decision to pivot to Unix, Apple was roughly 90 days from bankruptcy. Apple of 2000 was not the fearsome behemoth that Apple of 2012 is.
At the time, there were many commercial Linux distributors - Red Hat, Mandrake, SuSE all spring to mind - and any one of them could have beaten Apple to the punch in offering a Unix desktop environment that was easy to use and, for lack of a better term, pretty. They didn't, though. Why not? I think the need to build on top of X is one thing that hampered Linux desktop environments when competing with OSX.
When I remotely debug applications running in foreign countries, I always feel that X (and ssh -Y) are miraculous inventions. Unix gives a divine feeling of ubiquity.
You should look up how Plan 9 handles debugging, you just import /proc from the remove machine, and run the debugger locally.
This works even across different architectures, and the debugger is completely network unware, is all handled transparently by the file system interface to processes.
Some goes for the window system, which also works across the network even when it knows nothing about networks.
On the other hand, that very separation is X's greatest hindrance. It imposes a fairly large cost, not just in terms of performance, but also in terms of complexity. You have to deal with a network protocol just to display things on the screen. No other windowing system imposes this level of cost on the developer, and it can be argued that the reason we never got a Linux desktop was because of X. Simply put, Apple showed us what could be done with a Unix system if you abandon X and set allow your GUI API to talk directly to the hardware. A large part of Apple's success can be tied directly to their decision to forego X11.