If Apple has shown something, is what putting millions of dollars and more than a thousand full-time engineers working on a Unix OS with custom hardware looks like. Most Linux users don't care about the X server nor find it underperformant. Hell, not even GUI developers have to touch it.
I would argue that Linux isn't mainstream because it has nothing like Apple's budget, specially on design and marketing. Add to that those hardware vendors that are actively hostile to support from other operating systems, and software developers that don't care, and you have it. Still, Linux is doing fine, for what resources it has.
Linux doesn't have Apple's budget, true, but neither did Apple, when they started the OSX project. Remember, when Jobs came back on board and made the decision to pivot to Unix, Apple was roughly 90 days from bankruptcy. Apple of 2000 was not the fearsome behemoth that Apple of 2012 is.
At the time, there were many commercial Linux distributors - Red Hat, Mandrake, SuSE all spring to mind - and any one of them could have beaten Apple to the punch in offering a Unix desktop environment that was easy to use and, for lack of a better term, pretty. They didn't, though. Why not? I think the need to build on top of X is one thing that hampered Linux desktop environments when competing with OSX.
I would argue that Linux isn't mainstream because it has nothing like Apple's budget, specially on design and marketing. Add to that those hardware vendors that are actively hostile to support from other operating systems, and software developers that don't care, and you have it. Still, Linux is doing fine, for what resources it has.