Literature made an abrupt change of direction about 100 years ago. Compare, say, Thomas Hardy's Tess of the D'Urbervilles (1892) with James Joyce's Ulysses (1922). In the span of just 30 years (and you could use other examples to narrow the span further), the understanding of what constituted great literature changed dramatically. Writers like Hardy, Charles Dickens and Anthony Trollope gave way to writers like Joyce, Virginia Woolf and William Faulkner. Why?
What happened to the novel at about the time of the First World War was happening to other art forms, as well. Poetry no longer had to rhyme. Paintings no longer had to look like anything. The old rules no longer applied to novels either.
Thomas C. Foster analyses this change in his book How to Read Novels Like a Professor. He suggests that novelists' priorities changed from the 19th century to the 20th. In the 19th century, writers sought first to win an emotional response from their readers and after that, in order, an intellectual response and an aesthetic response. Early in the 20th century this hierarchy reversed itself. Writers sought first an aesthetic response, then an intellectual response and an emotional response.
Nineteenth century writers weren't interested in creating art. They were trying to make a living. People were more likely to buy books, or the publications in which novels were serialized, when they could be emotionally involved in the lives of the characters. Writers like Joyce, Woolf, Faulkner and those who followed in their footsteps did not have nearly as many readers because they didn't write stories that were as emotionally accessible. That's why so many of them have, like Faulkner, moonlighted as Hollywood screenwriters or, like most literary writers today, taught creative writing courses in colleges to supplement their incomes.
It is probably still too soon to predict what the priorities of 21st century novelists will be. Will aesthetics continue to be valued above all else, or will there be another change of direction? The ideal, it seems to me, would be a balance between the readers' emotional, aesthetic and intellectual responses. When I think of the two 21st century novels I most admire so far, State of Wonder by Ann Patchett and The Goldfinch by Donna Tartt, I think that perfect balance has been very nearly achieved. Readers respond emotionally to these stories, which have both been best sellers, yet they respond intellectually and aesthetically, as well. Both novels contain enough ambiguity for literature professors to argue about for years, yet enough clarity to give satisfaction to ordinary readers. They are smart, they are beautifully written and they can bring a tear to the reader's eye. Sounds like literary perfection to me.