The other night a friend of mine asked me to define literary fiction. "Hold on," I said, "and I'll get that for you." I had been asked the same question in an email several years back, and had written the definition down to email it back to my friend in Texas. I went to the "Sent" folder in my laptop and found the email, thus saving myself the embarrassment of trying to state my thoughts on the matter verbally. I believe I can stand by my original answer, offhand as it was, but here I've tried to state it more clearly:
Subject: What is literary fiction?
I'm going to answer this question according to my personal thinking, realizing that in general, it's probably more useful for us to think about the definition(s) used by our publishers.
First, I don't think of literary fiction as a genre, but as separate from genre fiction. It's not constrained by sets of parameters the way the genres are. On the contrary, part of what makes it literary is that in some sense, to some degree, the author has succeeded in stepping outside preexisting formulas. While all fiction needs to conform to “rules” of good writing, literary fiction also exists to challenge the rules, to find other ways to “work” for the reader as necessary to convey the author's artistic vision. Like any other true art, literary fiction moves civilization forward. As we see over and over again, that forward motion often meets resistance, but can eventually become absorbed in our culture.
What defines fiction as literary is that it is grounded in Truth in a way that genre fiction is not. By Truth, I mean that profound personal concern which the writer intentionally elucidates through the labor of writing. I don't mean that the writer has picked out something he believes, and sets out to convince readers of same; no, that would be what I like to call “poster fiction.” On the contrary, the Truth I'm referring to is not something that can be stated on a poster or even summarized. So, while a thriller might “say” that crime does not pay, or a romance might "say" that forbidden love is doomed and tragic, a literary novel's Truth cannot be reduced to fewer words than the number of words in the novel. I've been asked, on occasion, “What are you saying in this novel?” The correct answer is, "If I could tell you that in a few words, I wouldn't have had to write the novel."
It often seems that part of the definition of a literary novel is that it gainfully employs language. Fine, but any kind of fiction can be beautifully written. I would agree that literary stories and novels tend to employ language more richly than other forms. However, that is a result of the writer's artful quest to reveal a complex Truth, and not part of the definition of literary fiction.