See note on methodology

For the first time, but not for the last, I face the consequences of the break-up of the former Yugoslavia. Even those living there found it tricky to follow the shifting borders between the Slovenian referendum in 1990 and the independence of Kosovo in 2008. A lot of the books identified with "Serbia" tags on LibraryThing and Goodreads are simply set elsewhere, most often in Bosnia; others are tours d'horizon of the entire region.

I have come up with an answer to the question that satisfies me, though I admit that it has flaws. Published as recently as 2010, by a writer who was born in Belgrade but stresses her roots elsewhere in the former Yugoslavia, it is set in an unnamed Balkan state, but reviewers that I have checked have assumed it is meant to be Serbia (including one who irritably listed all the mistakes that were made with the Serbian setting, which pretty much proves the point). Winner of the 2011 Orange Prize, it is:

The Tiger's Wife, by Téa Olbreht.

Worth noting, but not sufficient answers to the question, are two well-known books about the former Yugoslavia as a whole, both of which were critiqued for feeding the convenient narrative that external intervention would not help during the most recent conflict. One of them is based in fact on events of the 1930s, published in 1941; the other was published in 1993. They are:

Black Lamb and Grey Falcon, by Rebecca West
Balkan Ghosts, by Robert Kaplan.

Digging down further, I hit the problem that few of the great writers of the former Yugoslavia set their best-known works in Serbia (often they were not themselves Serbs, of course). This is true for Danilo Kiš, Milorad Pavić, Ivo Andrić and Meša Selimović. The top book on Goodreads which I could identify clearly as set in Serbia, identifiable as such, is a 1910 classic Serbian novel about tradition versus modernisation and the changing position of women, made into a Serbian TV series in 2012. It seems to have several names in Enlgish translation; I'm giving first the one I could find in recent publication (2008) followed by the two given in Wikipedia. It is:

Bad Blood / Impure Blood / Sophka / Nečista krv, by  Borisav Stanković.

The top book on LibraryThing which I could identify clearly as set in Serbia, named as such, is a series of interviews conducted by a Norwegian journalist with her various Serbian contacts over the years of war and peace. It is:

With Their Backs to the World: Portraits from Serbia, by Åsne Seierstad

It's interesting that Serbian writers in general appear better represented on Goodreads.