Some difficult judgements to be made here. A famous 1864 SF novel must be disqualified because only a short but memorable section is set on Iceland, the main action taking place in an underground setting completely implausible to today's geologists. It is of course:
Journey to the Center of the Earth / Voyage au centre de la Terre, by Jules Verne
Ruling that out, the book that topped the Goodreads ownership by quite a long way was very far behind on LibraryThing, so far behind that I can't really consider it today's winner. It was published only in 2013; I wonder if this reflects a recent surge of relative popularity for GR? Anyway, it is a novel about the last woman to be executed for murder in Iceland, in 1854:
Burial Rites, by Hannah Kent
Second on both LibraryThing and Goodreads is one of the great works of ancient Norse literature. I fear I must disqualify it also, because it too is not set on Iceland, concentrating instead on the great issues of cosmogony and literature - though I'm willing to be persuaded about this. It is:
The Prose Edda, by Snorri Sturluson
Jockeying for position with it are the third and fourth volumes of Iceland's most famous series of crime novels. I'm going to proclaim the former of these today's winner, because it is top on LT, whereas the latter is only third on GR. (Each is fourth on the other system, respectively.) They are:
Jar City / Mýrin (Inspector Erlendur 3), by Arnaldur Indridason
Silence of the Grave / Grafarþögn (Inspector Erlendur 4), by Arnaldur Indridason
But it's close at the top. Third on LT and fifth on GR is the most famous novel by Iceland's only Nobel laureate writer:
Independent People, by Halldór Laxness
And not very far behind is the best known of the works of Norse literature actually set on Iceland:
Njál's Saga / Njáls saga, possibly by Sæmundr fróði
Many more sagas and detective noves fill up the lists.