Alfred Nobel left a cross-eyed legacy: he invented dynamite and he founded the Nobel Prizes. Nobel's talent for explosives combined smoothly with a head for business—by the year of his death, 1896, he had ninety-three dynamite factories in several different countries—but the greater his success, the more it rankled his altruism. One commentator remarked that the clause concerning the Peace Prize in Nobel's will "challenged the humanitarian liberals among his personal friends to solve the problems his discoveries had created." Nobel was a melancholic, discontented man, with problems of his own. He never married or had successful relations with women. To a request from a relative for a contribution to a family history, he responded ironically: "Greatest sin: Does not worship Mammon. Important events in his life: None." As a postscript, Nobel suggested that he "should have been strangled by a humanitarian doctor when he made his screeching entrance into the world."

Born into a family of Swedish engineers and inventors in 1833, Nobel was raised partly in Stockholm and partly in St. Petersburg. Despite having had only one year of formal schooling, he was fluent in several languages. He patented "Nobels Extradynamit" in 1867, and his fortune increased as the railways expanded across Europe and dynamite became popular in warfare. An "important event" took place in 1864, when Nobel's younger brother was blown to pieces at one of the factories. According to some accounts, another occurred in 1888. Aged fifty-five and now living in Paris ("every mongrel stinks of culture here," was his characteristic stamp of approval), Nobel read of his own death in a French newspaper. In fact, it was his brother Ludwig who had died, but a reporter got the names confused and Alfred learned that he would be remembered as a "merchant of death," a "dynamite king," a businessman who had made his fortune from blowing things (and people) up.

A first will was made in 1893, with plans for prizes in chemistry, physics, medicine and peace (the award for economics was introduced in the 1960s). Only in a revised, final testament drawn up in 1895 did Nobel stipulate that one of the awards should be given to "the person who shall have produced in the field of literature the most outstanding work of an idealistic tendency." It was not exactly an afterthought, but nor was it a priority. He died the following year, and the first round of Nobel Prizes was distributed at the beginning of the new century, in 1901.

Nobel's wretched personal life seems to conceal few secrets, but there was one little romantic matter that he kept largely confidential: he was a writer himself. "The famous chemist and experimenter in explosives was at heart a poet," insisted a pair of those personal friends charged with deflecting attention from his innovations in weaponry. To call him a poet is an exaggeration, but Nobel produced enough, in several genres, to suggest that he had serious literary intentions. He wrote fiction in middle life and drama in his last years, but his youthful efforts were in verse—a heavily shod Miltonic blank verse, written in English. None of it was published in Nobel's lifetime, and most was destroyed at the time of his death by the circumspect executors, but the poetry that survives reveals a deep dejection at the heart of the dynamite king:

To dream of immortality, till Time
O'er empty visions draws the closing veil
And a new life sets in—the life of worms,
Those hungry plunderers of the human breast.

These lines, part of a long autobiographical poem, were written when Nobel was eighteen, and suggest that his idealism—"To dream of immortality"—like his discontent, started early. The poem veers towards self-pity in its depiction of a life hanging by threads, which survives only to mature into joylessness: "My cradle looked a deathbed, and for years / A mother watched with ever anxious care"…"Begun in pain, in deeper torture ended, / This breathing clay, what business has it here?" and so on. Nobel took his puerile poetry seriously enough to show it to friends in later life, doubtless in hope of a positive appraisal (which it received). It is heartfelt stuff and displays a remarkable command of English, but the high-flown diction is kept aloft less by its own energy than by poetical gas.

The fiction was written in Swedish. In an unfinished novel, "Brothers and Sisters," sections of which were published in an official biography (Schück and Sohlman, 1929), the gloomy introspection of the verse has been replaced by "ideas"; yet an identical theme appears to have inspired the writer in both genres. "You say I am a riddle—it may be," the long, untitled poem begins, "For all of us are riddles unexplained." In "Brothers and Sisters," lengthy discussions are centered on the same ponderous thought: "We are certainly surrounded by an eternal riddle; there are mysteries that we can never solve," etc. The verse dramas based on classical models—which he wrote in his last years, and which he paid to have published shortly before his death—are no more inspiring.

Nobel's own literature is essentially, profoundly, middlebrow. And it was his taste which dominated at least the first three decades of literature prizes—to an extent that makes it surprising that the award not only survived but became "the gold standard against which all other awards are measured," to borrow a phrase from The New York Times of 1983. Like most literary enthusiasts, Nobel was eventually willing to let writers of genuine talent do the work for him; unlike most, he was in a position to reward those others—the monetary aspect of all the Nobel Prizes has done much to confirm their status as the "gold standard"—and he intended to enrich only authors of like mind. The literature prize was reserved for writers who exhibited his preferred "idealistic tendency." Nobel elected the Swedish Academy to carry out his wishes (the science, economics and peace prizes are judged by other institutions), and its members debated at length the meaning of the stipulation and tried to apply it to the candidates. (They are debating it still: the Nobel website currently has an essay on the interpretation of the phrase, by the Academy's former permanent secretary, Sture Allén.) Was it to be taken as writing that, like scientific work, had a combined humanitarian and constructive purpose? Sture Allén interprets Nobel's phrase as "literary excellence…in a direction towards an ideal," which is nothing if not broad-based. In any case, Nobel wished to be seen to deplore the kind of pessimism that had soured his own life. Man may be food for plundering worms, but mankind must light up the darkness with hope—or Hope, as it more loftily goes in the poem. Thoughts that "lift us to the spheres" might ensnare a Nobel Prize in literature, never "petty wants to chain us to the earth." The character Morena in "Brothers and Sisters" could be speaking for Nobel himself when he says: "Clever sophists are the most dangerous men there are, for they rob their fellow men of peace, not only in this, but also in a future world." It was the desire to avoid this, and the determination to shun the "experimental morality" of the likes of Zola, whom Nobel particularly reviled, that threw up the bewildering early choices of the Nobel Committee, so obscure as to appear now wilfully blind. They were not the choices of Nobel himself, of course, but of the members of the Swedish Academy trying to guess what the repentant merchant of death would like.

The literature prize is customarily announced at noon on the second Thursday in October. The Nobel Committee submits to the full Swedish Academy a shortlist of candidates, numbering about twenty. The Academy then enters into deliberations to reduce the shortlist to five. The winner is finally elected by secret ballot; for the election to be valid, a candidate must gain more than half the votes cast. The name is made known to the press in the Grand Hall of the Academy, housed in the palatial buildings of the former Stockholm Stock Exchange. All other details, including the remaining names on the shortlist, are kept confidential.

The initial stage of selecting the winner is broadly democratic. During the autumn of the previous year, the Nobel Committee, composed of half-a-dozen members of the Academy, sends out invitations to several hundred professors of literature and languages, presidents of authors' organizations, Nobel laureates, members of sister academies (the only other two, besides the Swedish, are in France and Spain), and other representative literary people. The replies will contain about a hundred different names. According to Lars Gyllestein, a former Chair of the Nobel Committee and a historian of the literature prize, "only a few are new names which have not been proposed before." Many names are sent in year after year. "It is very unusual for anyone who has been proposed for the first time to win the prize," writes Gyllestein.

Some writers have won because they had strong supporters inside the Academy; others have faced over-my-dead-body opposition at the decisive stage. Arthur Lundkvist, a prominent Academician, championed Pablo Neruda, whom he also translated, and vigorously opposed Graham Greene. Neruda won in 1971, while Greene never received the prize, despite being nominated repeatedly. When William Golding won in 1983, Lundkvist took the unprecedented step of issuing a public denouncement, calling Golding "a little English phenomenon of no special interest." There were mutterings of disapproval, but Lundkvist remained a sitting member of the Academy (which, formally, it is impossible to leave). One of those who has come closest to winning without having done so is Norman Mailer. When Mailer opened The Prisoner of Sex in 1971 with an account of his high expectations—his secretary is made aware of every step in his itinerary and given contact telephone numbers—even admirers were apt to put it down as another piece of clowning on the part of the fictional character, "Mailer." But the veteran Academician Knut Ahnlund is on record as saying (The New Yorker, October 5, 1998): "I argued so many times for him." Not enough, though, and it is unlikely that the post-Ancient Evenings, post-Tough Guys Don't Dance Mailer is going to hear from Stockholm.

With so much prestige and money at stake, it is not surprising that the Swedish Academy has been the scene of some ungentlemanly bickering and brawling. When the present permanent secretary, Horace Engdahl, was first nominated to the Academy in 1997 (new members may be nominated only to replace the dead), he was subject to what he has called "vicious, unrestrained" attacks on him in the press, because of his professional approach to literature. Engdahl is an academic who specializes in post-structuralism. The main attacker was his fellow Academician Ahnlund, who has also referred to the previous permanent secretary, Sture Allén, a computer linguist, as "an intellectual accountant."

In 1989, the Swedish Academy was the stage for what counts as a public scandal in this secretive world, when two members quit over the way in which the Academy responded to the fatwa imposed on Salman Rushdie. A majority decided that the Academy should not put its name to a petition in support of Rushdie, alleging a traditional "non-political stance." Instead, it issued a statement condemning "all attempts to stifle freedom of expression."

It is easy to make merry with the game of Great Writers Who Were Passed Over, in discussions of the Nobel. Many writers who have come to be seen as giants in twentieth-century literature, such as Proust or Joyce, were not regarded that way during their lifetimes (the prize cannot be awarded posthumously). Others, such as Kafka, were barely published. Some of the modernist innovators of the early part of the century—T. S. Eliot, William Faulkner—were properly honored in their seniority. No committee can be expected to please everybody, not even all its own members, and for every onlooker who tut-tuts over the omission of Graham Greene, there is another of the view that he was just a writer of elevated thrillers (and yet another who has never read Greene but holds the strongest views of all). One thing on which opinion appears to be united, however, is that the early choices of the Swedish Academy were eccentric, and none more so than the first.

The 1901 award to the French "Parnassian" poet Sully Prudhomme caused embarrassment in the Swedish literary world even at the time. The Parnassians were a loosely constituted group, active in Paris from the 1860s onwards, who addressed exotic, non-political themes in strictly formal verse. One of their leading practitioners has been light-heartedly praised (by Rimbaud's most recent biographer, Graham Robb) for writing "beautifully calculated poems of…a haunting dullness." Forty-two authors and artists signed an open letter denouncing the choice of Sully Prudhomme and protesting the neglect of Tolstoy, still alive and writing—nothing if not idealistically—on his estate at Yasnaya Polyana. The Academy defended itself by saying that Tolstoy could not have been awarded the prize since he had not been nominated by an official body, as was the requirement. Anders Osterling, a former permanent secretary, has explained that Sully Prudhomme, on the other hand, "had been proposed by a large number of prominent members of the French Academy, and to follow their advice was apparently regarded as a matter of courtesy." As one Academy curtsied before another, therefore, literary history began one of its extended jokes: the first twenty years of the Nobel Prize. Had the great wave of nineteenth-century French poetry, bearing Baudelaire, Verlaine, Rimbaud, Mallarmé, and Laforgue, survived into the next, as conceivably it might have done, it is unlikely that any of them would have been chosen before the Parnassian Sully Prudhomm (it was at a recital by leading Parnassians in Paris that Rimbaud raucously added "Merde!" to the end of every line). Nobel required his literature to spout an improving message.

The Swedish Academy had the opportunity to set matters right by awarding the prize to Tolstoy in 1902, by which time he had been nominated in due form. But the Academy's eminence, Carl af Wirsen, who dominated the prize in its first years, would not lose face. He issued a statement in which, speaking on the behalf of the Academy, he criticized Tolstoy for having "condemned all forms of civilization," adding "one feels dubious." The 1902 Nobel went to the German historian Theodor Mommsen instead.

So it continued, with one or two imaginative exceptions (Kipling in 1907, Knut Hamsun in 1920), for the next two decades. Zola was ignored out of respect for Nobel's personal taste; Rilke, Thomas Hardy and Henry James were passed over, while Bjørnstjerne Bjørnson, Henryk Sienkiewicz, Giosuè Carducci, R. C. Eucken, Selma Largerlöf, and Paule Heyse were all given the prize (worth a substantial $40,000) within its first decade. Karl Gjellerup, Henrik Pontoppidan and other now forgotten brightnesses of the turn of the century followed.

The Academy had a bias towards Scandinavian writers in those years, perhaps not unnaturally, but dutiful adherence to the "idealistic tendency" could mean ignoring the talent on the doorstep, too. The Spanish dramatist José de Echegaray Y Eizaguirre won in 1904, while Ibsen and Strindberg were dismissed. Strindberg had also made himself unpopular with the powerful permanent secretary Wirsen, first by participating in the protest over Tolstoy and later by satirizing Wirsen in a book.

It was only when strict reliance on the letter of the will was relaxed that the Nobel Prize appeared to be running at pace with the movement in modern literature. The change began around the 1920s and 30s (Thomas Mann won in 1929, Luigi Pirandello in 1934, Eugene O'Neill in 1936), and was given a hefty jolt by the Second World War. In 1946, Hermann Hesse, a novelist who was to become a hip cult favorite in the 1960s, was chosen; the next year the prize went to André Gide, the year after that to Eliot, and in 1949 to Faulkner. The successive selection of, perhaps, America's greatest twentieth-century poet and novelist (bearing in mind that Eliot was by then a naturalized British citizen) might be seen as compensation for a mix of neglect and eccentricity in earlier times. (No American won the prize for almost the first thirty years, and then it went to Sinclair Lewis; the award to Pearl Buck in 1938 is generally seen as one of the oddest of all Nobel Prizes.) With Eliot especially, the literature prize appears to have slipped the "idealistic" yoke; if the ghost of Nobel objected—how intensely he would have disliked the "riddles" of the sublimely pessimistic poem "The Hollow Men"—then the Academy could plead that it was the Eliot of Four Quartets, the churchly Eliot, who was being honoured, not the Eliot of "Unreal city /…I had not thought death had undone so many."

It scarcely amounts to an original observation to say that by the late 1940s the days when the "idealistic tendency" was regarded as essential to literature were at an end. But they were already gone by 1901. That was the year in which Queen Victoria died, having outlived the era named after her; it was the year when Wilhelm Kostrowitzky became "Apollinaire," a herald of new, "relativist" artistic practices based on close acquaintanceship with modern painters. Long before Nobel's death, the latter, cultured mongrels that they were, had ditched idealism as an inappropriate agent in art. A few years after the first prizes were distributed, Ezra Pound would arrive in London with a missionary purpose to cast out old devils of statement. The question provoked by modernist literature was already contemporary: not only (if at all) "What is the story?" but "Why did you tell it that way?" No modernist could utter the phrase "idealistic tendency" without ironic ambivalence (come to think of it, ironic ambivalence was what replaced the tendency).

It took the Swedish Academy most of the first half of the twentieth century to catch up, but catch up it did. The upheavals of the Second World War had something to do with it. Nobel's projection had occasionally rewarded a dubious idealism: two future Nazi sympathizers received the prize (the Swede Verner von Heidenstam, 1916, and the Norwegian Knut Hamsun, 1920), as well as a number of Europeans with a "strong commitment to German culture" (the phrase was used specifically of the 1917 recipient, Karl Gjellerup). It is not a coincidence that Hermann Hesse, the first winner after the end of the war, was a long-term exile from Hitler's Germany. The official citation to Eliot stated that he had "the ability to cut into the consciousness of our own generation with the sharpness of a diamond." The cliché might have brought a frown to Eliot's discreet brow, but the notion of the "consciousness" of a generation, as represented by the author of "The Waste Land," was not the sort of thing that had come naturally to earlier members of the Swedish Academy.

The Nobel Prize has been declined on only two occasions, although there have been waverers. George Bernard Shaw, when selected in 1925, signalled a conditional acceptance. However, he wrote to a friend: "I cannot persuade myself to accept the money." Shaw had a general dislike of prizes, and had previously called the Nobel a lottery. What's more, the munitions millionaire Andrew Undershaft, in his play Major Barbara (1905), was thought to be partly modelled on Nobel the dynamite king. Nevertheless, the Academy was determined to give it to him. When Shaw offered to accept the prize but not the money, it was pointed out that the two could not be separated, and so Shaw created the Anglo-Swedish Literary Foundation, to aid the translation into English of classical Swedish literature (it still exists, in redefined form).

In 1958, Boris Pasternak, having first gladly accepted, was forced to withdraw because of pressure from the Soviet government, and was reduced to pleading with Krushchev not to expel him from his native land. The affair damaged Pasternak's health and probably contributed to his death in 1960. The tension is etched into a poem he left behind, called "Nobel Prize":

I'm caught like a beast in a trap.
Somewhere there is freedom, light, people.
But the hunt is after me
and there is no way out.

The sole wholehearted refusal in the history of the Nobel Prize came half-a-dozen years later. On October 14, 1964, Jean-Paul Sartre read in Le Figaro littéraire that the Swedish Academy had him lined up for that year's prize. In fact, the official announcement was not due to be made until the following week; but however it came about, the prediction proved to be accurate. Sartre declined the honor immediately, and with genuine courtesy. He first wrote personally to the secretary of the Academy—"I cannot and do not want to, not in 1964 or ever, accept this great distinction"—and then dictated a statement to a Swedish journalist, in which he said that he had always turned down "official distinctions" in the past, out of a conviction that "the writer must not allow himself to be transformed by institutions." Sartre had previously declined the highest official accolade his country could bestow on him, the Légion d'honneur (it is a mystery why this committed anti-establishment radical was offered it), as well as a professorial chair at the Collège de France. His refusal of the Nobel, he said, was not "an improvised act," but the result of a thought-out position on honors and awards. Here was idealism in action, surely, of which even Nobel might have approved? Sartre was turning down a fortune, and modestly putting forward a high-minded reason for doing so. He simply wished to remain free. The Swedish Academy could not see things his way, however. It responded through tight lips—"The fact that he is declining does not alter in the least the validity of the nomination"—and went ahead with the prize-giving ceremony in Stockholm, in the absence of a winning head to anoint.

The will stated that "in awarding the prizes no consideration shall be given to the nationality of the candidates." It was Nobel's wish that "the most worthy shall receive the prize, whether he be a Scandinavian or not." In literature, unlike the precise sciences (and the thoroughly imprecise one, "peace"), there was an obstacle: language. It was easy to find committee members who read English, French, German, Spanish; less so to accommodate Chinese or Modern Greek or Arabic. Translations, especially from non-Latin or non-Germanic languages, could convey only the surface appearance of a writer's work. How was the Nobel Committee to know whether a poet was a master in his own language, or if his translator had misrepresented him (perhaps by improvement)? An accompanying consideration was that Asian and African forms of literature may be only slackly comparable to those in the West. These are problems which the Committee still faces, but in earlier times they appeared insurmountable, and for sixty years or more the prize was practically a Western preserve.

On the rare occasions when non-Western writers were invited to Stockholm, it was, in a mutually acknowledged term, as "strangers." The 1945 prize was awarded to the Chilean poet Gabriela Mistral, the first writer from Latin America to be chosen (and the fifth woman), described in a history sponsored by the Academy as "a stranger from afar with a half-Indian appearance." The only non-white writer to come before her was the Bengali poet Rabindranath Tagore, who won in 1913. Born in Calcutta and educated partly in England, Tagore had translated his own work into English, making it accessible to the members of the Committee. It is lyrical and sweet, full of moonlight and young love, a "maiden" who might modestly shade "the timid flame of her lamp," and birdsong which makes "the heart dance with gladness"—in other words, it is miles away from the explosion that was about to occur in English poetry. The Swedish Academy stressed his Christian links, and Tagore showed that he appreciated the effort by emphasizing in his acceptance telegram "the breadth of understanding which has…made a stranger a brother."

The Japanese writer Yasunari Kawabata won in 1968, and the dimensions of the prize have continued to expand ever since. The "idealistic tendency" persists as the flimsiest of criteria (it was raised in the presentation to Nadine Gordimer in 1991), giving way to an increasing use of the postmodern password "diversity" instead. A Yiddish writer won in 1978 (Singer), the first African in 1986 (Soyinka), an Afro-Caribbean in 1992 (Walcott), followed immediately by an African American (Toni Morrison). In 2000, Gao Xingjian became the first Chinese author to win the prize. The choice did not please the Beijing government—which is likely to have been partly the point. In spite of what it claimed in taking its stance over the Rushdie affair, the Swedish Academy has made many selections which seem intended to support politically engaged writers, though not of one particular stripe, in the past thirty years. The anti-Marxist Solzhenitsyn in 1970 was counterbalanced by the Marxist Neruda in 1971. Since then there have been the Polish exile Milosz (1980), the veteran of Nigerian prisons Soyinka, the former Soviet "vagabond" Brodsky (1987), and then Gordimer, Morrison, Szymborska (1996), Fo (1997), Grass (1999), and the Chinese exile Gao, all of them embattled in one way or another. In one of its most discreditable decisions, in 1965, the Nobel Committee chose Mikhail Sholokhov, a Soviet writer whose voice had been among those raised against Pasternak. It is widely believed in Russian literary circles today that Sholokhov did not single-handedly write the novel which did most to earn him the honor, And Quiet Flows the Don.

This drift towards political engagement has brought with it accusations that the Academy is too much influenced by extra-literary criteria. It is a legitimate complaint, but it should be set beside a response to Tagore's prize, in The New York Times in November 1913. The American reading public was said to be surprised that Western writers had been passed over in favour of a "Hindu bard," though disappointment was mitigated by recognition that the bard had had a Western education, and enough sense to translate his poems into "good sound English" before expecting anyone to read them. Tagore, "if not exactly one of us, is, as an Aryan, a distant relation of all white folk." (This item appeared in "Topics of the Times"; the paper also ran a more serious article on Tagore's award.)

Public perception of the Nobel Prize in literature has changed since then, but is still characterized by a split. On the one hand, there is general acceptance that it is indeed "the gold standard" against which other awards are measured; on the other, the constant objection is made that the annual awards are unrepresentative of the best in contemporary literature. It seems that these contradictory views are perpetually in tandem—the Nobel will never do its job properly, yet will not be dislodged from its pre-eminent position. That the Nobel survived its early misguided selections can be put down to two things: there is the prestige of the associated science prizes, and then there is the money. The latter—this year's Nobel Prize laureate will receive about $1,000,000—surely accounts for the fact that there have not been more refusals, according to the principles set down by Sartre. Many a hard-working writer, transformed by the institution, has seen his or her pen dry up in the warmer climate of fame and luxury. "Greatest sin: Does not worship Mammon." The same is probably true of most of the winners. But Sartre's decision to decline represents the apotheosis of Alfred Nobel's "idealistic tendency."

James Campbell works for the Times Literary Supplement. His new book, This Is the Beat Generation, will be published in November.