Emma Brockes takes a trip down the Yellow Brick Road to talk to those in the know about the making of a phenomenon
In pictures: The Wizard of Oz

In the book on which the 1939 film The Wizard Of Oz was based, Dorothy lived in a one-room shack on the Kansas prairie with her Uncle Henry and Aunt Em, a defeated pair who “did not know what joy was”. As in the film, it was Dorothy’s “little black dog” Toto who kept her out in the storm, and together they were whisked to Oz, a place she had longed to discover but on discovering was instantly feverish to leave. There was no Miss Gulch in L Frank Baum’s book and the Witch of the North, who travelled mysteriously but not by pink bubble, was an old lady the size of a Munchkin. Dorothy, on the other hand, was a ”well grown child for her age” – although not, perhaps, as well grown as her MGM incarnation who, generations on, still reigns as a symbol of hope in hard times.

It is 70 years since The Wizard Of Oz was made and almost no one from the production survives. Principal cast and crew are long gone. Of the 124 Munchkins, six remain. The cast’s children advocate for them now, along with the self-defined “Oz nuts” who attend conventions, collect memorabilia and fall into camps of gently warring interests. (The Baum-ites disdain the Judy-ites; the Oz scholars cut eyes at the collectors. Everyone loves the Munchkins.) That Baum’s story made it through the Hollywood sausage machine more or less intact is something devotees of both book and film see as practically mystical. The movie survived 10 writers, four directors and the propensity of Hollywood to find simple things and effortlessly scramble them.

Over the years, it also survived Marxism, Freudianism, postindustrialism, postcolonialism and the greatest threat of all to its meaning, the co-option of its charm into the hard, mean drive of American Idol-type aspiration. Over The Rainbow, beloved of auditioners everywhere, has come to stand loosely for the notion of Dreams Come True, if by dreams we mean standing on a stage, holding a microphone, while little people across the land look up at us in rapture. In the context of the film, Over The Rainbow means nothing of the sort, of course. The best and oddest thing about The Wizard Of Oz is its power as a critique of what it’s supposed to be striving for.

When MGM bought the rights in 1937, the scriptwriters faced a number of problems. Parts of the book were too graphic to film and others too complex. In Baum’s version, the Tin Man is not merely a victim of bad weather. Once a flesh-and-blood woodsman, his axe, we learn, has been cursed by the witch, causing him to hack off his own legs, arms, torso and eventually, in a feat of dexterity, his own head, to be replaced with tin prosthetics that the witch then treats to a downpour. Pre-CGI, the book’s Kalidahs, with “bodies like bears and heads like tigers”, were a headache the film-makers didn’t want to get into. Oz was a land surrounded by desert and the Emerald City, in Baum’s version, was not even green, but an illusion wrought by green-tinted spectacles, which Oz citizens were mandated by the wizard to have padlocked to their heads.

In 1899, Baum was writing against the background of a failed populist movement, an early civil and women’s rights lobby, which he broadly supported. His mother-in-law, Matilda Joslyn Gage, was a suffragette so radical she has been written out of American history – in her book Woman, Church And State, she attacked the Christian church for oppressing women and pointed to preferable feminist models among Native Americans. Baum had several failed careers behind him – actor, salesman – and it was Gage who encouraged him to write; his wife, Maud, meanwhile, inspired the story of a sensible little girl in a land ruled by women.

“Without Maud I don’t think any of this would have happened,” says Bob Baum, the author’s great-grandson, who taught the Oz books in the LA school system before he retired. “Frank saw in Maud a very practical, down-to-earth person who could make things happen. She gave him the time and encouragement and space to do what he did.”

The first scriptwriter to have a crack at adapting the book was Herman Mankiewicz, a former newspaper reporter who, as Aljean Harmetz chronicles in her 1977 book The Making Of The Wizard Of Oz, knocked out a script in four days. A year later he would win an Oscar for Citizen Kane, but his Oz was a mess. Mankiewicz draws Dorothy as a cheerful little girl full of trite observations (on the subject of corn, she squeals, “I guess it’s about the best food there is!”) and stuffs the script with terrible subplots (at one point a limousine pulls up outside Dorothy’s house in Kansas, dispensing a millionaire and her hilarious pekingese). The producer, Mervyn LeRoy, nearly invented postmodernism 20 years before Derrida by suggesting an opening shot of Dorothy reading a copy of The Wizard Of Oz in bed. Director George Cukor lasted just three days before huffing off in protest at the trashy material. (“I was brought up on grander things,” he sniffed. “I was brought up on Tennyson.”) Before he left, he scraped the make-up off Garland and warned against making her too “fancy-schmancy”, which his successor, Victor Fleming, heeded.

After several more false starts, the project fell into the hands of Noel Langley, a 26-year-old South African who wrote most of the final script, although, because the film is so reliant on song, a great deal of credit goes to the lyricist, a man barely recognised today. Yip Harburg was born and raised in New York’s Lower East Side at a time, says his 82-year-old son Ernie, when it had a ”higher density than Calcutta”. Yip’s parents were Russian immigrants. As a child he slept on two chairs pushed together and watched his parents – to whom he once referred with tender shame as “loose screws in the world” – melt in the fire of the sweatshops. Like Baum, he went into business; like Baum he went bust. In the early 30s he became a songwriter and wrote a song that defined the decade, Brother, Can You Spare A Dime? In 1939 he wrote a song with Harold Arlen that defined something larger. Ernie Harburg shrugs, batting the air. “If Rembrandt did it, they’d say it was a masterpiece.”

Harburg and Langley lobbied hard to stick to the ethos of the book. It was Harburg who wrote the scene in which the Wizard hands out satirical gifts to the Scarecrow, Lion and Tin Man, an idea he devised, he told Harmetz, “because I was so aware of our lives being the images of things rather than the things themselves”. In the Emerald City, everyone has cod English accents and royal green dress – colonial humbug. At the time of its release, the flying monkeys inevitably seemed like bombers darkening the skies over Europe and the Winkies like a fascist army.

The film’s biggest departure from its origins was the creation of the three farmhands who, in Oz, became Dorothy’s companions and who, on her return to Kansas, implied the whole thing was just a dream. Harburg hated this ending, but I never thought it at odds with the reality of Oz. When Aunt Em says dismissively, “We dream lots of silly things”, it’s like that moment in ET when the adults are blundering blindly about and only the children can see the real possibilities of the world.

When Margaret Pellegrini was 14, something curious happened to her at the Tennessee state fair. Her father was 5ft 9in, her sister 5ft 7in and her brother would grow to be 6ft 2in. As she wandered with her family through the fair, she was approached, she says, by a “little person” who saw in her something she hadn’t seen in herself. “I was asked if I’d like to join their midget show. I said no! I had no idea I was going to stay small.”

The near mishaps of casting are well known by now – Shirley Temple as Dorothy would have been like Lassie putting himself forward for Toto; Fanny Brice and Gracie Fields were both considered for Glinda before Billie Burke got the role; and the first version of the Wicked Witch was as a sleek and glamorous fallen woman, before the film-makers thought it would send the wrong signal.

Pellegrini was contacted by MGM via the people at the fair, took leave from school and bought a ticket to LA. “To be in Hollywood!” she says. She was a year younger than Judy Garland. “I was very, very excited. And very bashful. Now, not so much.”

Stories from the set fascinate for their contrast with the happy tenor of the movie. For most of the actors, it was a miserable time. Margaret Hamilton, the Wicked Witch of the West, nearly died when her make-up caught fire. Her stand-in was blown off her broomstick in the skywriting scene and spent a week in hospital. Bert Lahr (the Cowardly Lion), Jack Haley (the Tin Man) and Ray Bolger (the Scarecrow) weren’t allowed to eat in the canteen because their make-up was considered too disturbing. Even Toto, after being blown across the floor like a toupee when the wind machines were turned on, had to have a stunt double.

As for Garland, at 16 she was under tremendous pressure to carry a film that would cost $2.7m, MGM’s most expensive of the year. Between takes, Pellegrini sat with her on the Yellow Brick Road, chatting like “regular teenagers”, although in later years Garland and her colleagues got fond capital out of Munchkin-lore. “We had a hell of a time with those little guys,” Mervyn LeRoy said. “They got into sex orgies at the hotel. We had to have police on every floor.” Garland told similar tales, and “embellished the truth later to make a better story”, says her daughter, Lorna Luft.

“Some of us were a little older than others,” says Pellegrini, who was cast as the second Sleepy Head to stand up in the nest and as one of the Flower Pot Girls. “But it’s all out of proportion. Just a few of them liked their drinks too much.”

Does she remember how much she was paid? Pellegrini laughs. “$50 a week, including room and board and transportation. You know what we found out later? Toto made more than we did.”

Of the principal actors, Garland earned the least – $500 a week. Luft says her mother loved making the film. “Adored it. It was the movie that put her into another stratosphere, that made her a star.”

Bolger and Haley were both on $3,000 and Lahr, a veteran of vaudeville, on $2,500. As for the songwriters, “their place was near the janitor!” Ernie Harburg says. “They were just guys you called in to do a little job for you. Work for hire. “

In the wrong hands, the score could have been a soppy nightmare. But Harburg, unusually for a songwriter of the period, wasn’t sentimental. He was a social activist, who’d be blacklisted in the communist witch-hunt and go on to write Finian’s Rainbow, a mixed-race musical for a long time considered too challenging for mainstream tastes.

“Yip was a tough-love guy,” Ernie says. “He wrote the schmaltzy stuff at the start of his career, but most of the time it was like Paper Moon: ‘It’s a Barnum and Bailey world/Just as phoney as it can be.’ He said art was for struggle and hope, that’s what his lyrics were about. Look, in the late 20s, his business went bust, his wife left, his brother died, his father died. Bang, bang, bang. You have to – like Obama said, he’s the same kinda guy – create hope. With humour, then imagination. You have to imagine: how can I get out of this mess? Then use your smarts.”

“If Yip Harburg were alive,” says John Lahr, theatre critic for the New Yorker and son of Bert, “he’d tell you he was writing a musical version of FDR’s New Deal.”

Over The Rainbow is perhaps the most misperformed song of all time. “People don’t get the point,” says Luft. “They make it into something more than it is, a big dramatic piece. If you look at where it came from, it’s a simple song.”

Initially Harburg thought Arlen’s melody too histrionic for a little girl and toned it down with childlike lyrics. “Lemon drops,” Ernie says. Is it significant the song ends in a question? “Yes.” He nearly jumps out of his chair. “That’s Yip. He was a philosopher – it’s a Socratic method, engage the listener by asking questions. Why oh why can’t I?” Over The Rainbow was thrown out of the film three times before Louis B Mayer intervened and said, “Let the boys have their damned song.”

When it came out, Oz was neither commercially nor critically successful. Garland, Harburg and Arlen won Oscars, but the film lost money at the box office and was dismissed by critics as having “no trace of imagination, good taste or integrity” (the New Yorker) and being full of “freak characters” (the New Republic). The best the New York Times could say was that it was “genial”. It wasn’t until 1956, when CBS leased the film from MGM (they’d bid for Gone With The Wind and got Oz as a consolation prize), to show on TV every year, that it started to build into a phenomenon.

The key to its appeal, John Lahr says, is the way it addresses fundamental anxieties in American culture. “What the story speaks to is mastering a sense of inadequacy that’s built into the American system. In other words: you’re free to become who you want. Which is terrifying, because you have no support. This sense of can I make it, am I good enough, do I have the right stuff? Oz is a little capitalist bliss, everything’s perfect, shiny, grand. And I think it speaks about longing and the feeling – the hope – that we’re all right inside ourselves and can reach that.”

By the time the protagonists find the wizard, they no longer need his magic, which, of course, turns out to be fake. Oz is an illusion. What’s real is the travellers’ original, unpolished value and the camaraderie of the journey. Although Bob Baum has reservations about the film – “It’s basically half the original book!” – he thinks it is broadly faithful to the spirit of what his great-grandfather intended. Which is? “That you have within yourself far more power than you’re actually using.”

Bert Lahr was ambivalent about the fuss around the film and the fact that, although he did serious theatre later – he played Estragon in the first New York production of Waiting For Godot – he’d always be remembered as the Cowardly Lion. “My father had no education,” says John Lahr, “in the great tradition of the early American comedians. Buster Keaton had one day of schooling, my father got to sixth grade. They sent us to these great schools, we were highly educated, and my sister and I were always trying to get Dad to do more avant garde things. But these guys, they wanted big houses. They were businessmen. My father once said to us, ‘Put me in a jockstrap and if I go out there and entertain people for two hours, I’ve done my job.’”

John’s sister, Jane, is writing a book about the theosophical foundations of Oz – L Frank Baum and Noel Langley were theosophists. “It’s the philosophy of know thyself,” she says. “It was very popular in the early 1890s and it doesn’t go out of fashion.” Her father “would only watch the film towards the end of his life. I think we forced him to, when we were back from college. He finally acknowledged it was wonderful, though he thought Ray Bolger was a bit of a ham.”

In 1970, the witch’s hat Margaret Hamilton wore sold at auction for $450. In 1988 it was re-auctioned for $33,000. In 2005, the original dress worn by Judy Garland in the film sold for $267,000 and every anniversary is greeted by new merchandise and previously undiscovered material – new behind-the-scenes stills from the film, in the case of the Blu-Ray Ultimate Collector’s Edition box set to be released this autumn for the 70th anniversary, and not to be confused with the 2005 Collector’s Edition, or the 1999 Wizard Of Oz DVD Gift Set.

Oz has been turned into books on philosophy (“Was Oz the dream or was Kansas?”), Jungian therapy (“Psychological healing through the archetypes of Oz”) and, my favourite, a self-help book called The Wizard Of Oz And Other Narcissists: Coping With The One-Way Relationship In Work, Love, And Family. (Baum’s character wasn’t a narcissist; he was just a bad wizard.) New York this summer has seen a new theatre production of The Wiz, the Motown version of Oz, and Wicked, the prequel, continues to play to packed houses.

The nearest threat to Judy Garland’s ownership of Over The Rainbow was probably Eva Cassidy’s version in 1992. “Well,” snorts Luft, “I wasn’t crazy about the way she fooled around with the melody. Not to take anything away from her talent. She was a lovely singer.”

Why was Garland’s the definitive version? “Because it was written for her, written for the film. It was the most perfect song in one of the most perfect movies ever made. Frank Sinatra said to me, ‘I wouldn’t touch that song with a 10ft pole.’”

Neither Luft nor her sister Liza Minnelli has ever sung the song, nor, says Luft, will she, although she did play the Wicked Witch in The Wizard Of Oz in Salford last year. She doesn’t need to see the film again – “Not at the age of 56. I’ve seen it 80 million times.” But when she was 16, after her mother died, for a while it was “one of the only ways I had to be near her. I’ve a sense of her being near me when I see it.” The film endures, she says, because of “her honesty. Her innocence. Not only her, all of them. You believe every one of them.”

It hasn’t dated. The only line that might be taken out today is the un-PC suggestion by Glinda that “only bad witches are ugly”. As childhood changes and anxious parents increasingly limit their offspring’s roaming rights, the story finds new resonance. If Oz has any metaphorical use, it is for the sense of wonder that greets the child’s first steps out into the world, the small explorations that lead to imagination and, through it, art.

“I’m afraid that when the Munchkins are gone, the book may stumble for a while,” Bob Baum says. “When we lose touch with the actual people in it.” At 85, Margaret Pellegrini is in constant demand; later this year she will travel to Oz festivals in Tennessee, Indiana, Kansas and Missouri. She feels nothing but gratitude to the film. “I was the little girl who lived over the tracks,” she says. “And then, when I came back to Alabama having been in a motion picture, all the VIPs like the lawyers and the doctors’ wives invited me to tea. I was – how would I say? – I was noticed.”

Luft’s children see the film as a branch of the family tree, part of their heritage, she says. “You have to put your arms around it and hold on to it and say thank you. That’s what I do.” Not that it isn’t without occasional annoyances: when she was in Australia earlier this year, people kept shouting at Luft, “Isn’t it funny, you’re in Oz?”

Ernie Harburg, meanwhile, after retiring from anthropology, spends his time spreading the word about Yip via the Yip Harburg Foundation. To his delight, a new production of Finian’s Rainbow will open in New York later this year. “When Yip was dying, I handed him a bunch of papers and said, ‘Yip, here is your elegant legacy’ which is a line from Finian’s Rainbow. Simultaneously I saw in him a tear and a twinkle. He said to me, ‘You’re pulling my legacy.’ ” Ernie smiles. “It got him out of the schmaltz of the moment.”

When John Lahr sees The Wizard Of Oz in the TV listings, or anywhere else it might casually arise, his reaction is “fairly complicated. I love it that Dad’s there, and he’s alive, and he’s in his prime. He’s 45, so I wasn’t quite born. And I know, from my own research, that his performance is a compendium of all his burlesque and vaudeville up to that point. It gives me a deep gratification that his skill and great capacity for making joy is condensed in that performance, and has preserved Dad for ever, really. But, like the Michael Jackson funeral, I don’t understand the idolatry. It scares me. The point of Harburg’s lyrics is to disenchant. The thing about fairy tales is that it’s not the spellbound who are free, it’s the disenchanted.”

Baum always said he came up with the name by glancing at the O-Z index on his filing cabinet, but it’s a word, as John Updike observed, that gets extra push from its echo of Shelley’s Ozymandias. “My name is Ozymandias, king of kings/Look on my works, ye Mighty, and despair!” No need to despair, as it happened. The king was long gone and his words merely etched into a monument, a ”colossal wreck, boundless and bare” that in the final line of the sonnet is revealed to lie abandoned in a desert; a reminder, where “the lone and level sands stretch far away”, of hubris past.

• What Would Barbra Do? How Musicals Changed My Life, by Emma Brockes, is published by Black Swan.