And when it became clear there was no possibility of Labour continuing in charge, Mr Brown announced he was standing down.
Making a statement outside 10 Downing Street before leaving for Buckingham Palace with wife Sarah and their two sons to officially offer his resignation to the Queen, he said he would be asking the Queen to invite David Cameron to form a government.
In an emotional speech, Mr Brown said: "I wish the next Prime Minister well as he makes the important choices for the future.
"Only those who have held the office of prime minister can understand the full weight of its responsibilities and its great capacity for good.
"I have been privileged to learn much about the very best in human nature and a fair amount too about its frailties - including my own."
While Mr Brown has endured some tough times in office, particularly overseeing Britain's reaction to the global financial crisis, he insisted he had enjoyed his job.
He said: "I loved the job, not for its prestige, its titles and its ceremony, which I do not love at all.
"No, I loved the job for its potential to make this country I love fairer, more tolerant, more green, more democratic, more prosperous, more just - truly a greater Britain."
Mr Brown paid a special tribute to the armed services.
He said: "Now that the political season is over, let me stress that having shaken their hands and looked into their eyes, our troops represent all that is best in our country and I will never forget all those who have died in honour and whose families today live in grief."
And there was also special mention for his family, as he can look forward to devoting more time to them.
He said: "Above all I want to thank Sarah for her unwavering support, as well as her love.
"I thank my sons John and Fraser for the love and joy they bring to our lives.
"As I leave the second most important job I could ever hold, I cherish even more the first, as a husband and a father.
"Thank you, and goodbye."
Mr Brown's audience with the Queen lasted 20 minutes, and he was then driven to Labour headquarters.
Having announced yesterday he was standing down as Labour leader but would continue until his replacement was appointed, Mr Brown today said he was quitting with immediate effect.