Mr Gandhi, a 41-year-old presumed "prime-minister-in-waiting", led campaigning in Uttar Pradesh in a bid to revive Congress - his biggest challenge yet in a state where the party has a dismal record stretching back 22 years.
Early counting - just a handful of results have been confirmed - showed Congress winning 29 out of 403 seats, representing a small increase on their miserly tally in 2007 despite Gandhi's tireless campaigning.
The failure will likely feed the doubts about his ability to lead the world's biggest democracy and might rekindle interest in his sister Priyanka, whom some Gandhi loyalists still prefer.
"We are disappointed by the trends," Ashwani Kumar, minister of state for science and technology, admitted as the counting of nearly 76 million votes cast at 138,000 polling stations in UP gathered pace.
"These results if they eventually turn out the way they are now, are a cause for reflection and serious introspection on what went wrong."
The tragedy-plagued Gandhi family has dominated politics in India since independence in 1947, providing three prime ministers, two of whom were assassinated.
"One thing is clear, the Nehru-Gandhi charisma is no longer a major factor in winning elections," political analyst and commentator Parsa Venkateshwar Rao told AFP.
"The opposition can take satisfaction from the fact that Rahul Gandhi has not really emerged as a leader of national stature."
Prime Minister Singh, a former academic and economist, is also likely to face more searching questions about his hands-off style of leadership and his stewardship of the government since his re-election in 2009.
A series of high-profile corruption scandals and a popular anti-corruption campaign led by activist Anna Hazare last summer have badly damaged Singh's image and hurt the party in the state polls.
Following a record high turn-out of 59.5 per cent, UP's incumbent Chief Minister Mayawati, a colourful low-caste leader famed for her handbags and taste for expensive statues, was also heading for a thumping defeat.
Victory appeared almost certain for the regional Samajwadi Party (SP) which is headed by a former wrestler and draws on support from low-caste farmers and Muslims.
Congress lost heavily in the holiday state Goa, where it had been in power, and was struggling to regain the agricultural heartland of Punjab and the mountainous northern state of Uttarakhand.
Impoverished Manipur in the far northeast of India was the bright spot for Congress, where the party looked certain to remain in power.
Mr Rahul, who is an elected national lawmaker from UP, pitched himself to the voting masses as a man who understands the everyday problems they face, including corruption, unemployment and caste discrimination.
He said he wanted to bring development and better governance to notoriously corrupt and economically deprived UP, which has a population larger than Brazil.
The elections in the five states were held in stages over the last five weeks, beginning on January 28 in Manipur and finishing at the weekend in UP and Goa.