Demonstrators held a minute’s silence with their backs to parliament, then shouted “resign” with fists clenched, BBC reported.
Parliament was guarded by hundreds of police officers.
New figures this week showed about a quarter of working-age people in Spain were now unemployed.
Saturday’s protesters came from all over the country and were met by vans of riot police.
Just hours earlier, 300 police had staged their own protest in the capital, setting off fire crackers and blowing police whistles over the same issue – budget cuts.
One banner read: “The police can’t take it any more.”
Austerity protests also took place in Barcelona, Valencia and other cities.
One protester in Madrid, Sabine Alberdi, said, “I came to demonstrate because they’re taking everything away, our health, our education, our houses.”
Mr. Rajoy’s program will require spending cuts of 150bn euros between 2012 and 2014.
Spain will brace for its second general strike in less than one year on November 14.