The party had installed the previously little-known Grindeanu, 43, as premier in January because party boss Liviu Dragnea, 54, is barred from holding office due to a conviction for voter fraud.
Dragnea continued to pull the strings behind the scenes and Grindeanu, widely seen as his puppet, at first complied. But in recent months he began asserting his independence.
This reportedly infuriated Dragnea and the PSD last week withdrew its support for Grindeanu, accusing him of "delays" in implementing badly needed reforms in the European Union's second-poorest country.
Grindeanu however refused to resign and accused Dragnea of seeking to "concentrate all the power in his hands".
Wednesday's no-confidence motion passed with 241 in favour to 10 votes against. The centre-right opposition - usually no fans of Grindeanu - abstained.
Dragnea said he wanted to "put a halt to a government that has not respected its programme and electoral promises."
"Allowing things to continue as they were would have led to failure," he said.
A defiant Grindeanu called the no-confidence motion "incomprehensible", saying his government had "worked well".
The PSD and its small coalition partner, the ALDE party, are now expected to propose a new premier to centre-right President Klaus Iohannis.
Once nominated, the incoming prime minister will then have 10 days to secure a vote of confidence in parliament for his cabinet and policy plans.
Ahead of the vote, Iohannis had urged the government to resolve the crisis quickly.
Romania's economy has been doing well, enjoying the fastest growth rate - 5.6 percent - in the EU in the first quarter, while efforts to tackle corruption have begun to bear fruit.
But the country of 20 million inhabitants can ill afford political instability, with the International Monetary Fund and the European Commission warning more reforms are sorely needed.
It is the second major crisis to hit the PSD since it rode back into power in December, barely a year after being forced from office over a deadly nightclub blaze blamed on corruption.
In February, Romania's largest protests since the fall of communism forced the government to drop a bill aimed at watering down anti-corruption laws that critics said would help Dragnea himself.
Analysts say Grindeanu's fall from grace might be linked to his perceived failure to push through the legislation. Dragnea is also fighting another case against him for alleged abuse of power.
"Liviu Dragnea only wants one thing -- amendments to the anti-corruption laws," former PSD member Alin Teodorescu recently told AFP.