PM Modi made the remark while lashing out at the Congress for its criticism of the Rs 1.1 lakh crore project, which he said was "a negligible price", connecting Ahmedabad and Mumbai.
He said the Congress government too wanted the project but failed to get it and so was now opposing it.
"Those opposing the bullet train project should travel on bullock carts. We won't mind," he told an election rally at the Amod Sugar factory near this Gujarat city on the banks of the Narmada.
PM Modi said the Mumbai-Ahmedabad high-speed rail corridor to be built by the state-run Japan International Cooperation Agency would open up a huge employment opportunity in Gujarat.
"Imagine how much employment will it generate for the region, for Bharuch. I ask you where will the cement for the bullet train (infrastructure) come from, iron come from, labourers come from? Won't that be from India? And who will be buying that? Japan. Isn't it a big deal?" PM Modi asked.
He said the UPA government under Manmohan Singh also wanted the project but could not get it. "The NDA government got it at a negligible price. So the Congress didn't like that. My only grievance against the Congress is that if they could not achieve anything, why does it pain them when someone else does it."
Modi took a jibe at the Nehru-Gandhi family over what he said their claim to have Gujarati origins and doing nothing for the state.
"But since the BJP formed the government in the Center, all these ailments have gone. When I was the Chief Minister, there was not a single day when they would not hurt Gujarat."
He said the two districts of Kutch and Bharuch having large Muslim population witnessed "maximum development during the BJP rule".
He said the mega project of the "Statue of Unity" to install the world's tallest statue of Sardar Vallabhbhai Patel will also contribute to the development of the region and attract foreign visitors.
The Prime Minister said the government planned to develop some 1,300 small islands and islets around the marine border of the state, many of which are even bigger than Singapore.
"Imagine these islands getting developed as Singapore. So, for a better tomorrow of Bharuch and Gujarat you should vote for the BJP."
The Prime Minister is on a two-day visit to the state. On Monday, he will visit Dharmapur in Valsad district and then in Saurashtra's Bhavnagar, Junagadh and Jamnagar.