The incident took place early Sunday morning and has left Harkirat Singh, an immigrant from Punjab who moved to the US three years ago, scared.
"I'm so afraid. I don't want to work," Mr Singh told the New York Daily News. "It's an insult on my religion, also. An insult of my faith. It's horrible."
The report said the New York Police Department is investigating the incident as a possible hate crime. Mayor Bill de Blasio tweeted in his support, saying "Harkirat Singh - You are welcome here. What happened to you was wrong. You did the right thing by calling the NYPD."
Mr Singh said he picked up three men and a woman - all in their 20s - around 5 am Sunday from Madison Square Garden.
When they reached their destination in the Bronx, they began complaining that Mr Singh had taken them to the wrong address.
The passengers were allegedly drunk and could not give Mr Singh a straight answer about where they wanted to go, leaving him "confused."
The passengers then began hurling slurs and banging on the plastic partition in his cab, Mr Singh recalled.
"They're using bad words, also," he said recalling that those used expletives and called him "Ali Baba."
Mr Singh told the group to pay USD 41.76 and find another cab. While the women gave him the cash after Singh called 911, one of the men got back into the cab and tried to smash the meter. Then he punched Singh in the arm, he said.
"After that, he picked off my turban from my head," he said. "He wanted to snatch my phone also...It was too horrible."
"Why are you doing this, brother? We can sit. We can talk," he recalled telling the unruly passenger.
"At that time, I'm so afraid - they can do anything to me. They're gonna kill me."
With the police on its way, the group ran away from the scene with Singh's turban, a primary symbol of the faith. The incident occurred just hours after thousands of Sikhs had gathered at Times Square for the 'Turban day' event organised by a Sikh group to spread awareness about the faith.
The report said police sources described the suspect as a clean-shaven white Hispanic man in his 20s, around 5-foot-9.
Police say they are only seeking one suspect, though they'd like to speak to the other three passengers. Singh did not require a medical attention but filed a report with police. He was able to snap a photo of two of the passengers.
Harpreet Singh Toor, the chairman of public policy and external affairs at The Sikh Cultural Society, said the theft reflected prejudice against Sikhs, who often targeted in anti-Muslim bias despite following a different faith.
"I used to get mad - now I laugh at it, because if we are still ignorant about other faiths...who will make those people understand?" he said.
In the years since the 9/11 terror attacks, Sikhs remain more likely to be targeted in cases of profiling, bigotry and backlash than the average American.
In a latest incident of hate crime, a Sikh man was shot near Seattle last month after the gunman allegedly told him to "go back to your own country."