Mr Ali, 47, who works as a carpenter, had shifted base from Kolkata 20 years ago. His voter ID card is dated 1995. In 2016, he got his Aadhar number on his Kolkata address, where his ailing parents live. Even his parents' names in 1966 voter rolls were declared unacceptable by the court, which said those were not attested by the Election Commission.
According to the Assam Accord signed in 1985, anyone who entered the state after 1971 is considered an illegal resident. The rule was framed with a view to weed out illegal migrants from Bangladesh. But rights lawyers say most of those who get penalised are impoverished people like Mr Ali, who cannot maintain proper records.
The Foreigners Act is around 20 years old, but activists say detention under it has picked up pace after the BJP government - which made a pre-election promise of ousting migrants -- came to power in the state last year.
Until 2016, around 2,500 people on an average were declared foreigners each year. But over the last year, the number rose nearly six times -- to more than 12,000.
NDTV found government documents that say from 1985 to October 2017, 4.85 lakh cases have been disposed by the Foreigners Tribunal in Assam. Of them, 92,304 -- around 30 per cent -- people were found to be illegal migrants, which is 0.29 per cent of the state's population.
The rest were Indians who were suspected to be illegals.
In 1997, Shah Alom Bhuyan was marked as a D-voter or Doubtful Voter -- the first step in being labelled a foreigner. Mr Bhuyan is officer of Assam Police, and is posted as a security officer in Chief Minister Sarbananda Sonowal's residence. He said there was no verification before he was marked a D-voter.
"For 20 years I could not vote. I had to move tribunals, engage a lawyer and spend a lot of money to raise the issue in the tribunal," Mr Bhuyan said. He has been declared Indian this year by the tribunal.
The government claims the National Register of Citizens -- a massive database meant to determine who is Indian and who is not -- would solve the issue once for all.
"No bonfide Indian need to be worried. We are doing the NRC with the sole purpose of safeguarding the genuine Indians," said Chief Minister Sarbananda Sonowal.
But then, in homes like Asgar Ali's, patience is running out. "His mental condition is very bad. I cannot go to meet him leaving the child. My in-laws are old and bed-ridden... he was the only earning member," said his wife Sehnaz Begum.