Mr Pai blasted the activists for "harassing" his family members, saying they "crossed the line".
Mr Pai is facing backlash from his opponents for his proposal which seeks to reverse Obama era's net neutrality, which meant that all websites are treated equally by Internet providers.
His proposal would end net neutrality and give big Internet providers like Comcast and Verizon broad powers to determine what website should reach out to subscribers faster than others.
Net neutrality activists not only protest outside his home with signs directing at his children, they also reportedly sent pizzas to his home every half-and-hour.
Some media reports said the frequency was every 15 minutes.
This is the first instance that a senior Indian-American administration official - in any of the presidential administration - being harassed and threatened by opponents not agreeing to his views.
Signboards and placards placed by protestors outside Mr Pai's house in Virginia mocked him for his views. "Is this really the world you want Annabelle and Alexander to inherit? How will they ever look you in the eye again," was written in one of the posters.
"They will come to know the truth. Dad murdered democracy in cold blood. And for what? It's not too late for you Chairman Pai. You do not have to be evil," written in another poster.
In an interview to Fox and Friends, Mr Pai said that his opponents have "crossed the line".
"I understand that people are passionate about policy, but the one thing in America that should remain sacred is that families, wives and kids, should remain out of it. And stop harassing us at our homes," Mr Pai told Fox and Friends.
Mr Pai's proposal is scheduled to come out to vote before FCC on December 14.
This is expected to pass as the Republicans have five votes as against three from the Democratic Party. Mr Pai expressed his displeasure when asked about the protest outside his house in Virginia.
"It was a little nerve-racking, especially for my wife," he said.
"Internet regulation activists have crossed the line by threatening and harassing my family. They should leave my family out of this and focus on debating the merits of the issue," Mr Pai said in a statement reported by The Washington Post.
"That's one of the things I think is very unfortunate about all the vitriol and hot air that's out there is that if you keep going out there and peddling this misinformation like 'this is the guy who is going to break the Internet and destroy democracy', it's not surprising that some people get alarmed by it," he said.
In an interview to PBS News early this month, Mr Pai had said the digital landscape has changed, and the government should stop "micromanaging the Web".
Instead, he argued, "companies should be transparent about their policies".
Free Press Action, which has been seeking net neutrality condemned the personal attack against Pai and his family.
"If you are sending racist message to FCC Chairman Ajit Pai, mocking his name, heritage or family, we do not want you in this new neutrality fight," it said.
According to Brendan Bordelon, a cybers ecurity reporter at National Journal, "Pizzas also reportedly set to his house every half-hour last night," he said, adding that "the racist memes and the hyperbolic social media attacks are bad enough. This is deeply disturbing".
The Indian-American community, who are quick in defence of such attacks, have surprisingly been quite on this so far, leaving Pai, 44, to defend himself alone.