Rabindra Nath Ghosh, who is the minister for North Bengal Development in Mamata Banerjee's government, denied getting violent. "I just drove people away using my hand, that's all," the minister was quoted as saying by news agency ANI.
The man, he said, was a BJP agent who was trying to make off with a ballot box and a commotion started when the security officers caught hold of him. Mr Ghosh claimed that was when he tried to shoo off the people.
In the video, the man is seen trying to get away as several people from the mob aim blows at him. A policeman is seen among the mob, but he does nothing to protect the man.
Poll-related violence, being reported throughout the state, has claimed six lives since last evening. Two of the people belong to Trinamool Congress, two to the CPM.
Violence has also been reported from neighbouring Alipurduar district.
The opposition BJP and the CPM have called it a "murder of democracy".
State BJP chief Dilip Ghosh said, "There is not even a single district where violence has not taken place... TMC ministers are openly slapping BJP workers inside the polling booth and that too in presence of the local police. They are capturing booths... threatening people to vote in their favour... They are openly doing hooliganism".