Election officers said by midday the number of voters increased and voting was peaceful.
The bypolls were necessitated after Uttar Pradesh chief minister Yogi Adityanath and deputy chief minister Keshav Prasad Maurya vacated the Gorakhpur and Phulpur Lok Sabha seats, respectively, following their election to the state legislative council.
After casting his vote in Gorakhpur, Mr Adityanath said the BJP was getting widespread support from the people. "The people know that development was the only panacea," he told reporters.
Attacking the Bahujan Samaj Party (BSP) and the Samajwadi Party (SP), the chief minister said, "These parties are indulging in negative politics, politics of bargaining and politics of opportunism. The state has suffered the negative consequences."
In what could be a "rehearsal" for the 2019 Lok Sabha election, the by-elections are seeing a triangular contest involving the ruling BJP, the SP and the Congress.
The BJP has fielded Kaushalendra Singh Patel from Phulpur and Upendra Dutt Shukla from Gorakhpur against the SP's Pravin Nishad and Nagendra Pratap Singh Patel, respectively. The Congress has nominated Sureetha Kareem for Gorakhpur and Manish Mishra for Phulpur Lok Sabha seats.
Phulpur, once represented by the country's first Prime Minister Jawaharlal Nehru, saw a BJP surge in the 2014 Lok Sabha elections and Mr Maurya won there.
Apart from the Provincial Armed Constabulary (PAC) and Home Guard personnel, 65 companies of central paramilitary forces have been deployed to ensure peaceful bypolls.
In Gorakhpur parliamentary constituency, there are 970 polling centres and 2,141 polling booths, while in Phulpur, there are 793 polling centres and 2,059 polling booths. According to the Election Commission, there are 19.61 lakh voters in Phulpur parliamentary constituency, while the Gorakhpur Lok Sabha seat has 19.49 lakh electorates.
As many as 4,728 voter-verifiable paper audit trail machines are being used in the bypolls and provisions for web-casting from 95 critical booths have been made.