Araria has a high concentration of Yadavs or the Muslims, communities that have been traditional supporters of the RJD.
On the last day of campaigning, Mr Rai had allegedly said, "if Sarfaraz (RJD candidate) wins this election, Araria will become a safe haven for the ISI (the Pakistani spy agency, Inter-Services Intelligence). The victory of our candidate Pradip Singh will, on the other hand, give rise to patriotic fervour".
RJD candidate Sarfaraz Alam is the son of former lawmaker Mohammad Taslimuddin, whose death last year made the elections necessary. Taslimuddin had wrested the seat in 2014 from BJP's Pradeep Kumar Singh, whom the party has pitched again in this election.
Calling Mr Rai's remark "outrageous", RJD national vice president Shivanand Tiwary told news agency Press Trust of India that the statement shows the "desperation in the BJP, which is faced with the possibility of a defeat in Araria".
The by-election in Bihar is seen a referendum on Chief Minister Nitish Kumar's alliance with the BJP that was crafted last year at the expense of the ties with RJD and the Congress. Lalu Yadav's party, which held Araria, had repeatedly called it a violation of the anti-BJP mandate delivered by the people of Bihar in the 2015 assembly elections.
The BJP-JD(U) alliance has been out to win back Araria from the RJD, which won the seat in the 2014 general elections. It was one of the four seats the RJD had won in that election -- a huge contrast to the 31 of 40 Lok Sabha seats in the state that were bagged by the NDA. But the assembly elections held the next year melted the BJP's advantage in the state as the Grand Alliance of Nitish Kumar's JDU, the RJD and the Congress won 178 seats in the 243-member house. The RJD emerged as the leader of the pack with 80 seats.
But last year, Nitish Kumar ended the alliance and came back at the head of another government formed with the BJP - a move that has been repeatedly castigated by Lalu Yadav and his son Tejashwi Yadav. The youngest son of Lalu Yadav, seen as his political heir, has been leading the party campaign in the absence of his father, who has been in jail since January, following his conviction in two corruption cases.