Mr Kharge was inaugurating a training camp for Congress functionaries.
Congress president Mallikarjun Kharge on Friday said he had recently spoken to Prime Minister Narendra Modi about the BJP's "poaching" of leaders from the opposition ranks and asserted it was being done by "scaring" them.
Against the backdrop of former chief minister Ashok Chavan joining the BJP recently, Mr Kharge said such switching of sides was an act of "cowardice".
He was speaking after inaugurating a two-day state-level training camp in Lonavala for Congress functionaries.
"During a tea meeting in Parliament where Prime Minister Narendra Modi was present, I asked him how many people are you (BJP) going to poach as ministers and ex-chief ministers are joining your fold. I asked him about the appetite (to take leaders from opposition ranks into the ruling party)," Mr Kharge told the gathering.
"He (PM) said what could he do if people want to join the BJP. I told him they (BJP) are carrying out this work (poaching) by scaring people. He said these people want to join because of their (government's) work," the Congress chief added.
Slamming turncoats, Mr Kharge said party workers and voters made some persons "big leaders", who then ran away, "which is nothing but an act of cowardice".
"But we do not have to get scared. If we are scared, we will perish, But if we fight, we will live and, one day, victory will be ours. I am sure you are all ready for this," he told his party's workers.
Mr Kharge pointed out that Congress leader Rahul Gandhi has said minimum support price (MSP) will get a legal guarantee if his party comes to power at the Centre.
This assurance has been welcomed by people, Mr Kharge claimed.
Several farm outfits from Punjab have begun a march towards New Delhi seeking, among other things, legal guarantee for MSP, implementation of the Swaminathan Commission's recommendations, pension for farmers and farm labourers, farm debt waiver and no hike in electricity tariff.