Moscow has implemented a Fan ID system that exempts visiting fans from Russia's strict visa system and allows them to move freely between host cities for the tournament between June 15 and July 15.
Belarus, which signed an agreement with Russia allowing Fan ID holders to transit the country without a visa, said it had stopped four Moroccans and an unspecified number of Pakistanis as they tried to enter the European Union illegally across borders with Lithuania and Poland.
"More 'illegals' have entered Belarus under the guise of football fans," Andrey Sytenkov, an investigator with Belarus's State Border Committee, said at a press conference in Minsk on Wednesday.
The Moroccans are currently in temporary detention centres and face fines, deportation and a five-year ban on entering Belarus, he added. There was no word on the whereabouts of the Pakistani nationals.
Sytenkov said the Belarusian border guard service "has information on groups of pseudo-fans" on the country's Western borders.
"We are ready for the fact that Europe will not accept them and will be sending them back," he said.
Similar cases have also been reported in Finland, an EU member that also shares a border with Russia.
Finnish Border Guard official Marko Saareks told AFP there have been three cases of people crossing the Russian-Finnish border and applying for asylum in Finland after arriving in Russia with Fan IDs.
"There is a threat of this kind of phenomenon," he said, adding "these kinds of illegal crossings may go on until the end" of the World Cup.
Last Friday, a Chinese citizen flew to Helsinki from Russia and applied for asylum after arriving in Russia with a World Cup Fan ID.
In another case on the same day, a Nigerian with a fake Brazilian passport tried to cross the Finnish border. He had earlier arrived in Russia with a Nigerian passport and a Fan ID.
On Sunday, three Moroccan citizens with Fan IDs managed to cross the border illegally before being apprehended by border guards.
Saareks stressed that World Cup Fan IDs are not travel documents to the EU.
All five men have applied for asylum and the Finnish Border Guard has transferred their cases to the migration authorities.
The Fan ID system, introduced at this World Cup, also serves as an additional security measure, with Russian police conducting background checks with the help of their foreign counterparts.
Nearly 500 people were denied Fan IDs ahead of the tournament which is taking place in 11 cities across Russia, ending with the final in Moscow on July 15.
(Except for the headline, this story has not been edited by NDTV staff and is published from a syndicated feed.)