Military spokesman Colonel Ahmed Aly said in a statement that a "terrorist group" had been targeting a police vehicle but hit the workers' bus by mistake.
"The bus was attacked with a rocket-propelled grenade near Al-Arish airport. Three people died and 17 were injured in the attack," a security official said.
A medical source confirmed the toll, adding that "many of those injuries are critical." The bus was carrying workers from a cement factory.
The attack came hours after gunmen clashed with the Egyptian army near the Israeli border.
The fighting broke out in the area of Al-Wifaq, in northern Sinai, after militants tried unsuccessfully to blow up a police vehicle with explosives.
The restive Sinai peninsula has been hit by a surge of violence since president Mohamed Morsi's ouster on July 3, with militants killing a police officer early on Friday.
A Coptic Christian man found decapitated a day earlier, while two people died in an attack on a checkpoint in the peninsula on Wednesday.