Two of the planets are super-Earths located in the habitable zone of the star, tau Ceti, meaning they could support liquid surface water, said researchers at University of California (UC), Santa Cruz in the US.
The planets have masses as low as 1.7 Earth mass, making them among the smallest planets ever detected around nearby Sun-like stars, they said.
They were detected by observing the wobbles in the movement of tau Ceti. This required techniques sensitive enough to detect variations in the movement of the star as small as 30 centimetres per second.
"Our detection of such weak wobbles is a milestone in the search for Earth analogs and the understanding of the Earth's habitability through comparison with these analogs," said Fabo Feng from the University of Hertfordshire in the UK.
"We have introduced new methods to remove the noise in the data in order to reveal the weak planetary signals," said Feng, lead author of the study published in the Astrophysical Journal.
"We are slowly learning to tell the difference between wobbles caused by planets and those caused by stellar active surface. This enabled us to essentially verify the existence of the two outer, potentially habitable planets in the system," said Mikko Tuomi from the University of Hertfordshire.
Sun-like stars are thought to be the best targets in the search for habitable Earth-like planets due to their similarity to the Sun, researchers said.
Unlike more common smaller stars, such as the red dwarf stars Proxima Centauri and Trappist-1, they are not so faint that planets would be tidally locked, showing the same side to the star at all times.
Tau Ceti is very similar to the Sun in its size and brightness, and both stars host multi-planet systems.