A five-member bench of the Supreme Court on Friday disqualified 67-year-old Sharif for dishonesty and ruled that corruption cases be filed against him and his children over the Panama Papers scandal, forcing the embattled leader out of office.
Mr Sharif left the PM House with his wife Kulsum Nawaz, daughter Maryam Nawaz and his son-in-law Captain (Retd) Safdar and shifted to his private residence in the hill resort of Murree near Islamabad.
He met the staff before leaving. The luggage of the former first family was already shifted, local media reports said.
Ousted finance minister and Mr Sharif confidante Ishaq Dar also left for Murree with the Mr Sharif family.
The PM House, nestled in the foothills of Margalla Hills overlooking the capital Islamabad, would be renovated for the new occupant, officials said.
It is not known for how long Mr Sharif would stay in Murree, a place he likes a lot and has been visiting frequently since he was the chief minister of Punjab.
Mr Sharif stepped down after the Supreme Court disqualified him for failing to declare "receivable" salary from a UAE based company of his son in his 2013 nomination paper. The court termed the salary which was not paid to Mr Sharif as an "asset".
Senior PML-N leader and former petroleum minister Shahid Khaqan Abbasi will run the PML-N government as interim prime minister of Pakistan until Shehbaz Sharif gets set to succeed his brother Nawaz Sharif as the Pakistan prime minister.