Mr Gupta will be the new deputy chief minister in place of Nirmal Singh, who submitted his resignation last night, officials said.
The new BJP faces sworn in as ministers were the party's state unit chief Satpal Sharma and the legislators from Kathua and adjoining Sambha, Rajiv Jasrotia and Devinder Kumar Manyal, respectively.
The BJP also upgraded Sunil Sharma, at present minister of state for transport, to the rank of a cabinet minister. BJP legislator Shakti Raj took oath as a minister of state.
The PDP members included in the cabinet were Mohd Khalil Band, MLA from Pulwama, and Mohd Ashraf Mir, legislator from Sonwar.
The BJP has dropped Nirmal Singh, Health Minister Bali Bhagat and Minister of State (MoS) for Tourism Priya Sethi.
Their resignations have been accepted along with that of Law Minister Abdul Haq Khan by Governor NN Vohra, an official said, adding that Sunil Sharma, who was upgraded as cabinet minister, also resigned as MoS.
The oath of office and secrecy was administered by the governor at the convention centre instead of the Raj Bhavan as the state government has already shifted its base to summer capital Srinagar, part of the bi-annual darbar move under which the government functions for six months in Jammu and six months in Srinagar.
The BJP had on April 17 asked all its nine ministers in the PDP-BJP government in the state to submit their resignations to enable bringing in new faces in the two-year-old Mufti cabinet.
BJP ministers Lal Singh and Chander Prakash Ganga, who took part in a rally in support of those accused in the rape and murder of an eight-year-old girl in Kathua, had to resign from the government earlier this month.
The state can have a maximum of 25 ministers, including the chief minister, out of which 14 portfolios are with the PDP and the remaining with the BJP.
Follow NDTV for latest election news and live coverage of assembly elections 2019 in Maharashtra and Haryana.
Subscribe to our YouTube channel, like us on Facebook or follow us on Twitter and Instagram for latest news and live news updates.