Michael McCormack, 53, was chosen by the Nationals -- the junior partner in the governing Liberal-National coalition -- to take over as leader and deputy PM in an internal party vote.
"We are the party for small business and farmers and we want to make sure that continues and that can only continue with a close relationship with the Liberals," McCormack told reporters in Canberra after winning the vote.
"Look forward to having a good discussion with (Prime Minister and Liberals leader) Malcolm Turnbull in a few moments," he added.
McCormack, who had been the veterans' affairs minister, was sworn in as deputy PM and infrastructure minister shortly after his appointment.
He was widely expected to win the leadership role after other Nationals MPs withdrew from the race.
Maverick Queensland MP George Christensen launched a late bid for the leadership, but was defeated. The vote count was not released.
McCormack is not as well-known as Joyce, who made international headlines for threatening to euthanise Hollywood star Johnny Depp's dogs over a quarantine violation.
But he is seen as a safe pair of hands as the Liberal and National parties seek to repair their relationship after the Joyce scandal.
Tensions between the two parties rose after 50-year-old Joyce's affair with ex-staffer Vikki Campion, 33, was splashed across the frontpage of Sydney's Daily Telegraph in early February.
Joyce in turn slammed Turnbull's comments as "inept" and initially refused to step down, before resigning Friday after a separate sexual harassment complaint was lodged against him. He denied the allegation.
Turnbull welcomed McCormack's appointment, saying he expected the more than 70-year coalition between the two parties would continue under his leadership.
The coalition has endured a tumultuous few months, with the scandal coming on the heels of a dual citizenship crisis that involved some of its lawmakers, including Joyce, and threatened its wafer-thin parliamentary majority.
McCormack, first elected to the lower House of Representatives in 2010 representing the rural region of Riverina in southwest New South Wales state, is not without controversy.
More than two decades ago as a local newspaper editor he wrote a column in which he described homosexuality as "sordid behaviour".
In August, as he oversaw a voluntary nationwide postal vote on same-sex marriage as the minister in charge of the government agency running the poll, he "apologised wholeheartedly for the comments at the time".
McCormack later voted in parliament in support of amending the Marriage Act to legalise gay unions.
(Except for the headline, this story has not been edited by NDTV staff and is published from a syndicated feed.)
Get the latest election news, live updates and election schedule for Lok Sabha Elections 2019 on ndtv.com/elections. Like us on Facebook or follow us on Twitter and Instagram for updates from each of the 543 parliamentary seats for the 2019 Indian general elections.