Starboard in a letter to the board of directors implied that Mayer and her officers needed to go, without naming her specifically.
"To be successful, dramatically different thinking is required, together with significant changes across all aspects of the business starting at the board level, and including executive leadership," Starboard CEO Jeffrey Smith said in the letter on Wednesday.
Starboard, which owns about 0.75 per cent of Yahoo, has been pushing for changes at the Internet company since 2014, urging it to separate its Asian assets and auction off the core business.
Starboard has threatened to shake up the board if Yahoo's stock continued to suffer.
"Unfortunately, it appears that shareholders have no confidence that management and the board will be able to execute on a separation of these assets or improve the performance of the core business," Smith said in the letter.
Yahoo spokeswoman Rebecca Neufeld said the company will provide more details on its turnaround plan prior to its fourth quarter earnings call later this month.
"Our board and management team engage in and maintain regular, open dialogue with all our shareholders, and consistently strive to deliver and to maximize shareholder value," Neufeld said in an emailed statement.
Starboard, together with other shareholders, have demanded Yahoo separate the Asian assets, including stakes in Chinese e-commerce company Alibaba and Yahoo Japan Corp, and conduct an immediate public auction of the core business, including search and advertising businesses.
But Yahoo is resisting, instead pursuing a tax-free spinoff of the core business, which could take at least a year.
Some analysts think board upheaval may be in the works.
"If the board is sufficiently confident that their current management team has the right plan, then the board needs to prepare to be replaced because that's probably going to happen," said Pivotal Research Group analyst Brian Wieser, who recommends holding the stock. "I don't think many shareholders will get into the way of an activist slate."
Yahoo is scheduled to report quarterly earnings on Jan. 26. The deadline for shareholders to nominate directors for the anticipated May annual meeting is expected to be in late February through March 26.
MARKET VALUE HAS "COLLAPSED"
Smith said in the letter he is confident that buyers are expressing interest in Yahoo's core business, but the company is not pursuing the suitors.
Verizon Communications is among the technology, media and telecommunications companies seen as potential buyers of Yahoo's core business. Verizon's chief financial officer said in December such a sale could make sense though it was premature to discuss.
After heading Yahoo for more than three years, Mayer has been unable to revive growth in its revenues and faces stiff competition from the likes of Facebook Inc and Alphabet Inc's Google unit.
Some shareholders said they doubt whether Mayer has a robust plan to revive the struggling Internet media business.
They have criticized her for a series of ineffective acquisitions and being unable to stem a continued decline in the value of the Internet business, citing them as reasons to replace her.
In the letter Smith claimed Yahoo's market value, without its Alibaba shares, "has collapsed and is currently trading near zero."
The Alibaba stake, worth more than $30 billion, accounts for the bulk of Yahoo's market value, while its 35 percent stake in Yahoo Japan is worth $8.5 billion.
The plan to spin off the Alibaba stake hit a hurdle in September when the U.S. Internal Revenue Service did not make a decision on Yahoo's request for a ruling on whether the transaction would be tax-free. If taxable, the deal could cost shareholders billions of dollars.
In December, Yahoo shelved plans to spin off the Alibaba stake and said it would create a separate company that would house Yahoo's Internet business and its stake in Yahoo Japan.