The health ministry is to submit a proposed law to parliament by June that would ban smoking in many restaurants, as well as public facilities.
If approved, rule-breakers would face a penalty of 300,000 yen ($2,600).
But the proposals, unveiled last week, have already been watered down from an initial push for a broad ban on indoor smoking, which ran up against strong opposition from former state monopoly Japan Tobacco (JT) and many small business owners, who fear they'll lose customers.
Establishments can build a separate smoking room under the latest proposal.
Places with less than 30 square metres of floor space would not be affected by the restrictions at all - excluding a huge number of establishments in space-challenged Japan.
"It would be better if it's completely banned" in restaurants, Kumiko Tanaka, a 27-year-old non-smoker, told AFP.
"But I don't mind as long as smoking areas are strictly separated."
Even the less-expansive rules could be headed for trouble, however, as opposition brews within the ruling Liberal Democratic Party (LDP).
Gaffe-prone finance minister Taro Aso recently questioned the long-established link between smoking cigarettes and lung cancer.
This week, the LDP's parliamentary affairs chief Wataru Takeshita bemoaned the proposals, as a group in his party suggested restaurant owners instead be allowed to choose - and display their smoking status on the door.
"As a cigarette lover, I feel like this is not a good idea -- how would I live if smoking is banned everywhere?" Mr Takeshita told reporters.
This conflict leaves anti-smoking campaigners fuming in a country where cigarettes sell for around 430 yen ($3.75) a pack, and carry only modest health warning labels.
It is a far cry from other developed nations, including Australia and France, which have wide-ranging bans on smoking in public places and large, graphic health warnings on cigarette packs, which sell for two or three times more than in Japan.
Almost 50 countries have blanket bans on smoking indoors.
"We are really late," said Yumiko Mochizuki, a board member at the Japan Society for Tobacco Control (JSTC). "And we were given a very bad score by the World Health Organization."
The global health body has given Japan its lowest rating for efforts to prevent passive smoking - just 10 per cent of restaurants are non-smoking.
Japan even scores poorly among other Asian nations with smoking cultures such as China and South Korea.
"Countries such as the Republic of Korea, Hong Kong, Singapore, Australia, New Zealand, Vietnam and cities including Beijing and Shanghai China... have already introduced comprehensive smoking bans," said Susan Mercado, a Manila-based director at the WHO.
Smoking kills some 130,000 people in Japan annually, with another 15,000 dying of second-hand smoke related illnesses, Ms Mercado added.
Despite its many smoking venues, tobacco use in Japan has been falling in line with a broader global trend.
More than half of Japanese men smoked in 2000. That number dropped to just below 30 per cent in 2016.