The Goods and Services Tax (GST), he said, has brought more than 17 lakh new traders into the indirect tax system within two months.
He asked them to ensure that the benefits of the biggest tax reform since independence reach the common man and even small traders with turnover below Rs 20 lakh are registered in the new system.
Asking taxmen to improve their work culture and incorporate both a sense of urgency and measurability in their performance, PM Modi said officers must fix clear targets to improve the country's tax administration by 2022, the 75th anniversary of Independence.
Since coming to power in May 2014, PM Modi has assured taxpayers that he will remove the fear of tax officials hounding them by ushering in accountability and answerability.
Measures taken include cutting down physical interface between the taxpayer and official to curb corruption and easier online filing of returns and claiming of refund as also paperless email-based inquiry and e-scrutiny.
"The Prime Minister said human interface must be kept to a minimum in the tax administration's dealings. He asked for a push to be given to e-assessment and anonymity of proceedings using technology, so that vested interests do not impede the due course of law," an official statement said.
The government, he said, is working towards creation of an environment which shatters the confidence of the corrupt, and instills trust among the honest taxpayers.
PM Modi mentioned steps like demonetisation and implementation of stringent laws against black money and benami property in this regard.
Expressing dismay at the huge pendency of tax-related cases in adjudication and appeal, he said big sums of money locked up in these cases could have been used for the welfare of the poor.
He asked officers to come up with an action plan during the Rajaswa Gyan Sangam to eliminate pendency.
The closed-door two-day conference is being attended by senior officers of the Central Board of Direct Taxes (CBDT) and Central Board of Excise and Customs (CBEC).
CBEC in a Twitter post said, "Hon'ble PM @narendramodi directed the tax administrators to be friendly to the honest taxpayers."
Though efforts to increase tax revenue are made by officers each year, the estimated amounts of tax that should accrue to the system are often not realised, he said.
He asked officers to come up with a time-bound solution to 'tax raised and not realised', and asserted that the honest cannot continue to pay the price for the misdeeds of the dishonest.
In this regard, he also suggested complete reworking of human resource management in the tax departments to strengthen the data analytics and investigation wing.
The Prime Minister said that in order to enable all traders to take maximum benefit of GST, it should be ensured that all traders, including even relatively smaller traders with a turnover below Rs 20 lakhs, are registered with the GST system.
He asked the officers to make efforts in this regard by designing a system for this category.
CBEC said PM Modi appreciated the efforts put in by Central and state officials for smooth implementation of the GST, which unifies more than a dozen levies like excise duty, service tax and VAT.
GST, which PM Modi previously described as a 'good and simple tax' that will help end tax terrorism and inspector raj, was rolled out from July 1, making India one market and ending cascading effect of multiple taxes.
"Hon'ble PM @narendramodi desired that the benefit of GST should percolate to the common man," CBEC said. GST is expected to boost economic activity as same rate for goods and services makes business easy and also facilitates smooth movement of goods and services.
In his address, Finance Minister Arun Jaitley commended the efforts made by the central and state tax officials to make GST a reality.
He also thanked PM Modi for his "visionary policies and motivation", CBEC said in a tweet.
The objective of the two-day Rajaswa Gyan Sangam is to enable a two-way communication between the policymakers and the senior officers in the field offices with a view to increase revenue collection and facilitate effective implementation of law and policies, the finance ministry had earlier said.