Opposition parties have alleged that he only says what he wants to and does not listen to the people's voices.
He added that he had not thought the programme would be viewed politically when he launched it more than two years ago.
"Some people take 'Mann Ki Baat' as a monologue and some criticise it from a political angle," Mr Modi said in his radio programme, the first after the completion of three years of his government.
"When I started 'Mann Ki Baat', I had not thought so," he added.
Through the programme, he said he felt like he was conversing with "my family while sitting at home". He added that there are many families who have written to him these very feelings.
Prime Minister Modi then referred to the launch of an analytical book on 'Mann Ki Baat' by President Pranab Mukherjee two days ago in the presence of Vice President Hamid Ansari, Lok Sabha Speaker Sumitra Mahajan and a number of other dignitaries.
Expressing gratitude to the President, the Vice President and the Speaker, he said, "As an ordinary citizen and as an individual, this was very inspirational event for me."
He said the event had given a new dimension to 'Mann Ki Baat'.
While talking about the book, he had praise for Akbar, an artist living in Abu Dhabi, who had offered to sketch the topics on which various episodes of 'Mann Ki Baat' without taking a single rupee.
"Akbar sahab transformed 'Mann Ki Baat' into art as a gesture of his love. I am grateful," Mr Modi said.
During the 30-minute programme, he also emphasised the importance of preserving the environment, laid thrust on cleanliness and waste management, spoke about the importance of yoga and talked about freedom fighters.
Mentioning about environment and cleanliness, he said the Centre, with the help of state governments and people's representatives, will launch a major campaign for waste management from June 5, the World Environment Day.
Under the campaign, means for collecting solid and liquid waste will be launched in 4000 cities across the country.
In the context of cleanliness, the prime minister said Reasi block, a far-flung area of Jammu and Kashmir, has been declared as 'open defecation free' and hailed the people, particularly women, and administrators of the area for it.
Referring to the upcoming 3rd International Yoga Day on June 21, Modi invited selfies of three generations of a family - grandparents, parents and children - on the pattern of 'selfie with daughter' campaign earlier.
PM Modi also said the freedom fighters had suffered "despicable miseries and that is how we can now breathe in a free India." In this context, he referred to the cellular jail of Andaman and Nicobar Islands and said the younger generation should visit it to understand the trouble and torture the freedom fighters had to suffer to achieve freedom.