"For everything there is a Lakshmana Rekha,"Mr Goel, who is leading a 200-member Indian youth delegation to China, told the media.
"But when the line is crossed then a message has to be given. This (boycott) is a message," he said defending the policy not to have bilateral cricket series with Pakistan.
"We have clearly stated terrorism and sports cannot go side-by-side. As long as Pakistan doesn't stop cross border terror and (instigating) trouble in Kashmir..then there is no chance for bilateral series and also the national mood doesn't support it," Mr Goel said.
He declined to comment on the letter written by Ramachandra Guha, one of the four members of Supreme Court appointed Committee of Administrators (CoA), to implement the recommendations of the Justice R M Lodha panel to reform the Board of Control for Cricket in India (BCCI).
Mr Goel said cricket is managed by the BCCI as well as the apex court and that he would not like comment on the cricket related issues.
He, however, referred to his ministry's proposal to have a sports code in India which should include cricket too.
He outlined numerous programmes initiated by his ministry to improve sports on all fronts by expanding the sports infrastructure.
It included a new App through which children above the age of eight years can highlight their talents in sports for his ministry to initiate follow up action.
The Indian delegation is in China as part of an annual exchange programme.
Both countries have been conducting youth exchanges since 2006. Last year a Chinese delegation visited India.
The last bilateral series between India and Pakistan was played in 2012-13.