The organisers said in a statement that while they stood by the decision to invite Mr Modi, adverse reactions from among the stakeholders involved had prompted the decision.
"Our team felt that the potential polarising reactions from sub-segments of the alumni base, student body and our supporters might put Mr Modi in a compromising position, which we would like to avoid at all costs," the statement said.
"We do not endorse any political views and do not support any specific ideology. Our goal as a team is only to stimulate valuable dialogue on India's growth story," the statement said.
Mr Modi had been approached by the Wharton School's students' body, which organises the India Economic Forum, and he had agreed to deliver a keynote address through video conference on March 23.
Later, a group of Wharton's professors and students wrote a strongly-worded letter saying they were outraged to learn that Mr Modi had been invited as a speaker.
Wharton India Economic Forum is an annual student-run India-centric conference hosted by the Wharton School. The speakers for the meeting are usually chosen by the students' body.
A student from the group of organisers told NDTV they invited Mr Modi because they were impressed with Gujarat's growth story.
Last month, the Gujarat Chief Minister had faced protests while delivering a speech at Delhi's prestigious Shri Ram College of Commerce (SRCC). As he addressed the students, another group outside raised slogans and protested against him, saying the invitation overlooked the communal riots in Gujarat in 2002. The police had to use water cannons and canes to control the protests.
Get Breaking news, live coverage, and Latest News from India and around the world on NDTV.com. Catch all the Live TV action on NDTV 24x7 and NDTV India. Like us on Facebook or follow us on Twitter and Instagram for latest news and live news updates.