The organisers said in a statement that while they stood by the decision to invite Mr Modi, adverse reactions from among the stakeholders involved had prompted the decision.
"Our team felt that the potential polarising reactions from sub-segments of the alumni base, student body and our supporters might put Mr Modi in a compromising position, which we would like to avoid at all costs," the statement said.
"We do not endorse any political views and do not support any specific ideology. Our goal as a team is only to stimulate valuable dialogue on India's growth story," the statement said.
Mr Modi had been approached by the Wharton School's students' body, which organises the India Economic Forum, and he had agreed to deliver a keynote address through video conference on March 23.
Later, a group of Wharton's professors and students wrote a strongly-worded letter saying they were outraged to learn that Mr Modi had been invited as a speaker.
"This is the same politician who was refused a diplomatic visa by the United States State Department on March 18, 2005 on the ground that he, as Chief Minister, did nothing to prevent a series of orchestrated riots that targeted Muslims in Gujarat," their letter said.
Wharton India Economic Forum is an annual student-run India-centric conference hosted by the Wharton School. The speakers for the meeting are usually chosen by the students' body.
A student from the group of organisers told NDTV they invited Mr Modi because they were impressed with Gujarat's growth story.
Last month, the Gujarat Chief Minister had faced protests while delivering a speech at Delhi's prestigious Shri Ram College of Commerce (SRCC). As he addressed the students, another group outside raised slogans and protested against him, saying the invitation overlooked the communal riots in Gujarat in 2002. The police had to use water cannons and canes to control the protests.