One of the victims had her throat slit by the assailant, a man with a criminal record believed to be in his 30s who witnesses said shouted "Allahu Akbar" (God is Greatest) at the start of his rampage.
Troops serving in a special 7,000-strong force known as Sentinelle set up to guard vulnerable areas in terror-hit France responded to the stabbings and shot dead the attacker, whose identity remains unknown.
French Interior Minister Gerard Collomb said about a dozen witnesses were being questioned.
"This act could be terrorist in nature but at this time we cannot confirm that," he added.
"I was on the esplanade just in front of the station," Melanie Petit, an 18-year-old student, told AFP. "I heard someone shout 'Allahu Akbar' and I saw a man who seemed to be dressed all in black."
Later, ISIS's Amaq propaganda agency cited a "security source" saying: "The executor of the stabbing operation in the city of Marseille... is from the soldiers of the Islamic State."
Heavily armed police sealed off and evacuated the ornate rail terminus in the bustling heart of France's second-biggest city, stopping all train traffic on some of the country's busiest lines.
The latest deaths came with France still on high alert and under a state of emergency following a string of attacks in recent years by extremists linked to the ISIS group or Al-Qaeda.
Since 2015, a total of 239 people have been killed in France by jihadists, according to an AFP count before Sunday's incident.
After the lunchtime attack, anti-terror prosecutors said they had opened an investigation into "killings linked to a terrorist organisation" and the "attempted killing of a public official".
Travellers around the station described "controlled panic" as security forces evacuated passengers and looked for possible accomplices, while another witness said white sheets were placed over the bodies of the victims.
The two victims remained unidentified but were both "around 20 years old," a police source said.
Their assailant was known to the authorities for a string of common law offences and had "no papers on him," the source added.
Collomb said that video surveillance footage showed the assailant attacked the first victim, then fled -- only to return and kill the other woman.
Minute's silence at football match
The incident came only days after the ISIS released a recording of what it said was its leader Abu Bakr al-Baghdadi urging his followers to strike their enemies in the West.
France has deployed troops and its air force to the Middle East and is a leading partner in the US-led international coalition fighting ISIS in Iraq and Syria, where the jihadists are being driven back.
The country has suffered several major terror attacks since 2015, including on satirical magazine Charlie Hebdo in January of that year followed by an assault on Paris bars and the Bataclan concert hall by gunmen 10 months later.
In Nice in July 2016, 86 people were killed when an extremist drove a truck into crowds after a fireworks display on Bastille Day.
A minute's silence was observed in Nice on Sunday evening, ahead of a football match between Nice and Marseille.
Since November 2015, the country has been in a state of emergency which gives the government and security forces greater powers to combat extremists and launch anti-terror raids.
Controversial security law
New centrist President Emmanuel Macron has vowed to end the state of emergency with a new and controversial security law that will make many of the provisions of the emergency regime permanent.
He tweeted that he was "deeply angered by this barbaric act," and shared the pain of the families of friends of the victims.
Despite criticism from rights groups that the law reduces judicial oversight over the actions of the police, the lower house of parliament is set to vote on a first draft of the legislation on Tuesday.
French Prime Minister Edouard Philippe vowed: "We will not lower our guard."