Myanmar's recent return to elected rule has given it a hybrid government in which the military still has wide powers in the security domain and in Rakhine state, where the United Nations has reported ethnic cleansing.
"We're extraordinarily concerned by what is happening with Rohingya in Burma," Tillerson said.
"I've been in contact with Aung San Suu Kyi the leader of the civilian part of the government, as you know this is a power sharing government.
"We really hold the military leadership accountable for what's happening," he said, warning the world won't stand and "be witness to the atrocities that have been reported."
In the last seven weeks, more than half a million Rohingya have fled Rakhine and crossed into Bangladesh.
Their stories have shocked the world globe, with accounts of Myanmar soldiers and Buddhist mobs murdering and raping civilians before torching their villages to the ground.
The western region descended into chaos when Rohingya militants attacked Myanmar police posts on August 25, triggering the brutal military crackdown.
Tillerson said that Washington understands that Myanmar is facing "serious rebel terrorist elements" in Rakhine, but warned the military must be disciplined and "restrained."
And he said Myanmar must grant fuller access to aid agencies to aid civilians and to allow the world "a fuller picture of what is going on."
"Someone is going to be held to account for that and it's up to the military leadership of Burma to decide what role do they want to play in the future of Burma," he said.
"This is a real test for this power sharing government."