The shocking footage shows Mr Advani being told to "pipe down", before going on to make the comment "Brexit, go home" and after that he walked out the surgery and reported the abuse to police, it said.
"I was absolutely disgusted by what I had heard and couldn't believe that in 2018 people can be so bigoted. At the very first opportunity I told him to politely stop and hoped that would be the end of it," Mr Advani said.
"Any reasonable human in that situation would have accepted his wrong but he became unnecessarily aggressive to me," Mr Advani, who also runs a charity for homeless people, said.
"Investigations continue. No arrests have been made," a police spokesman was quoted as saying by the report. The Cornford House Surgery has also apologized.
Simon Gridley, a surgery manager at Cornford House Surgery, wrote in a letter to him: "I would like to apologise for our handling of the racial abuse you received on that day. My colleague mentioned that you felt you were unfairly treated as you (were made to feel) that you were the guilty party, when you had, quite rightly, stood up for another member of the public. "I know an apology is NOT the same as getting it right the first time, but again, I sincerely apologise."