"I toyed with the idea of giving out a statement," said KJo "It was a baseless accusation, and it really annoyed me," he added "I am, now, afraid to call up people to my home," he also said