"It's the worst thing in the world," said Saif Ali Khan "I don't think I will ever really be okay with that," he added Saif Ali Khan and Amrita Singh divorced in 2004