Chief Justice Dipak Misra said he will constitute a new bench and put off the hearing to 28 March.
The Supreme Court does not have any explicit rules around judges recusing themselves from hearing a case where which they may have dealt with in a different capacity and have had an interest.
In this case, Justice Khanwilkar, who practised as a lawyer before being elevated as a judge back in 2000, is learnt to have appeared in a related case.
The top court had taken up a petition by a BJP leader Ajay Agrawal that challenged the 2005 judgment of the Delhi High Court quashing the Bofors case against UK-based industrialists, the Hinduja brothers, and the Swedish firm.
The CBI, which hadn't appealed against the 12-year-old judgment, had earlier this month persuaded Attorney General KK Venugopal to let the agency also join the case. Mr Venugopal had earlier advised against filing an appeal at this late stage, pointing that the court would throw out the petition on account of the long delay.
The 1986 Bofors howitzer gun deal devastated the Congress government of Rajiv Gandhi and ruined the party's prospects of returning to power for several years.
The Congress had dismissed the CBI's appeal as an attempt to gain "cheap publicity" ahead of the 2019 Lok Sabha elections.
Last October, the CBI had indicated that it was considering an appeal against the High Court decision after Michael Hershman, a private detective from US-based firm Fairfax, alleged that the Rajiv Gandhi government had sabotaged his investigation into the case.