The Supreme Court has knocked back Justice Yashwant Varma's challenge to an in-house committee that recommended his impeachment over burnt piles of money found at his Delhi home in March.