"One family has done greatest injustice to Dr. Babasaheb Ambedkar and Sardar Patel. When Pandit Nehru's influence on the Congress was complete, the Congress ensured that Dr. Ambedkar found it tough to join the Constituent Assembly," PM Modi said today in Gujarat's Dhandhuka.