In a letter, the woman judge has said "until the committee is reconstituted and her queries answered, she won't appear before the panel on Tuesday."
The panel was set up on the orders of Chief Justice of India RM Lodha, who had asked for an inquiry into the allegations.
In a letter to the panel, the woman said the accused judge continues in his role and witnesses can't depose in a free and fair manner if he is not removed.
She also says one of the two judges in the panel had once transferred her and could be biased.
The accused judge has denied the allegations and said he is willing to face the "death penalty" if found guilty.
The woman, an additional district judge in Gwalior, resigned on July 15 after detailing her alleged harassment in letters to the President, the Chief Justice and Union Law Minister. She said in the letters that the judge had asked her to "dance to an item song" and influenced her transfer to a remote location.
"If this is how a mother, sister and wife can be treated, who is herself no less than a judicial officer duty-bound to protect society and law, what constitutional goals are we serving?" wrote the woman, saying she had no option but to resign in "compelling, humiliating and disgraceful circumstances."
In December, for the first time, the Supreme Court conducted an internal inquiry into sexual harassment allegations by an intern against a former judge. The inquiry determined that the allegations seemed correct but no action could be taken as the judge had since retired.
In January, another retired Supreme Court judge was accused by an intern of sexual harassment. The Supreme Court is now determining whether to set up a mechanism that can look at charges of sexual harassment against serving as well as retired judges.