20 C.F.R. § 404.940
An administrative law judge shall not conduct a hearing if he or she is prejudiced or partial with respect to any party or has any interest in the matter pending for decision. If you object to the administrative law judge who will conduct the hearing, you must notify the administrative law judge at your earliest opportunity. The administrative law judge shall consider your objections and shall decide whether to proceed with the hearing or withdraw. If he or she withdraws, the Associate Commissioner for Hearings and Appeals, or his or her delegate, will appoint another administrative law judge to conduct the hearing. If the administrative law judge does not withdraw, you may, after the hearing, present your objections to the Appeals Council as reasons why the hearing decision should be revised or a new hearing held before another administrative law judge.
20 C.F.R. §404.940