That's one of the ideas often toyed with in Amazon. It hasn't worked out so well in Amazon because of rapid expansion of the company. As an SDE/SDM you have only so much time to devote to hiring, and with lot of pressure to hire, people like to devote time to hiring for their own team. Hence, the BR becomes the "balancer" to the process.
A BR can not be a "bar raiser" on his own team's interview loops.