Xbox One and Xbox Series X independently with same controller?

I have both an Xbox One and Xbox Series X connected to my AV system. I'd like to control each seperately. Is there a way to so this? It seems like presently both are powering on when I try to control one of them. Thanks!