What matters is which side the controller is turned towards you. When they stand with their right side to you and stretch the right arm forward, movement from the side of their right shoulder is forbidden. So in this position you may not go — you must stop and wait until the controller changes the position of their arms.