We investigate the impact of confinement density (i.e the number of individual in a group per unit area of available space) on transitions from polarized to milling state, using groups of rummy-nose tetrafish (Hemigrammus rhodostomus) under controlled experimental conditions. We demonstrate for the first time a continuous state transition controlled by confinement density in a group of live animals. During this transition, the school exhibits a bistable state, wherein both polarization and milling states coexist, with the group randomly alternating between them. A simple two state Markov process describe the observed transition remarkably well. Importantly, the confinement density influences the statistics of this bistability, shaping the distribution of transition times between states. Our findings suggest that confinement plays a crucial role in state transitions for moving animal groups, and, more generally, they constitute a solid experimental benchmark for active matter models of macroscopic, self-propelled, confined agents.