We revisit birefringence effects associated with the evolution of the polarization of light as it propagates through axion dark matter or the background of a passing gravitational wave (GW). We demonstrate that this can be described by a unified formalism, highlighting a synergy between searches for axions and high-frequency GWs. We show that by exploiting this framework, the optical cavities used by the ALPS II experiment can potentially probe axion masses in the range $m_a \sim 10^{-9} - 10^{-...