The gravitational wave (GW) spectrum at frequencies above a kHz is a largely unexplored frontier. We show that detectors with sensitivity to single-phonon excitations in crystal targets can search for GWs with frequencies, $\mathrm{THz} \lesssim f \lesssim 100 \, \mathrm{THz}$, corresponding to the range of optical phonon energies, $\mathrm{meV} \lesssim ω\lesssim 100 \, \mathrm{meV}$. Such detectors are already being built to search for light dark matter (DM), and therefore sensitivity to high-...