We investigate how the quantum friction experienced by a polarizable atom moving with constant velocity parallel to a planar interface is modified when the latter consists of chiral or nonreciprocal media, with special focus on topological insulators. Macroscopic quantum electrodynamics is used to obtain the velocity-dependent Casimir–Polder frequency shift and decay rate. These results are a generalization to matter with time-reversal symmetry breaking. Our findings are illustrated by examining the nonretarded and retarded limits for five examples: a perfectly conducting mirror, a perfectly reflecting nonreciprocal mirror, a three-dimensional topological insulator, a perfectly reflecting chiral mirror and an isotropic chiral medium. Different asymptotic power laws are found for all these materials. Interestingly, two bridges between chirality and nonreciprocity through the frequency shift, which arise as a consequence of the magnetoelectric coupling. Specifically, the position-dependent Casimir–Polder frequency shift for the nonreciprocal case on a geometric magnetic field associated with photoionization of chiral molecules, while the Casimir–Polder depending on the velocity for the chiral case has the optical rotatory strength as the atomic response, and those for the nonreciprocal case depend on an analog of the optical rotatory strength.