Steady illumination of a non-centrosymmetric semiconductor results in a bulk photovoltaic current, which is contributed by real-space displacements (`shifts’) of charged quasiparticles as they transit between Bloch states. The shift induced by interband excitation via absorption of photons has received the prevailing attention. However, this excitation-induced shift can be far outweighed (<<) by the shift induced by intraband relaxation, or by the shift induced by radiative recombination of electron-hole pairs. This outweighing (<<) is attributed to (i) time-reversal-symmetric, intraband Berry curvature, which results in an anomalous shift of quasiparticles as they scatter with phonons, as well as to (ii) topological singularities in the interband Berry phase (`optical vortices'), which makes the photovoltaic current extraordinarily sensitive to the linear polarization vector of the light source. Both (i-ii) potentially lead to nonlinear conductivities of order mA/V^2, without finetuning of the incident radiation frequency, band gap, or joint density of states. A case study of BiTeI showcases the anomalous shift and optical vorticity in a realistic material.