There has been some discussion rolling about that very same subject, and, in theory, the inverting one was *Very* slightly better (in distortion but more importantly in noise figure) because you "kill" one of the input differential transistors, or, more precisely, ground its base (or gate, if a FET). Anyway, the consense was that the difference is practically inaudible, so, the choice becomes based more in architecture, such as: multi-input or channel mixers become easier with inverting stages (which provide a virtual-ground input) ; and high impedance stages become easier to build with inverting op-amps, as you correctly suggested.