I have to believe what we're seeing is within the normal limitations of the gyro chips. It's probably one of those "ignorance is bliss" situations, since it would happen with any other FC, though they don't have the ability or opportunity to see it like we do with the AQ calibration ritual.
For calibrations, as Steve said, it's evenly distributed above and below the normal line, so it should average out.
For actual flight, I would imagine it could cause a momentary bobble, but maybe too small to really notice. The sensitivity of the chip is listed as 2mv per degree per second, and the max deviation I've seen from high noise to low is about 16mv. At worst, that would mean the FC would think there was an 8 degree per second movement in the X axis, which is slow enough that you'd probably have to be in a controlled environment with no wind to even notice it.
I'm officially not worrying about it
Cheers,
Rusty