Bandpass spectral correction

I have a need to have a very flat passband for some of the
analysis I’m doing of neutral hydrogen. Real hardware, of course,
produces passbands that aren’t quite flat–some worse than others.

I’ve tried an experiment in one of my flow-graphs where I take the
(aggressively-averaged) FFT magnitudes by taking each bin magnitude and
dividing by the maximum, and then using the inverse of that to produce a
correction curve, as normalized magnitudes.

I then use those
normalized magnitudes in an FFT filter (after converting into
time-domain), and the result works less-well than I would expect. The
result is a much-flatter spectrum, but at the fine-scales I’m doing
analysis at (0.5dB), it’s not nearly as flat – the outer edges are
under-corrected, and the inner bits are over-corrected.

Anybody have
any pointers to good papers on passband flattening techniques?