Out-band rejection is often exprerssed in dB. It is a important characteristic for a passive bandpass filter or an active LNA.
One can use the network analyzer or spectrum to measure the out-band rejection by setting the instrument to the desired frequency. You can see the power level "out of" the central frequency and such power levels in different frequencies are the corresponding out-band rejections (in dB).