ppcpy.qc#
ppcpy.qc.pollySaturationDetect#
- ppcpy.qc.pollySaturationDetect.pollySaturationDetect(data_cube, rfill: float = 250, sigSaturateThresh: float = 500) ndarray[source]#
Detect the bins which are fully saturated by the clouds.
- Parameters:
- data_cubeobject
Main PicassoProc object.
- rfillfloat, optional
Minimum range gap to fill [m]. Default is 250.
- sigSaturateThreshfloat, optional
Threshold of saturated signal [photon count]. Default is 500.
- Returns:
- flagSaturationboolean ndarray
True indicates current range bin is saturated by clouds.
- ** History **
- 2018-12-21: First Edition by Zhenping
- 2019-07-08: Fix the bug of converting signal to PCR.
- 2025-05-14: translated and changed the algorithm to use scipy.ndimage
ppcpy.qc.transCor#
- ppcpy.qc.transCor.transCorGHK_cube(data_cube, signal: str = 'BGCor', collect_debug: bool = False) tuple[source]#
Perform GHK transmission correction.
- Parameters:
- data_cubeobject
Main PicassoProc object.
- signalstr
Signal type. Default is ‘BGCor’.
- collect_debugbool
If True, collects debug information. Default is False.
- Returns:
- sigTCorndarray
Transmission corrected signal [Photon counts].
- BGTCorndarray
Transmission corrected background.
- ppcpy.qc.transCor.transCor_E16_channel(sigT: ndarray, bgT: ndarray, voldepol: ndarray, HT: float) tuple[source]#
Transmission correction for the total channel using the Mattis 2009/Engelmann 2016 method
- Parameters:
- sigTndarray
Signal in total channel (background-corrected)
- bgTndarray
Background in total channel
- voldepolndarray
Volume depolarization ratio
- HTfloat
Transmission ratio of total channel in GHK notation
- Returns:
- sigTCorndarray
Signal in total channel corrected for polarization induced transmission effects
- bgTCorndarray
Background of total signal
Notes
Following [1] in the notation of [2]
\[P_{i, \text{corr}} = P_i \frac{1 + R_i \delta^V}{1+\delta^V} \]with the signal \(P_i\), the transmission ratio \(R_i\) and the volume depolarization ratio \(\delta^V\)
Todo
Clarify the background treatment. The bgTCor should not change (i.e. assuming the vdr is 0?)
References
- ppcpy.qc.transCor.transCorGHK_channel(sigT: ndarray, bgT: ndarray, sigC: ndarray, bgC: ndarray, transGT: float = 1, transGR: float = 1, transHT: float = 0, transHR: float = -1, polCaliEta: float = 1, polCaliEtaStd: float = 0) tuple[source]#
Corrects the effect of different polarization-dependent transmission inside the total and depol channel.
Follows the matlab code at: PollyNET/Pollynet_Processing_Chain
- Parameters:
- sigTndarray
Signal in total channel.
- bgTndarray
Background in total channel.
- sigCndarray
Signal in cross channel.
- bgCndarray
Background in cross channel.
- transGTfloat
G parameter in total channel.
- transGRfloat
G parameter in cross channel.
- transHTfloat
H parameter in total channel.
- transHRfloat
H parameter in cross channel.
- polCaliEtafloat
Depolarization calibration constant (eta).
- polCaliEtaStdfloat
Uncertainty of the depolarization calibration constant.
- Returns:
- sigTCorndarray
Transmission corrected elastic signal.
- bgTCorndarray
Background of transmission corrected elastic signal.
Notes
Todo
Input parameter ´polCaliEtaStd´ is not used.
References
Mattis, I., Tesche, M., Grein, M., Freudenthaler, V., and Müller, D.: Systematic error of lidar profiles caused by a polarization-dependent receiver transmission: Quantification and error correction scheme, Appl. Opt., 48, 2742-2751, 2009.
Freudenthaler, V. About the effects of polarising optics on lidar signals and the Delta90 calibration. Atmos. Meas. Tech., 9, 4181–4255 (2016).
** History **
2021-05-27: First edition by Zhenping.
2024-08-14: Change to GHK parameterization by Moritz.
2024-12-28: AI translation
ppcpy.qc.overlapEst#
- ppcpy.qc.overlapEst.run_frnr_cldFreeGrps(data_cube, collect_debug: bool = True) list[source]#
Estimate overlap function using far-range-near-range method.
- Parameters:
- data_cubeobject
Main PicassoProc object.
- collect_debugbool
If true, collect debug information. Default is True.
- Returns:
- overlapndarray
Overlap function.
- overlapStdndarray
Standard deviation of overlap function.
- sigRatiofloat
Signal ratio between near-range and far-range signals.
- normRangelist
Height index of the signal normalization range.
Notes
The matlab version preforms some convertio from PC to PCR after calculating the overlap function (this might just be relevent at all)
The matlab version also uses calculates 1 overlap function for each channel while we calculate 1 per clFreeGrp per channel…
The frnr overlap calculations are very unstable and frequently results in values outside the range [0,1].
- ppcpy.qc.overlapEst.overlapCalc(height: ndarray, sigFR: ndarray, bgFR: ndarray, sigNR: ndarray, bgNR: ndarray, hFullOverlap: float = 600, collect_debug=False) dict[source]#
Calculate overlap function.
- Parameters:
- heightndarray
Height above ground (m).
- sigFRndarray
Far-field signal (photon count).
- bgFRndarray
Background of far-field signal (photon count).
- sigNRndarray
Near-field signal (photon count).
- bgNRndarray
Background of near-field signal (photon count).
- Returns:
- olFuncndarray
Overlap function.
- olFuncStdndarray
Standard deviation of overlap function.
- sigRatiofloat
Signal ratio between near-range and far-range signals.
- normRangelist
Height index of the signal normalization range.
- Other Parameters:
- hFullOverlapfloat, optional
Minimum height with full overlap for far-range signal (default: 600 m).
Notes
Todo
Define the Default values of the output ei. olFunc = 1 & olFuncStd = 0.
Todo
Is the first if-statment needed?
Todo
are bgFR and bgNR actually ndarrays of floats??
History
2021-05-18: First edition by Zhenping
- ppcpy.qc.overlapEst.run_raman_cldFreeGrps(data_cube, collect_debug: bool = True) list[source]#
Estimate overlap function using Raman method.
- Parameters:
- data_cubeobject
Main PicassoProc object.
- collect_debugbool
If true, collect debug information. Default is True.
- Returns:
- olFuncndarray
Overlap function.
- olFunc_rawndarray
Overlap function with no smoothing.
- LRfloat
Optimal Lidar Ratio for …
- normRangelist
Height index of the signal normalization range.
- ppcpy.qc.overlapEst.overlapCalcRaman(Lambda_el: float, Lambda_Ra: float, height: ndarray, sigFRel: ndarray, sigFRRa: ndarray, molExt: ndarray, molBsc_r: ndarray, molExt_r: ndarray, aerBsc: ndarray, hFullOverlap: float = 600, smoothbins: float = 1, AE: float = 1, hres: float = 150, collect_debug=False) dict[source]#
Calculate overlap function from Polly measurements based on Wandinger and Ansmann 2002 https://doi.org/10.1364/AO.41.000511
- Parameters:
- Lambda_elfloat
Elastic wavelength.
- Lambda_Rafloat
Raman wavelength.
- heightndarray
Height above ground [m].
- sigFRelndarray
Far-field elastic signal.
- sigFRRandarray
Far-field Raman signal.
- molExtndarray
Molecular extinction (elastic).
- molBsc_rndarray
Molecular Backscatter (inelastic).
- molExt_rndarray
Molecular extinction (inelastic).
- aerBscndarray
Particle backscattering derived with the Raman method [m^{-1}].
- hFullOverlapfloat, optional
Minimum height with complete overlap [m]. Default is 600.
- smoothbinsint, optional
Number of bins for smoothing. Default is 1.
- AEfloat, optional
Angstroem exponent. Default is 1.
- hresfloat, optional
Instrument height resolution [m]. Default is 150.
- Returns:
- olFuncndarray
Overlap function.
- olFunc_rawndarray
Overlap function with no smoothing.
- LRfloat
Optimal Lidar Ratio for …
- normRangelist
Height index of the signal normalization range.
Notes
Todo
Finish docsting! (LR output…)
Todo
This function uses a mix of ppcpy.misc.helper.unifrom_filter and scipy.ndimage.uniform_filter1d for smoothing. This is done to avoid NaN-value related errors. A better more cohesive solution should be precude in the future.
Todo
Be cearfull with NaN values! scipy.ndimage.uniform_filter1d used for smoothing in this module will propagate any NaN values present in the signal throughot the rest of the smoothed signal. Additionaly, here we are using mode ‘reflect’ for padding (see scipy.ndimage.uniform_filter1d documentation). This might not be the most optimal mode, the other availabel mode should also be considered. Optimally, should we designe our own filter for this purpuse that do not have the issue with propagating NaN values, Like what is used in the rest of the modules. However, without reducing dimension or filling in NaN values.
Todo
Is the if statment that privusly checked if len(aerBsc) > 0, now len(aerBsc) > 6 to be consistant with the action of the statment actually needed. I believe it might be a leftover from the Matlab version where they often used staments like this to check if the aerBsc existed or not…
** History **
2023-06-06: First edition by Cristofer
xxxx-xx-xx: Translated to python
References
Wandinger, U. and Ansmann, A. (2002) ‘Experimental determination of the lidar overlap profile with Raman lidar’, Applied Optics, 41(3), pp. 511–514. Available at: https://doi.org/10.1364/ao.41.000511.
ppcpy.qc.overlapCor#
- ppcpy.qc.overlapCor.spread(data_cube)[source]#
Select the correct overlap method, spread the profiles to 2d for each wavelength.
design decision for now: drop the signal glue option (overlapCorMode == 3)
- Parameters:
- data_cubeobject
Main PicassoProc object.
- Returns:
- dict
2d overlap function per channel.
Notes
in the matlab version any olFunc is additionally smoothed with olSm = smooth(olFuncDeft, p.Results.overlapSmWin, ‘sgolay’, 2); (e.g. here PollyNET/Pollynet_Processing_Chain) This should probably be done more explicitly.
Also the ‘glueing’ in function [sigCor] = olCor(sigFR, overlap, height, normRange) seems wired (PollyNET/Pollynet_Processing_Chain).
Todo
Using different overlap function at different periods of the signal introduces harsh transitions as long as the overlap functions are not stable. Look into options for smoothing the overlap function in the time dimension to ease the transition from one period to another.
** History **
xxxx-xx-xx: First edition by …
xxxx-xx-xx: Translated to python.
- ppcpy.qc.overlapCor.apply_cube(data_cube)[source]#
Apply overlap function to 2d (time, height) signal.
- Parameters:
- data_cubeobject
Main PicassoProc object.
- Returns:
- sigOLCorndarray
Overlap corrected signal (time, height).
- BGOLCorndarray
Overlap corrected background (time, ).
- heightFullOverlapCorndarray
Height of full overlap [m?].
Notes
- Shoud we apply the overlap correction on the GHK-Transmission corrected signal or the Background Corrrected signal??
Currentlly we are using a wired mix here. sigOLCor is inintialized with the sigTCor but the correction is applied based on the sigBGCor. The correction to the BG is alwaysed based on the BGTCor.
Also, in highres the GHK-Transmission correction is again applied on sigOLCor before the OC_attBsc is calculated.
Todo
At times the retrieved overlap function is outside the range [0, 1], the values of the function should be checked and corrected before applying. Currnetly only values less then 0.07 are corrected (set to NaN).
Todo
This function could be easaly vectorized by cunstructing all the 2d overlap function into a 3d Matrix / Tensor Channels that do not have overlap function can be replaced by ones. This alongside the use of alternative overlap functions (using 355 insted of 386 etc.) can be handled in the construction of the 3d Matrix / Tensor. Then the correction itself can be applied like this: sigOLCor = sigTCor / olFunc_3d
** History **
xxxx-xx-xx: First edition by …
xxxx-xx-xx: Translated to python.
- ppcpy.qc.overlapCor.fixLowest(overlap: ndarray, indexsearchmax: int, thres: float = 0.05)[source]#
Very rough fix for exploding values in the very near range of the overlap function.
in the lowest heights (below indexsearchmax, e.g. 800m) search for chunks, where the overlap function is smaller than a treshold (thres e.g. 0.05) in that chunk take the miniumum and fill heights below
- Parameters:
- overlaplist
List of dicts including ovelap functions per cloud free period.
- indexsearchmax :int
Maximum height index for the applying the fix.
- thresfloat
Threshold for searching for a stable low overlap value. Default is 0.05.
ppcpy.qc.qualityMask#
- ppcpy.qc.qualityMask.qualityMask(data_cube) ndarray[source]#
Estimate quality mask.
- Parameters:
- data_cubeobject
Main picassoProc object.
- useQuasiSNRbool
If True, use lowSNRMask_quasi. Otherwise use lowSNRMask.
- Returns:
- quality_maskndarray
Quality mask with categories listed above.
- ** History **
- xxxx-xx-xx: First edition by …
- xxxx-xx-xx: translated to python
- 2026-07-03: vectorized and added option to use lowSNRMask_quasi
- 2026-07-03: added resistance toward missing mask-variables
- ppcpy.qc.qualityMask.improvedSNR(data_cube) tuple[source]#
Artificially improve SNR by smoothing the signal and background.
- Parameters:
- data_cubeobject
Main picassoProc object.
- Returns:
- SNRndarray
Improved signal to noise ratio
- lowSNRMaskndarray
True if SNR is lower than the config variable ‘mask_SNRmin’. Otherwise False.
Notes
Could also be in the collection retrieval together with ´calc_snr´.
** History **
xxxx-xx-xx: First edition by …
2026-07-03: translated to python
2026-07-12: vectorized for better efficiency