ppcpy.qc#

ppcpy.qc.pollySaturationDetect#

ppcpy.qc.pollySaturationDetect.pollySaturationDetect(data_cube, rfill: float = 250, sigSaturateThresh: float = 500) → ndarray[source]#

Detect the bins which are fully saturated by the clouds.

Parameters:
data_cubeobject

Main PicassoProc object.

rfillfloat, optional

Minimum range gap to fill [m]. Default is 250.

sigSaturateThreshfloat, optional

Threshold of saturated signal [photon count]. Default is 500.

Returns:
flagSaturationboolean ndarray

True indicates current range bin is saturated by clouds.

** History **
  • 2018-12-21: First Edition by Zhenping
  • 2019-07-08: Fix the bug of converting signal to PCR.
  • 2025-05-14: translated and changed the algorithm to use scipy.ndimage

ppcpy.qc.transCor#

ppcpy.qc.transCor.transCorGHK_cube(data_cube, signal: str = 'BGCor', collect_debug: bool = False) → tuple[source]#

Perform GHK transmission correction.

Parameters:
data_cubeobject

Main PicassoProc object.

signalstr

Signal type. Default is ‘BGCor’.

collect_debugbool

If True, collects debug information. Default is False.

Returns:
sigTCorndarray

Transmission corrected signal [Photon counts].

BGTCorndarray

Transmission corrected background.

ppcpy.qc.transCor.transCor_E16_channel(sigT: ndarray, bgT: ndarray, voldepol: ndarray, HT: float) → tuple[source]#

Transmission correction for the total channel using the Mattis 2009/Engelmann 2016 method

Parameters:
sigTndarray

Signal in total channel (background-corrected)

bgTndarray

Background in total channel

voldepolndarray

Volume depolarization ratio

HTfloat

Transmission ratio of total channel in GHK notation

Returns:
sigTCorndarray

Signal in total channel corrected for polarization induced transmission effects

bgTCorndarray

Background of total signal

Notes

Following [1] in the notation of [2]

\[P_{i, \text{corr}} = P_i \frac{1 + R_i \delta^V}{1+\delta^V} \]

with the signal \(P_i\), the transmission ratio \(R_i\) and the volume depolarization ratio \(\delta^V\)

Todo

Clarify the background treatment. The bgTCor should not change (i.e. assuming the vdr is 0?)

References

[1]

Mattis et al 2009

[2]

Engelmann et al 2016

ppcpy.qc.transCor.transCorGHK_channel(sigT: ndarray, bgT: ndarray, sigC: ndarray, bgC: ndarray, transGT: float = 1, transGR: float = 1, transHT: float = 0, transHR: float = -1, polCaliEta: float = 1, polCaliEtaStd: float = 0) → tuple[source]#

Corrects the effect of different polarization-dependent transmission inside the total and depol channel.

Follows the matlab code at: PollyNET/Pollynet_Processing_Chain

Parameters:
sigTndarray

Signal in total channel.

bgTndarray

Background in total channel.

sigCndarray

Signal in cross channel.

bgCndarray

Background in cross channel.

transGTfloat

G parameter in total channel.

transGRfloat

G parameter in cross channel.

transHTfloat

H parameter in total channel.

transHRfloat

H parameter in cross channel.

polCaliEtafloat

Depolarization calibration constant (eta).

polCaliEtaStdfloat

Uncertainty of the depolarization calibration constant.

Returns:
sigTCorndarray

Transmission corrected elastic signal.

bgTCorndarray

Background of transmission corrected elastic signal.

Notes

Todo

Input parameter ´polCaliEtaStd´ is not used.

References

  • Mattis, I., Tesche, M., Grein, M., Freudenthaler, V., and Müller, D.: Systematic error of lidar profiles caused by a polarization-dependent receiver transmission: Quantification and error correction scheme, Appl. Opt., 48, 2742-2751, 2009.

  • Freudenthaler, V. About the effects of polarising optics on lidar signals and the Delta90 calibration. Atmos. Meas. Tech., 9, 4181–4255 (2016).

** History **

  • 2021-05-27: First edition by Zhenping.

  • 2024-08-14: Change to GHK parameterization by Moritz.

  • 2024-12-28: AI translation

ppcpy.qc.overlapEst#

ppcpy.qc.overlapEst.run_frnr_cldFreeGrps(data_cube, collect_debug: bool = True) → list[source]#

Estimate overlap function using far-range-near-range method.

Parameters:
data_cubeobject

Main PicassoProc object.

collect_debugbool

If true, collect debug information. Default is True.

Returns:
overlapndarray

Overlap function.

overlapStdndarray

Standard deviation of overlap function.

sigRatiofloat

Signal ratio between near-range and far-range signals.

normRangelist

Height index of the signal normalization range.

Notes

  • The matlab version preforms some convertio from PC to PCR after calculating the overlap function (this might just be relevent at all)

  • The matlab version also uses calculates 1 overlap function for each channel while we calculate 1 per clFreeGrp per channel…

  • The frnr overlap calculations are very unstable and frequently results in values outside the range [0,1].

ppcpy.qc.overlapEst.overlapCalc(height: ndarray, sigFR: ndarray, bgFR: ndarray, sigNR: ndarray, bgNR: ndarray, hFullOverlap: float = 600, collect_debug=False) → dict[source]#

Calculate overlap function.

Parameters:
heightndarray

Height above ground (m).

sigFRndarray

Far-field signal (photon count).

bgFRndarray

Background of far-field signal (photon count).

sigNRndarray

Near-field signal (photon count).

bgNRndarray

Background of near-field signal (photon count).

Returns:
olFuncndarray

Overlap function.

olFuncStdndarray

Standard deviation of overlap function.

sigRatiofloat

Signal ratio between near-range and far-range signals.

normRangelist

Height index of the signal normalization range.

Other Parameters:
hFullOverlapfloat, optional

Minimum height with full overlap for far-range signal (default: 600 m).

Notes

Todo

Define the Default values of the output ei. olFunc = 1 & olFuncStd = 0.

Todo

Is the first if-statment needed?

Todo

are bgFR and bgNR actually ndarrays of floats??

History

  • 2021-05-18: First edition by Zhenping

ppcpy.qc.overlapEst.run_raman_cldFreeGrps(data_cube, collect_debug: bool = True) → list[source]#

Estimate overlap function using Raman method.

Parameters:
data_cubeobject

Main PicassoProc object.

collect_debugbool

If true, collect debug information. Default is True.

Returns:
olFuncndarray

Overlap function.

olFunc_rawndarray

Overlap function with no smoothing.

LRfloat

Optimal Lidar Ratio for …

normRangelist

Height index of the signal normalization range.

ppcpy.qc.overlapEst.overlapCalcRaman(Lambda_el: float, Lambda_Ra: float, height: ndarray, sigFRel: ndarray, sigFRRa: ndarray, molExt: ndarray, molBsc_r: ndarray, molExt_r: ndarray, aerBsc: ndarray, hFullOverlap: float = 600, smoothbins: float = 1, AE: float = 1, hres: float = 150, collect_debug=False) → dict[source]#

Calculate overlap function from Polly measurements based on Wandinger and Ansmann 2002 https://doi.org/10.1364/AO.41.000511

Parameters:
Lambda_elfloat

Elastic wavelength.

Lambda_Rafloat

Raman wavelength.

heightndarray

Height above ground [m].

sigFRelndarray

Far-field elastic signal.

sigFRRandarray

Far-field Raman signal.

molExtndarray

Molecular extinction (elastic).

molBsc_rndarray

Molecular Backscatter (inelastic).

molExt_rndarray

Molecular extinction (inelastic).

aerBscndarray

Particle backscattering derived with the Raman method [m^{-1}].

hFullOverlapfloat, optional

Minimum height with complete overlap [m]. Default is 600.

smoothbinsint, optional

Number of bins for smoothing. Default is 1.

AEfloat, optional

Angstroem exponent. Default is 1.

hresfloat, optional

Instrument height resolution [m]. Default is 150.

Returns:
olFuncndarray

Overlap function.

olFunc_rawndarray

Overlap function with no smoothing.

LRfloat

Optimal Lidar Ratio for …

normRangelist

Height index of the signal normalization range.

Notes

Todo

Finish docsting! (LR output…)

Todo

This function uses a mix of ppcpy.misc.helper.unifrom_filter and scipy.ndimage.uniform_filter1d for smoothing. This is done to avoid NaN-value related errors. A better more cohesive solution should be precude in the future.

Todo

Be cearfull with NaN values! scipy.ndimage.uniform_filter1d used for smoothing in this module will propagate any NaN values present in the signal throughot the rest of the smoothed signal. Additionaly, here we are using mode ‘reflect’ for padding (see scipy.ndimage.uniform_filter1d documentation). This might not be the most optimal mode, the other availabel mode should also be considered. Optimally, should we designe our own filter for this purpuse that do not have the issue with propagating NaN values, Like what is used in the rest of the modules. However, without reducing dimension or filling in NaN values.

Todo

Is the if statment that privusly checked if len(aerBsc) > 0, now len(aerBsc) > 6 to be consistant with the action of the statment actually needed. I believe it might be a leftover from the Matlab version where they often used staments like this to check if the aerBsc existed or not…

** History **

  • 2023-06-06: First edition by Cristofer

  • xxxx-xx-xx: Translated to python

References

Wandinger, U. and Ansmann, A. (2002) ‘Experimental determination of the lidar overlap profile with Raman lidar’, Applied Optics, 41(3), pp. 511–514. Available at: https://doi.org/10.1364/ao.41.000511.

ppcpy.qc.overlapEst.load(data_cube) → list[source]#

Read the overlap function from files into a structure similar to the others.

Parameters:
data_cubeobject

Main PicassoProc object.

Returns:
overlaplist of dicts

…

ppcpy.qc.overlapCor#

ppcpy.qc.overlapCor.spread(data_cube)[source]#

Select the correct overlap method, spread the profiles to 2d for each wavelength.

design decision for now: drop the signal glue option (overlapCorMode == 3)

Parameters:
data_cubeobject

Main PicassoProc object.

Returns:
dict

2d overlap function per channel.

Notes

  • in the matlab version any olFunc is additionally smoothed with olSm = smooth(olFuncDeft, p.Results.overlapSmWin, ‘sgolay’, 2); (e.g. here PollyNET/Pollynet_Processing_Chain) This should probably be done more explicitly.

  • Also the ‘glueing’ in function [sigCor] = olCor(sigFR, overlap, height, normRange) seems wired (PollyNET/Pollynet_Processing_Chain).

Todo

Using different overlap function at different periods of the signal introduces harsh transitions as long as the overlap functions are not stable. Look into options for smoothing the overlap function in the time dimension to ease the transition from one period to another.

** History **

  • xxxx-xx-xx: First edition by …

  • xxxx-xx-xx: Translated to python.

ppcpy.qc.overlapCor.apply_cube(data_cube)[source]#

Apply overlap function to 2d (time, height) signal.

Parameters:
data_cubeobject

Main PicassoProc object.

Returns:
sigOLCorndarray

Overlap corrected signal (time, height).

BGOLCorndarray

Overlap corrected background (time, ).

heightFullOverlapCorndarray

Height of full overlap [m?].

Notes

  • Shoud we apply the overlap correction on the GHK-Transmission corrected signal or the Background Corrrected signal??

    Currentlly we are using a wired mix here. sigOLCor is inintialized with the sigTCor but the correction is applied based on the sigBGCor. The correction to the BG is alwaysed based on the BGTCor.

  • Also, in highres the GHK-Transmission correction is again applied on sigOLCor before the OC_attBsc is calculated.

Todo

At times the retrieved overlap function is outside the range [0, 1], the values of the function should be checked and corrected before applying. Currnetly only values less then 0.07 are corrected (set to NaN).

Todo

This function could be easaly vectorized by cunstructing all the 2d overlap function into a 3d Matrix / Tensor Channels that do not have overlap function can be replaced by ones. This alongside the use of alternative overlap functions (using 355 insted of 386 etc.) can be handled in the construction of the 3d Matrix / Tensor. Then the correction itself can be applied like this: sigOLCor = sigTCor / olFunc_3d

** History **

  • xxxx-xx-xx: First edition by …

  • xxxx-xx-xx: Translated to python.

ppcpy.qc.overlapCor.fixLowest(overlap: ndarray, indexsearchmax: int, thres: float = 0.05)[source]#

Very rough fix for exploding values in the very near range of the overlap function.

in the lowest heights (below indexsearchmax, e.g. 800m) search for chunks, where the overlap function is smaller than a treshold (thres e.g. 0.05) in that chunk take the miniumum and fill heights below

Parameters:
overlaplist

List of dicts including ovelap functions per cloud free period.

indexsearchmax :int

Maximum height index for the applying the fix.

thresfloat

Threshold for searching for a stable low overlap value. Default is 0.05.

ppcpy.qc.overlapCor.hFullOLbyGrp(clFreeGrps, heightFullOverCor)[source]#

ppcpy.qc.qualityMask#

ppcpy.qc.qualityMask.qualityMask(data_cube) → ndarray[source]#

Estimate quality mask.

Parameters:
data_cubeobject

Main picassoProc object.

useQuasiSNRbool

If True, use lowSNRMask_quasi. Otherwise use lowSNRMask.

Returns:
quality_maskndarray

Quality mask with categories listed above.

** History **
  • xxxx-xx-xx: First edition by …
  • xxxx-xx-xx: translated to python
  • 2026-07-03: vectorized and added option to use lowSNRMask_quasi
  • 2026-07-03: added resistance toward missing mask-variables
ppcpy.qc.qualityMask.improvedSNR(data_cube) → tuple[source]#

Artificially improve SNR by smoothing the signal and background.

Parameters:
data_cubeobject

Main picassoProc object.

Returns:
SNRndarray

Improved signal to noise ratio

lowSNRMaskndarray

True if SNR is lower than the config variable ‘mask_SNRmin’. Otherwise False.

Notes

  • Could also be in the collection retrieval together with ´calc_snr´.

** History **

  • xxxx-xx-xx: First edition by …

  • 2026-07-03: translated to python

  • 2026-07-12: vectorized for better efficiency