ppcpy.qc

ppcpy.qc.pollySaturationDetect

ppcpy.qc.pollySaturationDetect.pollySaturationDetect(data_cube, rfill=250, sigSaturateThresh=500)

detect the bins which are fully saturated by the clouds.

INPUTS:

data: dict

Data dictionary. See documentation for format.

KEYWORDS:

hfill: float

Minimum range gap to fill (m). Default: 250

sigSaturateThresh: float

Threshold of saturated signal (photon count). Default: 500

OUTPUTS:

flag: boolean ndarray

True indicates current range bin is saturated by clouds.

HISTORY:

• 2018-12-21: First Edition by Zhenping

• 2019-07-08: Fix the bug of converting signal to PCR.

• 2025-05-14: translated and changed the algorithm to use scipy.ndimage

ppcpy.qc.transCor

ppcpy.qc.transCor.transCorGHK_cube(data_cube, signal='BGCor')

ppcpy.qc.transCor.transCor_E16_channel(sigT, bgT, voldepol, HT)

transmission correction for the total channel using the Mattis 2009/Engelmann 2016 method

Parameters:

sigTarray

Signal in total channel (background-corrected)

bgTarray

Background in total channel

voldepolarray

Volume depolarization ratio

HTfloat

Transmission ratio of total channel in GHK notation

Returns:

sigTCorarray

Signal in total channel corrected for polarization induced transmission effects

bgTCorarray

Background of total signal

Notes

Following [1] in the notation of [2]

\[P_{i, \text{corr}} = P_i \frac{1 + R_i \delta^V}{1+\delta^V} \]

with the signal \(P_i\), the transmission ratio \(R_i\) and the volume depolarization ratio \(\delta^V\)

Todo

Clarify the background treatment. The bgTCor should not change (i.e. assuming the vdr is 0?)

References

[1]

Mattis et al 2009

[2]

Engelmann et al 2016

ppcpy.qc.transCor.transCorGHK_channel(sigT, bgT, sigC, bgC, transGT=1, transGR=1, transHT=0, transHR=-1, polCaliEta=1, polCaliEtaStd=0)

Corrects the effect of different polarization-dependent transmission inside the total and depol channel.

PollyNET/Pollynet_Processing_Chain

INPUTS:

sigT: array

Signal in total channel.

bgT: array

Background in total channel.

sigC: array

Signal in cross channel.

bgC: array

Background in cross channel.

transGT: float

G parameter in total channel.

transGR: float

G parameter in cross channel.

transHT: float

H parameter in total channel.

transHR: float

H parameter in cross channel.

polCaliEta: float

Depolarization calibration constant (eta).

polCaliEtaStd: float

Uncertainty of the depolarization calibration constant.

OUTPUTS:

sigTCor: array

Transmission corrected elastic signal.

bgTCor: array

Background of transmission corrected elastic signal.

REFERENCES:

Mattis, I., Tesche, M., Grein, M., Freudenthaler, V., and Müller, D.: Systematic error of lidar profiles caused by a polarization-dependent receiver transmission: Quantification and error correction scheme, Appl. Opt., 48, 2742-2751, 2009. Freudenthaler, V. About the effects of polarising optics on lidar signals and the Delta90 calibration. Atmos. Meas. Tech., 9, 4181–4255 (2016).

HISTORY:

• 2021-05-27: First edition by Zhenping.

• 2024-08-14: Change to GHK parameterization by Moritz.

• 2024-12-28: AI translation

Authors: - zhenping@tropos.de, haarig@tropos.de

ppcpy.qc.overlapEst

ppcpy.qc.overlapEst.run_frnr_cldFreeGrps(data_cube, collect_debug: bool = True) → list

Estimate overlap function using far-range-near-range method.

Parameters:

data_cubeobject

Main PicassoProc object.

collect_debugbool

If true, collect debug information. Default is True.

Returns:

overlaplist of dicts

Per channel per cloud free period:

overlapndarray

Overlap function.

overlapStdndarray

Standard deviation of overlap function.

sigRatiofloat

Signal ratio between near-range and far-range signals.

normRangelist

Height index of the signal normalization range.

Notes

• The matlab version preforms some convertio from PC to PCR after calculating the overlap function (this might just be relevent at all)

• The matlab version also uses calculates 1 overlap function for each channel while we calculate 1 per clFreeGrp per channel…

• The frnr overlap calculations are very unstable and frequently results in values outside the range [0,1].

ppcpy.qc.overlapEst.overlapCalc(height: ndarray, sigFR: ndarray, bgFR: ndarray, sigNR: ndarray, bgNR: ndarray, hFullOverlap: float = 600, collect_debug=False) → dict

Calculate overlap function.

Parameters:

heightndarray

Height above ground (m).

sigFRndarray

Far-field signal (photon count).

bgFRndarray

Background of far-field signal (photon count).

sigNRndarray

Near-field signal (photon count).

bgNRndarray

Background of near-field signal (photon count).

Returns:

overlapndarray

Overlap function.

overlapStdndarray

Standard deviation of overlap function.

sigRatiofloat

Signal ratio between near-range and far-range signals.

normRangelist

Height index of the signal normalization range.

Other Parameters:

hFullOverlapfloat, optional

Minimum height with full overlap for far-range signal (default: 600 m).

Notes

History

• 2021-05-18: First edition by Zhenping

ppcpy.qc.overlapEst.run_raman_cldFreeGrps(data_cube, collect_debug: bool = True) → list

Estimate overlap function using Raman method.

Parameters:

data_cubeobject

Main PicassoProc object.

collect_debugbool

If true, collect debug information. Default is True.

Returns:

overlaplist of dicts

Per channel per cloud free period:

olFuncndarray

Overlap function.

olStdfloat

Standard deviation of overlap function.

olFunc0ndarray

Overlap function with no smoothing.

ppcpy.qc.overlapEst.overlapCalcRaman(Lambda_el: float, Lambda_Ra: float, height: ndarray, sigFRel: ndarray, sigFRRa: ndarray, molExt: ndarray, molBsc_r: ndarray, molExt_r: ndarray, aerBsc: ndarray, hFullOverlap: float = 600, smoothbins: float = 1, AE: float = 1, hres: float = 150, collect_debug=False) → dict

Calculate overlap function from Polly measurements based on Wandinger and Ansmann 2002 https://doi.org/10.1364/AO.41.000511

Parameters:

Lambda_elfloat

Elastic wavelength.

Lambda_Rafloat

Raman wavelength.

heightndarray

Height above ground [m].

sigFRelndarray

Far-field elastic signal.

sigFRRandarray

Far-field Raman signal.

molExtndarray

Molecular extinction (elastic).

molBsc_rndarray

Molecular Backscatter (inelastic).

molExt_rndarray

Molecular extinction (inelastic).

aerBscndarray

Particle backscattering derived with the Raman method [m^{-1}].

hFullOverlapfloat, optional

Minimum height with complete overlap [m]. Default is 600.

smoothbinsint, optional

Number of bins for smoothing. Default is 1.

AEfloat, optional

Angstroem exponent. Default is 1.

hresfloat, optional

Instrument height resolution [m]. Default is 150.

Returns:

olFuncndarray

Overlap function.

olStdfloat

Standard deviation of overlap function.

olFunc0ndarray

Overlap function with no smoothing.

Notes

Todo

What is returned by the function and what is described in the docstring does not corresponed.

Todo

This function uses a mix of ppcpy.misc.helper.unifrom_filter and scipy.ndimage.uniform_filter1d for smoothing. This is done to avoid NaN-value related errors. A better more cohesive solution should be precude in the future.

Todo

Be cearfull with NaN values! scipy.ndimage.uniform_filter1d used for smoothing in this module will propagate any NaN values present in the signal throughot the rest of the smoothed signal. Additionaly, here we are using mode ‘reflect’ for padding (see scipy.ndimage.uniform_filter1d documentation). This might not be the most optimal mode, the other availabel mode should also be considered. Optimally, should we designe our own filter for this purpuse that do not have the issue with propagating NaN values, Like what is used in the rest of the modules. However, without reducing dimension or filling in NaN values.

** History **

• 2023-06-06: First edition by Cristofer

ppcpy.qc.overlapEst.load(data_cube) → list

read the overlap function from files into a structure similar to the others.

Parameters:

data_cubeobject

Main PicassoProc object.

Returns:

overlaplist of dicts

ppcpy.qc.overlapCor

ppcpy.qc.overlapCor.spread(data_cube)

select the correct overlap method, spread the profiles to 2d for each wavelength

design decision for now: drop the signal glue option (overlapCorMode == 3)

in the matlab version any olFunc is additionally smoothed with olSm = smooth(olFuncDeft, p.Results.overlapSmWin, ‘sgolay’, 2); (e.g. here PollyNET/Pollynet_Processing_Chain) This should probably be done more explicitly

Also the ‘glueing’ in function [sigCor] = olCor(sigFR, overlap, height, normRange) seems wired (PollyNET/Pollynet_Processing_Chain)

ppcpy.qc.overlapCor.apply_cube(data_cube)

ppcpy.qc.overlapCor.fixLowest(overlap: ndarray, indexsearchmax: int, thres: float = 0.05)

very rough fix for exploding values in the very near range of the overlap function

in the lowest heights (below indexsearchmax, e.g. 800m) search for chunks, where the overlap function is smaller than 0.05 in that chunk take the miniumum and fill heights below

ppcpy.qc.overlapCor.hFullOLbyGrp(clFreeGrps, heightFullOverCor)

ppcpy.qc.qualityMask

ppcpy.qc.qualityMask.qualityMask(data_cube)

# 0 in quality_mask means good data # 1 in quality_mask means low-SNR data # 2 in quality_mask means depolarization calibration periods # 3 in quality_mask means shutter on # 4 in quality_mask means fog # 5 in quality_mask means saturated (NEW)

The lowSNRMask is actually calculated twice, once in pollyPreprocess.m and then again when the quality mask is evaluated in picassoProcV3.m. Also the original processing chain has a quality_mask_vdr, which should be a composite of cross and total, maybe this can be handled more logically here.