ppcpy.calibration#

ppcpy.calibration.lidarconstant#

Testtext for documentation

contains get_best_LC()

ppcpy.calibration.lidarconstant.loadDefaults(data_cube, **defaults) → dict[source]#

Prepare default Lidar calibration values.

Parameters:
data_cubeobject

Main PicassoProc object.

LClist, optional

Default Lidar constant value per channel.

LCStdlist, optional

Default Lidar constant error per channel.

Returns:
defaultDictdict

Default Lidar calibration result per channel.

Each channel contains a list with one single sub-dict with entries:

LCfloat

Default Lidar calibration constant.

LCStdfloat

Default uncertainty of lidar calibration constant.

methodstr

Name of retrieval method.

Notes

Default values are by standard taken from their config variable but can be overwritten if passed as an input to this function. The order of LC and LCStd must match the channel order in data_cube.retrievals_highres['channel'].

Todo

Consider allowing default values for a single channel to passed as input.

History

  • 2026-08-07: First edition by Buholdt

ppcpy.calibration.lidarconstant.lc_for_cldFreeGrps(data_cube, retrieval: str, collect_debug: bool = False) → dict[source]#

Estimate lidar calibration constant from the optical profiles.

Parameters:
data_cubeobject

Main PicassoProc object.

retrievalstr

Retrieval type. ‘klett’ or ‘raman’.

collect_debugbool, optional

If true, collects debug information. Default is False

Returns:
LCsdict

Lidar calibration results for retrieval retrieved optical profiles per channel.

Each channel contains a list of sub-dicts with entries:

LCfloat

Lidar calibration constant.

LCStdfloat

Uncertainty of lidar calibration constant.

time_start, time_endint

Start and stop times for successful calibration.

methodstr

Name of retrieval method.

The number of elements in each list depends on the number of successful retrievals.

Notes

For NR channels, the LC is calculated directly form the optical profiles, whereas in the matlab version, it is estimated by multiplying the respective FR LC with olAttri387.sigRatio.

The function uses the following configuration flags:

  • flagUseRetrievedExt4LCCalc: If enabled the retrieved extinction when calculating the LCs.

    If disabled the extinction will be estimated by the retrieved backscatter times the assumed LR.

The options for Rotational Raman and Aeronet LC retrievals are currently missing.

Todo

Check if LC’s are normalized with respect to the mean of the profiles.

Todo

Add option for Aeronet and rotational Raman retrieved LC.

History

xxxx-xx-xx: First edition by … 2026-03-18: Changed beta_mol for inelastic wavelengths and added the ‘flagUseRetrievedExt4LCCalc’ variable.

ppcpy.calibration.polarization#

ppcpy.calibration.polarization.onemx_onepx(x: float | ndarray) → float | ndarray[source]#

Calculate the fraction of (1-x)/(1+x)

ppcpy.calibration.polarization.smooth_signal(signal: ndarray, window_len: int) → ndarray[source]#

Uniformly smooth the input signal

Parameters:
singalndarray

Signal to be smooth

window_lenint

Width of the applied uniform filter

Returns:
ndarray

Smoothed signal

Notes

History

  • 2026-02-04: Changed from scipy.ndimage.uniform_filter1d to ppcpy.misc.helper.uniform_filter

ppcpy.calibration.polarization.loadDefaults(data_cube, **defaults) → dict[source]#

Prepare default Depol calibration values.

Parameters:
data_cubeobject

Main PicassoProc object.

polCaliEta355float, optional

Default Depol calibration constant at 355 nm.

polCaliEtaStd355float, optional

Default Depol calibration constant error at 355 nm.

polCaliEta532float, optional

Default Depol calibration constant at 532 nm.

polCaliEtaStd532float, optional

Default Depol calibration constant error at 532 nm.

polCaliEta1064float, optional

Default Depol calibration constant at 1064 nm.

polCaliEtaStd1064float, optional

Default Depol calibration constant error at 1064 nm.

Returns:
defaultDictdict

Default polarization calibration result per wavelength.

Each wavelength contains a list with one single sub-dict with entries:

etafloat

Default Depol calibration constant.

eta_stdfloat

Defaults uncertainty of Depol calibration constant.

methodstr

Name of retrieval method.

Notes

Default values are by standard taken from their config variable but can be overwritten if passed as an input to this function.

History

  • 2026-08-07: First edition by Buholdt

ppcpy.calibration.polarization.loadGHK(data_cube)[source]#

Prepare the GHK parameters, especially if given in TR convert them into GHK.

Parameters:
data_cubeobject

Main PicassoProc object.

Attributes:
data_cube.polly_config_dictdict
Updated to parameters:

TR –> Removed G –> filled and converted to array H –> filled and converted to array K –> filled and converted to array voldepol_error_355 –> converted to array voldepol_error_532 –> converted to array voldepol_error_1064 –> converted to array

Notes

Todo

  • Write a proper docstring.

** History **

  • xx-xx-xxxx: First edition by …

ppcpy.calibration.polarization.calibrateGHK(data_cube, collect_debug: bool = False) → dict[source]#

Estimate the polarization calibration from the Delta-90° method [1].

Parameters:
data_cubeobject

Main PicassoProc object

Returns:
pol_calidict

polarization calibration results from Delta-90° method for each wavelength.

Each wavelength contains a list of sub-dicts. One per successful retrieval period, with entries:

etafloat

Depol calibration constant.

eta_stdfloat

Uncertainty of Depol calibration constant.

time_start, time_endint

Start and stop times for successful calibration.

methodstr

Name of retrieval method.

The number of element in each list depends on the number of successful retrievals.

Notes

Function is called here PollyNET/Pollynet_Processing_Chain The two most relevant functions here are PollyNET/Pollynet_Processing_Chain which also calls PollyNET/Pollynet_Processing_Chain

References

[1]

Freudenthaler, V. About the effects of polarising optics on lidar signals and the Delta90 calibration. Atmos. Meas. Tech., 9, 4181–4255 (2016).

ppcpy.calibration.polarization.depol_cali_ghk(signal_t: ndarray, bg_t: ndarray, signal_x: ndarray, bg_x: ndarray, time: ndarray, pol_cali_pang_start_time: ndarray, pol_cali_pang_stop_time: ndarray, pol_cali_nang_start_time: ndarray, pol_cali_nang_stop_time: ndarray, K: float, cali_h_indx_range: list | tuple, SNRmin: list, sig_max: list, rel_std_dplus: float, rel_std_dminus: float, segment_len: int, smooth_win: int, collect_debug: bool = False, flagPicassoComparison: bool = False) → list[source]#

Polarization calibration for PollyXT lidar system.

Parameters:
signal_tndarray

Background-removed photon count signal at the total channel. Shape: (n_profiles, n_bins)

bg_tndarray

Background at the total channel. Shape: (n_profiles)

signal_xndarray

Background-removed photon count signal at the cross channel. Shape: (n_profiles, n_bins)

bg_xndarray

Background at the cross channel. Shape: (n_profiles)

timendarray

Datetime array representing the measurement time of each profile.

pol_cali_pang_start_time, pol_cali_pang_stop_timendarray

Start and stop times when the polarizer rotates to the positive angle.

pol_cali_nang_start_time, pol_cali_nang_stop_timendarray

Start and stop times when the polarizer rotates to the negative angle.

Kfloat

Parameter from GHK to correct the calibration.

cali_h_indx_rangelist or tuple

Range of height indexes to use for polarization calibration.

SNRminlist

Minimum SNR for calibration. Length: 4

sig_maxlist

Maximum signal allowed for calibration to prevent pulse pileup.

rel_std_dplus, rel_std_dminusfloat

Maximum relative uncertainty of dplus and dminus allowed.

segment_lenint

Segment length for testing the variability of calibration results.

smooth_winint

Width of the sliding window for smoothing the signal.

collect_debugbool, optional

Store and return the intermediate results. Default is False.

Returns:
resultslist of dicts

Containing successful retrieved depol calibration constant and retrieval information.

Each dict contains the following entries:

etafloat

Eta values from polarization calibration.

eta_stdfloat

Uncertainty of eta values from calibration.

time_start, time_endint

Start and stop times for successful calibration.

methodstr

Name of retrieval method.

The number of element in the list depends on the number of successful retrievals.

Notes

Todo

Why is pol_cali_nang_start_time returned as time_start of the calibration period instead of pol_cali_start_time???

History

  • xxxx-xx-xx: First edition by …

  • 2026-07-30: Made output datatype consistent and changed from nanmean to

    nansum in signal aggregation to be consistent with SNR requirement.

  • 2026-08-06: Overhalled function structure and input output consistency.

ppcpy.calibration.polarization.analyze_segments(dplus: ndarray, dminus: ndarray, segment_len: int, rel_std_dplus: float, rel_std_dminus: float) → tuple[source]#

Analyze calibration segment.

Parameters:
dplusndarray

Plus 45° calibration…

dminusndarray

Minus 45° calibration…

segment_lenint

Segment length for testing the variability of calibration results.

rel_std_dplus, rel_std_dminusfloat

Maximum relative uncertainty of dplus and dminus allowed.

Returns:
mean_dplusesndarray

Mean plus 45° calibration per segment.

std_dplusesndarray

Standard deviation of plus 45° calibration per segment.

mean_dminusesndarray

Mean minus 45° calibration per segment.

std_dminusesndarray

Standard deviation of minus 45° calibration per segment.

ppcpy.calibration.polarization.calibrateMol(data_cube) → dict[source]#

Calibrate the polarization with the molecular signal.

Parameters:
data_cubeobject

Main PicassoProc object.

Returns:
etandarray

Polarization calibration eta.

etaStdndarray

Uncertainty of polarization calibration eta.

facarray

Polarization calibration factor.

facStdndarray

Uncertainty of polarization calibration factor.

statusint
Retrieval status

0 : Bad 1 : Good

time_startint

Start time of the cloud free segment for the retrieval in unixtime.

end_timeint

End time of the cloud free segment for the retrieval in unixtime.

Notes

  • Converted from the matlab code to the best knowledge, but not cross-validated yet.

Todo

  • had to calculate TR_t, TR_c again when calling depol_cali_mol().

  • Finish docstring.

** History **

  • xx-xx-xxxx: First edition by …

  • xx-xx-xxxx: Translated to python

ppcpy.calibration.polarization.depol_cali_mol(signal_t: ndarray, background_t: ndarray, signal_c: ndarray, background_c: ndarray, TR_t: float, TR_t_std: float, TR_c: float, TR_c_std: float, minSNR: float, mdr: float, mdrStd: float, flagPicassoComparison: bool = False) → dict[source]#

Molecular polarization calibration.

Parameters:
signal_tndarray

Total signal [photon count].

background_tndarray

Background at total channel [photon count].

signal_cndarray

Cross signal [photon count].

background_cndarray

Background at cross channel [photon count].

TR_tfloat

Transmission ratio at total channel.

TR_t_stdfloat

Uncertainty of the transmission ratio at total channel.

TR_cfloat

Transmission ratio at cross channel.

TR_c_stdfloat

Uncertainty of the transmission ratio at cross channel.

minSNRfloat

The SNR constraint for the signal strength at reference height.

mdrfloat

Default molecular depolarization ratio.

mdrStdfloat

Default standard deviation of molecular depolarization ratio.

Returns:
etandarray

Polarization calibration eta.

etaStdndarray

Uncertainty of polarization calibration eta.

facarray

Polarization calibration factor.

facStdndarray

Uncertainty of polarization calibration factor.

statusint
Retrieval status

0 : Bad 1 : Good

Notes

Todo

  • The inputs TR_t_std & TR_c_std are currnetly not used in the function!

History

  • 2021-07-06: First edition by Zhenping

  • 2024-12-23: converted to python

References

Baars, H., et al., Aerosol profiling with lidar in the Amazon Basin during the wet and dry season, J Geophys Res-Atmos, 117, 10.1029/2012jd018338, 2012.

ppcpy.calibration.rayleighfit#

ppcpy.calibration.rayleighfit.getVal(array: ndarray, indices: list) → list[source]#

Get values of array for a set of indices, including robustness to NaN indices.

Parameters:
heightndarray

Array to be searched.

indicesarry_like

Indeces of the array, may include NaN values.

Returns:
outlist

Values of the array at given indices. If indx is NaN value is also NaN.

ppcpy.calibration.rayleighfit.rayleighfit(data_cube, collect_debug: bool = False) → list[source]#

Perform Rayleighfit procedure using Douglas Peucker algorithm for each channel and cloud free period.

Parameters:
data_cubeobject

Main PicassoProc object.

collect_debugbool

If true, collects debug information.

Returns:
refHlist of dicts

Reference heights per channel per cloud free period.

Notes

  • This function uses the Douglas Peucker algorithm to segmentize the signal inside a search range specified by the config variables ‘heightFullOverlap’ and ‘maxDecomHeight’. These segments are considered as potential reference height. The final reference height is chosen as the segment with the best fit to the molecular signal.

  • Currently, only the FR reference heights are calculated, while the NR reference heights are taken from the config files.

ppcpy.calibration.rayleighfit.smooth_signal(signal: ndarray, window_len: int) → ndarray[source]#

Uniformly smooth the input signal

Parameters:
singalndarray

Signal to be smooth

window_lenint

Width of the applied uniform filter

Returns:
ndarray

Smoothed signal

Notes

History

  • 2026-02-04: Changed from scipy.ndimage.uniform_filter1d to ppcpy.misc.helper.uniform_filter

  • 2026-04-17: Changed to ppcpy.misc.helper.moving_average to get values at edges.

ppcpy.calibration.rayleighfit.DouglasPeucker(signal: ndarray, height: ndarray, epsilon: float, heightBase: float, heightTop: float, maxHThick: float, window_size: int = 1, showDetails: bool = False) → ndarray[source]#

Simplify signal according to Douglas-Peucker algorithm.

Parameters:
signalndarray

Molecule corrected signal. [MHz]

heightndarray

Height. [m]

epsilonfloat

Maximum distance.

heightBasefloat

Minimum height for the algorithm. [m]

heightTopfloat

Maximum height for the algorithm. [m]

maxHThickfloat

Maximum spatial thickness of each segment. [m]

window_sizeint

Size of the average smooth window. Default is 1.

showDetailsbool

If true, collect debug information. Default is False.

Returns:
sigIndxarray

Index of the signal that stands for different segments of the signal.

Notes

History

  • 2017-12-29: First edition by Zhenping.

  • 2018-07-29: Add the height range for the searching instead of SNR restriction.

  • 2018-07-31: Add the maxHThick argument to control the maximum thickness of each output segment.

  • 2024-12-20: Direct translated from matlab with ai

References

https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm

ppcpy.calibration.rayleighfit.DP_algorithm(pointList: list, epsilon: float, maxHThick: float, recursive_depth: int = 0, showDetails: bool = False) → list[source]#

Recursive implementation of the Douglas-Peucker algorithm.

Parameters:
pointListlist

List of points [[x1, y1], [x2, y2], …].

epsilonfloat

Maximum distance.

maxHThickfloat

Maximum thickness for each segment.

showDetailsbool

If True, print debug information. Default is False.

Returns:
sigIndxlist

Indices of simplified points.

ppcpy.calibration.rayleighfit.my_dist(pointM: list, pointS: list, pointE: list, full: bool = True) → float[source]#

Calculate the perpendicular distance between pointM and the line connecting pointS and pointE.

Parameters:
pointMlist

Middle point [x, y].

pointSlist

Start point [x, y].

pointElist

End point [x, y].

fullbool

If true, calculate the full distance, othrewise calculate only the numerator. Default is True.

Returns:
dfloat

Distance.

Notes

  • If pointS and pointE are constant only the numerator is neede to calculate the extreme points with respect to pointM.

ppcpy.calibration.rayleighfit.chi2fit(x: ndarray, y: ndarray, measure_error: ndarray) → tuple[source]#

CHI2FIT Chi-2 fitting. All the code are translated from the exemplified code in Numerical Recipes in C (2nd Edition). Great help comes from Birgit Heese.

Parameters:
xndarray

The length of x should be larger than 1.

yndarray

The measured signal.

measure_errorndarray

Measurement errors for the y values.

Returns:
afloat

Intercept of the linear regression.

bfloat

Slope of the linear regression.

sigmaAfloat

Uncertainty of the intercept.

sigmaBfloat

Uncertainty of the slope.

chi2float

Chi-square value.

Qfloat

Goodness of fit.

Notes

History

  • 2018-08-03: First edition by Zhenping.

Examples

a, b, sigmaA, sigmaB, chi2, Q = chi2fit(x, y, measure_error)

ppcpy.calibration.rayleighfit.fit_profile(height: ndarray, sig_aer: ndarray, pc: ndarray, bg: ndarray, sig_mol: ndarray, dpIndx: ndarray, layerThickConstrain: float, slopeConstrain: float, SNRConstrain: float, flagShowDetail: bool = False, flagPicassoComparison: bool = False) → tuple[source]#

Search the clean region with rayleigh fit algorithm.

Parameters:
heightarray_like

height [m]

sig_aerarray_like

range corrected signal

pcarray_like

photon count signal

bgarray_like

background

sig_molarray_like

range corrected molecular signal

dpIndxarray_like

index of the region calculated by Douglas-Peucker algorithm

layerThickConstrainfloat

constrain for the reference layer thickness [m]

slopeConstrainfloat

constrain for the uncertainty of the regressed extinction coefficient

SNRConstrainfloat

minimum SNR for the signal at the reference height

flagShowDetailbool, optional

If true, calculation information will be printed. Default is False.

flagPicassoComparisonbool, optional

If true, use Picasso values and logic. Default is False.

Returns:
tuple

(hBIndx, hTIndx) - indices of bottom and top of searched region Returns (nan, nan) if region not found

Notes

  • There are known numerical discrepancies between this and the Picasso versions especially in the residual calculations due to the subtraction and mean of very small numbers [-1e-24, 1e-24]. these numerical discrepancies may cause a different reference height to be chosen compared to Picasso.

ppcpy.calibration.select#

ppcpy.calibration.select.single_best(d: dict, name_val: str, name_min: str, name_method: str, relative: bool = False) → dict[source]#

Select the best calibration constant

Generalization of lidarconstant.get_best_LC

Parameters:
ddict

dict per channel of list of calibration constants

name_valstr

designator for the actual value

name_minstr

designator of the minimum

relativebool

If true, choose calibration constant based on the relative error.

Returns:
bestdict

Lidar constants/Etas with lowest standard deviation per channel.

History
  • 2026-02-16: Added additional checks to hinder negative LCs to be chosen.
  • 2026-03-27: generalized to also hold for depolarization calibration.
ppcpy.calibration.select.plot_cals(d: dict, param: str, used: dict = None)[source]#

Plot the calibration constants.

Produces a scatter plot of the stored calibration constants (CC). CCs retrieved form the measurement are marked as filled circles, while those loaded from the database are marked as hollow circles. The chosen optimal CC and its default value are marked by horizontal gray dotted and dashed lines respectively.

Parameters:
ddict

Dict storing all CCs. pol_cali or LC.

paramstr

Name of the parameter to extract.

useddict, optional

Dict storing the used CCs. LCused or etaused. Will produce a horizontal dashed line in the plot if added. Default is None.

Returns:
figfigure

Matplotlib figure object.

axaxis

Matplotlib axis object.

Examples

>>> plot_cals(data_cube.pol_cali, 'eta', used=data_cube.etaused)
>>> plot_cals(data_cube.LC, 'LC', used=data_cube.LCused)