ppcpy.calibration#
ppcpy.calibration.lidarconstant#
Testtext for documentation
contains get_best_LC()
- ppcpy.calibration.lidarconstant.loadDefaults(data_cube, **defaults) dict[source]#
Prepare default Lidar calibration values.
- Parameters:
- data_cubeobject
Main PicassoProc object.
- LClist, optional
Default Lidar constant value per channel.
- LCStdlist, optional
Default Lidar constant error per channel.
- Returns:
- defaultDictdict
Default Lidar calibration result per channel.
Each channel contains a list with one single sub-dict with entries:
LCfloatDefault Lidar calibration constant.
LCStdfloatDefault uncertainty of lidar calibration constant.
methodstrName of retrieval method.
Notes
Default values are by standard taken from their config variable but can be overwritten if passed as an input to this function. The order of
LCandLCStdmust match the channel order indata_cube.retrievals_highres['channel'].Todo
Consider allowing default values for a single channel to passed as input.
History
2026-08-07: First edition by Buholdt
- ppcpy.calibration.lidarconstant.lc_for_cldFreeGrps(data_cube, retrieval: str, collect_debug: bool = False) dict[source]#
Estimate lidar calibration constant from the optical profiles.
- Parameters:
- data_cubeobject
Main PicassoProc object.
- retrievalstr
Retrieval type. ‘klett’ or ‘raman’.
- collect_debugbool, optional
If true, collects debug information. Default is False
- Returns:
- LCsdict
Lidar calibration results for
retrievalretrieved optical profiles per channel.Each channel contains a list of sub-dicts with entries:
LCfloatLidar calibration constant.
LCStdfloatUncertainty of lidar calibration constant.
time_start,time_endintStart and stop times for successful calibration.
methodstrName of retrieval method.
The number of elements in each list depends on the number of successful retrievals.
Notes
For NR channels, the LC is calculated directly form the optical profiles, whereas in the matlab version, it is estimated by multiplying the respective FR LC with
olAttri387.sigRatio.The function uses the following configuration flags:
flagUseRetrievedExt4LCCalc: If enabled the retrieved extinction when calculating the LCs.If disabled the extinction will be estimated by the retrieved backscatter times the assumed LR.
The options for Rotational Raman and Aeronet LC retrievals are currently missing.
Todo
Check if LC’s are normalized with respect to the mean of the profiles.
Todo
Add option for Aeronet and rotational Raman retrieved LC.
History
xxxx-xx-xx: First edition by … 2026-03-18: Changed beta_mol for inelastic wavelengths and added the ‘flagUseRetrievedExt4LCCalc’ variable.
ppcpy.calibration.polarization#
- ppcpy.calibration.polarization.onemx_onepx(x: float | ndarray) float | ndarray[source]#
Calculate the fraction of (1-x)/(1+x)
- ppcpy.calibration.polarization.smooth_signal(signal: ndarray, window_len: int) ndarray[source]#
Uniformly smooth the input signal
- Parameters:
- singalndarray
Signal to be smooth
- window_lenint
Width of the applied uniform filter
- Returns:
- ndarray
Smoothed signal
Notes
History
2026-02-04: Changed from scipy.ndimage.uniform_filter1d to ppcpy.misc.helper.uniform_filter
- ppcpy.calibration.polarization.loadDefaults(data_cube, **defaults) dict[source]#
Prepare default Depol calibration values.
- Parameters:
- data_cubeobject
Main PicassoProc object.
- polCaliEta355float, optional
Default Depol calibration constant at 355 nm.
- polCaliEtaStd355float, optional
Default Depol calibration constant error at 355 nm.
- polCaliEta532float, optional
Default Depol calibration constant at 532 nm.
- polCaliEtaStd532float, optional
Default Depol calibration constant error at 532 nm.
- polCaliEta1064float, optional
Default Depol calibration constant at 1064 nm.
- polCaliEtaStd1064float, optional
Default Depol calibration constant error at 1064 nm.
- Returns:
- defaultDictdict
Default polarization calibration result per wavelength.
Each wavelength contains a list with one single sub-dict with entries:
etafloatDefault Depol calibration constant.
eta_stdfloatDefaults uncertainty of Depol calibration constant.
methodstrName of retrieval method.
Notes
Default values are by standard taken from their config variable but can be overwritten if passed as an input to this function.
History
2026-08-07: First edition by Buholdt
- ppcpy.calibration.polarization.loadGHK(data_cube)[source]#
Prepare the GHK parameters, especially if given in TR convert them into GHK.
- Parameters:
- data_cubeobject
Main PicassoProc object.
- Attributes:
- data_cube.polly_config_dictdict
- Updated to parameters:
TR –> Removed G –> filled and converted to array H –> filled and converted to array K –> filled and converted to array voldepol_error_355 –> converted to array voldepol_error_532 –> converted to array voldepol_error_1064 –> converted to array
Notes
Todo
Write a proper docstring.
** History **
xx-xx-xxxx: First edition by …
- ppcpy.calibration.polarization.calibrateGHK(data_cube, collect_debug: bool = False) dict[source]#
Estimate the polarization calibration from the Delta-90° method [1].
- Parameters:
- data_cubeobject
Main PicassoProc object
- Returns:
- pol_calidict
polarization calibration results from Delta-90° method for each wavelength.
Each wavelength contains a list of sub-dicts. One per successful retrieval period, with entries:
etafloatDepol calibration constant.
eta_stdfloatUncertainty of Depol calibration constant.
time_start,time_endintStart and stop times for successful calibration.
methodstrName of retrieval method.
The number of element in each list depends on the number of successful retrievals.
Notes
Function is called here PollyNET/Pollynet_Processing_Chain The two most relevant functions here are PollyNET/Pollynet_Processing_Chain which also calls PollyNET/Pollynet_Processing_Chain
References
[1]Freudenthaler, V. About the effects of polarising optics on lidar signals and the Delta90 calibration. Atmos. Meas. Tech., 9, 4181–4255 (2016).
- ppcpy.calibration.polarization.depol_cali_ghk(signal_t: ndarray, bg_t: ndarray, signal_x: ndarray, bg_x: ndarray, time: ndarray, pol_cali_pang_start_time: ndarray, pol_cali_pang_stop_time: ndarray, pol_cali_nang_start_time: ndarray, pol_cali_nang_stop_time: ndarray, K: float, cali_h_indx_range: list | tuple, SNRmin: list, sig_max: list, rel_std_dplus: float, rel_std_dminus: float, segment_len: int, smooth_win: int, collect_debug: bool = False, flagPicassoComparison: bool = False) list[source]#
Polarization calibration for PollyXT lidar system.
- Parameters:
- signal_tndarray
Background-removed photon count signal at the total channel. Shape: (n_profiles, n_bins)
- bg_tndarray
Background at the total channel. Shape: (n_profiles)
- signal_xndarray
Background-removed photon count signal at the cross channel. Shape: (n_profiles, n_bins)
- bg_xndarray
Background at the cross channel. Shape: (n_profiles)
- timendarray
Datetime array representing the measurement time of each profile.
- pol_cali_pang_start_time, pol_cali_pang_stop_timendarray
Start and stop times when the polarizer rotates to the positive angle.
- pol_cali_nang_start_time, pol_cali_nang_stop_timendarray
Start and stop times when the polarizer rotates to the negative angle.
- Kfloat
Parameter from GHK to correct the calibration.
- cali_h_indx_rangelist or tuple
Range of height indexes to use for polarization calibration.
- SNRminlist
Minimum SNR for calibration. Length: 4
- sig_maxlist
Maximum signal allowed for calibration to prevent pulse pileup.
- rel_std_dplus, rel_std_dminusfloat
Maximum relative uncertainty of dplus and dminus allowed.
- segment_lenint
Segment length for testing the variability of calibration results.
- smooth_winint
Width of the sliding window for smoothing the signal.
- collect_debugbool, optional
Store and return the intermediate results. Default is False.
- Returns:
- resultslist of dicts
Containing successful retrieved depol calibration constant and retrieval information.
Each dict contains the following entries:
etafloatEta values from polarization calibration.
eta_stdfloatUncertainty of eta values from calibration.
time_start,time_endintStart and stop times for successful calibration.
methodstrName of retrieval method.
The number of element in the list depends on the number of successful retrievals.
Notes
Todo
Why is
pol_cali_nang_start_timereturned astime_startof the calibration period instead ofpol_cali_start_time???History
xxxx-xx-xx: First edition by …
- 2026-07-30: Made output datatype consistent and changed from nanmean to
nansum in signal aggregation to be consistent with SNR requirement.
2026-08-06: Overhalled function structure and input output consistency.
- ppcpy.calibration.polarization.analyze_segments(dplus: ndarray, dminus: ndarray, segment_len: int, rel_std_dplus: float, rel_std_dminus: float) tuple[source]#
Analyze calibration segment.
- Parameters:
- dplusndarray
Plus 45° calibration…
- dminusndarray
Minus 45° calibration…
- segment_lenint
Segment length for testing the variability of calibration results.
- rel_std_dplus, rel_std_dminusfloat
Maximum relative uncertainty of dplus and dminus allowed.
- Returns:
- mean_dplusesndarray
Mean plus 45° calibration per segment.
- std_dplusesndarray
Standard deviation of plus 45° calibration per segment.
- mean_dminusesndarray
Mean minus 45° calibration per segment.
- std_dminusesndarray
Standard deviation of minus 45° calibration per segment.
- ppcpy.calibration.polarization.calibrateMol(data_cube) dict[source]#
Calibrate the polarization with the molecular signal.
- Parameters:
- data_cubeobject
Main PicassoProc object.
- Returns:
- etandarray
Polarization calibration eta.
- etaStdndarray
Uncertainty of polarization calibration eta.
- facarray
Polarization calibration factor.
- facStdndarray
Uncertainty of polarization calibration factor.
- statusint
- Retrieval status
0 : Bad 1 : Good
- time_startint
Start time of the cloud free segment for the retrieval in unixtime.
- end_timeint
End time of the cloud free segment for the retrieval in unixtime.
Notes
Converted from the matlab code to the best knowledge, but not cross-validated yet.
Todo
had to calculate TR_t, TR_c again when calling depol_cali_mol().
Finish docstring.
** History **
xx-xx-xxxx: First edition by …
xx-xx-xxxx: Translated to python
- ppcpy.calibration.polarization.depol_cali_mol(signal_t: ndarray, background_t: ndarray, signal_c: ndarray, background_c: ndarray, TR_t: float, TR_t_std: float, TR_c: float, TR_c_std: float, minSNR: float, mdr: float, mdrStd: float, flagPicassoComparison: bool = False) dict[source]#
Molecular polarization calibration.
- Parameters:
- signal_tndarray
Total signal [photon count].
- background_tndarray
Background at total channel [photon count].
- signal_cndarray
Cross signal [photon count].
- background_cndarray
Background at cross channel [photon count].
- TR_tfloat
Transmission ratio at total channel.
- TR_t_stdfloat
Uncertainty of the transmission ratio at total channel.
- TR_cfloat
Transmission ratio at cross channel.
- TR_c_stdfloat
Uncertainty of the transmission ratio at cross channel.
- minSNRfloat
The SNR constraint for the signal strength at reference height.
- mdrfloat
Default molecular depolarization ratio.
- mdrStdfloat
Default standard deviation of molecular depolarization ratio.
- Returns:
- etandarray
Polarization calibration eta.
- etaStdndarray
Uncertainty of polarization calibration eta.
- facarray
Polarization calibration factor.
- facStdndarray
Uncertainty of polarization calibration factor.
- statusint
- Retrieval status
0 : Bad 1 : Good
Notes
Todo
The inputs TR_t_std & TR_c_std are currnetly not used in the function!
History
2021-07-06: First edition by Zhenping
2024-12-23: converted to python
References
Baars, H., et al., Aerosol profiling with lidar in the Amazon Basin during the wet and dry season, J Geophys Res-Atmos, 117, 10.1029/2012jd018338, 2012.
ppcpy.calibration.rayleighfit#
- ppcpy.calibration.rayleighfit.getVal(array: ndarray, indices: list) list[source]#
Get values of array for a set of indices, including robustness to NaN indices.
- Parameters:
- heightndarray
Array to be searched.
- indicesarry_like
Indeces of the array, may include NaN values.
- Returns:
- outlist
Values of the array at given indices. If indx is NaN value is also NaN.
- ppcpy.calibration.rayleighfit.rayleighfit(data_cube, collect_debug: bool = False) list[source]#
Perform Rayleighfit procedure using Douglas Peucker algorithm for each channel and cloud free period.
- Parameters:
- data_cubeobject
Main PicassoProc object.
- collect_debugbool
If true, collects debug information.
- Returns:
- refHlist of dicts
Reference heights per channel per cloud free period.
Notes
This function uses the Douglas Peucker algorithm to segmentize the signal inside a search range specified by the config variables ‘heightFullOverlap’ and ‘maxDecomHeight’. These segments are considered as potential reference height. The final reference height is chosen as the segment with the best fit to the molecular signal.
Currently, only the FR reference heights are calculated, while the NR reference heights are taken from the config files.
- ppcpy.calibration.rayleighfit.smooth_signal(signal: ndarray, window_len: int) ndarray[source]#
Uniformly smooth the input signal
- Parameters:
- singalndarray
Signal to be smooth
- window_lenint
Width of the applied uniform filter
- Returns:
- ndarray
Smoothed signal
Notes
History
2026-02-04: Changed from scipy.ndimage.uniform_filter1d to ppcpy.misc.helper.uniform_filter
2026-04-17: Changed to ppcpy.misc.helper.moving_average to get values at edges.
- ppcpy.calibration.rayleighfit.DouglasPeucker(signal: ndarray, height: ndarray, epsilon: float, heightBase: float, heightTop: float, maxHThick: float, window_size: int = 1, showDetails: bool = False) ndarray[source]#
Simplify signal according to Douglas-Peucker algorithm.
- Parameters:
- signalndarray
Molecule corrected signal. [MHz]
- heightndarray
Height. [m]
- epsilonfloat
Maximum distance.
- heightBasefloat
Minimum height for the algorithm. [m]
- heightTopfloat
Maximum height for the algorithm. [m]
- maxHThickfloat
Maximum spatial thickness of each segment. [m]
- window_sizeint
Size of the average smooth window. Default is 1.
- showDetailsbool
If true, collect debug information. Default is False.
- Returns:
- sigIndxarray
Index of the signal that stands for different segments of the signal.
Notes
History
2017-12-29: First edition by Zhenping.
2018-07-29: Add the height range for the searching instead of SNR restriction.
2018-07-31: Add the maxHThick argument to control the maximum thickness of each output segment.
2024-12-20: Direct translated from matlab with ai
References
https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm
- ppcpy.calibration.rayleighfit.DP_algorithm(pointList: list, epsilon: float, maxHThick: float, recursive_depth: int = 0, showDetails: bool = False) list[source]#
Recursive implementation of the Douglas-Peucker algorithm.
- Parameters:
- pointListlist
List of points [[x1, y1], [x2, y2], …].
- epsilonfloat
Maximum distance.
- maxHThickfloat
Maximum thickness for each segment.
- showDetailsbool
If True, print debug information. Default is False.
- Returns:
- sigIndxlist
Indices of simplified points.
- ppcpy.calibration.rayleighfit.my_dist(pointM: list, pointS: list, pointE: list, full: bool = True) float[source]#
Calculate the perpendicular distance between pointM and the line connecting pointS and pointE.
- Parameters:
- pointMlist
Middle point [x, y].
- pointSlist
Start point [x, y].
- pointElist
End point [x, y].
- fullbool
If true, calculate the full distance, othrewise calculate only the numerator. Default is True.
- Returns:
- dfloat
Distance.
Notes
If pointS and pointE are constant only the numerator is neede to calculate the extreme points with respect to pointM.
- ppcpy.calibration.rayleighfit.chi2fit(x: ndarray, y: ndarray, measure_error: ndarray) tuple[source]#
CHI2FIT Chi-2 fitting. All the code are translated from the exemplified code in Numerical Recipes in C (2nd Edition). Great help comes from Birgit Heese.
- Parameters:
- xndarray
The length of x should be larger than 1.
- yndarray
The measured signal.
- measure_errorndarray
Measurement errors for the y values.
- Returns:
- afloat
Intercept of the linear regression.
- bfloat
Slope of the linear regression.
- sigmaAfloat
Uncertainty of the intercept.
- sigmaBfloat
Uncertainty of the slope.
- chi2float
Chi-square value.
- Qfloat
Goodness of fit.
Notes
History
2018-08-03: First edition by Zhenping.
Examples
a, b, sigmaA, sigmaB, chi2, Q = chi2fit(x, y, measure_error)
- ppcpy.calibration.rayleighfit.fit_profile(height: ndarray, sig_aer: ndarray, pc: ndarray, bg: ndarray, sig_mol: ndarray, dpIndx: ndarray, layerThickConstrain: float, slopeConstrain: float, SNRConstrain: float, flagShowDetail: bool = False, flagPicassoComparison: bool = False) tuple[source]#
Search the clean region with rayleigh fit algorithm.
- Parameters:
- heightarray_like
height [m]
- sig_aerarray_like
range corrected signal
- pcarray_like
photon count signal
- bgarray_like
background
- sig_molarray_like
range corrected molecular signal
- dpIndxarray_like
index of the region calculated by Douglas-Peucker algorithm
- layerThickConstrainfloat
constrain for the reference layer thickness [m]
- slopeConstrainfloat
constrain for the uncertainty of the regressed extinction coefficient
- SNRConstrainfloat
minimum SNR for the signal at the reference height
- flagShowDetailbool, optional
If true, calculation information will be printed. Default is False.
- flagPicassoComparisonbool, optional
If true, use Picasso values and logic. Default is False.
- Returns:
- tuple
(hBIndx, hTIndx) - indices of bottom and top of searched region Returns (nan, nan) if region not found
Notes
There are known numerical discrepancies between this and the Picasso versions especially in the residual calculations due to the subtraction and mean of very small numbers [-1e-24, 1e-24]. these numerical discrepancies may cause a different reference height to be chosen compared to Picasso.
ppcpy.calibration.select#
- ppcpy.calibration.select.single_best(d: dict, name_val: str, name_min: str, name_method: str, relative: bool = False) dict[source]#
Select the best calibration constant
Generalization of lidarconstant.get_best_LC
- Parameters:
- ddict
dict per channel of list of calibration constants
- name_valstr
designator for the actual value
- name_minstr
designator of the minimum
- relativebool
If true, choose calibration constant based on the relative error.
- Returns:
- bestdict
Lidar constants/Etas with lowest standard deviation per channel.
- History
- 2026-02-16: Added additional checks to hinder negative LCs to be chosen.
- 2026-03-27: generalized to also hold for depolarization calibration.
- ppcpy.calibration.select.plot_cals(d: dict, param: str, used: dict = None)[source]#
Plot the calibration constants.
Produces a scatter plot of the stored calibration constants (CC). CCs retrieved form the measurement are marked as filled circles, while those loaded from the database are marked as hollow circles. The chosen optimal CC and its default value are marked by horizontal gray dotted and dashed lines respectively.
- Parameters:
- ddict
Dict storing all CCs.
pol_caliorLC.- paramstr
Name of the parameter to extract.
- useddict, optional
Dict storing the used CCs. LCused or etaused. Will produce a horizontal dashed line in the plot if added. Default is None.
- Returns:
- figfigure
Matplotlib figure object.
- axaxis
Matplotlib axis object.
Examples
>>> plot_cals(data_cube.pol_cali, 'eta', used=data_cube.etaused) >>> plot_cals(data_cube.LC, 'LC', used=data_cube.LCused)