ppcpy.preprocess#
ppcpy.preprocess.pollyPreprocess#
- ppcpy.preprocess.pollyPreprocess.pollyPreprocess(rawdata_dict: dict, collect_debug: bool = False, flagPicassoComparison: bool = False, **param: dict) dict[source]#
Preprocessing of Lidar-data.
- Includes the following processes in order:
Deadtime correction
Background correction
First-bin shift
Mask for low-SNR
Mask for depolarization-calibration process
Range correction.
- Parameters:
- rawdata_dictdict
- rawSignalndarray
Signal [Photon Count].
- mShotsndarray
Number of the laser shots for each profile.
- mTimendarray
Datetime array for the measurement time of each profile.
- depCalAngndarray
Angle of the polarizer in the receiving channel (>0 means calibration process starts).
- zenithAngndarray
Zenith angle of the laer beam.
- repRatefloat
Laser pulse repetition rate [s^-1].
- hResfloat
Spatial resolution [m].
- mSitestr
Measurement site.
- collect_debugbool
If true, collects debug information. Default is False.
- flagPicassoComparisonbool
If true, use Picasso values and logic.
- Returns:
- data_dictdict
- mShotsndarray
Number of the laser shots for each profile.
- sigDTCorndarray
Dead time corrected signal [photon count].
- BGndarray
Background.
- sigBGCorndarray
Backgound corrected signal [photon count].
- rangendarray
Height above ground at zenith angle [m].
- heightndarray
Height above ground [m].
- altndarray
Altitude [m].
- timelist
Measurement time in unix time.
- time64ndarray
Measurement time in numpy.datetime.
- SNRndarray
Signal to noise ratio.
- lowSNRMaskndarray
True if SNR is less SNRmin. Otherwise False.
- shutterOnMaskndarray
True if at timesteps where the shutters was on. Otherwise False.
- fogMaskndarray
If it is foggy which means the signal will be very weak, fogMask will be set True. Otherwise, False.
- mask607Offndarray
Mask of PMT on/off status at 607 nm channel.
- mask387Offndarray
Mask of PMT on/off status at 387 nm channel.
- mask407Offndarray
Mask of PMT on/off status at 407 nm channel.
- mask355RROffndarray
Mask of PMT on/off status at 355 nm rotational Raman channel.
- mask532RROffndarray
Mask of PMT on/off status at 532 nm rotational Raman channel.
- mask1064RROffndarray
Mask of PMT on/off status at 1064 nm rotational Raman channel.
- depCal_P_Ang_time_startlist
Time for the first profile with valid positive angle depolarization calibration.
- depCal_P_Ang_time_endlist
Time for the last profile with valid positive angle depolarization calibration.
- depCal_N_Ang_time_startlist
Time for the first profile with valid negative angle depolarization calibration.
- depCal_N_Ang_time_endlist
Time for the last profile with valid negative angle depolarization calibration.
- maskDepCalndarray
If polly was doing polarization calibration, depCalMask is set True. Otherwise, False.
- RCSndarray
Range Corrected signal [MCPS].
- PCR_slicendarray
Background corrected signal in photon count rate [MCPS].
Notes
Todo
Remove all unecesary comments.
Why does this function not take data_cube as an input like the rest of the functions?
Implement Temperature correction.
Is using mShots_norm to convert back needed?
History
2018-12-16: First edition by Zhenping.
2019-07-10: Add mask for laser shutter due to approaching airplanes.
2019-08-27: Add mask for turnoff of PMT at 607 and 387nm.
2021-01-19: Add keyword of ‘flagForceMeasTime’ to align measurement time.
2021-01-20: Re-sample the profiles into temporal resolution of 30-s.
xxxx-xx-xx: Translated to Python by …
2026-06-24: Cleaned and revamped by Buholdt.
- ppcpy.preprocess.pollyPreprocess.photonCount2PCR(signal: ndarray, mShots: ndarray, hRes: float) ndarray[source]#
Compute PCR from photon counts.
PCR = (c * signal)/(2 * hRes * mShots)
- Parameters:
- signalndarray
Signal [Photon Count] (shape: [M, N, P]).
- mShotsndarray
Mesurments shots [counts] (shape: [M, P]).
- hResfloat
Reight or range resolution [m].
- Returns:
- PCRndarray
Photon count rate signal [MCPS].
- ppcpy.preprocess.pollyPreprocess.PCR2PhotonCount(PCR: ndarray, mShots: ndarray, hRes: float) ndarray[source]#
Compute photon counts from PCR.
photonCount = 2 * PCR * mShot * hRes / c
- Parameters:
- PCRndarray
Photon count rate signal [MCPS] (shape: [M, N, P]).
- mShotsndarray
Mesurments shots [counts] (shape: [M, P]).
- hResfloat
Reight or range resolution [m].
- Returns:
- photonCountndarray
Signal [Photon Count] (shape: [M, N, P]).
- ppcpy.preprocess.pollyPreprocess.faster_polyval(p: ndarray, x: float | ndarray) float | ndarray[source]#
Faster version of np.polyval().
If p is of length N, this function returns:
y = p[N]*x**(N-1) + p[N-1]*x**(N-2) + ... + p[1]*x + p[0]
- Parameters:
- pArrayLike
Polynomial coefficient including coefficients equal to 0, from constant term to highest order term.
- xfloat or ArrayLike
Value(s) at which to evaluate the polynomial p.
- Returns:
- yfloat or ArrayLike
Polynomial p evaluated at values x.
Notes
Using numba would provide 10% increase, but with different function.
** History **
xxxx-xx-xx: first edition by
- ppcpy.preprocess.pollyPreprocess.pollyDTCor(PCR: ndarray, **varargin: dict) ndarray[source]#
Dead Time Correction.
- Parameters:
- PCRndarray
Photon count rate signal [MCPS].
- Returns:
- PCR_DTCorndarray
Dead time corrected photon count rate signal [MCPS].
Notes
** History **
2021-05-16: first edition by Zhenping
2025-02-19: edition by Cristofer Jimenez
xxxx-xx-xx: translated to python
2026-06-24: Removed signal conversion (photon count to PCR) from function
- ppcpy.preprocess.pollyPreprocess.pollyRemoveBG(rawSignal: ndarray, bgCorrectionIndexLow: list, bgCorrectionIndexHigh: list, bgCorrectionIndexLow_upper: list, bgCorrectionIndexHigh_upper: list, maxHeightBin: int = 3000, firstBinIndex: list | None = None) tuple[ndarray, ndarray][source]#
Background correction. Remove mean background noise from signal.
- Parameters:
- rawSignalndarray
Signal to be processed [MCPS or Photon counts].
- bgCorrectionIndexLowlist of int
Lower index of background noise per channel.
- bgCorrectionIndexHighlist of int
Upper index of background noise per channel.
- maxHeightBinint
Maximum height bin index. Default is 3000).
- firstBinIndexlist of int
First height bin index per channel. Default is 0 per channel.
- Returns:
- signal_outndarray
Background corrected signal [MCPs or Photon counts].
- bgndarray
Removed background noise [MCPS or Photon counts].
Notes
** History **
2021-05-16: first edition by Zhenping
xxxx-xx-xx: translated to pyhton
2025-08-13: allow channel dependent background range
- ppcpy.preprocess.pollyPreprocess.slicerange(array: ndarray, maxHeightBin: int, firstBinIndex: list) ndarray[source]#
Slice a given array across the height/range dimension from firstBinIndex to maxHeightBin + firstBinIndex.
- Parameters:
- arrayndarray
Array to be sliced.
- maxHeightBinint
Length of slice.
- firstBinIndexlist of int
Start hight/range index of slice per channel.
- Returns:
- outndarray
Sliced array.
- ** History **
- xxxx-xx-xx: first edition by …
- 2025-08-13: vectorized by Buholdt
- ppcpy.preprocess.pollyPreprocess.pollyPolCaliTime(depCalAng: ndarray, mTime: list, init_depAng: float, maskDepCalAng: list) tuple[source]#
Retrieve the time for the polly depolarization calibration period. depolarization calibration: 5 min (+45°) + 5 min (-45°) + 0.5 min.
- Parameters:
- depCalAngndarray
Angle of the polarizer in the receiving channel (>0 means calibration process starts).
- mTimelist
Datetime ndarray for the measurement time of each profile.
- init_depAngfloat
Initial polarization angle of the polarizer for polarization calibration. Default is 0.
- maskDepCalAnglist
Mask for positive and negative calibration angle of the polarizer, in which ‘p’ stands for positive angle, while ‘n’ for negative angle. Default is {}.
- Returns:
- depCal_P_Ang_time_startlist
Time for the first profile with valid positive angle depolarization calibration.
- depCal_P_Ang_time_endlist
time for the last profile with valid positive angle depolarization calibration.
- depCal_N_Ang_time_startlist
time for the first profile with valid negative angle depolarization calibration.
- depCal_N_Ang_time_endlist
time for the last profile with valid negative angle depolarization calibration.
- maskDepCalndarray
If polly was doing polarization calibration, depCalMask is set True. Otherwise, False.
Notes
Todo
Clean comments of the function.
History
2021-04-21: First edition by Zhenping
xxxx-xx-xx: Translated to Python by …
- ppcpy.preprocess.pollyPreprocess.calculate_rcs(signal: ndarray, ranges: ndarray) ndarray[source]#
Function for calculating RCS.
- Parameters:
- signalndarray
Signal to range correct [PCR].
- rangesndarray
Ranges dimension [m].
- Returns:
- RCSndarray
Range corrected signal [PCR].
Notes
** History **
xxxx-xx-xx: First edition by …
- ppcpy.preprocess.pollyPreprocess.any_signal(sig: ndarray) ndarray[source]#
Check if there is any signal POLLYISLASERSHUTTERON determine whether the laser shutter is on due to the flying object.
- Parameters:
- sig: ndarray
BGCor signal with shape [height, time].
- Returns:
- flag: ndarray
Boolean array of shape [time,] where True indicates the laser shutter is turned on.
Notes
History
2021-04-21: first edition by Zhenping
2025-05-14: translated and generalized pollyIsLaserShutterOn,
ppcpy.preprocess.profiles#
- ppcpy.preprocess.profiles.aggregate_clFreeGrps(data_cube, var: str, func=<function nansum>, flagVal: ~numpy.ndarray = None) ndarray[source]#
Aggregate the highres signal over the periods of the cloud free signal.
- Parameters:
- data_cubeobject
Main PicassoProc object.
- varstring
name of variable to be aggregated.
- funcfunction
function to do the aggregateion (mean, sum, median, etc), defult: np.nansum.
- flagValNone | np.ndarray
boolean flag for valid profiles
- Returns:
- outnp.ndarray
Aggregated highres signal for each cloud free segment.