ppcpy.preprocess#

ppcpy.preprocess.pollyPreprocess#

ppcpy.preprocess.pollyPreprocess.pollyPreprocess(rawdata_dict: dict, collect_debug: bool = False, flagPicassoComparison: bool = False, **param: dict) → dict[source]#

Preprocessing of Lidar-data.

Includes the following processes in order:
  1. Deadtime correction

  2. Background correction

  3. First-bin shift

  4. Mask for low-SNR

  5. Mask for depolarization-calibration process

  6. Range correction.

Parameters:
rawdata_dictdict
rawSignalndarray

Signal [Photon Count].

mShotsndarray

Number of the laser shots for each profile.

mTimendarray

Datetime array for the measurement time of each profile.

depCalAngndarray

Angle of the polarizer in the receiving channel (>0 means calibration process starts).

zenithAngndarray

Zenith angle of the laer beam.

repRatefloat

Laser pulse repetition rate [s^-1].

hResfloat

Spatial resolution [m].

mSitestr

Measurement site.

collect_debugbool

If true, collects debug information. Default is False.

flagPicassoComparisonbool

If true, use Picasso values and logic.

Returns:
data_dictdict
mShotsndarray

Number of the laser shots for each profile.

sigDTCorndarray

Dead time corrected signal [photon count].

BGndarray

Background.

sigBGCorndarray

Backgound corrected signal [photon count].

rangendarray

Height above ground at zenith angle [m].

heightndarray

Height above ground [m].

altndarray

Altitude [m].

timelist

Measurement time in unix time.

time64ndarray

Measurement time in numpy.datetime.

SNRndarray

Signal to noise ratio.

lowSNRMaskndarray

True if SNR is less SNRmin. Otherwise False.

shutterOnMaskndarray

True if at timesteps where the shutters was on. Otherwise False.

fogMaskndarray

If it is foggy which means the signal will be very weak, fogMask will be set True. Otherwise, False.

mask607Offndarray

Mask of PMT on/off status at 607 nm channel.

mask387Offndarray

Mask of PMT on/off status at 387 nm channel.

mask407Offndarray

Mask of PMT on/off status at 407 nm channel.

mask355RROffndarray

Mask of PMT on/off status at 355 nm rotational Raman channel.

mask532RROffndarray

Mask of PMT on/off status at 532 nm rotational Raman channel.

mask1064RROffndarray

Mask of PMT on/off status at 1064 nm rotational Raman channel.

depCal_P_Ang_time_startlist

Time for the first profile with valid positive angle depolarization calibration.

depCal_P_Ang_time_endlist

Time for the last profile with valid positive angle depolarization calibration.

depCal_N_Ang_time_startlist

Time for the first profile with valid negative angle depolarization calibration.

depCal_N_Ang_time_endlist

Time for the last profile with valid negative angle depolarization calibration.

maskDepCalndarray

If polly was doing polarization calibration, depCalMask is set True. Otherwise, False.

RCSndarray

Range Corrected signal [MCPS].

PCR_slicendarray

Background corrected signal in photon count rate [MCPS].

Notes

Todo

  • Remove all unecesary comments.

  • Why does this function not take data_cube as an input like the rest of the functions?

  • Implement Temperature correction.

  • Is using mShots_norm to convert back needed?

History

  • 2018-12-16: First edition by Zhenping.

  • 2019-07-10: Add mask for laser shutter due to approaching airplanes.

  • 2019-08-27: Add mask for turnoff of PMT at 607 and 387nm.

  • 2021-01-19: Add keyword of ‘flagForceMeasTime’ to align measurement time.

  • 2021-01-20: Re-sample the profiles into temporal resolution of 30-s.

  • xxxx-xx-xx: Translated to Python by …

  • 2026-06-24: Cleaned and revamped by Buholdt.

ppcpy.preprocess.pollyPreprocess.photonCount2PCR(signal: ndarray, mShots: ndarray, hRes: float) → ndarray[source]#

Compute PCR from photon counts.

PCR = (c * signal)/(2 * hRes * mShots)

Parameters:
signalndarray

Signal [Photon Count] (shape: [M, N, P]).

mShotsndarray

Mesurments shots [counts] (shape: [M, P]).

hResfloat

Reight or range resolution [m].

Returns:
PCRndarray

Photon count rate signal [MCPS].

ppcpy.preprocess.pollyPreprocess.PCR2PhotonCount(PCR: ndarray, mShots: ndarray, hRes: float) → ndarray[source]#

Compute photon counts from PCR.

photonCount = 2 * PCR * mShot * hRes / c

Parameters:
PCRndarray

Photon count rate signal [MCPS] (shape: [M, N, P]).

mShotsndarray

Mesurments shots [counts] (shape: [M, P]).

hResfloat

Reight or range resolution [m].

Returns:
photonCountndarray

Signal [Photon Count] (shape: [M, N, P]).

ppcpy.preprocess.pollyPreprocess.faster_polyval(p: ndarray, x: float | ndarray) → float | ndarray[source]#

Faster version of np.polyval().

If p is of length N, this function returns:

y = p[N]*x**(N-1) + p[N-1]*x**(N-2) + ... + p[1]*x + p[0]
Parameters:
pArrayLike

Polynomial coefficient including coefficients equal to 0, from constant term to highest order term.

xfloat or ArrayLike

Value(s) at which to evaluate the polynomial p.

Returns:
yfloat or ArrayLike

Polynomial p evaluated at values x.

Notes

  • Using numba would provide 10% increase, but with different function.

** History **

  • xxxx-xx-xx: first edition by

ppcpy.preprocess.pollyPreprocess.pollyDTCor(PCR: ndarray, **varargin: dict) → ndarray[source]#

Dead Time Correction.

Parameters:
PCRndarray

Photon count rate signal [MCPS].

Returns:
PCR_DTCorndarray

Dead time corrected photon count rate signal [MCPS].

Notes

** History **

  • 2021-05-16: first edition by Zhenping

  • 2025-02-19: edition by Cristofer Jimenez

  • xxxx-xx-xx: translated to python

  • 2026-06-24: Removed signal conversion (photon count to PCR) from function

ppcpy.preprocess.pollyPreprocess.pollyRemoveBG(rawSignal: ndarray, bgCorrectionIndexLow: list, bgCorrectionIndexHigh: list, bgCorrectionIndexLow_upper: list, bgCorrectionIndexHigh_upper: list, maxHeightBin: int = 3000, firstBinIndex: list | None = None) → tuple[ndarray, ndarray][source]#

Background correction. Remove mean background noise from signal.

Parameters:
rawSignalndarray

Signal to be processed [MCPS or Photon counts].

bgCorrectionIndexLowlist of int

Lower index of background noise per channel.

bgCorrectionIndexHighlist of int

Upper index of background noise per channel.

maxHeightBinint

Maximum height bin index. Default is 3000).

firstBinIndexlist of int

First height bin index per channel. Default is 0 per channel.

Returns:
signal_outndarray

Background corrected signal [MCPs or Photon counts].

bgndarray

Removed background noise [MCPS or Photon counts].

Notes

** History **

  • 2021-05-16: first edition by Zhenping

  • xxxx-xx-xx: translated to pyhton

  • 2025-08-13: allow channel dependent background range

ppcpy.preprocess.pollyPreprocess.slicerange(array: ndarray, maxHeightBin: int, firstBinIndex: list) → ndarray[source]#

Slice a given array across the height/range dimension from firstBinIndex to maxHeightBin + firstBinIndex.

Parameters:
arrayndarray

Array to be sliced.

maxHeightBinint

Length of slice.

firstBinIndexlist of int

Start hight/range index of slice per channel.

Returns:
outndarray

Sliced array.

** History **
  • xxxx-xx-xx: first edition by …
  • 2025-08-13: vectorized by Buholdt
ppcpy.preprocess.pollyPreprocess.pollyPolCaliTime(depCalAng: ndarray, mTime: list, init_depAng: float, maskDepCalAng: list) → tuple[source]#

Retrieve the time for the polly depolarization calibration period. depolarization calibration: 5 min (+45°) + 5 min (-45°) + 0.5 min.

Parameters:
depCalAngndarray

Angle of the polarizer in the receiving channel (>0 means calibration process starts).

mTimelist

Datetime ndarray for the measurement time of each profile.

init_depAngfloat

Initial polarization angle of the polarizer for polarization calibration. Default is 0.

maskDepCalAnglist

Mask for positive and negative calibration angle of the polarizer, in which ‘p’ stands for positive angle, while ‘n’ for negative angle. Default is {}.

Returns:
depCal_P_Ang_time_startlist

Time for the first profile with valid positive angle depolarization calibration.

depCal_P_Ang_time_endlist

time for the last profile with valid positive angle depolarization calibration.

depCal_N_Ang_time_startlist

time for the first profile with valid negative angle depolarization calibration.

depCal_N_Ang_time_endlist

time for the last profile with valid negative angle depolarization calibration.

maskDepCalndarray

If polly was doing polarization calibration, depCalMask is set True. Otherwise, False.

Notes

Todo

Clean comments of the function.

History

  • 2021-04-21: First edition by Zhenping

  • xxxx-xx-xx: Translated to Python by …

ppcpy.preprocess.pollyPreprocess.calculate_rcs(signal: ndarray, ranges: ndarray) → ndarray[source]#

Function for calculating RCS.

Parameters:
signalndarray

Signal to range correct [PCR].

rangesndarray

Ranges dimension [m].

Returns:
RCSndarray

Range corrected signal [PCR].

Notes

** History **

  • xxxx-xx-xx: First edition by …

ppcpy.preprocess.pollyPreprocess.any_signal(sig: ndarray) → ndarray[source]#

Check if there is any signal POLLYISLASERSHUTTERON determine whether the laser shutter is on due to the flying object.

Parameters:
sig: ndarray

BGCor signal with shape [height, time].

Returns:
flag: ndarray

Boolean array of shape [time,] where True indicates the laser shutter is turned on.

Notes

History

  • 2021-04-21: first edition by Zhenping

  • 2025-05-14: translated and generalized pollyIsLaserShutterOn,

ppcpy.preprocess.profiles#

ppcpy.preprocess.profiles.aggregate_clFreeGrps(data_cube, var: str, func=<function nansum>, flagVal: ~numpy.ndarray = None) → ndarray[source]#

Aggregate the highres signal over the periods of the cloud free signal.

Parameters:
data_cubeobject

Main PicassoProc object.

varstring

name of variable to be aggregated.

funcfunction

function to do the aggregateion (mean, sum, median, etc), defult: np.nansum.

flagValNone | np.ndarray

boolean flag for valid profiles

Returns:
outnp.ndarray

Aggregated highres signal for each cloud free segment.