ppcpy.interface#
ppcpy.interface.picassoProc#
- class ppcpy.interface.picassoProc.PicassoProc(rawdata_dict: dict, polly_config_dict: dict, picasso_config_dict: dict)[source]#
Picasso Processor.
This class is responsible for perfoming the processing of PollyXT data.
Initialize the PicassoProc object.
- Parameters:
- rawdata_dictdict
The dict returned by readPollyRawData.readPollyRawData(filename=rawfile).
- polly_config_dictdict
The configuration specific to the specific polly loadConfigs.loadPollyConfig(polly_config_file_fullname, polly_default_config_file).
- picasso_config_dictdict
The general picasso config loadConfigs.loadPicassoConfig(args.picasso_config_file,picasso_default_config_file).
- Attributes:
- self.rawfilestr
Path to level0-file.
- self.rawdata_dictdict
The input parameter rawdata_dict.
- self.polly_config_dictdict
The input parameter polly_config_dict.
- self.picasso_config_dictdict
The input parameter picasso_config_dict.
- self.devicestr
Name of pollyXT device.
- self.location…
Location of PollyXT device during the measurement.
- self.datestr
Measurement date.
- self.num_of_channelsint
Number of measurment channels.
- self.num_of_profilesint
Number of profiles in the measurment (time-dimension).
- self.retrievals_highresdict
Dictionary to store high resolution time, height data.
- self.retrievals_profiledict
Dictionary to store profile date.
- self.retrievals_profile[‘avail_optical_profiles’]list
Availabele optical profiles.
- self.pol_calidict
Dictionary to store polarization calibration constants.
- self.LCdict
Dictionary to store lidar calibration constants.
- self.wv_calidict
Dictionary to store water vapor calibration constants.
Notes
The polly_default_dict is not longer available as a separate variable, but is included into the polly_config_dict.
- counter = 0#
- gf(wavelength: float | int | str, meth: str, telescope: str) ndarray[source]#
Get flag shorthand.
i.e., the following two calls are equivalent
` data_cube.flag_532_total_FR data_cube.gf(532, 'total', 'FR') `where the pattern {wavelength}_{total|cross|parallel|rr}_{NR|FR|DFOV} from PollyNET/Pollynet_Processing_Chain#303 is obeyed
- Parameters:
- wavelengthfloat or int or str
Wavelength tag.
- methstr
Method type.
- telescopestr
Telescope type.
- Returns:
- ndarray
With bool flag
- check_for_correct_mshots() ndarray[source]#
Check if mshots are more than 1.1 * laser_prf * deltatime or smaller 0.
- Returns:
- condition_check_matrixndarray
Boolean array. True for elements where the condition above holds, otherwise False.
- filter_or_correct_false_mshots()[source]#
Filter or correct the mshots (currently only logging).
Todo
that might be covered via the mcps conversion –> How exactly???
- Attributes:
- self.rawdata_dictdict
With filtered or corrected enteries for timestamps where data_cube.check_for_correct_mshots()=True.
Notes
Now includes a first try for the function. I Do not know if it actually is needed yet.
Todo
Not yet crosschecked with the matlab version.
** History **
2026-06-24: Translated to python
- mdate_consistency() bool[source]#
Check mdate consistency.
- Returns:
- bool
True, if measurement date of file name equals that of in the file, otherwise False.
- reset_date_infile()[source]#
Correct the date in the file.
- Attributes:
- self.rawdata_dict[‘measurement_time’][‘var_data’]ndarray
Updated ….
Notes
Todo
Finish docstring.
- setChannelTags()[source]#
Set the channel tags.
they are stored as dictionary in
` data_cube.channel_dict `and as list in
` data_cube.retrievals_highres['channel'] data_cube.polly_config_dict['channelTags'] `as an array of boolean flags in
` data_cube.flags `as flags per channel in
` data_cube.flag_355_total_FR `- Attributes:
- self.retrievals_highres[‘channel’]list
Updated channel tags.
- self.polly_config_dict[‘channelTags’]list
updated channel tags.
- self.flag_355_total_FRndarray
True if the channel is 355nm total FR else False.
- self.flag_355_cross_FRndarray
True if the channel is 355nm cross FR else False.
- self.flag_355_parallel_FRndarray
True if the channel is 355nm parallel FR else False.
- self.flag_355_total_NRndarray
True if the channel is 355nm total NR else False.
- self.flag_387_total_FRndarray
True if the channel is 387nm total FR else False.
- self.flag_387_total_NRndarray
True if the channel is 387nm total NR else False.
- self.flag_407_total_FRndarray
True if the channel is 407nm total FR else False.
- self.flag_407_total_NRndarray
True if the channel is 407nm total NR else False.
- self.flag_532_total_FRndarray
True if the channel is 532nm total FR else False.
- self.flag_532_cross_FRndarray
True if the channel is 532nm cross FR else False.
- self.flag_532_parallel_FRndarray
True if the channel is 532nm parallel FR else False.
- self.flag_532_total_NRndarray
True if the channel is 532nm total NR else False.
- self.flag_532_cross_DFOVndarray
True if the channel is 532nm cross DFOV else False.
- self.flag_532_rr_FRndarray
True if the channel is 532nm rr FR else False.
- self.flag_607_total_FRndarray
True if the channel is 607nm total FR else False.
- self.flag_607_total_NRndarray
True if the channel is 607nm total NR else False.
- self.flag_1058_total_FRndarray
True if the channel is 1058nm total FR else False.
- self.flag_1064_total_FRndarray
True if the channel is 1064nm total FR else False.
- self.flag_1064_cross_FRndarray
True if the channel is 1064nm cross FR else False.
- self.flag_1064_total_NRndarray
True if the channel is 1064nm total NR else False.
- preprocessing(collect_debug: bool = False)[source]#
- Preprocessing of Lidar data. Includes the followin processes in order:
Deadtime correction
Background correction
First-bin shift
Mask for low-SNR
Mask bins with laser shutter on
Mask bins with fog
Mask for depolarization-calibration process
Range correction.
- Parameters:
- collect_debugbool, optional
If true, collects debug information. Default is False.
- Attributes:
- self.retrievals_highresdict
With preprocessed lidar data.
Notes
Todo
This is just a first draft for a docstring. Improve it. There is more processes and outputs of the function.
Also go over the inputs and see if all of them are actually needed.
- SaturationDetect()[source]#
Saturation Detection.
- Attributes:
- self.flagSaturationndarray
1 dimensional temporal boolean array per channel. True if the channel is saturated, else False.
- polarizationCaliD90(db_path: str = None, collect_debug: bool = False, **defaults)[source]#
Calculate/Estimate and select optimal Depol calibration constants with the Delta-90° method.
The function calculates/estimates Depol calibration constants (DCs) through the Delta-90° method. If configured, DCs from a defined database are also loaded. The optimal DC per channel is chosen as the DC with the lowest relative error from both the calculated/estimated and loaded DCs. If no DC is found for a channel its default value is taken.
- Parameters:
- db_pathstr, optional
Path to calibration constant database. If
db_pathisNoneand config variableflagUsePreviousDepolCaliis enabled, a standard path is constructed from the config variablecalibrationDB. Default isNone.- collect_debugbool, optional
If True, collects debug information. Default is False.
- polCaliEta355float, optional
Default depol calibration constant at 355 nm.
- polCaliEtaStd355float, optional
Default depol calibration constant error at 355 nm.
- polCaliEta532float, optional
Default depol calibration constant at 532 nm.
- polCaliEtaStd532float, optional
Default depol calibration constant error at 532 nm.
- polCaliEta1064float, optional
Default depol calibration constant at 1064 nm.
- polCaliEtaStd1064float, optional
Default depol calibration constant error at 1064 nm.
- Attributes:
- self.pol_cali[‘D90’ & ‘D90_db’]dict
All retrieved and read delta-90° depol calibration constants and retrieval information.
- self.pol_cali[‘default’]dict
Default depol calibration constants and retrieval information.
- self.etauseddict
The optimal depol calibration constants per channel.
Notes
The function uses the following configuration flags:
flagLCCalibration: Enables or disables calculation of Depolcalibration constants through Delta-90° method.
flagUsePreviousLC: Enables or disables loading previously determinedDepol calibration constants from the calibration database.
All calibration constants are stored in
self.pol_cali. The selected optimal constants are stored inself.etaused.Default values are by standard taken from their config variable but can be overwritten if passed as an input to this function.
The stuff that starts here in the matlab version PollyNET/Pollynet_Processing_Chain
History
xxxx-xx-xx: First edition by …
- 2026-10-08: Consider retrieved and database constants equally
and added default calibration constants.
- cloudScreen(collect_debug: bool = False)[source]#
Basic cloud screening.
- Parameters:
- collect_debugbool, optional
If true, collects debug information. Default is False.
- Attributes:
- self.flagCloudFreendarray
1 dimensional temporal boolean array. 0 = cloudy, 1 = cloud free.
Notes
Matlab equivalent code at: PollyNET/Pollynet_Processing_Chain
- genFlagValPrf()[source]#
combine the cloudFree, depCal, shutterOn and fog masks to obtain valid profiles
- cloudFreeSeg()[source]#
Cloud free profile segmentation.
- Attributes:
- self.clFreeGrpsnested list
List with start, stop index of each cloud free segement.
Notes
The matlab eqvivalent code starts here: PollyNET/Pollynet_Processing_Chain
Todo
Write a more extensive example for the docstring.
- aggregate_profiles(var: str | list = None, func=<function nansum>, flagVal: bool = None)[source]#
Aggregate highres profiles over cloud free segments.
- Parameters:
- varstr or array_like
Name(s) of variable to aggregate. Default is None.
- funcfunction, optional
Function to do the aggregation (mean, sum, median, etc.). Default is np.nansum.
- flagValNone | bool | np.ndarray
cut out single invalid profiles form clFreeGrps. If True the data_cube.flagValPrf is used. Individual array can also be supplied.
- Attributes:
- self.retrievals_profiles[`var`]ndarray
Aggregated profile.
Notes
The variable(s) var need to be stored in the dictionary data_cube.retrievals_highres. All aggregated profiles will be stored in the dictionary data_cube.retrievals_profiles under the the same key.
If var is None. The following variables will be aggregated: sigBGCor, BG, RCS, mShots, mask387Off, mask607Off, mask407Off
func=np.nansum is needed for correct SNR calculations.
** History **
xxxx-xx-xx: First edition by …
- 2026-07-28: Added option to aggregate multiple variables in the same call.
And corrected the aggregation for PCR-signals like RCS.
- loadMeteo()[source]#
Load meteorological data.
Meteorological data like; - Temperatue - Pressure - Relative Humidity - Spesific Humidity
is read form the cloudnet ECMWF file specified by the config variable; - meteorDataSource - meteo_folder - meteo_file
and saved in the dataFrame self.met.
- Attributes:
- self.metobject
Object to handle meterological data.
Notes
Only meteorDataSource = ‘nc_cloudnet’ is currently suported.
- loadAOD()[source]#
Load the AOD from a co-located fotometer.
Notes
Todo
Implement the option to load AOD from co-located fotometer.
- calcMolecular()[source]#
Calculate the molecular scattering for the cloud free periods with the strategy of first averaging the met data and then calculating the rayleigh scattering.
- Attributes:
- self.mol_profilesdict
Dictionary containing calculated molecular profiles for each channel.
Notes
Todo
How are these molecular profiles aggregated (mean or sum??).
Todo
Idea: As we already need the highres molecular signal for the WV-product. Would it not make more sens to retrieve this first and then use the aggregate_profile() function to get the cloud free profiles? –> This only makes sens if we can directly retrieve the highres molecular signal which I dout we can.
- watervaporCali(db_path: str = None, collect_debug: bool = False, **defaults)[source]#
Perform water vapor calibration on aggregated profiles.
- Parameters:
- db_pathstr, optional
Path to calibration constant database. If
db_pathisNoneand config variableflagUsePreviousWVconstis enabled, a standard path is constructed from the config variablecalibrationDB. Default isNone.- collect_debugbool, optional
If True, collects debug information. Default is False.
- wvconstfloat
Default water vapor calibration constant.
- wvconstStdfloat
Default water vapor calibration constant error.
- Attributes:
- self.wv_cali[‘model_profile’ & ‘model_profile_db’]dict
All retrieved and read water vapor calibration constants and retrieval information from model data with profile method.
- self.wv_cali[‘model_regression’ & ‘model_regression_db’]dict
All retrieved and read water vapor calibration constants and retrieval information from model data with regression method.
- self.wv_cali[‘default’]dict
Default depol calibration constants and retrieval information.
- self.WVCuseddict
The optimal depol calibration constants.
Notes
The function uses the following configuration flags:
flagWVCalibration: Enables or disables calculation of water vaporcalibration constants through Delta-90° method.
flagUsePreviousWVconst: Enables or disables loading previously determinedwater vapor calibration constants from the calibration database.
All calibration constants are stored in
self.wv_cali. The selected optimal constants are stored inself.WVCused.Default values are by standard taken from their config variable but can be overwritten if passed as an input to this function.
At the moment, only calibration results form the model regression method is considerd when selecting the optimal constant. There currently exist no option to select the best WVC over all calibration instruments and methods.
Todo
How to select best WVC over all calibration instruments and methods.
History
xxxx-xx-xx: First edition by …
2026-09-11: Implemented in python by Jakob.
- rayleighFit(collect_debug: bool = False)[source]#
Perform the rayleigh fit procedure.
- Parameters:
- collect_debugbool, optional
If true, collects debug information. Default is False.
- Attributes:
- self.retrievals_profiles[‘refH’][idx][ch]dict
Reference height information for cloud free segment idx and channel ch.
Notes
Direct translation from the matlab code. There might be noticeable numerical discrepancies (especially in the residual) seemed to work ok for 532, 1064, but with issues for 355.
The Douglas Peucker algorithm works fine (gives the same result as the Matlab version). However, due to the numerical discrepancies In the residuals of the fit algorithm sometimes different refH segments are chosen.
- polarizationCaliMol()[source]#
Calibration with molecular signal in reference height.
- Attributes:
- self.pol_cali[‘mol’][ch][idx]dict
Molecular depol calibration constants and retrieval information for channel ch and cloud free index idx.
Notes
Calibration is only performed if the config variable flagMolDepolCali = True.
- transCor()[source]#
Perform GHK-transmission correction on the signal.
- Attributes:
- self.retrievals_highresdict
With corrected signal.
Notes
if the config entry flagTransCor = False, no GHK transmission correction will be done and the background corrected signal will replace the transmission corrected signal for the rest of the processing steps.
Todo
Overwriting sigTCor with sigBGCor when flagTransCor = False can cause confiusion among useres. It would be more intutive to not have a sigTCor variable in this case. However, this requiers re… the rest of the processing chain.
- retrievalKlett(oc: bool = False, nr: bool = False)[source]#
Apply Klett retrieval.
- Parameters:
- ocbool, optional
If true, apply the retrieval on the overlap corrected profiles. The outupt will then also be saved under the key ‘klett_OC’ insted of ‘klett’. Default is False.
- nrbool, optional
If true, apply the retrieval on the near range profiles as well as the far range profiles. Default is False.
- Attributes:
- self.retrievals_profile[‘klett’ or ‘klett_OC’][idx][ch]dict
Klett retrieved optical profiles and retrieval information for cloud free segment idx and channel ch.
Notes
The retrieval is by default applied on the FR profiles. if nr=True the retrieval will be applied both on the NR and FR profiles.
- retrievalRaman(oc: bool = False, nr: bool = False, collect_debug: bool = False)[source]#
Apply raman retrieval (nighttime only).
- Parameters:
- ocbool, optional
If true, apply the retrieval on the overlap corrected profiles. The outupt will then also be saved under the key ‘raman_OC’ insted of ‘raman’. Default is False.
- nrbool, optional
If true, apply the retrieval on the near range profiles as well as the far range profiles. Default is False.
- collect_debugbool, optional
If true, collects debug information. Default is False.
- Attributes:
- self.retrievals_profile[‘raman’ or ‘raman_OC’][idx][ch]dict
Raman retrieved optical profiles and retrieval information for cloud free segment idx and channel ch.
Notes
The retrieval is by default applied on the FR profiles. if nr=True the retrieval will be applied both on the NR and FR profiles.
- overlapCalc(collect_debug: bool = False)[source]#
Estimate the overlap function.
- Parameters:
- collect_debugstr, optional
If true, collects debug information. Default is False.
- Attributes:
- self.retrievals_profile[‘overlap’][‘frnr’][idx][ch]list of dicts
Far-range-Near-range retrieved overlap function and retrieval information for cloud free segment idx and channel ch.
Notes
Different to the matlab version, where an average over all cloud free periods is taken. Here the overlap is applied per cloud free period.
- overlapFixLowestBins()[source]#
The lowest bins are affected by stange near range effects.
- Attributes:
- self.retrievals_profile[‘overlap’][‘frnr’][idx][ch][‘olFunc’]ndarray
FRNR retrieved overlap func with subdued near-range effect.
- self.retrievals_profile[‘overlap’][‘raman’][idx][ch][‘olFunc’]ndarray
Raman retrieved overlap func with subdued near-range effect.
- overlapCor()[source]#
Overlap correction.
- Attributes:
- self.retrieval_highresdict
With overlap corrected signal and retrieval information.
Notes
the overlap correction is implemented differently to the matlab version first a 2d (time, height) correction array is constructed then it is applied. In future this will allow for time varing overlap functions
- calcDepol()[source]#
Calculate the volume depol and the particle depol.
- Attributes:
- self.retrievals_profiles[sig][idx][ch]dict
Volume and particle depolarization profiles and retrieval info for signal type sig for cloud free segment idx and channel ch.
Notes
This retrieval is done for all avaiable optical profiles.
- estQualityMask()[source]#
Estimate the quality mask.
- Attributes:
- self.retrievals_highres[‘quality_mask’]ndarray
- High resolution (time, height) quality masks per channel.
0 : good data 1 : low-SNR data 2 : depolarization calibration periods 3 : shutter on 4 : fog 5 : saturated (NEW)
Notes
If the config variables flagUseImprovedSNR=True, SNR and the lowSNRMask are recalculated with quasi-smoothed signals to improve data quality.
** History **
xxxx-xx-xx: First edition by …
2026-07-17: Added quasi-smoothed SNR and lowSNRMask.
- Angstroem()[source]#
Calculate the angstrom exponent.
- Attributes:
- self.retrievals_profiles[sig][idx][ch]dict
Volume and particle depolarization profiles and retrieval info for signal type sig for cloud free segment idx and channel ch.
Notes
AE_{prod}_{wv1}_{wv2} = log(prod_wv1 / prod_wv2) / log(wv2 / wv1)
This retrieval is done for all avaiable optical profiles.
- LidarCalibration(db_path: str = None, collect_debug: bool = False, **defaults)[source]#
Calculate/Estimate and select optimal lidar calibration constants.
The function calculates/estimates lidar calibration constants (LCs) from both klett and Raman retrieved optical profiles. If configured, LCs from a defined database are also loaded. The optimal LC per channel is chosen as the LC with the lowest relative error per channel form both the calculated/estimated and loaded LCs. If no LC is found for a channel its default value is taken.
- Parameters:
- db_pathstr or Path, optional
Path to calibration constant database. If
db_pathisNoneand config variableflagUsePreviousLCis enabled, a standard path is constructed from the config variablecalibrationDB. Default isNone.- collect_debugbool, optional
If
True, collects debug information. Default isFalse.- LClist, optional
Default Lidar constant value per channel.
- LCStdlist, optional
Default Lidar constant error per channel.
- Attributes:
- self.LC[‘klett’ & ‘klett_db’]dict
All retrieved and read Klett lidar calibration constants and retrieval information.
- self.LC[‘raman’ & ‘raman_db’]dict
All retrieved and read Raman lidar calibration constants and retrieval information.
- self.LC[‘default’]dict
Default lidar calibration constants and retrieval information.
- self.LCuseddict
The optimal lidar calibration constant per channel.
Notes
The function uses the following configuration flags:
flagLCCalibration: Enables or disables calculation of Lidarcalibration constants from klett and Raman retrieval.
flagUsePreviousLC: Enables or disables loading previously determinedlidar calibration constants from the calibration database.
All calibration constants are stored in
self.LC. The selected optimal constants are stored inself.LCused.Default values are by standard taken from their config variable but can be overwritten if passed as an input to this function.
History
xxxx-xx-xx: First edition by …
- 2026-10-08: Consider retrieved and database constants equally
and added default calibration constants.
- attBsc_volDepol()[source]#
High resolution attenuated backscatter and volume depolarization.
- Attributes:
- self.retrievals_highresdict
With attenuated backscatter.
- molecularHighres()[source]#
Calculate the molecular signal for the 2d high resolution.
- Attributes:
- self.mol_2dxr.Dataset
xarray dataset with dimensions time and height and variables molecular backscatter (mBsc) and molecular extinction (mExt) per wavelength.
Notes
mol_2d’s time dimension differe from the time dimension of the measurement.
- wvmr()[source]#
High resolution water vapor mixing ratio.
- Attributes:
- self.retrieval_highresdict
With water vapor mixing rato.
- quasiV1()[source]#
QuasiV1 retrievals and target categorisation.
- Attributes:
- self.retrieval_highresdict
With high resolution Quasi version 1 products as well as Target categorization.
Notes
QuasiV1 products uses Klett-retrieved products.
- quasiV2()[source]#
QuasiV2 retrievals and target categorisation.
- Attributes:
- self.retrieval_highresdict
With high resolution Quasi version 2 products as well as Target categorization.
Notes
QuasiV2 products uses Raman-retrieved products.
- write_2_sql_db(parameter: str, db_path: str | None = None, method: str | None = None)[source]#
Write LC, eta or WVC to sqlite db table.
- Parameters:
- parameterstr
can be LC (Lidar-calibration-constant), DC (Depol-calibration-constant) or WVC (WaterVapor-calibration-constant)
- methodstr
‘raman’, ‘klett’, ‘D90’, ‘transfer’, ‘profile’ or ‘regression’
- db_pathstr
location of the sqlite db-file
Notes
The unique columns are needed that new entries overwrite old ones, otherwise they are just added to the table with same timestamps.
- read_calibration_db(db_path: str | None = None)[source]#
Read the calibration constants from database.
- Parameters:
- db_pathstr or NoneType
path to database file… Default is None.
- Attributes:
- self.LCdict
Original LC with all additinal lidar calibration constant availabel in the datafram db_path in the time period of the measurement +-24h.
- self.pol_calidict
Original pol_cali with all additinal polarization calibration constant availabel in the datafram db_path in the time period of the measurement +-24h.
Notes
Time interval includes 24h before start of the measurement and after the end of the measurement.