ppcpy.interface#

ppcpy.interface.picassoProc#

class ppcpy.interface.picassoProc.PicassoProc(rawdata_dict: dict, polly_config_dict: dict, picasso_config_dict: dict)[source]#

Picasso Processor.

This class is responsible for perfoming the processing of PollyXT data.

Initialize the PicassoProc object.

Parameters:
rawdata_dictdict

The dict returned by readPollyRawData.readPollyRawData(filename=rawfile).

polly_config_dictdict

The configuration specific to the specific polly loadConfigs.loadPollyConfig(polly_config_file_fullname, polly_default_config_file).

picasso_config_dictdict

The general picasso config loadConfigs.loadPicassoConfig(args.picasso_config_file,picasso_default_config_file).

Attributes:
self.rawfilestr

Path to level0-file.

self.rawdata_dictdict

The input parameter rawdata_dict.

self.polly_config_dictdict

The input parameter polly_config_dict.

self.picasso_config_dictdict

The input parameter picasso_config_dict.

self.devicestr

Name of pollyXT device.

self.location…

Location of PollyXT device during the measurement.

self.datestr

Measurement date.

self.num_of_channelsint

Number of measurment channels.

self.num_of_profilesint

Number of profiles in the measurment (time-dimension).

self.retrievals_highresdict

Dictionary to store high resolution time, height data.

self.retrievals_profiledict

Dictionary to store profile date.

self.retrievals_profile[‘avail_optical_profiles’]list

Availabele optical profiles.

self.pol_calidict

Dictionary to store polarization calibration constants.

self.LCdict

Dictionary to store lidar calibration constants.

self.wv_calidict

Dictionary to store water vapor calibration constants.

Notes

  • The polly_default_dict is not longer available as a separate variable, but is included into the polly_config_dict.

counter = 0#
mdate_filename() → str[source]#

Get the date from filename in YYYYMMDD.

Returns:
str

Measurement date.

gf(wavelength: float | int | str, meth: str, telescope: str) → ndarray[source]#

Get flag shorthand.

i.e., the following two calls are equivalent ` data_cube.flag_532_total_FR data_cube.gf(532, 'total', 'FR') `

where the pattern {wavelength}_{total|cross|parallel|rr}_{NR|FR|DFOV} from PollyNET/Pollynet_Processing_Chain#303 is obeyed

Parameters:
wavelengthfloat or int or str

Wavelength tag.

methstr

Method type.

telescopestr

Telescope type.

Returns:
ndarray

With bool flag

mdate_infile() → str[source]#

First date in file as string.

Returns:
str

Measurement date.

check_for_correct_mshots() → ndarray[source]#

Check if mshots are more than 1.1 * laser_prf * deltatime or smaller 0.

Returns:
condition_check_matrixndarray

Boolean array. True for elements where the condition above holds, otherwise False.

filter_or_correct_false_mshots()[source]#

Filter or correct the mshots (currently only logging).

Todo

that might be covered via the mcps conversion –> How exactly???

Attributes:
self.rawdata_dictdict

With filtered or corrected enteries for timestamps where data_cube.check_for_correct_mshots()=True.

Notes

  • Now includes a first try for the function. I Do not know if it actually is needed yet.

Todo

Not yet crosschecked with the matlab version.

** History **

  • 2026-06-24: Translated to python

mdate_consistency() → bool[source]#

Check mdate consistency.

Returns:
bool

True, if measurement date of file name equals that of in the file, otherwise False.

reset_date_infile()[source]#

Correct the date in the file.

Attributes:
self.rawdata_dict[‘measurement_time’][‘var_data’]ndarray

Updated ….

Notes

Todo

Finish docstring.

setChannelTags()[source]#

Set the channel tags.

they are stored as dictionary in ` data_cube.channel_dict `

and as list in ` data_cube.retrievals_highres['channel'] data_cube.polly_config_dict['channelTags'] `

as an array of boolean flags in ` data_cube.flags `

as flags per channel in ` data_cube.flag_355_total_FR `

Attributes:
self.retrievals_highres[‘channel’]list

Updated channel tags.

self.polly_config_dict[‘channelTags’]list

updated channel tags.

self.flag_355_total_FRndarray

True if the channel is 355nm total FR else False.

self.flag_355_cross_FRndarray

True if the channel is 355nm cross FR else False.

self.flag_355_parallel_FRndarray

True if the channel is 355nm parallel FR else False.

self.flag_355_total_NRndarray

True if the channel is 355nm total NR else False.

self.flag_387_total_FRndarray

True if the channel is 387nm total FR else False.

self.flag_387_total_NRndarray

True if the channel is 387nm total NR else False.

self.flag_407_total_FRndarray

True if the channel is 407nm total FR else False.

self.flag_407_total_NRndarray

True if the channel is 407nm total NR else False.

self.flag_532_total_FRndarray

True if the channel is 532nm total FR else False.

self.flag_532_cross_FRndarray

True if the channel is 532nm cross FR else False.

self.flag_532_parallel_FRndarray

True if the channel is 532nm parallel FR else False.

self.flag_532_total_NRndarray

True if the channel is 532nm total NR else False.

self.flag_532_cross_DFOVndarray

True if the channel is 532nm cross DFOV else False.

self.flag_532_rr_FRndarray

True if the channel is 532nm rr FR else False.

self.flag_607_total_FRndarray

True if the channel is 607nm total FR else False.

self.flag_607_total_NRndarray

True if the channel is 607nm total NR else False.

self.flag_1058_total_FRndarray

True if the channel is 1058nm total FR else False.

self.flag_1064_total_FRndarray

True if the channel is 1064nm total FR else False.

self.flag_1064_cross_FRndarray

True if the channel is 1064nm cross FR else False.

self.flag_1064_total_NRndarray

True if the channel is 1064nm total NR else False.

preprocessing(collect_debug: bool = False)[source]#
Preprocessing of Lidar data. Includes the followin processes in order:
  1. Deadtime correction

  2. Background correction

  3. First-bin shift

  4. Mask for low-SNR

  5. Mask bins with laser shutter on

  6. Mask bins with fog

  7. Mask for depolarization-calibration process

  8. Range correction.

Parameters:
collect_debugbool, optional

If true, collects debug information. Default is False.

Attributes:
self.retrievals_highresdict

With preprocessed lidar data.

Notes

Todo

  • This is just a first draft for a docstring. Improve it. There is more processes and outputs of the function.

  • Also go over the inputs and see if all of them are actually needed.

SaturationDetect()[source]#

Saturation Detection.

Attributes:
self.flagSaturationndarray

1 dimensional temporal boolean array per channel. True if the channel is saturated, else False.

polarizationCaliD90(db_path: str = None, collect_debug: bool = False, **defaults)[source]#

Calculate/Estimate and select optimal Depol calibration constants with the Delta-90° method.

The function calculates/estimates Depol calibration constants (DCs) through the Delta-90° method. If configured, DCs from a defined database are also loaded. The optimal DC per channel is chosen as the DC with the lowest relative error from both the calculated/estimated and loaded DCs. If no DC is found for a channel its default value is taken.

Parameters:
db_pathstr, optional

Path to calibration constant database. If db_path is None and config variable flagUsePreviousDepolCali is enabled, a standard path is constructed from the config variable calibrationDB. Default is None.

collect_debugbool, optional

If True, collects debug information. Default is False.

polCaliEta355float, optional

Default depol calibration constant at 355 nm.

polCaliEtaStd355float, optional

Default depol calibration constant error at 355 nm.

polCaliEta532float, optional

Default depol calibration constant at 532 nm.

polCaliEtaStd532float, optional

Default depol calibration constant error at 532 nm.

polCaliEta1064float, optional

Default depol calibration constant at 1064 nm.

polCaliEtaStd1064float, optional

Default depol calibration constant error at 1064 nm.

Attributes:
self.pol_cali[‘D90’ & ‘D90_db’]dict

All retrieved and read delta-90° depol calibration constants and retrieval information.

self.pol_cali[‘default’]dict

Default depol calibration constants and retrieval information.

self.etauseddict

The optimal depol calibration constants per channel.

Notes

The function uses the following configuration flags:

  • flagLCCalibration: Enables or disables calculation of Depol

    calibration constants through Delta-90° method.

  • flagUsePreviousLC: Enables or disables loading previously determined

    Depol calibration constants from the calibration database.

All calibration constants are stored in self.pol_cali. The selected optimal constants are stored in self.etaused.

Default values are by standard taken from their config variable but can be overwritten if passed as an input to this function.

The stuff that starts here in the matlab version PollyNET/Pollynet_Processing_Chain

History

  • xxxx-xx-xx: First edition by …

  • 2026-10-08: Consider retrieved and database constants equally

    and added default calibration constants.

cloudScreen(collect_debug: bool = False)[source]#

Basic cloud screening.

Parameters:
collect_debugbool, optional

If true, collects debug information. Default is False.

Attributes:
self.flagCloudFreendarray

1 dimensional temporal boolean array. 0 = cloudy, 1 = cloud free.

Notes

genFlagValPrf()[source]#

combine the cloudFree, depCal, shutterOn and fog masks to obtain valid profiles

cloudFreeSeg()[source]#

Cloud free profile segmentation.

Attributes:
self.clFreeGrpsnested list

List with start, stop index of each cloud free segement.

Notes

Todo

Write a more extensive example for the docstring.

aggregate_profiles(var: str | list = None, func=<function nansum>, flagVal: bool = None)[source]#

Aggregate highres profiles over cloud free segments.

Parameters:
varstr or array_like

Name(s) of variable to aggregate. Default is None.

funcfunction, optional

Function to do the aggregation (mean, sum, median, etc.). Default is np.nansum.

flagValNone | bool | np.ndarray

cut out single invalid profiles form clFreeGrps. If True the data_cube.flagValPrf is used. Individual array can also be supplied.

Attributes:
self.retrievals_profiles[`var`]ndarray

Aggregated profile.

Notes

  • The variable(s) var need to be stored in the dictionary data_cube.retrievals_highres. All aggregated profiles will be stored in the dictionary data_cube.retrievals_profiles under the the same key.

  • If var is None. The following variables will be aggregated: sigBGCor, BG, RCS, mShots, mask387Off, mask607Off, mask407Off

  • func=np.nansum is needed for correct SNR calculations.

** History **

  • xxxx-xx-xx: First edition by …

  • 2026-07-28: Added option to aggregate multiple variables in the same call.

    And corrected the aggregation for PCR-signals like RCS.

loadMeteo()[source]#

Load meteorological data.

Meteorological data like; - Temperatue - Pressure - Relative Humidity - Spesific Humidity

is read form the cloudnet ECMWF file specified by the config variable; - meteorDataSource - meteo_folder - meteo_file

and saved in the dataFrame self.met.

Attributes:
self.metobject

Object to handle meterological data.

Notes

  • Only meteorDataSource = ‘nc_cloudnet’ is currently suported.

loadAOD()[source]#

Load the AOD from a co-located fotometer.

Notes

Todo

Implement the option to load AOD from co-located fotometer.

calcMolecular()[source]#

Calculate the molecular scattering for the cloud free periods with the strategy of first averaging the met data and then calculating the rayleigh scattering.

Attributes:
self.mol_profilesdict

Dictionary containing calculated molecular profiles for each channel.

Notes

Todo

How are these molecular profiles aggregated (mean or sum??).

Todo

Idea: As we already need the highres molecular signal for the WV-product. Would it not make more sens to retrieve this first and then use the aggregate_profile() function to get the cloud free profiles? –> This only makes sens if we can directly retrieve the highres molecular signal which I dout we can.

watervaporCali(db_path: str = None, collect_debug: bool = False, **defaults)[source]#

Perform water vapor calibration on aggregated profiles.

Parameters:
db_pathstr, optional

Path to calibration constant database. If db_path is None and config variable flagUsePreviousWVconst is enabled, a standard path is constructed from the config variable calibrationDB. Default is None.

collect_debugbool, optional

If True, collects debug information. Default is False.

wvconstfloat

Default water vapor calibration constant.

wvconstStdfloat

Default water vapor calibration constant error.

Attributes:
self.wv_cali[‘model_profile’ & ‘model_profile_db’]dict

All retrieved and read water vapor calibration constants and retrieval information from model data with profile method.

self.wv_cali[‘model_regression’ & ‘model_regression_db’]dict

All retrieved and read water vapor calibration constants and retrieval information from model data with regression method.

self.wv_cali[‘default’]dict

Default depol calibration constants and retrieval information.

self.WVCuseddict

The optimal depol calibration constants.

Notes

The function uses the following configuration flags:

  • flagWVCalibration: Enables or disables calculation of water vapor

    calibration constants through Delta-90° method.

  • flagUsePreviousWVconst: Enables or disables loading previously determined

    water vapor calibration constants from the calibration database.

All calibration constants are stored in self.wv_cali. The selected optimal constants are stored in self.WVCused.

Default values are by standard taken from their config variable but can be overwritten if passed as an input to this function.

At the moment, only calibration results form the model regression method is considerd when selecting the optimal constant. There currently exist no option to select the best WVC over all calibration instruments and methods.

Todo

How to select best WVC over all calibration instruments and methods.

History

  • xxxx-xx-xx: First edition by …

  • 2026-09-11: Implemented in python by Jakob.

rayleighFit(collect_debug: bool = False)[source]#

Perform the rayleigh fit procedure.

Parameters:
collect_debugbool, optional

If true, collects debug information. Default is False.

Attributes:
self.retrievals_profiles[‘refH’][idx][ch]dict

Reference height information for cloud free segment idx and channel ch.

Notes

  • Direct translation from the matlab code. There might be noticeable numerical discrepancies (especially in the residual) seemed to work ok for 532, 1064, but with issues for 355.

  • The Douglas Peucker algorithm works fine (gives the same result as the Matlab version). However, due to the numerical discrepancies In the residuals of the fit algorithm sometimes different refH segments are chosen.

polarizationCaliMol()[source]#

Calibration with molecular signal in reference height.

Attributes:
self.pol_cali[‘mol’][ch][idx]dict

Molecular depol calibration constants and retrieval information for channel ch and cloud free index idx.

Notes

  • Calibration is only performed if the config variable flagMolDepolCali = True.

transCor()[source]#

Perform GHK-transmission correction on the signal.

Attributes:
self.retrievals_highresdict

With corrected signal.

Notes

  • if the config entry flagTransCor = False, no GHK transmission correction will be done and the background corrected signal will replace the transmission corrected signal for the rest of the processing steps.

Todo

Overwriting sigTCor with sigBGCor when flagTransCor = False can cause confiusion among useres. It would be more intutive to not have a sigTCor variable in this case. However, this requiers re… the rest of the processing chain.

retrievalKlett(oc: bool = False, nr: bool = False)[source]#

Apply Klett retrieval.

Parameters:
ocbool, optional

If true, apply the retrieval on the overlap corrected profiles. The outupt will then also be saved under the key ‘klett_OC’ insted of ‘klett’. Default is False.

nrbool, optional

If true, apply the retrieval on the near range profiles as well as the far range profiles. Default is False.

Attributes:
self.retrievals_profile[‘klett’ or ‘klett_OC’][idx][ch]dict

Klett retrieved optical profiles and retrieval information for cloud free segment idx and channel ch.

Notes

  • The retrieval is by default applied on the FR profiles. if nr=True the retrieval will be applied both on the NR and FR profiles.

retrievalRaman(oc: bool = False, nr: bool = False, collect_debug: bool = False)[source]#

Apply raman retrieval (nighttime only).

Parameters:
ocbool, optional

If true, apply the retrieval on the overlap corrected profiles. The outupt will then also be saved under the key ‘raman_OC’ insted of ‘raman’. Default is False.

nrbool, optional

If true, apply the retrieval on the near range profiles as well as the far range profiles. Default is False.

collect_debugbool, optional

If true, collects debug information. Default is False.

Attributes:
self.retrievals_profile[‘raman’ or ‘raman_OC’][idx][ch]dict

Raman retrieved optical profiles and retrieval information for cloud free segment idx and channel ch.

Notes

  • The retrieval is by default applied on the FR profiles. if nr=True the retrieval will be applied both on the NR and FR profiles.

overlapCalc(collect_debug: bool = False)[source]#

Estimate the overlap function.

Parameters:
collect_debugstr, optional

If true, collects debug information. Default is False.

Attributes:
self.retrievals_profile[‘overlap’][‘frnr’][idx][ch]list of dicts

Far-range-Near-range retrieved overlap function and retrieval information for cloud free segment idx and channel ch.

Notes

  • Different to the matlab version, where an average over all cloud free periods is taken. Here the overlap is applied per cloud free period.

overlapFixLowestBins()[source]#

The lowest bins are affected by stange near range effects.

Attributes:
self.retrievals_profile[‘overlap’][‘frnr’][idx][ch][‘olFunc’]ndarray

FRNR retrieved overlap func with subdued near-range effect.

self.retrievals_profile[‘overlap’][‘raman’][idx][ch][‘olFunc’]ndarray

Raman retrieved overlap func with subdued near-range effect.

overlapCor()[source]#

Overlap correction.

Attributes:
self.retrieval_highresdict

With overlap corrected signal and retrieval information.

Notes

  • the overlap correction is implemented differently to the matlab version first a 2d (time, height) correction array is constructed then it is applied. In future this will allow for time varing overlap functions

calcDepol()[source]#

Calculate the volume depol and the particle depol.

Attributes:
self.retrievals_profiles[sig][idx][ch]dict

Volume and particle depolarization profiles and retrieval info for signal type sig for cloud free segment idx and channel ch.

Notes

  • This retrieval is done for all avaiable optical profiles.

estQualityMask()[source]#

Estimate the quality mask.

Attributes:
self.retrievals_highres[‘quality_mask’]ndarray
High resolution (time, height) quality masks per channel.

0 : good data 1 : low-SNR data 2 : depolarization calibration periods 3 : shutter on 4 : fog 5 : saturated (NEW)

Notes

  • If the config variables flagUseImprovedSNR=True, SNR and the lowSNRMask are recalculated with quasi-smoothed signals to improve data quality.

** History **

  • xxxx-xx-xx: First edition by …

  • 2026-07-17: Added quasi-smoothed SNR and lowSNRMask.

Angstroem()[source]#

Calculate the angstrom exponent.

Attributes:
self.retrievals_profiles[sig][idx][ch]dict

Volume and particle depolarization profiles and retrieval info for signal type sig for cloud free segment idx and channel ch.

Notes

  • AE_{prod}_{wv1}_{wv2} = log(prod_wv1 / prod_wv2) / log(wv2 / wv1)

  • This retrieval is done for all avaiable optical profiles.

LidarCalibration(db_path: str = None, collect_debug: bool = False, **defaults)[source]#

Calculate/Estimate and select optimal lidar calibration constants.

The function calculates/estimates lidar calibration constants (LCs) from both klett and Raman retrieved optical profiles. If configured, LCs from a defined database are also loaded. The optimal LC per channel is chosen as the LC with the lowest relative error per channel form both the calculated/estimated and loaded LCs. If no LC is found for a channel its default value is taken.

Parameters:
db_pathstr or Path, optional

Path to calibration constant database. If db_path is None and config variable flagUsePreviousLC is enabled, a standard path is constructed from the config variable calibrationDB. Default is None.

collect_debugbool, optional

If True, collects debug information. Default is False.

LClist, optional

Default Lidar constant value per channel.

LCStdlist, optional

Default Lidar constant error per channel.

Attributes:
self.LC[‘klett’ & ‘klett_db’]dict

All retrieved and read Klett lidar calibration constants and retrieval information.

self.LC[‘raman’ & ‘raman_db’]dict

All retrieved and read Raman lidar calibration constants and retrieval information.

self.LC[‘default’]dict

Default lidar calibration constants and retrieval information.

self.LCuseddict

The optimal lidar calibration constant per channel.

Notes

The function uses the following configuration flags:

  • flagLCCalibration: Enables or disables calculation of Lidar

    calibration constants from klett and Raman retrieval.

  • flagUsePreviousLC: Enables or disables loading previously determined

    lidar calibration constants from the calibration database.

All calibration constants are stored in self.LC. The selected optimal constants are stored in self.LCused.

Default values are by standard taken from their config variable but can be overwritten if passed as an input to this function.

History

  • xxxx-xx-xx: First edition by …

  • 2026-10-08: Consider retrieved and database constants equally

    and added default calibration constants.

attBsc_volDepol()[source]#

High resolution attenuated backscatter and volume depolarization.

Attributes:
self.retrievals_highresdict

With attenuated backscatter.

molecularHighres()[source]#

Calculate the molecular signal for the 2d high resolution.

Attributes:
self.mol_2dxr.Dataset

xarray dataset with dimensions time and height and variables molecular backscatter (mBsc) and molecular extinction (mExt) per wavelength.

Notes

  • mol_2d’s time dimension differe from the time dimension of the measurement.

wvmr()[source]#

High resolution water vapor mixing ratio.

Attributes:
self.retrieval_highresdict

With water vapor mixing rato.

quasiV1()[source]#

QuasiV1 retrievals and target categorisation.

Attributes:
self.retrieval_highresdict

With high resolution Quasi version 1 products as well as Target categorization.

Notes

  • QuasiV1 products uses Klett-retrieved products.

quasiV2()[source]#

QuasiV2 retrievals and target categorisation.

Attributes:
self.retrieval_highresdict

With high resolution Quasi version 2 products as well as Target categorization.

Notes

  • QuasiV2 products uses Raman-retrieved products.

write_2_sql_db(parameter: str, db_path: str | None = None, method: str | None = None)[source]#

Write LC, eta or WVC to sqlite db table.

Parameters:
parameterstr

can be LC (Lidar-calibration-constant), DC (Depol-calibration-constant) or WVC (WaterVapor-calibration-constant)

methodstr

‘raman’, ‘klett’, ‘D90’, ‘transfer’, ‘profile’ or ‘regression’

db_pathstr

location of the sqlite db-file

Notes

  • The unique columns are needed that new entries overwrite old ones, otherwise they are just added to the table with same timestamps.

read_calibration_db(db_path: str | None = None)[source]#

Read the calibration constants from database.

Parameters:
db_pathstr or NoneType

path to database file… Default is None.

Attributes:
self.LCdict

Original LC with all additinal lidar calibration constant availabel in the datafram db_path in the time period of the measurement +-24h.

self.pol_calidict

Original pol_cali with all additinal polarization calibration constant availabel in the datafram db_path in the time period of the measurement +-24h.

Notes

  • Time interval includes 24h before start of the measurement and after the end of the measurement.

adding_retrieving_infos_2_polly_config_dict()[source]#

Some infos from the polly_config_dict should have there own keys, e.g. reference_search_range.

Attributes:
self.polly_config_dictdict

With additional info.

Notes

Todo

  • Finish docstring.

  • Is this function necessary?

runProcessing()[source]#

Idea: Include a run function in the object as an alternative to the default run script…