Picasso Functions

Picasso main module

Picasso calibration

lib.calibration.DouglasPeucker(signal, height, epsilon, heightBase, heightTop, maxHThick, window_size)

DOUGLASPEUCKER simplify signal according to Douglas-Peucker algorithm.

USAGE:

[ sigIndx ] = DouglasPeucker(signal, height, epsilon, heightBase, heightTop, maxHThick, window_size)

INPUTS:

signal: array: Molecule corrected signal. [MHz]
height: array: height. [m]
epsilon: float: maximum distance.
heightBase: float: minimun height for the algorithm. [m]
heightTop: float: maximum height for the algorithm. [m]
maxHThick:: maximum spatial thickness of each segment. [m]
window_size: integer: size of the average smooth window.

OUTPUTS:

sigIndx: array: index of the signal that stands for different segments of the signal.

REFERERENCES:

https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm

HISTORY:

2017-12-29: First edition by Zhenping.
2018-07-29: Add the height range for the searching instead of SNR restriction.
2018-07-31: Add the maxHThick argument to control the maximum thickness of each output segment.x

lib.calibration.depolCali(signal_t, bg_t, signal_x, bg_x, time, polCaliPAngStartTime, polCaliPAngStopTime, polCaliNAngStartTime, polCaliNAngStopTime, TR_t, TR_x, caliHIndxRange, SNRmin, sigMax, rel_std_dplus, rel_std_dminus, segmentLen, smoothWin)

DEPOLCALI polarization calibration for PollyXT lidar system.

USAGE:

[polCaliEta, polCaliEtaStd, polCaliFac, polCaliFacStd, depol_cal_time] = depolCali(signal_t,: bg_t, signal_x, bg_x, time, polCaliPAngStartTime, polCaliPAngStopTime, polCaliNAngStartTime, polCaliNAngStopTime, TR_t, TR_x, caliHIndxRange, SNRmin, sigMax, rel_std_dplus, rel_std_dminus, segmentLen, smoothWin, flagShowResults)

INPUTS:

signal_t: matrix: background-removed photon count signal at total channel. (nBins * nProfiles)
bg_t: matrix: background at total channel. (nBins * nProfiles)
signal_x: matrix: background-removed photon count signal at cross channel. (nBins * nProfiles)
bg_x: matrix: background at cross channel. (nBins * nProfiles)
time: array: datenum array represents the measurement time of each profile.
polCaliPAngStartTime: array: datenum array represents the start time that the polarizer rotates to the positive angle.
polCaliPAngStopTime: array: datenum array represents the stop time that the polarizer rotates to the positive angle.
polCaliNAngStartTime array: datenum array represents the start time that the polarizer rotates to the negative angle.
polCaliNAngStopTime: array: datenum array represents the end time that the polarizer rotates to the negative angle.
TR_t: float: tranmission at total channel.
TR_x: float: transmision at cross channel.
caliHIndxRange: 2-element array: range of height indexes at which the signal can be used for polarization calibration.
SNRmin: array: minimum SNR for calibration.
sigMax: array: maximum signal that could be used in the calibration to prevent pulse pileup effects. (Photon Count)
rel_std_dplus: float: maximum relative uncertainty of dplus that is allowed.
rel_std_dplus: float: maximum relative uncertainty of dminus that is allowed.
segmentLen: integer: segement length for testing the variability of the calibration results to prevent of cloud contamintaion.
smoothWin: integer: width of the sliding window for smoothing the signal.
flagShowResults: logical: flag to control whether to save the intermediate results.

OUTPUTS:

polCaliEta: array: eta from polarization calibration.
polCaliEtaStd: array: uncertainty of eta from polarizatrion calibration.
polCaliFac: array: polarization calibration factor.
polCaliFacStd: array: uncertainty of polarization calibration factor.
polCaliStartTime: array: start time for each successful calibration.
polCaliStopTime: array: stop time for each successful calibration.
globalAttri: struct: all the information about the depol calibration.

REFERENCE:

Freudenthaler, V.: About the effects of polarising optics on lidar signals and the Δ90 calibration, Atmos. Meas. Tech., 9, 4181-4255, 10.5194/amt-9-4181-2016, 2016.

HISTORY:

2018-07-25: First edition by Zhenping.
2019-06-08: If no depol cali, return empty array.
2019-09-06: Remove the part to replace the bins of low SNR with NaN, because it will lead to bias when doing smoothing.

lib.calibration.pollyMDR(sigT, bgT, sigC, bgC, Rt, RtStd, Rc, RcStd, depolConst, depolConstStd, minSNR, deftMDR, deftMDRStd)

POLLYMDR etimate the molecular depolarization ratio according to the measurements at reference height.

USAGE:

[MDR, MDRStd, flagDeft] = pollyMDR(sigT, bgT, … sigC, bgC, Rt, RtStd, Rc, RcStd, depolConst, depolConstStd, … minSNR, deftMDR, deftMDRStd)

INPUTS:

sigT: array: signal strength of the total channel at reference height. [photon count]
bgT: array: background of the total channel at reference height. [photon count]
sigCross: array: signal strength of the cross channel at reference height. [photon count]
bgCross: array: background of the cross channel at reference height. [photon count]
Rt: scalar: transmission ratio in total channel
RtStd: scalar: uncertainty of the transmission ratio in total channel
Rc: scalar: transmission ratio in cross channel
RcStd: scalar: uncertainty of the transmission ratio in cross channel
depolConst: scalar: depolarzation calibration constant. (transmission ratio for the parallel component in cross channel and total channel)
depolConstStd: scalar: uncertainty of the depolarization calibration constant.
minSNR: float: the SNR constrain for the the signal strength at reference height. Choose a strong constrain for ensuring a stable result, like 50 or 100.
deftMDR: float: default molecular depolarization ratio.
deftMDRStd: float: default std of molecular depolarization ratio.

OUTPUTS:

MDR: float: retrieved molecular depolarization ratio.
MDRStd: float: std of retrieved molecular depolarization ratio.
flagDeft: logical: flag to show whether using the default values. If true, it means default MDR and MDRStd were used.

HISTORY:

2021-05-31: first edition by Zhenping

lib.calibration.pollyMolPolCali(tSig, bgTSig, cSig, bgCSig, TR_t, TR_t_std, TR_c, TR_c_std, minSNR, mdr, mdrStd)

POLLYMOLPOLCALI molecular polarization calibration.

USAGE:

[polCaliEta, polCaliEtaStd, polCaliFac, polCaliFacStd] = pollyMolPolCali(tSig, bgTSig, cSig, bgCSig, TR_t, TR_t_std, TR_c, TR_c_std, minSNR, mdr, mdrStd)

INPUTS:

tSig: numeric: total signal. (photon count)
bgTSig: numeric: background at total channel. (photon count)
cSig: numeric: cross signal. (photon count)
bgCSig: numeric: background at cross channel. (photon count)
TR_t: scalar: transmission ratio at total channel
TR_t_std: scalar: uncertainty of the transmission ratio at total channel
TR_c: scalar: transmission ratio at cross channel
TR_c_std: scalar: uncertainty of the transmission ratio at cross channel.
minSNR: float: the SNR constrain for the the signal strength at reference height. Choose a strong constrain for ensuring a stable result, like 50 or 100.
mdr: float: default molecular depolarization ratio.
mdrStd: float: default std of molecular depolarization ratio.

OUTPUTS:

polCaliEta: array: polarization calibration eta.
polCaliEtaStd: array: uncertainty of polarization calibration eta.
polCaliFac: array: polarization calibration factor.
polCaliFacStd: array: uncertainty of polarization calibration factor.

References

Baars, H., Ansmann, A., Althausen, D., Engelmann, R., Heese, B., Muller, D., Artaxo, P., Paixao, M., Pauliquevis, T., and Souza, R.: Aerosol profiling with lidar in the Amazon Basin during the wet and dry season, J Geophys Res-Atmos, 117, 10.1029/2012jd018338, 2012.

HISTORY:

2021-07-06: first edition by Zhenping

lib.calibration.pollyPolCali(data, transRatio, varargin)

POLLYPOLCALI calibrate the PollyXT cross channels for 355 and 532 nm with ±45° method.

USAGE:

[polCaliEta, polCaliEtaStd,polCaliFac, polCaliFacStd, polCaliTime, polCaliAttri] = pollyPolCali(data, transRatio)

INPUTS:

data: struct: data
transRatio: array: transmission ratios at each channel.

KEYWORDS:

wavelength: char: ‘355nm’ or ‘532nm’.
depolCaliMinBin: numeric: minimum search index for stable polarization calibration constants.
depolCaliMaxBin: maximum search index for stable polarization calibration constants.
depolCaliMinSNR: numeric: minimum signal-noise ratio for calculating polarization calibration constants.
depolCaliMaxSig: numeric: maximum signal in photon count (to avoid signal saturation).
relStdDPlus: numeric: maximum relative std of dplus that is allowed.
relStdDMinus: numeric: maximum relative std of dminus that is allowed.
depolCaliSegLen: numeric: segement length for testing the variability of the calibration results to prevent of cloud contamintaion.
depolCaliSmWin: numeric: width of the sliding window for smoothing the signal.
dbFile: char: absolute path of the calibration database file.
pollyType: char: polly version. (‘arielle’)
flagUsePrevDepolConst: logical: whether to use previous calibration constants.
flagDepolCali: logical: whether to perform depolarization calibration.
default_polCaliEta: numeric: default eta for polarization calibration.
default_polCaliEtaStd: uncertainty of default eta for polarization calibration.

OUTPUTS:

polCaliEta: numeric: polarization calibration eta.
polCaliEtaStd: numeric: uncertainty of eta for polarization calibration.
polCaliFac: numeric: polarization calibration constant.
polCaliFacStd: numeric: uncertainty of polarization calibration constant.
polCaliTime: 2-element array: time of depolarization calibration.
polCaliAttri: struct: polly polarization calibration attributes.

HISTORY:

2018-12-17: First edition by Zhenping
2019-08-28: Add flag to control whether to do polarization calibration.
2020-04-18: Generalise the interface.

lib.calibration.pollyRayleighFit(height, sig, sigPCR, bg, mSig, varargin)

POLLYRAYLEIGHFIT search reference height with Rayleigh fit algorithm.

USAGE:

[refHInd, DPInd] = pollyRayleighFit(height, sig, sigPCR, bg, mSig)

INPUTS:

height: array: height. (m)
sig: array: lidar signal. (photon count)
sigPCR: array: lidar signal. (photon count rate)
bg: array: background. (photon count)
mSig: array: molecular signal.

KEYWORDS:

minDecomLogDist: float: maximum distance for Douglas-Peucker algorithm (default: 0.2).
maxDecomHeight: numeric: maximum height for signal decomposition (default: 10000). (m)
maxDecomThickness: numeric: maximum spatial thickness for each segment (default: 1500). [m]
decomSmWin: numeric: smoothing window for signal as input for Douglas-Peucker algorithm (default: 40).
minRefThickness: numeric: minimum spatial thickness for each segment
minRefDeltaExt: numeric: constrain for the uncertainty of the regressed extinction coefficient in the reference height. (see test 3 in Baars et al, ACP, 2016)
minRefSNR: numeric: minimum SNR for the signal at the reference height (default: 5).
heightFullOverlap: numeric: minimum height with full overlap (default: 600). (m)
flagSameRef: logical: flag to determine whether use default reference height and decomposition points (default: false).
defaultRefH: 2-element array: default reference height (default: [NaN, NaN]). (m)
defaultDPInd: array: default decomposition points by Douglas-Peucker algorithm (default: []).
printLevel: numeric: print level. 0, 1, 2: print details while running. 3, 4, 5: hide prompts while running.

OUTPUTS:

refHInd: 2-element array: [base, top] index of the reference height.
DPInd: array: index of the signal that stands for different segments of the signal.

HISTORY:

2021-05-25: first edition by Zhenping

lib.calibration.pollyWVCali(height, sig387, bg387, sig407, E_tot_1064_IWV, E_tot_1064_cali, E_tot_1064_cali_std, wvCaliStarttime, wvCaliStoptime, IWV, flagWVCali, flag407On, trans387, trans407, rhoAir, sunriseTime, sunsetTime, varargin)

POLLYWVCALI water vapor calibration.

USAGE:

[wvconst, wvconstStd, globalAttri] = pollyWVCali(height, sig387, bg387, …: sig407, E_tot_1064_IWV, E_tot_1064_cali, E_tot_1064_cali_std, … wvCaliStarttime, wvCaliStoptime, IWV, flagWVCali, flag407On, … trans387, trans407, rhoAir)

INPUTS:

height: numeric: height. (m)
sig387: numeric: signal at 387 nm.
bg387: numeric: background at 387 nm
sig407: numeric: signal at 407 nm
E_tot_1064_IWV: numeric: integral 1064 nm signal when external IWV measurements were done.
E_tot_1064_cali: numeric: integral 1064 nm signal.
E_tot_1064_cali_std: numeric: integral of signal at 1064 nm.
wvCaliStarttime: numeric: water vapor calibration start time.
wvCaliStoptime: numeric: water vapor calibration stop time.
IWV: numeric: integral water vapor.
flagWVCali: logical: water vapor calibration flag.
flag407On: logical: flag of channel status at 407 nm.
trans387: numeric: transmittance at 387 nm.
trans407: numeric: transmittance at 407 nm.
rhoAir: numeric: air density.
sunriseTime: numeric: sunrise time.
sunsetTime: numeric: sunset time.

KEYWORDS

hWVCaliBase: numeric: base height of water vapor calibration range.
hFullOL387: numeric: minimum height of full overlap at 387 nm.
minSNRWVCali: numeric: minimum SNR for water vapor calibration.

OUTPUTS:

wvconst: array

water vapor calibration constant. [g/kg]

wvconstStd: array

uncertainty of water vapor calibration constant. [g/kg]

globalAttri: struct

cali_start_time: array: water vapor calibration start time. [datenum]
cali_stop_time: array: water vapor calibration stop time. [datenum]
WVCaliInfo: cell: calibration information for each calibration period.
IntRange: matrix: index of integration range for calculate the raw IWV from lidar.

References

Dai, G., Althausen, D., Hofer, J., Engelmann, R., Seifert, P., Bühl, J., Mamouri, R.-E., Wu, S., and Ansmann, A.: Calibration of Raman lidar water vapor profiles by means of AERONET photometer observations and GDAS meteorological data, Atmospheric Measurement Techniques, 11, 2735-2748, 2018.

HISTORY:

2018-12-26: First Edition by Zhenping
2019-08-08: Add the sunrise and sunset to exclude the low SNR calibration periods.

lib.calibration.rayleighfit(height, sig_aer, pc, bg, sig_mol, dpIndx, layerThickConstrain, slopeConstrain, SNRConstrain, flagShowDetail)

RAYLEIGHFIT search the clean region with rayleigh fit algorithm.

USAGE:

[ hBIndx, hTIndx ] = rayleighfit(height, sig_aer, sig_mol, dpIndx,: layerThickConstrain, slopeConstrain, SNRConstrain, flagShowDetail)

INPUTS:

height: array: height. [m]
sig_aer: array: range corrected signal.
pc: array: photon count signal.
bg: array: background.
sig_mol: array: range corrected molecular signal.
dpIndx: array: index of the region which is calculated by Douglas-Peucker algorithm.
layerThickConstrain: float: constrain for the reference layer thickness. [m]
slopeConstrain: float: constrain for the uncertainty of the regressed extinction coefficient in the reference height. (see test 3 in Baars et al, ACP, 2016)
SNRConstrain: float: minimum SNR for the signal at the reference height.
flagShowDetail: boolean: if flagShowDetail is true, the calculation information will be printed. Default is false.

OUTPUTS:

hBIndx: int: index of the bottom of the searched region. If the region is not found, NaN will be returned.
hTIndx: int: index of the top of the searched region. If the region is not found, NaN will be returned.

References

Baars, H., et al. (2016). “An overview of the first decade of Polly NET: an emerging network of automated Raman-polarization lidars for continuous aerosol profiling.” Atmospheric Chemistry and Physics 16(8): 5111-5137.

HISTORY:

2018-01-01: First edition by Zhenping.
2018-07-05: Add the SNR constrain for the reference height.
2019-01-01: change the single array to double to avoid overflow in chi2fit.
2019-05-26: Strengthen the criteria for Near-Far Range test. Old: (meanSig_aer + deltaSig_aer) >= meanSig_mol; New: (meanSig_aer + deltaSig_aer/3) >= meanSig_mol
2019-08-03: Using the SNR for the final determination. The higher SNR of the reference, the better.
2019-09-15: Fix the bug in slope criteria.

Picasso classification

lib.classification.VDE_cld(signal, height, BG, minLayerDepth, minHeight, smoothWin, minSNR)

VDE_CLD cloud layer detection with VDE method. THis method only required elstic signal.

INPUTS:

signal: array: raw signal without background. [photon count]
height: array: height above the ground. [km]
BG: numeric: background signal. [photon count]
minLayerDepth: numeric: minimun layer geometrical depth (default: 0.2). [km]
minHeight: numeric: minimum height to start the searching with (default: 0.4). [km]
smoothWin: numeric: smoothindow in bins (default: 3).
minSNR: numeric: minimum layer SNR to filter the fake features (default: 5).

OUTPUTS:

layerInfo: struct array

id: numeric: identity of the layer.
baseHeight: numeric: the layer base height. [km]
topHeight: numeric: the layer top height. [km]
peakHeight: numeric: the layer height with maximum backscatter signal. [km]
layerDepth: numeric: geometrical depth of the layer. [km]
flagCloud: logical: cloud flag.

PD: array

SDP signal.

PN: array

VDE signal

References

Zhao, C., Y. Wang, Q. Wang, Z. Li, Z. Wang, and D. Liu (2014), A new cloud and aerosol layer detection method based on micropulse lidar measurements, Journal of Geophysical Research: Atmospheres, 119(11), 6788-6802.

HISTORY:

2021-06-13: first edition by Zhenping

lib.classification.detectLiquidBits(height, bsc1064, varargin)

DETECTLIQUIDBITS detect liquid cloud bits.

USAGE:

flagLiquid = detectLiquidBits(height, bsc1064, cloudThresBsc1064, minAttnBsc1064, p.Results.searchCloudAbove, p.Results.searchCloudBelow)

INPUTS:

height: numeric: height. (m)
bsc1064: matrix (height x time): particle backscatter at 1064 nm.

KEYWORDS:

cloudThresBsc1064: numeric: threshold of cloud backscatter at 1064 nm.
minAttnBsc1064: numeric: minimum attanuation required to detect liquid cloud.
searchCloudAbove: numeric: cloud search window above current bit. (m)
searchCloudBelow: cloud search window below current bit. (m)

OUTPUTS:

flagLiquid: logical (height x time)

HISTORY:

2021-06-05: first edition by Zhenping

lib.classification.targetClassify(height, attBeta532, quasiBsc1064, quasiBsc532, quasiPDR532, VDR532, quasiAE, varargin)

TARGETCLASSIFY aerosol/cloud target classification.

USAGE:

[tc_mask] = targetClassify(height, attBeta532, quasiBsc1064, quasiBsc532, quasiPDR532, VDR532, quasiAE)

INPUTS:

height: numeric: height. (m)
attBeta532: matrix (height x time): attenuated backscatter at 532 nm.
quasiBsc1064: matrix (height x time): quasi particle backscatter at 1064 nm. (m^{-1}sr^{-1})
quasiBsc532: matrix (height x time): quasi particle backscatter at 532 nm. (m^{-1}sr^{-1})
quasiPDR532: matrix (height x time): quais particle depolarization ratio at 532 nm
VDR532: matrix (height x time): volume depolarization ratio at 532 nm
quasiAE: matrix (height x time): quasi Ångström exponents.

KEYWORDS:

clearThresBsc1064: numeric turbidThresBsc1064: numeric turbidThresBsc532: numeric dropletThresPDR: numeric spheriodThresPDR: numeric unspheroidThresPDR: numeric iceThresVDR: numeric iceThresPDR: numeric largeThresAE: numeric smallThresAE: numeric cloudThresBsc1064: numeric minAttnRatioBsc1064: numeric searchCloudAbove: numeric searchCloudBelow: numeric hFullOL: numeric

OUTPUTS:

tc_mask: matrix: ‘0: No signal’ ‘1: Clean atmosphere’ ‘2: Non-typed particles/low conc.’ ‘3: Aerosol: small’ ‘4: Aerosol: large, spherical’ ‘5: Aerosol: mixture, partly non-spherical’ ‘6: Aerosol: large, non-spherical’ ‘7: Cloud: non-typed’ ‘8: Cloud: water droplets’ ‘9: Cloud: likely water droplets’ ‘10: Cloud: ice crystals’ ‘11: Cloud: likely ice crystal

References

Baars, H., Seifert, P., Engelmann, R. & Wandinger, U. Target categorization of aerosol and clouds by continuous multiwavelength-polarization lidar measurements. Atmospheric Measurement Techniques 10, 3175-3201, doi:10.5194/amt-10-3175-2017 (2017).

HISTORY:

2021-06-05: first edition by Zhenping

Picasso cloud-mask

lib.cloudmask.cloudDetect_Zhao(time, height, signal, bg, varargin)

CLOUDDETECT_ZHAO cloud layer detection based on Zhao’s algorithm.

USAGE:

% Usecase 1: get the cloud mask flagCloudFree = cloudDetect_Zhao(time, height, signal, bg);

% Usecase 2: specify the detection range [flagCloudFree, layerStatus] = cloudDetect_Zhao(time, height, signal, bg, ‘detectRange’, [0, 8000])

% Usecase 3: specify the minimum layer depth [flagCloudFree, layerStatus] = cloudDetect_Zhao(time, height, signal, bg, ‘minDepth’, 500);

INPUTS:

time: array: measurement time for each profile. (datenum)
height: array: height above ground. (m)
signal: matrix (height x time): signal. (photon count)
bg: array: background. (photon count)

KEYWORDS:

minDepth: double: minimum layer depth (default: 100). (m)
detectRange: 2-element array: bottom and top height for cloud detection (default: [0, 10000]). (m)
heightFullOverlap: double: minimum height with full overlap (default: 600). (m)
smoothWin: integer: smooth window (default: 8). (bins)
minSNR: double: minimum layer mean signal-noise-ratio (default: 1).

OUTPUTS:

flagCloudFree: array: cloud free mask for each profile.
layerStatus: matrix (height x time): layer status for each bin. (0: unknown; 1: cloud; 2: aerosol)

References

Zhao, C., Y. Wang, Q. Wang, Z. Li, Z. Wang, and D. Liu (2014), A new cloud and aerosol layer detection method based on micropulse lidar measurements, Journal of Geophysical Research: Atmospheres, 119(11), 6788-6802.

HISTORY:

2021-05-18: first edition by Zhenping

lib.cloudmask.cloudScreen(time, height, signal, varargin)

CLOUDSCREEN cloud screen.

USAGE:

% Usecase 1: get the cloud mask flagCloudFree = cloudScreen(time, height, signal);

% Usecase 2: cloudscreen with using signal gradient flagCloudFree = cloudScreen(time, height, signal, ‘mode’, 1, ‘detectRange’, [0, 8000], ‘slope_thres’, 1e7)

% Usecase 3: cloudscreen with using Zhao’s algorithm [flagCloudFree, layerStatus] = cloudScreen(time, height, signal, ‘mode’, 2, ‘minDepth’, 500);

INPUTS:

time: array: measurement time for each profile. (datenum)
height: array: height above ground. (m)
signal: matrix (height x time): signal. (photon count)

KEYWORDS:

mode: integer: 1: with using signal gradient (default) 2: with using Zhao’s algorithm
minDepth: double: minimum layer depth (default: 100). (m)
detectRange: 2-element array: bottom and top height for cloud detection (default: [0, 10000]). (m)
heightFullOverlap: double: minimum height with full overlap (default: 600). (m)
smoothWin: integer: smooth window (default: 8). (bins)
minSNR: double: minimum layer mean signal-noise-ratio (default: 1).
background: array: background (default: 0 for each profile). (photon count)
slope_thres: double: threshold of the slope to determine whether there is strong backscatter signal (default: 0). [MHz*m]

OUTPUTS:

flagCloudFree: array: cloud free mask for each profile.
layerStatus: matrix (height x time): layer status for each bin. (0: unknown; 1: cloud; 2: aerosol)

References

Zhao, C., Y. Wang, Q. Wang, Z. Li, Z. Wang, and D. Liu (2014), A new cloud and aerosol layer detection method based on micropulse lidar measurements, Journal of Geophysical Research: Atmospheres, 119(11), 6788-6802.

HISTORY:

2021-05-18: first edition by Zhenping

lib.cloudmask.cloudScreen_MSG(time, height, signal, slope_thres, search_region)

CLOUDSCREEN_MSG cloud screen with maximum signal gradient.

USAGE:

flagCloudFree = cloudScreen_MSG(height, signal, slope_thres, search_region)

INPUTS:

time: array: measurement time for each profile.
height: array: height. [m]
signal: matrix (height * time): photon count rate. [MHz]
slope_thres: float: threshold of the slope to determine whether there is strong backscatter signal. [MHz*m]
search_region: 2-elements array: [baseHeight, topHeight]. [m]

OUTPUTS:

flagCloudFree: boolean: whether the profile is cloud free.
layerStatus: matrix (height x time): layer status for each bin. (0: unknown; 1: cloud; 2: aerosol)

HISTORY:

2021-05-18: first edition by Zhenping

Picasso io

lib.io.decompressPollyData(startDate, endDate, saveFolder, PicassoConfigFile, varargin)

DECOMPRESSPOLLYDATA Unzip the polly data and write data info to the todolist file for pollynet processing chain.

USAGE:

decompressPollyData(startDate, endDate, saveFolder, PicassoConfigFile)

INPUTS:

startDate: numeric: start date of polly data to be decompressed. i.e, datenum(2015, 1, 1) stands for Jan 1st, 2015.
endDate: numeric: stop date of polly data to be decompressed.
saveFolder: char: polly data folder. e.g., /oceanethome/pollyxt
PicassoConfigFile: char: the absolute path of the pollynet configuration file. e.g., /home/picasso/Pollynet_Processing_Chain/config/pollynet_processing_chain_config.json

KEYWORDS:

pollyType: char: polly instrument. e.g., arielle
mode: char: If mode was ‘a’, the polly data info will be appended. If ‘w’, a new todofile will be created.

HISTORY:

2019-07-21: First Edition by Zhenping
2019-10-16: Add warnings when no polly data files were found.

lib.io.extract_cali_results(dbFile, csvFilepath, varargin)

EXTRACT_CALI_RESULTS extract calibration results from SQLite database to ASCII files.

USAGE:

% Usecase 1: convert single table extract_cali_results(‘/path/to/dbFile’, ‘/path/to/csvFile’, ‘tablename’, ‘lidar_calibration_constant’);

% Usecase 2: convert all tables extract_cali_results(‘/path/to/dbFile’, ‘/path/to/csvFile’);

% Usecase 3: add prefix for csv files extract_cali_results(‘/path/to/dbFile’, ‘/path/to/csvFile’, ‘prefix’, ‘arielle-‘);

INPUTS:

dbFile: char: absolute path of database file.
csvFilepath: char: output folder for the csv file.

KEYWORDS:

tablename: char: table name that needs to be extracted (regular expression is supported). (defaults: ‘.*’)
prefix: char: prefix for the ASCII filename.
SQLiteReadMode: char: ‘database_toolbox’ (default) or ‘jdbc’

OUTPUTS:

csvFilenames: cell: absolute path for extracted ASCII files.
csvFileID: cell: identifier (table name) for respective csv file.

HISTORY:

2021-06-13: first edition by Zhenping

lib.io.fillmissing(xIn, varargin)

FILLMISSING fill the missing values in input array.

USAGE:

[xOut] = fillmissing(xIn, varargin)

INPUTS:

xIn: array or matrix: input array which could contain some values that you want to replace.

OUTPUTS:

xOut: array or matrix: after the missing values were filled.

HISTORY:

2021-06-26: first edition by Zhenping

lib.io.loadConfig(configFile, globalConfigFile)

LOADCONFIG load configurations

USAGE:

[config] = loadConfig(configFile, globalConfigFile)

INPUTS:

configFile: char: absolute path of the configuration file.
globalConfigFile: char: absolute path of the global configuration file.

OUTPUTS:

config: struct: configurations.

HISTORY:

2021-04-07: first edition by Zhenping

lib.io.loadConfigPrivate(configFile)

LOADCONFIGPRIVATE load key-value paired configuration file, and convert it into matlab struct.

USAGE:

[thisConfig] = loadConfigPrivate(configFile)

INPUTS:

configFile: char: absolute path the configuration file. This file should only contain the key=value pair and with comments start with ‘#’. e.g., # This is an example user=”Zhenping” password=’123’

OUTPUTS:

thisConfig: struct: this struct contains all the valid key-value pairs in the configuration file. The comments will be filtered and any line start with whitespace will be filtered as well.

HISTORY:

2019-09-02: Source code comes from the answer in matlab forum under the link [https://de.mathworks.com/matlabcentral/answers/16494-periodically-updated-static-text-and-reading-from-key-value-file]: Great thanks to the author Meric Ozturk.
2019-09-02: Modified by Zhenping.

lib.io.loadDepolConst(queryTime, dbFile, pollyType, wavelength, varargin)

LOADDEPOLCONST load depolarization calibration constant from database.

USAGE:

[depolconst, depolconstStd, caliStartTime, caliStopTime] = loadDepolConst(queryTime, dbFile, pollyType, wavelength)

INPUTS:

queryTime: datenum: query time.
dbFile: char: absolute path of the SQLite database.
pollyType: char: polly name. (case-sensitive)
wavelength: char: wavelength (‘355’ or ‘532’).

KEYWORDS:

deltaTime: datenum: search range for the query time. (default: NaN)
flagClosest: logical: flag to control whether to return the closest value only. (default: false) (default: false)
flagBeforeQuery: logical: flag to control whether to return records with calibration time before queryTime. (default: false)

OUTPUTS:

depolconst: array: depolarization calibration constant.
depolconstStd: array: uncertainty of depolarization calibration constant.
caliStartTime: array: calibration start time for each record.
caliStopTime: array: calibration stop time for each record.

HISTORY:

2021-06-08: first edition by Zhenping

lib.io.loadLiConst(queryTime, dbFile, pollyType, wavelength, caliMethod, telescope, varargin)

LOADLICONST load lidar calibration constant from database.

USAGE:

[liconst, liconstStd, caliStartTime, caliStopTime] = loadLiConst(queryTime, dbFile, pollyType, wavelength, caliMethod)

INPUTS:

queryTime: datenum: query time.
dbFile: char: absolute path of the SQLite database.
pollyType: char: polly name. (case-sensitive)
wavelength: char: wavelength (‘355’, ‘532’, ‘1064’, ‘387’ or ‘607’).
caliMethod: char: calibration method (‘Klett_Method’, ‘Raman_Method’, ‘AOD_Constrained_Method’)
telescope: char: detection range. (‘far_range’, or ‘near_range’)

KEYWORDS:

deltaTime: datenum: search range for the query time. (default: NaN)
flagClosest: logical: flag to control whether to return the closest value only. (default: false) (default: false)
flagBeforeQuery: logical: flag to control whether to return records with calibration time before queryTime. (default: false)

OUTPUTS:

liconst: array: lidar calibration constant.
liconstStd: array: uncertainty of lidar calibration constant.
caliStartTime: array: calibration start time for each record.
caliStopTime: array: calibration stop time for each record.

HISTORY:

2021-06-11: first edition by Zhenping

lib.io.loadMeteor(mTime, asl, varargin)

LOADMETEOR read meteorological data.

USAGE:

[temp, pres, relh, wins, wind, meteorAttri] = loadMeteor(mTime, asl)

INPUTS:

mTime: array: query time.
asl: array: height above sea level. (m)

KEYWORDS:

meteorDataSource: str: meteorological data type. e.g., ‘gdas1’(default), ‘standard_atmosphere’, ‘websonde’, ‘radiosonde’
gdas1Site: str: the GDAS1 site for the current campaign.
gdas1_folder: str: the main folder of the GDAS1 profiles.
radiosondeSitenum: integer: site number, which can be found in doc/radiosonde-station-list.txt.
radiosondeFolder: str: the folder of the sonding files.
radiosondeType: integer: file type of the radiosonde file. 1: radiosonde file for MOSAiC (default) 2: radiosonde file for MUA
flagReadLess: logical: flag to determine whether access meteorological data by certain time interval. (default: false)
method: char: Interpolation method. (default: ‘nearest’)
isUseLatestGDAS: logical: whether to search the latest available GDAS profile (default: false).

OUTPUTS:

temp: matrix (time * height)

temperature for each range bin. [°C]

pres: matrix (time * height)

pressure for each range bin. [hPa]

relh: matrix (time * height)

relative humidity for each range bin. [%]

wins: matrix (time * height)

wind speed. (m/s)

meteorAttri: struct

dataSource: cell: The data source used in the data processing for each cloud-free group.
URL: cell: The data file info for each cloud-free group.
datetime: array: datetime label for the meteorlogical data.

HISTORY:

2021-05-22: first edition by Zhenping

lib.io.loadPollyConfig(pollyConfigFile, pollyGlobalConfigFile)

LOADPOLLYCONFIG load polly configurations from polly config file.

USAGE:

[pollyConfig] = loadPollyConfig(pollyConfigFile, pollyGlobalConfigFile)

INPUTS:

pollyConfigFile: char: absolute path of polly config file.
pollyGlobalConfigFile: char: absolute path of polly global config file.

OUTPUTS:

pollyConfig: struct: polly configurations. Details can be found in doc/polly_config.md

HISTORY:

2018-12-16: First edition by Zhenping
2019-08-01: Remove the conversion of depol cali time. (Don’t need to set the depol cali time any more)
2019-08-03: Add global polly config for unify the defaults polly settings.

lib.io.loadWVConst(queryTime, dbFile, pollyType, varargin)

LOADWVCONST load water vapor calibration constant from database.

USAGE:

[wvconst, wvconstStd, caliStartTime, caliStopTime, caliInstrument, instrumentMeasTime] = loadWVConst(queryTime, dbFile, pollyType)

INPUTS:

queryTime: datenum: query time.
dbFile: char: absolute path of the SQLite database.
pollyType: char: polly name. (case-sensitive)

KEYWORDS:

deltaTime: datenum: search range for the query time. (default: NaN)
flagClosest: logical: flag to control whether to return the closest value only. (default: false)
flagBeforeQuery: logical: flag to control whether to return records with calibration time before queryTime. (default: false)

OUTPUTS:

wvconst: array: water vapor calibration constant. (g*kg^{-1})
wvconstStd: array: uncertainty of water vapor calibration constant. (g*kg^{-1})
caliStartTime: array: calibration start time for each record.
caliStopTime: array: calibration stop time for each record.
caliInstrument: cell: intrument used in the water vapor calibration for each record.
instrumentMeasTime: array: timestamp for the data measured by the standard instrument.

HISTORY:

2021-06-11: first edition by Zhenping

lib.io.pollyReadTemps(file)

POLLYREADTEMPS read polly housekeeping data saved in a temporary file.

USAGE:

[res] = pollyReadTemps(file)

INPUTS:

file: char: filename. (absolute filename)

OUTPUTS:

res: struct

time: datenum array: log time.
T1064: array: temperature of the 1064 PMT
pyro: array: laser energy
T1: array: temperature 1
RH1: array: RH 1
T2: array: temperature 2
RH2: array: RH 2
Tout: array: temperature outside
RHout: array: RH outside
Status: array: status
Dout: array: D outside

HISTORY:

2019-09-28: First Edition by Zhenping
2021-02-14: Add error catch processing.

lib.io.pollySaveAttnBeta(data)

POLLYSAVEATTNBETA save attenuated backscatter.

USAGE:

pollySaveAttnBeta(data)

INPUTS:

data: struct

HISTORY:

2019-01-10: First Edition by Zhenping
2019-05-16: Extended the attributes for all the variables and comply with the ACTRIS convention.
2019-09-27: Turn on the netCDF4 compression.

lib.io.pollySaveCloudInfo(data)

POLLYSAVECLOUDINFO save cloud layering information to netcdf file.

USAGE:

pollySaveCloudInfo(data)

INPUTS:

data: struct

HISTORY:

2021-06-09: first edition by Zhenping

lib.io.pollySaveNRAttnBeta(data)

POLLYSAVENRATTNBETA save near-field attenuated backscatter.

USAGE:

pollySaveNRAttnBeta(data)

INPUTS:

data: struct

HISTORY:

2021-06-09: first edition by Zhenping

lib.io.pollySaveNRProfiles(data)

POLLYSAVENRPROFILES save profiles of aerosol optical properties.

Usage:

pollySaveNRProfiles(data)

INPUTS:

data.struct

History:

2019-08-06: First Edition by Zhenping
2019-09-27: Turn on the netCDF4 compression

lib.io.pollySaveOCAttnBeta(data)

POLLYSAVEOCATTNBETA save overlap corrected attenuated backscatter.

USAGE:

pollySaveOCAttnBeta(data)

INPUTS:

data: struct

HISTORY:

2021-06-09: first edition by Zhenping

lib.io.pollySaveOCProfiles(data)

POLLYSAVEOCPROFILES saving vertical profiles of aerosol properties.

USAGE:

pollySaveOCProfiles(data)

INPUTS:

data.struct

HISTORY:

2021-06-08: first edition by Zhenping

lib.io.pollySaveOverlap(data, file)

POLLYSAVEOVERLAP save overlap function.

USAGE:

pollySaveOverlap(data, file)

INPUTS:

data: struct file: char

HISTORY:

2018-12-21. First Edition by Zhenping
2019-05-16. Extended the attributes for all the variables and comply with the ACTRIS convention.
2019-09-27. Turn on the netCDF4 compression.

lib.io.pollySaveProfiles(data)

POLLYSAVEPROFILES save the retrieved results, including backscatter, extinction coefficients, lidar ratio, volume/particles depolarization ratio and so on.

USAGE:

pollySaveProfiles(data)

INPUTS:

data: struct

HISTORY:

2021-06-08: first edition by Zhenping

lib.io.pollySaveQsiV1(data)

POLLYSAVEQSIV1 save quasi-retrieving (V1) results.

USAGE:

pollySaveQsiV1(data)

INPUTS:

data: struct

HISTORY:

2018-12-30: First Edition by Zhenping
2019-05-16: Extended the attributes for all the variables and comply with the ACTRIS convention.
2019-09-27: Turn on the netCDF4 compression.

lib.io.pollySaveQsiV2(data)

pollySaveQsiV2 save quasi-retrieving (V2) results.

USAGE:

pollySaveQsiV2(data)

INPUTS:

data: struct

HISTORY:

2019-08-03: First Edition by Zhenping
2019-09-27: Turn on the netCDF4 compression.

lib.io.pollySaveTCV1(data)

POLLYSAVETCV1 save aerosol/cloud target classification products.

USAGE:

pollySaveTCV1(data)

INPUTS:

data: struct

HISTORY:

2018-12-30: First Edition by Zhenping
2019-05-16: Extended the attributes for all the variables and comply with the ACTRIS convention.
2019-09-27: Turn on the netCDF4 compression.

lib.io.pollySaveTCV2(data)

POLLYSAVETCV2 save target classification results (V2)

USAGE:

pollySaveTCV2(data)

INPUTS:

data: struct

HISTORY:

2018-12-30: First Edition by Zhenping
2019-05-16: Extended the attributes for all the variables and comply with the ACTRIS convention.
2019-09-27: Turn on the netCDF4 compression.

lib.io.pollySaveVDR(data)

POLLYSAVEVDR save volume depolarization ratio.

USAGE:

pollySaveVDR(data)

INPUTS:

data: struct

HISTORY:

2019-01-10: First Edition by Zhenping
2019-05-16: Extended the attributes for all the variables and comply with the ACTRIS convention.
2019-09-27: Turn on the netCDF4 compression.

lib.io.pollySaveWV(data)

POLLYSAVEWV save water vapor products.

USAGE:

pollySaveWV(data)

INPUTS:

data: struct

HISTORY:

2019-03-15: First Edition by Zhenping
2019-05-16: Extended the attributes for all the variables and comply with the ACTRIS convention.
2019-09-27: Turn on the netCDF4 compression.

lib.io.pollyWriteDonelist(data)

POLLYWRITEDONELIST Write processing information to done list file and generate report for the current task. These report can be used for further examination.

USAGE:

[reportStr] = pollyWriteDonelist(data)

INPUTS:

data: struct

OUTPUTS:

reportStr: char

HISTORY:

2021-06-21: first edition by Zhenping

lib.io.print_msg(inStr, varargin)

PRINT_MSG Print message.

USAGE:

% Usercase 1: print message print_msg(‘Hello world!’); % Usercase 2: print message with timestamp print_msg(‘Hello world!’, ‘flagTimestamp’, true);

INPUTS:

inStr: char: input char array

KEYWORDS:

mode: digit: 0: normal mode 1: log mode (default)
flagSimpleMsg: logical: simple message flag. (default: true)
flagTimestamp: logical: flag to control whether add timestamp for the message. (default: false)

HISTORY:

2021-04-06: first edition by Zhenping

lib.io.readAERONET(site, mdate, level, flagFilterNegAOD)

READAERONET This function determines the Aerosol Optical Depth (AOD) from a collocated photometer. Available AOD values for a specified day are returned. Data is downloaded for the specified location from Aeronet website: http://aeronet.gsfc.nasa.gov/new_web/aerosols.html The function accesses the appropriate aeronet website first. This way the website is triggered to create a compressed file with the requested data. This file is accessed and unzipped into a temporary file. The temporary file is then read and finally deleted. AOD values and corresponding time along with the link to the aeronet website are returned.

USAGE:

[measTime, AOD, wavelength, IWV, angstrexp440_870, AERONETAttri] = readAERONET(site, date, level)

INPUTS:

site: char: AERONET site. You can find the nearest site by referring to doc/AERONET-station-list.txt
mdate: integer or two-element array: the measurement day or [startDate, endDate]. [datenum]
level: char: product level. (‘10’, ‘15’, ‘20’)
flagFilterNegAOD: logical: flag to control whether to filter out the negative AOD values. (default: true)

OUTPUTS:

measTime: array

time of each measurment point.

AOD_{wavelength}: array

AOD at wavelength.

wavelength: array

wavelength of each channel. [nm]

IWV: array

Integrated Water Vapor. [kg * m^{-2}]

angstrexp440_870: array

angstroem exponent 440-870 nm

AERONETAttri: struct

URL: char: URL to retrieve the data.
level: char: product level. (‘10’, ‘15’, ‘20’)
status: logical: status to show whether retrieve the data successfully.
IWVUnit: char: unit of integrated water vapor. [kg * m^{-2}]
location: char: AERONET site
PI: char: PI of the current AERONET site.
contact: char: email of the PI.

HISTORY:

2017-12-19: First edition by Zhenping.
2018-06-22: Add ‘TreatAsEmpty’ keyword to textscan function to filter N/A field in AERONET data.
2018-12-23: Second Edition by Zhenping
2019-02-06: Add ‘flagFilterNegAOD’ to keyword to enable filtering out negative AOD values.
2019-09-01: Enable download the AERONET data between two dates.

lib.io.readDefaultOL(file)

READDEFAULTOL Read default overlap function from file.

USAGE:

[height, overlap] = readDefaultOL(file)

INPUTS:

file: char: overlap file. The format of this file can be referred to doc/polly_defaults.md

OUTPUTS:

height: array: height for each range bin. [m]
overlap: array: overlap function.

HISTORY:

2021-05-22: first edition by Zhenping

lib.io.readGDAS1(tRange, gdas1site, folder, varargin)

READGDAS1 read gdas1 file

Example

[alt, temp, pres, relh] = readGDAS1(tRange, gdas1site, folder)

INPUTS:

tRange: 2-element array: search range.
gdas1site: char: the location for gdas1. Our server will automatically produce the gdas1 products for all our pollynet location. You can find it in /lacroshome/cloudnet/data/model/gdas1

KEYWORDS:

isUseLatestGDAS: logical: whether to search the latest available GDAS profile (default: false).

OUTPUTS:

alt: array: altitute for each range bin. [m]
temp: array: temperature for each range bin. If no valid data, NaN will be filled. [C]
pres: array: pressure for each range bin. If no valid data, NaN will be filled. [hPa]
rh: array: relative humidity for each range bin. If no valid data, NaN will be filled. [%]
wins: array: wind speed [m/s]
wind: array: wind direction. [degree]
gdas1file: char: filename of gdas1 file.

HISTORY:

2018-12-22.:First Edition by Zhenping

lib.io.readIWV(instrument, clFreTime, varargin)

READIWV read integrated water vapor from external instruments, like sunphotometer or microwave radiometer.

USAGE:

[IWV, globalAttri] = readIWV(instrument, clFreTime, varargin)

INPUTS:

instrument: char: instrument for providing integral water vapor (IWV) measurements. ‘AERONET’ or ‘MWR’
clFreTime: matrix: start and stop time of cloud free segments. [[startTime1, stopTime1]; [startTime2, stopTime2], …]

KEYWORDS:

AERONETSite: char: AERONET site (default: leipzig).
AERONETIWV: numeric: IWV from AERONET (default: []). [kg * m^{-2}]
AERONETTime: numeric: measurement time for IWV measurements (default: []).
MWRFolder: char: folder of microwave radiometer (MWR) results (default: ‘’).
MWRSite: char: microwave radiometer deployed site (default: ‘leipzig’).
maxIWVTLag: numeric: maximum temporal offset between IWV measurements and lidar measurements. (default: datenum(0,1,0,2,0,0))
PI: char: PI (default: ‘’).
contact: char: contact of PI (default: ‘’).

OUTPUTS:

IWV: array

integrated water vapor. (kg * m^{-2})

globalAttri:

source: char: data source. (‘AERONET’, ‘MWR’ or else)
site: char: measurement site.
datetime: array: datetime of applied IWV.
PI: char: PI
contact: char: contact

HISTORY:

2018-12-26: First Edition by Zhenping
2019-05-19: Fix the bug of returning empty IWV when more than 2 MWR files were found.

lib.io.readLogBook(logbookFile, nChannel)

READLOGBOOK read logbook information from logbookFile.

USAGE:

[logbook] = readLogBook(logbookFile, nChannel)

INPUTS:

logbookFile: char: filename of the logbook file.
nChannel: int32: number of all the channels.

OUTPUTS:

logbook: struct

datetime: array: datetime for applying the changes.
changes: struct: flagOverlap: logical flagWindowwipe: logical flagFlashlamps: logical flagPulsepower: logical flagRestart: logical
flag_CH_NDChange: logical matrix (IDs * nChannel): logical to show whether there is ND filter changes in certain channels.

HISTORY:

2021-06-11: first edition by Zhenping

lib.io.readMWR(files)

READMWR read integrated water vapor from the microwave radiometer outputs.

USAGE:

[tIWV, IWV, IWVErr, IWVAttri] = readMWR(files)

INPUTS:

files: char | cell: absolute paths of the netcdf files for saving the IWV results from HATPRO. Generally, you can find the data in our rsd server. Detailed information you can contact with Patric Seifert.

OUTPUTS:

IWV: array

intergrated water vapor. [kg*m^{-2}]

tIWV: array

time for each bin. [datenum]

IWVErr: float

standar deviation of IWV. [kg*m^{-2}]

IWVAttri: struct

institution: char: Institution
contact: char: contact
source: char: data source or instrument.
site: char: site

HISTORY:

2019-01-03: First Edition by Zhenping
2019-12-02: Add support for reading multiple files.

lib.io.readMeteor(measTime, varargin)

READMETEOR Read meteorological data according the input meteorological data type.

USAGE:

[alt, temp, pres, relh, attri] = readMeteor(measTime)

INPUTS:

measTime: datenum: the measurement time.

KEYWORDS:

meteorDataSource: str: meteorological data type. e.g., ‘gdas1’(default), ‘standard_atmosphere’, ‘websonde’, ‘radiosonde’
gdas1Site: str: the GDAS1 site for the current campaign.
gdas1_folder: str: the main folder of the GDAS1 profiles.
radiosondeSitenum: integer: site number, which can be found in doc/radiosonde-station-list.txt.
radiosondeFolder: str: the folder of the sonding files.
radiosondeType: integer: file type of the radiosonde file. - 1: radiosonde file for MOSAiC (default) - 2: radiosonde file for MUA
isUseLatestGDAS: logical: whether to search the latest available GDAS profile (default: false).

OUTPUTS:

alt: array

height above the mean sea surface. [m]

temp: array

temperature for each range bin. [C]

pres: array

pressure for each range bin. [hPa]

relh: array

relative humidity for each range bin. [%]

wins: array

wind speed. [m/s]

wind: array

wind direction. [°]

attri: struct

dataSource: cell: meteorological data used in the data processing for each cloud-free group.
URL: cell: data file info for each cloud-free group.
datetime: array: datetime label for the meteorlogical data.

HISTORY:

2019-07-20: First Edition by Zhenping
2020-04-16: rewrite the function interface

lib.io.readPolly1stLaserlogbook(file, flagDeleteData)

READPOLLY1STLASERLOGBOOK Read housekeeping data from the zipped laserlogbook file.

USAGE:

health = readPolly1stLaserlogbook(file, flagDeleteData)

INPUTS:

file: char: the full filename.
flagDeleteData: logical: flag to control whether to delete the laserlogbook file.

OUTPUTS:

health: struct

time: datenum array: time
AD: array: laser energy (measured inside laser head.) [a.u.]
EN: array: laser energy (measured inside laser head.) [mJ]
counts: array: flashlamp used counts.
Temp1: array: temperature for the transmitting chamber. [degree celsius]
Temp2: array: temperature for the receiving chamber. [degree celsius]
WT: array: water temperature. [degree celsius]
LS: array: laser shutter. (031010000 stands for ‘shutter open’, otherwise means ‘closed’)
Temp1064: array: 1064 PMT temperature. [degree celsius]
OutsideTemp: array: environment temperature. [degree celsius]
OutsideRH: array: environment relative humidity. [%]

HISTORY:

2021-04-12: first edition by Zhenping

lib.io.readPolly1v2Laserlogbook(file, flagDeleteData)

readPolly1v2Laserlogbook Read housekeeping data from the zipped laserlogbook file

USAGE:

[output] = readPolly1v2Laserlogbook(file, flagDeleteData)

INPUTS:

file: char: the full filename.
flagDeleteData: logical: flag to control whether to delete the laserlogbook file.

OUTPUTS:

health: struct

time: datenum array: time.
AD: array: laser energy (measured inside laser head.) [a.u.]
EN: array: laser energy (measured inside laser head.) [mJ]
counts: array: flashlamp used counts.
ExtPyro: array: raw output energy (ExtPyro). [mJ]
Temp1064: array: temperature for the PMT at 1064nm channel. [degree celsius]
Temp1: array: temperature for the transmitting chamber. [degree celsius]
Temp2: array: temperature for the receiving chamber. [degree celsius]
OutsideRH: array: RH outside the polly system. [%]
OutsideT: array: temperature outside the Polly system. [degree celsius]
roof: array: status to show whether the roof is closed.
rain: array: status to show whether it is raining.
shutter: array: status to show whether the shutter is closed.

HISTORY:

2021-04-12: first edition by Zhenping

lib.io.readPollyDefaults(deftFile, globalDeftFile)

READPOLLYDEFAULTS Read polly default settings.

USAGE:

[defts] = readPollyDefaults(deftFile)

INPUTS:

deftFile: char: absoluta path of the polly defaults file.

OUTPUTS:

defts:: default settings for polly lidar system.

HISTORY:

2021-04-10: first edition by Zhenping

lib.io.readPollyLaserlogbook(laserlogFile, varargin)

READPOLLYLASERLOGBOOK Read polly laserlog book file.

USAGE:

[laserlogs] = readPollyLaserlogbook(laserlogFile)

INPUTS:

laserlogFile: char: absolute path of the laserlog book file.

KEYWORDS:

flagDeleteData: logical: flag to control whether to delete the laserlog book file.
pollyType: char: polly type.

OUTPUTS:

laserlogs: struct

time: datenum array: time.
AD: array: laser energy (measured inside laser head.) [a.u.]
EN: array: laser energy (measured inside laser head.) [mJ]
counts: array: flashlamp used counts.
ExtPyro: array: raw output energy (ExtPyro). [mJ]
Temp1064: array: temperature for the PMT at 1064nm channel. [degree celsius]
Temp1: array: temperature for the transmitting chamber. [degree celsius]
Temp2: array: temperature for the receiving chamber. [degree celsius]
OutsideRH: array: RH outside the polly system. [%]
OutsideT: array: temperature outside the Polly system. [degree celsius]
roof: array: status to show whether the roof is closed.
rain: array: status to show whether it is raining.
shutter: array: status to show whether the shutter is closed.

HISTORY:

2021-04-10: first edition by Zhenping

lib.io.readPollyRawData(file, varargin)

READPOLLYRAWDATA Read polly raw data.

USAGE:

data = readPollyRawData(file)

INPUTS:

file: char: absolute path of the polly data.

KEYWORDS:

flagFilterFalseMShots: logical: whether to filter out profiles with false shots number.
flagCorrectFalseMShots: logical: whether to correct false shots number.
flagDeleteData: logical: flag to control whether to delete the data files after extracting the data.
dataFileFormat: char: parsing rules for polly data filename.
deltaT: numeric: integration time (in seconds) for single profile. (default: 30)

OUTPUTS:

data: struct

rawSignal: array: signal. [Photon Count]
mShots: array: number of the laser shots for each profile.
mTime: array: datetime array for the measurement time of each profile.
depCalAng: array: angle of the polarizer in the receiving channel. (>0 means calibration process starts). [degree]
zenithAng: numeric: zenith angle of the laser beam. [degree]
repRate: float: laser pulse repetition rate. [s^-1]
hRes: float: spatial resolution [m]
mSite: char: measurement site.
deadtime: matrix (channel x polynomial_orders): deadtime correction parameters.
lat: float: latitude of measurement site. [degree]
lon: float: longitude of measurement site. [degree]
alt: float: altitude of measurement site. [degree]
filenameStartTime: datenum: start time extracted from filename.

HISTORY:

2018-12-16: First edition by Zhenping.
2019-07-08: Read the ‘laser_rep_rate’.
2020-04-16: Unify the argument interface.
2021-02-03: Extract start time from polly data filename.

lib.io.readPollyXTCGELaserlogbook(file, flagDeleteData)

READPOLLYXTCGELASERLOGBOOK Read housekeeping data from the zipped laserlogbook file

USAGE:

health = readPollyXTCGELaserlogbook(file, flagDeleteData)

INPUTS:

file: char: the full filename.
flagDeleteData: logical: flag to control whether to delete the laserlogbook file.

OUTPUTS:

health: struct

time: datenum array: time.
AD: array: laser energy (measured inside laser head.) [a.u.]
EN: array: laser energy (measured inside laser head.) [mJ]
counts: array: flashlamp used counts.
Temp1: array: temperature for the transmitting chamber. [degree celsius]
Temp2: array: temperature for the receiving chamber. [degree celsius]
WT: array: water temperature. [degree celsius]
LS: array: laser shutter.
HV1064: array: high voltage for 1064. [V]

HISTORY:

2021-04-12: first edition by Zhenping

lib.io.readPollyXTDWDLaserlogbook(file, flagDeleteData)

READPOLLYXTDWDLASERLOGBOOK Read housekeeping data from the zipped laserlogbook file

USAGE:

health = readPollyXTDWDLaserlogbook(file, flagDeleteData)

INPUTS:

file: char: the full filename.
flagDeleteData: logical: flag to control whether to delete the laserlogbook file.

OUTPUTS:

health: struct

time: datenum array: time.
AD: array: laser energy (measured inside laser head.) [a.u.]
EN: array: laser energy (measured inside laser head.) [mJ]
counts: array: flashlamp used counts.
ExtPyro: array: raw output energy (ExtPyro). [mJ]
Temp1064: array: temperature for the PMT at 1064nm channel. [degree celsius]
Temp1: array: temperature for the transmitting chamber. [degree celsius]
Temp2: array: temperature for the receiving chamber. [degree celsius]
OutsideRH: array: RH outside the polly system. [%]
OutsideT: array: temperature outside the Polly system. [degree celsius]
roof: array: status to show whether the roof is closed.
rain: array: status to show whether it is raining.
shutter: array: status to show whether the shutter is closed.

HISTORY:

2018-08-05: First edition by Zhenping.
2019-08-04: Parse nearly all available information in the laserlogbook. (That’s cool.)

lib.io.readPollyXTIfTLaserlogbook(file, flagDeleteData)

READPOLLYXTIFTLASERLOGBOOK Read housekeeping data from the zipped laserlogbook file

USAGE:

health = readPollyXTIfTLaserlogbook(file, flagDeleteData)

INPUTS:

file: char: the full filename.
flagDeleteData: logical: flag to control whether to delete the laserlogbook file.

OUTPUTS:

health: struct

time: datenum array: time.
AD: array: laser energy (measured inside laser head.) [a.u.]
EN: array: laser energy (measured inside laser head.) [mJ]
counts: array: flashlamp used counts.
Temp1: array: temperature for the transmitting chamber. [degree celsius]
Temp2: array: temperature for the receiving chamber. [degree celsius]
WT: array: water temperature. [degree celsius]
LS: array: laser shutter. (031010000 stands for ‘shutter open’, otherwise means ‘closed’)
Temp1064: array: 1064 PMT temperature. [degree celsius]
OutsideTemp: array: environment temperature. [degree celsius]
OutsideRH: array: environment relative humidity. [%]

HISTORY:

2021-04-12: first edition by Zhenping

lib.io.readPollyXTLaserlogbook(file, flagDeleteData)

READPOLLYXTLASERLOGBOOK Read housekeeping data from the zipped laserlogbook file

USAGE:

health = readPollyXTLaserlogbook(file, flagDeleteData)

INPUTS:

file: char: the full filename.
flagDeleteData: logical: flag to control whether to delete the laserlogbook file.

OUTPUTS:

health: struct

time: datenum array: time.
AD: array: laser energy (measured inside laser head.) [a.u.]
EN: array: laser energy (measured inside laser head.) [mJ]
counts: array: flashlamp used counts.
ExtPyro: array: raw output energy (ExtPyro). [mJ]
Temp1064: array: temperature for the PMT at 1064nm channel. [degree celsius]
Temp1: array: temperature for the transmitting chamber. [degree celsius]
Temp2: array: temperature for the receiving chamber. [degree celsius]
OutsideRH: array: RH outside the polly system. [%]
OutsideT: array: temperature outside the Polly system. [degree celsius]
roof: array: status to show whether the roof is closed.
rain: array: status to show whether it is raining.
shutter: array: status to show whether the shutter is closed.

HISTORY:

2018-08-05: First edition by Zhenping.
2019-08-04: Parse nearly all available information in the laserlogbook.

lib.io.readSonde(file, fileType, missingValue)

READSONDE read radiosonde data from netCDF file.

USAGE:

[alt, temp, pres, relh, datetime] = readSonde(file, fileType)

INPUTS:

file: str: filename of radiosonde data file.
fileType: integer: file type of the radiosonde file. - 1: radiosonde file for MOSAiC (default) - 2: radiosonde file for MUA
missingValue: double: missing value for filling the empty bins. These values need to be replaced with NaN to be compatible with the processing program.

OUTPUTS:

alt: array: altitute for each range bin. [m]
temp: array: temperature for each range bin. If no valid data, NaN will be filled. [C]
pres: array: pressure for each range bin. If no valid data, NaN will be filled. [hPa]
relh: array: relative humidity for each range bin. If no valid data, NaN will be filled. [%]
wins: array: wind speed [m/s]
wind: array: wind direction. [degree]
datetime: datenum: datetime for the radiosonde data.

Note

The radiosonde file should be in netCDF and must contain the variable of ‘altitude’, ‘temperature’, ‘pressure’ and ‘RH’. Below is the description of each variable. (detailed information please see example in ‘..exampleconvert_radiosonde_data')

variables:

double altitude(altitude=6728);: :unit = “m”; :long_name = “Height of lidar above mean sea level”; :standard_name = “altitude”; :axis = “Z”;
double pressure(altitude=6728);: :unit = “hPa”; :long_name = “air pressure”; :standard_name = “pressure”; :_FillValue = -999.0; // double
double temperature(altitude=6728);: :unit = “degree celsius”; :long_name = “air temperature”; :standard_name = “temperature”; :_FillValue = -999.0; // double
double RH(altitude=6728);: :unit = “%”; :long_name = “relative humidity”; :standard_name = “RH”; :_FillValue = -999.0; // double

HISTORY:

2019-07-19: First Edition by Zhenping
2019-07-28: Add the criteria for empty file.
2019-12-18: Add fileType to specify the type of the radiosonde file.

lib.io.readWebsonde(measTime, tRange, sitenum)

READWEBSONDE search the closest radionsde based on the ui in http://weather.uwyo.edu/upperair/sounding.html. And read the data.

USAGE:

[alt, temp, pres, rh, wins, wind, globalAttri] = readWebsonde(measTime, tRange, sitenum)

INPUTS:

measTime: float: polly measurement time. [datenum]
tRange: 2-element array: search range for the online radiosonde. [current whole day]
sitenum: integer: site number, which can be found in doc/radiosonde-station-list.txt. You can update the list with using download_radiosonde_list.m

OUTPUTS:

alt: array: altitute for each range bin. [m]
temp: array: temperature for each range bin. If no valid data, NaN will be filled. [C]
pres: array: pressure for each range bin. If no valid data, NaN will be filled. [hPa]
rh: array: relative humidity for each range bin. If no valid data, NaN will be filled. [%]
wind: array: wind direction. [degree]
wins: array: wind speed [m/s]
globalAttri: struct: URL: URL which can be used to retrieve the current returned values. datetime: measurement time for current used sonde. [datenum] sitenum: site number for current used sonde.

HISTORY:

2021-05-24: first edition by Zhenping

lib.io.read_camp_and_config(PicassoLinkFile)

READ_CAMP_AND_CONFIG read pollynet campaign link file.

USAGE:

[PicassoCampLinks] = read_camp_and_config(PicassoLinkFile)

INPUTS:

PicassoLinkFile: char: absolute path of the pollynet config link file.

OUTPUTS:

PicassoCampLinks: struct: instrument: cell location: cell camp_starttime: array camp_stoptime: array latitude: array longitude: array asl: array config_starttime: array config_stoptime: array config_file: cell process_func: cell default_file: cell caption: cell comment: cell

HISTORY:

2021-04-07: first edition by Zhenping

lib.io.read_fileinfo_new(file)

READ_FILEINFO_NEW read new file info.

USAGE:

[fileinfo_new] = read_fileinfo_new(file)

INPUTS:

file: char: filename of the fileinfo_new which locates in todo_filelist

OUTPUTS:

fileinfo_new: struct

todoPath: cell: path of the todo_filelist
dataPath: cell: directory to the respective polly lidar data
dataFilename: cell: filename of the polly data
zipFile: cell: filename of the zipped polly data
dataSize: array: file size of the zipped polly data
pollyType: cell: polly lidar label. e.g., ‘POLLYXT_TROPOS’

HISTORY:

2021-06-13: first edition by Zhenping

lib.io.read_processed_data(instrument, parameter, hRange, tRange, resFolder)

READ_PROCESSED_DATA read processed data from pollynet processing program in the given range.

USAGE:

[time, height, res] = read_processed_data(instrument, parameter, tRange, resFolder)

INPUTS:

instrument: str

polly instrument. {‘pollyxt_lacros’, ‘pollyxt_noa’,: ‘pollyxt_tropos’, ‘pollyxt_fmi’, ‘arielle’}

parameter: str

the label of the parameter which you want to extract. data LC_aeronet_1064nm LC_aeronet_355nm LC_aeronet_532nm LC_klett_1064nm LC_klett_355nm LC_klett_532nm LC_raman_1064nm LC_raman_355nm LC_raman_532nm attenuated_backscatter_1064nm attenuated_backscatter_355nm attenuated_backscatter_532nm target_classification target_classification_V2 volume_depolarization_ratio_355nm volume_depolarization_ratio_532nm

hRange: 2-element array

vertical range. [km]

tRange: 2-element array (datenum)

start time and end time of the extraction.

resFolder: str

the folder of saving the processign results.

OUTPUTS:

time: array: measurement time.
height: array: height above ground. (m)
res: matrix (height x time): target parameters.

HISTORY:

2021-06-13: first edition by Zhenping

lib.io.saveDepolConst(dbFile, depolconst, depolconstStd, dcStartTime, dcStopTime, pollyDataFilename, pollyType, wavelength)

SAVEDEPOLCONST save polarization calibration results.

USAGE:

saveDepolConst(dbFile, depolconst, depolconstStd, dcStartTime,: dcStopTime, pollyDataFilename, pollyType, wavelength)

INPUTS:

dbFile: char: absolute path of the database.
depolconst: array: polarization calibration constants.
depolconstStd: array: uncertainty of polarization calibration constants.
dcStartTime: array: start time of each calibration period.
dcStopTime: array: stop time of each calibration period.
pollyDataFilename: char: the polly netcdf data file.
pollyType: char: polly type. (case-sensitive)
wavelength: char: wavelength. (‘355’, ‘532’ or ‘1064’)

HISTORY:

2021-06-08: first edition by Zhenping

lib.io.saveLiConst(dbFile, liconst, liconstStd, lcStartTime, lcStopTime, pollyDataFilename, pollyType, wavelength, caliMethod, telescope)

SAVELICONST save lidar calibration results.

USAGE:

saveLiConst(dbFile, liconst, liconstStd, lcStartTime, lcStopTime, pollyDataFilename, pollyType, wavelength, caliMethod)

INPUTS:

dbFile: char: absolute path of the database.
liconst: array: lidar calibration constants.
liconstStd: array: uncertainty of lidar calibration constants.
lcStartTime: array: start time of each calibration period.
lcStopTime: array: stop time of each calibration period.
pollyDataFilename: char: the polly netcdf data file.
pollyType: char: polly type. (case-sensitive)
wavelength: char: wavelength. (‘355’, ‘532’, ‘1064’, ‘387’ or ‘607’)
caliMethod: char: applied lidar calibration method. (‘Klett_Method’, ‘Raman_Method’ or ‘AOD_Constrained_Method’)
telescope: char: detection range. (‘near_range’, or ‘far_range’)

HISTORY:

2021-06-08: first edition by Zhenping

lib.io.saveWVConst(dbFile, wvconst, wvconstStd, WVCaliInfo, IWVAttri, pollyDataFilename, pollyType)

SAVEWVCONST save water vapor calibration results.

USAGE:

saveWVConst(dbFile, wvconst, wvconstStd, WVCaliInfo, IWVAttri, pollyDataFilename, pollyType)

INPUTS:

dbFile: char

absolute path of the database.

wvconst: array

water vapor calibration constants. [g*kg^{-1}]

wvconstStd: array

uncertainty of water vapor calibration constants. [g*kg^{-1}]

WVCaliInfo: struct

source: char: data source. (‘AERONET’, ‘MWR’ or else)
site: char: measurement site.
datetime: array: datetime of applied IWV.
PI: char: PI.
contact: char: contact.

IWVAttri: struct

cali_start_time: array: water vapor calibration start time. [datenum]
cali_stop_time: array: water vapor calibration stop time. [datenum]
WVCaliInfo: cell: calibration information for each calibration period.
IntRange: matrix: index of integration range for calculate the raw IWV from lidar.

pollyDataFilename: char

the polly netcdf data file.

pollyType: char

polly type. (case-sensitive)

HISTORY:

2018-12-19: First Edition by Zhenping
2019-02-12: Remove the bug for saving flagCalibration at some time with no calibration constants.
2019-08-09: Saving the real applied water vapor constant instead of the defaults. And remove the outputs of the function.

lib.io.selectDepolConst(depolconst, depolconstStd, depolCaliStartTime, depolCaliStopTime, queryTime, dbFile, pollyType, wavelength, varargin)

SELECTDEPOLCONST select the most suitable eta of polarization calibration.

USAGE:

[dcUsed, dcUsedStd, dcUsedStartTime, dcUsedStopTime] = selectDepolConst(depolconst, depolconstStd, depolCaliTime, queryTime, dbFile, pollyType, wavelength)

INPUTS:

depolconst: array: depolarization calibration constant.
depolconstStd: array: uncertainty of depolarization calibration constant.
depolCaliStartTime: array: start time for each depolarization calibration period. (datenum)
depolCaliStopTime: array: stop time for each depolarization calibration period. (datenum)
queryTime: datenum: query time for searching the depolarization calibration constant.
dbFile: char: full path of the depol calibration file.
pollyType: char: polly type. (case-sensitive)
wavelength: char: wavelength (‘355’ or ‘532’)

KEYWORDS:

flagUsePrevDepolConst: logical: flag to control whether to search for depolarization calibration constants from the database.
flagDepolCali: logical: flag to control whether to use depolarization calibration constants.
deltaTime: datenum: maximum time lapse between query time and calibration time of old calibration results.
default_polCaliEta: double: default depolarization calibration constant, which will be used if no suitable calibration results were found.
default_polCaliEtaStd: double: uncertainty of default depolarization calibration constant.

OUTPUTS:

dcUsed: double: depolarization calibration constants.
dcUsedStd: double: uncertainty of depolarization calibration constants.
dcUsedStartTime: double: depolarization calibration start time. (0 was set if there was no successful depolarization calibration.)
dcUsedStopTime: double: depolarization calibration stop time. (0 was set if there was no successful depolarization calibration.)

HISTORY:

2021-06-08: first edition by Zhenping

lib.io.selectLiConst(liconst, liconstStd, caliStartTime, caliStopTime, queryTime, dbFile, pollyType, wavelength, telescope, varargin)

SELECTLICONST select the most suitable lidar constant.

USAGE:

[lcUsed, lcUsedStd, lcUsedTag, lcUsedWarning] = selectLiConst(liconst, liconstStd, caliStartTime, caliStopTime, queryTime, dbFile, pollyType, wavelength)

INPUTS:

liconst: array: lidar calibration constants.
liconstStd: array: uncertainty of lidar constants.
caliStartTime: array: start time for each lidar calibration period. (datenum)
caliStopTime: array: stop time for each lidar calibration period. (datenum)
queryTime: datenum: query time for searching the lidar calibration constant.
dbFile: char: path of SQLite database, which contains lidar calibration results.
pollyType: char: polly type name. (case-sensitive)
wavelength: char: wavelength (‘355’, ‘532’, ‘1064’, ‘387’ or ‘607’)
telescope: char: detection range. (‘near_range’, or ‘far_range’)

KEYWORDS:

flagUsePrevLC: logical: flag to control whether to search for lidar calibration constants from the database.
flagLCCalibration: logical: flag to control whether to use lidar calibration constants.
deltaTime: datenum: maximum time lapse between query time and calibration time of old calibration results.
default_liconst: double: default lidar calibration constant, which will be used if no suitable calibration results were found.
default_liconstStd: double: uncertainty of default lidar calibration constant.

OUTPUTS:

lcUsed: double: lidar constant that will be used. (photon_count * m^3 * sr)
lcUsedStd: double: uncertainty of lidar constant. (photon_count * m^3 * sr)
lcUsedTag: integer: source of the applied lidar constant. (1: klett; 2: raman; 3: defaults; 4: history)
lcUsedWarning: logical: flag to show whether the calibration constant is unstable.

HISTORY:

2021-06-13: first edition by Zhenping

lib.io.selectWVConst(wvconst, wvconstStd, IWVAttri, queryTime, dbFile, pollyType, varargin)

SELECTWVCONST select the most appropriate water vapor calibration constant.

USAGE:

[wvconstUsed, wvconstUsedStd, wvconstUsedInfo] = selectWVConst(wvconst, wvconstStd, IWVAttri, queryTime, dbFile, pollyType)

INPUTS:

wvconst: array

water vapor calibration constants. [g*kg^{-1}]

wvconstStd: array

uncertainty of water vapor calibration constants. [g*kg^{-1}]

IWVAttri: struct

datetime: array: water vapor calibration time. [datenum]
WVCaliInfo: cell: calibration information for each calibration period.
IntRange: matrix: index of integration range for calculate the raw IWV from lidar.

queryTime: datenum

query time for searching the water vapor calibration constant.

dbFile: char

path of SQLite database, which contains water vapor calibration results.

pollyType: char

polly type name. (case-sensitive)

KEYWORDS:

flagUsePrevWVConst: logical: flag to control whether to search for the water vapor calibration constants from the database.
flagWVCalibration: logical: flag to control whether to use water vapor calibration constants.
deltaTime: datenum: maximum time lapse between query time and calibration time of old calibration results.
default_wvconst: double: default water vapor calibration constant, which will be used if no suitable calibration results were found.
default_wvconstStd: double: uncertainty of default water vapor calibration constant.

OUTPUTS:

wvconstUsed: float

applied water vapor calibration constants.[g*kg^{-1}]

wvconstUsedStd: float

uncertainty of applied water vapor calibration constants. [g*kg^{-1}]

wvconstUsedInfo: struct

flagCalibrated: logical: flag to show whether the applied constant comes from a successful calibration. If not, the result comes from the defaults.
IWVInstrument: char: the instrument for external standard IWV measurement
nIWVCali: integer: number of successful water vapor calibration.

HISTORY:

2018-12-19: First Edition by Zhenping
2019-08-16: Fix bug for taking defaults when there was no calibration instead of taking the previous calibration results.
2020-04-18: Update the interface.

lib.io.writeTodoListAuto(pollyAppConfigFile, picassoConfigFile, pollyDataBaseDir, minDataSize, tSearchStart, tSearchRange, flagCheckGDAS1)

WRITETODOLISTAUTO Search the updated polly data in the server with comparing its file size with the file size saved in the database. And also checked the GDAS1 status together if ‘flagCheckGDAS1’ was set true. The modified zipped files will be extracted to the todopath and the fileinfo_new file will be created to trigger the Picasso.

USAGE:

[flag] = writeTodoListAuto(pollyAppConfigFile, picassoConfigFile, pollyDataBaseDir, minDataSize, tSearchStart, tSearchRange, flagCheckGDAS1)

INPUTS:

pollyAppConfigFile: char: filename of the pollyAPP private configuration file. e.g., ‘~/pollyAPP/config/config.private’
picassoConfigFile: char: filename of the picasso global configuration file. e.g., ‘~/Pollynet_Processing_Chain/config/pollynet_processing_chain_config.json’
pollyDataBaseDir: char: root directory for holding polly data e.g., ‘/pollyhome’
minDataSize: integer: minumum size of the polly data to trigger the processing. [bytes]
tSearchStart: datenum: start time for searching the polly data file.
tSearchRange: datenum: search range for searching the polly data file before the tSearchStart.
flagCheckGDAS1: logical: flag to control whether to reprocess the data when GDAS1 files were ready.

OUTPUTS:

flag: logical: status for the whole process.

HISTORY:

2021-06-13: first edition by Zhenping

lib.io.write_2_donelist(file, permission, lidar, location, startTime, stopTime, last_update, lambda, imageFile, level, thisinfo, nc_zip_file, nc_zip_file_size, active, GDAS1, GDAS1_timestamp, lidar_ratio, software_version, product_type, product_starttime, product_stoptime)

WRITE_2_DONELIST Write info of each generated pic to donelist file.

USAGE:

write_2_donelist(file, permission, lidar, location, startTime, endTime, last_update, lambda, imageFile, level, thisinfo, nc_zip_file, nc_zip_file_size, active, GDAS1, GDAS1_timestamp, lidar_ratio, software_version, comment);

INPUTS:

file: char: filename of the done list file.
permission: char: file access type.
lidar: char: lidar label. Please go to /doc/pollynet.md for detailed information.
location: char: location of the current measurement campaign. This info can be found in /config/pollynet_processing_chain_link.txt
startTime: char: start time of the current measurement. (yyyy-mm-dd HH:MM:SS)
stopTime: char: stop time for the current logged data file. (yyyy-mm-dd HH:MM:SS)
last_update: char: last updated time for the image. (yyyy-mm-dd HH:MM:SS)
lambda: char: wavelength label for the image.
imageFile: char: relative directory of the image.
level: char: level of the image. (Need to be updated)
thisinfo: char: information about the image. (like which configurations you’ve used for the retrieving.)
nc_zip_file: char: data file.
nc_zip_file_size: char: size of the data file.
active: char: flag to show whether the lidar is well operated.
GDAS1: char: flag to show whether the GDAS1 data is used for the retrievign.
GDAS1_timestamp: char: timestamp for the used GDAS1 file. (yyyymmddHH)
lidar_ratio: char: lidar ratio.
software_version: char: software version
product_type: char: identification for different lidar product.
product_starttime: char: the start time for the current product. (yyyymmdd HH:MM:SS)
product_stoptime: char: the stop time for the current product. (yyyymmdd HH:MM:SS)

HISTORY:

2019-01-04. First Edition by Zhenping
2019-02-15. Add two params of ‘product_starttime’ and ‘nproduct_stoptime’.
2019-03-12. Add input parameter of ‘product_type’ according to the requirement of new pollyWebApplication.
2019-08-16. Add the criteria for ‘imageFile’. If the image doesn’t exist, throw an warning instead of writing to the done_fielist.

lib.io.write_laserlogbook(file, data, mode)

WRITE_LASERLOGBOOK create laserlogbook file with the given laserlogbook data.

USAGE:

write_laserlogbook(file, data, mode)

INPUTS:

file: char: absolute file path of the laserlogbook file.
data: struct: laserlogbook data.
mode: char: file creation mode.

HISTORY:

2021-06-13: first edition by Zhenping

Picasso pre-process

lib.preprocess.pollyDTCor(sigI, mShots, hRes, varargin)

POLLYDTCOR deadtime correction for polly photon counting data.

USAGE:

[sigO] = pollyDTCor(sigI, mShots, hRes, varargin)

INPUTS:

sigI: matrix (channel x height x time): raw photon counting signal.
mShots: matrix (channel x time): number of accumulated shots for each profile.
hRes: numeric: height resolution. (m)

KEYWORDS:

pollyType: char: polly version. (default: ‘arielle’)
flagDeadTimeCorrection: logical: flag to control whether to apply deadtime correction. (default: false)
deadtimeCorrectionMode: numeric: deadtime correction mode. (default: 2) 1: polynomial correction with parameters saved in data file. 2: non-paralyzable correction 3: polynomail correction with user defined parameters 4: disable deadtime correction
deadtimeParams: numeric: deadtime parameters. (default: [])
deadtime: matrix (channel x polynomial_orders): deadtime correction parameters.

OUTPUTS:

sigO: matrix (channel x height x time): deadtime corrected signal in photon count.

HISTORY:

2021-05-16: first edition by Zhenping

lib.preprocess.pollyPreprocess(data, varargin)

POLLYPREPROCESS Deadtime correction, background correction, first-bin shift, mask for low-SNR and mask for depolarization-calibration process.

USAGE:

[data] = pollyPreprocess(data)

INPUTS:

data: struct

rawSignal: array: signal. [Photon Count]
mShots: array: number of the laser shots for each profile.
mTime: array: datetime array for the measurement time of each profile.
depCalAng: array: angle of the polarizer in the receiving channel. (>0 means calibration process starts)
zenithAng: array: zenith angle of the laer beam.
repRate: float: laser pulse repetition rate. [s^-1]
hRes: float: spatial resolution [m]
mSite: string: measurement site.

KEYWORDS:

deltaT: numeric: integration time (in seconds) for single profile. (default: 30)
flagForceMeasTime: logical: flag to control whether to align measurement time with file creation time, instead of taking the measurement time in the data file. (default: false)
maxHeightBin: numeric: number of range bins to read out from data file. (default: 3000)
firstBinIndex: numeric: index of first bin to read out. (default: 1)
pollyType: char: polly version. (default: ‘arielle’)
flagDeadTimeCorrection: logical: flag to control whether to apply deadtime correction. (default: false)
deadtimeCorrectionMode: numeric: deadtime correction mode. (default: 2) 1: polynomial correction with parameters saved in data file. 2: non-paralyzable correction 3: polynomail correction with user defined parameters 4: disable deadtime correction
deadtimeParams: numeric: deadtime parameters. (default: [])
flagSigTempCor: logical: flag to implement signal temperature correction.
tempCorFunc: cell: symbolic function for signal temperature correction. “1”: no correction “exp(-0.001*T)”: exponential correction function. (Unit: Kelvin)
meteorDataSource: str: meteorological data type. e.g., ‘gdas1’(default), ‘standard_atmosphere’, ‘websonde’, ‘radiosonde’
gdas1Site: str: the GDAS1 site for the current campaign.
gdas1_folder: str: the main folder of the GDAS1 profiles.
radiosondeSitenum: integer: site number, which can be found in doc/radiosonde-station-list.txt.
radiosondeFolder: str: the folder of the sonding files.
radiosondeType: integer: file type of the radiosonde file. - 1: radiosonde file for MOSAiC (default) - 2: radiosonde file for MUA
bgCorrectionIndex: 2-element array: base and top index of bins for background estimation. (defaults: [1, 2])
asl: numeric: above sea level in meters. (default: 0)
initialPolAngle: numeric: initial polarization angle of the polarizer for polarization calibration. (default: 0)
maskPolCalAngle: cell: mask for positive and negative calibration angle of the polarizer, in which ‘p’ stands for positive angle, while ‘n’ for negative angle. (default: {})
minSNRThresh: numeric: lower bound of signal-noise ratio.
minPC_fog: numeric: minimun number of photon count after strong attenuation by fog.
flagFarRangeChannel: logical: flags of far-range channel.
flag532nmChannel: logical: flags of channels with central wavelength (CW) at 532 nm.
flagTotalChannel: logical: flags of channels receiving total elastic signal.
flag355nmChannel: logical: flags of channels with CW at 355 nm.
flag607nmChannel: logical: flags of channels with CW at 607 nm.
flag387nmChannel: logical: flags of channels with CW at 387 nm.
flag407nmChannel: logical: flags of channels with CW at 407 nm.
flag532nmRotRaman: logical: flags of rotational Raman channels with CW at 532 nm.
flag1064nmRotRaman: logical: flags of rotational Raman channels with CW at 1064 nm.

OUTPUTS:

data: struct

rawSignal: array: signal. [Photon Count]
mShots: array: number of the laser shots for each profile.
mTime: array: datetime array for the measurement time of each profile.
depCalAng: array: angle of the polarizer in the receiving channel. (>0 means calibration process starts)
zenithAng: array: zenith angle of the laer beam.
repRate: float: laser pulse repetition rate. [s^-1]
hRes: float: spatial resolution [m]
mSite: string: measurement site.
deadtime: matrix (channel x polynomial_orders): deadtime correction parameters.
signal: array: Background removed signal
bg: array: background
height: array: height. [m]
lowSNRMask: logical: If SNR less SNRmin, mask is set true. Otherwise, false
depCalMask: logical: If polly was doing polarization calibration, depCalMask is set true. Otherwise, false.
fogMask: logical: If it is foggy which means the signal will be very weak, fogMask will be set true. Otherwise, false
mask607Off: logical: mask of PMT on/off status at 607 nm channel.
mask387Off: logical: mask of PMT on/off status at 387 nm channel.
mask407Off: logical: mask of PMT on/off status at 407 nm channel.
mask355RROff: logical: mask of PMT on/off status at 355 nm rotational Raman channel.
mask532RROff: logical: mask of PMT on/off status at 532 nm rotational Raman channel.
mask1064RROff: logical: mask of PMT on/off status at 1064 nm rotational Raman channel.

HISTORY:

2018-12-16: First edition by Zhenping.
2019-07-10: Add mask for laser shutter due to approaching airplanes.
2019-08-27: Add mask for turnoff of PMT at 607 and 387nm.
2021-01-19: Add keyword of ‘flagForceMeasTime’ to align measurement time.
2021-01-20: Re-sample the profiles into temporal resolution of 30-s.

lib.preprocess.pollyRemoveBG(sigI, varargin)

POLLYREMOVEBG remove signal background.

USAGE:

[sigO, bg] = pollyRemoveBG(sigI, varargin)

INPUTS:

sigI: matrix (channel x height x time): lidar signal.

KEYWORDS:

maxHeightBin: numeric: number of range bins to read out from data file. (default: 3000)
firstBinIndex: numeric: index of first bin to read out. (default: 1)
bgCorrectionIndex: 2-element array: base and top index of bins for background estimation. (defaults: [1, 2])

OUTPUTS:

sigO: matrix (channel x height x time): background-substracted signal.
bg: matrix (channel x height x time): background.

HISTORY:

2021-05-16: first edition by Zhenping

Picasso quality control (QC)

lib.qc.olCor(sigFR, overlap, height, normRange)

OLCOR overlap correction.

USAGE:

sigCor = olCor(sigFR, overlap, height, normRange)

INPUTS:

sigFR: matrix (height * time): far-range signal
overlap: array: overlap function.
height: array: height above ground. (m)
normRange: array: signal normalization range. (m)

OUTPUTS:

sigCor: matrix (height * time): glued signal.

HISTORY:

2021-05-22: first edition by Zhenping

lib.qc.overlapCalc(height, sigFR, bgFR, sigNR, bgNR, varargin)

OVERLAPCALC calculate overlap function.

USAGE:

overlap = overlapCalc(sigFR, bgFR, sigNR, bgNR, height);

INPUTS:

height: array: height above ground. (m)
sigFR: array: far-range signal. (photon count)
bgFR: array: background of far-range signal. (photon count)
sigNR: array: near-range signal. (photon count)
bgNR: array: background of near-range signal. (photon count)

KEYWORDS:

hFullOverlap: numeric: minimum height with full overlap function for far-range signal (default: 600). (m)
overlapCalMode: integer: overlap calculation mode. 1: signal ratio between near-range and far-range channel 2: Raman method

OUTPUTS:

overlap: array: overlap function. If no overlap function was calculated, overlap will be empty.
overlapStd: array: error of the overlap function. If no overlap function was calculated, overlapStd will be empty.
sigRatio: numeric: signal ratio between near-range and far-range signal.
normRange: 2-element array: height index of the signal normalization range.

HISTORY:

2021-05-18: first edition by Zhenping

lib.qc.pollyOLCor(height, sigFR, bgFR, varargin)

POLLYOLCOR overlap correction.

USAGE:

[sigOLCor, bgOLCor, olFuncDeft, flagOLDeft] = pollyOLCor(height, sigFR, bgFR)

INPUTS:

height: array: height above ground. (m)
sigFR: array: far-field channel signal.
bgFR: array: far-field channel background

KEYWORDS:

signalNR: array: near-field channel signal
bgNR: array: near-field channel signal
signalRatio: numeric: ratio between near-field and far-field signal.
normRange: 2-element array: normalization range (index) for signal ratio between near-field and far-field signal.
defaultOLFile: char: absolute path of default overlap file.
overlapCorMode: numeric: overlap correction mode. 0: no overlap correction; 1:overlap correction with using the default overlap function; 2: overlap correction with using the calculated overlap function; 3: overlap correction with gluing near-range and far-range signal
overlapSmWin: numeric: smoothing window for overlap function (in bins)
overlap: array: overlap function.

OUTPUTS:

sigOLCor: array: overlap corrected signal.
bgOLCor: array: overlap corrected background.
olFuncDeft: default overlap function.
flagOLDeft: flag to determine whether default overlap function was applied in the overlap correction.

HISTORY:

2021-05-22: first edition by Zhenping

lib.qc.pollyOVLCalc(height, sigFR, sigNR, bgFR, bgNR, varargin)

POLLYOVLCALC calculate overlap function from polly measurements.

USAGE:

[olFunc, olStd, olAttri] = pollyOVLCalc(height, sigFR, sigNR, bgFR, bgNR)

INPUTS:

height: array: height above ground. (m)
sigFR: array: far-field signal.
sigNR: array: near-field signal.
bgFR: array: far-field background.
bgNR: array: near-field background.

KEYWORDS:

hFullOverlap: numeric: minimum height with complete overlap (default: 600). (m)
overlapCalMode: numeric: overlap calculation mode. (default: 1) 0: no overlap correction 1:overlap correction with using the default overlap function 2: overlap correction with using the calculated overlap function 3: overlap correction with gluing near-range and far-range signal
PC2PCR: numeric: conversion factor from photon count to photon count rate (default: 1).

OUTPUTS:

olFunc: numeric

overlap function.

olStd: numeric

standard deviation of overlap function.

olAttri: struct

sigFR: numeric: far-field signal.
sigNR: numeric: near-field signal.
sigRatio: numeric: signal ratio of near-field and far-field signal.
normRange: 2-element array: normalization range.

HISTORY:

2021-05-20: first edition by Zhenping

lib.qc.pollySaturationDetect(data, varargin)

POLLYSATURATIONDETECT detect the bins which is fully saturated by the clouds.

USAGE:

[flag] = pollySaturationDetect(data)

INPUTS:

data: struct: More detailed information can be found in docpollynet_processing_program.md

KEYWORDS:

hFullOverlap: double: minimum height with full overlap (m). (default: 500)
sigSaturateThresh: double: threshold of saturated signal (photon count). (default: 500)

OUTPUTS:

flag: logical matrix: if it is true, it means the current range bin should be saturated by clouds. Vice versa.

HISTORY:

2018-12-21: First Edition by Zhenping
2019-07-08: Fix the bug of converting signal to PCR.

lib.qc.sigGlue(sigFR, sigNR, sigRatio, height, normRange)

SIGGLUE glue near-range and far-range signal.

USAGE:

sigGl = sigGlue(sigFR, sigNR, height, normRange)

INPUTS:

sigFR: matrix (height * time): far-range signal
sigNR: matrix (height * time): near-range signal.
sigRatio: numeric: ratio of lidar constants between near-range and far-range signal.
height: array: height above ground. (m)
normRange: array: signal normalization range. (m)

OUTPUTS:

sigGl: matrix (height * time): glued signal.

HISTORY:

2021-05-22: first edition by Zhenping

lib.qc.transCor(sigT, bgT, sigC, bgC, varargin)

TRANSCOR correct the effect of different transmission inside the Tot and depol channel.

USAGE:

[sigTCor, bgTCor, sigTCorStd] = transCor(sigEl, bgT, sigC, bgC)

INPUTS:

sigT: array: signal in total channel.
bgT: array: background in total channel.
sigC: array: signal in cross channel.
bgC: array: background in total channel.

KEYWORDS:

transRatioTotal: float: transmission ratio of perpendicular and parallel component in total channel.
transRatioTotalStd: float: uncertainty of the transmission ratio of perpendicular and parallel componnet in total channel.
transRatioCross: float: transmission ratio of perpendicular and parallel component in cross channel.
transRatioCrossStd: float: uncertainty of the transmission ratio of perpendicular and parallel componnet in total channel.
polCaliFactor: float: depolarization calibration constant.
polCaliFacStd: float: uncertainty of the depolarization calibration constant.

OUTPUTS:

sigTCor: array: transmission corrected elastic signal.
bgTCor: array: background of transmission corrected elastic signal.
sigTCorStd: array: uncertainty of transmission corrected elastic signal.

References

Mattis, I., Tesche, M., Grein, M., Freudenthaler, V., and Müller, D.: Systematic error of lidar profiles caused by a polarization-dependent receiver transmission: Quantification and error correction scheme, Appl. Opt., 48, 2742-2751, 2009.

HISTORY:

2021-05-27: first edition by Zhenping.

Picasso retrievals

lib.retrievals.pollyAE(param1, param1_std, param2, param2_std, wavelength1, wavelength2, smoothWindow)

POLLYAE calculate the angstroem exponent and its uncertainty.

USAGE:

[angexp, angexpStd] = pollyAE(param1, param1_std, param2, param2_std, wavelength1, wavelength2)

INPUTS:

param1: array: extinction or backscatter coefficient at wavelength1.
param1_std: array: uncertainty of param1.
param2: array: extinction or backscatter coefficient at wavelength2.
param2_std: array: uncertainty of param2.
wavelength1: float: the wavelength for the input parameter 1. [nm]
wavelength2: float: the wavelength for the input parameter 2. [nm]

OUTPUTS:

angexp: array: angstroem exponent based on param1 and param2
angexpStd: array: uncertainty of angstroem exponent.

HISTORY:

2021-05-31: first edition by Zhenping

lib.retrievals.pollyConstrainedKlett(height, signal, SNR, refHeight, refBeta, molBsc, maxIterations, minLR, maxLR, AOD_AERONET, minAODDev, minHeight, minSNR, window_size)

POLLYCONSTRAINEDKLETT retrieve the aerosol backscatter coefficient with AOD constrain from AERONET.

USAGE:

[aerBsc, bestLR, biasAOD, nIters] = pollyConstrainedKlett(height, signal, SNR, refHeight, refBeta, molBsc, maxIterations, minLR, maxLR, AOD_AERONET, minAODDev, minHeight, minSNR, window_size)

INPUTS:

height: array: height. [m]
signal: array: measured signal.
SNR: array: signal-noise-ratio
refHeight: 2-element array: reference height. [m]
refBeta: float: reference value. [m^{-1}sr^{-1}]
molBsc: array: molecular backscatter coefficient. [m^{-1}*sr^{-1}]
maxIterations: integer: maximum number of the iterations.
minLR: float: minimum aerosol Lidar ratio. [sr]
maxLR: float: maximum aerosol Lidar ratio. [sr]
AOD_AERONET: float: AOD measured by AERONET.
minAODDev: float: minimum deviation of AOD between Lidar and AERONET that can be tolerable.
LR: float: aerosol lidar ratio. [sr]
minHeight: float: minimum height for the full FOV. [m]
minSNR: float: minimum SNR
window_size: integer: smoothing window size.

OUTPUTS:

aerBsc: array: aerosol backscatter coefficient. [m^{-1}sr^{-1}]
bestLR: float: the best lidar ratio with the AOD deviation less that required.
biasAOD: float: deviation of AOD between lidar and AERONET.
nIters: integer: number of iterations to achieve the minimun AOD deviation.

HISTORY:

2018-02-03: First edition by Zhenping
2019-03-29: Fix the bug of returning NaN for lidar ratio.
2019-04-09: Screen out the negative backscatter coefficient during the calculation.

lib.retrievals.pollyFernald(alt, signal, bg, LR_aer, refAlt, refBeta, molBsc, window_size)

POLLYFERNALD retrieve aerosol backscatter coefficient with Fernald method.

USAGE:

[aerBsc, aerBscStd, aerBR, aerBRStd] = pollyFernald(alt, signal, bg, LR_aer, refAlt, refBeta, molBsc, window_size)

INPUTS:

alt: array: height. [m]
signal: array: elastic signal without background. [Photon Count]
bg: array: background. [Photon count]
LR_aer: float or array: aerosol lidar ratio. [sr]
refAlt: float or 2-element array: reference altitude or region. [m]
refBeta: float: aerosol backscatter coefficient at the reference region. [m^{-1}sr^{-1}]
molBsc: array: molecular backscattering coefficient. Unit: m^{-1}*sr^{-1}
window_size: int32: Bins of the smoothing window for the signal. [default, 40 bins]

OUTPUTS:

aerBsc: array: aerosol backscatter coefficient. [m^{-1}*sr^{-1}]
aerBscStd: array: statistical uncertainty of aerosol backscatter. [m^{-1}*sr^{-1}]
aerBR: array: aerosol backscatter ratio.
aerBRStd: array: statistical uncertainty of aerosol backscatter ratio.

References

Fernald, F. G.: Analysis of atmospheric lidar observations: some comments, Appl. Opt., 23, 652-653, 10.1364/AO.23.000652, 1984.

HISTORY:

2021-05-30: first edition by Zhenping

lib.retrievals.pollyLR(aerExt, aerBsc, varargin)

POLLYLR calculate aerosol lidar ratio.

USAGE:

[aerLR, effRes, aerLRStd] = pollyLR(aerExt, aerBsc)

INPUTS:

aerExt: numeric: aerosol extinction coefficient. (m^-1)
aerBsc: numeric: aerosol backscatter coefficient. (m^-1sr^-1)

KEYWORDS:

hRes: numeric: vertical resolution of each height bin. (m)
aerExtStd: numeric: uncertainty of aerosol extinction coefficient. (m^-1)
aerBscStd: numeric: uncertainty of aerosol backscatter coefficient. (m^-1sr^-1)
smoothWinExt: numeric: applied smooth window length for calculating aerosol extinction coefficient.
smoothWinBsc: numeric: applied smooth window length for calculating aerosol backscatter coefficient.

OUTPUTS:

aerLR: numeric: aerosol lidar ratio. (sr)
effRes: numeric: effective resolution of lidar ratio. (m)
aerLRStd: numeric: uncertainty of aerosol lidar ratio. (sr)

References

Mattis, I., D’Amico, G., Baars, H., Amodeo, A., Madonna, F., and Iarlori, M.: EARLINET Single Calculus Chain–technical–Part 2: Calculation of optical products, Atmospheric Measurement Techniques, 9, 3009-3029, 2016.

HISTORY:: 2021-07-20: first edition by Zhenping

lib.retrievals.pollyPDR(volDepol, volDepolStd, aerBsc, aerBscStd, molBsc, molDepol, molDepolStd)

POLLYPDR calculate the particle depolarization ratio and estimate the standard deviation of particle depolarization ratio.

USAGE:

[parDepol, parDepolStd] = pollyPDR(volDepol, volDepolStd, aerBsc, aerBscStd, molBsc, molDepol, molDepolStd)

INPUTS:

volDepol: array: volume depolarization ratio.
volDepolStd: array: standard deviation of volume depolarization ratio.
aerBsc: array: aerosol backscatter coefficient. [m^{-1}Sr^{-1}]
aerBscStd: array: standard deviation of aerosol backscatter coefficient. [m^{-1}Sr^{-1}]
molBsc: array: molecule backscatter coefficient. [m^{-1}Sr^{-1}]
molDepol: scalar: molecule depolarization ratio. This value is highly dependent on the central wavelength and FWHM of the narrow IF in the depol channel.
molDepolStd: scalar: standard deviation of molecule depolarization ratio.

OUTPUTS:

parDepol: array: particle depolarization ratio.
parDepolStd: array: standard deviation of particle depolarization ratio.

References

Freudenthaler, V., Esselborn, M., Wiegner, M., Heese, B., Tesche, M., Ansmann, A., Müller, D., Althausen, D., Wirth, M., and Fix, A.: Depolarization ratio profiling at several wavelengths in pure Saharan dust during SAMUM 2006, Tellus B, 61, 165-179, 2009.

HISTORY:: 2021-05-31: first edition by Zhenping

lib.retrievals.pollyRamanBsc(height, sigElastic, sigVRN2, ext_aer, angstroem, ext_mol, beta_mol, HRef, wavelength, betaRef, window_size, flagSmoothBefore)

POLLYRAMANBSC calculate the aerosol backscatter coefficient with Raman method.

USAGE:

[ beta_aer, LR ] = pollyRamanBsc(height, sigElastic, sigVRN2, ext_aer, ext_mol, beta_mol, HRef, wavelength, betaRef, window_size, flagSmoothBefore)

INPUTS:

height: array: height. [m]
sigElastic: array: elastic photon count signal.
sigVRN2: array: N2 vibration rotational raman photon count signal.
ext_aer: array: aerosol extinction coefficient. [m^{-1}]
angstroem: array: aerosol angstroem exponent.
ext_mol: array: molecular extinction coefficient. [m^{-1}]
beta_mol: array: molecular backscatter coefficient. [m^{-1}Sr^{-1}]
HRef: 2 element array: reference region. [m]
wavelength: integer: wavelength of the corresponding elastic signal. [nm]
betaRef: float: aerosol backscatter coefficient at the reference region. [m^{-1}Sr^{-1}]
window_size: integer: number of the bins of the sliding window for the signal smooth. [default: 40]
flagSmoothBefore: logical: flag bit to control whether to smooth the signal before or after calculating the signal ratio.

OUTPUTS:

beta_aer: array: aerosol backscatter coefficient. [m^{-1}Sr^{-1}]
LR: array: aerosol lidar ratio.

References

netcdf-florian Ansmann, A., et al. (1992). “Independent measurement of extinction and backscatter profiles in cirrus clouds by using a combined Raman elastic-backscatter lidar.” Applied optics 31(33): 7113-7131.

HISTORY:

2018-01-02: First edition by Zhenping.
2018-07-24: Add the ext_mol_factor and ext_aer_factor for wavelength of 1064nm
2018-09-04: Change the smoothing order. Previous way is smoothing the signal. This will create large drift at the signal ridges.
2018-09-05: Keep the original smoothing order for 355, which makes the retrieving results at the far range bins quite stable.

lib.retrievals.pollyRamanBscStd(height, sigElastic, bgElastic, sigVRN2, bgVRN2, ext_aer, sigma_ext_aer, angstroem, sigma_angstroem, ext_mol, beta_mol, HRef, wavelength, betaRef, smoothWindow, nSamples, method, flagSmoothBefore)

POLLYRAMANBSCSTD calculate uncertainty of aerosol backscatter coefficient with Monte-Carlo simulation.

USAGE:

[ aerBscStd ] = pollyRamanBscStd(height, sigElastic, bgElastic, …: sigVRN2, bgVRN2, ext_aer, sigma_ext_aer, angstroem, … sigma_angstroem, ext_mol, beta_mol, HRef, wavelength, … betaRef, smoothWindow, nSamples, method, flagSmoothBefore)

INPUTS:

height: array: height. [m]
sigElastic: array: elastic photon count signal.
bgElastic: array: background of elastic signal.
sigVRN2: array: N2 vibration rotational raman photon count signal.
bgVRN2: array: background of N2 vibration rotational signal.
ext_aer: array: aerosol extinction coefficient. [m^{-1}]
sigma_ext_aer: array: uncertainty of aerosol extinction coefficient. [m^{-1}]
angstroem: array: aerosol angstroem exponent.
ext_mol: array: molecular extinction coefficient. [m^{-1}]
beta_mol: array: molecular backscatter coefficient. [m^{-1}Sr^{-1}]
HRef: 2 element array: reference region. [m]
wavelength: integer: wavelength of the corresponding elastic signal. [nm]
betaRef: float: aerosol backscatter coefficient at the reference region. [m^{-1}Sr^{-1}]
smoothWindow: integer or n*3 matrix: number of the bins of the sliding window for the signal smooth. [default: 40]
nSamples: scalar or matrix: samples for each error source. [samples_angstroem, samples_aerExt, samples_signal, samples_aerBscRef]
method: char: computational method. [‘monte-carlo’ or ‘analytical’]
flagSmoothBefore: logical: flag to control the smooth order.

OUTPUTS:

aerBscStd: array: uncertainty of aerosol backscatter coefficient. [m^{-1}Sr^{-1}]

HISTORY:

2021-07-16: first edition by Zhenping

lib.retrievals.pollyRamanExt(height, sig, lambda_emit, lambda_rec, angstrom, pressure, temperature, window_size, C, rh, method, measure_error)

POLLYRAMANEXT etrieve the aerosol extinction coefficient with Raman method

USAGE:

[ ext_aer ] = pollyRamanExt(height, sig, lambda_emit, lambda_rec, angstrom, pressure, temperature, window_size, C, rh, method, measure_error)

INPUTS:

height: array: height[m]
sig: array: measured raman signal. Unit: Photon Count
lambda_emit: float: the wavelength of the emitted laser beam.[nm]
lambda_rec: float: the wavelength of raman sigal.[nm]
angstrom: float: the angstrom exponent for aerosol extinction coefficient
pressure: array: pressure of the atmosphere. [hPa]
temperature: array: temperature of the atmosphere. [K]
window_size: integer: window_size for smoothing the signal with sgolay filter.
order: integer: order of the implemented sgolay filter.
C: array: CO2 concentration.[ppmv]
rh: array: relative humidity.
method: char: specify the method to calculate the slope of the signal. You can choose from ‘moving’, ‘smoothing’ and ‘chi2’.
measure_error: array: measurement error for each bin.

OUTPUTS:

ext_aer: array: aerosol extinction coefficient [m^{-1}]

References

https://bitbucket.org/iannis_b/lidar_processing

Ansmann, A. et al. Independent measurement of extinction and backscatter profiles in cirrus clouds by using a combined Raman elastic-backscatter lidar. Applied Optics Vol. 31, Issue 33, pp. 7113-7131 (1992)

HISTORY:

2021-05-31: first edition by Zhenping

lib.retrievals.pollyRamanExtStd(height, signal, bg, lambda_emit, lambda_rec, angstrom, pressure, temperature, window_size, C, rh, nProfiles, method, measure_error)

POLLYRAMANEXTSTD calcualte uncertainty of aerosol extinction coefficient with Monte-Carlo simulation.

USAGE:

[ext_std] = pollyRamanExtStd(height, signal, bg, lambda_emit, lambda_rec, angstrom, pressure, temperature, window_size, C, rh, nProfiles, method, measure_error)

INPUTS:

height: array: height[m]
signal: array: measured raman signal. [Photon Count]
bg: array: background. [Photon Count]
lambda_emit: float: the wavelength of the emitted laser beam.[nm]
lambda_rec: float: the wavelength of raman sigal.[nm]
angstrom: float: the angstrom exponent for aerosol extinction coefficient
pressure: array: pressure of the atmosphere. [hPa]
temperature: array: temperature of the atmosphere. [K]
window_size: integer: window_size for smoothing the signal.
C: array: CO2 concentration.[ppmv]
rh: array: relative humidity.
nProfiles: integer: number of the generated profiles to calculate the uncertainty.

OUTPUTS:

ext_std: array: uncertainty of aerosol extinction coefficient [m^{-1}]

HISTORY:

2021-07-16: first edition by Zhenping

lib.retrievals.pollySNR(signal, bg)

POLLYSNR calculate signal-noise ratio (SNR).

USAGE:

[SNR] = pollySNR(signal, bg)

INPUTS:

signal: array: signal strength.
bg: array: background. (bg should have the same size as signal)

OUTPUTS:

SNR: array: signal-noise-ratio. For negative signal, the SNR was set to be 0.

REFERENCES: Heese, B., Flentje, H., Althausen, D., Ansmann, A., and Frey, S.: Ceilometer lidar comparison: backscatter coefficient retrieval and signal-to-noise ratio determination, Atmospheric Measurement Techniques, 3, 1763-1770, 2010.

HISTORY:

2021-04-21: first edition by Zhenping

lib.retrievals.pollyVDR(sigTot, bgTot, sigCross, bgCross, Rt, RtStd, Rc, RcStd, depolConst, depolConstStd, smoothWindow, flagSmoothBefore)

POLLYVDR calculate volume depolarization ratio.

USAGE:

[volDepol, volDepolStd] = pollyVDR(sigTot, bgTot, sigCross, bgCross, Rt, RtStd, Rc, RcStd, depolConst, depolConstStd, smoothWindow)

INPUTS:

sigTot: array: signal strength of the total channel. [photon count]
bgTot: array: background of the total channel. [photon count]
sigCross: array: signal strength of the cross channel. [photon count]
bgCross: array: background of the cross channel. [photon count]
Rt: scalar: transmission ratio in total channel
RtStd: scalar: uncertainty of the transmission ratio in total channel
Rc: scalar: transmission ratio in cross channel
RcStd: scalar: uncertainty of the transmission ratio in cross channel
depolConst: scalar: depolarzation calibration constant. (transmission ratio for the parallel component in cross channel and total channel)
depolConstStd: scalar: uncertainty of the depolarization calibration constant.
smoothWindow: scalar or m*3 matrix: the width of the sliding smoothing window for the signal.
flagSmoothBefore: logical: flag to control the vol-depol smoothing whether before or after the signal ratio.

OUTPUTS:

volDepol: array: volume depolarization ratio.
volDepolStd: array: uncertainty of the volume depolarization ratio

REFERENCE:

Engelmann, R. et al. The automated multiwavelength Raman polarization and water-vapor lidar Polly XT: the neXT generation. Atmospheric Measurement Techniques 9, 1767-1784 (2016). Freudenthaler, V. et al. Depolarization ratio profiling at several wavelengths in pure Saharan dust during SAMUM 2006. Tellus B 61, 165-179 (2009).

HISTORY:

2018-09-02: First edition by Zhenping
2018-09-04: Change the smoothing order. Smoothing the signal ratio instead of smoothing the signal.
2019-05-24: Add ‘flagSmoothBefore’ to control the smoothing order.

lib.retrievals.pollyVDR2(sigTot, sigCross, Rt, Rc, depolConst)

POLLYVDR2 calculate 2-dimensional volume depolarization ratio for pollyXT system.

USAGE:

[volDepol] = pollyVDR2(sigTot, sigCross, Rt, Rc, depolConst)

INPUTS:

sigTot: array: signal strength of the total channel. [photon count]
sigCross: array: signal strength of the cross channel. [photon count]
Rt: scalar: transmission ratio in total channel
Rc: scalar: transmission ratio in cross channel
depolConst: scalar: depolarzation calibration constant. (transmission ratio for the parallel component in cross channel and total channel)

OUTPUTS:

volDepol: array: volume depolarization ratio.

REFERENCE:

Engelmann, R. et al. The automated multiwavelength Raman polarization and water-vapor lidar Polly XT: the neXT generation. Atmospheric Measurement Techniques 9, 1767-1784 (2016). Freudenthaler, V. et al. Depolarization ratio profiling at several wavelengths in pure Saharan dust during SAMUM 2006. Tellus B 61, 165-179 (2009).

HISTORY:

2021-06-04: first edition by Zhenping

lib.retrievals.quasiRetrieval(height, att_beta, molExt, molBsc, LRaer, varargin)

QUASIRETRIEVAL Retrieve the aerosol optical properties with quasi retrieving method

USAGE:

[quasi_par_bsc, quasi_par_ext] = quasiRetrieval(height, att_beta, …: molExt, molBsc, LRaer)

INPUTS:

height: array: height. [m]
att_beta: matrix: attenuated backscatter. [m^{-1}Sr^{-1}]
molExt: matrix: molecule extinction coefficient. [m^{-1}]
molBsc: matrix: molecule backscatter coefficient. [m^{-1}Sr^{-1}]
LRaer: float: aerosol lidar ratio. [Sr]

KEYWORDS:

nIters: numeric: iteration times.

OUTPUTS:

quasi_par_bsc: matrix: quasi particle backscatter coefficient. [m^{-1}Sr^{-1}]
quasi_par_ext: matrix: quasi particle extinction coefficient. [m^{-1}]

References

Baars, H., Seifert, P., Engelmann, R. & Wandinger, U. Target categorization of aerosol and clouds by continuous multiwavelength-polarization lidar measurements. Atmospheric Measurement Techniques 10, 3175-3201, doi:10.5194/amt-10-3175-2017 (2017).

HISTORY:

2018-12-25: First Edition by Zhenping
2019-03-31: Add the keywork of ‘nIters’ to control the iteration times.

lib.retrievals.quasiRetrieval2(height, att_beta_el, att_beta_ra, wavelength, molExtEl, molBscEl, molExtRa, AE, LR, varargin)

QUASIRETRIEVAL2 Retrieve the aerosol optical properties with quasi retrieving method (V2). This method was improved from the quasi-retrieving method, which also takes use of the Raman signal.

USAGE:

[quasi_par_bsc, quasi_par_ext] = quasiRetrieval2(height, … att_beta_el, att_beta_ra, wavelength, molExtEl, molBscEl, molExtRa, … AE, LR)

INPUTS:

height: array: height. [m]
att_beta_el: matrix: attenuated backscatter at elastic wavelength. [m^{-1}sr^{-1}]
att_beta_ra: matrix: attenuated backscatter at corresponding Raman backscatter wavelength. [m^{-1}sr^{-1}]
wavelength: integer: the wavelength of the elastic backscatter to guide to choose the suitable Raman wavelength. [nm]
molExtEl: matrix: molecule extinction coefficient at the Elastic wavelength. [m^{-1}]
molBscEl: matrix: molecule backscatter coefficient at the Elastic wavelength. [m^{-1}sr^{-1}]
molExtRa: matrix: molecule extinction coefficient at the Raman backscatter wavelength. [m^{-1}]
AE: float: Extinction related Angstroem exponent.
LR: float: aerosol lidar ratio. [sr]

KEYWORDS:

nIters: numeric: iteration times.

OUTPUTS:

quasi_par_bsc: matrix: quasi particle backscatter coefficient. [m^{-1}sr^{-1}]
quasi_par_ext: matrix: quasi particle extinction coefficient. [m^{-1}]

HISTORY:

2021-06-07: first edition by Zhenping

lib.retrievals.sigGenWithNoise(signal, noise, nProfile, method)

SIGGENWITHNOISE generate the noise containing signal with certain noise adding algorithm.

USAGE:

[signalGen] = sigGenWithNoise(signal, noise, nProfile, method)

INPUTS:

signal: array: signal strength.
noise: array: noise. Unit should be keep the same with signal.
nProfile: array: number of signal profiles should be generated.
method: char: ‘norm’: normal distributed noise -> signalGen = signal + norm * noise ‘poisson’: poisson distributed noise -> signal = poisson(signal, nProfile)

OUTPUTS:

signalGen: matrix length(signal) * nProfile: noise containing signal.

HISTORY:

2021-06-13: first edition by Zhenping

lib.retrievals.signalStd(signal, bg, smoothWindow, dimension)

SIGNALSTD The uncertainty of the signal with taking into account of background noise and additional smoothing.

USAGE:

[sigStd] = signalStd(signal, bg, smoothWindow, dimension)

INPUTS:

signal: array: signal strength
bg: array: background
smoothWindow: scalar or matrix: the width of the smoothing window. If smoothWindow is a scalar, the width is fixed. Otherwise, the width of smoothing window is dependent of the height.
dimension: integer: the dimension which the smoothing is along with.

OUTPUTS:

sigStd: array: uncertainty of the signal

HISTORY:

2021-06-13: first edition by Zhenping

Picasso visualization

lib.visualization.pollyDisplayAttnBsc(data)

POLLYDISPLAYATTNBSC display attenuated backscatter.

USAGE:

pollyDisplayAttnBsc(data)

INPUTS:

data: struct

HISTORY:

2021-06-10: first edition by Zhenping

lib.visualization.pollyDisplayHousekeeping(data)

POLLYDISPLAYHOUSEKEEPING display housekeeping data.

USAGE:

pollyDisplayHousekeeping(data)

INPUTS:

data: struct

HISTORY:

2021-06-09: first edition by Zhenping

lib.visualization.pollyDisplayLC(data)

POLLYDISPLAYLC display lidar calibration constants.

USAGE:

pollyDisplayLC(data)

INPUTS:

data: struct

HISTORY:

2021-06-11: first edition by Zhenping

lib.visualization.pollyDisplayLTLCali(data, dbFile)

POLLYDISPLAYLTLCALI display long-term calibration results.

USAGE:

pollyDisplayLTLCali(data)

INPUTS:

data: struct dbFile: char

HISTORY:

2021-06-09: first edition by Zhenping

lib.visualization.pollyDisplayOCProfiles(data)

POLLYDISPLAYPROFILES display (overlap-corrected) averaged profiles.

USAGE:

pollyDisplayOCProfiles(data)

INPUTS:

data: struct

HISTORY:

2021-06-11: first edition by Zhenping

lib.visualization.pollyDisplayOL(data)

pollyDisplayOL display overlap function.

USAGE:

pollyDisplayOL(data)

INPUTS:

data: struct

HISTORY:

2021-06-10: first edition by Zhenping

lib.visualization.pollyDisplayPolCali(data)

POLLYDISPLAYPOLCALI display polarization calibration results.

USAGE:

pollyDisplayPolCali(data)

INPUTS:

data: struct

HISTORY:

2021-06-10: first edition by Zhenping

lib.visualization.pollyDisplayProfiles(data)

POLLYDISPLAYPROFILES display averaged profiles.

USAGE:

pollyDisplayProfiles(data)

INPUTS:

data: struct

HISTORY:

2021-06-11: first edition by Zhenping

lib.visualization.pollyDisplayQsiV1(data)

POLLYDISPLAYQSIV1 display quasi-retrieved prodcuts (V1)

USAGE:

pollyDisplayQsiV1(data)

INPUTS:

data: struct

HISTORY:

2021-06-10: first edition by Zhenping

lib.visualization.pollyDisplayQsiV2(data)

POLLYDISPLAYQSIV2 display quasi-retrieved prodcuts (V2)

USAGE:

pollyDisplayQsiV2(data)

INPUTS:

data: struct

HISTORY:

2021-06-10: first edition by Zhenping

lib.visualization.pollyDisplayRCS(data)

POLLYDISPLAYRCS display range corrected signal.

USAGE:

pollyDisplayRCS(data)

INPUTS:

data: struct

HISTORY:

2021-06-09: first edition by Zhenping

lib.visualization.pollyDisplaySigStatus(data)

POLLYDISPLAYSIGSTATUS display signal status.

USAGE:

pollyDisplaySigStatus(data)

INPUTS:

data: struct

HISTORY:

2021-06-10: first edition by Zhenping

lib.visualization.pollyDisplayTCV1(data)

POLLYDISPLAYTCV1 display aerosol/cloud target classification mask (V1).

USAGE:

pollyDisplayTCV1(data)

INPUTS:

data: struct

HISTORY:

2021-06-11: first edition by Zhenping

lib.visualization.pollyDisplayTCV2(data)

POLLYDISPLAYTCV2 display aerosol/cloud target classification mask (V2).

USAGE:

pollyDisplayTCV2(data)

INPUTS:

data: struct

HISTORY:

2021-06-11: first edition by Zhenping

lib.visualization.pollyDisplayVDR(data)

POLLYDISPLAYVDR display volume depolarization ratio.

USAGE:

pollyDisplayVDR(data)

INPUTS:

data: struct

HISTORY:

2021-06-10: first edition by Zhenping

lib.visualization.pollyDisplayWV(data)

POLLYDISPLAYWV display water vapor products.

USAGE:

pollyDisplayWV(data)

INPUTS:

data: struct

HISTORY:

2021-06-10: first edition by Zhenping

lib.visualization.polly_1st_displayHousekeeping(data)

POLLY_1ST_DISPLAYHOUSEKEEPING display housekeeping data.

USAGE:

polly_1st_displayHousekeeping(data)

INPUTS:

data: struct

HISTORY:

2021-06-09: first edition by Zhenping

lib.visualization.polly_1st_displayLTLCali(data, dbFile)

POLLY_1ST_DISPLAYLTLCALI display long-term calibration results.

USAGE:

polly_1st_displayLTLCali(data, dbFile)

INPUTS:

data: struct dbFile: char

HISTORY:

2021-06-11: first edition by Zhenping

lib.visualization.polly_1v2_displayHousekeeping(data)

POLLY_1V2_DISPLAYHOUSEKEEPING display housekeeping data.

USAGE:

polly_1v2_displayHousekeeping(data)

INPUTS:

data: struct

HISTORY:

2021-06-09: first edition by Zhenping

lib.visualization.polly_1v2_displayLTLCali(data, dbFile)

POLLY_1V2_DISPLAYLTLCALI display long-term lidar calibration results.

USAGE:

polly_1v2_displayLTLCali(data, dbFile)

INPUTS:

data: struct dbFile: char

HISTORY:

2021-06-11: first edition by Zhenping

lib.visualization.pollyxt_cge_displayHousekeeping(data)

POLLYXT_CGE_DISPLAYHOUSEKEEPING display housekeeping data.

USAGE:

pollyxt_cge_displayHousekeeping(data)

INPUTS:

data: struct

HISTORY:

2021-06-09: first edition by Zhenping

lib.visualization.pollyxt_cge_displayLTLCali(data, dbFile)

POLLYXT_CGE_DISPLAYLTLCALI display long-term lidar calibration results.

USAGE:

pollyxt_cge_displayLTLCali(data, dbFile)

INPUTS:

data: struct dbFile: char

HISTORY:

2021-06-11: first edition by Zhenping

lib.visualization.pollyxt_cpv_displayLTLCali(data, dbFile)

POLLYXT_CPV_DISPLAYLTLCALI display lidar calibration results.

USAGE:

pollyxt_cpv_displayLTLCali(data)

INPUTS:

data: struct dbFile: char

HISTORY:

2021-06-11: first edition by Zhenping

lib.visualization.pollyxt_displayHousekeeping(data)

POLLYXT_DISPLAYHOUSEKEEPING display housekeeping data.

USAGE:

pollyxt_displayHousekeeping(data)

INPUTS:

data: struct

HISTORY:

2021-06-09: first edition by Zhenping

lib.visualization.pollyxt_displayLTLCali(data, dbFile)

POLLYXT_DISPLAYLTLCALI display lidar calibration results.

USAGE:

pollyxt_displayLTLCali(data)

INPUTS:

data: struct dbFile: char

HISTORY:

2021-06-11: first edition by Zhenping

lib.visualization.pollyxt_dwd_displayHousekeeping(data)

POLLYXT_DWD_DISPLAYHOUSEKEEPING display housekeeping data.

USAGE:

pollyxt_dwd_displayHousekeeping(data)

INPUTS:

data: struct

HISTORY:

2021-06-09: first edition by Zhenping

lib.visualization.pollyxt_dwd_displayLTLCali(data, dbFile)

POLLYXT_DWD_DISPLAYLTLCALI display long-term lidar calibration results.

USAGE:

pollyxt_dwd_displayLTLCali(data, dbFile)

INPUTS:

data: struct dbFile: char

HISTORY:

2021-06-11: first edition by Zhenping

lib.visualization.pollyxt_ift_displayHousekeeping(data)

POLLYXT_IFT_DISPLAYHOUSEKEEPING display housekeeping data.

USAGE:

pollyxt_ift_displayHousekeeping(data)

INPUTS:

data: struct

HISTORY:

2021-06-09: first edition by Zhenping

lib.visualization.pollyxt_ift_displayLTLCali(data, dbFile)

POLLYXT_IFT_DISPLAYLTLCALI display long-term lidar calibration results.

USAGE:

pollyxt_ift_displayLTLCali(data, dbFile)

INPUTS:

data: struct dbFile: char

HISTORY:

2021-06-11: first edition by Zhenping

lib.visualization.subfigPos(pos, nRow, nColumn, xPad, yPad)

SUBFIGPOS calculate the normalized position of each subfigure.

USAGE:

[figPos] = subfigPos(pos, nRow, nColumn, xpad, ypad)

INPUTS:

pos: 4-element array: [left, bottom, width, height]
nRow: integer: number of the total rows. (default, 1)
nColumn: integer: number of the total columns. (default, 1)
xPad: numeric: x-padding
yPad: numeric: y-padding

OUTPUTS:

figPos: matrix: returned postition of each subfigures. The first figure is the top-right one and as followed by from left-to-right and top-to-base.

HISTORY:

2018-11-09: First edition by Zhenping
2019-12-15: Enable set the x-y padding.

lib.visualization.update_colormap()

UPDATE_COLORMAP Update user defined colormaps.

USAGE:

[status_colormap, TC_colormap] = update_colormap()

OUTPUTS:

status_colormap: matrix: color table for the signal status plot.
TC_colormap: matrix: color table for the aerosol target classification plot.

HISTORY:

2021-06-13: first edition by Zhenping