pyFAI.engines package

pyFAI.engines.CSC_engine module

CSC rebinning engine implemented in pure python (with bits of scipy !)

class pyFAI.engines.CSC_engine.CSCIntegrator(input_size, output_size, lut=None, empty=0.0)

Bases: object

__init__(input_size, output_size, lut=None, empty=0.0)

Constructor of the abstract class

Parameters:

  • input_size – input image size

  • output_size – output histogram size (int or 2-tuple)

  • lut – tuple of 3 arrays with data, indices and indptr, index of the start of line in the CSC matrix

  • empty – value for empty pixels

integrate(signal, variance=None, error_model=None, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0, weighted_average=True)

Actually perform the CSR matrix multiplication after preprocessing.

Parameters:

  • signal – array of the right size with the signal in it.

  • variance – Variance associated with the signal

  • error_model – Enum or string, set to “poisson” to use signal as variance (minimum 1), set to “azimuthal” to use azimuthal model.

  • dummy – values which have to be discarded (dynamic mask)

  • delta_dummy – precision for dummy values

  • dark – noise to be subtracted from signal

  • flat – flat-field normalization array

  • flat – solidangle normalization array

  • polarization – :solidangle normalization array

  • absorption – :absorption normalization array

  • normalization_factor – scale all normalization with this scalar

  • weighted_average (bool) – set to False to use an unweighted mean (similar to legacy) instead of the weighted average

Returns:

the preprocessed data integrated as array nbins x 4 which contains: regrouped signal, variance, normalization, pixel count, sum_norm²

Nota: all normalizations are grouped in the preprocessing step.

set_matrix(data, indices, indptr)

Actually set the CSR sparse matrix content

Parameters:

  • data – the non zero values NZV

  • indices – the column number of the NZV

  • indptr – the index of the start of line

class pyFAI.engines.CSC_engine.CscIntegrator1d(image_size, lut=None, empty=0.0, unit=None, bin_centers=None, mask_checksum=None)

Bases: CSCIntegrator

__init__(image_size, lut=None, empty=0.0, unit=None, bin_centers=None, mask_checksum=None)

Constructor of the abstract class for 1D integration

Parameters:

  • image_size – size of the image

  • lut – (data, indices, indptr) of the CSR matrix

  • empty – value for empty pixels

  • unit – the kind of radial units

  • bin_center – position of the bin center

  • mask_checksum – just a place-holder to track which mask was used

Nota: bins value is deduced from the dimentionality of bin_centers

property check_mask
integrate(signal, variance=None, error_model=None, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0, weighted_average=True)

Actually perform the 1D integration

Parameters:

  • signal – array of the right size with the signal in it.

  • variance – Variance associated with the signal

  • error_model – Enum or str, set to “poisson” to use signal as variance (minimum 1), set to “azimuthal” to use azimuthal model.

  • dummy – values which have to be discarded (dynamic mask)

  • delta_dummy – precision for dummy values

  • dark – noise to be subtracted from signal

  • flat – flat-field normalization array

  • flat – solidangle normalization array

  • polarization – :solidangle normalization array

  • absorption – :absorption normalization array

  • normalization_factor – scale all normalization with this scalar

  • weighted_average (bool) – set to False to use an unweighted mean (similar to legacy) instead of the weighted average

Returns:

Integrate1dResult or Integrate1dWithErrorResult object depending on variance

integrate_ng(signal, variance=None, error_model=None, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0, weighted_average=True)

Actually perform the 1D integration

Parameters:

  • signal – array of the right size with the signal in it.

  • variance – Variance associated with the signal

  • error_model – Enum or str, set to “poisson” to use signal as variance (minimum 1), set to “azimuthal” to use azimuthal model.

  • dummy – values which have to be discarded (dynamic mask)

  • delta_dummy – precision for dummy values

  • dark – noise to be subtracted from signal

  • flat – flat-field normalization array

  • flat – solidangle normalization array

  • polarization – :solidangle normalization array

  • absorption – :absorption normalization array

  • normalization_factor – scale all normalization with this scalar

  • weighted_average (bool) – set to False to use an unweighted mean (similar to legacy) instead of the weighted average

Returns:

Integrate1dResult or Integrate1dWithErrorResult object depending on variance

set_matrix(data, indices, indptr)

Actually set the CSR sparse matrix content

Parameters:

  • data – the non zero values NZV

  • indices – the column number of the NZV

  • indptr – the index of the start of line

sigma_clip(data, dark=None, dummy=None, delta_dummy=None, variance=None, dark_variance=None, flat=None, solidangle=None, polarization=None, absorption=None, safe=True, error_model=None, normalization_factor=1.0, cutoff=4.0, cycle=5)

Perform a sigma-clipping iterative filter within each along each row. see the doc of scipy.stats.sigmaclip for more descriptions.

If the error model is “azimuthal”: the variance is the variance within a bin, which is refined at each iteration, can be costly !

Else, the error is propagated according to:

\[signal = (raw - dark) variance = variance + dark_variance normalization = normalization_factor*(flat * solidangle * polarization * absortoption) count = number of pixel contributing\]

Integration is performed using the CSR representation of the look-up table on all arrays: signal, variance, normalization and count

Formula for azimuthal variance from: https://dbs.ifi.uni-heidelberg.de/files/Team/eschubert/publications/SSDBM18-covariance-authorcopy.pdf

Parameters:

  • dark – array of same shape as data for pre-processing

  • dummy – value for invalid data

  • delta_dummy – precesion for dummy assessement

  • variance – array of same shape as data for pre-processing

  • dark_variance – array of same shape as data for pre-processing

  • flat – array of same shape as data for pre-processing

  • solidangle – array of same shape as data for pre-processing

  • polarization – array of same shape as data for pre-processing

  • safe – Unused in this implementation

  • error_model – Enum or str, “azimuthal” or “poisson”

  • normalization_factor – divide raw signal by this value

  • cutoff – discard all points with |value - avg| > cutoff * sigma. 3-4 is quite common

  • cycle – perform at maximum this number of cycles. 5 is common.

Returns:

namedtuple with “position intensity error signal variance normalization count”

class pyFAI.engines.CSC_engine.CscIntegrator2d(image_size, lut=None, empty=0.0, unit=None, bin_centers0=None, bin_centers1=None, checksum=None, mask_checksum=None)

Bases: CSCIntegrator

__init__(image_size, lut=None, empty=0.0, unit=None, bin_centers0=None, bin_centers1=None, checksum=None, mask_checksum=None)

Constructor of the abstract class for 2D integration

Parameters:

  • image_size – input image size

  • lut – tuple of 3 arrays with data, indices and indptr, index of the start of line in the CSR matrix

  • empty – value for empty pixels

  • unit – unit to be used

  • bin_center0 – position of the bin center along dim0

  • bin_center1 – position of the bin center along dim1

  • checksum – checksum for the LUT, if not provided, recalculated

  • mask_checksum – just a place-holder to track which mask was used

Nota: bins are deduced from bin_centers0, bin_centers1

property check_mask
integrate(signal, variance=None, error_model=ErrorModel.NO, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0, **kwargs)

Actually perform the 2D integration

Parameters:

  • signal – array of the right size with the signal in it.

  • variance – Variance associated with the signal

  • error_model – enum ErrorModel

  • dummy – values which have to be discarded (dynamic mask)

  • delta_dummy – precision for dummy values

  • dark – noise to be subtracted from signal

  • flat – flat-field normalization array

  • flat – solidangle normalization array

  • polarization – :solidangle normalization array

  • absorption – :absorption normalization array

  • normalization_factor – scale all normalization with this scalar

Returns:

Integrate2dtpl namedtuple: “radial azimuthal intensity error signal variance normalization count”

integrate_ng(signal, variance=None, error_model=ErrorModel.NO, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0, **kwargs)

Actually perform the 2D integration

Parameters:

  • signal – array of the right size with the signal in it.

  • variance – Variance associated with the signal

  • error_model – enum ErrorModel

  • dummy – values which have to be discarded (dynamic mask)

  • delta_dummy – precision for dummy values

  • dark – noise to be subtracted from signal

  • flat – flat-field normalization array

  • flat – solidangle normalization array

  • polarization – :solidangle normalization array

  • absorption – :absorption normalization array

  • normalization_factor – scale all normalization with this scalar

Returns:

Integrate2dtpl namedtuple: “radial azimuthal intensity error signal variance normalization count”

set_matrix(data, indices, indptr)

Actually set the CSR sparse matrix content

Parameters:

  • data – the non zero values NZV

  • indices – the column number of the NZV

  • indptr – the index of the start of line

pyFAI.engines.CSR_engine module

CSR rebinning engine implemented in pure python (with bits of scipy !)

class pyFAI.engines.CSR_engine.CSRIntegrator(image_size, lut=None, empty=0.0)

Bases: object

__init__(image_size, lut=None, empty=0.0)

Constructor of the abstract class

Parameters:

  • size – input image size

  • lut – tuple of 3 arrays with data, indices and indptr, index of the start of line in the CSR matrix

  • empty – value for empty pixels

integrate(signal, variance=None, error_model=None, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0, weighted_average=True)

Actually perform the CSR matrix multiplication after preprocessing.

Parameters:

  • signal – array of the right size with the signal in it.

  • variance – Variance associated with the signal

  • error_model – Enum or string, set to “poisson” to use signal as variance (minimum 1), set to “azimuthal” to use azimuthal model.

  • dummy – values which have to be discarded (dynamic mask)

  • delta_dummy – precision for dummy values

  • dark – noise to be subtracted from signal

  • flat – flat-field normalization array

  • flat – solidangle normalization array

  • polarization – :solidangle normalization array

  • absorption – :absorption normalization array

  • normalization_factor – scale all normalization with this scalar

  • weighted_average (bool) – set to False to use an unweighted mean (similar to legacy) instead of the weighted average WIP

Returns:

the preprocessed data integrated as array nbins x 4 which contains: regrouped signal, variance, normalization, pixel count, sum_norm²

Nota: all normalizations are grouped in the preprocessing step.

set_matrix(data, indices, indptr)

Actually set the CSR sparse matrix content

Parameters:

  • data – the non zero values NZV

  • indices – the column number of the NZV

  • indptr – the index of the start of line

class pyFAI.engines.CSR_engine.CsrIntegrator1d(image_size, lut=None, empty=0.0, unit=None, bin_centers=None, mask_checksum=None)

Bases: CSRIntegrator

__init__(image_size, lut=None, empty=0.0, unit=None, bin_centers=None, mask_checksum=None)

Constructor of the abstract class for 1D integration

Parameters:

  • image_size – size of the image

  • lut – (data, indices, indptr) of the CSR matrix

  • empty – value for empty pixels

  • unit – the kind of radial units

  • bin_center – position of the bin center

  • mask_checksum – just a place-holder to track which mask was used

Nota: bins value is deduced from the dimentionality of bin_centers

property check_mask
integrate(signal, variance=None, error_model=None, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0, weighted_average=True)

Actually perform the 1D integration

Parameters:

  • signal – array of the right size with the signal in it.

  • variance – Variance associated with the signal

  • error_model – Enum or str, set to “poisson” to use signal as variance (minimum 1), set to “azimuthal” to use azimuthal model.

  • dummy – values which have to be discarded (dynamic mask)

  • delta_dummy – precision for dummy values

  • dark – noise to be subtracted from signal

  • flat – flat-field normalization array

  • flat – solidangle normalization array

  • polarization – :solidangle normalization array

  • absorption – :absorption normalization array

  • normalization_factor – scale all normalization with this scalar

  • weighted_average (bool) – set to False to use an unweighted mean (similar to legacy) instead of the weighted average

Returns:

Integrate1dResult or Integrate1dWithErrorResult object depending on variance

integrate_ng(signal, variance=None, error_model=None, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0, weighted_average=True)

Actually perform the 1D integration

Parameters:

  • signal – array of the right size with the signal in it.

  • variance – Variance associated with the signal

  • error_model – Enum or str, set to “poisson” to use signal as variance (minimum 1), set to “azimuthal” to use azimuthal model.

  • dummy – values which have to be discarded (dynamic mask)

  • delta_dummy – precision for dummy values

  • dark – noise to be subtracted from signal

  • flat – flat-field normalization array

  • flat – solidangle normalization array

  • polarization – :solidangle normalization array

  • absorption – :absorption normalization array

  • normalization_factor – scale all normalization with this scalar

  • weighted_average (bool) – set to False to use an unweighted mean (similar to legacy) instead of the weighted average

Returns:

Integrate1dResult or Integrate1dWithErrorResult object depending on variance

set_matrix(data, indices, indptr)

Actually set the CSR sparse matrix content

Parameters:

  • data – the non zero values NZV

  • indices – the column number of the NZV

  • indptr – the index of the start of line

sigma_clip(data, dark=None, dummy=None, delta_dummy=None, variance=None, dark_variance=None, flat=None, solidangle=None, polarization=None, absorption=None, safe=True, error_model=None, normalization_factor=1.0, cutoff=4.0, cycle=5)

Perform a sigma-clipping iterative filter within each along each row. see the doc of scipy.stats.sigmaclip for more descriptions.

If the error model is “azimuthal”: the variance is the variance within a bin, which is refined at each iteration, can be costly !

Else, the error is propagated according to:

\[signal = (raw - dark) variance = variance + dark_variance normalization = normalization_factor*(flat * solidangle * polarization * absortoption) count = number of pixel contributing\]

Integration is performed using the CSR representation of the look-up table on all arrays: signal, variance, normalization and count

Formula for azimuthal variance from: https://dbs.ifi.uni-heidelberg.de/files/Team/eschubert/publications/SSDBM18-covariance-authorcopy.pdf

Parameters:

  • dark – array of same shape as data for pre-processing

  • dummy – value for invalid data

  • delta_dummy – precesion for dummy assessement

  • variance – array of same shape as data for pre-processing

  • dark_variance – array of same shape as data for pre-processing

  • flat – array of same shape as data for pre-processing

  • solidangle – array of same shape as data for pre-processing

  • polarization – array of same shape as data for pre-processing

  • safe – Unused in this implementation

  • error_model – Enum or str, “azimuthal” or “poisson”

  • normalization_factor – divide raw signal by this value

  • cutoff – discard all points with |value - avg| > cutoff * sigma. 3-4 is quite common

  • cycle – perform at maximum this number of cycles. 5 is common.

Returns:

namedtuple with “position intensity error signal variance normalization count”

class pyFAI.engines.CSR_engine.CsrIntegrator2d(image_size, lut=None, empty=0.0, unit=None, bin_centers0=None, bin_centers1=None, checksum=None, mask_checksum=None)

Bases: CSRIntegrator

__init__(image_size, lut=None, empty=0.0, unit=None, bin_centers0=None, bin_centers1=None, checksum=None, mask_checksum=None)

Constructor of the abstract class for 2D integration

Parameters:

  • image_size – input image size

  • lut – tuple of 3 arrays with data, indices and indptr, index of the start of line in the CSR matrix

  • empty – value for empty pixels

  • unit – unit to be used

  • bin_center – position of the bin center

  • checksum – checksum for the LUT, if not provided, recalculated

  • mask_checksum – just a place-holder to track which mask was used

Nota: bins are deduced from bin_centers0, bin_centers1

property check_mask
integrate(signal, variance=None, error_model=ErrorModel.NO, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0, **kwargs)

Actually perform the 2D integration

Parameters:

  • signal – array of the right size with the signal in it.

  • variance – Variance associated with the signal

  • error_model – enum ErrorModel

  • dummy – values which have to be discarded (dynamic mask)

  • delta_dummy – precision for dummy values

  • dark – noise to be subtracted from signal

  • flat – flat-field normalization array

  • flat – solidangle normalization array

  • polarization – :solidangle normalization array

  • absorption – :absorption normalization array

  • normalization_factor – scale all normalization with this scalar

Returns:

Integrate2dtpl namedtuple: “radial azimuthal intensity error signal variance normalization count”

integrate_ng(signal, variance=None, error_model=ErrorModel.NO, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0, **kwargs)

Actually perform the 2D integration

Parameters:

  • signal – array of the right size with the signal in it.

  • variance – Variance associated with the signal

  • error_model – enum ErrorModel

  • dummy – values which have to be discarded (dynamic mask)

  • delta_dummy – precision for dummy values

  • dark – noise to be subtracted from signal

  • flat – flat-field normalization array

  • flat – solidangle normalization array

  • polarization – :solidangle normalization array

  • absorption – :absorption normalization array

  • normalization_factor – scale all normalization with this scalar

Returns:

Integrate2dtpl namedtuple: “radial azimuthal intensity error signal variance normalization count”

set_matrix(data, indices, indptr)

Actually set the CSR sparse matrix content

Parameters:

  • data – the non zero values NZV

  • indices – the column number of the NZV

  • indptr – the index of the start of line

pyFAI.engines.histogram_engine module

simple histogram rebinning engine implemented in pure python (with the help of numpy !)

pyFAI.engines.histogram_engine.histogram1d_engine(radial, npt, raw, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, mask=None, dummy=None, delta_dummy=None, normalization_factor=1.0, empty=None, split_result=False, variance=None, dark_variance=None, error_model=ErrorModel.NO, weighted_average=True, radial_range=None)

Implementation of rebinning engine using pure numpy histograms

Parameters:

  • radial – radial position 2D array (same shape as raw)

  • npt – number of points to integrate over

  • raw – 2D array with the raw signal

  • dark – array containing the value of the dark noise, to be subtracted

  • flat – Array containing the flatfield image. It is also checked for dummies if relevant.

  • solidangle – the value of the solid_angle. This processing may be performed during the rebinning instead. left for compatibility

  • polarization – Correction for polarization of the incident beam

  • absorption – Correction for absorption in the sensor volume

  • mask – 2d array of int/bool: non-null where data should be ignored

  • dummy – value of invalid data

  • delta_dummy – precision for invalid data

  • normalization_factor – final value is divided by this

  • empty – value to be given for empty bins

  • variance – provide an estimation of the variance

  • dark_variance – provide an estimation of the variance of the dark_current,

  • error_model – Use the provided ErrorModel, only “poisson” and “variance” is valid

  • weighted_average (bool) – set to False to use an unweighted mean (similar to legacy) instead of the weighted average

NaN are always considered as invalid values

if neither empty nor dummy is provided, empty pixels are left at 0.

Nota: “azimuthal_range” has to be integrated into the

mask prior to the call of this function

Returns:

Integrate1dtpl named tuple containing: position, average intensity, std on intensity, plus the various histograms on signal, variance, normalization and count.

pyFAI.engines.histogram_engine.histogram2d_engine(radial, azimuthal, bins, raw, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, mask=None, dummy=None, delta_dummy=None, normalization_factor=1.0, empty=None, variance=None, dark_variance=None, error_model=ErrorModel.NO, radial_range=None, azimuth_range=None, allow_radial_neg=False, chiDiscAtPi=True, clip_pos1=True)

Implementation of 2D rebinning engine using pure numpy histograms

Parameters:

  • radial – radial position 2D array (same shape as raw)

  • azimuthal – azimuthal position 2D array (same shape as raw)

  • bins – number of points to integrate over in (radial, azimuthal) dimensions

  • raw – 2D array with the raw signal

  • dark – array containing the value of the dark noise, to be subtracted

  • flat – Array containing the flatfield image. It is also checked for dummies if relevant.

  • solidangle – the value of the solid_angle. This processing may be performed during the rebinning instead. left for compatibility

  • polarization – Correction for polarization of the incident beam

  • absorption – Correction for absorption in the sensor volume

  • mask – 2d array of int/bool: non-null where data should be ignored

  • dummy – value of invalid data

  • delta_dummy – precision for invalid data

  • normalization_factor – final value is divided by this

  • empty – value to be given for empty bins

  • variance – provide an estimation of the variance

  • dark_variance – provide an estimation of the variance of the dark_current,

  • error_model – set to “poisson” for assuming the detector is poissonian and variance = raw + dark

  • radial_range – enforce boundaries in radial dimention, 2tuple with lower and upper bound

  • azimuth_range – enforce boundaries in azimuthal dimention, 2tuple with lower and upper bound

  • allow_radial_neg – clip negative radial position (can a dimention be negative ?)

  • chiDiscAtPi – boolean; by default the chi_range is in the range ]-pi,pi[ set to 0 to have the range ]0,2pi[ TODO: unimplemented

  • clip_pos1 – clip the azimuthal range to [-pi pi] (or [0 2pi]), set to False to deactivate behavior TODO: unimplemented

NaN are always considered as invalid values

if neither empty nor dummy is provided, empty pixels are left at 0.

Nota: “azimuthal_range” has to be integrated into the

mask prior to the call of this function

Returns:

Integrate1dtpl named tuple containing: position, average intensity, std on intensity, plus the various histograms on signal, variance, normalization and count.

pyFAI.engines.preproc module

Module providing common pixel-wise pre-processing of data.

pyFAI.engines.preproc.preproc(raw, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, mask=None, dummy=None, delta_dummy=None, normalization_factor=1.0, empty=None, split_result=False, variance=None, dark_variance=None, error_model=ErrorModel.NO, apply_normalization=False, dtype=<class 'numpy.float32'>, out=None)

Common preprocessing step for all integration engines

Parameters:

  • data – raw value, as a numpy array, 1D or 2D

  • mask – array non null where data should be ignored

  • dummy – value of invalid data

  • delta_dummy – precision for invalid data

  • dark – array containing the value of the dark noise, to be subtracted

  • flat – Array containing the flatfield image. It is also checked for dummies if relevant.

  • solidangle – the value of the solid_angle. This processing may be performed during the rebinning instead. left for compatibility

  • polarization – Correction for polarization of the incident beam

  • absorption – Correction for absorption in the sensor volume

  • normalization_factor – final value is divided by this

  • empty – value to be given for empty bins

  • split_result – set to true to separate signal from normalization and return an array of float2, float3 (with variance) ot float4 (including counts)

  • variance – provide an estimation of the variance, enforce split_result=True and return an float3 array with variance in second position.

  • dark_variance – provide an estimation of the variance of the dark_current, enforce split_result=True and return an float3 array with variance in second position.

  • error_model – set to “Poisson” for assuming the detector is poissonian and variance = max(1, raw + dark)

  • apply_normalization – correct (directly) the raw signal & variance with normalization, WIP

  • dtype – dtype for all processing

  • out – output buffer to save a malloc

All calculation are performed in single precision floating point (32 bits).

NaN are always considered as invalid values

if neither empty nor dummy is provided, empty pixels are 0. Empty pixels are always zero in “split_result” mode.

When set to False, i.e the default, the pixel-wise operation is:

\[I = \frac{raw - dark}{flat \cdot solidangle \cdot polarization \cdot absorption}\]

Invalid pixels are set to the dummy or empty value.

When split_result is set to True, each result is a float2 or a float3 (with an additional value for the variance) as such:

I = [\(raw - dark\), \(variance\), \(flat \cdot solidangle \cdot polarization \cdot absorption\)]

If split_result is 4, then the count of pixel is appended to the list, i.e. 1 or 0 for masked pixels Empty pixels will have all their 2 or 3 or 4 values to 0 (and not to dummy or empty value)

If poissonian is set to True, the variance is evaluated as raw + dark, with a minimum of 1.

Module contents

This sub-module contains various rebinning and pre-processing engines defined at the Python level.

class pyFAI.engines.Engine(engine=None)

Bases: object

This class defines a regrid-engine with its locking mechanism

__init__(engine=None)

Constructor of the class

reset()
set_engine(engine)

should be called from a locked region