Filtering signal in azimuthal space#
Usually, diffraction signal presents a polar symmetry, this means all pixels with the same azimuthal angle (χ) have similar intensities. The best way to exploit this is to take the mean, what is called azimuthal average. But the average is very sensitive to outliers, like gaps, missing pixels, shadows, cosmic rays or reflections coming from larger crystallites. In this tutorial we will see two alternative ways to remove those unwanted signals and focus on the majority of pixels: sigma clipping and median filtering.
import os
os.environ["PYOPENCL_COMPILER_OUTPUT"]="0"
import pyFAI
print(f"pyFAI version: {pyFAI.version}")
pyFAI version: 2026.9.1-dev0
%matplotlib inline
from matplotlib.pyplot import subplots
from pyFAI.gui import jupyter
import numpy
import fabio
from pyFAI.test.utilstest import UtilsTest
import pyFAI.benchmark
figsize = (10,5)
ai = pyFAI.load(UtilsTest.getimage("Pilatus6M.poni"))
img = fabio.open(UtilsTest.getimage("Pilatus6M.cbf")).data
fig, ax = subplots(1, 2, figsize=figsize)
jupyter.display(img, ax=ax[1])
jupyter.plot1d(ai.integrate1d(img, 1000), ax=ax[0])
ax[1].set_title("With a few Bragg peaks");
WARNING:pyFAI.gui.matplotlib:Matplotlib already loaded with backend `inline`, setting its backend to `QtAgg` may not work!
Azimuthal sigma-clipping#
The idea is to discard pixels which look like outliers in the distribution of all pixels contributing to a single azimuthal bin. It requires an error model like poisson but it has been proven to be better to use the variance in the given azimuthal ring. All details are available in this publication: https://doi.org/10.1107/S1600576724011038 also available at https://doi.org/10.48550/arXiv.2411.09515
fig, ax = subplots(1, 2, figsize=figsize)
jupyter.display(img, ax=ax[1])
jupyter.plot1d(ai.sigma_clip(img, 1000, error_model="hybrid", method=("no", "csr", "cython")), ax=ax[0])
ax[1].set_title("With a few Bragg peaks")
ax[0].set_title("Sigma_clipping");
Of course, sigma-clip takes several extra parameters like the number of iterations to perform, the cut-off, the error model, … There are also a few limitations:
The algorithm needs to be the CSR-sparse matrix multiplication: since several integrations are needed, it makes no sense to use a histogram based algorithm.
The algorithm is available with any implementation: Python (using scipy.saprse), Cython and OpenCL, and it runs just fine on GPU.
Sigma-clipping is incompatible with any kind of pixel splitting: With pixel splitting, a single pixel can contribute to several azimuthal bins and discarding a pixel in one ring could disable it in the neighboring ring (or not, since bins are processed in parallel).
Sigma-clipping performances:#
method = ["no", "csr", "cython"]
%%time
perfs_integrate_python = {}
perfs_integrate_cython = {}
perfs_integrate_opencl = {}
perfs_sigma_clip_python = {}
perfs_sigma_clip_cython = {}
perfs_sigma_clip_opencl = {}
for ds in pyFAI.benchmark.PONIS:
ai = pyFAI.load(UtilsTest.getimage(ds))
if ai.wavelength is None: ai.wavelength=1.54e-10
img = fabio.open(UtilsTest.getimage(pyFAI.benchmark.datasets[ds])).data
size = numpy.prod(ai.detector.shape)
print(ds)
print(" Cython")
meth = tuple(method)
nbin = max(ai.detector.shape)
print(" * integrate ", end="")
perfs_integrate_cython[size] = %timeit -o ai.integrate1d(img, nbin, method=meth)
print(" * sigma-clip", end="")
perfs_sigma_clip_cython[size] = %timeit -o ai.sigma_clip(img, nbin, method=meth, error_model="azimuthal")
print(" Python")
meth = tuple(method[:2]+["python"])
print(" * integrate ", end="")
perfs_integrate_python[size] = %timeit -o ai.integrate1d(img, nbin, method=meth)
print(" * sigma-clip", end="")
perfs_sigma_clip_python[size] = %timeit -o ai.sigma_clip(img, nbin, method=meth, error_model="azimuthal")
print(" OpenCL")
meth = tuple(method[:2]+["opencl"])
print(" * integrate ", end="")
perfs_integrate_opencl[size] = %timeit -o ai.integrate1d(img, nbin, method=meth)
print(" * sigma-clip", end="")
perfs_sigma_clip_opencl[size] = %timeit -o ai.sigma_clip(img, nbin, method=meth, error_model="azimuthal")
Pilatus1M.poni
Cython
* integrate
11.1 ms ± 2.22 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
* sigma-clip
15.8 ms ± 1.57 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
Python
* integrate
10.9 ms ± 68.6 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
* sigma-clip
162 ms ± 3.86 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
OpenCL
* integrate
/users/kieffer/.venv/py314/lib/python3.14/site-packages/pyopencl/__init__.py:570: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
lambda: self._prg.build(options_bytes, devices),
772 μs ± 73.7 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
2.49 ms ± 3.13 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Pilatus2M.poni
Cython
* integrate
25.1 ms ± 3.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
17.9 ms ± 3.21 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Python
* integrate
34.1 ms ± 50.3 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
* sigma-clip
550 ms ± 1.75 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
OpenCL
* integrate
/users/kieffer/.venv/py314/lib/python3.14/site-packages/pyopencl/__init__.py:570: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
lambda: self._prg.build(options_bytes, devices),
1.23 ms ± 133 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
6.11 ms ± 14.6 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Eiger4M.poni
Cython
* integrate
35.1 ms ± 3.51 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
33.2 ms ± 614 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Python
* integrate
63.1 ms ± 1.13 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
* sigma-clip
1.12 s ± 3.51 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
OpenCL
* integrate
/users/kieffer/.venv/py314/lib/python3.14/site-packages/pyopencl/__init__.py:570: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
lambda: self._prg.build(options_bytes, devices),
2.22 ms ± 94.2 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
10.9 ms ± 34.7 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Pilatus6M.poni
Cython
* integrate
43.4 ms ± 2.09 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
36.5 ms ± 1.9 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Python
* integrate
85.9 ms ± 402 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
* sigma-clip
1.59 s ± 4.26 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
OpenCL
* integrate
/users/kieffer/.venv/py314/lib/python3.14/site-packages/pyopencl/__init__.py:570: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
lambda: self._prg.build(options_bytes, devices),
3.11 ms ± 31.5 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
14 ms ± 28.1 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Eiger9M.poni
Cython
* integrate
101 ms ± 11.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
84.2 ms ± 15.2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Python
* integrate
154 ms ± 512 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
* sigma-clip
3.09 s ± 11.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
OpenCL
* integrate
/users/kieffer/.venv/py314/lib/python3.14/site-packages/pyopencl/__init__.py:570: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
lambda: self._prg.build(options_bytes, devices),
4.74 ms ± 42.1 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
26.3 ms ± 220 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Mar3450.poni
Cython
* integrate
65.1 ms ± 2.77 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
66.6 ms ± 1.66 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Python
* integrate
167 ms ± 688 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
* sigma-clip
3.43 s ± 12.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
OpenCL
* integrate
/users/kieffer/.venv/py314/lib/python3.14/site-packages/pyopencl/__init__.py:570: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
lambda: self._prg.build(options_bytes, devices),
5.33 ms ± 59.2 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
28.9 ms ± 124 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Fairchild.poni
Cython
* integrate
110 ms ± 2.33 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
159 ms ± 36.5 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Python
* integrate
386 ms ± 1.94 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
9.56 s ± 40.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
OpenCL
* integrate
/users/kieffer/.venv/py314/lib/python3.14/site-packages/pyopencl/__init__.py:570: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.
lambda: self._prg.build(options_bytes, devices),
4.95 ms ± 36.2 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)
* sigma-clip
33.2 ms ± 46.3 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
CPU times: user 38min 7s, sys: 40.5 s, total: 38min 48s
Wall time: 5min 23s
fig, ax = subplots()
ax.set_xlabel("Image size (Mpix)")
ax.set_ylabel("Frames per seconds")
sizes = numpy.array(list(perfs_integrate_python.keys()))/1e6
ax.plot(sizes, [1/i.best for i in perfs_integrate_python.values()], label="Integrate/Python", color='green', linestyle='dashed', marker='1')
ax.plot(sizes, [1/i.best for i in perfs_integrate_cython.values()], label="Integrate/Cython", color='orange', linestyle='dashed', marker='1')
ax.plot(sizes, [1/i.best for i in perfs_integrate_opencl.values()], label="Integrate/OpenCL", color='blue', linestyle='dashed', marker='1')
ax.plot(sizes, [1/i.best for i in perfs_sigma_clip_python.values()], label="Sigma-clip/Python", color='green', linestyle='dotted', marker='2')
ax.plot(sizes, [1/i.best for i in perfs_sigma_clip_cython.values()], label="Sigma-clip/Cython", color='orange', linestyle='dotted', marker='2')
ax.plot(sizes, [1/i.best for i in perfs_sigma_clip_opencl.values()], label="Sigma-clip/OpenCL", color='blue', linestyle='dotted', marker='2')
ax.set_yscale("log")
ax.legend()
ax.set_title("Performance of Sigma-clipping vs integrate");
The penalty is very limited in Cython, much more in Python.
The biggest limitation of sigma-clipping is its incompatibility with pixel-splitting, a feature needed when oversampling, i.e. taking many more points than the size of the diagonal of the image. While oversampling is not recommended in the general case (due to the cross-correlation between bins it creates), it can be a necessary evil, especially when performing Rietveld refinement where 5 points per peak are needed, resolution that cannot be obtained with the pixel-size/distance couple accessible by the experimental setup.
Median filter in Azimuthal space#
The idea is to sort all pixels contributing to an azimuthal bin and to average out all pixels between the lower and upper quantile. When those two thresholds are at one half, this filter provides actually the median. In order to be compatible with pixel splitting, each pixel is duplicated as many times as it contributes to different bins. After sorting fragments of pixels according to their normalization corrected signal, the cumulative sum of normalization is performed in order to determine which fragments to average out.
ai = pyFAI.load(UtilsTest.getimage("Pilatus6M.poni"))
img = fabio.open(UtilsTest.getimage("Pilatus6M.cbf")).data
method = ["full", "csr", "cython"]
percentile=(40,60)
pol=0.99
fig, ax = subplots(1, 2, figsize=figsize)
jupyter.display(img, ax=ax[1])
jupyter.plot1d(ai.medfilt1d_ng(img, 1000, method=method, percentile=percentile, polarization_factor=pol), ax=ax[0])
ax[1].set_title("With a few Bragg peaks")
ax[0].set_title("Median filtering");
Unlike the sigma-clipping, this median filter does not require any error model; but the computational cost induced by the sort is huge. In addition, the median is very sensitive and requires a good geometry and modelisation of the polarization.
%%time
perf2_integrate_python = {}
perf2_integrate_cython = {}
perf2_integrate_opencl = {}
perf2_medfilt_python = {}
perf2_medfilt_cython = {}
perf2_medfilt_opencl = {}
for ds in pyFAI.benchmark.PONIS:
ai = pyFAI.load(UtilsTest.getimage(ds))
if ai.wavelength is None: ai.wavelength=1.54e-10
img = fabio.open(UtilsTest.getimage(pyFAI.benchmark.datasets[ds])).data
size = numpy.prod(ai.detector.shape)
print(ds)
print(" Cython")
meth = tuple(method)
nbin = max(ai.detector.shape)
print(" * integrate ", end="")
perf2_integrate_cython[size] = %timeit -o ai.integrate1d(img, nbin, method=meth)
print(" * medianfilter", end="")
perf2_medfilt_cython[size] = %timeit -o ai.medfilt1d_ng(img, nbin, method=meth)
print(" Python")
meth = tuple(method[:2]+["python"])
print(" * integrate ", end="")
perf2_integrate_python[size] = %timeit -o ai.integrate1d(img, nbin, method=meth)
print(" * medianfilter", end="")
perf2_medfilt_python[size] = %timeit -o ai.medfilt1d_ng(img, nbin, method=meth)
print(" OpenCL")
meth = tuple(method[:2]+["opencl"])
print(" * integrate ", end="")
perf2_integrate_opencl[size] = %timeit -o ai.integrate1d(img, nbin, method=meth)
print(" * medianfilter", end="")
perf2_medfilt_opencl[size] = %timeit -o ai.medfilt1d_ng(img, nbin, method=meth)
Pilatus1M.poni
Cython
* integrate
16 ms ± 3.05 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* medianfilter
19.7 ms ± 934 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Python
* integrate
13.6 ms ± 51.7 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
* medianfilter
1.18 s ± 9.05 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
OpenCL
* integrate
742 μs ± 2.22 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
* medianfilter
2.56 ms ± 14.8 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Pilatus2M.poni
Cython
* integrate
26.8 ms ± 3.37 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* medianfilter
69.9 ms ± 1.66 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Python
* integrate
46.9 ms ± 188 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
* medianfilter
4 s ± 25.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
OpenCL
* integrate
1.26 ms ± 2.27 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
* medianfilter
6.32 ms ± 33 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Eiger4M.poni
Cython
* integrate
34.3 ms ± 2.41 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* medianfilter
117 ms ± 1.25 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Python
* integrate
82.2 ms ± 178 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
* medianfilter
6.97 s ± 22.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
OpenCL
* integrate
2.37 ms ± 5.78 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
* medianfilter
10.9 ms ± 38.1 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Pilatus6M.poni
Cython
* integrate
42.8 ms ± 2.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* medianfilter
169 ms ± 40.5 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
Python
* integrate
105 ms ± 296 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
* medianfilter
10.3 s ± 35.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
OpenCL
* integrate
3.44 ms ± 55.3 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
* medianfilter
14.9 ms ± 48.6 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Eiger9M.poni
Cython
* integrate
65.6 ms ± 5.06 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* medianfilter
416 ms ± 189 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Python
* integrate
192 ms ± 349 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)
* medianfilter
17.1 s ± 37 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
OpenCL
* integrate
5.16 ms ± 12.8 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)
* medianfilter
24.2 ms ± 321 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Mar3450.poni
Cython
* integrate
71.7 ms ± 3.04 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* medianfilter
620 ms ± 163 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Python
* integrate
247 ms ± 1.24 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Compiler time: 0.23 s
* medianfilter
19.6 s ± 58.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
OpenCL
* integrate
5.99 ms ± 47.7 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)
* medianfilter
27.8 ms ± 7.91 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Fairchild.poni
Cython
* integrate
112 ms ± 4.24 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
* medianfilter
417 ms ± 3.89 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Python
* integrate
311 ms ± 359 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)
* medianfilter
24.1 s ± 20.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
OpenCL
* integrate
5.34 ms ± 41.6 μs per loop (mean ± std. dev. of 7 runs, 1 loop each)
* medianfilter
35.4 ms ± 18.2 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)
CPU times: user 24min 38s, sys: 30 s, total: 25min 8s
Wall time: 13min 39s
fig, ax = subplots()
ax.set_xlabel("Image size (Mpix)")
ax.set_ylabel("Frames per seconds")
sizes = numpy.array(list(perf2_integrate_python.keys()))/1e6
ax.plot(sizes, [1/i.best for i in perf2_integrate_python.values()], label="Integrate/Python", color='green', linestyle='dashed', marker='1')
ax.plot(sizes, [1/i.best for i in perf2_integrate_cython.values()], label="Integrate/Cython", color='orange', linestyle='dashed', marker='1')
ax.plot(sizes, [1/i.best for i in perf2_integrate_opencl.values()], label="Integrate/OpenCL", color='blue', linestyle='dashed', marker='1')
ax.plot(sizes, [1/i.best for i in perf2_medfilt_python.values()], label="Medfilt/Python", color='green', linestyle='dotted', marker='2')
ax.plot(sizes, [1/i.best for i in perf2_medfilt_cython.values()], label="Medfilt/Cython", color='orange', linestyle='dotted', marker='2')
ax.plot(sizes, [1/i.best for i in perf2_medfilt_opencl.values()], label="Medfilt/OpenCL", color='blue', linestyle='dotted', marker='2')
ax.set_yscale("log")
ax.legend()
ax.set_title("Performance of Median filtering vs integrate");
As one can see, the penalties are much larger for OpenCL and Python than for Cython.
Conclusion#
Sigma-clipping and median-filtering are alternatives to azimuthal integration and offer the ability to reject outliers. They are not more difficult to use but slightly slower owing to their greater complexity.