Logarithmic Binning

An important problem when dealing with experimental data, whether this data originated in a “real” experiment in a numerical one, is to reduce the ammount of stochastic noise. This is specially important when trying to determine the distribution P(x) of a given quantity quantity x. A common approach is to apply some form of logarithmic binning to the data and then proceed to analyze the cleaner, “binned” data. In this case, and contrarily to what is done for histograms, for example, the bin size is not constant and is, instead, given by:

[tex]\Delta=\frac{1}{N}\log\left(\frac{b}{a}\right)[/tex]

where N is the desired number of bins, and b and a are, respectively, the maximum and th minimum of the dataset. Each data point, x_i is assigned to the bin:

[tex]n_i=\frac{1}{\Delta}\log\left(\frac{x_i}{a}\right)[/tex]

to which corresponds the new xx value X, given by:

[tex]X_i=ae^{\Delta\left(n_i+1\right)}[/tex]

and the value of the distribution P(X_i) is simply the arithmetic average of all the points in the bin:

[tex]P\left(X_i\right)=\frac{1}{N_i}\sum_{x_i\in \{n_i\}}x_i[/tex]

where N_i is the number of data points in bin n_i and the sum is taken over all x_i, in the bin. This method preserves the functional behavior of the distribution while averaging out fluctuations which are specially dominant in regions with poor statistics. An example is shown below where we clearly see that the statistical fluctuations in the tail are smoothed out.

This procedure can be easily implemented in just a few lines of awk, allowing you to quickly log bin your datafiles. You can download my script from the link below, where I have also included some code to properly deal with error bars (assuming that’s the third column on your file). What other techniques do you use to clean up your data?