xhistogram: Fast, flexible, label-aware histograms for numpy and xarray

Histograms (a.k.a “binning”) are much more than just a visualization tool. They are the foundation of a wide range of scientific analyses including [joint] probability distributions and coordinate transformations. Xhistogram makes it easier to calculate flexible, complex histograms with multi-dimensional data. It integrates (optionally) with Dask, in order to scale up to very large datasets and with Xarray, in order to consume and produce labelled, annotated data structures. It is useful for a wide range of scientific tasks.

Why a new histogram package?

The main problem with the standard histogram function in numpy and dask is that they automatically act over the entire input array (i.e. they “flatten” the data). Xhistogram allows you to choose which axes / dimensions you want to preserve and which you want to flatten. It also allows you to combine N arbitrary inputs to produce N-dimensional histograms. A good place to start is the Xhistogram Tutorial.

Contents

Installation

Requirements

xhistogram is compatible with python 3. It requires numpy and, optionally, xarray.

Installation from Conda Forge

The easiest way to install xhistogram along with its dependencies is via conda forge:

conda install -c conda-forge xhistogram

Installation from Pip

An alternative is to use pip:

pip install xhistogram

This will install the latest release from pypi.

Installation from GitHub

xhistogram is under active development. To obtain the latest development version, you may clone the source repository and install it:

git clone https://github.com/xgcm/xhistogram.git
cd xhistogram
python setup.py install

or simply:

pip install git+https://github.com/xgcm/xhistogram.git

Users are encouraged to fork xhistogram and submit issues and pull requests.

Xhistogram Tutorial

Histograms are the foundation of many forms of data analysis. The goal of xhistogram is to make it easy to calculate weighted histograms in multiple dimensions over n-dimensional arrays, with control over the axes. Xhistogram builds on top of xarray, for automatic coordiantes and labels, and dask, for parallel scalability.

2D Histogram

Now let’s say we have multiple input arrays. We can calculate their joint distribution:

[7]:

db = xr.DataArray(np.random.randn(nt, nx), dims=['time', 'x'],
                  name='bar') - 2

histogram(da, db, bins=[bins, bins]).plot()

[7]:

<matplotlib.collections.QuadMesh at 0x7f0705fd0070>

Dask Integration

Should just work, but need examples.

API

Core Module

Numpy API for xhistogram.

xhistogram.core.histogram(*args, bins=None, range=None, axis=None, weights=None, density=False, block_size='auto')[source]

Histogram applied along specified axis / axes.

Parameters

argsarray_like

Input data. The number of input arguments determines the dimensionality of the histogram. For example, two arguments produce a 2D histogram. All args must have the same size.

binsint, str or numpy array or a list of ints, strs and/or arrays, optional

If a list, there should be one entry for each item in args. The bin specifications are as follows:

If int; the number of bins for all arguments in args.

If str; the method used to automatically calculate the optimal bin width for all arguments in args, as defined by numpy histogram_bin_edges.

If numpy array; the bin edges for all arguments in args.

If a list of ints, strs and/or arrays; the bin specification as above for every argument in args.

When bin edges are specified, all but the last (righthand-most) bin include the left edge and exclude the right edge. The last bin includes both edges.

A TypeError will be raised if args or weights contains dask arrays and bins are not specified explicitly as an array or list of arrays. This is because other bin specifications trigger computation.

range(float, float) or a list of (float, float), optional

If a list, there should be one entry for each item in args. The range specifications are as follows:

If (float, float); the lower and upper range(s) of the bins for all arguments in args. Values outside the range are ignored. The first element of the range must be less than or equal to the second. range affects the automatic bin computation as well. In this case, while bin width is computed to be optimal based on the actual data within range, the bin count will fill the entire range including portions containing no data.

If a list of (float, float); the ranges as above for every argument in args.

If not provided, range is simply (arg.min(), arg.max()) for each arg.

axisNone or int or tuple of ints, optional

Axis or axes along which the histogram is computed. The default is to compute the histogram of the flattened array

weightsarray_like, optional

An array of weights, of the same shape as a. Each value in a only contributes its associated weight towards the bin count (instead of 1). If density is True, the weights are normalized, so that the integral of the density over the range remains 1.

densitybool, optional

If False, the result will contain the number of samples in each bin. If True, the result is the value of the probability density function at the bin, normalized such that the integral over the range is 1. Note that the sum of the histogram values will not be equal to 1 unless bins of unity width are chosen; it is not a probability mass function.

block_sizeint or ‘auto’, optional

A parameter which governs the algorithm used to compute the histogram. Using a nonzero value splits the histogram calculation over the non-histogram axes into blocks of size block_size, iterating over them with a loop (numpy inputs) or in parallel (dask inputs). If 'auto', blocks will be determined either by the underlying dask chunks (dask inputs) or an experimental built-in heuristic (numpy inputs).

Returns

histarray: The values of the histogram.
bin_edgeslist of arrays: Return the bin edges for each input array.

Xarray Module

Xarray API for xhistogram.

xhistogram.xarray.histogram(*args, bins=None, range=None, dim=None, weights=None, density=False, block_size='auto', keep_coords=False, bin_dim_suffix='_bin')[source]

Histogram applied along specified dimensions.

Parameters

argsxarray.DataArray objects

Input data. The number of input arguments determines the dimensonality of the histogram. For example, two arguments prodocue a 2D histogram. All args must be aligned and have the same dimensions.

binsint, str or numpy array or a list of ints, strs and/or arrays, optional

If a list, there should be one entry for each item in args. The bin specifications are as follows:

If int; the number of bins for all arguments in args.

If str; the method used to automatically calculate the optimal bin width for all arguments in args, as defined by numpy histogram_bin_edges.

If numpy array; the bin edges for all arguments in args.

If a list of ints, strs and/or arrays; the bin specification as above for every argument in args.

When bin edges are specified, all but the last (righthand-most) bin include the left edge and exclude the right edge. The last bin includes both edges.

A TypeError will be raised if args or weights contains dask arrays and bins are not specified explicitly as an array or list of arrays. This is because other bin specifications trigger computation.

range(float, float) or a list of (float, float), optional

If a list, there should be one entry for each item in args. The range specifications are as follows:

If (float, float); the lower and upper range(s) of the bins for all arguments in args. Values outside the range are ignored. The first element of the range must be less than or equal to the second. range affects the automatic bin computation as well. In this case, while bin width is computed to be optimal based on the actual data within range, the bin count will fill the entire range including portions containing no data.

If a list of (float, float); the ranges as above for every argument in args.

If not provided, range is simply (arg.min(), arg.max()) for each arg.

dimtuple of strings, optional

Dimensions over which which the histogram is computed. The default is to compute the histogram of the flattened array.

weightsarray_like, optional

An array of weights, of the same shape as a. Each value in a only contributes its associated weight towards the bin count (instead of 1). If density is True, the weights are normalized, so that the integral of the density over the range remains 1. NaNs in the weights input will fill the entire bin with NaNs. If there are NaNs in the weights input call .fillna(0.) before running histogram().

densitybool, optional

If False, the result will contain the number of samples in each bin. If True, the result is the value of the probability density function at the bin, normalized such that the integral over the range is 1. Note that the sum of the histogram values will not be equal to 1 unless bins of unity width are chosen; it is not a probability mass function.

block_sizeint or ‘auto’, optional

A parameter which governs the algorithm used to compute the histogram. Using a nonzero value splits the histogram calculation over the non-histogram axes into blocks of size block_size, iterating over them with a loop (numpy inputs) or in parallel (dask inputs). If 'auto', blocks will be determined either by the underlying dask chunks (dask inputs) or an experimental built-in heuristic (numpy inputs).

keep_coordsbool, optional

If True, keep all coordinates. Default: False

bin_dim_suffixstr, optional

Suffix to append to input arg names to define names of output bin dimensions

Returns

histxarray.DataArray: The values of the histogram. For each bin, the midpoint of the bin edges is given along the bin coordinates.

Contributor Guide

This package is in very early stages. Lots of work is needed.

You can help out just by using xhistogram and reporting issues.

The following sections cover some general guidelines for maintainers and contributors wanting to help develop xhistogram.

Feature requests, suggestions and bug reports

We are eager to hear about any bugs you have found, new features you would like to see and any other suggestions you may have. Please feel free to submit these as issues.

When suggesting features, please make sure to explain in detail how the proposed feature should work and to keep the scope as narrow as possible. This makes features easier to implement in small PRs.

When report bugs, please include:

Any details about your local setup that might be helpful in troubleshooting, specifically the Python interpreter version, installed libraries, and xhistogram version.
Detailed steps to reproduce the bug, ideally a Minimal, Complete and Verifiable Example.
If possible, a demonstration test that currently fails but should pass when the bug is fixed.

Write documentation

Adding documentation is always helpful. This may include:

More complementary documentation. Have you perhaps found something unclear?
Docstrings.
Example notebooks of xhistogram being used in real analyses.

The xhistogram documentation is written in reStructuredText. You can follow the conventions in already written documents. Some helpful guides can be found here and here.

When writing and editing documentation, it can be useful to see the resulting build without having to push to Github. You can build the documentation locally by running:

$ # Install the packages required to build the docs in a conda environment
$ conda env create -f doc/environment.yml
$ conda activate xhistogram_doc_env
$ # Install the latest xhistogram
$ pip install --no-deps -e .
$ cd doc/
$ make html

This will build the documentation locally in doc/_build/. You can then open _build/html/index.html in your web browser to view the documentation. For example, if you have xdg-open installed:

$ xdg-open _build/html/index.html

To lint the reStructuredText documentation files run:

$ doc8 doc/*.rst

Preparing Pull Requests

Fork the xhistogram GitHub repository. It’s fine to use xhistogram as your fork repository name because it will live under your username.

Clone your fork locally, connect your repository to the upstream (main project), and create a branch to work on:

$ git clone git@github.com:YOUR_GITHUB_USERNAME/xhistogram.git
$ cd xhistogram
$ git remote add upstream git@github.com:xgcm/xhistogram.git
$ git checkout -b your-bugfix-feature-branch-name master

If you need some help with Git, follow this quick start guide

Install dependencies into a new conda environment:

$ conda env create -f ci/environment-3.9.yml
$ conda activate xhistogram_test_env

Install xhistogram using the editable flag (meaning any changes you make to the package will be reflected directly in your environment):
```
$ pip install --no-deps -e .
```
Start making your edits. Please try to type annotate your additions as much as possible. Adding type annotations to existing unannotated code is also very welcome. You can read about Python typing here.
Break your edits up into reasonably sized commits:
```
$ git commit -a -m "<commit message>"
$ git push -u
```
It can be useful to manually run pre-commit as you make your edits. pre-commit will run checks on the format and typing of your code and will show you where you need to make changes. This will mean your code is more likely to pass the CI checks when you push it:
```
$ pip install pre_commit # you only need to do this once
$ pre-commit run --all-files
```
Run the tests (including those you add to test your edits!):
```
$ pytest xhistogram
```
You can also test that your contribution and tests increased the test coverage:
```
$ coverage run --source xhistogram -m py.test
$ coverage report
```
Add a new entry describing your contribution to the Release History in doc/contributing.rst. Please try to follow the format of the existing entries.
Submit a pull request through the GitHub website.

Note that you can create the Pull Request while you’re working on your PR. The PR will update as you add more commits. xhistogram developers and contributors can then review your code and offer suggestions.

Release History

v0.3.2

Fix bug producing TypeError when weights is provided with keep_coords=True GH78. By Dougie Squire.
Raise TypeError when weights is a dask array and bin edges are not explicitly provided GH12. By Dougie Squire.

v0.3.1

Add DOI badge and CITATION.cff. By Julius Busecke.

v0.3.0

Add support for histograms over non-float dtypes (e.g. datetime objects) GH25. By Dougie Squire.
Refactor histogram calculation to use dask.array.blockwise when input arguments are dask arrays, resulting in significant performance improvements GH49. By Ryan Abernathy, Tom Nicholas and Gabe Joseph.
Fixed bug with density calculation when NaNs are present GH51. By Dougie Squire.
Implemented various options for users for providing bins to xhistogram that mimic the numpy histogram API. This included adding a range argument to the xhistogram API GH13. By Dougie Squire.
Added a function to check if the object passed to xhistogram is an xarray.DataArray and if not, throw an error. GH14. By Yang Yunyi.

v0.2.0

Added FutureWarning for upcoming changes to core API GH13. By Dougie Squire.
Added documentation on how to deal with NaNs in weights GH26. By Shanice Bailey.
Move CI to GitHub actions GH32. By James Bourbeau.
Add documentation for contributors. By Dougie Squire.
Add type checking with mypy GH32. By Dougie Squire.

v0.1.3

Update dependencies to exclude incompatible dask version GH27. By Ryan Abernathey.

v0.1.2

Aligned definition of bins with numpy.histogram GH18. By Dougie Squire.

v0.1.1

Minor bugfix release

Imroved documentation examples. By Dhruv Balwada.
Fixed issue GH5 related to incorrect dimension order and dropping of dimension coordinates. By Ryan Abernathey.

v0.1

First release