xarray NetCDF LRU cache replaced autoclose

Python code using xarray.open_dataset() or xarray.open_dataarray() or similar functions reading from NetCDF4 use an LRU cache that automatically closes unneeded files.

As of xarray 0.11.0, the obsolete autoclose=True option should no longer be used.

Problems fixed by LRU cache

The LRU cache used to open NetCDF4 files with xarray fixes these problems, and gives high performance:

  • random segmentation fault while reading NetCDF4 .nc files, where the same file is reopened in the program.
  • OSError from too many open files, where even increasing ulimit doesn’t help

Tags: ,

Categories:

Written by Michael Hirsch, Ph.D. //

Comments