Increase ulimit number of open files

Why increase ulimit -n maximum number of open files?

Making Matplotlib/Seaborn plots in parallel executing threads in Python is one area where you may need to increase ulimit -n setting. That is, increase the limit for the maximum number of open files. The default maximum number of open file limits are a bit low

Linux (Ubuntu) Mac OS X (Yosemite/El Capitan)
1024 256

This can lead to errors from Python with Matplotlib like:

OSError: [Errno 24] Too many open files: ‘/usr/share/fonts/truetype/msttcorefonts/arial.ttf’

Fix

Platform independent ulimit -n increase “ulimit_nofile.py”

In general I avoid one-off or sudo-requiring system parameter setting techniques, and would defer to using platform-independent Python since virtually all of my programs at least incidentally use Python. This does not use sudo. It instead makes temporary setting changes where feasible or at most user-profile specific system changes. This change will be only for this instance of Python; once Python terminates even the same terminal session will have the original ulimit again.

#!/usr/bin/env python
try:
    import resource as res
except ImportError: #Windows
    res = None

def raise_nofile(nofile_atleast=4096):
    """
    sets nofile soft limit to at least 4096, useful for running matlplotlib/seaborn on
    parallel executing plot generators vs. Ubuntu 16.04 default ulimit -n 1024 or OS X El Captian 256
    temporary setting extinguishing with Python session.
    """
    if res is None:
        return (None,)*2
#%% (0) what is current ulimit -n setting?
    soft,ohard = res.getrlimit(res.RLIMIT_NOFILE)
    hard = ohard
#%% (1) increase limit (soft and even hard) if needed
    if soft<nofile_atleast:
        soft = nofile_atleast

        if hard<soft:
            hard = soft

        print('setting soft & hard ulimit -n {} {}'.format(soft,hard))
        try:
            res.setrlimit(res.RLIMIT_NOFILE,(soft,hard))
        except (ValueError,res.error):
            try:
               hard = soft
               print('trouble with max limit, retrying with soft,hard {},{}'.format(soft,hard))
               res.setrlimit(res.RLIMIT_NOFILE,(soft,hard))
            except Exception:
               print('failed to set ulimit, giving up')
               soft,hard = res.getrlimit(res.RLIMIT_NOFILE)

    return soft,hard


if __name__ == '__main__':
    from argparse import ArgumentParser
    p = ArgumentParser()
    p.add_argument('-n','--nofile',help='max number of open files',type=int,default=4096)
    p = p.parse_args()

    soft,hard = raise_nofile(p.nofile)
    print('ulimit -n soft,hard: {},{}'.format(soft,hard))

Leave a Comment