Linux: Increasing the number of open file descriptors

See additional more recent notes at the bottom of this file

A small number of open file descriptors (sockets) can significantly reduce both the performance of an Internet Server and the load that workload generator like httperf can generate. This is meant to provide some information about how to increase the limits on the number of open file descriptors (sockets) on Linux. Note: the actual numbers used below are examples. The numbers you should use will depend on weather you are modifying a system that will be used as a client or a server and the load being generated. In this example we increase the limit to 65535.

Also note that some of these steps may or may not be required depending on whether you are using PAM and if some of the stuff is being done remotely using ssh.

  1. To check and modify system limits.
    [The current limit shown is 8192]
    % cat /proc/sys/fs/file-max
    8192
    
    [To increase this to 65535 (as root)]
    # echo "65535" > /proc/sys/fs/file-max
    

    If you want this new value to survive across reboots you can at it to /etc/sysctl.conf

    # Maximum number of open files permited
    fs.file-max = 65535
    
    Note: that this isn't proc.sys.fs.file-max as one might expect.

    To list the available parameters that can be modified using sysctl do

    % sysctl -a 
    

    To load new values from the sysctl.conf file.

    % sysctl -p /etc/sysctl.conf
    

  2. Modify your software to make use of a larger number of open FDs.
    [Find out where __FD_SETSIZE is defined]
    % grep "#define __FD_SETSIZE" /usr/include/*.h /usr/include/*/*.h
    /usr/include/bits/types.h:#define __FD_SETSIZE  1024
    /usr/include/linux/posix_types.h:#define __FD_SETSIZE   1024
    
    Alternately your system may use tabs in which case
    something like this should do the trick:
    % grep -E "#define\W+__FD_SETSIZE" /usr/include/*.h /usr/include/*/*.h
    [Thanks to madRat for this tip].
    
    
    [Make a local copy of these files]
    % cp /usr/include/bits/types.h include/bits/types.h
    % cp /usr/include/linux/posix_types.h include/linux/posix_types.h
    
    [Modify them so that they look something like
    #define __FD_SETSIZE  65535
    
    [Recompile httperf and/or your server so that it uses a larger file
    descriptor set size by using -I include during compliation, this
    will allow the compiler/preprocessor to use the new include files
    rather than the default versions]
    
    
  3. To check and modify limits per shell.
    [Using csh: openfiles and descriptors show that the limit here is 1024]
    % limit
    cputime         unlimited
    filesize        unlimited
    datasize        unlimited
    stacksize       8192 kbytes
    coredumpsize    0 kbytes
    memoryuse       unlimited
    descriptors     1024 
    memorylocked    unlimited
    maxproc         8146 
    openfiles       1024 
    
    [To increase this to 65535 for all users (as root)]
    # vi /etc/security/limits.conf
    
    [Modify or add "nofile" (number of file) entries - note
    that a userid can be used in place of *]
    *                soft    nofile          65535
    *                hard    nofile          65535
    
    # vi /etc/pam.d/login
    [Add the line]
    session    required   /lib/security/pam_limits.so
    
    [On many systems this will be sufficient - log in as a regular
    user and try it before doing the following steps]
    
    [These steps may be required depending on how PAM and ssh are configured
    [Note on some systems - Debian?]
    # vi /etc/pam.d/ssh
    [Note on other systems - RedHat]
    # vi /etc/pam.d/sshd
    [Add the line]
    session    required   /lib/security/pam_limits.so
    
    # vi /etc/ssh/sshd_config
    [May need to modify or add the line]
    UsePrivilegeSeparation no
    
    [Restart the ssh daemon]
    [Note on some systems - Debian?]
    # /etc/init.d/ssh reload
    [Note on other systems - RedHat]
    # /etc/rc.d/init.d/sshd reload
    
    
    NOTE: it may still be necessary in some cases to adjust the limits manually.
    
    In tcsh
    limit descriptors 65535
    
    In bash
    ulimit -n 65535
    
    
I've had problems with these instructions in some cases (IIRC, Fedora 7). I haven't used this but one suggestion is after sshing to the next machine. su - $USER Another bit of info I've seen is that in /etc/ssh/sshd_config You want PAMAuthenticationViaKbdInt yes

Created: Wed Jan 22 15:08:58 EST 2003 Last modified: Fri Dec 18 12:27:15 EST 2009