500 - Internal Server Error: Failed to create socket: Too many open files

1. offizieller Beitrag

    I'm trying to add a drive to my system but keep getting hung in Storage | File Systems with a 500 - Internal Server Error. Where do I start in resolving this?

    Thanks.

    ---

    Failed to create socket: Too many open files

    OMV\Rpc\Exception: Failed to create socket: Too many open files in /usr/share/php/openmediavault/rpc/serviceabstract.inc:615

    Stack trace:

    #0 /usr/share/openmediavault/engined/rpc/filesystemmgmt.inc(214): OMV\Rpc\ServiceAbstract->asyncProc(Object(Closure))

    #1 [internal function]: Engined\Rpc\OMVRpcServiceFileSystemMgmt->enumerateFilesystems(NULL, Array)

    #2 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)

    #3 /usr/share/openmediavault/engined/rpc/filesystemmgmt.inc(380): OMV\Rpc\ServiceAbstract->callMethod('enumerateFilesy...', NULL, Array)

    #4 [internal function]: Engined\Rpc\OMVRpcServiceFileSystemMgmt->getList(Array, Array)

    #5 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)

    #6 /usr/share/php/openmediavault/rpc/serviceabstract.inc(149): OMV\Rpc\ServiceAbstract->callMethod('getList', Array, Array)

    #7 /usr/share/php/openmediavault/rpc/serviceabstract.inc(588): OMV\Rpc\ServiceAbstract->OMV\Rpc\{closure}('/tmp/bgstatusKx...', '/tmp/bgoutput0Q...')

    #8 /usr/share/php/openmediavault/rpc/serviceabstract.inc(159): OMV\Rpc\ServiceAbstract->execBgProc(Object(Closure))

    #9 /usr/share/openmediavault/engined/rpc/filesystemmgmt.inc(519): OMV\Rpc\ServiceAbstract->callMethodBg('getList', Array, Array)

    #10 [internal function]: Engined\Rpc\OMVRpcServiceFileSystemMgmt->getListBg(Array, Array)

    #11 /usr/share/php/openmediavault/rpc/serviceabstract.inc(123): call_user_func_array(Array, Array)

    #12 /usr/share/php/openmediavault/rpc/rpc.inc(86): OMV\Rpc\ServiceAbstract->callMethod('getListBg', Array, Array)

    #13 /usr/sbin/omv-engined(537): OMV\Rpc\Rpc::call('FileSystemMgmt', 'getListBg', Array, Array, 1)

    #14 {main}

    Version7.0-32 (Sandworm)
    ProcessorAMD EPYC 7302P 16-Core Processor
    KernelLinux 6.1.15-1-pve
    HardwareDell R7515

    Is there a way to fix this error, so I would be able to use the graphical interface?

    If not how do I add a new drive to the system without web access?

    Version7.0-32 (Sandworm)
    ProcessorAMD EPYC 7302P 16-Core Processor
    KernelLinux 6.1.15-1-pve
    HardwareDell R7515

    our telepathic capabilities are quite limited and there is no mention of what you tried to fix this already

    omv 6.9.6-2 (Shaitan) on RPi CM4/4GB with 64bit Kernel 6.1.21-v8+

    2x 6TB 3.5'' HDDs (CMR) formatted with ext4 via 2port PCIe SATA card with ASM1061R chipset providing hardware supported RAID1


    omv 6.9.3-1 (Shaitan) on RPi4/4GB with 32bit Kernel 5.10.63 and WittyPi 3 V2 RTC HAT

    2x 3TB 3.5'' HDDs (CMR) formatted with ext4 in Icy Box IB-RD3662-C31 / hardware supported RAID1

    For Read/Write performance of SMB shares hosted on this hardware see forum here

    Too many open files means, some programs hold many file handles (sockets are files in unix).

    Either find out who is eating up your file handles or

    increase the limits or

    try if a reboot fixes that.

    If you got help in the forum and want to give something back to the project click here (omv) or here (scroll down) (plugins) and write up your solution for others.

    Zitat von Zoki

    try if a reboot fixes that.

    would be my first approach for any issue too :)


    ajaja I'd recommend to read https://opensource.com/life/16…w-ask-technical-questions specifically the part "Research"

    omv 6.9.6-2 (Shaitan) on RPi CM4/4GB with 64bit Kernel 6.1.21-v8+

    2x 6TB 3.5'' HDDs (CMR) formatted with ext4 via 2port PCIe SATA card with ASM1061R chipset providing hardware supported RAID1


    omv 6.9.3-1 (Shaitan) on RPi4/4GB with 32bit Kernel 5.10.63 and WittyPi 3 V2 RTC HAT

    2x 3TB 3.5'' HDDs (CMR) formatted with ext4 in Icy Box IB-RD3662-C31 / hardware supported RAID1

    For Read/Write performance of SMB shares hosted on this hardware see forum here

    Einmal editiert, zuletzt von mi-hol ()

    Zitat von Zoki

    Too many open files means, some programs hold many file handles (sockets are files in unix).

    Either find out who is eating up your file handles or

    increase the limits or

    try if a reboot fixes that.

    Does not go away with reboot. Other than basic file services via SMB/CIFS, I run a number of Docker containers; unfortunately this error continues when I shut down all my containers.

    How might I deduce what is eating my file handlers, and/or how do I increase the limits?

    Version7.0-32 (Sandworm)
    ProcessorAMD EPYC 7302P 16-Core Processor
    KernelLinux 6.1.15-1-pve
    HardwareDell R7515

    these are plenty of file handles. You are not hit by this limit. Try the other solutions in the thread. Use the lsof ... to find who is using sockets files.

    If you got help in the forum and want to give something back to the project click here (omv) or here (scroll down) (plugins) and write up your solution for others.

    at end of post is the part:

    "NGINX worker processes were launched by NGINX user. Ulimit (soft and hard) for nginx user was 65536. The ulimit and setting limits.conf did not work.

    The rlimit setting in nginx.conf did not help either: worker_rlimit_nofile 65536;

    The solution that worked was:

    Code
    $ mkdir -p /etc/systemd/system/nginx.service.d
$ nano /etc/systemd/system/nginx.service.d/nginx.conf    [Service]    LimitNOFILE=30000
$ systemctl daemon-reload
$ systemctl restart nginx.service

    "

    omv 6.9.6-2 (Shaitan) on RPi CM4/4GB with 64bit Kernel 6.1.21-v8+

    2x 6TB 3.5'' HDDs (CMR) formatted with ext4 via 2port PCIe SATA card with ASM1061R chipset providing hardware supported RAID1


    omv 6.9.3-1 (Shaitan) on RPi4/4GB with 32bit Kernel 5.10.63 and WittyPi 3 V2 RTC HAT

    2x 3TB 3.5'' HDDs (CMR) formatted with ext4 in Icy Box IB-RD3662-C31 / hardware supported RAID1

    For Read/Write performance of SMB shares hosted on this hardware see forum here

    Zitat von Zoki

    Try the other solutions in the thread. Use the lsof ... to find who is using sockets files.

    root@omv:~# lsof -u `whoami` | wc -l

    3902

    I have no idea what that means.

    Version7.0-32 (Sandworm)
    ProcessorAMD EPYC 7302P 16-Core Processor
    KernelLinux 6.1.15-1-pve
    HardwareDell R7515
    Zitat von mi-hol

    NGINX

    Should I be doing this if my initial thought is, "What is NGINX?"
    It looks like a load balancer. Why would I do that?

    Version7.0-32 (Sandworm)
    ProcessorAMD EPYC 7302P 16-Core Processor
    KernelLinux 6.1.15-1-pve
    HardwareDell R7515

    Einmal editiert, zuletzt von ajaja ()

    ryecoaaron any chance there is a simple way out of this loop?

    ryecoaaron

    ryecoaaron

    ryecoaaron

    ryecoaaron

    Version7.0-32 (Sandworm)
    ProcessorAMD EPYC 7302P 16-Core Processor
    KernelLinux 6.1.15-1-pve
    HardwareDell R7515
    • Offizieller Beitrag

    As a temporary check to see if this fixes, try:

    echo -e "* soft nofile unlimited\n* hard nofile unlimited" | sudo tee -a /etc/security/limits.conf


    Then reboot

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Unfortunately no difference.

    Zitat von ryecoaaron

    As a temporary check to see if this fixes, try:

    echo -e "* soft nofile unlimited\n* hard nofile unlimited" | sudo tee -a /etc/security/limits.conf


    Then reboot

    root@omv:~# echo -e "* soft nofile unlimited\n* hard nofile unlimited" | sudo tee -a /etc/security/limits.conf

    * soft nofile unlimited

    * hard nofile unlimited


    Version7.0-32 (Sandworm)
    ProcessorAMD EPYC 7302P 16-Core Processor
    KernelLinux 6.1.15-1-pve
    HardwareDell R7515
    • Offizieller Beitrag

    It is unlikely to be an OS setting then. Maybe change rlimit_files in /etc/php/7.4/fpm/php-fpm.conf and restart php-fpm? But I'm really curious why that would be needed.

    omv 7.0.5-1 sandworm | 64 bit | 6.8 proxmox kernel

    plugins :: omvextrasorg 7.0 | kvm 7.0.13 | compose 7.1.4 | k8s 7.1.0-3 | cputemp 7.0.1 | mergerfs 7.0.4


    omv-extras.org plugins source code and issue tracker - github - changelogs


    Please try ctrl-shift-R and read this before posting a question.

    Please put your OMV system details in your signature.
    Please don't PM for support... Too many PMs!

    Zitat von ryecoaaron

    It is unlikely to be an OS setting then. Maybe change rlimit_files in /etc/php/7.4/fpm/php-fpm.conf and restart php-fpm? But I'm really curious why that would be needed.

    Doubled from 1024 to 2048; no go.

    Version7.0-32 (Sandworm)
    ProcessorAMD EPYC 7302P 16-Core Processor
    KernelLinux 6.1.15-1-pve
    HardwareDell R7515

    How many open files does your system have:


    lsof | wc -l # may take a while to complete


    I have 89812 open file handles in total


    for omv:

    Code
    root@omv:/etc# lsof -u openmediavault | wc -l
233


    for www-data (web server)

    Code
    root@omv:/etc# lsof -u www-data | wc -l
1124


    for root:

    Code
    root@omv:/etc# lsof -u root | wc -l
4851


    and for my docker apps (your path may be different)


    Code
    root@omv:/etc# lsof +D /srv/docker-apps | wc -l
943

    and for docker itself (your path my be different e.g. /var/lib/docker)


    user 1013 is a user inside a container I o not need a user in the host.



    Look here for more examples:

    https://www.thegeekstuff.com/2012/08/lsof-command-examples/

    If you got help in the forum and want to give something back to the project click here (omv) or here (scroll down) (plugins) and write up your solution for others.

    root@omv:~# /etc# lsof -u openmediavault | wc -l

    -bash: /etc#: No such file or directory

    0

    root@omv:~# /etc# lsof -u www-data | wc -l

    -bash: /etc#: No such file or directory

    0

    root@omv:~# /etc# lsof -u root | wc -l

    -bash: /etc#: No such file or directory

    0

    root@omv:~# /etc# lsof +D /srv/docker-apps | wc -l

    -bash: /etc#: No such file or directory

    0

    root@omv:~# /etc# lsof +D /srv/docker-root | wc -l

    -bash: /etc#: No such file or directory

    0

    zero


    Version7.0-32 (Sandworm)
    ProcessorAMD EPYC 7302P 16-Core Processor
    KernelLinux 6.1.15-1-pve
    HardwareDell R7515

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!

Benutzerkonto erstellen Anmelden