Not even sure where to start - some services failing to run on 3.0.64

    • OMV 3.x

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • Not even sure where to start - some services failing to run on 3.0.64

      Put together an offsite backup NAS and got it mostly working, then took the machine to my parents' house 4 hours away and put it in place. It's a Core i3-530 with 8GB ECC RAM and 3x1TB drives in ZFS RAIDZ1. I was able to do a zfs send/receive to get a baseline on the server for the data, with the intent to be using syncthing via a custom tinc VPN install to sync data between this and my main NAS at my house. Note, I've been doing this for some time, but the reason for the change was to upgrade to 3.x so the ZFS plugin functions with snapshots, and to get some better hardware in there - the old one was an Atom D525 with 4GB RAM.

      Now, the Tinc VPN setup is working. I can SSH into the server just fine, web UI is working great, Dashboard page shows all services that have been activated as green (in this case, Syncthing, Samba, TFTP, and SSH). The only one that works is SSH. Samba, Syncthing, and TFTP (mostly activated to just see if it worked) all show, after "systemctl restart xxxxx" that everything started, but it's "Active (Exited)" in status:


      Source Code

      1. root@mjolnir:~# systemctl restart syncthing
      2. root@mjolnir:~# systemctl status syncthing
      3. ● syncthing.service - LSB: syncthing
      4. Loaded: loaded (/etc/init.d/syncthing)
      5. Active: active (exited) since Mon 2017-03-06 20:19:32 EST; 1s ago
      6. Process: 16675 ExecStop=/etc/init.d/syncthing stop (code=exited, status=0/SUCCESS)
      7. Process: 16682 ExecStart=/etc/init.d/syncthing start (code=exited, status=0/SUCCESS)
      8. Mar 06 20:19:32 mjolnir syncthing[16682]: Starting Syncthing for user root
      9. Mar 06 20:19:32 mjolnir systemd[1]: Started LSB: syncthing.
      10. root@mjolnir:~#
      Display All
      Note, "systemctl start syncthing" did not do anything - it was showing "Active (Exited)" then timestamped 27 hours previous even after the "start". Issuing "restart" re-runs it, but the web interface isn't accessible. Same with SMB, start does nothing, restart reloads it, there's no errors I can find anywhere, but the system is completely unreachable via SMB/CIFS.

      Even better, if I just run syncthing from a bash prompt, not only does it run, but the web interface is accessible. I've not attempted something similar with SMB, because I can't quite see how that would work out, and I'm now 4 hours away over mildly slow internet connection.

      I even tried rebooting. Really can't figure out what the heck is going on here, so if anyone's got any ideas, I'm willing to try them. I've attached the Report file from the System Information tab in case that provides something useful.

      Thanks!
      Files
    • Not that I can see. A search for error in the syslog pulls up nothing worth mentioning (SMART completed without error, etc.), and searching syncthing comes up with this, and nothing else:

      Source Code

      1. Mar 6 20:19:20 mjolnir systemd[1]: Started LSB: syncthing.
      2. Mar 6 20:19:32 mjolnir systemd[1]: Stopping LSB: syncthing...
      3. Mar 6 20:19:32 mjolnir systemd[1]: Starting LSB: syncthing...
      4. Mar 6 20:19:32 mjolnir syncthing[16682]: Starting Syncthing for user root
      5. Mar 6 20:19:32 mjolnir systemd[1]: Started LSB: syncthing.
      6. M
      The Syncthing logs are empty, which since it doesn't actually run at all, isn't overly surprising.
    • Looking in daemon.log, I do see this a couple of times from yesterday:


      Source Code

      1. Mar 5 16:44:43 mjolnir systemd[1]: Stopping LSB: syncthing...
      2. Mar 5 16:44:43 mjolnir systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument
      3. Mar 5 16:44:43 mjolnir systemd[1]: Starting LSB: syncthing...
      4. Mar 5 16:44:43 mjolnir syncthing[4241]: Starting Syncthing for user thing
      5. Mar 5 16:44:43 mjolnir systemd[1]: Started LSB: syncthing.
      6. M
      I was assuming that that's because it's stopping services that aren't actually running, and that's what was causing the error, but I'm including it for completeness' sake.
    • I just installed the virtualbox, syncthing, remotedesktop, and zfs plugins and enabled samba and tftp as well. Everything is working. Very strange. Is this on an ssd?
      omv 4.0.6 arrakis | 64 bit | 4.12 backports kernel | omvextrasorg 4.1.0
      omv-extras.org plugins source code and issue tracker - github.com/OpenMediaVault-Plugin-Developers

      Please don't PM for support... Too many PMs!
    • ryecoaaron wrote:

      I just installed the virtualbox, syncthing, remotedesktop, and zfs plugins and enabled samba and tftp as well. Everything is working. Very strange. Is this on an ssd?
      No it isn't, it's on a 2.5" 60GB 5400RPM drive. I did check the SMART data and there's no reallocs, no pendings, nothing that would indicate any real issues whatsoever. There IS an error listed, but I had it in a 2.5" portable eSATA enclosure at one point that was seriously flaked out and I know that's when the error happened. It's an older drive sir, but it checks out. :)

      I did try uninstalling and reinstalling the Syncthing plugin, but I haven't tried uninstalling anything else or adding any additional ones. I just can't figure out why the heck existing services seem to work (SSH) but these three won't and why they aren't apparently throwing errors when whatever it is happening actually happens.
    • Something else screwy going on. I went into my main NAS to check for updates, saw there was updates for something related to Mono, the latest omv-extras package (3.4.22), and the latest Syncthing (0.14.24). Ran the updates, all good.

      Went in to the problem box and checked for updates because I hadn't had any since I installed it, which was bothering me for some reason, and I wanted to make sure it was working right. Only thing waiting was Syncthing. Mono isn't enabled, but still should have 3.4.22 of omv-extras. Did a Check, no change.

      SSH into the box, do an apt-get update, goes through everything, get to the end and see:

      Source Code

      1. Reading package lists... Done
      2. W: Conflicting distribution: https://downloads.plex.tv public InRelease (expected public but got )
      3. W: GPG error: https://dl.bintray.com jessie Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7B9C5D187860A018
      4. W: GPG error: https://dl.bintray.com jessie Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7B9C5D187860A018
      5. W: GPG error: https://dl.bintray.com jessie Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7B9C5D187860A018

      Alright, something weird there too, not sure how to deal with that or if it's even related. Then I do an apt-get upgrade, and I see a WHOLE bunch of stuff that isn't coming up:


      Source Code

      1. root@mjolnir:~# apt-get upgrade
      2. Reading package lists... Done
      3. Building dependency tree
      4. Reading state information... Done
      5. Calculating upgrade... The following package was automatically installed and is no longer required:
      6. libasprintf0c2
      7. Use 'apt-get autoremove' to remove it.
      8. Done
      9. The following packages have been kept back:
      10. btrfs-tools ifupdown jq libsystemd0 libudev1 python3-pyudev rsyslog systemd udev xrdp
      11. The following packages will be upgraded:
      12. bash-completion dh-python dmidecode gettext-base iucode-tool libapparmor1 libasprintf0c2 libcomerr2 libdrm2 libgeoip1 libgl1-mesa-glx libglapi-mesa libglib2.0-0 libgpg-error0 libjs-jquery libjs-sphinxdoc
      13. libss2 manpages monit python-dnspython shared-mime-info smartmontools socat syncthing systemd-sysv tinc
      14. 26 upgraded, 0 newly installed, 0 to remove and 10 not upgraded.
      15. Need to get 7,544 kB/13.7 MB of archives.
      16. After this operation, 1,356 kB of additional disk space will be used.
      17. Do you want to continue? [Y/n] n
      18. Abort.
      19. root@mjolnir:~#
      Display All

      Doing a dist-upgrade now to get all the kept-back packages; will see where things stand afterwards. I'll probably need to do a reboot, which I'm leery of with the box 4 hours away and two elderly parents as the only remote hands available on a headless box, but we'll see.
    • If the reboot doesn't help, I might suspect a bad dimm? I can't think of any other reason all of these services wouldn't start especially since they aren't really related. Syncthing might have a problem if storage (zfs) isn't ready but samba shouldn't care.

      The bintray key is the old omv-extras signing key. There is still packages in the repos signed with that key. I will have to add it back to omv-extras. Until then, it is just a warning.

      As for the upgrades, I don't see any real dangerous upgrades in the list.
      omv 4.0.6 arrakis | 64 bit | 4.12 backports kernel | omvextrasorg 4.1.0
      omv-extras.org plugins source code and issue tracker - github.com/OpenMediaVault-Plugin-Developers

      Please don't PM for support... Too many PMs!
    • Syncthing isn't even configured - the only time I got it to run was running it on the command line.

      Update went all Hulk on my VPN connection (TINC SMASH!) and I had to back door into it via port forwarding on my parents' router (thank goodness for "Linksys Smart Wifi"). Reboot restored that, and SSH never stopped working or I'd really be hosed, but the reboot did not start up SMB, Syncthing, TFTP, or XRDP.

      Just to try it, I'm going through and removing plugins I don't really need. In this case, it's Sensors (which didn't work anyhow; errors every time I try and apply configs - sensors command in shell works just fine though), Remote-mount, remotedesktop, and locate.

      Nope, still won't run. I was kind of hoping this would get somewhere since the update included a change from Systemd 215 to 230, but doesn't look like it.
    • I think we should figure the samba issue out and maybe the other services will have a similar fix. What is the output of:

      systemctl status smbd.service
      systemctl status nmbd.service
      omv 4.0.6 arrakis | 64 bit | 4.12 backports kernel | omvextrasorg 4.1.0
      omv-extras.org plugins source code and issue tracker - github.com/OpenMediaVault-Plugin-Developers

      Please don't PM for support... Too many PMs!
    • Works for me. As a note - and I have no idea why this is - I was able to do "systemctl enable syncthing@user.service" and "systemctl start syncthing@user.service" and syncthing is functioning now. I did NOT need to do that on my main NAS, and trying to do systemctl status syncthing@user.service" comes back with the result that it's dead, despite that being the user it's running as, so clearly something weird going on here.

      On to SMB, here's the output for smbd.service:

      Source Code

      1. wolfstar@mjolnir:~$ systemctl status smbd.service
      2. ● smbd.service - LSB: start Samba SMB/CIFS daemon (smbd)
      3. Loaded: loaded (/etc/init.d/smbd; generated; vendor preset: enabled)
      4. Active: active (exited) since Tue 2017-03-07 10:45:41 EST; 1h 1min ago
      5. Docs: man:systemd-sysv-generator(8)
      6. CGroup: /system.slice/smbd.service
      7. Mar 07 10:45:40 mjolnir systemd[1]: Starting LSB: start Samba SMB/CIFS daemon (smbd)...
      8. Mar 07 10:45:41 mjolnir smbd[5792]: Starting SMB/CIFS daemon: smbd.
      9. Mar 07 10:45:41 mjolnir systemd[1]: Started LSB: start Samba SMB/CIFS daemon (smbd).
      10. wolfstar@mjolnir:~$
      Display All

      And here's nmbd.service:

      Source Code

      1. wolfstar@mjolnir:~$ systemctl status nmbd.service
      2. ● nmbd.service - LSB: start Samba NetBIOS nameserver (nmbd)
      3. Loaded: loaded (/etc/init.d/nmbd; generated; vendor preset: enabled)
      4. Active: active (exited) since Tue 2017-03-07 10:30:24 EST; 1h 17min ago
      5. Docs: man:systemd-sysv-generator(8)
      6. CGroup: /system.slice/nmbd.service
      7. Mar 07 10:30:07 mjolnir systemd[1]: Starting LSB: start Samba NetBIOS nameserver (nmbd)...
      8. Mar 07 10:30:24 mjolnir nmbd[3174]: Starting NetBIOS name server: nmbd.
      9. Mar 07 10:30:24 mjolnir systemd[1]: Started LSB: start Samba NetBIOS nameserver (nmbd).
      10. wolfstar@mjolnir:~$
      Display All

      Note in both cases the "active (exited)" is in green text.
    • Here you go:

      Source Code

      1. wolfstar@mjolnir:~$ ls -al /var/log/
      2. total 3080
      3. drwxr-xr-x 11 root root 4096 Mar 6 07:35 .
      4. drwxr-xr-x 12 root root 4096 Mar 2 00:10 ..
      5. -rw-r--r-- 1 root root 9488 Mar 7 10:42 alternatives.log
      6. drwxr-xr-x 2 root root 4096 Mar 2 00:10 apt
      7. -rw-r----- 1 root adm 125259 Mar 7 18:53 auth.log
      8. -rw-r----- 1 root adm 112665 Mar 5 11:12 auth.log.1
      9. -rw-r--r-- 1 root root 0 Dec 29 06:40 bootstrap.log
      10. -rw------- 1 root utmp 2304 Mar 2 13:49 btmp
      11. drwxr-xr-x 2 root root 4096 Mar 5 00:23 cron-apt
      12. -rw-r----- 1 root adm 195977 Mar 7 17:30 daemon.log
      13. -rw-r----- 1 root adm 415422 Mar 5 11:12 daemon.log.1
      14. -rw-r----- 1 root adm 35350 Mar 7 10:30 debug
      15. -rw-r----- 1 root adm 89788 Mar 5 00:18 debug.1
      16. -rw-r----- 1 root adm 0 Dec 29 06:40 dmesg
      17. -rw-r--r-- 1 root root 346984 Mar 7 10:44 dpkg.log
      18. -rw-r--r-- 1 root root 32064 Mar 5 01:13 faillog
      19. -rw-r--r-- 1 root root 1220 Mar 7 10:44 fontconfig.log
      20. drwxr-xr-x 2 root root 4096 Mar 2 00:10 fsck
      21. -rw-r----- 1 root adm 146308 Mar 7 12:16 kern.log
      22. -rw-r----- 1 root adm 364469 Mar 5 09:49 kern.log.1
      23. -rw-rw-r-- 1 root utmp 292584 Mar 7 18:53 lastlog
      24. -rw-r----- 1 root adm 8193 Mar 7 18:17 mail.err
      25. -rw-r----- 1 root adm 1825 Mar 3 00:17 mail.err.1
      26. -rw-r----- 1 root adm 8193 Mar 7 18:17 mail.info
      27. -rw-r----- 1 root adm 2409 Mar 3 00:17 mail.info.1
      28. -rw-r----- 1 root adm 8193 Mar 7 18:17 mail.log
      29. -rw-r----- 1 root adm 2409 Mar 3 00:17 mail.log.1
      30. -rw-r----- 1 root adm 8193 Mar 7 18:17 mail.warn
      31. -rw-r----- 1 root adm 2409 Mar 3 00:17 mail.warn.1
      32. -rw-r----- 1 root adm 120200 Mar 7 12:16 messages
      33. -rw-r----- 1 root adm 289882 Mar 5 11:12 messages.1
      34. -rw-r----- 1 root adm 0 Mar 7 09:20 monit.log
      35. drwxr-xr-x 2 root adm 4096 Mar 5 00:23 nginx
      36. drwxr-xr-x 2 root root 4096 Mar 2 00:14 openmediavault
      37. -rw------- 1 root root 814 Mar 7 10:30 php5-fpm.log
      38. -rw------- 1 root root 2178 Mar 5 00:19 php5-fpm.log.1
      39. drwxr-xr-x 2 root root 4096 Mar 2 00:10 proftpd
      40. drwxr-x--- 2 root adm 4096 Dec 17 16:20 samba
      41. drwxr-xr-x 2 root root 4096 Mar 5 01:14 syncthing
      42. -rw-r----- 1 root adm 161784 Mar 7 18:45 syslog
      43. -rw-r----- 1 root adm 272890 Mar 7 09:18 syslog.1
      44. -rw-r----- 1 root adm 134836 Mar 5 11:12 syslog.2.gz
      45. -rw-r--r-- 1 root root 63936 Mar 7 10:40 tallylog
      46. -rw-r----- 1 root adm 10632 Mar 7 08:28 user.log
      47. -rw-r----- 1 root adm 66304 Mar 2 04:28 user.log.1
      48. drwxr-xr-x 2 root root 4096 Dec 4 2014 watchdog
      49. -rw-rw-r-- 1 root utmp 43392 Mar 7 18:53 wtmp
      50. wolfstar@mjolnir:~$
      Display All

      On the assumption you'd want it as well, there's nothing in /var/log/samba:


      Source Code

      1. wolfstar@mjolnir:/var/log/samba$ ls -al
      2. total 8
      3. drwxr-x--- 2 root adm 4096 Dec 17 16:20 .
      4. drwxr-xr-x 11 root root 4096 Mar 6 07:35 ..
      5. wolfstar@mjolnir:/var/log/samba$
      Confirmed, JUST in case, that it's also empty for root (sudo).
    • Longer posts sometimes trigger the spam filter. I approved it.

      Didn't find anything odd in your output. How about dpkg -l | grep samba
      omv 4.0.6 arrakis | 64 bit | 4.12 backports kernel | omvextrasorg 4.1.0
      omv-extras.org plugins source code and issue tracker - github.com/OpenMediaVault-Plugin-Developers

      Please don't PM for support... Too many PMs!
    • ryecoaaron wrote:

      Longer posts sometimes trigger the spam filter. I approved it.

      Didn't find anything odd in your output. How about dpkg -l | grep samba
      Here we go; heading home now (had a night maintenance) so won't be able to reply until much later, if at all tonight.


      Source Code

      1. wolfstar@mjolnir:~$ dpkg -l | grep samba
      2. ii python-samba 2:4.2.14+dfsg-0+deb8u2 amd64 Python bindings for Samba
      3. ii samba 2:4.2.14+dfsg-0+deb8u2 amd64 SMB/CIFS file, print, and login server for Unix
      4. ii samba-common 2:4.2.14+dfsg-0+deb8u2 all common files used by both the Samba server and client
      5. ii samba-common-bin 2:4.2.14+dfsg-0+deb8u2 amd64 Samba common files used by both the server and the client
      6. ii samba-dsdb-modules 2:4.2.14+dfsg-0+deb8u2 amd64 Samba Directory Services Database
      7. ii samba-libs:amd64 2:4.2.14+dfsg-0+deb8u2 amd64 Samba core libraries
      8. ii samba-vfs-modules 2:4.2.14+dfsg-0+deb8u2 amd64 Samba Virtual FileSystem plugins
      9. wolfstar@mjolnir:~$
      Thanks Ryecoaaron!
    • ryecoaaron wrote:

      Unfortunately, that didn't show me any problems. I guess I am back to maybe bad ram?
      Two 4GB ECC UDIMMs in the box. I guess it COULD be, but it was running in my main NAS without issues. Unfortunately, running memtest86 on it is non-trivial. On the other hand, Samba is a nice-to-have in this situation, so as long as I can get syncthing running. I'll track it down eventually.

      Thanks for the help!
    • So, poking around some more, not really trying to "solve" the issue, but just because I can't leave it alone, I tried to figure out why manually doing systemctl start syncthing@root.service worked where systemctl start syncthing.service didn't, I discovered that the former is Syncthing's native systemd setup, whereas the latter is a sysvinit wrapper that systemd uses.

      So, looking into it, tftpd-hpa, smbd, and probably xrdp are all using sysvinit scripts and systemd-sysv-install as a wrapper to let systemd control it.

      I have no idea WHY sysvinit scripts aren't properly running, but that's what looks like is going on. Any thoughts?