How to spotlight-index an AFP share - best methods?

    • OMV 3.x
    • How to spotlight-index an AFP share - best methods?

      I inadvertently hijacked another thread as my issue was related to a side point that was mentioned - indexing an AFP share. See (post 37 to 64; AFP transfer sometimes very(!) slow) for several methods that have been tried to spotlight-index an AFP share hosted on OMV3.

      So far, the only method which works for me is to store my data in a sparse bundle which is formatted in HFS+, which I then mount within OSX. And then enabling indexing on the mounted sparsebundle with: mdutil /Volumes/<share> -i on

      Unfortunately, the mdutil command doesn't work natively on an AFP share (for me). It would be ideal to index the AFP share natively.
      NAS OMV 3.0.59 Rig: DFI Lanparty nForce4 SLI-DR | Opteron 165 Dual Core (no overclock) | 4GB Corsair Non-ECC DDR400 RAM | OMV3: 1x100GB IDE | Data: 2x 250GB WD 7200rpm, 2x 3TB WD Red, 1x 4TB Samsung 5400rpm
    • Just did the steps I gave you, and I am able to index my shares just fine. The indexing started right away and a progress bar now shows up in Spotlight. Did you try to use sudo before enabling it? The first line was enough to get mine rolling. Pasting the command here for future reference:


      Source Code

      1. mdutil -i on /Volumes/MyOMVshare/
      It just seems bizarre that is only indexing sparse volumes and not the share directly.
      Absolutely no support through PMs!!
      I work for the HPE servers R&D division. Have tech questions about ProLiant servers? Ask me!
      Learn more about HPE's iLO4 Management Engine included in all ProLiant servers.
      HPE Repositories for Linux are available here!

      [IMG:http://home.thepoorgeek.com/misc/Hewlett-Packard-Enterprise-logo.jpg]
    • That is so strange. Do you think you'd be able to create a Test share? Are you running OMV3 and MacOs Sierra or configured the AFP plugin in any particular way e.g. extra options? I'm just trying to understand how I may essentially 'copy' what you have done in order to get it to work.

      Do you have any additional special ACL's setup? Thanks
      NAS OMV 3.0.59 Rig: DFI Lanparty nForce4 SLI-DR | Opteron 165 Dual Core (no overclock) | 4GB Corsair Non-ECC DDR400 RAM | OMV3: 1x100GB IDE | Data: 2x 250GB WD 7200rpm, 2x 3TB WD Red, 1x 4TB Samsung 5400rpm
    • I've been trying to figure this out (getting nowhere); however, I think that the root cause lies within permissions. As you pointed out, I can index the sparse bundle but not the share. Within OSX, the permissions for the share and sparsebundle respectively are slightly different. Any thoughts?

      Share permissions (as seen within OSX):

      Name Privilege

      (unknown) read and write
      everyone no access

      Sparsebundle permissions (as seen within OSX):

      Name Privilege

      (unknown) read and write
      staff read and write
      everyone read only

      Does this shed any light on the matter?

      Update: Don't know if this is relevant, but when I try to change the permissions of the shared folder within OSX (terminal as root), I receive a permission is denied message. I think this is to be expected, but in case it isn't, I thought I'd mention it.
      NAS OMV 3.0.59 Rig: DFI Lanparty nForce4 SLI-DR | Opteron 165 Dual Core (no overclock) | 4GB Corsair Non-ECC DDR400 RAM | OMV3: 1x100GB IDE | Data: 2x 250GB WD 7200rpm, 2x 3TB WD Red, 1x 4TB Samsung 5400rpm

      The post was edited 1 time, last by RS1 ().

    • There's nothing other than default for all my shares. Make a new share as public and allow guest to write/read, make sure you can write to it and if you can, try indexing that.
      Absolutely no support through PMs!!
      I work for the HPE servers R&D division. Have tech questions about ProLiant servers? Ask me!
      Learn more about HPE's iLO4 Management Engine included in all ProLiant servers.
      HPE Repositories for Linux are available here!

      [IMG:http://home.thepoorgeek.com/misc/Hewlett-Packard-Enterprise-logo.jpg]
    • Just gave it a try via AFP and SMB for good measure. Same result - 'indexing disabled'. I think that I'm going to have to accept defeat on this issue.

      Thank you for taking the time to investigate!
      NAS OMV 3.0.59 Rig: DFI Lanparty nForce4 SLI-DR | Opteron 165 Dual Core (no overclock) | 4GB Corsair Non-ECC DDR400 RAM | OMV3: 1x100GB IDE | Data: 2x 250GB WD 7200rpm, 2x 3TB WD Red, 1x 4TB Samsung 5400rpm
    • Quick update - I had decided to use a sparsebundle to get me over the line re. indexing - however, I experienced some mounting issues out of the blue. Ran first aid in OS X - and it reported that the sparsebundle is corrupt. This has diminished my confidence to say the least. Since then, I have decided to be careful with sparsebundles and make frequent backups - all because of indexing! The ideal solution would be to move away from spotlight altogether.

      I reached out to the original author that @Spy Alelo was kind enough to have posted:

      Spy Alelo wrote:

      OS X does not index network shares by default, but you can make it happen with the little guide I posted below.
      discussions.apple.com/thread/130462?tstart=0

      Gerrit returned with a very thoughtful and detailed response which essentially entails exploring 'Gnome Tracker' with an SMB implementation:

      subzero79 wrote:

      Sorry to disappoint you but spotlight does not refer to spotlight OS X (took me a while to figure this out). Is a feature intended for gnome tracker (search engine for Linux desktop).

      @subzero79, did you play around with setting up Gnome Tracker at all?
      Is it beyond the scope of any typical OMV installation to get it working?
      From your experience, is it even worth the effort to implement it?

      I am debating in my mind whether implementing Gnome Tracker to work with one SMB share that stores my documents is too large a task or whether it is doable with the right guidance. Perhaps its implementation could be seen as an experiment of sorts?

      I have included Gerrit's response to my email below. Would be great to receive your thoughts (albeit general in nature) @subzero79, @Spy Alelo.

      Thanks!

      Gerrit DeWitt wrote:

      Hi Ricky,

      Yes, getting Spotlight or search in general working for AFP or SMB volumes can be a little difficult. First, a little background:

      - For proper server-side indexing, in addition to providing AFP or SMB services, your server must have some kind of indexing service, and the AFP or SMB server must be able to communicate with said indexing service. Server.app on macOS leverages the local Spotlight daemons and databases. Apple's AFP and SMB server processes communicate with local Spotlight stores on the server itself to answer queries from clients.

      - My 11-year old post was pretty rough around the edges. The reason it worked is because it tricked the local Spotlight indexing service on the server (then running as root under OS X 10.4) to pick up and continue indexing after a client kickstarted the process. To be honest with you, my suggestion would not have scaled very well. Mac OS X 10.4 had really half-baked searching capabilities. Apple didn't attempt to properly fix this until 10.5. I remember an Apple engineer telling me that there would be some level of interaction between AFP and Spotlight in 10.4, so imagine my disappointment when there was none by default! Even basic file name searches failed unless Spotlight was enabled using the hacky procedure I described! As with so many things Apple, there was really no effort made for enterprise. File name searches worked in Server 10.3, but not by default in Server 10.4. Apple didn't properly add a means to enable server side indexing of shared folders until 10.5.

      - These days with macOS Server, enabling Spotlight search for a shared folder means letting the Spotlight system user read and traverse the shared folder's hierarchy. This is usually done via an ACL entry made by Server.app. There might also be some backend communication with Spotlight to prepare a special index for the share point; this is not documented as far as I know. Still, the Spotlight indices would reside on the server in the various Spotlight folders at the top level of volumes or those in /var/db/ (for "removable" volumes, unless this is also used to hold special share point indices). With some experimentation, it would be possible to find out if macOS is making a special "share point index" and where it's putting it.

      - Client macOS systems are supposed to automatically index removable and mounted server volumes, creating their own per-client index databases in /var/db if the server doesn't support searching. How this determination is accomplished is unknown. Presumably the protocol version of AFP or SMB provides the client with a hint as to whether or not it should treat the mounted share point for searching. As you and others have found, this is hit-or-miss. Some tricks include enabling Spotlight on the volume each time it is mounted. Others do things like add the mounted volume to Spotlight's privacy "exclude path" list, then remove it from that list. Presumably this tells the local Spotlight processes that maintaining an index for /Volumes/whatever is OK, and that it should be handled like a removable disk. These solutions provide no server side indexing, and, even when they do work, each client has to create the index for the mounted share point. The takeaway I have from this is that macOS clients should not be considered reliable when making per client indices of mounted volumes. I will say that they seem to be more reliable than in the past, though.

      Now onto the useful bits...

      - Fortunately, some smart people have figured out how to add searching capabilities to AFP and SMB server implementations that don't run on macOS Server! These implementations are purely server side, and they create databases and search processes that either Samba or Netatalk can query. These implementations don't create Spotlight compatible-databases, but they do allow searching from clients through the AFP or SMB to non macOS servers.

      - My first recommendation is this: Migrate away from AFP. Move to SMB, protocol version 3. SMB has been the first class file sharing protocol version since OS X 10.9 Mavericks. OS X 10.10 Yosemite added support for SMB protocol version 3. Final Cut Pro 10.3.1 can store video libraries on SMB 3 volumes. Server.app even creates home directory auto mounts using SMB instead of AFP these days. I say this because I believe there will come a day very soon when macOS no longer will handle AFP volumes. There are several oddities about the AFP protocol compared to SMB or even NFSv4 that make it stick out like a sore thumb. It encodes file names with decomposed Unicode. Clients reference files by special IDs instead of by path. These are peculiarities that will be hard to justify maintaining, especially since macOS is one of four client operating systems Apple considers itself making these days. :)

      - Not withstanding the forgoing, it is possible to have searchable AFP volumes hosted by Netatalk on Linux. There is an absolutely excellent write-up of AFP in depth here: netatalk.sourceforge.net/3.1/htmldocs/configuration.html

      The Netatalk team clearly understands inner workings of AFP, probably far better than most people at Apple do. There is in-depth information about ACL mappings on the server side (something that event Apple didn't have to worry about).

      Netatalk supports Spotlight indexing using the Gnome Tracker as its search database provider. There's information about how to configure AFP on the Netatalk page, but you'll need Gnome Tracker to be installed (wiki.gnome.org/Projects/Tracker), and Netatalk will need to have been compiled with the path to the dbus-daemon. (Think of dbus as a messaging and interprocess communications system for Linux. It has some similarities to Mach ports, XPC, and NSDistributedNotificationCenter.)

      I don't know if the Netatalk included with OMV includes dbus support. If it does, then you should be able to get searching working if you: (a) install and configure Gnome Tracker, and (b) configure Netatalk share points to be searchable as described in its documentation: this configuration will probably take you out of the OMV UI and involve editing Netatalk config files manually. This shouldn't be to hard, but I haven't done it, and I think it might be a legitimate question for the maintainers of OMV.

      - Back to what I was saying about SMB: you can have searchable SMB volumes via Samba, too! Just as for Netatalk, the Gnome Tracker is the search provider, and Samba must be explicitly compiled with dbus and "Spotlight" support: wiki.samba.org/index.php/Spotlight

      Depending on your needs, it might make sense to spend time getting searchable SMB working and migrate to SMB protocol version 3 as you'll probably have to do this at some point anyway.

      Again, I haven't done this myself, but it seems that you're in capable hands with community support from Samba.

      - One point that I should add is that the Gnome Tracker may not index all of the same attributes that an Apple Spotlight server would. But it's certainly better than having no search at all. It would be worth doing some experimentation to find out what Spotlight attributes work and what ones don't.

      I hope this helps you, and I would enjoy hearing about any successes you may have!

      --Gerrit
      NAS OMV 3.0.59 Rig: DFI Lanparty nForce4 SLI-DR | Opteron 165 Dual Core (no overclock) | 4GB Corsair Non-ECC DDR400 RAM | OMV3: 1x100GB IDE | Data: 2x 250GB WD 7200rpm, 2x 3TB WD Red, 1x 4TB Samsung 5400rpm
    • netatalk for omv has not being compiled with spotlight support and you can check yourself by running netatalk -v

      You're using OSX and you want to use now a linux desktop just to search on spotlight network shares? seems a little bit too much doesn't it?

      You can still search from the finder search box is not instant but should work
      New wiki
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server
    • I agree, I do think it is a little much. I was thinking more along the lines of Gnome Tracker/SMB share support. In any event, I managed to get the functionality I needed from sparsebundle/SMB/Spotlight. I found the topic interesting though :)
      NAS OMV 3.0.59 Rig: DFI Lanparty nForce4 SLI-DR | Opteron 165 Dual Core (no overclock) | 4GB Corsair Non-ECC DDR400 RAM | OMV3: 1x100GB IDE | Data: 2x 250GB WD 7200rpm, 2x 3TB WD Red, 1x 4TB Samsung 5400rpm
    • Quicksilver is certainly better than Easyfind which I found to be extremely slow. Thank you for the suggestion. I've also been experimenting with Devonthink - good article here for anyone else concerned with indexing: macdrifter.com/2015/01/searching-without-spotlight.html



      subzero79 wrote:

      netatalk for omv has not being compiled with spotlight support and you can check yourself by running netatalk -v

      I ran this command and you're quite right, netatalk hasn't been compiled accordingly. Is there a reason that OMV has decided to discard Gnome Tracker? Or more to the point, discard Samba and Netatalk spotlight indexing support? Is it considered a significant effort?
      NAS OMV 3.0.59 Rig: DFI Lanparty nForce4 SLI-DR | Opteron 165 Dual Core (no overclock) | 4GB Corsair Non-ECC DDR400 RAM | OMV3: 1x100GB IDE | Data: 2x 250GB WD 7200rpm, 2x 3TB WD Red, 1x 4TB Samsung 5400rpm
    • This binary is provided by omv packages as Debian doesn't have netatalk anymore. The default option was no. I believe the developer grabbed a debianized src package a build it. The default options are sufficient imo,he probably doesn't even know the option exists.
      New wiki
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server
    • Thanks for your thoughts on this. Perhaps Samba spotlight is the better way to go as Apple prefer SMB to AFP. I saw an interesting wiki here : wiki.samba.org/index.php/Spotlight

      This doesn't appear to be a priority for many people perhaps, but I think it'd be neat to have server side spotlight indexing. I've submitted a plugin/feature request to just put it out there.. :D
      NAS OMV 3.0.59 Rig: DFI Lanparty nForce4 SLI-DR | Opteron 165 Dual Core (no overclock) | 4GB Corsair Non-ECC DDR400 RAM | OMV3: 1x100GB IDE | Data: 2x 250GB WD 7200rpm, 2x 3TB WD Red, 1x 4TB Samsung 5400rpm
    • RS1 wrote:

      I've submitted a plugin/feature request to just put it out there..
      IMO, that won't happen as samba is provided as stable and maintained binary by Debian. Netatalk is a case apart as the package did not made into debian jessie. It took a lot of time to convince to use an out of debian package (the plugin was going probably to disappear).
      Also the name "spotlight" is misleading, as is clearly has nothing to do with OSX search feature, as it doesn't aid at all in the search function, as this is done at the client side in OSX.
      Again "spotlight" for netatalk and samba is to be used only in Gnome linux desktop
      New wiki
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server
    • subzero79 wrote:

      Again "spotlight" for netatalk and samba is to be used only in Gnome linux desktop

      Okay, understood. The link wiki.samba.org/index.php/Spotlight is extremely misleading as it states "compiling and configuring Samba with support for OS X Spotlight support. " Hence the confusion..
      NAS OMV 3.0.59 Rig: DFI Lanparty nForce4 SLI-DR | Opteron 165 Dual Core (no overclock) | 4GB Corsair Non-ECC DDR400 RAM | OMV3: 1x100GB IDE | Data: 2x 250GB WD 7200rpm, 2x 3TB WD Red, 1x 4TB Samsung 5400rpm
    • I am little bit confused about this now. Look at this
      davidschlachter.com/misc/netatalk3rpi

      This
      serverfault.com/questions/5611…-spotlight-indexing-hangs

      Notice in the first page how the guy points out that spotlight works only from the finder, he doesn't mention the global one cmd + space
      New wiki
      chat support at #openmediavault@freenode IRC | Spanish & English | GMT+10
      telegram.me/openmediavault broadcast channel
      openmediavault discord server
    • Thanks for the links! Quite interesting. I sent David Schlachter an email re. the global shortcut. But in theory, it should work. The second link is vague, but shows that server side indexing via afp seems to be okay?

      Any thoughts on the samba implementation? Does OMV3 feature the latest samba build (4.3+)?

      At present, since I couldn't get spotlight to work with OMV, I did away with AFP for everything other than Time Machine, solely relying on SMB.
      NAS OMV 3.0.59 Rig: DFI Lanparty nForce4 SLI-DR | Opteron 165 Dual Core (no overclock) | 4GB Corsair Non-ECC DDR400 RAM | OMV3: 1x100GB IDE | Data: 2x 250GB WD 7200rpm, 2x 3TB WD Red, 1x 4TB Samsung 5400rpm
    • subzero79 wrote:

      Notice in the first page how the guy points out that spotlight works only from the finder, he doesn't mention the global one cmd + space

      The guy got back to me:


      "What I'd observed is that Spotlight only gives results from the shared drive if you search with the search box in the top-right of a Finder window while you're browsing the shared drive. I haven't used the Spotlight support in netatalk for a while though (haven't figured it out on Ubuntu 16 on a new server)."

      I think that it would still be worth doing (even if it isn't possible to do a cmd+space search).

      I'd be interested in your thoughts on the samba + OSX spotlight implementation though as this may be more future proof / in the interest of others and OMV3 going forward..
      NAS OMV 3.0.59 Rig: DFI Lanparty nForce4 SLI-DR | Opteron 165 Dual Core (no overclock) | 4GB Corsair Non-ECC DDR400 RAM | OMV3: 1x100GB IDE | Data: 2x 250GB WD 7200rpm, 2x 3TB WD Red, 1x 4TB Samsung 5400rpm