Moving data between shares on the same volume

    • OMV 4.x
    • Moving data between shares on the same volume

      Let's say I have 2 shared folders on volume1, one called downloads and one called video. When I try to move (cut and paste) a file between those folders, it will do a network copy.
      It does this both with AFP and SMB. It even does this on the system itself (when Sickrage tries to move a file from downloads to video while postprocessing).

      When I move a file between folders on the same shared folder, the move is instant.

      Is this by design? Should I just create one shared folder and create subfolders within that shared folder?
    • Moving data between shares on the same volume

      If you copy from another machine via a share, it will always do a network copy. This is because the data comes out to the machine you are using, then back to the share and is inherently inefficient. This happens because the machine that is copying has no idea where the shares are located and sees them as two entirely separate drives regardless of where they actually are. The reason that it’s instant within the same share is because (unless you’ve done some stuff with symlinks/shortcuts) the data is more or less guaranteed to be on the same partition within the drive just like a USB drive for example. Whilst it’s inefficient, you can do it without issue - it will just take longer. A better way would be to login to the server and copy via SSH. So the network copy thing is by design (or rather, unavoidable).

      However, it definitely shouldn’t happen with internal stuff too like Sickrage - providing that the files are located within the same partition. If the server is just moving a file from within a partition to another folder in that partition, it should be instant. So it’s probably one of two things:

      1. Sickrage’s data is on the OS drive and not the data drive (which of course will result in a copy when it completes and moves files): check the location of Sickrage’s data

      Or

      2. A permissions problem: I would check that your permissions are correct for each of the shared folders and that Sickrage user is able to access them.


      Sent from my iPhone using Tapatalk
    • Each share are a separate file system as far as the client is concerned. even if they are parts of the same file system on the server.

      I usually have one big share per disk, and subfolders. That way it is easy to move files and folders around

      And if data moves between filesystems then they usually pass through the client. So the data has to traverse the network TWICE. From the server to the client and from the client back to the server. (Or to a different server.)

      This is very wasteful so there are many tools available to avoid this. They let data flow directly from the source to the destination, without having to pass through the client. scp, rcp, rsync and a lot more. Several of these tools use ssh, as Grape mentioned.

      My favorite tool to handle this is mc. Midnight Commander. It is like a steam punk cross between a multi tool, a cement mixer and a communications satellite.

      When DOS was a thing, before Linux even existed, I used Norton Commander a lot. And mc looks and handles a lot like nc, superficially at least.

      Midnight Commander has a very elegant way to handle ssh called FISH. Also it can do ftp, sftp, syncs, shells and a lot more. And it has a simple easy to use editor built in.

      You can run mc from a client and connect to a server using FISH/ssh or you can ssh in to a server and then run mc. And when on the server you can ssh in to another server using mc. And you can use FISH without mc. And on an on.

      I have mc installed on all my Linux computers, laptop, desktop and servers. Usually it is what I install first, before anything else.

      sudo apt install mc

      FISH: en.wikipedia.org/wiki/Files_transferred_over_shell_protocol

      Midnight Commander: midnight-commander.org/
      OMV 4: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4
    • Mr.Grape wrote:

      Or log in to ssh and move what you need in /srv/dev-disk-by-label-justtim/
      That worked, thanks, using WinSCP, the move was instant.

      ellnic wrote:

      This happens because the machine that is copying has no idea where the shares are located and sees them as two entirely separate drives regardless of where they actually are. The reason that it’s instant within the same share is because (unless you’ve done some stuff with symlinks/shortcuts) the data is more or less guaranteed to be on the same partition within the drive just like a USB drive for example.
      That makes sense! Thanks for clearing that up.

      ellnic wrote:

      However, it definitely shouldn’t happen with internal stuff too like Sickrage - providing that the files are located within the same partition. If the server is just moving a file from within a partition to another folder in that partition, it should be instant. So it’s probably one of two things:

      1. Sickrage’s data is on the OS drive and not the data drive (which of course will result in a copy when it completes and moves files): check the location of Sickrage’s data

      Or

      2. A permissions problem: I would check that your permissions are correct for each of the shared folders and that Sickrage user is able to access them.
      Sickrage doesn't have any data folders configured on the OS drive. It also has sufficient access rights, because it is able to move the data eventually, it is just not instant (I can see the filesize growing slowly when it is processing).
      Could it be because Sickrage is running in Docker, and that it is therefore somehow unaware of the partitioning (as if it was an external system)?


      Adoby wrote:

      Each share are a separate file system as far as the client is concerned. even if they are parts of the same file system on the server.

      I usually have one big share per disk, and subfolders. That way it is easy to move files and folders around

      And if data moves between filesystems then they usually pass through the client. So the data has to traverse the network TWICE. From the server to the client and from the client back to the server. (Or to a different server.)

      This is very wasteful so there are many tools available to avoid this. They let data flow directly from the source to the destination, without having to pass through the client. scp, rcp, rsync and a lot more. Several of these tools use ssh, as Grape mentioned.

      My favorite tool to handle this is mc. Midnight Commander. It is like a steam punk cross between a multi tool, a cement mixer and a communications satellite.

      When DOS was a thing, before Linux even existed, I used Norton Commander a lot. And mc looks and handles a lot like nc, superficially at least.

      Midnight Commander has a very elegant way to handle ssh called FISH. Also it can do ftp, sftp, syncs, shells and a lot more. And it has a simple easy to use editor built in.

      You can run mc from a client and connect to a server using FISH/ssh or you can ssh in to a server and then run mc. And when on the server you can ssh in to another server using mc. And you can use FISH without mc. And on an on.

      I have mc installed on all my Linux computers, laptop, desktop and servers. Usually it is what I install first, before anything else.

      sudo apt install mc

      FISH: en.wikipedia.org/wiki/Files_transferred_over_shell_protocol

      Midnight Commander: midnight-commander.org/
      Thanks for your extensive response! Will look into Midnight Commander, looks promising!
    • ellnic wrote:

      If you copy from another machine via a share, it will always do a network copy
      Not necessarily. The Macs I administrated already over a decade ago did not showed this (rather stupid) behavior since the servers we used had the kSupportsCopyFile bit set and so the Mac instead of copying stuff through the network twice simply asked the server to do the job. This is protocol dependent and worked only with Apple's AFP.

      Microsoft started with something similar later when defining SMB2 and even Samba supports this starting with Samba 4.1: wiki.samba.org/index.php/Server-Side_Copy

      Of course both server and client (software) need to support such APIs.
    • tkaiser wrote:

      Not necessarily. The Macs I administrated already over a decade ago did not showed this (rather stupid) behavior since the servers we used had the kSupportsCopyFile bit set and so the Mac instead of copying stuff through the network twice simply asked the server to do the job. This is protocol dependent and worked only with Apple's AFP.

      Microsoft started with something similar later when defining SMB2 and even Samba supports this starting with Samba 4.1: wiki.samba.org/index.php/Server-Side_Copy

      Of course both server and client (software) need to support such APIs.
      Thanks. The article you linked to does list a limitation, namely: "Both source and destination files must reside on the same Samba share!".
      This is probably why it is still doing a network copy. It does work within the same shared folder.
    • justtim wrote:

      I just realised that Sickrage looks at /sharedfolders/data and /sharedfolders/video/.

      I should probably change that to /srv/dev-disk-by-label-justtim/data and /srv/dev-disk-by-label-justtim/video, right? Could this exlpain the behaviour?
      Did try and change this, didn't matter.
      Posted an issue here github.com/linuxserver/docker-sickrage/issues/34 as I think this is probably an issue with this Sickrage Docker.
    • @justtim if you mount the folder /srv/dev-disk-by-label-justtim/ into your docker and then point sickrage to folders within that mount point it should be able to move files instantly (although I've not tried this yet!)



      I am not sure why moving files in the /srv/dev-disk-by-label-***/ is instant, while moving files between /sharedfolders/a to /sharedfolders/b is done as a copy. Can anyone explain?

      The post was edited 1 time, last by finnsloss ().

    • It is because before moving the files there is a check to see what filesystem each file lives on. If both filesystems are the same, then the file is simply renamed. And that is instantaneous.

      If the filesystems reported by the operating system are different then the file is copied to the new location and the old file is deleted. This can be very slow if it has to be done via a client that is connected over the network.

      It is easy to confuse the operating system and make it fail to see that the filesystem is the same. For instance if you are connecting via different shares or via different mount points. You must make the paths easy and obvious for the move to be instantaneous.

      /sharedfolders/a and /sharedfolders/b are on the same filesystem if you access them locally. But they are two different shares and two completely different filesystems if accessed remotely.

      To avoid this hassle I only have one shared folder per NAS. Then every file I move within that share is moved instantaneous.
      OMV 4: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4
    • @Adoby I have tried moving files via ssh command line and using mc between:

      /sharedfolders/public/
      /sharedfolders/video/

      which results in a copy & delete

      and moving between:

      /srv/dev-disk-by-id-ata-data-drive/public/
      /srv/dev-disk-by-id-ata-data-drive/video/

      which results in a move

      as far as I can tell the public folders are hard links of each other (same for the video folders) the i-nodes are the same as below:

      finn@odroidnas1:/$ ls sharedfolders/ -i
      217579521 config 15794177 home 89522177 music 176881665 photo 140705793 public 189595649 video
      finn@odroidnas1:/$ ls srv/dev-disk-by-id-data-drive/ -i
      217579521 config 15794177 home 11 lost+found 89522177 music 176881665 photo 140705793 public 189595649 video

      so i can't understand the difference in behaviour.

      I do understand that a copy will happen if the file is moved remotely using shares.

      Thanks, Finn.
    • Well, then a and b could be separate filesystems, both separately mounted at /sharedfolders. Not hardlinks.

      Since I have never had more than one shared folder, I never noticed that. KISS.

      When I move files around in my single shared folder it is always instantaneous. If you are sure you don't want or need instantaneous moves, then keep copying between two different filesystems.

      Try doing a search for something like: "how do I check if two files are on the same filesystem" for more information on how to figure out if two files are on the same filesystem or not. Or simply try to move a file and see if it is slow or fast.

      Edit: I do have more than one shared folder, when I think about it. But only one that is actually shared. The other one is for docker stuff. I don't ever move files between the two.
      OMV 4: 9 x Odroid HC2 + 1 x Odroid HC1 + 1 x Raspberry Pi 4

      The post was edited 1 time, last by Adoby ().