Moving data between shares on the same volume

    • OMV 4.x
    • Moving data between shares on the same volume

      Let's say I have 2 shared folders on volume1, one called downloads and one called video. When I try to move (cut and paste) a file between those folders, it will do a network copy.
      It does this both with AFP and SMB. It even does this on the system itself (when Sickrage tries to move a file from downloads to video while postprocessing).

      When I move a file between folders on the same shared folder, the move is instant.

      Is this by design? Should I just create one shared folder and create subfolders within that shared folder?
    • Moving data between shares on the same volume

      If you copy from another machine via a share, it will always do a network copy. This is because the data comes out to the machine you are using, then back to the share and is inherently inefficient. This happens because the machine that is copying has no idea where the shares are located and sees them as two entirely separate drives regardless of where they actually are. The reason that it’s instant within the same share is because (unless you’ve done some stuff with symlinks/shortcuts) the data is more or less guaranteed to be on the same partition within the drive just like a USB drive for example. Whilst it’s inefficient, you can do it without issue - it will just take longer. A better way would be to login to the server and copy via SSH. So the network copy thing is by design (or rather, unavoidable).

      However, it definitely shouldn’t happen with internal stuff too like Sickrage - providing that the files are located within the same partition. If the server is just moving a file from within a partition to another folder in that partition, it should be instant. So it’s probably one of two things:

      1. Sickrage’s data is on the OS drive and not the data drive (which of course will result in a copy when it completes and moves files): check the location of Sickrage’s data

      Or

      2. A permissions problem: I would check that your permissions are correct for each of the shared folders and that Sickrage user is able to access them.


      Sent from my iPhone using Tapatalk
    • Each share are a separate file system as far as the client is concerned. even if they are parts of the same file system on the server.

      I usually have one big share per disk, and subfolders. That way it is easy to move files and folders around

      And if data moves between filesystems then they usually pass through the client. So the data has to traverse the network TWICE. From the server to the client and from the client back to the server. (Or to a different server.)

      This is very wasteful so there are many tools available to avoid this. They let data flow directly from the source to the destination, without having to pass through the client. scp, rcp, rsync and a lot more. Several of these tools use ssh, as Grape mentioned.

      My favorite tool to handle this is mc. Midnight Commander. It is like a steam punk cross between a multi tool, a cement mixer and a communications satellite.

      When DOS was a thing, before Linux even existed, I used Norton Commander a lot. And mc looks and handles a lot like nc, superficially at least.

      Midnight Commander has a very elegant way to handle ssh called FISH. Also it can do ftp, sftp, syncs, shells and a lot more. And it has a simple easy to use editor built in.

      You can run mc from a client and connect to a server using FISH/ssh or you can ssh in to a server and then run mc. And when on the server you can ssh in to another server using mc. And you can use FISH without mc. And on an on.

      I have mc installed on all my Linux computers, laptop, desktop and servers. Usually it is what I install first, before anything else.

      sudo apt install mc

      FISH: en.wikipedia.org/wiki/Files_transferred_over_shell_protocol

      Midnight Commander: midnight-commander.org/
      OMV 4, 5 x ODROID HC2, 2 x 12TB, 2 x 8 TB, 1 x 500 GB SSD, GbE, WiFi mesh
    • Mr.Grape wrote:

      Or log in to ssh and move what you need in /srv/dev-disk-by-label-justtim/
      That worked, thanks, using WinSCP, the move was instant.

      ellnic wrote:

      This happens because the machine that is copying has no idea where the shares are located and sees them as two entirely separate drives regardless of where they actually are. The reason that it’s instant within the same share is because (unless you’ve done some stuff with symlinks/shortcuts) the data is more or less guaranteed to be on the same partition within the drive just like a USB drive for example.
      That makes sense! Thanks for clearing that up.

      ellnic wrote:

      However, it definitely shouldn’t happen with internal stuff too like Sickrage - providing that the files are located within the same partition. If the server is just moving a file from within a partition to another folder in that partition, it should be instant. So it’s probably one of two things:

      1. Sickrage’s data is on the OS drive and not the data drive (which of course will result in a copy when it completes and moves files): check the location of Sickrage’s data

      Or

      2. A permissions problem: I would check that your permissions are correct for each of the shared folders and that Sickrage user is able to access them.
      Sickrage doesn't have any data folders configured on the OS drive. It also has sufficient access rights, because it is able to move the data eventually, it is just not instant (I can see the filesize growing slowly when it is processing).
      Could it be because Sickrage is running in Docker, and that it is therefore somehow unaware of the partitioning (as if it was an external system)?


      Adoby wrote:

      Each share are a separate file system as far as the client is concerned. even if they are parts of the same file system on the server.

      I usually have one big share per disk, and subfolders. That way it is easy to move files and folders around

      And if data moves between filesystems then they usually pass through the client. So the data has to traverse the network TWICE. From the server to the client and from the client back to the server. (Or to a different server.)

      This is very wasteful so there are many tools available to avoid this. They let data flow directly from the source to the destination, without having to pass through the client. scp, rcp, rsync and a lot more. Several of these tools use ssh, as Grape mentioned.

      My favorite tool to handle this is mc. Midnight Commander. It is like a steam punk cross between a multi tool, a cement mixer and a communications satellite.

      When DOS was a thing, before Linux even existed, I used Norton Commander a lot. And mc looks and handles a lot like nc, superficially at least.

      Midnight Commander has a very elegant way to handle ssh called FISH. Also it can do ftp, sftp, syncs, shells and a lot more. And it has a simple easy to use editor built in.

      You can run mc from a client and connect to a server using FISH/ssh or you can ssh in to a server and then run mc. And when on the server you can ssh in to another server using mc. And you can use FISH without mc. And on an on.

      I have mc installed on all my Linux computers, laptop, desktop and servers. Usually it is what I install first, before anything else.

      sudo apt install mc

      FISH: en.wikipedia.org/wiki/Files_transferred_over_shell_protocol

      Midnight Commander: midnight-commander.org/
      Thanks for your extensive response! Will look into Midnight Commander, looks promising!
    • ellnic wrote:

      If you copy from another machine via a share, it will always do a network copy
      Not necessarily. The Macs I administrated already over a decade ago did not showed this (rather stupid) behavior since the servers we used had the kSupportsCopyFile bit set and so the Mac instead of copying stuff through the network twice simply asked the server to do the job. This is protocol dependent and worked only with Apple's AFP.

      Microsoft started with something similar later when defining SMB2 and even Samba supports this starting with Samba 4.1: wiki.samba.org/index.php/Server-Side_Copy

      Of course both server and client (software) need to support such APIs.
    • tkaiser wrote:

      Not necessarily. The Macs I administrated already over a decade ago did not showed this (rather stupid) behavior since the servers we used had the kSupportsCopyFile bit set and so the Mac instead of copying stuff through the network twice simply asked the server to do the job. This is protocol dependent and worked only with Apple's AFP.

      Microsoft started with something similar later when defining SMB2 and even Samba supports this starting with Samba 4.1: wiki.samba.org/index.php/Server-Side_Copy

      Of course both server and client (software) need to support such APIs.
      Thanks. The article you linked to does list a limitation, namely: "Both source and destination files must reside on the same Samba share!".
      This is probably why it is still doing a network copy. It does work within the same shared folder.
    • justtim wrote:

      I just realised that Sickrage looks at /sharedfolders/data and /sharedfolders/video/.

      I should probably change that to /srv/dev-disk-by-label-justtim/data and /srv/dev-disk-by-label-justtim/video, right? Could this exlpain the behaviour?
      Did try and change this, didn't matter.
      Posted an issue here github.com/linuxserver/docker-sickrage/issues/34 as I think this is probably an issue with this Sickrage Docker.