SCP/SSHFS recursive copy with exclusions

1 minute read

SCP does not have an option to exclude files while copying remote files over SSH. This is a problem when you have Git-managed code you’ve modified on say:

  • offline computer
  • HPC, and don’t want to put your Git host credentials on the HPC

If you just use scp -r, you’ll also overwrite the .git directory which can destroy work done on this or other branches. We want to just copy the code files, NOT the .git tree.

Solution

  1. on your laptop, install SSHFS, which uses SSH and FUSE to make a virtual directory on your laptop, where the true files reside on the remote computer.

    • Linux / Windows Subsystem for Linux: apt install sshfs
    • Mac: brew install sshfs
  2. mount to a location on your laptop, say ~/X
    mkdir ~/X    # one-time
       
    sshfs -o follow_symlinks myserver: ~/X
    

    Now the ~ directory on myserver is connected to ~/X on your laptop.

  3. use cp as usual to copy a directory while excluding .git
    shopt -s extglob   # enables fancy globbing
       
    cp -rv ~/X/myprog/*!(.git/|bin/) .
    

    Note I also excluded things under bin/ of the repository, as the HPC probably runs a different Linux distro and the compilation is optimized for a different CPU than my laptop, so the HPC binaries wouldn’t generally be useful on my laptop.

Leave a Comment