# "One shot" sftp (a step further) Previously, I wrote about using sftp to simulate the behaviour of scp, on systems where scp could not be used. => 2022-01-14_One_shot_sftp.gmi "One shot" sftp (simulating scp) Since then Szczeżuja wrote a little response on his tiny log about his own troubles getting scp working when uploading files to SDF and how he used tar pipes over ssh to transfer content. This is a good idea, especially if you want to transfer whole directory trees. It also allows you use compression, to speed up transfers and/or reduce bandwidth. => gemini://szczezuja.space/tinylog.gmi Szczeżuja's tinylog § Fri 14 Jan 2022 04:34:56 PM CET However, I should perhaps better explain why I wrote my initial post. Often systems are setup to limit to just sftp. In such cases there is no shell (and hence tar pipes cannot work) and no scp, which on many ssh severs (like OpenSSH) is tied to shell access and cannot be enabled on its own—at least not without installing and configuring further software like scponly (to better contain the shell environment). => https://github.com/scponly/scponly scponly The reason why scp is limited and usually tied to shell access seems to relate to security. Consider the following from the OpenSSH 8.0 release announcement (from 2019-04-17). => https://www.openssh.com/txt/release-8.0 OpenSSH 8.0 release announcement § Security > This release contains mitigation for a weakness in the scp(1) tool and protocol (CVE-2019-6111): when copying files from a remote system to a local directory, scp(1) did not verify that the filenames that the server sent matched those requested by the client. This could allow a hostile server to create or clobber unexpected local files with attacker-controlled content. > > This release adds client-side checking that the filenames sent from the server match the command-line request, > > The scp protocol is outdated, inflexible and not readily fixed. We recommend the use of more modern protocols like sftp and rsync for file transfer instead. Regardless of the reasons why scp is often unavailable, the fact remains that the workflow is more convenient and faster than the default behaviour of the standard sftp client. A scp transfer is a one liner, while sftp requires the connection step, one or more 'put' commands and a disconnect step ('exit', 'quit', 'bye' or '^D'). It is just a lot of typing for a quick file transfer. To be fair you do not have to use the classic sftp client. There are plenty of others. Perhaps the nicest for me is mounting the sftp connection using sshfs and indeed that is what I do on Linux. However I also tend to flip around between systems and 'sftp' is always present, while other options may not be. Now a little shell script wrapper is technically 'extra software' but it is tiny and does not need to be properly 'installed'. Ok, I have written a lot already, so for anyone getting bored, here is the script. [✍ 12:10 +0100: updated to use printf instead of echo] [✍ 16:30 +0100: removed extra quoting again as it prevented wildcards] [✍ 18:44 +0100: added a fix for spaces in filenames] [✍ 2022-01-17 08:43 +0100: another fix for wildcards] [✍ 2022-01-17 11:06 +0100: stops on errors or unset variables; compacted slightly] [✍ 2022-01-19 18:06 +0100: removed the need for 'sed' and fixed shebang] ``` #!/usr/bin/env -S bash -eu if [ "${1:-}" = '-P' ]; then p="${@:1:2}" shift 2 fi eval l=\${$#} if [[ "$l" == *:* ]]; then for f in "${@:1:$(($#-1))}"; do printf '%s\n' "put ${f// /\\ }" done | sftp ${p:-} "$l" else f="${1#*:}" printf '%s\n' "get ${f// /\\ } \"$2\"" | sftp ${p:-} "${1%%:*}" fi ``` Before you run away, if you are considering using this, read § Limitations. ## The new additions For anyone sticking around, my final example from the last post had the following issues. * It actually failed if you tried to upload more than 10 files at once. * It created a new connection for every uploaded file, which is especially problematic with 'interactive authentication' (password-based). * It only handled uploads. * It did not handle non-standard ports. * [✍ 18:44 +0100] There were quoting problems with spaces in filenames. The first one is easy, I just forgot a couple of curly brackets (braces '{}'). The second was also obvious once I looked at it again. I just moved the sftp command out of the loop, so that it is a single process that receives all the puts together. To handle uploads and have things work like scp, we have to do a couple of things. Firstly scp expects the file(s) to be listed first on upload and the server first on download. Secondly, the server name always has a colon (':') at the end, followed by the destination. The colon actually helps because we can look for it in the final argument and if present we know it is an upload, otherwise we assume it is a download. For downloads we simplify things as do not need to run through a loop but just pick the correct part (server or file), either side of the colon. Personally, I need to connect to servers that use non-standard ports (e.g. 🐟flounder). For this I just collect the first couple of arguments, if the initial one is '-P'. More options could be supported using 'getopts' but I don't (generally) need them and it would more than double the length of the script. I'll add more if I ever need them or perhaps just use sftp directly. [✍ 18:28 +0100] I am now filtering filenames with 'printf %q' to escape spaces before they are piped to sftp—I cannot hardcode quotes around all filenames or wildcards will fail. [✍ 2022-01-17 08:43 +0100] 'printf %q' was problematic as it prevented wildcards, like '*'. Instead, I now filter filenames with sed, purely to escape 'spaces' before they are piped to sftp but not other characters that 'printf %q ' might be able to handle. The downside for this is if you had a file that actually contained an 'odd' character like '*' it would need to be double escaped. And that is it… for now 😉. I hope this is useful to someone. If not… 🤷🏼 ### Limitations * You can't download multiple files like so, 'server:\{file1,file2\} .'—I am not sure how many people knew about that scp feature anyway. You can still grab more than one file at a time using wild cards however, e.g. 'server:file\?.txt .' * You can't download to a directory with colon in the name, or 'on-the-fly' rename a download to filename that contains a colon. * You can't rename a file on upload like so, 'file1 server:file2'—you can still copy to an already present remote directory, e.g. 'file1 server:dir'. * Only the -P (port) option is supported and (if you want to use it) it must be the first argument with a space to separate it from the port number. * [✍ 16:30 +0100] You can't do server to server copies—another feature that might not even be widely known or used. * [✍ 2022-01-17 08:43 +0100] If certain unusual characters actually appear in filenames—and are not intended as wildcards—they would need to be double escaped (e.g. a file named 'file*1', would need to be uploaded as 'file1 server:"file\*1"'). ⁂ [✍ 16:30 +0100] Here is another version using only POSIX shell (tested against dash). [✍ 18:44 +0100: fixed an issue with quoting problems when a filename contains spaces] [✍ 2022-01-17 11:06 +0100: stops on errors or unset variables; compacted slightly] ``` #!/bin/sh -eu if [ "${1:-}" = '-P' ]; then p="$1 ${2:-}" shift 2 fi eval l=\${$#} s () { printf '%s\n' "$1" | sed 's/ /\\ /g'; } case "$l" in *:*) c=0; t="$(($#-1))" for f in "$@"; do if [ "$c" -lt "$t" ]; then printf '%s\n' "put $(s "$f")" c="$(($c+1))" fi done | sftp ${p:-} "$l" ;; *) printf '%s\n' "get $(s "${1#*:}") \"$2\"" | sftp ${p:-} "${1%%:*}" ;; esac ``` ⁂ => ../contact.gmi 📝 Comment => . 🔙 Gemlog index => .. 🔝 Capsule index