In this guide we are going to explore how to use Golang script to do operations on an SFTP server.

SFTP (Secure File Transfer Protocol) is a file transfer protocol that leverages a set of utilities that provide secure access to a remote computer to deliver secure communications. It relies on SSH.

# Related Content

• How to work with SFTP client in Linux – 10 sftp commands
• How to set up an SFTP server on Debian 11 Server
• Download Files from SFTP server Using a python script
• List, Upload and Download files from an SFTP Server using golang
• How to set up an SFTP server on OpenSUSE Leap 15.3 Server
• How to install and set up sftp server in Ubuntu 20.04
• How to set up an SFTP server on CentOS 8 /RHEL 8 Server

# Prerequisites

To follow along:

• Ensure you have golang installed locally.
• Ensure you have access to an SFTP server – Username and password
• Ensure you are familiar with the terminal

# Table of content

1. Creating the directory structure and initializing golang module
2. Creating the script: Imports
3. Creating the script: Function to list files
4. Creating the script: Function to download files
5. Creating the script: Full code
6. Building and testing the code

# 1. Creating the directory structure and initializing golang module

We need a directory that will have our content. Create it with this command:

mkdir gosftp

Switch to the directory and initialize a golang module:

➜ cd gosftp
➜ go mod init gosftp
go: creating new go.mod: module gosftp
go: to add module requirements and sums:
    go mod tidy

This will create a file go.mod with this content:

module gosftp

go 1.17

# 2. Creating the script: Imports

Let’s not create the script. Create a file called main.go and add these imports:

package main

import (
    "bufio"
    "fmt"
    "io"
    "log"
    "net"
    "net/url"
    "os"
    "path/filepath"
    "strings"
    "time"

    "golang.org/x/crypto/ssh"
    "golang.org/x/crypto/ssh/agent"

    "github.com/pkg/sftp"
)

# 2. Creating the script: Connecting to the server

Now that we have the imports, let’s use this code to initialize connection to the sftp server:

    // Create a url 
    rawurl := fmt.Sprintf("sftp://%v:%v@%v", sftpUser, sftpPass, sftpHost)

    // Parse the URL 
    parsedUrl, err := url.Parse(rawurl)
    if err != nil {
        log.Fatalf("Failed to parse SFTP To Go URL: %s", err)
    }

    // Get user name and pass
    user := parsedUrl.User.Username()
    pass, _ := parsedUrl.User.Password()

    // Parse Host and Port
    host := parsedUrl.Host

    // Get hostkey 
    hostKey := getHostKey(host)

    log.Printf("Connecting to %s ...\n", host)

    var auths []ssh.AuthMethod

    // Try to use $SSH_AUTH_SOCK which contains the path of the unix file socket that the sshd agent uses
    // for communication with other processes.
    if aconn, err := net.Dial("unix", os.Getenv("SSH_AUTH_SOCK")); err == nil {
        auths = append(auths, ssh.PublicKeysCallback(agent.NewClient(aconn).Signers))
    }

    // Use password authentication if provided
    if pass != "" {
        auths = append(auths, ssh.Password(pass))
    }

    // Initialize client configuration
    config := ssh.ClientConfig{
        User: user,
        Auth: auths,
        // Auth: []ssh.AuthMethod{
        //  ssh.KeyboardInteractive(SshInteractive),
        // },

        // Uncomment to ignore host key check
        // HostKeyCallback: ssh.InsecureIgnoreHostKey(),
        HostKeyCallback: ssh.FixedHostKey(hostKey),
        // HostKeyCallback: func(hostname string, remote net.Addr, key ssh.PublicKey) error {
        //  return nil
        // },
        Timeout: 30 * time.Second,
    }

    addr := fmt.Sprintf("%s:%s", host, sftpPort)

    // Connect to server
    conn, err := ssh.Dial("tcp", addr, &config)
    if err != nil {
        log.Fatalf("Failed to connec to host [%s]: %v", addr, err)
    }

    defer conn.Close()

    // Create new SFTP client
    sc, err := sftp.NewClient(conn)
    if err != nil {
        log.Fatalf("Unable to start SFTP subsystem: %v", err)
    }
    defer sc.Close()

# 3. Creating the script: Function to list files

Now let’s create a function to list files. We are using the connection to the sftp server to read contents of the remote dir then add to a list of of structs to return.

func listFiles(sc sftp.Client, remoteDir string) (theFiles []remoteFiles, err error) {

    files, err := sc.ReadDir(remoteDir)
    if err != nil {
        return theFiles, fmt.Errorf("Unable to list remote dir: %v", err)
    }

    for _, f := range files {
        var name, modTime, size string

        name = f.Name()
        modTime = f.ModTime().Format("2006-01-02 15:04:05")
        size = fmt.Sprintf("%12d", f.Size())

        if f.IsDir() {
            name = name + "/"
            modTime = ""
            size = "PRE"
        }

        theFiles = append(theFiles, remoteFiles{
            Name:    name,
            Size:    size,
            ModTime: modTime,
        })
    }

    return theFiles, nil
}

# 4. Creating the script: Function to upload files

Let’s create a function to upload files to the sftp server. We will use the connection to open a file, then make remote directories recursively then copy the data from the local file

// Upload file to sftp server
func uploadFile(sc sftp.Client, localFile, remoteFile string) (err error) {
    log.Printf("Uploading [%s] to [%s] ...", localFile, remoteFile)

    srcFile, err := os.Open(localFile)
    if err != nil {
        return fmt.Errorf("Unable to open local file: %v", err)
    }
    defer srcFile.Close()

    // Make remote directories recursion
    parent := filepath.Dir(remoteFile)
    path := string(filepath.Separator)
    dirs := strings.Split(parent, path)
    for _, dir := range dirs {
        path = filepath.Join(path, dir)
        sc.Mkdir(path)
    }

    // Note: SFTP Go doesn't support O_RDWR mode
    dstFile, err := sc.OpenFile(remoteFile, (os.O_WRONLY | os.O_CREATE | os.O_TRUNC))
    if err != nil {
        return fmt.Errorf("Unable to open remote file: %v", err)
    }
    defer dstFile.Close()

    bytes, err := io.Copy(dstFile, srcFile)
    if err != nil {
        return fmt.Errorf("Unable to upload local file: %v", err)
    }
    log.Printf("%d bytes copied", bytes)

    return nil
}

# 5. Creating the script: Function to download files

This function downloads a file from the remote server given the remote path. In this function, we are creating a file in the tmp directory then copying the data from the remote file path.

// Download file from sftp server
func downloadFile(sc sftp.Client, remoteFile, localFile string) (err error) {

    localPath := "/tmp/" + localFile

    log.Printf("Downloading [%s] to [%s] ...", remoteFile, localFile)
    // Note: SFTP To Go doesn't support O_RDWR mode
    srcFile, err := sc.OpenFile(remoteFile, (os.O_RDONLY))
    if err != nil {
        return fmt.Errorf("unable to open remote file: %v", err)
    }
    defer srcFile.Close()

    dstFile, err := os.Create(localPath)
    if err != nil {
        return fmt.Errorf("unable to open local file: %v", err)
    }
    defer dstFile.Close()

    bytes, err := io.Copy(dstFile, srcFile)
    if err != nil {
        return fmt.Errorf("unable to download remote file: %v", err)
    }
    log.Printf("%d bytes copied to %v", bytes, localPath)

    return nil
}

# 5. Creating the script: Full code

This is the full code for the script to do operations with SFTP using Golang:

package main

import (
    "bufio"
    "fmt"
    "io"
    "log"
    "net"
    "net/url"
    "os"
    "path/filepath"
    "strings"
    "time"

    "golang.org/x/crypto/ssh"
    "golang.org/x/crypto/ssh/agent"

    "github.com/pkg/sftp"
)

const (
    sftpUser = "citizix"
    sftpPass = "Str0ngP4ss"
    sftpHost = "10.2.11.10"
    sftpPort = "22"
)

func main() {
    // Create a url 
    rawurl := fmt.Sprintf("sftp://%v:%v@%v", sftpUser, sftpPass, sftpHost)

    // Parse the URL 
    parsedUrl, err := url.Parse(rawurl)
    if err != nil {
        log.Fatalf("Failed to parse SFTP To Go URL: %s", err)
    }

    // Get user name and pass
    user := parsedUrl.User.Username()
    pass, _ := parsedUrl.User.Password()

    // Parse Host and Port
    host := parsedUrl.Host

    // Get hostkey 
    hostKey := getHostKey(host)

    log.Printf("Connecting to %s ...\n", host)

    var auths []ssh.AuthMethod

    // Try to use $SSH_AUTH_SOCK which contains the path of the unix file socket that the sshd agent uses
    // for communication with other processes.
    if aconn, err := net.Dial("unix", os.Getenv("SSH_AUTH_SOCK")); err == nil {
        auths = append(auths, ssh.PublicKeysCallback(agent.NewClient(aconn).Signers))
    }

    // Use password authentication if provided
    if pass != "" {
        auths = append(auths, ssh.Password(pass))
    }

    // Initialize client configuration
    config := ssh.ClientConfig{
        User: user,
        Auth: auths,
        // Auth: []ssh.AuthMethod{
        //  ssh.KeyboardInteractive(SshInteractive),
        // },

        // Uncomment to ignore host key check
        // HostKeyCallback: ssh.InsecureIgnoreHostKey(),
        HostKeyCallback: ssh.FixedHostKey(hostKey),
        // HostKeyCallback: func(hostname string, remote net.Addr, key ssh.PublicKey) error {
        //  return nil
        // },
        Timeout: 30 * time.Second,
    }

    addr := fmt.Sprintf("%s:%s", host, sftpPort)

    // Connect to server
    conn, err := ssh.Dial("tcp", addr, &config)
    if err != nil {
        log.Fatalf("Failed to connec to host [%s]: %v", addr, err)
    }

    defer conn.Close()

    // Create new SFTP client
    sc, err := sftp.NewClient(conn)
    if err != nil {
        log.Fatalf("Unable to start SFTP subsystem: %v", err)
    }
    defer sc.Close()

    // List files in the root directory .
    theFiles, err := listFiles(*sc, ".")
    if err != nil {
        log.Fatalf("failed to list files in .: %v", err)
    }

    log.Printf("Found Files in . Files")
    // Output each file name and size in bytes
    log.Printf("%19s %12s %s", "MOD TIME", "SIZE", "NAME")
    for _, theFile := range theFiles {
        log.Printf("%19s %12s %s", theFile.ModTime, theFile.Size, theFile.Name)
    }

    // Upload local file
    err = uploadFile(*sc, "/Users/etowett/Desktop/data.csv", "./citizix/data.csv")
    if err != nil {
        log.Fatalf("could not upload file: %v", err)
    }

    // Download remote file to local file.
    err = downloadFile(*sc, "citizix/data.csv", "data.csv")
    if err != nil {
        log.Fatalf("Could not download file data.csv; %v", err)
    }
    return
}

func SshInteractive(user, instruction string, questions []string, echos []bool) (answers []string, err error) {
    // Hack, check https://stackoverflow.com/questions/47102080/ssh-in-go-unable-to-authenticate-attempted-methods-none-no-supported-method
    answers = make([]string, len(questions))
    // The second parameter is unused
    for n, _ := range questions {
        answers[n] = sftpPass
    }

    return answers, nil
}

type remoteFiles struct {
    Name    string
    Size    string
    ModTime string
}

func listFiles(sc sftp.Client, remoteDir string) (theFiles []remoteFiles, err error) {

    files, err := sc.ReadDir(remoteDir)
    if err != nil {
        return theFiles, fmt.Errorf("Unable to list remote dir: %v", err)
    }

    for _, f := range files {
        var name, modTime, size string

        name = f.Name()
        modTime = f.ModTime().Format("2006-01-02 15:04:05")
        size = fmt.Sprintf("%12d", f.Size())

        if f.IsDir() {
            name = name + "/"
            modTime = ""
            size = "PRE"
        }

        theFiles = append(theFiles, remoteFiles{
            Name:    name,
            Size:    size,
            ModTime: modTime,
        })
    }

    return theFiles, nil
}

// Upload file to sftp server
func uploadFile(sc sftp.Client, localFile, remoteFile string) (err error) {
    log.Printf("Uploading [%s] to [%s] ...", localFile, remoteFile)

    srcFile, err := os.Open(localFile)
    if err != nil {
        return fmt.Errorf("Unable to open local file: %v", err)
    }
    defer srcFile.Close()

    // Make remote directories recursion
    parent := filepath.Dir(remoteFile)
    path := string(filepath.Separator)
    dirs := strings.Split(parent, path)
    for _, dir := range dirs {
        path = filepath.Join(path, dir)
        sc.Mkdir(path)
    }

    // Note: SFTP Go doesn't support O_RDWR mode
    dstFile, err := sc.OpenFile(remoteFile, (os.O_WRONLY | os.O_CREATE | os.O_TRUNC))
    if err != nil {
        return fmt.Errorf("Unable to open remote file: %v", err)
    }
    defer dstFile.Close()

    bytes, err := io.Copy(dstFile, srcFile)
    if err != nil {
        return fmt.Errorf("Unable to upload local file: %v", err)
    }
    log.Printf("%d bytes copied", bytes)

    return nil
}

// Download file from sftp server
func downloadFile(sc sftp.Client, remoteFile, localFile string) (err error) {

    log.Printf("Downloading [%s] to [%s] ...\n", remoteFile, localFile)
    // Note: SFTP To Go doesn't support O_RDWR mode
    srcFile, err := sc.OpenFile(remoteFile, (os.O_RDONLY))
    if err != nil {
        return fmt.Errorf("unable to open remote file: %v", err)
    }
    defer srcFile.Close()

    dstFile, err := os.Create(localFile)
    if err != nil {
        return fmt.Errorf("unable to open local file: %v", err)
    }
    defer dstFile.Close()

    bytes, err := io.Copy(dstFile, srcFile)
    if err != nil {
        return fmt.Errorf("unable to download remote file: %v", err)
    }
    log.Printf("%d bytes copied to %v", bytes, dstFile)

    return nil
}

// Get host key from local known hosts
func getHostKey(host string) ssh.PublicKey {
    // parse OpenSSH known_hosts file
    // ssh or use ssh-keyscan to get initial key
    file, err := os.Open(filepath.Join(os.Getenv("HOME"), ".ssh", "known_hosts"))
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to read known_hosts file: %v\n", err)
        os.Exit(1)
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    var hostKey ssh.PublicKey
    for scanner.Scan() {
        fields := strings.Split(scanner.Text(), " ")
        if len(fields) != 3 {
            continue
        }
        if strings.Contains(fields[0], host) {
            var err error
            hostKey, _, _, _, err = ssh.ParseAuthorizedKey(scanner.Bytes())
            if err != nil {
                fmt.Fprintf(os.Stderr, "Error parsing %q: %v\n", fields[2], err)
                os.Exit(1)
            }
            break
        }
    }

    if hostKey == nil {
        fmt.Fprintf(os.Stderr, "No hostkey found for %s", host)
        os.Exit(1)
    }

    return hostKey
}

# 6. Building and testing the code

Now that we have our code, let’s build and test it.

First ensure all dependencies are downloaded using the go mod tidy command.
This is my output:

 ❯ go mod tidy
go: finding module for package golang.org/x/crypto/ssh
go: finding module for package github.com/pkg/sftp
go: found github.com/pkg/sftp in github.com/pkg/sftp v1.13.4
go: found golang.org/x/crypto/ssh in golang.org/x/crypto v0.0.0-20210921155107-089bfa567519

Next let’s build our code into a gosftp binary in the current directory:

➜ go build -o gosftp

Now run the script. This is my output:

➜ ./gosftp
2021/10/08 13:10:36 Connecting to 10.2.11.10 ...
2021/10/08 13:10:43 Found Files in . Files
2021/10/08 13:10:43            MOD TIME         SIZE NAME
2021/10/08 13:10:43                              PRE etowett/
2021/10/08 13:10:43                              PRE citizix/
2021/10/08 13:10:43                              PRE PAYMENTDATA/
2021/10/08 13:10:43 Uploading [/Users/etowett/Desktop/data.csv] to [./citizix/data.csv] ...
2021/10/08 13:10:44 24 bytes copied
2021/10/08 13:10:45 Downloading [citizix/data.csv] to [data.csv] ...
2021/10/08 13:10:46 24 bytes copied to &{0xc000090a20}

# Conclusion

In this article, we managed to create a script to list files in a remote sftp server, Upload files and download files.