Related
I'm currently developing a custom scanner in go for OpenVAS. Problem is that the handshake with my custom server fails.
I traced the problem down to error -73: GNUTLS_E_ASN1_TAG_ERROR from gnutls_handshake, but I can't find any resources on that problem. I read something about the certificates being incorrect then but I can't do anything other than regenerating the OpenVAS certificates. The tls functionality in the go server just uses a simple ListenAndServeTLS and gets the server cert and key.
edit:
So this is the relevant network part on the custom scanner:
var (
port = ":1234"
cert = "/usr/local/var/lib/openvas/CA/servercert.pem"
ca = "/usr/local/var/lib/openvas/CA/cacert.pem"
key = "/usr/local/var/lib/openvas/private/CA/serverkey.pem"
)
func start_server() {
ca_file, err := ioutil.ReadFile(ca)
if err != nil {
panic(err)
}
blocks, _ := pem.Decode( ca_file )
ca, err := x509.ParseCertificate(blocks.Bytes)
if err != nil {
panic(err)
}
priv_file, _ := ioutil.ReadFile(key)
blocks2, _ := pem.Decode( priv_file )
priv, err := x509.ParsePKCS1PrivateKey(blocks2.Bytes)
if err != nil {
panic(err)
}
pool := x509.NewCertPool()
pool.AddCert(ca)
cert := tls.Certificate{
Certificate: [][]byte{ca_file},
PrivateKey: priv,
}
config := tls.Config{
ClientAuth: tls.RequireAndVerifyClientCert,
Certificates: []tls.Certificate{cert},
ClientCAs: pool,
}
config.Rand = rand.Reader
service := "0.0.0.0" + port
listener, err := tls.Listen("tcp", service, &config)
if err != nil {
log.Fatalf("server: listen: %s", err)
}
log.Print("server: listening")
for {
conn, err := listener.Accept()
if err != nil {
log.Printf("server: accept: %s", err)
break
}
defer conn.Close()
log.Printf("server: accepted from %s", conn.RemoteAddr())
go handle(conn)
}
}
func handle(conn net.Conn) {
str := "Hello"
defer conn.Close()
buf := make([]byte, 512)
for {
log.Print("server: conn: waiting")
conn.Write( ([]byte)(str) )
n, err := conn.Read(buf)
if err != nil {
if err != nil {
fmt.Println (err)
}
break
}
tlscon, ok := conn.(*tls.Conn)
if ok {
state := tlscon.ConnectionState()
sub := state.PeerCertificates[0].Subject
log.Println(sub)
}
log.Printf("server: conn: echo %q\n", string(buf[:n]))
n, err = conn.Write(buf[:n])
n, err = conn.Write(buf[:n])
log.Printf("server: conn: wrote %d bytes", n)
if err != nil {
log.Printf("server: write: %s", err)
break
}
}
log.Println("server: conn: closed")
}
func main() {
start_server()
}
It's taken from an example but it didn't work properly at first ( there was no decode before parsecertificates). Maybe the certificate is mal-formatted now because of that? Before adding the two decodes I had a similar error about the asn1 tags not matching. So also an asn1 error. I thought of generating my own certificate but I don't know if this will not break OpenVAS for the other scanners. I had the same results when just using listenandservetls from go. The error is definitely produced in gnutls_handshake. It's frustrating that I only get an error code from that.
I am working on some code for a device that will not be able to query a DNS. I will only have ports 80 and 443 available.
The following works, but of course hits the DNS. The domain used is my personal domain, not the real domain the problem is for - it’s work related and redacted. This is simply used to illustrate the issue.
package main
import (
“log”
“net/http”
)
func main() {
client := &http.Client{}
req, err := http.NewRequest(“GET”, “https://donatstudios.com/images/Spacecat/spacecat.svg”, nil)
if err != nil {
log.Fatal(err)
}
_, err = client.Do(req)
if err != nil {
log.Fatal(err)
}
log.Fatal(“no errors”)
}
I change the code to hit the specific IP address ala:
package main
import (
“log”
“net/http”
)
func main() {
client := &http.Client{}
req, err := http.NewRequest(“GET”, “https://162.243.23.224/images/Spacecat/spacecat.svg”, nil)
if err != nil {
log.Fatal(err)
}
req.Host = “donatstudios.com”
_, err = client.Do(req)
if err != nil {
log.Fatal(err)
}
log.Fatal(“no errors”)
}
And now receive “cannot validate certificate for 162.243.23.224 because it doesn't contain any IP SANs”
When not using an https domain the above code works.
Presumably this is something to do with SSL. #go-nuts told me they believe this to happen before it ever hit the HTTP layer? I’ve been poking this for hours and cannot figure out how to make it work.
I'm assuming your server is using SNI like the one used in the example. Give this a try and see if it works for you.
package main
import (
"crypto/tls"
"log"
"net/http"
)
func main() {
tlsConfig := &tls.Config{
ServerName: "moupon.co",
}
tlsConfig.BuildNameToCertificate()
transport := &http.Transport{TLSClientConfig: tlsConfig}
client := &http.Client{Transport: transport}
req, err := http.NewRequest("GET", "https://216.239.32.21/s/img/logo.png", nil)
if err != nil {
log.Fatal(err)
}
req.Host = "moupon.co"
_, err = client.Do(req)
if err != nil {
log.Fatal(err)
}
log.Fatal("no errors")
}
I'm trying to run multiple commands through ssh but seems that Session.Run allows only one command per session ( unless I'm wrong). I'm wondering how can I bypass this limitation and reuse the session or send a sequence of commands.
The reason is that I need to run sudo su within the same session with the next command ( sh /usr/bin/myscript.sh )
Session.Shell allows for more than one command to be run, by passing your commands in via session.StdinPipe().
Be aware that using this approach will make your life more complicated; instead of having a one-shot function call that runs the command and collects the output once it's complete, you'll need to manage your input buffer (don't forget a \n at the end of a command), wait for output to actually come back from the SSH server, then deal with that output appropriately (if you had multiple commands in flight and want to know what output belongs to what input, you'll need to have a plan to figure that out).
stdinBuf, _ := session.StdinPipe()
err := session.Shell()
stdinBuf.Write([]byte("cd /\n"))
// The command has been sent to the device, but you haven't gotten output back yet.
// Not that you can't send more commands immediately.
stdinBuf.Write([]byte("ls\n"))
// Then you'll want to wait for the response, and watch the stdout buffer for output.
While for your specific problem, you can easily run sudo /path/to/script.sh, it shock me that there wasn't a simple way to run multiple commands on the same session, so I came up with a bit of a hack, YMMV:
func MuxShell(w io.Writer, r io.Reader) (chan<- string, <-chan string) {
in := make(chan string, 1)
out := make(chan string, 1)
var wg sync.WaitGroup
wg.Add(1) //for the shell itself
go func() {
for cmd := range in {
wg.Add(1)
w.Write([]byte(cmd + "\n"))
wg.Wait()
}
}()
go func() {
var (
buf [65 * 1024]byte
t int
)
for {
n, err := r.Read(buf[t:])
if err != nil {
close(in)
close(out)
return
}
t += n
if buf[t-2] == '$' { //assuming the $PS1 == 'sh-4.3$ '
out <- string(buf[:t])
t = 0
wg.Done()
}
}
}()
return in, out
}
func main() {
config := &ssh.ClientConfig{
User: "kf5",
Auth: []ssh.AuthMethod{
ssh.Password("kf5"),
},
}
client, err := ssh.Dial("tcp", "127.0.0.1:22", config)
if err != nil {
panic(err)
}
defer client.Close()
session, err := client.NewSession()
if err != nil {
log.Fatalf("unable to create session: %s", err)
}
defer session.Close()
modes := ssh.TerminalModes{
ssh.ECHO: 0, // disable echoing
ssh.TTY_OP_ISPEED: 14400, // input speed = 14.4kbaud
ssh.TTY_OP_OSPEED: 14400, // output speed = 14.4kbaud
}
if err := session.RequestPty("xterm", 80, 40, modes); err != nil {
log.Fatal(err)
}
w, err := session.StdinPipe()
if err != nil {
panic(err)
}
r, err := session.StdoutPipe()
if err != nil {
panic(err)
}
in, out := MuxShell(w, r)
if err := session.Start("/bin/sh"); err != nil {
log.Fatal(err)
}
<-out //ignore the shell output
in <- "ls -lhav"
fmt.Printf("ls output: %s\n", <-out)
in <- "whoami"
fmt.Printf("whoami: %s\n", <-out)
in <- "exit"
session.Wait()
}
If your shell prompt doesn't end with $ ($ followed by a space), this will deadlock, hence why it's a hack.
NewSession is a method of a connection. You don't need to create a new connection each time. A Session seems to be what this library calls a channel for the client, and many channels are multiplexed in a single connection. Hence:
func executeCmd(cmd []string, hostname string, config *ssh.ClientConfig) string {
conn, err := ssh.Dial("tcp", hostname+":8022", config)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
var stdoutBuf bytes.Buffer
for _, command := range cmd {
session, err := conn.NewSession()
if err != nil {
log.Fatal(err)
}
defer session.Close()
session.Stdout = &stdoutBuf
session.Run(command)
}
return hostname + ": " + stdoutBuf.String()
}
So you open a new session(channel) and you run command within the existing ssh connection but with a new session(channel) each time.
You can use a small trick: sh -c 'cmd1&&cmd2&&cmd3&&cmd4&&etc..'
This is a single command, the actual commands are passed as argument to the shell which will execute them. This is how Docker handles multiple commands.
This works for me.
package main
import (
"fmt"
"golang.org/x/crypto/ssh"
// "io"
"log"
"os"
// Uncomment to store output in variable
//"bytes"
)
type MachineDetails struct {
username, password, hostname, port string
}
func main() {
h1 := MachineDetails{"root", "xxxxx", "x.x.x.x", "22"}
// Uncomment to store output in variable
//var b bytes.Buffer
//sess.Stdout = &b
//sess.Stderr = &b
commands := []string{
"pwd",
"whoami",
"echo 'bye'",
"exit",
}
connectHost(h1, commands)
// Uncomment to store in variable
//fmt.Println(b.String())
}
func connectHost(hostParams MachineDetails, commands []string) {
// SSH client config
config := &ssh.ClientConfig{
User: hostParams.username,
Auth: []ssh.AuthMethod{
ssh.Password(hostParams.password),
},
// Non-production only
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
}
// Connect to host
client, err := ssh.Dial("tcp", hostParams.hostname+":"+hostParams.port, config)
if err != nil {
log.Fatal(err)
}
defer client.Close()
// Create sesssion
sess, err := client.NewSession()
if err != nil {
log.Fatal("Failed to create session: ", err)
}
defer sess.Close()
// Enable system stdout
// Comment these if you uncomment to store in variable
sess.Stdout = os.Stdout
sess.Stderr = os.Stderr
// StdinPipe for commands
stdin, err := sess.StdinPipe()
if err != nil {
log.Fatal(err)
}
// Start remote shell
err = sess.Shell()
if err != nil {
log.Fatal(err)
}
// send the commands
for _, cmd := range commands {
_, err = fmt.Fprintf(stdin, "%s\n", cmd)
if err != nil {
log.Fatal(err)
}
}
// Wait for sess to finish
err = sess.Wait()
if err != nil {
log.Fatal(err)
}
// return sess, stdin, err
}
func createSession() {
}
Really liked OneOfOne's answer which inspired me with a more generalized solution to taken a variable that could match the tail of the read bytes and break the blocking read (also no need to fork two extra threads for blocking read and writes). The known limitation is (as in the original solution) if the matching string comes after 64 * 1024 bytes, then this code will spin forever.
package main
import (
"fmt"
"golang.org/x/crypto/ssh"
"io"
"log"
)
var escapePrompt = []byte{'$', ' '}
func main() {
config := &ssh.ClientConfig{
User: "dummy",
Auth: []ssh.AuthMethod{
ssh.Password("dummy"),
},
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
}
client, err := ssh.Dial("tcp", "127.0.0.1:22", config)
if err != nil {
panic(err)
}
defer client.Close()
session, err := client.NewSession()
if err != nil {
log.Fatalf("unable to create session: %s", err)
}
defer session.Close()
modes := ssh.TerminalModes{
ssh.ECHO: 0, // disable echoing
ssh.TTY_OP_ISPEED: 14400, // input speed = 14.4kbaud
ssh.TTY_OP_OSPEED: 14400, // output speed = 14.4kbaud
}
if err := session.RequestPty("xterm", 80, 40, modes); err != nil {
log.Fatal(err)
}
w, err := session.StdinPipe()
if err != nil {
panic(err)
}
r, err := session.StdoutPipe()
if err != nil {
panic(err)
}
if err := session.Start("/bin/sh"); err != nil {
log.Fatal(err)
}
readUntil(r, escapePrompt) //ignore the shell output
write(w, "ls -lhav")
out, err := readUntil(r, escapePrompt)
fmt.Printf("ls output: %s\n", *out)
write(w, "whoami")
out, err = readUntil(r, escapePrompt)
fmt.Printf("whoami: %s\n", *out)
write(w, "exit")
session.Wait()
}
func write(w io.WriteCloser, command string) error {
_, err := w.Write([]byte(command + "\n"))
return err
}
func readUntil(r io.Reader, matchingByte []byte) (*string, error) {
var buf [64 * 1024]byte
var t int
for {
n, err := r.Read(buf[t:])
if err != nil {
return nil, err
}
t += n
if isMatch(buf[:t], t, matchingByte) {
stringResult := string(buf[:t])
return &stringResult, nil
}
}
}
func isMatch(bytes []byte, t int, matchingBytes []byte) bool {
if t >= len(matchingBytes) {
for i := 0; i < len(matchingBytes); i++ {
if bytes[t - len(matchingBytes) + i] != matchingBytes[i] {
return false
}
}
return true
}
return false
}
get inspiration from this
i spent several days and that answer inspires me to try about using sdtin to run multiple commands, finally succeed. and i want to say i dont know golang at all , hence it may be redundant ,but the code works.
if _, err := w.Write([]byte("sys\r")); err != nil {
panic("Failed to run: " + err.Error())
}
if _, err := w.Write([]byte("wlan\r")); err != nil {
panic("Failed to run: " + err.Error())
}
if _, err := w.Write([]byte("ap-id 2099\r")); err != nil {
panic("Failed to run: " + err.Error())
}
if _, err := w.Write([]byte("ap-group xuebao-free\r")); err != nil {
panic("Failed to run: " + err.Error())
}
if _, err := w.Write([]byte("y\r")); err != nil {
panic("Failed to run: " + err.Error())
}
its function is the same asterminal operation
here is the whole code:
/* switch ssh
*/
package main
import (
"flag"
"fmt"
"io"
"log"
"net"
"os"
"strings"
"sync"
)
import (
"golang.org/x/crypto/ssh"
)
func main() {
//go run ./testConfig.go --username="aaa" --passwd='aaa' --ip_port="192.168.6.87" --cmd='display version'
username := flag.String("username", "aaa", "username")
passwd := flag.String("passwd", "aaa", "password")
ip_port := flag.String("ip_port", "1.1.1.1:22", "ip and port")
cmdstring := flag.String("cmd", "display arp statistics all", "cmdstring")
flag.Parse()
fmt.Println("username:", *username)
fmt.Println("passwd:", *passwd)
fmt.Println("ip_port:", *ip_port)
fmt.Println("cmdstring:", *cmdstring)
config := &ssh.ClientConfig{
User: *username,
Auth: []ssh.AuthMethod{
ssh.Password(*passwd),
},
Config: ssh.Config{
Ciphers: []string{"aes128-cbc", "aes128-ctr"},
},
HostKeyCallback: func(hostname string, remote net.Addr, key ssh.PublicKey) error {
return nil
},
}
// config.Config.Ciphers = append(config.Config.Ciphers, "aes128-cbc")
clinet, err := ssh.Dial("tcp", *ip_port, config)
checkError(err, "connet "+*ip_port)
session, err := clinet.NewSession()
defer session.Close()
checkError(err, "creae shell")
modes := ssh.TerminalModes{
ssh.ECHO: 1, // disable echoing
ssh.TTY_OP_ISPEED: 14400, // input speed = 14.4kbaud
ssh.TTY_OP_OSPEED: 14400, // output speed = 14.4kbaud
}
if err := session.RequestPty("vt100", 80, 40, modes); err != nil {
log.Fatal(err)
}
w, err := session.StdinPipe()
if err != nil {
panic(err)
}
r, err := session.StdoutPipe()
if err != nil {
panic(err)
}
e, err := session.StderrPipe()
if err != nil {
panic(err)
}
in, out := MuxShell(w, r, e)
if err := session.Shell(); err != nil {
log.Fatal(err)
}
<-out //ignore the shell output
in <- *cmdstring
fmt.Printf("%s\n", <-out)
if _, err := w.Write([]byte("sys\r")); err != nil {
panic("Failed to run: " + err.Error())
}
if _, err := w.Write([]byte("wlan\r")); err != nil {
panic("Failed to run: " + err.Error())
}
if _, err := w.Write([]byte("ap-id 2099\r")); err != nil {
panic("Failed to run: " + err.Error())
}
if _, err := w.Write([]byte("ap-group xuebao-free\r")); err != nil {
panic("Failed to run: " + err.Error())
}
if _, err := w.Write([]byte("y\r")); err != nil {
panic("Failed to run: " + err.Error())
}
in <- "quit"
_ = <-out
session.Wait()
}
func checkError(err error, info string) {
if err != nil {
fmt.Printf("%s. error: %s\n", info, err)
os.Exit(1)
}
}
func MuxShell(w io.Writer, r, e io.Reader) (chan<- string, <-chan string) {
in := make(chan string, 5)
out := make(chan string, 5)
var wg sync.WaitGroup
wg.Add(1) //for the shell itself
go func() {
for cmd := range in {
wg.Add(1)
w.Write([]byte(cmd + "\n"))
wg.Wait()
}
}()
go func() {
var (
buf [1024 * 1024]byte
t int
)
for {
n, err := r.Read(buf[t:])
if err != nil {
fmt.Println(err.Error())
close(in)
close(out)
return
}
t += n
result := string(buf[:t])
if strings.Contains(string(buf[t-n:t]), "More") {
w.Write([]byte("\n"))
}
if strings.Contains(result, "username:") ||
strings.Contains(result, "password:") ||
strings.Contains(result, ">") {
out <- string(buf[:t])
t = 0
wg.Done()
}
}
}()
return in, out
}
The following code works for me.
func main() {
key, err := ioutil.ReadFile("path to your key file")
if err != nil {
panic(err)
}
signer, err := ssh.ParsePrivateKey([]byte(key))
if err != nil {
panic(err)
}
config := &ssh.ClientConfig{
User: "ubuntu",
Auth: []ssh.AuthMethod{
ssh.PublicKeys(signer),
},
}
client, err := ssh.Dial("tcp", "52.91.35.179:22", config)
if err != nil {
panic(err)
}
session, err := client.NewSession()
if err != nil {
panic(err)
}
defer session.Close()
session.Stdout = os.Stdout
session.Stderr = os.Stderr
session.Stdin = os.Stdin
session.Shell()
session.Wait()
}
I have a question regarding how to send input and receive output from a terminal subprocess such as ssh. An example in python would be something like this:
how to give subprocess a password and get stdout at the same time
I cannot find a simple example in Golang that is similar how the above work.
In Golang I would want to do something like this but it does not seem to work:
cmd := exec.Command("ssh", "user#x.x.x.x")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
stdin, _ := cmd.StdinPipe()
stdin.Write([]byte("password\n"))
cmd.Run()
However; I'm not sure how to do this in go because every time i exec this ssh command I am only able to get the output. I am unable to input my password automatically from code.
Does anyone have examples of writing to terminal processes such as ssh? If so, please share.
Thanks to the comments above, I was able to get ssh access working with a password. I used golang's ssh api library. It was fairly simple as I followed the examples from:
https://code.google.com/p/go/source/browse/ssh/example_test.go?repo=crypto
Specifically:
func ExampleDial() {
// An SSH client is represented with a ClientConn. Currently only
// the "password" authentication method is supported.
//
// To authenticate with the remote server you must pass at least one
// implementation of AuthMethod via the Auth field in ClientConfig.
config := &ClientConfig{
User: "username",
Auth: []AuthMethod{
Password("yourpassword"),
},
}
client, err := Dial("tcp", "yourserver.com:22", config)
if err != nil {
panic("Failed to dial: " + err.Error())
}
// Each ClientConn can support multiple interactive sessions,
// represented by a Session.
session, err := client.NewSession()
if err != nil {
panic("Failed to create session: " + err.Error())
}
defer session.Close()
// Once a Session is created, you can execute a single command on
// the remote side using the Run method.
var b bytes.Buffer
session.Stdout = &b
if err := session.Run("/usr/bin/whoami"); err != nil {
panic("Failed to run: " + err.Error())
}
fmt.Println(b.String())
}
This is a modified/complete version of above example https://godoc.org/golang.org/x/crypto/ssh#example-Dial
First get terminal package by go get golang.org/x/crypto/ssh
package main
import (
"bufio"
"bytes"
"fmt"
"os"
"strings"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/terminal"
)
func main() {
if len(os.Args) < 3 {
usage := "\n./remote-ssh {host} {port}"
fmt.Println(usage)
} else {
host := os.Args[1]
port := os.Args[2]
username, password := credentials()
config := &ssh.ClientConfig{
User: username,
Auth: []ssh.AuthMethod{
ssh.Password(password),
},
}
connectingMsg := fmt.Sprintf("\nConnecting to %s:%v remote server...", host, port)
fmt.Println(connectingMsg)
hostAddress := strings.Join([]string{host, port}, ":")
// fmt.Println("Host add %s ", hostAddress)
client, err := ssh.Dial("tcp", hostAddress, config)
if err != nil {
panic("Failed to dial: " + err.Error())
}
for {
session, err := client.NewSession()
if err != nil {
panic("Failed to create session: " + err.Error())
}
defer session.Close()
// Once a Session is created, can execute a single command on remote side
var cmd string
str := "\nEnter command (e.g. /usr/bin/whoami OR enter 'exit' to return) : "
fmt.Print(str)
fmt.Scanf("%s", &cmd)
if cmd == "exit" || cmd == "EXIT" {
break
}
s := fmt.Sprintf("Wait for command '%s' run and response...", cmd)
fmt.Println(s)
var b bytes.Buffer
session.Stdout = &b
if err := session.Run(cmd); err != nil {
panic("Failed to run: " + err.Error())
}
fmt.Println(b.String())
}
}
}
func credentials() (string, string) {
reader := bufio.NewReader(os.Stdin)
fmt.Print("Enter Username: ")
username, _ := reader.ReadString('\n')
fmt.Print("Enter Password: ")
bytePassword, err := terminal.ReadPassword(0)
if err != nil {
panic(err)
}
password := string(bytePassword)
return strings.TrimSpace(username), strings.TrimSpace(password)
}
https://play.golang.org/p/4Ad1vKNXmI
I'm trying to upload images from my computer to a website using go. Usually, I use a bash script that sends a file and a key to the server:
curl -F "image"=#"IMAGEFILE" -F "key"="KEY" URL
it works fine, but I'm trying to convert this request into my golang program.
http://matt.aimonetti.net/posts/2013/07/01/golang-multipart-file-upload-example/
I tried this link and many others, but, for each code that I try, the response from the server is "no image sent", and I've no idea why. If someone knows what's happening with the example above.
Here's some sample code.
In short, you'll need to use the mime/multipart package to build the form.
package main
import (
"bytes"
"fmt"
"io"
"mime/multipart"
"net/http"
"net/http/httptest"
"net/http/httputil"
"os"
"strings"
)
func main() {
var client *http.Client
var remoteURL string
{
//setup a mocked http client.
ts := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
b, err := httputil.DumpRequest(r, true)
if err != nil {
panic(err)
}
fmt.Printf("%s", b)
}))
defer ts.Close()
client = ts.Client()
remoteURL = ts.URL
}
//prepare the reader instances to encode
values := map[string]io.Reader{
"file": mustOpen("main.go"), // lets assume its this file
"other": strings.NewReader("hello world!"),
}
err := Upload(client, remoteURL, values)
if err != nil {
panic(err)
}
}
func Upload(client *http.Client, url string, values map[string]io.Reader) (err error) {
// Prepare a form that you will submit to that URL.
var b bytes.Buffer
w := multipart.NewWriter(&b)
for key, r := range values {
var fw io.Writer
if x, ok := r.(io.Closer); ok {
defer x.Close()
}
// Add an image file
if x, ok := r.(*os.File); ok {
if fw, err = w.CreateFormFile(key, x.Name()); err != nil {
return
}
} else {
// Add other fields
if fw, err = w.CreateFormField(key); err != nil {
return
}
}
if _, err = io.Copy(fw, r); err != nil {
return err
}
}
// Don't forget to close the multipart writer.
// If you don't close it, your request will be missing the terminating boundary.
w.Close()
// Now that you have a form, you can submit it to your handler.
req, err := http.NewRequest("POST", url, &b)
if err != nil {
return
}
// Don't forget to set the content type, this will contain the boundary.
req.Header.Set("Content-Type", w.FormDataContentType())
// Submit the request
res, err := client.Do(req)
if err != nil {
return
}
// Check the response
if res.StatusCode != http.StatusOK {
err = fmt.Errorf("bad status: %s", res.Status)
}
return
}
func mustOpen(f string) *os.File {
r, err := os.Open(f)
if err != nil {
panic(err)
}
return r
}
Here's a function I've used that uses io.Pipe() to avoid reading in the entire file to memory or needing to manage any buffers. It handles only a single file, but could easily be extended to handle more by adding more parts within the goroutine. The happy path works well. The error paths have not hand much testing.
import (
"fmt"
"io"
"mime/multipart"
"net/http"
"os"
)
func UploadMultipartFile(client *http.Client, uri, key, path string) (*http.Response, error) {
body, writer := io.Pipe()
req, err := http.NewRequest(http.MethodPost, uri, body)
if err != nil {
return nil, err
}
mwriter := multipart.NewWriter(writer)
req.Header.Add("Content-Type", mwriter.FormDataContentType())
errchan := make(chan error)
go func() {
defer close(errchan)
defer writer.Close()
defer mwriter.Close()
w, err := mwriter.CreateFormFile(key, path)
if err != nil {
errchan <- err
return
}
in, err := os.Open(path)
if err != nil {
errchan <- err
return
}
defer in.Close()
if written, err := io.Copy(w, in); err != nil {
errchan <- fmt.Errorf("error copying %s (%d bytes written): %v", path, written, err)
return
}
if err := mwriter.Close(); err != nil {
errchan <- err
return
}
}()
resp, err := client.Do(req)
merr := <-errchan
if err != nil || merr != nil {
return resp, fmt.Errorf("http error: %v, multipart error: %v", err, merr)
}
return resp, nil
}
After having to decode the accepted answer for this question for use in my unit testing I finally ended up with the follow refactored code:
func createMultipartFormData(t *testing.T, fieldName, fileName string) (bytes.Buffer, *multipart.Writer) {
var b bytes.Buffer
var err error
w := multipart.NewWriter(&b)
var fw io.Writer
file := mustOpen(fileName)
if fw, err = w.CreateFormFile(fieldName, file.Name()); err != nil {
t.Errorf("Error creating writer: %v", err)
}
if _, err = io.Copy(fw, file); err != nil {
t.Errorf("Error with io.Copy: %v", err)
}
w.Close()
return b, w
}
func mustOpen(f string) *os.File {
r, err := os.Open(f)
if err != nil {
pwd, _ := os.Getwd()
fmt.Println("PWD: ", pwd)
panic(err)
}
return r
}
Now it should be pretty easy to use:
b, w := createMultipartFormData(t, "image","../luke.png")
req, err := http.NewRequest("POST", url, &b)
if err != nil {
return
}
// Don't forget to set the content type, this will contain the boundary.
req.Header.Set("Content-Type", w.FormDataContentType())
Here is an option that works for files or strings:
package main
import (
"bytes"
"io"
"mime/multipart"
"os"
"strings"
)
func createForm(form map[string]string) (string, io.Reader, error) {
body := new(bytes.Buffer)
mp := multipart.NewWriter(body)
defer mp.Close()
for key, val := range form {
if strings.HasPrefix(val, "#") {
val = val[1:]
file, err := os.Open(val)
if err != nil { return "", nil, err }
defer file.Close()
part, err := mp.CreateFormFile(key, val)
if err != nil { return "", nil, err }
io.Copy(part, file)
} else {
mp.WriteField(key, val)
}
}
return mp.FormDataContentType(), body, nil
}
Example:
package main
import "net/http"
func main() {
form := map[string]string{"image": "#IMAGEFILE", "key": "KEY"}
ct, body, err := createForm(form)
if err != nil {
panic(err)
}
http.Post("https://stackoverflow.com", ct, body)
}
https://golang.org/pkg/mime/multipart#Writer.WriteField
Send file from one service to another:
func UploadFile(network, uri string, f multipart.File, h *multipart.FileHeader) error {
buf := new(bytes.Buffer)
writer := multipart.NewWriter(buf)
part, err := writer.CreateFormFile("file", h.Filename)
if err != nil {
log.Println(err)
return err
}
b, err := ioutil.ReadAll(f)
if err != nil {
log.Println(err)
return err
}
part.Write(b)
writer.Close()
req, _ := http.NewRequest("POST", uri, buf)
req.Header.Add("Content-Type", writer.FormDataContentType())
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
b, _ = ioutil.ReadAll(resp.Body)
if resp.StatusCode >= 400 {
return errors.New(string(b))
}
return nil
}
To extend on #attila-o answer, here is the code I went with to perform a POST HTTP req in Go with:
1 file
configurable file name (f.Name() didn't work)
extra form fields.
Curl representation:
curl -X POST \
http://localhost:9091/storage/add \
-H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
-F owner=0xc916Cfe5c83dD4FC3c3B0Bf2ec2d4e401782875e \
-F password=$PWD \
-F file=#./internal/file_example_JPG_500kB.jpg
Go way:
client := &http.Client{
Timeout: time.Second * 10,
}
req, err := createStoragePostReq(cfg)
res, err := executeStoragePostReq(client, req)
func createStoragePostReq(cfg Config) (*http.Request, error) {
extraFields := map[string]string{
"owner": "0xc916cfe5c83dd4fc3c3b0bf2ec2d4e401782875e",
"password": "pwd",
}
url := fmt.Sprintf("http://localhost:%d%s", cfg.HttpServerConfig().Port(), lethstorage.AddRoute)
b, w, err := createMultipartFormData("file","./internal/file_example_JPG_500kB.jpg", "file_example_JPG_500kB.jpg", extraFields)
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", url, &b)
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", w.FormDataContentType())
return req, nil
}
func executeStoragePostReq(client *http.Client, req *http.Request) (lethstorage.AddRes, error) {
var addRes lethstorage.AddRes
res, err := client.Do(req)
if err != nil {
return addRes, err
}
defer res.Body.Close()
data, err := ioutil.ReadAll(res.Body)
if err != nil {
return addRes, err
}
err = json.Unmarshal(data, &addRes)
if err != nil {
return addRes, err
}
return addRes, nil
}
func createMultipartFormData(fileFieldName, filePath string, fileName string, extraFormFields map[string]string) (b bytes.Buffer, w *multipart.Writer, err error) {
w = multipart.NewWriter(&b)
var fw io.Writer
file, err := os.Open(filePath)
if fw, err = w.CreateFormFile(fileFieldName, fileName); err != nil {
return
}
if _, err = io.Copy(fw, file); err != nil {
return
}
for k, v := range extraFormFields {
w.WriteField(k, v)
}
w.Close()
return
}
I have found this tutorial very helpful to clarify my confusions about file uploading in Go.
Basically you upload the file via ajax using form-data on a client and use the following small snippet of Go code on the server:
file, handler, err := r.FormFile("img") // img is the key of the form-data
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
fmt.Println("File is good")
fmt.Println(handler.Filename)
fmt.Println()
fmt.Println(handler.Header)
f, err := os.OpenFile(handler.Filename, os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
fmt.Println(err)
return
}
defer f.Close()
io.Copy(f, file)
Here r is *http.Request. P.S. this just stores the file in the same folder and does not perform any security checks.