The third parameter of http.Post() allows io.Reader and that means the return value of os.Open() should work. But the below code gets unexpected result, in other words, it won't set Content-Length properly. Perhaps File type doesn't implement something. Is there any proper way to set Content-Length with *File?
package main
import (
"bytes"
"io/ioutil"
"log"
"net/http"
"net/http/httptest"
"os"
)
var sample = []byte(`hello`)
func main() {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
log.Println(r.Header)
if int(r.ContentLength) != len(sample) {
log.Fatal("Unexpected Content-Length:", r.ContentLength)
}
w.Header().Set("Content-Type", "application/json")
w.Write([]byte(`{}`))
}))
defer ts.Close()
file, err := ioutil.TempFile(os.TempDir(), "")
if err != nil {
log.Fatal(err)
}
defer os.Remove(file.Name())
file.Write(sample)
// This works
buf, err := ioutil.ReadFile(file.Name())
if err != nil {
log.Fatal(err)
}
_, err = http.Post(ts.URL, "application/octet-stream", bytes.NewBuffer(buf))
if err != nil {
log.Fatal(err)
}
// This looks fine in my opinion, though it doesn't set Content-Length
f, err := os.Open(file.Name())
if err != nil {
log.Fatal(err)
}
_, err = http.Post(ts.URL, "application/octet-stream", f)
if err != nil {
log.Fatal(err)
}
}
Output:
2009/11/10 23:00:00 map[Content-Type:[application/octet-stream] Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1] Content-Length:[5]]
2009/11/10 23:00:00 map[Content-Type:[application/octet-stream] Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]]
2009/11/10 23:00:00 Unexpected Content-Length:-1
https://play.golang.org/p/hJLN2H9Y9p
If you look at source for NewRequest you can see that contentLength is handled specially for specific input types, and the file reader isn't one of them. You'll have to manually set the Content-Length header if that's important [chunked should also work fine, unless you're sending to an old server impl].
If you want to add a add the Content-Length, you need to stat the file to get the size. The ContentLength isn't calculated automatically because an os.File may not have a useful size.
f, err := os.Open(file.Name())
if err != nil {
log.Fatal(err)
}
req, err := http.NewRequest("POST", ts.URL, f)
if err != nil {
log.Fatal(err)
}
stat, err := f.Stat()
if err != nil {
log.Fatal(err)
}
req.ContentLength = stat.Size()
req.Header.Set("Content-Type", "application/octet-stream")
resp, err = http.Do(req)
...
Anybody who know how to watch docker event with engine-api the go client for docker, or has any example for this ?
I know How to do this now :
package main
import (
"encoding/json"
"fmt"
"io"
"log"
"github.com/docker/engine-api/client"
"github.com/docker/engine-api/types"
"github.com/docker/engine-api/types/events"
"golang.org/x/net/context"
)
func main() {
defaultHeaders := map[string]string{"User-Agent": "engine-api-cli-1.0"}
cli, err := client.NewClient("http://172.17.150.101:2376", "v1.24", nil, defaultHeaders)
if err != nil {
panic(err)
}
options := types.ContainerListOptions{All: true}
containers, err := cli.ContainerList(context.Background(), options)
if err != nil {
panic(err)
}
for _, c := range containers {
fmt.Println(c.ID)
}
body, err := cli.Events(context.Background(), types.EventsOptions{})
if err != nil {
log.Fatal(err)
}
dec := json.NewDecoder(body)
for {
var event events.Message
err := dec.Decode(&event)
if err != nil && err == io.EOF {
break
}
log.Println(event)
}
}
I am working on some code for a device that will not be able to query a DNS. I will only have ports 80 and 443 available.
The following works, but of course hits the DNS. The domain used is my personal domain, not the real domain the problem is for - it’s work related and redacted. This is simply used to illustrate the issue.
package main
import (
“log”
“net/http”
)
func main() {
client := &http.Client{}
req, err := http.NewRequest(“GET”, “https://donatstudios.com/images/Spacecat/spacecat.svg”, nil)
if err != nil {
log.Fatal(err)
}
_, err = client.Do(req)
if err != nil {
log.Fatal(err)
}
log.Fatal(“no errors”)
}
I change the code to hit the specific IP address ala:
package main
import (
“log”
“net/http”
)
func main() {
client := &http.Client{}
req, err := http.NewRequest(“GET”, “https://162.243.23.224/images/Spacecat/spacecat.svg”, nil)
if err != nil {
log.Fatal(err)
}
req.Host = “donatstudios.com”
_, err = client.Do(req)
if err != nil {
log.Fatal(err)
}
log.Fatal(“no errors”)
}
And now receive “cannot validate certificate for 162.243.23.224 because it doesn't contain any IP SANs”
When not using an https domain the above code works.
Presumably this is something to do with SSL. #go-nuts told me they believe this to happen before it ever hit the HTTP layer? I’ve been poking this for hours and cannot figure out how to make it work.
I'm assuming your server is using SNI like the one used in the example. Give this a try and see if it works for you.
package main
import (
"crypto/tls"
"log"
"net/http"
)
func main() {
tlsConfig := &tls.Config{
ServerName: "moupon.co",
}
tlsConfig.BuildNameToCertificate()
transport := &http.Transport{TLSClientConfig: tlsConfig}
client := &http.Client{Transport: transport}
req, err := http.NewRequest("GET", "https://216.239.32.21/s/img/logo.png", nil)
if err != nil {
log.Fatal(err)
}
req.Host = "moupon.co"
_, err = client.Do(req)
if err != nil {
log.Fatal(err)
}
log.Fatal("no errors")
}
I'm trying to upload images from my computer to a website using go. Usually, I use a bash script that sends a file and a key to the server:
curl -F "image"=#"IMAGEFILE" -F "key"="KEY" URL
it works fine, but I'm trying to convert this request into my golang program.
http://matt.aimonetti.net/posts/2013/07/01/golang-multipart-file-upload-example/
I tried this link and many others, but, for each code that I try, the response from the server is "no image sent", and I've no idea why. If someone knows what's happening with the example above.
Here's some sample code.
In short, you'll need to use the mime/multipart package to build the form.
package main
import (
"bytes"
"fmt"
"io"
"mime/multipart"
"net/http"
"net/http/httptest"
"net/http/httputil"
"os"
"strings"
)
func main() {
var client *http.Client
var remoteURL string
{
//setup a mocked http client.
ts := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
b, err := httputil.DumpRequest(r, true)
if err != nil {
panic(err)
}
fmt.Printf("%s", b)
}))
defer ts.Close()
client = ts.Client()
remoteURL = ts.URL
}
//prepare the reader instances to encode
values := map[string]io.Reader{
"file": mustOpen("main.go"), // lets assume its this file
"other": strings.NewReader("hello world!"),
}
err := Upload(client, remoteURL, values)
if err != nil {
panic(err)
}
}
func Upload(client *http.Client, url string, values map[string]io.Reader) (err error) {
// Prepare a form that you will submit to that URL.
var b bytes.Buffer
w := multipart.NewWriter(&b)
for key, r := range values {
var fw io.Writer
if x, ok := r.(io.Closer); ok {
defer x.Close()
}
// Add an image file
if x, ok := r.(*os.File); ok {
if fw, err = w.CreateFormFile(key, x.Name()); err != nil {
return
}
} else {
// Add other fields
if fw, err = w.CreateFormField(key); err != nil {
return
}
}
if _, err = io.Copy(fw, r); err != nil {
return err
}
}
// Don't forget to close the multipart writer.
// If you don't close it, your request will be missing the terminating boundary.
w.Close()
// Now that you have a form, you can submit it to your handler.
req, err := http.NewRequest("POST", url, &b)
if err != nil {
return
}
// Don't forget to set the content type, this will contain the boundary.
req.Header.Set("Content-Type", w.FormDataContentType())
// Submit the request
res, err := client.Do(req)
if err != nil {
return
}
// Check the response
if res.StatusCode != http.StatusOK {
err = fmt.Errorf("bad status: %s", res.Status)
}
return
}
func mustOpen(f string) *os.File {
r, err := os.Open(f)
if err != nil {
panic(err)
}
return r
}
Here's a function I've used that uses io.Pipe() to avoid reading in the entire file to memory or needing to manage any buffers. It handles only a single file, but could easily be extended to handle more by adding more parts within the goroutine. The happy path works well. The error paths have not hand much testing.
import (
"fmt"
"io"
"mime/multipart"
"net/http"
"os"
)
func UploadMultipartFile(client *http.Client, uri, key, path string) (*http.Response, error) {
body, writer := io.Pipe()
req, err := http.NewRequest(http.MethodPost, uri, body)
if err != nil {
return nil, err
}
mwriter := multipart.NewWriter(writer)
req.Header.Add("Content-Type", mwriter.FormDataContentType())
errchan := make(chan error)
go func() {
defer close(errchan)
defer writer.Close()
defer mwriter.Close()
w, err := mwriter.CreateFormFile(key, path)
if err != nil {
errchan <- err
return
}
in, err := os.Open(path)
if err != nil {
errchan <- err
return
}
defer in.Close()
if written, err := io.Copy(w, in); err != nil {
errchan <- fmt.Errorf("error copying %s (%d bytes written): %v", path, written, err)
return
}
if err := mwriter.Close(); err != nil {
errchan <- err
return
}
}()
resp, err := client.Do(req)
merr := <-errchan
if err != nil || merr != nil {
return resp, fmt.Errorf("http error: %v, multipart error: %v", err, merr)
}
return resp, nil
}
After having to decode the accepted answer for this question for use in my unit testing I finally ended up with the follow refactored code:
func createMultipartFormData(t *testing.T, fieldName, fileName string) (bytes.Buffer, *multipart.Writer) {
var b bytes.Buffer
var err error
w := multipart.NewWriter(&b)
var fw io.Writer
file := mustOpen(fileName)
if fw, err = w.CreateFormFile(fieldName, file.Name()); err != nil {
t.Errorf("Error creating writer: %v", err)
}
if _, err = io.Copy(fw, file); err != nil {
t.Errorf("Error with io.Copy: %v", err)
}
w.Close()
return b, w
}
func mustOpen(f string) *os.File {
r, err := os.Open(f)
if err != nil {
pwd, _ := os.Getwd()
fmt.Println("PWD: ", pwd)
panic(err)
}
return r
}
Now it should be pretty easy to use:
b, w := createMultipartFormData(t, "image","../luke.png")
req, err := http.NewRequest("POST", url, &b)
if err != nil {
return
}
// Don't forget to set the content type, this will contain the boundary.
req.Header.Set("Content-Type", w.FormDataContentType())
Here is an option that works for files or strings:
package main
import (
"bytes"
"io"
"mime/multipart"
"os"
"strings"
)
func createForm(form map[string]string) (string, io.Reader, error) {
body := new(bytes.Buffer)
mp := multipart.NewWriter(body)
defer mp.Close()
for key, val := range form {
if strings.HasPrefix(val, "#") {
val = val[1:]
file, err := os.Open(val)
if err != nil { return "", nil, err }
defer file.Close()
part, err := mp.CreateFormFile(key, val)
if err != nil { return "", nil, err }
io.Copy(part, file)
} else {
mp.WriteField(key, val)
}
}
return mp.FormDataContentType(), body, nil
}
Example:
package main
import "net/http"
func main() {
form := map[string]string{"image": "#IMAGEFILE", "key": "KEY"}
ct, body, err := createForm(form)
if err != nil {
panic(err)
}
http.Post("https://stackoverflow.com", ct, body)
}
https://golang.org/pkg/mime/multipart#Writer.WriteField
Send file from one service to another:
func UploadFile(network, uri string, f multipart.File, h *multipart.FileHeader) error {
buf := new(bytes.Buffer)
writer := multipart.NewWriter(buf)
part, err := writer.CreateFormFile("file", h.Filename)
if err != nil {
log.Println(err)
return err
}
b, err := ioutil.ReadAll(f)
if err != nil {
log.Println(err)
return err
}
part.Write(b)
writer.Close()
req, _ := http.NewRequest("POST", uri, buf)
req.Header.Add("Content-Type", writer.FormDataContentType())
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
b, _ = ioutil.ReadAll(resp.Body)
if resp.StatusCode >= 400 {
return errors.New(string(b))
}
return nil
}
To extend on #attila-o answer, here is the code I went with to perform a POST HTTP req in Go with:
1 file
configurable file name (f.Name() didn't work)
extra form fields.
Curl representation:
curl -X POST \
http://localhost:9091/storage/add \
-H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
-F owner=0xc916Cfe5c83dD4FC3c3B0Bf2ec2d4e401782875e \
-F password=$PWD \
-F file=#./internal/file_example_JPG_500kB.jpg
Go way:
client := &http.Client{
Timeout: time.Second * 10,
}
req, err := createStoragePostReq(cfg)
res, err := executeStoragePostReq(client, req)
func createStoragePostReq(cfg Config) (*http.Request, error) {
extraFields := map[string]string{
"owner": "0xc916cfe5c83dd4fc3c3b0bf2ec2d4e401782875e",
"password": "pwd",
}
url := fmt.Sprintf("http://localhost:%d%s", cfg.HttpServerConfig().Port(), lethstorage.AddRoute)
b, w, err := createMultipartFormData("file","./internal/file_example_JPG_500kB.jpg", "file_example_JPG_500kB.jpg", extraFields)
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", url, &b)
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", w.FormDataContentType())
return req, nil
}
func executeStoragePostReq(client *http.Client, req *http.Request) (lethstorage.AddRes, error) {
var addRes lethstorage.AddRes
res, err := client.Do(req)
if err != nil {
return addRes, err
}
defer res.Body.Close()
data, err := ioutil.ReadAll(res.Body)
if err != nil {
return addRes, err
}
err = json.Unmarshal(data, &addRes)
if err != nil {
return addRes, err
}
return addRes, nil
}
func createMultipartFormData(fileFieldName, filePath string, fileName string, extraFormFields map[string]string) (b bytes.Buffer, w *multipart.Writer, err error) {
w = multipart.NewWriter(&b)
var fw io.Writer
file, err := os.Open(filePath)
if fw, err = w.CreateFormFile(fileFieldName, fileName); err != nil {
return
}
if _, err = io.Copy(fw, file); err != nil {
return
}
for k, v := range extraFormFields {
w.WriteField(k, v)
}
w.Close()
return
}
I have found this tutorial very helpful to clarify my confusions about file uploading in Go.
Basically you upload the file via ajax using form-data on a client and use the following small snippet of Go code on the server:
file, handler, err := r.FormFile("img") // img is the key of the form-data
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
fmt.Println("File is good")
fmt.Println(handler.Filename)
fmt.Println()
fmt.Println(handler.Header)
f, err := os.OpenFile(handler.Filename, os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
fmt.Println(err)
return
}
defer f.Close()
io.Copy(f, file)
Here r is *http.Request. P.S. this just stores the file in the same folder and does not perform any security checks.