Related
I am currently in the process of creating a little Go App. Right now I am working on the DB part. The Library I use is this one: https://github.com/jackc/pgx
The problem I have is that every time I try to execute the database read, it tells me that my 'conn is busy'. I read about using a pgxpool instead of a single connection, but it still does not work. What am I doing wrong?
func (postgre *PostgreClient) read(query string) (pgx.Row, error) {
client, err := postgre.client.Acquire(context.TODO())
transaction, err := client.BeginTx(context.TODO(), pgx.TxOptions{})
if err != nil {
return nil, err
}
defer transaction.Rollback(context.TODO())
rows := transaction.QueryRow(context.TODO(), query)
if err != nil {
return nil, err
}
err = transaction.Commit(context.TODO())
return rows, err
}
Thanks in advance.
You have to scan the row before you commit the transaction.
If you want the handling of the transaction to remain within the function you can pass an interface that does the scanning also inside the function.
For example:
// implemented by *sql.Row & *sql.Rows
type Row interface {
Scan(dst ...interface{}) error
}
// implemented by your "models"
type RowScanner interface {
ScanRow(r Row) error
}
type User struct {
Id int
Email string
}
func (u *User) ScanRow(r Row) error {
return r.Scan(
&u.Id,
&u.Email,
)
}
func (postgre *PostgreClient) read(query string, rs RowScanner) (err error) {
conn, err := postgre.client.Acquire(context.TODO())
if err != nil {
return err
}
defer conn.Release()
tx, err := conn.BeginTx(context.TODO(), pgx.TxOptions{})
if err != nil {
return err
}
defer func() {
if err != nil {
tx.Rollback(context.TODO())
} else {
tx.Commit(context.TODO())
}
}()
row := tx.QueryRow(context.TODO(), query)
if err != nil {
return nil, err
}
return rs.ScanRow(row)
}
u := new(User)
if err := pg.read("select id, email from users limit 1", u); err != nil {
panic(err)
}
For scanning a list of models:
type UserList []*User
func (ul *UserList) ScanRow(r Row) error {
u := new(User)
if err := u.ScanRow(r); err != nil {
return err
}
*ul = append(*ul, u)
return nil
}
func (postgre *PostgreClient) list(query string, rs RowScanner) (err error) {
conn, err := postgre.client.Acquire(context.TODO())
if err != nil {
return err
}
defer conn.Release()
tx, err := conn.BeginTx(context.TODO(), pgx.TxOptions{})
if err != nil {
return err
}
defer func() {
if err != nil {
tx.Rollback(context.TODO())
} else {
tx.Commit(context.TODO())
}
}()
rows, err := tx.Query(context.TODO(), query)
if err != nil {
return err
}
defer rows.Close()
for rows.Next() {
if err := rs.ScanRow(rows); err != nil {
return err
}
}
return rows.Err()
}
ul := new(UserList)
if err := pg.list("select id, email from users", ul); err != nil {
panic(err)
}
I am going through a list of files and Unmarshalling the xml data in them into an array of structs rArray. I intend to process about 18000 files. When I get to about 1300 files processed, the program panics and says that too many files are open. If I limit the amount of files processed to a safe amount of 1000, the program does not crash.
As seen below, I am using ioutil.ReadFile to read the file data.
for _, f := range files {
func() {
data, err := ioutil.ReadFile("./" + recordDir + "/" + f.Name())
if err != nil {
fmt.Println("error reading %v", err)
return
} else {
if (strings.Contains(filepath.Ext(f.Name()), "xml")) {
//unmarshal data and put into struct array
err = xml.Unmarshal([]byte(data), &rArray[a])
if err != nil {
fmt.Println("error decoding %v: %v",f.Name(), err)
return
}
}
}
}()
}
I am not sure if Go is using too many file descriptors or not closing the files fast enough.
After reading https://groups.google.com/forum/#!topic/golang-nuts/7yXXjgcOikM and viewing the ioutil source in http://golang.org/src/pkg/io/ioutil/ioutil.go, the code for ioutil.ReadFile shows that it uses defer to close the file. defer runs when calling function is returned and ReadFile() is the calling function. Am I correct in this understanding?
I also tried wrapping the ioutil.ReadFile part of my code in a function, but it makes no difference.
My ulimit is set to unlimited.
UPDATE:
I believe that the error of too many files is actually occurring during my Unzip function.
func Unzip(src, dest string) error {
r, err := zip.OpenReader(src)
if err != nil {
return err
}
for _, f := range r.File {
rc, err := f.Open()
if err != nil {
panic(err)
}
path := filepath.Join(dest, f.Name)
if f.FileInfo().IsDir() {
os.MkdirAll(path, f.Mode())
} else {
f, err := os.OpenFile(
path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, f.Mode())
if err != nil {
panic(err)
}
_, err = io.Copy(f, rc)
if err != nil {
panic(err)
}
f.Close()
}
rc.Close()
}
r.Close()
return nil
}
I initially got the Unzip function from https://gist.github.com/hnaohiro/4572580, but upon further inspection, the use of defer in the gist author's function seemed wrong as the file would only be closed after the Unzip() function returned which is too late becuase then 18000 file descriptors will be open. ;)
I replaced the deferred Closes with explicit Close() as shown above, but am still getting the same "too many open files" error. Is there a problem with my modified Unzip function?
UPDATE # 2
Oops, I was running this on Heroku and was pushing to the wrong app with my changes this entire time. Lesson learned: verify target app in heroku toolbelt.
Unzip code from https://gist.github.com/hnaohiro/4572580 does not work as it does not close files until all files processed.
My unzip code with explicit close above works and so does the defer version in #peterSO's answer.
I would modify the Unzip function from https://gist.github.com/hnaohiro/4572580 to the following:
package main
import (
"archive/zip"
"io"
"log"
"os"
"path/filepath"
)
func unzipFile(f *zip.File, dest string) error {
rc, err := f.Open()
if err != nil {
return err
}
defer rc.Close()
path := filepath.Join(dest, f.Name)
if f.FileInfo().IsDir() {
err := os.MkdirAll(path, f.Mode())
if err != nil {
return err
}
} else {
f, err := os.OpenFile(
path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, f.Mode())
if err != nil {
return err
}
defer f.Close()
_, err = io.Copy(f, rc)
if err != nil {
return err
}
}
return nil
}
func Unzip(src, dest string) error {
r, err := zip.OpenReader(src)
if err != nil {
return err
}
defer r.Close()
for _, f := range r.File {
err := unzipFile(f, dest)
if err != nil {
return err
}
}
return nil
}
func main() {
err := Unzip("./sample.zip", "./out")
if err != nil {
log.Fatal(err)
}
}
I would like to run WMI queries from Go. There are ways to call DLL functions from Go. My understanding is that there must be some DLL somewhere which, with the correct call, will return some data I can parse and use. I'd prefer to avoid calling into C or C++, especially since I would guess those are wrappers over the Windows API itself.
I've examined the output of dumpbin.exe /exports c:\windows\system32\wmi.dll, and the following entry looks promising:
WmiQueryAllDataA (forwarded to wmiclnt.WmiQueryAllDataA)
However I'm not sure what to do from here. What arguments does this function take? What does it return? Searching for WmiQueryAllDataA is not helpful. And that name only appears in a comment of c:\program files (x86)\windows kits\8.1\include\shared\wmistr.h, but with no function signature.
Are there better methods? Is there another DLL? Am I missing something? Should I just use a C wrapper?
Running a WMI query in Linqpad with .NET Reflector shows the use of WmiNetUtilsHelper:ExecQueryWmi (and a _f version), but neither have a viewable implementation.
Update: use the github.com/StackExchange/wmi package which uses the solution in the accepted answer.
Welcome to the wonderful world of COM, Object Oriented Programming in C from when C++ was "a young upstart".
On github mattn has thrown together a little wrapper in Go, which I used to throw together a quick example program. "This repository was created for experimentation and should be considered unstable." instills all sorts of confidence.
I'm leaving out a lot of error checking. Trust me when I say, you'll want to add it back.
package main
import (
"github.com/mattn/go-ole"
"github.com/mattn/go-ole/oleutil"
)
func main() {
// init COM, oh yeah
ole.CoInitialize(0)
defer ole.CoUninitialize()
unknown, _ := oleutil.CreateObject("WbemScripting.SWbemLocator")
defer unknown.Release()
wmi, _ := unknown.QueryInterface(ole.IID_IDispatch)
defer wmi.Release()
// service is a SWbemServices
serviceRaw, _ := oleutil.CallMethod(wmi, "ConnectServer")
service := serviceRaw.ToIDispatch()
defer service.Release()
// result is a SWBemObjectSet
resultRaw, _ := oleutil.CallMethod(service, "ExecQuery", "SELECT * FROM Win32_Process")
result := resultRaw.ToIDispatch()
defer result.Release()
countVar, _ := oleutil.GetProperty(result, "Count")
count := int(countVar.Val)
for i :=0; i < count; i++ {
// item is a SWbemObject, but really a Win32_Process
itemRaw, _ := oleutil.CallMethod(result, "ItemIndex", i)
item := itemRaw.ToIDispatch()
defer item.Release()
asString, _ := oleutil.GetProperty(item, "Name")
println(asString.ToString())
}
}
The real meat is the call to ExecQuery, I happen to grab Win32_Process from the available classes because it's easy to understand and print.
On my machine, this prints:
System Idle Process
System
smss.exe
csrss.exe
wininit.exe
services.exe
lsass.exe
svchost.exe
svchost.exe
atiesrxx.exe
svchost.exe
svchost.exe
svchost.exe
svchost.exe
svchost.exe
spoolsv.exe
svchost.exe
AppleOSSMgr.exe
AppleTimeSrv.exe
... and so on
go.exe
main.exe
I'm not running it elevated or with UAC disabled, but some WMI providers are gonna require a privileged user.
I'm also not 100% that this won't leak a little, you'll want to dig into that. COM objects are reference counted, so defer should be a pretty good fit there (provided the method isn't crazy long running) but go-ole may have some magic inside I didn't notice.
I'm commenting over a year later, but there is a solution here on github (and posted below for posterity).
// +build windows
/*
Package wmi provides a WQL interface for WMI on Windows.
Example code to print names of running processes:
type Win32_Process struct {
Name string
}
func main() {
var dst []Win32_Process
q := wmi.CreateQuery(&dst, "")
err := wmi.Query(q, &dst)
if err != nil {
log.Fatal(err)
}
for i, v := range dst {
println(i, v.Name)
}
}
*/
package wmi
import (
"bytes"
"errors"
"fmt"
"log"
"os"
"reflect"
"runtime"
"strconv"
"strings"
"sync"
"time"
"github.com/mattn/go-ole"
"github.com/mattn/go-ole/oleutil"
)
var l = log.New(os.Stdout, "", log.LstdFlags)
var (
ErrInvalidEntityType = errors.New("wmi: invalid entity type")
lock sync.Mutex
)
// QueryNamespace invokes Query with the given namespace on the local machine.
func QueryNamespace(query string, dst interface{}, namespace string) error {
return Query(query, dst, nil, namespace)
}
// Query runs the WQL query and appends the values to dst.
//
// dst must have type *[]S or *[]*S, for some struct type S. Fields selected in
// the query must have the same name in dst. Supported types are all signed and
// unsigned integers, time.Time, string, bool, or a pointer to one of those.
// Array types are not supported.
//
// By default, the local machine and default namespace are used. These can be
// changed using connectServerArgs. See
// http://msdn.microsoft.com/en-us/library/aa393720.aspx for details.
func Query(query string, dst interface{}, connectServerArgs ...interface{}) error {
dv := reflect.ValueOf(dst)
if dv.Kind() != reflect.Ptr || dv.IsNil() {
return ErrInvalidEntityType
}
dv = dv.Elem()
mat, elemType := checkMultiArg(dv)
if mat == multiArgTypeInvalid {
return ErrInvalidEntityType
}
lock.Lock()
defer lock.Unlock()
runtime.LockOSThread()
defer runtime.UnlockOSThread()
err := ole.CoInitializeEx(0, ole.COINIT_MULTITHREADED)
if err != nil {
oleerr := err.(*ole.OleError)
// S_FALSE = 0x00000001 // CoInitializeEx was already called on this thread
if oleerr.Code() != ole.S_OK && oleerr.Code() != 0x00000001 {
return err
}
} else {
// Only invoke CoUninitialize if the thread was not initizlied before.
// This will allow other go packages based on go-ole play along
// with this library.
defer ole.CoUninitialize()
}
unknown, err := oleutil.CreateObject("WbemScripting.SWbemLocator")
if err != nil {
return err
}
defer unknown.Release()
wmi, err := unknown.QueryInterface(ole.IID_IDispatch)
if err != nil {
return err
}
defer wmi.Release()
// service is a SWbemServices
serviceRaw, err := oleutil.CallMethod(wmi, "ConnectServer", connectServerArgs...)
if err != nil {
return err
}
service := serviceRaw.ToIDispatch()
defer serviceRaw.Clear()
// result is a SWBemObjectSet
resultRaw, err := oleutil.CallMethod(service, "ExecQuery", query)
if err != nil {
return err
}
result := resultRaw.ToIDispatch()
defer resultRaw.Clear()
count, err := oleInt64(result, "Count")
if err != nil {
return err
}
// Initialize a slice with Count capacity
dv.Set(reflect.MakeSlice(dv.Type(), 0, int(count)))
var errFieldMismatch error
for i := int64(0); i < count; i++ {
err := func() error {
// item is a SWbemObject, but really a Win32_Process
itemRaw, err := oleutil.CallMethod(result, "ItemIndex", i)
if err != nil {
return err
}
item := itemRaw.ToIDispatch()
defer itemRaw.Clear()
ev := reflect.New(elemType)
if err = loadEntity(ev.Interface(), item); err != nil {
if _, ok := err.(*ErrFieldMismatch); ok {
// We continue loading entities even in the face of field mismatch errors.
// If we encounter any other error, that other error is returned. Otherwise,
// an ErrFieldMismatch is returned.
errFieldMismatch = err
} else {
return err
}
}
if mat != multiArgTypeStructPtr {
ev = ev.Elem()
}
dv.Set(reflect.Append(dv, ev))
return nil
}()
if err != nil {
return err
}
}
return errFieldMismatch
}
// ErrFieldMismatch is returned when a field is to be loaded into a different
// type than the one it was stored from, or when a field is missing or
// unexported in the destination struct.
// StructType is the type of the struct pointed to by the destination argument.
type ErrFieldMismatch struct {
StructType reflect.Type
FieldName string
Reason string
}
func (e *ErrFieldMismatch) Error() string {
return fmt.Sprintf("wmi: cannot load field %q into a %q: %s",
e.FieldName, e.StructType, e.Reason)
}
var timeType = reflect.TypeOf(time.Time{})
// loadEntity loads a SWbemObject into a struct pointer.
func loadEntity(dst interface{}, src *ole.IDispatch) (errFieldMismatch error) {
v := reflect.ValueOf(dst).Elem()
for i := 0; i < v.NumField(); i++ {
f := v.Field(i)
isPtr := f.Kind() == reflect.Ptr
if isPtr {
ptr := reflect.New(f.Type().Elem())
f.Set(ptr)
f = f.Elem()
}
n := v.Type().Field(i).Name
if !f.CanSet() {
return &ErrFieldMismatch{
StructType: f.Type(),
FieldName: n,
Reason: "CanSet() is false",
}
}
prop, err := oleutil.GetProperty(src, n)
if err != nil {
errFieldMismatch = &ErrFieldMismatch{
StructType: f.Type(),
FieldName: n,
Reason: "no such struct field",
}
continue
}
defer prop.Clear()
switch val := prop.Value().(type) {
case int, int64:
var v int64
switch val := val.(type) {
case int:
v = int64(val)
case int64:
v = val
default:
panic("unexpected type")
}
switch f.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
f.SetInt(v)
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
f.SetUint(uint64(v))
default:
return &ErrFieldMismatch{
StructType: f.Type(),
FieldName: n,
Reason: "not an integer class",
}
}
case string:
iv, err := strconv.ParseInt(val, 10, 64)
switch f.Kind() {
case reflect.String:
f.SetString(val)
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
if err != nil {
return err
}
f.SetInt(iv)
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
if err != nil {
return err
}
f.SetUint(uint64(iv))
case reflect.Struct:
switch f.Type() {
case timeType:
if len(val) == 25 {
mins, err := strconv.Atoi(val[22:])
if err != nil {
return err
}
val = val[:22] + fmt.Sprintf("%02d%02d", mins/60, mins%60)
}
t, err := time.Parse("20060102150405.000000-0700", val)
if err != nil {
return err
}
f.Set(reflect.ValueOf(t))
}
}
case bool:
switch f.Kind() {
case reflect.Bool:
f.SetBool(val)
default:
return &ErrFieldMismatch{
StructType: f.Type(),
FieldName: n,
Reason: "not a bool",
}
}
default:
typeof := reflect.TypeOf(val)
if isPtr && typeof == nil {
break
}
return &ErrFieldMismatch{
StructType: f.Type(),
FieldName: n,
Reason: fmt.Sprintf("unsupported type (%T)", val),
}
}
}
return errFieldMismatch
}
type multiArgType int
const (
multiArgTypeInvalid multiArgType = iota
multiArgTypeStruct
multiArgTypeStructPtr
)
// checkMultiArg checks that v has type []S, []*S for some struct type S.
//
// It returns what category the slice's elements are, and the reflect.Type
// that represents S.
func checkMultiArg(v reflect.Value) (m multiArgType, elemType reflect.Type) {
if v.Kind() != reflect.Slice {
return multiArgTypeInvalid, nil
}
elemType = v.Type().Elem()
switch elemType.Kind() {
case reflect.Struct:
return multiArgTypeStruct, elemType
case reflect.Ptr:
elemType = elemType.Elem()
if elemType.Kind() == reflect.Struct {
return multiArgTypeStructPtr, elemType
}
}
return multiArgTypeInvalid, nil
}
func oleInt64(item *ole.IDispatch, prop string) (int64, error) {
v, err := oleutil.GetProperty(item, prop)
if err != nil {
return 0, err
}
defer v.Clear()
i := int64(v.Val)
return i, nil
}
// CreateQuery returns a WQL query string that queries all columns of src. where
// is an optional string that is appended to the query, to be used with WHERE
// clauses. In such a case, the "WHERE" string should appear at the beginning.
func CreateQuery(src interface{}, where string) string {
var b bytes.Buffer
b.WriteString("SELECT ")
s := reflect.Indirect(reflect.ValueOf(src))
t := s.Type()
if s.Kind() == reflect.Slice {
t = t.Elem()
}
if t.Kind() != reflect.Struct {
return ""
}
var fields []string
for i := 0; i < t.NumField(); i++ {
fields = append(fields, t.Field(i).Name)
}
b.WriteString(strings.Join(fields, ", "))
b.WriteString(" FROM ")
b.WriteString(t.Name())
b.WriteString(" " + where)
return b.String()
}
To access the winmgmts object or a namespace (which is the same), you can use the code below. Basically, you need to specify the namespace as parameter, which is not documented properly in go-ole.
In the code below, you can also see how to access a class within this namespace and execute a method.
package main
import (
"log"
"github.com/go-ole/go-ole"
"github.com/go-ole/go-ole/oleutil"
)
func main() {
ole.CoInitializeEx(0, ole.COINIT_MULTITHREADED)
defer ole.CoUninitialize()
unknown, err := oleutil.CreateObject("WbemScripting.SWbemLocator")
if err != nil {
log.Panic(err)
}
defer unknown.Release()
wmi, err := unknown.QueryInterface(ole.IID_IDispatch)
if err != nil {
log.Panic(err)
}
defer wmi.Release()
// Connect to namespace
// root/PanasonicPC = winmgmts:\\.\root\PanasonicPC
serviceRaw, err := oleutil.CallMethod(wmi, "ConnectServer", nil, "root/PanasonicPC")
if err != nil {
log.Panic(err)
}
service := serviceRaw.ToIDispatch()
defer serviceRaw.Clear()
// Get class
setBiosRaw, err := oleutil.CallMethod(service, "Get", "SetBIOS4Conf")
if err != nil {
log.Panic(err)
}
setBios := setBiosRaw.ToIDispatch()
defer setBiosRaw.Clear()
// Run method
resultRaw, err := oleutil.CallMethod(setBios, "AccessAuthorization", "letmein")
resultVal := resultRaw.Value().(int32)
log.Println("Return Code:", resultVal)
}
import(
"os/exec"
)
func (lcu *LCU) GrabToken() {
cmd := exec.Command("powershell", "$cmdline = Get-WmiObject -Class Win32_Process")
out, err := cmd.CombinedOutput()
if err != nil {
fmt.Println(err)
}
outstr := string(out)
}
I'm trying to upload images from my computer to a website using go. Usually, I use a bash script that sends a file and a key to the server:
curl -F "image"=#"IMAGEFILE" -F "key"="KEY" URL
it works fine, but I'm trying to convert this request into my golang program.
http://matt.aimonetti.net/posts/2013/07/01/golang-multipart-file-upload-example/
I tried this link and many others, but, for each code that I try, the response from the server is "no image sent", and I've no idea why. If someone knows what's happening with the example above.
Here's some sample code.
In short, you'll need to use the mime/multipart package to build the form.
package main
import (
"bytes"
"fmt"
"io"
"mime/multipart"
"net/http"
"net/http/httptest"
"net/http/httputil"
"os"
"strings"
)
func main() {
var client *http.Client
var remoteURL string
{
//setup a mocked http client.
ts := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
b, err := httputil.DumpRequest(r, true)
if err != nil {
panic(err)
}
fmt.Printf("%s", b)
}))
defer ts.Close()
client = ts.Client()
remoteURL = ts.URL
}
//prepare the reader instances to encode
values := map[string]io.Reader{
"file": mustOpen("main.go"), // lets assume its this file
"other": strings.NewReader("hello world!"),
}
err := Upload(client, remoteURL, values)
if err != nil {
panic(err)
}
}
func Upload(client *http.Client, url string, values map[string]io.Reader) (err error) {
// Prepare a form that you will submit to that URL.
var b bytes.Buffer
w := multipart.NewWriter(&b)
for key, r := range values {
var fw io.Writer
if x, ok := r.(io.Closer); ok {
defer x.Close()
}
// Add an image file
if x, ok := r.(*os.File); ok {
if fw, err = w.CreateFormFile(key, x.Name()); err != nil {
return
}
} else {
// Add other fields
if fw, err = w.CreateFormField(key); err != nil {
return
}
}
if _, err = io.Copy(fw, r); err != nil {
return err
}
}
// Don't forget to close the multipart writer.
// If you don't close it, your request will be missing the terminating boundary.
w.Close()
// Now that you have a form, you can submit it to your handler.
req, err := http.NewRequest("POST", url, &b)
if err != nil {
return
}
// Don't forget to set the content type, this will contain the boundary.
req.Header.Set("Content-Type", w.FormDataContentType())
// Submit the request
res, err := client.Do(req)
if err != nil {
return
}
// Check the response
if res.StatusCode != http.StatusOK {
err = fmt.Errorf("bad status: %s", res.Status)
}
return
}
func mustOpen(f string) *os.File {
r, err := os.Open(f)
if err != nil {
panic(err)
}
return r
}
Here's a function I've used that uses io.Pipe() to avoid reading in the entire file to memory or needing to manage any buffers. It handles only a single file, but could easily be extended to handle more by adding more parts within the goroutine. The happy path works well. The error paths have not hand much testing.
import (
"fmt"
"io"
"mime/multipart"
"net/http"
"os"
)
func UploadMultipartFile(client *http.Client, uri, key, path string) (*http.Response, error) {
body, writer := io.Pipe()
req, err := http.NewRequest(http.MethodPost, uri, body)
if err != nil {
return nil, err
}
mwriter := multipart.NewWriter(writer)
req.Header.Add("Content-Type", mwriter.FormDataContentType())
errchan := make(chan error)
go func() {
defer close(errchan)
defer writer.Close()
defer mwriter.Close()
w, err := mwriter.CreateFormFile(key, path)
if err != nil {
errchan <- err
return
}
in, err := os.Open(path)
if err != nil {
errchan <- err
return
}
defer in.Close()
if written, err := io.Copy(w, in); err != nil {
errchan <- fmt.Errorf("error copying %s (%d bytes written): %v", path, written, err)
return
}
if err := mwriter.Close(); err != nil {
errchan <- err
return
}
}()
resp, err := client.Do(req)
merr := <-errchan
if err != nil || merr != nil {
return resp, fmt.Errorf("http error: %v, multipart error: %v", err, merr)
}
return resp, nil
}
After having to decode the accepted answer for this question for use in my unit testing I finally ended up with the follow refactored code:
func createMultipartFormData(t *testing.T, fieldName, fileName string) (bytes.Buffer, *multipart.Writer) {
var b bytes.Buffer
var err error
w := multipart.NewWriter(&b)
var fw io.Writer
file := mustOpen(fileName)
if fw, err = w.CreateFormFile(fieldName, file.Name()); err != nil {
t.Errorf("Error creating writer: %v", err)
}
if _, err = io.Copy(fw, file); err != nil {
t.Errorf("Error with io.Copy: %v", err)
}
w.Close()
return b, w
}
func mustOpen(f string) *os.File {
r, err := os.Open(f)
if err != nil {
pwd, _ := os.Getwd()
fmt.Println("PWD: ", pwd)
panic(err)
}
return r
}
Now it should be pretty easy to use:
b, w := createMultipartFormData(t, "image","../luke.png")
req, err := http.NewRequest("POST", url, &b)
if err != nil {
return
}
// Don't forget to set the content type, this will contain the boundary.
req.Header.Set("Content-Type", w.FormDataContentType())
Here is an option that works for files or strings:
package main
import (
"bytes"
"io"
"mime/multipart"
"os"
"strings"
)
func createForm(form map[string]string) (string, io.Reader, error) {
body := new(bytes.Buffer)
mp := multipart.NewWriter(body)
defer mp.Close()
for key, val := range form {
if strings.HasPrefix(val, "#") {
val = val[1:]
file, err := os.Open(val)
if err != nil { return "", nil, err }
defer file.Close()
part, err := mp.CreateFormFile(key, val)
if err != nil { return "", nil, err }
io.Copy(part, file)
} else {
mp.WriteField(key, val)
}
}
return mp.FormDataContentType(), body, nil
}
Example:
package main
import "net/http"
func main() {
form := map[string]string{"image": "#IMAGEFILE", "key": "KEY"}
ct, body, err := createForm(form)
if err != nil {
panic(err)
}
http.Post("https://stackoverflow.com", ct, body)
}
https://golang.org/pkg/mime/multipart#Writer.WriteField
Send file from one service to another:
func UploadFile(network, uri string, f multipart.File, h *multipart.FileHeader) error {
buf := new(bytes.Buffer)
writer := multipart.NewWriter(buf)
part, err := writer.CreateFormFile("file", h.Filename)
if err != nil {
log.Println(err)
return err
}
b, err := ioutil.ReadAll(f)
if err != nil {
log.Println(err)
return err
}
part.Write(b)
writer.Close()
req, _ := http.NewRequest("POST", uri, buf)
req.Header.Add("Content-Type", writer.FormDataContentType())
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
b, _ = ioutil.ReadAll(resp.Body)
if resp.StatusCode >= 400 {
return errors.New(string(b))
}
return nil
}
To extend on #attila-o answer, here is the code I went with to perform a POST HTTP req in Go with:
1 file
configurable file name (f.Name() didn't work)
extra form fields.
Curl representation:
curl -X POST \
http://localhost:9091/storage/add \
-H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
-F owner=0xc916Cfe5c83dD4FC3c3B0Bf2ec2d4e401782875e \
-F password=$PWD \
-F file=#./internal/file_example_JPG_500kB.jpg
Go way:
client := &http.Client{
Timeout: time.Second * 10,
}
req, err := createStoragePostReq(cfg)
res, err := executeStoragePostReq(client, req)
func createStoragePostReq(cfg Config) (*http.Request, error) {
extraFields := map[string]string{
"owner": "0xc916cfe5c83dd4fc3c3b0bf2ec2d4e401782875e",
"password": "pwd",
}
url := fmt.Sprintf("http://localhost:%d%s", cfg.HttpServerConfig().Port(), lethstorage.AddRoute)
b, w, err := createMultipartFormData("file","./internal/file_example_JPG_500kB.jpg", "file_example_JPG_500kB.jpg", extraFields)
if err != nil {
return nil, err
}
req, err := http.NewRequest("POST", url, &b)
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", w.FormDataContentType())
return req, nil
}
func executeStoragePostReq(client *http.Client, req *http.Request) (lethstorage.AddRes, error) {
var addRes lethstorage.AddRes
res, err := client.Do(req)
if err != nil {
return addRes, err
}
defer res.Body.Close()
data, err := ioutil.ReadAll(res.Body)
if err != nil {
return addRes, err
}
err = json.Unmarshal(data, &addRes)
if err != nil {
return addRes, err
}
return addRes, nil
}
func createMultipartFormData(fileFieldName, filePath string, fileName string, extraFormFields map[string]string) (b bytes.Buffer, w *multipart.Writer, err error) {
w = multipart.NewWriter(&b)
var fw io.Writer
file, err := os.Open(filePath)
if fw, err = w.CreateFormFile(fileFieldName, fileName); err != nil {
return
}
if _, err = io.Copy(fw, file); err != nil {
return
}
for k, v := range extraFormFields {
w.WriteField(k, v)
}
w.Close()
return
}
I have found this tutorial very helpful to clarify my confusions about file uploading in Go.
Basically you upload the file via ajax using form-data on a client and use the following small snippet of Go code on the server:
file, handler, err := r.FormFile("img") // img is the key of the form-data
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
fmt.Println("File is good")
fmt.Println(handler.Filename)
fmt.Println()
fmt.Println(handler.Header)
f, err := os.OpenFile(handler.Filename, os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
fmt.Println(err)
return
}
defer f.Close()
io.Copy(f, file)
Here r is *http.Request. P.S. this just stores the file in the same folder and does not perform any security checks.
I'm new to Go, and can't figure out how to use the compress/gzip package to my advantage. Basically, I just want to write something to a file, gzip it and read it directly from the zipped format through another script. I would really appreciate if someone could give me an example on how to do this.
All the compress packages implement the same interface. You would use something like this to compress:
var b bytes.Buffer
w := gzip.NewWriter(&b)
w.Write([]byte("hello, world\n"))
w.Close()
And this to unpack:
r, err := gzip.NewReader(&b)
io.Copy(os.Stdout, r)
r.Close()
Pretty much the same answer as Laurent, but with the file io:
import (
"bytes"
"compress/gzip"
"io/ioutil"
)
// ...
var b bytes.Buffer
w := gzip.NewWriter(&b)
w.Write([]byte("hello, world\n"))
w.Close() // You must close this first to flush the bytes to the buffer.
err := ioutil.WriteFile("hello_world.txt.gz", b.Bytes(), 0666)
For the Read part, something like the useful ioutil.ReadFile for .gz files could be :
func ReadGzFile(filename string) ([]byte, error) {
fi, err := os.Open(filename)
if err != nil {
return nil, err
}
defer fi.Close()
fz, err := gzip.NewReader(fi)
if err != nil {
return nil, err
}
defer fz.Close()
s, err := ioutil.ReadAll(fz)
if err != nil {
return nil, err
}
return s, nil
}
Here the func for unpack gzip file to destination file:
func UnpackGzipFile(gzFilePath, dstFilePath string) (int64, error) {
gzFile, err := os.Open(gzFilePath)
if err != nil {
return 0, fmt.Errorf("open file %q to unpack: %w", gzFilePath, err)
}
dstFile, err := os.OpenFile(dstFilePath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0660)
if err != nil {
return 0, fmt.Errorf("create destination file %q to unpack: %w", dstFilePath, err)
}
defer dstFile.Close()
ioReader, ioWriter := io.Pipe()
defer ioReader.Close()
go func() { // goroutine leak is possible here
gzReader, _ := gzip.NewReader(gzFile)
// it is important to close the writer or reading from the other end of the
// pipe or io.copy() will never finish
defer func(){
gzFile.Close()
gzReader.Close()
ioWriter.Close()
}()
io.Copy(ioWriter, gzReader)
}()
written, err := io.Copy(dstFile, ioReader)
if err != nil {
return 0, err // goroutine leak is possible here
}
return written, nil
}
I decided to combine ideas from others answers and just provide a full example program. Obviously there are many different ways to do the same thing. This is just one way:
package main
import (
"compress/gzip"
"fmt"
"io/ioutil"
"os"
)
var zipFile = "zipfile.gz"
func main() {
writeZip()
readZip()
}
func writeZip() {
handle, err := openFile(zipFile)
if err != nil {
fmt.Println("[ERROR] Opening file:", err)
}
zipWriter, err := gzip.NewWriterLevel(handle, 9)
if err != nil {
fmt.Println("[ERROR] New gzip writer:", err)
}
numberOfBytesWritten, err := zipWriter.Write([]byte("Hello, World!\n"))
if err != nil {
fmt.Println("[ERROR] Writing:", err)
}
err = zipWriter.Close()
if err != nil {
fmt.Println("[ERROR] Closing zip writer:", err)
}
fmt.Println("[INFO] Number of bytes written:", numberOfBytesWritten)
closeFile(handle)
}
func readZip() {
handle, err := openFile(zipFile)
if err != nil {
fmt.Println("[ERROR] Opening file:", err)
}
zipReader, err := gzip.NewReader(handle)
if err != nil {
fmt.Println("[ERROR] New gzip reader:", err)
}
defer zipReader.Close()
fileContents, err := ioutil.ReadAll(zipReader)
if err != nil {
fmt.Println("[ERROR] ReadAll:", err)
}
fmt.Printf("[INFO] Uncompressed contents: %s\n", fileContents)
// ** Another way of reading the file **
//
// fileInfo, _ := handle.Stat()
// fileContents := make([]byte, fileInfo.Size())
// bytesRead, err := zipReader.Read(fileContents)
// if err != nil {
// fmt.Println("[ERROR] Reading gzip file:", err)
// }
// fmt.Println("[INFO] Number of bytes read from the file:", bytesRead)
closeFile(handle)
}
func openFile(fileToOpen string) (*os.File, error) {
return os.OpenFile(fileToOpen, openFileOptions, openFilePermissions)
}
func closeFile(handle *os.File) {
if handle == nil {
return
}
err := handle.Close()
if err != nil {
fmt.Println("[ERROR] Closing file:", err)
}
}
const openFileOptions int = os.O_CREATE | os.O_RDWR
const openFilePermissions os.FileMode = 0660
Having a full example like this should be helpful for future reference.
To compress any Go object of interface type as input
func compress(obj interface{}) ([]byte, error) {
var b bytes.Buffer
objBytes, err := json.Marshal(obj)
if err != nil {
return nil, err
}
gz := gzip.NewWriter(&b)
defer gz.Close() //NOT SUFFICIENT, DON'T DEFER WRITER OBJECTS
if _, err := gz.Write(objBytes); err != nil {
return nil, err
}
// NEED TO CLOSE EXPLICITLY
if err := gz.Close(); err != nil {
return nil, err
}
return b.Bytes(), nil
}
To decompress the same,
func decompress(obj []byte) ([]byte, error) {
r, err := gzip.NewReader(bytes.NewReader(obj))
if err != nil {
return nil, err
}
defer r.Close()
res, err := ioutil.ReadAll(r)
if err != nil {
return nil, err
}
return res, nil
}
Note, ioutil.ReadAll(r) returns io.EOF or io.ErrUnexpectedEOF if you do not close the Writer object after writing. I assumed defer on Close() would close the object properly, but it won't. Don't defer writer objects.