Golang file upload to s3 using OS Open - file-upload

I am trying to upload an Image to my s3 account using Golang and the amazon s3 api . I can get the imagine uploaded if I hard code the direct path such as
file, err := os.Open("/Users/JohnSmith/Documents/pictures/cars.jpg")
defer file.Close()
if err != nil {
fmt.Printf("err opening file: %s", err)
}
if I hard code the file path like that then the picture will be uploaded to my s3 account . However that approach is not good as I can't obviously hard code the direct image path to every image that I want to upload . My question is how can I upload images without having to Hardcode the path . This will be apart of an API where users will upload images so I clearly can not have a hard coded path . This is my code first the HTML
<form method="post" enctype="multipart/form-data" action="profile_image">
<h2>Image Upload</h2>
<p><input type="file" name="file" id="file"/> </p>
<p> <input type="submit" value="Upload Image"></p>
</form>
then this is my HTTP Post function method
func UploadProfile(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
var resultt string
resultt = "Hi"
sess, _ := session.NewSession(&aws.Config{
Region: aws.String("us-west-2"),
Credentials: credentials.NewStaticCredentials(aws_access_key_id,aws_secret_access_key, ""),
})
svc := s3.New(sess)
file, err := os.Open("Users/JohnSmith/Documents/pictures/cars.jpg")
defer file.Close()
if err != nil {
fmt.Printf("err opening file: %s", err)
}
fileInfo, _ := file.Stat()
size := fileInfo.Size()
buffer := make([]byte, size) // read file content to buffer
file.Read(buffer)
fileBytes := bytes.NewReader(buffer)
fileType := http.DetectContentType(buffer)
path := file.Name()
params := &s3.PutObjectInput{
Bucket: aws.String("my-bucket"),
Key: aws.String(path),
Body: fileBytes,
ContentLength: aws.Int64(size),
ContentType: aws.String(fileType),
}
resp, err := svc.PutObject(params)
if err != nil {
fmt.Printf("bad response: %s", err)
}
fmt.Printf("response %s", awsutil.StringValue(resp))
}
That is my full code above however when I try to do something such as
file, err := os.Open("file")
defer file.Close()
if err != nil {
fmt.Printf("err opening file: %s", err)
}
I get the following error
http: panic serving [::1]:55454: runtime error: invalid memory address or nil pointer dereference
goroutine 7 [running]:
err opening file: open file: no such file or directorynet/http.(*conn).serve.func1(0xc420076e80)
I can't use absolute path (filepath.Abs()) because some of the files will be outside of the GoPath and as stated other users will be uploading. Is there anyway that I can get a relative path ..

After POST to your API, images are temporarily saved in a OS's temp directory (different for different OS's) by default. To get this directory you can use, for example:
func GetTempLoc(filename string) string {
return strings.TrimRight(os.TempDir(), "/") + "/" + filename
}
Where:
filename is a header.Filename, i.e. file name received in your POST request. In Gin-Gonic framework you get it in your request handler as:
file, header, err := c.Request.FormFile("file")
if err != nil {
return out, err
}
defer file.Close()
Example: https://github.com/gin-gonic/gin#another-example-upload-file.
I'm sure in your framework there will be an analogue.
os.TempDir() is a function go give you a temp folder (details: https://golang.org/pkg/os/#TempDir).
TrimRight is used to ensure result of os.TempDir is consistent on different OSs
And then you use it as
file, err := os.Open(GetTempLoc(fileName))
...

Related

Can't do a simple Request-reply with RabbitMQ (RPC) and Apache Camel - not working

I hope you are well! First, I am new to the EIP world. I am trying to do a simple request reply with:
A Golang rabbitMQ client
An apache Camel route in Kotlin acting as a RabbitMQ server
I have tried to read all the docs I could and search for answers but I could't find nothing. I am basically desperate. Mainly I saw this and nothing has worked yet.
My goal is to do a sync request-reply as the image.
My Golang client looks like this:
func (r *RabbitMQConn) GetQueue(name string) *amqp.Queue {
ch := r.GetChannel()
defer ch.Close()
q, err := ch.QueueDeclare(
name,
false,
false,
true,
false,
nil,
)
if err != nil {
panic(err)
}
return &q
}
func (r *RabbitMQConn) PublishAndWait(routingKey string, correlationId string, event domain.SyncEventExtSend) (domain.SyncEventExtReceive, error) {
message, err := json.Marshal(event)
if err != nil {
return domain.SyncEventExtReceive{}, apperrors.ErrInternal
}
ch := r.GetChannel()
defer ch.Close()
q := r.GetQueue("response")
h, err := ch.Consume(
q.Name,
"",
true,
false,
false,
false,
nil,
)
if err != nil {
return domain.SyncEventExtReceive{}, err
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
err = ch.PublishWithContext(
ctx,
"",
routingKey,
false,
false,
amqp.Publishing{
ContentType: "application/json",
Body: message,
CorrelationId: correlationId,
ReplyTo: q.Name,
},
)
if err != nil {
return domain.SyncEventExtReceive{}, err
}
for d := range h {
fmt.Println("Received a message:", string(d.Body))
if d.CorrelationId == correlationId {
var event domain.SyncEventExtReceive
err = json.Unmarshal(d.Body, &event)
return event, err
}
}
return domain.SyncEventExtReceive{}, apperrors.ErrInternal
}
Basically, just consuming from the default exchange with a named response queue. Also, I send the queue name as the ReplyTo parameter and I give it a correlation id. The routing-key that is sent is daily-weather in this case.
On the server side, I tried to do the server with the default exchange, but Apache Camel forbids me to do nothing with that exchange.
from("rabbitmq:?queue=daily-weather&autoAck=true&autoDelete=false")
So, I assigned it the amq.direct exchange. However, that didn't also worked.
"rabbitmq:amq.direct?queue=daily-weather&autoAck=true&autoDelete=false"
Then, I added a second RabbitMQ endpoint to see if it would sent it, but nothing.
from("rabbitmq:amq.direct?queue=daily-weather&autoAck=true&autoDelete=false")
.log(LoggingLevel.INFO, "weather-daily", "Received message: \${body}")
.to("rabbitmq:amq.direct?queue=response&autoAck=true&autoDelete=false")
I ask if anybody has any simple example to do this with Apache Camel, because I am ultra lost. Any further detail can be shared if you contact me.
Thank you very much!!!! :)
SOLVED
Hi! After some time I decided to take a look to the spring-rabbitmq Camel component. I realised that Camel has exchange patterns, and rabbitmq, by default, sets it to inOut. This way, automatically returns the information back to the replyTo property.
val RABBIMQ_ROUTE =
"spring-rabbitmq:default?queues={{rabbitmq.weather.daily.routing_key}}"
default refers to the default exchange queue.

Getting "multipart: NextPart: EOF" error when reading multipart/form-data

I am building a router layer for an API where I take the response and extract a parameter (in this case a service_code, the name can vary - service_code, serviceCode, code need to handle all) from the request body and pass it to the service that needs it. The data can be in raw json format application/json , query params, path params or as form data multipart/form-data. I am able to do it with the others but with form, when I pass the payload to the destination service I get the error multipart: NextPart: EOF.
Here is how I am reading the data:
c.MultipartForm()
postForm := c.Request.PostForm
queryParams := c.Request.URL.Query()
route := c.Request.URL.Path
body, _ := ioutil.ReadAll(c.Request.Body)
serviceCode, destUrl, payload := getCityCode(queryParams, route, body, postForm)
Then I pass this payload to http.NewRequest
http.NewRequest(method, destUrl, strings.NewReader(payload))
At the destination when I read the data:
data := models.Data{}
err := c.Bind(&data); err != nil {
c.JSON(400, gin.H{"error": err.Error()})
return
}
The error is multipart: NextPart: EOF. Can someone help please!
This error happens when the form is empty because you are reading all the stream data with ioutil before binding it. You should change your code to use the information you need after binding it.

Tarantool broadcast call

I have cluster with several replicasets. I want to call some stored function on all nodes without calculate bucket_id, and after to map results. How should I do it?
You can use module cartridge.rpc function get_candidates for getting all nodes with some role, which you want to call and after to use module cartridge.pool function map_call for calling your function and mapping results. This function available from 1.2.0-17 version of cartridge. So your code could be like this:
local cartridge = require('cartridge')
local nodes = cartridge.rpc_get_candidates('my_role_name', { leaders_only = true, healthy_only = true })
local pool = require('cartridge.pool')
local results, err = pool.map_call('_G.my_function_name', { func_args }, { uri_list = nodes, timeout = 10 })
if (err ~= nil) then
#your error handling#
end
All function response will be saved to results variable and mapped for every URI. All errors will be saved to err variable as map with keys: line, class_name, err, file, suberrors, str
Another proposal.
If you use vshard and want to perform map-reduce over storages:
local replicaset, err = vshard.router.routeall()
for _, replica in pairs(replicaset) do
local _, err = replica:callrw('function', { args })
if err ~= nil then
return nil, err
end
end

Write datas from Stripe API in a CSV file in golang

I am trying to retrieve Stripe datas and parse them into a CSV file.
Here is my code:
package main
import (
"github.com/stripe/stripe-go"
"github.com/stripe/stripe-go/invoice"
"fmt"
"os"
"encoding/csv"
)
func main() {
stripe.Key = "" // I can't share the API key
params := &stripe.InvoiceListParams{}
params.Filters.AddFilter("limit", "", "3")
params.Filters.AddFilter("status", "", "paid")
i := invoice.List(params)
// Create a CSV file
csvdatafile, err := os.Create("./mycsvfile.csv")
if err != nil {
fmt.Println(err)
}
defer csvdatafile.Close()
// Write Unmarshaled json data to CSV file
w := csv.NewWriter(csvdatafile)
//Column title
var header []string
header = append(header, "ID")
w.Write(header)
for i.Next() {
in := i.Invoice()
fmt.Printf(in.ID) // It is working
w.Write(in) // It is not working
}
w.Flush()
fmt.Println("Appending succed")
}
When I am running my program with go run *.go I obtain the following error:
./main.go:35:10: cannot use in (type *stripe.Invoice) as type []string in argument to w.Write
I think I am not far from the solution.
I just need to understand how to write correctly in the CSV file thank's to w.Write() command.
According to the doc, the Write function is:
func (w *Writer) Write(record []string) error
That is, it is expecting you to pass a slice of strings representing a line of CSV data with each string being a slice. So, if you have only one field, you have to pass a string slice of 1:
w.Write([]string{in.ID})

Google bucket SignedUrls 403

I'm doing a simple rest API which does the following:
get base64 encoded image
decode it
stores it on on specific google bucket
Now, on the GET verb, my api returns a signed url targeting the bucket image.
I've coded a test that works:
initialization stuff
...
BeforeEach(func() {
mockStringGenerator.On("GenerateUuid").Return("image1")
// First store
image, _ = ioutil.ReadFile("test_data/DSCF6458.JPG")
encodedImage = b64.RawStdEncoding.EncodeToString(image)
fileName, storeError = storage.Store(ctx, encodedImage, "image/jpeg")
// Then get
uri, getError = storage.Get(ctx, fileName)
getResponse, _ = http.Get(uri)
// Finally delete
deleteError = storage.Delete(ctx, fileName)
})
// Only 1 test to avoid making too much connexion
It("should create, get and delete the image", func() {
// Store
Expect(storeError).To(BeNil())
Expect(fileName).To(Equal("image1.jpg"))
// Get
Expect(getError).To(BeNil())
Expect(getResponse.StatusCode).To(Equal(http.StatusOK))
b, _ := ioutil.ReadAll(getResponse.Body)
Expect(b).To(Equal(image))
// Delete
Expect(deleteError).To(BeNil())
})
But when I run the .exe and try to make ssome request with postman, I get a 403 error in the signed url:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>AccessDenied</Code>
<Message>Access denied.</Message>
<Details>Anonymous caller does not have storage.objects.get access to teddycare-images/08d8c508-d97d-48d3-947b-a7f216f622db.jpg.</Details>
</Error>
Any ideas ? I really don't understand...
Save me guys
[EDIT] Here after the code I use to create signedUrl:
func (s *GoogleStorage) Get(ctx context.Context, fileName string) (string, error) {
url, err := storage.SignedURL(s.Config.BucketImagesName, fileName, &storage.SignedURLOptions{
GoogleAccessID: s.Config.BucketServiceAccountDetails.ClientEmail,
PrivateKey: []byte(s.Config.BucketServiceAccountDetails.PrivateKey),
Method: http.MethodGet,
Expires: time.Now().Add(time.Second * 180),
})
if err != nil {
return "", err
}
return url, nil
}
Ok, after I woke up, I found the answer.
It turns out that when the json is marshalized into string, all the special characters are encoded.
Example: & -> \u0026
So the url I tested in my UT had &, while the url returned by the api had \u0026, and google does not seem to have the same behaviour on both cases.
So the solution is to disable HTML escaping:
encoder := json.NewEncoder(w)
encoder.SetEscapeHTML(false)
return encoder.Encode(response)