Google bucket SignedUrls 403 - api

I'm doing a simple rest API which does the following:
get base64 encoded image
decode it
stores it on on specific google bucket
Now, on the GET verb, my api returns a signed url targeting the bucket image.
I've coded a test that works:
initialization stuff
...
BeforeEach(func() {
mockStringGenerator.On("GenerateUuid").Return("image1")
// First store
image, _ = ioutil.ReadFile("test_data/DSCF6458.JPG")
encodedImage = b64.RawStdEncoding.EncodeToString(image)
fileName, storeError = storage.Store(ctx, encodedImage, "image/jpeg")
// Then get
uri, getError = storage.Get(ctx, fileName)
getResponse, _ = http.Get(uri)
// Finally delete
deleteError = storage.Delete(ctx, fileName)
})
// Only 1 test to avoid making too much connexion
It("should create, get and delete the image", func() {
// Store
Expect(storeError).To(BeNil())
Expect(fileName).To(Equal("image1.jpg"))
// Get
Expect(getError).To(BeNil())
Expect(getResponse.StatusCode).To(Equal(http.StatusOK))
b, _ := ioutil.ReadAll(getResponse.Body)
Expect(b).To(Equal(image))
// Delete
Expect(deleteError).To(BeNil())
})
But when I run the .exe and try to make ssome request with postman, I get a 403 error in the signed url:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>AccessDenied</Code>
<Message>Access denied.</Message>
<Details>Anonymous caller does not have storage.objects.get access to teddycare-images/08d8c508-d97d-48d3-947b-a7f216f622db.jpg.</Details>
</Error>
Any ideas ? I really don't understand...
Save me guys
[EDIT] Here after the code I use to create signedUrl:
func (s *GoogleStorage) Get(ctx context.Context, fileName string) (string, error) {
url, err := storage.SignedURL(s.Config.BucketImagesName, fileName, &storage.SignedURLOptions{
GoogleAccessID: s.Config.BucketServiceAccountDetails.ClientEmail,
PrivateKey: []byte(s.Config.BucketServiceAccountDetails.PrivateKey),
Method: http.MethodGet,
Expires: time.Now().Add(time.Second * 180),
})
if err != nil {
return "", err
}
return url, nil
}

Ok, after I woke up, I found the answer.
It turns out that when the json is marshalized into string, all the special characters are encoded.
Example: & -> \u0026
So the url I tested in my UT had &, while the url returned by the api had \u0026, and google does not seem to have the same behaviour on both cases.
So the solution is to disable HTML escaping:
encoder := json.NewEncoder(w)
encoder.SetEscapeHTML(false)
return encoder.Encode(response)

Related

Write datas from Stripe API in a CSV file in golang

I am trying to retrieve Stripe datas and parse them into a CSV file.
Here is my code:
package main
import (
"github.com/stripe/stripe-go"
"github.com/stripe/stripe-go/invoice"
"fmt"
"os"
"encoding/csv"
)
func main() {
stripe.Key = "" // I can't share the API key
params := &stripe.InvoiceListParams{}
params.Filters.AddFilter("limit", "", "3")
params.Filters.AddFilter("status", "", "paid")
i := invoice.List(params)
// Create a CSV file
csvdatafile, err := os.Create("./mycsvfile.csv")
if err != nil {
fmt.Println(err)
}
defer csvdatafile.Close()
// Write Unmarshaled json data to CSV file
w := csv.NewWriter(csvdatafile)
//Column title
var header []string
header = append(header, "ID")
w.Write(header)
for i.Next() {
in := i.Invoice()
fmt.Printf(in.ID) // It is working
w.Write(in) // It is not working
}
w.Flush()
fmt.Println("Appending succed")
}
When I am running my program with go run *.go I obtain the following error:
./main.go:35:10: cannot use in (type *stripe.Invoice) as type []string in argument to w.Write
I think I am not far from the solution.
I just need to understand how to write correctly in the CSV file thank's to w.Write() command.
According to the doc, the Write function is:
func (w *Writer) Write(record []string) error
That is, it is expecting you to pass a slice of strings representing a line of CSV data with each string being a slice. So, if you have only one field, you have to pass a string slice of 1:
w.Write([]string{in.ID})

I am trying to get data over an OpenWeather API in Rust but I am facing some iusse regarding parsing I guess

extern crate openweather;
use openweather::LocationSpecifier;
static API_KEY: &str = "e85e0a3142231dab28a2611888e48f22";
fn main() {
let loc = LocationSpecifier::Coordinates {
lat: 24.87,
lon: 67.03,
};
let weather = openweather::get_current_weather(loc, API_KEY).unwrap();
print!(
"Right now in Minneapolis, MN it is {}K",
weather.main.humidity
);
}
error : thread 'main' panicked at 'called Result::unwrap() on an
Err value: ErrorReport { cod: 0, message: "Got unexpected response:
\"{\\"coord\\":{\\"lon\\":67.03,\\"lat\\":24.87},\\"weather\\":[{\\"id\\":803,\\"main\\":\\"Clouds\\",\\"description\\":\\"broken
clouds\\",\\"icon\\":\\"04n\\"}],\\"base\\":\\"stations\\",\\"main\\":{\\"temp\\":294.15,\\"pressure\\":1018,\\"humidity\\":60,\\"temp_min\\":294.15,\\"temp_max\\":294.15},\\"visibility\\":6000,\\"wind\\":{\\"speed\\":5.1,\\"deg\\":30},\\"clouds\\":{\\"all\\":70},\\"dt\\":1574012543,\\"sys\\":{\\"type\\":1,\\"id\\":7576,\\"country\\":\\"PK\\",\\"sunrise\\":1573955364,\\"sunset\\":1573994659},\\"timezone\\":18000,\\"id\\":1174872,\\"name\\":\\"Karachi\\",\\"cod\\":200}\""
}
The issue is a JSON parsing error due to the deserialized struct not matching OpenWeather's JSON, perhaps the API recently added this? With your example, the OpenWeatherCurrent struct is missing timezone.
But it looks like there is an open PR that will fix this, you can test it by doing the following:
Change your Cargo.toml dependency to openweather = { git = "https://github.com/caemor/openweather" }.
The PR author has also updated the get_current_weather signature so you'll need to change lines 2, 10 to the following:
use openweather::{LocationSpecifier, Settings};
let weather = openweather::get_current_weather(&loc, API_KEY, &Settings::default()).unwrap();

How to use URLSession downloadTaskWithResumeData to start download again when AfterdidCompleteWithError Called..?

I have the code to download two files from Server and store It to In local using URLSession (let dataTask = defaultSession.downloadTask(with: url)). Everything Is working fine only the problem is it's downloading first file it's giving me success but the second file is not downloading completely.. So, I hope there is a way to restart download for the second file that gives error ..
I think there is way of doing that and start looking into it and I found this delegate method .. but not much help .. can anyone please help me out how to restart download if it fails .. Do i have to use handleEventsForBackgroundURLSession to clear up previous downloads..?
// bellow download method will triggered when i get filenames I am passing it to this and path is optional here..
func download(path: String?, filenames: [String]) -> Int {
for filename in filenames {
var downloadFrom = "ftp://" + username! + ":"
downloadFrom += password!.addingPercentEncoding(withAllowedCharacters: .urlPasswordAllowed)! + "#" + address!
if let downloadPort = port {
downloadFrom += ":" + String(downloadPort) + "/"
} else {
downloadFrom += "/"
}
if let downloadPath = path {
if !downloadPath.isEmpty {
downloadFrom += downloadPath + "/"
}
}
downloadFrom += filename
if let url = URL(string: downloadFrom) {
let dataTask = defaultSession.downloadTask(with: url)
dataTask.resume()
}
}
return DLResponseCode.success
}
Please find delegate methods bellow ..
func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didFinishDownloadingTo location: URL) {
var responseCode = DLResponseCode.success
// Move the file to a new URL
let fileManager = FileManager.default
let filename = downloadTask.originalRequest?.url?.lastPathComponent
let destUrl = cacheURL.appendingPathComponent(filename!)
do {
let data = try Data(contentsOf: location)
// Delete it if it exists first
if fileManager.fileExists(atPath: destUrl.path) {
do{
try fileManager.removeItem(at: destUrl)
} catch let error {
danLogError("Clearing failed downloadFOTA file failed: \(error)")
responseCode = DLResponseCode.datalogger.failToCreateRequestedProtocolPipe
}
}
try data.write(to: destUrl)
} catch {
danLogError("Issue saving data locally")
responseCode = DLResponseCode.datalogger.noDataConnection
}
// Complete the download message
let message = DLBLEDataloggerChannel.Commands.download(responseCode: responseCode).description
connectionManagerDelegate?.sendMessageToDatalogger(msg: message)
}
func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
if error == nil {
print("session \(session) download completed")
} else {
print("session \(session) download failed with error \(String(describing: error?.localizedDescription))")
// session.downloadTask(withResumeData: <#T##Data#>)
}
guard error != nil else {
return
}
danLogError("Session \(session) invalid with error \(String(describing: error))\n")
let responseCode = DLResponseCode.datalogger.failToCreateRequestedProtocolPipe
let message = DLBLEDataloggerChannel.Commands.download(responseCode: responseCode).description
connectionManagerDelegate?.sendMessageToDatalogger(msg: message)
}
// When I call didWriteData delegate method it's printing below data seems not dowloaded complete data ..
session <__NSURLSessionLocal: 0x103e37970> download task <__NSCFLocalDownloadTask: 0x108d2ee60>{ taskIdentifier: 2 } { running } wrote an additional 30028 bytes (total 988980 bytes) out of an expected 988980 bytes.
//error that I am getting for second file .. this error is coming some times not always but most of the times..
session <__NSURLSessionLocal: 0x103e37970> download failed with error Optional("cancelled")
Please help me out to figure it out .. If there is any way to handle download again after it fails or why it fails ..
The resume data, if the request is resumable, should be in the NSError object's userInfo dictionary.
Unfortunately, Apple seems to have completely trashed the programming guide for NSURLSession (or at least I can't find it in Google search results), and the replacement content in the reference is missing all of the sections that talk about how to do proper error handling (even the constant that you're looking for is missing), so I'm going to have to describe it all from memory with the help of looking at the headers. Ick.
The key you're looking for is NSURLSessionDownloadTaskResumeData.
If that key is present, its value is a small NSData blob. Store that, then use the Reachability API (with the actual hostname from that request's URL) to decide when to retry the request.
After Reachability tells you that the server is reachable, create a new download task with the resume data and start it.

400 error getting oauth user token in golang?

I tried to get the ebay user token.
I've alreay gotten auth code in ebay, but can't get user token with auth code.
Code is following:
status := ctx.UserValue("status").(string)
applicationToken := string(ctx.QueryArgs().Peek("code"))
log.Println("ApplicationToken: ", applicationToken)
log.Println("Status: ", status)
if status == "declined" {
    fmt.Printf("User doesn't give permission. Go back to your dashboard.")
    ctx.Redirect("/dashboard", fasthttp.StatusSeeOther)
}
//var appConfig = config.Config()
client := &http.Client{}
applicationTokenURLEncoded, _ := url.Parse(applicationToken)
body := url.Values{
    "grant_type": {"authorization_code"},
    "code": {applicationTokenURLEncoded.String()},
    "redirect_uri": {Runame},
}
reqBody := bytes.NewBufferString(body.Encode())
log.Println("Reqbody: ", reqBody)
req, _ := http.NewRequest("POST", "https://api.sandbox.ebay.com/identity/v1/oauth2/token", reqBody)
authorization := “AppID” + ":" + “CertID”
authorizationBase64 := base64.StdEncoding.EncodeToString([]byte(authorization))
req.Header.Add("Authorization", "Basic "+authorizationBase64)
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
log.Println("Body: ", req)
resp, _ := client.Do(req)
log.Println("resp: ", resp)
log.Println("ResBody: ", resp.Body)
And the error is like that:
Bad Request 400
I looks like the message isn't formed correctly (which is why the server returns 400).
When you're setting your headers you write:
req.Header.Add("Authorization", "Basic "+authorizationBase64)
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
Golang has a nice built in way to add Authorization to a Header: SetBasicAuth, which is a method of http.Request.
func (r *Request) SetBasicAuth(username, password string)
SetBasicAuth sets the request's Authorization header to use HTTP Basic Authentication with the provided username and password.
With HTTP Basic Authentication the provided username and password are not encrypted.
So instead of setting the Authorization Header manually, you could try:
req.SetBasicAuth("AppID”, “CertID”)
You were using something called CertID, but I think you meant ClientId and ClientSecret -- respectively user and password
I don't know if this is the cause of your problem :). One way to help find out, is consistent error checking, see your line:
applicationTokenURLEncoded, _ := url.Parse(applicationToken)

Golang file upload to s3 using OS Open

I am trying to upload an Image to my s3 account using Golang and the amazon s3 api . I can get the imagine uploaded if I hard code the direct path such as
file, err := os.Open("/Users/JohnSmith/Documents/pictures/cars.jpg")
defer file.Close()
if err != nil {
fmt.Printf("err opening file: %s", err)
}
if I hard code the file path like that then the picture will be uploaded to my s3 account . However that approach is not good as I can't obviously hard code the direct image path to every image that I want to upload . My question is how can I upload images without having to Hardcode the path . This will be apart of an API where users will upload images so I clearly can not have a hard coded path . This is my code first the HTML
<form method="post" enctype="multipart/form-data" action="profile_image">
<h2>Image Upload</h2>
<p><input type="file" name="file" id="file"/> </p>
<p> <input type="submit" value="Upload Image"></p>
</form>
then this is my HTTP Post function method
func UploadProfile(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
var resultt string
resultt = "Hi"
sess, _ := session.NewSession(&aws.Config{
Region: aws.String("us-west-2"),
Credentials: credentials.NewStaticCredentials(aws_access_key_id,aws_secret_access_key, ""),
})
svc := s3.New(sess)
file, err := os.Open("Users/JohnSmith/Documents/pictures/cars.jpg")
defer file.Close()
if err != nil {
fmt.Printf("err opening file: %s", err)
}
fileInfo, _ := file.Stat()
size := fileInfo.Size()
buffer := make([]byte, size) // read file content to buffer
file.Read(buffer)
fileBytes := bytes.NewReader(buffer)
fileType := http.DetectContentType(buffer)
path := file.Name()
params := &s3.PutObjectInput{
Bucket: aws.String("my-bucket"),
Key: aws.String(path),
Body: fileBytes,
ContentLength: aws.Int64(size),
ContentType: aws.String(fileType),
}
resp, err := svc.PutObject(params)
if err != nil {
fmt.Printf("bad response: %s", err)
}
fmt.Printf("response %s", awsutil.StringValue(resp))
}
That is my full code above however when I try to do something such as
file, err := os.Open("file")
defer file.Close()
if err != nil {
fmt.Printf("err opening file: %s", err)
}
I get the following error
http: panic serving [::1]:55454: runtime error: invalid memory address or nil pointer dereference
goroutine 7 [running]:
err opening file: open file: no such file or directorynet/http.(*conn).serve.func1(0xc420076e80)
I can't use absolute path (filepath.Abs()) because some of the files will be outside of the GoPath and as stated other users will be uploading. Is there anyway that I can get a relative path ..
After POST to your API, images are temporarily saved in a OS's temp directory (different for different OS's) by default. To get this directory you can use, for example:
func GetTempLoc(filename string) string {
return strings.TrimRight(os.TempDir(), "/") + "/" + filename
}
Where:
filename is a header.Filename, i.e. file name received in your POST request. In Gin-Gonic framework you get it in your request handler as:
file, header, err := c.Request.FormFile("file")
if err != nil {
return out, err
}
defer file.Close()
Example: https://github.com/gin-gonic/gin#another-example-upload-file.
I'm sure in your framework there will be an analogue.
os.TempDir() is a function go give you a temp folder (details: https://golang.org/pkg/os/#TempDir).
TrimRight is used to ensure result of os.TempDir is consistent on different OSs
And then you use it as
file, err := os.Open(GetTempLoc(fileName))
...