I have a spreadsheet on my Google Drive and I want to download a CSV from another website and put it into my spreadsheet. The problem is that I have to login to the website first, so I need to use some HTTP request to do that.
I have found this site and this. If either of these sites has the answer on it, then I clearly don't understand them enough to figure it out. Could someone help me figure this out? I feel that the second site is especially close to what I need, but I don't understand what it is doing.
To clarify again, I want to login with an HTTP request and then make a call to the same website with a different URL that is the call to get the CSV file.
I have done a lot of this in the past month so I should be able to help you, we are trying to emulate the browsers behaviour here so first you need to use chrome's developer tools(or something similar) and note down the exact things the browser does like the form values posted, the url that is called and so on. The following example shows the general techinique to be used:
The first step is to login to the website and get the session cookie:
var payload =
{
"user_session[email]" : "username",
"user_session[password]" : "password",
};// The actual values of the post variables (like user_session[email]) depends on the site so u need to get it either from the html of the login page or using the developer tools I mentioned.
var options =
{
"method" : "post",
"payload" : payload,
"followRedirects" : false
};
var login = UrlFetchApp.fetch("https://www.website.com/login" , options);
var sessionDetails = login.getAllHeaders()['Set-Cookie'];
We have logged into the website (In order to confirm just log the sessionDetails and match it with the cookies set by chrome). The next step is purely dependent on the website so I will give u a general example
var downloadPayload =
{
"__EVENTTARGET" : 'ctl00$ActionsPlaceHolder$exportDownloadLink1',
};// This is just an example it may or may not be needed, if needed u need to trace the values from the developer tools.
var downloadCsv = UrlFetchApp.fetch("https://www.website.com/",
{"headers" : {"Cookie" : sessionDetails},
"method" : "post",
"payload" : downloadPayload,
});
Logger.log(downloadCsv.getContentText())
The file should now be logged, you can then parse the csv using hte GAS inbuilt function and dump the data in the spreadsheet.
A few points to note:
I have assumed that all form post values are static and can be
hardcoded, in case this is not true then let me know I will give you
a function that can extract values from the html.
Some websites require the browser to send a token value(the value will be present in the html) along with the credentials. In this case you need to extract the values and then post it.
Related
I'm using the search API for Google Photos documented here. I'd like the results in the response to be sorted from newest to oldest, but by default, the results are sorted from oldest to newest. Is there a way to reverse the sorting?
I believe your goal as follows.
You want to sort the result values from the method of "Method: mediaItems.search".
You want to sort the values from oldest to newest.
Issue and workaround:
Unfortunately, in the current stage, it seems that there is no parameter for sorting the returned values for the the method of "Method: mediaItems.search" in Google Photos API. Also, it seems that such parameter is not existing in the method of "mediaItems.list".
By the way, it was found that when albumId is used in the request body for the method of "Method: mediaItems.search", the returned values are sorted as the ascending order. If you use the albumn ID, I think that your goal can be achieve by this.
On the other hand, when albumId is NOT used in the request body, the returned values are sorted as the descending order. And also, it seems that when filteres is used in the request body, the returned values are sorted as the descending order.
From your question, I thought that in your situation, albumId might be not used. So in this case, as the current workaround, how about sorting the values using a script after the values are retrieved? In this answer, I would like to propose to use the Web Apps created by Google Apps Script as a wrapper API.
Usage:
1. Create new project of Google Apps Script.
Sample script of Web Apps is a Google Apps Script. So please create a project of Google Apps Script.
If you want to directly create it, please access to https://script.new/. In this case, if you are not logged in Google, the log in screen is opened. So please log in to Google. By this, the script editor of Google Apps Script is opened.
2. Linking Cloud Platform Project to Google Apps Script Project.
About this, you can see the detail flow at here.
And also, please enable Google Photos API at API console.
3. Add scope.
In this case, please addt the scope of https://www.googleapis.com/auth/photoslibrary to the manifest file (appsscript.json).
4. Script.
Please copy and paste the following script (Google Apps Script) to the script editor. This script is for the Web Apps. This Web Apps is used as an API.
function doGet(e) {
const key = "sampleKey"; // This is used for using this Web Apps.
try {
if (e.parameter.key != key) throw new Error("Invalid key.");
const albumId = e.parameter.albumId;
const filters = e.parameter.filters;
const sort = e.parameter.sort;
const headers = {"Authorization": "Bearer " + ScriptApp.getOAuthToken()};
const url = "https://photoslibrary.googleapis.com/v1/mediaItems:search";
let mediaItems = [];
let pageToken = "";
const metadata = {pageSize: 100, pageToken: pageToken};
if (albumId) metadata.albumId = albumId;
if (filters) metadata.filters = JSON.parse(filters);
do {
const params = {
method: "post",
headers: headers,
contentType: "application/json",
payload: JSON.stringify(metadata),
}
const res = UrlFetchApp.fetch(url, params);
const obj = JSON.parse(res.getContentText());
mediaItems = mediaItems.concat(obj.mediaItems);
pageToken = obj.nextPageToken || "";
} while (pageToken);
if (mediaItems.length > 0) {
if (sort && sort == "ascending") {
mediaItems.sort((a, b) => new Date(a.mediaMetadata.creationTime) < new Date(b.mediaMetadata.creationTime) ? -1 : 1);
}
return ContentService.createTextOutput(JSON.stringify({values: mediaItems}));
}
return ContentService.createTextOutput(JSON.stringify({error: "No values."}));
} catch(err) {
return ContentService.createTextOutput(JSON.stringify({error: err.message}));
}
}
5. Deploy Web Apps.
The detail information can be seen at the official document.
On the script editor, at the top right of the script editor, please click "click Deploy" -> "New deployment".
Please click "Select type" -> "Web App".
Please input the information about the Web App in the fields under "Deployment configuration".
Please select "Me" for "Execute as".
This is the important of this workaround.
Please select "Anyone" for "Who has access".
In this case, the user is not required to use the access token. So please use this as a test case.
When you want to use the access token, please set it to Anyone with Google account or Only myself. By this, the user can access to the Web Apps using the access token. When you use the access token, please include the scope of https://www.googleapis.com/auth/drive.readonly or https://www.googleapis.com/auth/drive.
Please click "Deploy" button.
When "The Web App requires you to authorize access to your data" is shown, please click "Authorize access".
Automatically open a dialog box of "Authorization required".
Select own account.
Click "Advanced" at "This app isn't verified".
Click "Go to ### project name ###(unsafe)"
Click "Allow" button.
Copy the URL of Web App. It's like https://script.google.com/macros/s/###/exec.
When you modified the Google Apps Script, please redeploy as new version. By this, the modified script is reflected to Web Apps. Please be careful this.
6. Testing.
As the test of this Web Apps, I would like to propose to use the following curl command. Please replace https://script.google.com/macros/s/###/exec with your Web Apps URL.
Simple use:
In this curl command, the result value is returned as the ascending order of oldest to newest.
$ curl -GL -d "key=sampleKey" -d "sort=ascending" https://script.google.com/macros/s/###/exec
Use albumId:
When you want to use the album ID, please use the following curl command.
$ curl -GL -d "albumId=###" -d "key=sampleKey" -d "sort=ascending" https://script.google.com/macros/s/###/exec
In this case, even when -d "sort=ascending" is not used, the result value is returned as the ascending order of oldest to newest.
Use filters:
When you want to use the filters, please use the following curl command.
$ curl -GL -d 'filters={"dateFilter":{"ranges":[{"startDate":{"year":2020},"endDate":{"year":2021}}]}}' -d "key=sampleKey" -d "sort=ascending" https://script.google.com/macros/s/###/exec
In this command, the values of 2020 - 2021 are returned as the ascending order of oldest to newest.
Note:
Although when I searched this at the Google issue tracker, I couldn't find about it. So how about reporting this as the future request? Ref
References:
Method: mediaItems.search
Related thread.
How to use Google Photos API Method: mediaItems.search in Google apps script for a spreadsheet
Google photos api adding photos not working, upload seems to work
Google Apps Scripts: import (upload) media from Google Drive to Google Photos?
I'd like to automate my log in to my bank to automatically fetch my transactions to stay up-to-date with spendings and earnings, but I am stuck.
The bank's login webpage is: https://login.bancochile.cl/bancochile-web/persona/login/index.html#/login
I am using python's request module with sessions:
urlLoginPage = 'https://login.bancochile.cl/bancochile-web/persona/login/index.html'
urlLoginSubmit = 'https://login.bancochile.cl/oam/server/auth_cred_submit'
username = '11.111.111-1' # this the format of a Chilean National ID ("RUT")
usernameFormatted = '111111111' # same id but formatted
pw = "password"
payload = [
("username2", usernameFormatted),
("username2", username),
("userpassword", pw),
("request_id", ''),
("ctx", "persona"),
("username", usernameFormatted),
("password", pw),
]
with requests.Session() as session:
login = session.get(urlLoginPage)
postLogin = session.post(
urlLoginSubmit,
data=payload,
allow_redirects=False,
)
redirectUrl = postLogin.headers["Location"]
First I find that the form data has duplicated keys, so I am using the payload as a list of tuples. From Chrome's inspect I find the form data to be like this:
username2=111111111&username2=11.111.111-1&userpassword=password&request_id=&ctx=persona&username=111111111&password=password
I've checked the page's source code to look for the use of a csrf token, but couldn't find any hint of it.
What happens is that the site does a redirect upon submitting the login data. I set allow_redirects=False to catch the redirect url of the post under the Location-header. However, here is the problem. Using the web-browser I know that the redirect url should be https://portalpersonas.bancochile.cl/mibancochile/rest/persona/perfilamiento/home, but I always end up on an error page when using the above method (https://login.bancochile.cl/bancochile-web/contingencia/error404.html). (I am using my own, correct login credentials to try this)
If I submit the payload in a wrong format (e.g. by dropping a key) I am redirected to the same error-page. This tells me that probably something with the payload is incorrect, but I don't know how to find out what may be wrong.
I am kind of stuck and don't know how I can figure out where/how to look for errors and possible solutions. Any suggestions on how to debug this and continue or ideas for other approaches would be very welcome!
Thanks!
I'm trying to write a react native app which will stream some tracks from Soundcloud. As a test, I've been playing with the API using python, and I'm able to make requests to resolve the url, pull the playlists/tracks, and everything else I need.
With that said, when making a request to the stream_url of any given track, I get a 401 error.
The current url in question is:
https://api.soundcloud.com/tracks/699691660/stream?client_id=PGBAyVqBYXvDBjeaz3kSsHAMnr1fndq1
I've tried it without the ?client_id..., I have tried replacing the ? with &, I've tried getting another client_id, I've tried it with allow_redirects as both true and false, but nothing seems to work. Any help would be greatly appreciated.
The streamable property of every track is True, so it shouldn't be a permissions issue.
Edit:
After doing a bit of research, I've found a semi-successful workaround. The /stream endpoint of the API is still not working, but if you change your destination endpoint to http://feeds.soundcloud.com/users/soundcloud:users:/sounds.rss, it'll give you an RSS feed that's (mostly) the same as what you'd get by using the tracks or playlists API endpoint.
The link contained therein can be streamed.
Okay, I think I have found a generalized solution that will work for most people. I wish it were easier, but it's the simplest thing I've found yet.
Use API to pull tracks from user. You can use linked_partitioning and the next_href property to gather everything because there's a maximum limit of 200 tracks per call.
Using the data pulled down in the JSON, you can use the permalink_url key to get the same thing you would type into the browser.
Make a request to the permalink_url and access the HTML. You'll need to do some parsing, but the url you'll want will be something to the effect of:
"https://api-v2.soundcloud.com/media/soundcloud:tracks:488625309/c0d9b93d-4a34-4ccf-8e16-7a87cfaa9f79/stream/progressive"
You could probably use a regex to parse this out simply.
Make a request to this url adding ?client_id=... and it'll give you YET ANOTHER url in its return json.
Using the url returned from the previous step, you can link directly to that in the browser, and it'll take you to your track content. I checked on VLC by inputting the link and it streams correctly.
Hopefully this helps some of you out with your developing.
Since I have the same problem, the answer from #Default motivated me to look for a solution. But I did not understand the workaround with the permalink_url in the steps 2 and 3. The easier solution could be:
Fetch for example user track likes using api-v2 endpoint like this:
https://api-v2.soundcloud.com/users/<user_id>/track_likes?client_id=<client_id>
In the response we can finde the needed URL like mentioned from #Default in his answer:
collection: [
{
track: {
media: {
transcodings:[
...
{
url: "https://api-v2.soundcloud.com/media/soundcloud:tracks:713339251/0ab1d60e-e417-4918-b10f-81d572b862dd/stream/progressive"
...
}
]
}
}
...
]
Make request to this URL with client_id as a query param and you get another URL with that you can stream/download the track
Note that the api-v2 is still not public and the request from your client probably will be blocked by CORS.
As mentioned by #user208685 the solution can be a bit simpler by using the SoundCloud API v2:
Obtain the track ID (e.g. using the public API at https://developers.soundcloud.com/docs)
Get JSON from https://api-v2.soundcloud.com/tracks/TRACK_ID?client_id=CLIENT_ID
From JSON parse MP3 progressive stream URL
From stream URL get MP3 file URL
Play media from MP3 file URL
Note: This link is only valid for a limited amount of time and can be regenerated by repeating steps 3. to 5.
Example in node (with node-fetch):
const clientId = 'YOUR_CLIENT_ID';
(async () => {
let response = await fetch(`https://api.soundcloud.com/resolve?url=https://soundcloud.com/d-o-lestrade/gabriel-ananda-maceo-plex-solitary-daze-original-mix&client_id=${clientId}`);
const track = await response.json();
const trackId = track.id;
response = await fetch(`https://api-v2.soundcloud.com/tracks/${trackId}?client_id=${clientId}`);
const trackV2 = await response.json();
const streamUrl = trackV2.media.transcodings.filter(
transcoding => transcoding.format.protocol === 'progressive'
)[0].url;
response = await fetch(`${streamUrl}?client_id=${clientId}`);
const stream = await response.json();
const mp3Url = stream.url;
console.log(mp3Url);
})();
For a similar solution in Python, check this GitHub issue: https://github.com/soundcloud/soundcloud-python/issues/87
I'm currently trying to implement few Postman requests with CSV data sources.
For instance let assume I have request named "Open as user".
In csv file, I have bunch of user credentials with description field that describes user role.
I would like to have the ability to change request names to reflect each user roles.
For instance, if the request is made as the admin user I would like request name in reports and runner to be "Open as user admin".
In documentation, I've found pm.info.requestName variable but seems it is read-only.
I put following in Pre-request Script
pm.info.requestName = "1";
console.log(pm.info.requestName);
but got "Open as user" value instead of assigned "1".
Have anyone tried to do the same trick or know whether it is possible at all?
I was looking for a solution to this as well and i have solved it in the following way:
Use the 'Pre-request Script' to determine what role the user has.
Set the url to a environment- or global-variable
Use that url in the request, like {{url}}
Clear the environment- or global-variable in the 'Tests' tab
In my case i used the environment variable to get the environment, instead of you, using the data variables.
var environment = pm.environment.get("environment");
var url;
switch(environment) {
case "test":
url = pm.globals.get("test-url");
break;
case "acc":
url = pm.globals.get("acc-url");
break;
case "prod":
url = pm.globals.get("prod-url");
break;
default:
url = pm.globals.get("test-url");
break;
}
pm.environment.set("url", url);
Hope this helps!
I'm afraid that you can't do that. pm.request object is available only after request execution. I even think that you can't access the request name the way you want (I though I would find it in the 'id' member, but it was empty)
Have a look here to see what's available in terms of members and methods concerning the request object.
You may find another way of proceeding (maybe duplicate your test, rename it with admin and, under proper condition, launch the admin request instead of the common user request ? it's kind of a hassle just for a test name)
I think about something else: you could just customize your assertions label, that's what I do to have 'readable' test names in TFS reports. In the Tests tab :
test_name = "[ "+ request.name + " ] - ";
and for each assertion (example):
tests[test_name + "Status code is 200"] = responseCode.code === 200;
This gives, in the response Tests tab, something like :
PASS [Get all configuration]-Status code is 200
Under a particular condition, you can replace request.name with a custom string or do request.name + "admin" ...
hope this helps
When I open the URL with driver.get(url), how can I get the response content of the page? Please refer to the image for more information.
In a separate post I saw this answer. As per it there is a ticket opened for Selenium.
I'm using Python and Django, but it's actually simple to get the response. I'm using a StaticLiveServerTestCase as my base test for the test. The .get() method on self.client actually returns the response itself. For example:
response = self.client.get(url)
However, it looks like what you're really trying to get is the cookie based on what you're pointing to in the picture. I use Django and the Django test suite to authenticate a user session to be used in the test.
def create_pre_authenticated_session(self, username, url="/"):
user = User.objects.create(username=username)
session = SessionStore()
session[SESSION_KEY] = user.pk
session[BACKEND_SESSION_KEY] = settings.AUTHENTICATION_BACKENDS[0]
session[HASH_SESSION_KEY] = user.get_session_auth_hash()
session.save()
# to set a cookie we need to first visit the domain.
# 404 pages load the quickest!
self.browser.get(self.live_server_url + '/404_no_such_url/')
self.browser.add_cookie(dict(
name=settings.SESSION_COOKIE_NAME,
value=session.session_key,
secure=False,
path='/',
))
self.browser.get(self.live_server_url + url)
return user
This has some other stuff in it that I borrowed from Percival's Test-Driven Development with Python, but I hope that it can provide some guidance on what you're trying to accomplish.