I'd like to automate my log in to my bank to automatically fetch my transactions to stay up-to-date with spendings and earnings, but I am stuck.
The bank's login webpage is: https://login.bancochile.cl/bancochile-web/persona/login/index.html#/login
I am using python's request module with sessions:
urlLoginPage = 'https://login.bancochile.cl/bancochile-web/persona/login/index.html'
urlLoginSubmit = 'https://login.bancochile.cl/oam/server/auth_cred_submit'
username = '11.111.111-1' # this the format of a Chilean National ID ("RUT")
usernameFormatted = '111111111' # same id but formatted
pw = "password"
payload = [
("username2", usernameFormatted),
("username2", username),
("userpassword", pw),
("request_id", ''),
("ctx", "persona"),
("username", usernameFormatted),
("password", pw),
]
with requests.Session() as session:
login = session.get(urlLoginPage)
postLogin = session.post(
urlLoginSubmit,
data=payload,
allow_redirects=False,
)
redirectUrl = postLogin.headers["Location"]
First I find that the form data has duplicated keys, so I am using the payload as a list of tuples. From Chrome's inspect I find the form data to be like this:
username2=111111111&username2=11.111.111-1&userpassword=password&request_id=&ctx=persona&username=111111111&password=password
I've checked the page's source code to look for the use of a csrf token, but couldn't find any hint of it.
What happens is that the site does a redirect upon submitting the login data. I set allow_redirects=False to catch the redirect url of the post under the Location-header. However, here is the problem. Using the web-browser I know that the redirect url should be https://portalpersonas.bancochile.cl/mibancochile/rest/persona/perfilamiento/home, but I always end up on an error page when using the above method (https://login.bancochile.cl/bancochile-web/contingencia/error404.html). (I am using my own, correct login credentials to try this)
If I submit the payload in a wrong format (e.g. by dropping a key) I am redirected to the same error-page. This tells me that probably something with the payload is incorrect, but I don't know how to find out what may be wrong.
I am kind of stuck and don't know how I can figure out where/how to look for errors and possible solutions. Any suggestions on how to debug this and continue or ideas for other approaches would be very welcome!
Thanks!
Related
I am trying to log in to a site using requests as follows:
s = requests.Session()
login_data = {"userName":"username", "password":"pass", "loginPath":"/d2l/login"}
resp = requests.post("https://d2l.pima.edu/d2l/login?login=1", login_data)
although I am getting a 200 response, when I say
print(resp.content)
b"<!DOCTYPE html><html><head><meta charset='utf-8' /><script>var hash = window.location.hash;if( hash ) hash = '%23' + hash.substring( 1 );window.location.replace('/d2l/login?sessionExpired=0&target=%2fd2l%2ferror%2f404%2flog%3ftargetUrl%3dhttp%253A%252F%252Fd2l.pima.edu%253A80%252Fd2l%252Flogin%253Flogin%253D1' + hash );</script><title></title></head><body></body></html>"
notice it says session expired.
What I've tried:
logging back out and in in the actual browser, no success.
http basic auth, no success.
I'm thinking maybe I need to authenticate myself to this site using cookies?
If so how do I determine which cookies to send it?
I tried figuring this out by saying
resp.cookies
Out[4]: <RequestsCookieJar[]>
shouldn't this be giving me names of cookies? I'm not sure what to do with such output.
Main Point: HOW DO I FIGURE OUT HOW TO AUTHENTICATE MYSLEF TO THIS WEBSITE?
Help is appreciated.
I would rather not use selenium.
From loading this page https://d2l.pima.edu/d2l/login and viewing its source, you'll notice the POST target path is /d2l/lp/auth/login/login.d2l. Try using that as your POST path. Your other fields look consistent with the form's expectations.
Note: with python requests if you create a session object use it to make your requests:
resp = s.post(<blah blah>, login_data)
The session will hold any cookies set by the login server, and you can continue to use the s object to make requests in the authenticated session.
I developed a chrome extension using Rally's WSAPI v2.0, and it basically does the following things:
get user and project, and store them
get current iteration everytime
send a post request to create a workitem
For the THIRD step, I sometimes get error ["Not authorized to perform action: Invalid key"] since end of last month.
[updated]Error can be reproduced everytime if I log in Rally website via SSO before using the extension to send requests via apikey.
What's the best practice to send subsequent requests via apikey in my extension since I can't control end users' habits?
I did see some similar posts but none of them is helpful... and in case it helps:
I'm adding ZSESSIONID:apikey in my request header, instead of user /
password to authenticate, so I believe no security token is needed
(https://comm.support.ca.com/kb/api-key-and-oauth-client-faq/kb000011568)
url starts with https://rally1.rallydev.com/slm/webservice/v2.0/
issue is fixed after clearing cookies for
https://rally1.rallydev.com/, but somehow it appears again some time
later
I checked the cookie when the issue was reproduced, and found one with name of ZSESSIONID and its value became something else rather than the apikey. Not sure if that matters though...
code for request:
function initXHR(method, url, apikey, cbFunc) {
let httpRequest = new XMLHttpRequest();
...
httpRequest.open(method, url);
httpRequest.setRequestHeader('Content-Type', ' application\/json');
httpRequest.setRequestHeader('Accept', ' application\/json');
httpRequest.setRequestHeader('ZSESSIONID', apikey);
httpRequest.onreadystatechange = function() {
...
};
return httpRequest;
}
...
usReq = initXHR ('POST', baseURL+'hierarchicalrequirement/create', apikey, function(){...});
Anyone has any idea / suggestion? Thanks a million!
I've seen this error when the API key had both read-only and full-access grants configured. I would start by making sure your key only has the full-access grant.
When I open the URL with driver.get(url), how can I get the response content of the page? Please refer to the image for more information.
In a separate post I saw this answer. As per it there is a ticket opened for Selenium.
I'm using Python and Django, but it's actually simple to get the response. I'm using a StaticLiveServerTestCase as my base test for the test. The .get() method on self.client actually returns the response itself. For example:
response = self.client.get(url)
However, it looks like what you're really trying to get is the cookie based on what you're pointing to in the picture. I use Django and the Django test suite to authenticate a user session to be used in the test.
def create_pre_authenticated_session(self, username, url="/"):
user = User.objects.create(username=username)
session = SessionStore()
session[SESSION_KEY] = user.pk
session[BACKEND_SESSION_KEY] = settings.AUTHENTICATION_BACKENDS[0]
session[HASH_SESSION_KEY] = user.get_session_auth_hash()
session.save()
# to set a cookie we need to first visit the domain.
# 404 pages load the quickest!
self.browser.get(self.live_server_url + '/404_no_such_url/')
self.browser.add_cookie(dict(
name=settings.SESSION_COOKIE_NAME,
value=session.session_key,
secure=False,
path='/',
))
self.browser.get(self.live_server_url + url)
return user
This has some other stuff in it that I borrowed from Percival's Test-Driven Development with Python, but I hope that it can provide some guidance on what you're trying to accomplish.
I have a spreadsheet on my Google Drive and I want to download a CSV from another website and put it into my spreadsheet. The problem is that I have to login to the website first, so I need to use some HTTP request to do that.
I have found this site and this. If either of these sites has the answer on it, then I clearly don't understand them enough to figure it out. Could someone help me figure this out? I feel that the second site is especially close to what I need, but I don't understand what it is doing.
To clarify again, I want to login with an HTTP request and then make a call to the same website with a different URL that is the call to get the CSV file.
I have done a lot of this in the past month so I should be able to help you, we are trying to emulate the browsers behaviour here so first you need to use chrome's developer tools(or something similar) and note down the exact things the browser does like the form values posted, the url that is called and so on. The following example shows the general techinique to be used:
The first step is to login to the website and get the session cookie:
var payload =
{
"user_session[email]" : "username",
"user_session[password]" : "password",
};// The actual values of the post variables (like user_session[email]) depends on the site so u need to get it either from the html of the login page or using the developer tools I mentioned.
var options =
{
"method" : "post",
"payload" : payload,
"followRedirects" : false
};
var login = UrlFetchApp.fetch("https://www.website.com/login" , options);
var sessionDetails = login.getAllHeaders()['Set-Cookie'];
We have logged into the website (In order to confirm just log the sessionDetails and match it with the cookies set by chrome). The next step is purely dependent on the website so I will give u a general example
var downloadPayload =
{
"__EVENTTARGET" : 'ctl00$ActionsPlaceHolder$exportDownloadLink1',
};// This is just an example it may or may not be needed, if needed u need to trace the values from the developer tools.
var downloadCsv = UrlFetchApp.fetch("https://www.website.com/",
{"headers" : {"Cookie" : sessionDetails},
"method" : "post",
"payload" : downloadPayload,
});
Logger.log(downloadCsv.getContentText())
The file should now be logged, you can then parse the csv using hte GAS inbuilt function and dump the data in the spreadsheet.
A few points to note:
I have assumed that all form post values are static and can be
hardcoded, in case this is not true then let me know I will give you
a function that can extract values from the html.
Some websites require the browser to send a token value(the value will be present in the html) along with the credentials. In this case you need to extract the values and then post it.
i am having some problem with the authentication in urllib2, it is hitting some pages
like i have
https://localhost:5260/user
I am using a basic authentication for this and its working fine, a page is retrieved correctly
username = 'test'
password = 'test'
base64string = base64.encodestring('%s:%s' % (username, password))[:-1]
authheader = "Basic %s" % base64string
but when i apply the same thing to another page which the user is an admin and trying to access, its not returning the page with authentication
https://localhost:5260/post/250
I understand this is something do with the state which is saved, but i am not able to figure it out.
This seems trivial now, I need to set the cookie and pass that information to the server :)
The link below gives a good description of how to look at it.
http://www.voidspace.org.uk/python/articles/cookielib.shtml