I'm using Selenium on Python 3, to perform automated actions on an internal webpage on Chrome v81 browser, where google authentication (logging into gmail) is required.
Getting cookies (after manually logging in):
pickle.dump(driver.get_cookies(), open(path_to_cookie, 'wb'))
Using cookies:
driver.get(url)
cookies = pickle.load(open(path_to_cookie, "rb"))
for cookie in cookies:
if 'expiry' in cookie:
del cookie['expiry']
driver.add_cookie(cookie)
time.sleep(2)
driver.get(url)
Problem:
The cookies are stored in a pickle file. There are 2 types of cookies - csrftoken and sessionid.
The sessionid one typically expires in 2 weeks, and even if i change the expiry date of the cookies in the json (in the cookies.pkl file), it will still expire after 2 weeks (and when i run the script it shows the gmail login page).
I've tried using user-data-dir and chrome profile (a duplicate of my existing chrome profile). It works, but the same problem still happens and i need to relogin again after 2 weeks. Any idea how i can make the cookies last for e.g months / a year instead of 2 weeks?
Related
I'm a CS student, and I've learned that (at least in most cases) sessions are stored in cookies, that's why you keep logged in even after a computer restart.
I wanted to see it with my own eyes, so I made a simple experiment. I just logged in a popular website (e.g. youtube, gmail, facebook, etc), and typed document.cookie into the the console then received the content. After that I opened a new browser, went to the same site, and paste the cookie with this command document.cookie = .... I refreshed the page, but it didn't work. What am I missing? Based on my knowledge, it should work.
For security, login cookies should be marked as HTTPonly which sends them with HTTP requests, but prevents them from being read by JavaScript. Using document.cookie in the console isn't going to show all the cookies, especially not the secure login cookies.
To get a list of all the cookies you would have to use your browser's cookie manager that shows the HTTPonly cookies as well.
I want to automate requests on a website and when doing so, I need a session cookie in order to identify myself.
When checking the network tab, I can clearly see the session cookie, but when checking the Application tab, this cookie is not shown. After accessing this website with selenium and calling driver.get_cookies() with Python, I only get the cookies shown in the Application tab.
I need to do this with selenium because this way it's possible to login. Only using requests will not work.
I do not have many clues on how to get that cookie and have almost 0 experience in this field, hence my question.
Is it possible to continue a request session in selenium will all its cookies, i have seen many people doing it the other way arround but i have no clue how to do it proper the way i want.
def open_selenium_session(self):
# get or set cookies
driver = get_chromedriver(self.proxy, use_proxy=True, user_agent=True)
driver.get("https://www.instagram.com")
cookies = driver.get_cookies()
for cookie in cookies:
self.session.cookies.set(cookie['name'], cookie['value'])
Incase you have stored the cookies previesly from an active session using pickle:
import pickle
pickle.dump( driver.get_cookies() , open("cookies.pkl","wb"))
You can always set them back as follows:
# loading the stored cookies
cookies = pickle.load(open("cookies.pkl", "rb"))
for cookie in cookies:
# adding the cookies to the session through webdriver instance
driver.add_cookie(cookie)
Reference
You can find a couple of detailed discussions in:
org.openqa.selenium.InvalidCookieDomainException: Document is cookie-averse using Selenium and WebDriver
selenium.common.exceptions.InvalidCookieDomainException: Message: invalid cookie domain while executing tests in Django with Selenium
I have an Express server using PassportJS authentication. I am using standard XMLHTTPRequests to retrieve files.
Chrome and Firefox work without problems. Safari (Mac and iOS) is failing to send a request cookie with some requests and doing this inconsistently.
Logged failed requests all have a different session id that isn't authenticated with PassportJS.
Chrome and Firefox have a single session id for all requests.
I'm not using cross-origin requests, these are all same-origin. I have tried using withCredentials true with the same result but this property shouldn't be needed anyway as it's same-origin and Chrome and Firefox work without it.
There doesn't seem to be a pattern to which files fail to send the request cookie. Sometimes at random (but rarely), all requests load ok.
It's not a sequential failure, most requests succeed to begin with then a sequence of fails with some successes in between.
The Express server isn't destroying the sessions, they keep accumulating. Failed requests have session ids that are not authenticated sessions.
What would cause Safari to randomly not send the request cookie for some XMLHTTPRequests?
It turned out to be the crossOrigin property, I must have been doing cross-origin requests:
https://developer.mozilla.org/en-US/docs/Web/HTML/CORS_settings_attributes
Most of my loaded files had their crossOrigin property set as "anonymous". For some reason, Chrome and Firefox seemed to ignore this and sent the cookie credentials anyway but Safari didn't. Setting this property to "use-credentials" worked fine:
img.crossOrigin = "use-credentials";
When using the Postman Chrome App with the Interceptor extension it's easy to reuse the browser's cookies in order to log into an app and then call the services within.
Since moving to the Postman standalone app, this process has become somewhat manual. After logging in from the browser, I have to access the JSESSIONID cookie in the developer tools and copy its value over to postman.
When my session expires I need to repeat the process.
I would like to automate this synchronization or at least understand how I could obtain the new authenticated value in postman. It's important to note that none of the authentication mechanisms available in Postman work with my app which is why the manual login in the browser is necessary.
You can get JSESSIONID cookie in Postman Standalone in similar way your browser do it - by send proper requests (probably POST "login" request with user credentials) to server