Is there a way or setting in scrapy to ignore pages having an basic http authentication while the crawling is in progress.
Thanks
Related
We have our Apache Websites secured using the password security which works well. Ive noticed If I add the following header to my browser requests I can bypass the security Authorization: Basicxxxxxxxxx. However, many other websites I visit use this same headers which requires me to always disble this header before visiting other sites.
Is there a way to configure Apache to recognize an additional headers to bypass security, so I can store this header in my browser settings and be able to visit other sites without having to disable it.
Is there a way to write a cloudflare rewrite/proxy similar to netlify as per https://docs.netlify.com/routing/redirects/rewrites-proxies/#proxy-to-another-service and adjust the header with basic auth credentials?
For example I would need /api/* to redirect to a WordPress at domain.com/wp-json/* with basic auth headers (ideally not visible to client itself - not sure if this is also possible)
I'm trying to fetch articles from https://journals.sagepub.com/, the website is accessible though my browser but I keep getting a 503 error when I try to crawl in scrapy shell. When I view the response in browser it shows the generic cloudflare ddos protection page. I have tried changing user agents and download delay but nothing works. I am new to scrapy and web scraping in general so some help would be much appreciated.
I have created a vue js app for frontend development.
In our case, we will be getting auth info in authorization header from some other web app.
I dont understand how can i fetch authorization header from URL.
I know we can fetch params from URL.
Can anyone please help me with this - How can i fetch authorization header from URL.
Thanks & Regards,
Jyoti
I'm pretty sure it's not possible for client-side JS to access headers in the request that initially loads the page. You can add headers to outbound requests made by client-side JS, but only the server getting the request has access to the headers. If it makes the information in that header available to the client, either as a cookie or in the source of the page, you can work with it, but depending on your application, that might be a security risk (e.g. for replay attacks).
I have two different domains/sites, one http and one https. The http site requires login and then users are shown a non-secure page with an iframe that shows content from the https site.
I would like to seamlessly pass login credentials from the http site to the iframe'd https site. I do not want to use jquery. Is it possible to have the iframe use a POST request instead of GET? I would like to encrypt the login info from the http site and POST the encrypted bundle to the https site.
I'm working with php and apache, if it makes a difference.
I found this article on iframe/POST but wasn't sure how to get the form auto-submitted when the http page loads. Maybe that's a line of javascript? Also, while the login page on the http site has a login form, the post-login page that has the iframe on it does not (currently) have a form. Maybe I could make an invisible form to create this POST request?
Thanks!
You can use Javascript like this:
document.getElementById('someId').submit();