Proxy Error when Requests connecting from HTTPS - ssl

I have been testing a site with a social media login validation. It's quite primitive aside from that I built in the ability to connect through a proxy. This works fine, including the proxy, when I run it from my box or the server the site is on. However, once I enabled https/ssl on my server and used the proxy, it stopped working with this error:
File "myprogram.py", line 274, in login
r = self.s.get(self.url)
File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 476, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 464, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 424, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='socialmediawebsite', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(111, 'Connection refused')))
I've read a solid amount and have looked through the code for requests and urllib3, but it seems more and more like I have a fundamental misunderstanding. I've tried HTTPAuth, setting verify to False, and setting environment variables. I have also tried without the proxy, and it works, which has added to my confusion. It seems to me like there might be another piece of technology required to send this https request through the proxy. Or is it just that I have to open up more ports somewhere or something equally simple?
There is one question similar to mine here: Python Requests doesnt work for https proxy
But the accepted answer is factually inaccurate and didn't work for me.
What my code looks like:
proxies = {
'http': 'http://user:pass#10.10.1.10:3128',
'https': 'http://user:pass#10.10.1.10:3128',
}
self.s.update(proxies)
self.s.get(self.url)

Related

Getting (insecure_transport) OAuth 2 MUST utilize https with CERT managed by Heroku. I have a subdomain attached pointed to Server

I'm trying to user google sheets API service, which requires an HTTPS connection. I'm getting the following error
Exception Type: InsecureTransportError at my_site/google/success/
Exception Value: (insecure_transport) OAuth 2 MUST utilize https.
I am using Heroku, and on my settings it says AMC Status: ok. I verified that I'm using HTTPS by running curl -vI https://my_site/google/success
which returned:
SSL certificate verify ok
From my perspective it seems that I am using HTTPS, but I am getting this error. What could I be doing wrong? Surely I have something misconfigured Anything else I need to provide from troubleshooting? Here is the Full traceback:
traceback (most recent call last):
File "/app/.heroku/python/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/app/.heroku/python/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/app/google_api/views.py", line 56, in authorize_success
flow.fetch_token(authorization_response=code)
File "/app/.heroku/python/lib/python3.8/site-packages/google_auth_oauthlib/flow.py", line 286, in fetch_token
return self.oauth2session.fetch_token(self.client_config["token_uri"], **kwargs)
File "/app/.heroku/python/lib/python3.8/site-packages/requests_oauthlib/oauth2_session.py", line 239, in fetch_token
self._client.parse_request_uri_response(
File "/app/.heroku/python/lib/python3.8/site-packages/oauthlib/oauth2/rfc6749/clients/web_application.py", line 203, in parse_request_uri_response
response = parse_authorization_code_response(uri, state=state)
File "/app/.heroku/python/lib/python3.8/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 256, in parse_authorization_code_response
raise InsecureTransportError()
import os
os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = '1'

ncclient: connecting to a NETCONF server

I want use the python library ncclient 0.6.6 with Python 2.7.15 to connect to a NETCONF server (netopeer2) and read out the running config.
I tried to follow the example from the manual, running this code in the console:
with manager.connect(host="*the IP adress*", port=*the port*, timeout=None, username="*user*", password="*pwd*") as m:
c = m.get_config(source='running').data_xml
with open("%s.xml" % host, 'w') as f:
f.write(c)
As written in the manual, I try to disable public-key authentification with allow_agent and look_for_keys as False. Unfortunately, this does not work properly, because I get the error message:
File "<stdin>", line 1, in <module>
File "/home/sisc/.local/lib/python2.7/site-packages/ncclient/manager.py", line 177, in connect
return connect_ssh(*args, **kwds)
File "/home/sisc/.local/lib/python2.7/site-packages/ncclient/manager.py", line 143, in connect_ssh
session.connect(*args, **kwds)
File "/home/sisc/.local/lib/python2.7/site-packages/ncclient/transport/ssh.py", line 481, in connect
raise SSHUnknownHostError(known_hosts_lookup, fingerprint)
ncclient.transport.errors.SSHUnknownHostError: Unknown host key [e3:8d:35:a9:43:f9:3c:8a:f4:d3:88:5b:a9:36:93:59] for [[192.168.56.2]:1831]
I do not get why it still complains about the unknown host key, even though I explicitly disabled public-key authentification.
The netopeer NETCONF server is definitely running, for I get a "Hello" Message as soon as I try to SSH into it from out of the terminal.
Did I miss something?
m = manager.connect(host="172.17.0.2", port=830, username="netconf", password="netconf", hostkey_verify=False)
Did the trick. Hostkey_verify has to be false.

Google EarthEngine Python in VirtualEnv: requests can connect to google.com but ee.Initialize() fails with SSL get certificate error

Trying to get a working installation of the Google EarthExplorer. I am inside a robust corporate firewall, and had to examine the certificate used by Chrome to verify the server/address being verified (using Chrome's dev tools).
After some reading - it looks like the situation is that:
1) when calling Initialize method on ee object, ee uses requests to manage the connection.
2) To configure the VirtualEnv correctly, I had to configure my virtualenv to use the organization provided certificate. Based on this SE (SE Python SSL Requests...) I was clued in to the fact that the python stack was using requests, which along with certifi manage a cert bundle for SSL on python.
3) After configuring the supplied certificate (matching that used with Chrome), I can open a connection to google inside my VirtualEnv using requests. Great!
(earthengine) X:\_01_VirtualEnvs\earthengine>python
Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("https://earthengine.google.com/", verify = True)
<Response [200]>
>>>
I think this is 'working' now ... when I try the same outside of the VirtualEnv, I get failed SSL3_GET_SERVER_CERTIFICATE... message. This lets me know that I got my certificate in the right place, and it seems to be working correctly.
However, I'm still getting errors on the ee.Initialize():
>>> ee.Initialize()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "X:\_01_VirtualEnvs\earthengine\lib\site-packages\ee\__init__.py", line 9
3, in Initialize
ApiFunction.initialize()
File "X:\_01_VirtualEnvs\earthengine\lib\site-packages\ee\apifunction.py", lin
e 151, in initialize
signatures = data.getAlgorithms()
File "X:\_01_VirtualEnvs\earthengine\lib\site-packages\ee\data.py", line 410,
in getAlgorithms
return send_('/algorithms', {}, 'GET')
File "X:\_01_VirtualEnvs\earthengine\lib\site-packages\ee\data.py", line 738,
in send_
response, content = send_with_backoff()
File "X:\_01_VirtualEnvs\earthengine\lib\site-packages\ee\data.py", line 735,
in send_with_backoff
'Unexpected HTTP error: %s' % e.message)
ee.ee_exception.EEException: Unexpected HTTP error: [Errno 1] _ssl.c:510: error:
14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Anyone have any ideas on what is going wrong here? I thought that ee was using requests, therefore setting the python environments' requests properly.
From looking at the exception trace-back, it seems that ee has sub-class of requests based on the similarity of the trace-back signature... am I reading this right? The trace-back in both cases points towards the same _ssl.c:510 failure - failing on the same filename on the same line?
Is there a way to get more info from the exception? I'm really at a loss at this point.
So it looks like the EE stack is using httplib2 to handle the authentication. This gives a few options:
In data.py ~ line 700 Override the SSL check (the quick and dirty):
http = httplib2.Http(timeout=(_deadline_ms / 1000.0) or None,
disable_ssl_certificate_validation=True)
It looks like you should be able to explicitly direct httplib2 when instantiating the http connection object with:
HTTPLIB_CA_CERTS_PATH = os.environ.get('HTTPLIB_CA_CERTS_PATH')
http = httplib2.Http(timeout=(_deadline_ms / 1000.0) or None,
ca_certs=HTTPLIB_CA_CERTS_PATH)
I found this in the ca_certs_locator module, __init__.py. It is being sourced in the ee.Initialize() method (probably through something in data.py but I can't back track it. Regardless, the second option (explicitly passing the ca_certs path) doesn't solve the problem.
I'm rolling with the disable SSL validation, and using only earthengine.google.com endpoint.
In more recent versions, the ee.Initialize() method is accepting a http_transport argument, so we no longer need to modify its source code, but rather create it in our own:
_http_transport = httplib2.Http(disable_ssl_certificate_validation=True)
ee.Initialize(credentials, http_transport=_http_transport)
This way you can also control the ca_certs option, but I haven't tried that one.

Botocore - how to handle failover condition

I have more than 4 servers up and running s3-auth server on them.
I am able to authenticate user from first server.
if we turn off first server, how to validate that s3 server is not running on first server through botocore, so it should automatically use next server.
When i turn off first server and send request to list users, response never comes from botocore. It retry operation for 5 times and after that it do nothing.
Botocore Version: 1.3.30
Boto3 version: 1.2.2
Please help on this.
See below botocore logs:
DEBUG:botocore.endpoint:Response received to retry, sleeping for 7.72060814652 seconds
DEBUG:botocore.hooks:Event request-created.iam.ListUsers: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f238444ccd0>>
DEBUG:botocore.auth:Calculating signature using v4 auth.
DEBUG:botocore.auth:CanonicalRequest:
POST
/
host:iam.test.com:8085
user-agent:Boto3/1.2.2 Python/2.7.5 Linux/3.10.0-229.11.1.el7.x86_64 Botocore/1.3.30
x-amz-date:20170322T131218Z
host;user-agent;x-amz-date
b6359072c78d70ebee1e81adcbab4f01bf2c23245fa365ef83fe8f1f955085e2
DEBUG:botocore.auth:StringToSign:
AWS4-HMAC-SHA256
20170322T131218Z
20170322/us-east-1/iam/aws4_request
e74bb593aaf7d92c8dfb517a4daedfe353ec4f9806a7b1c50bca7d7ed2e9e45e
DEBUG:botocore.auth:Signature:
adc96eb87a11f5b69214163bfafab7a82574113be0bcb0cddb43dfa70cbbc789
DEBUG:botocore.endpoint:Sending http request: <PreparedRequest [POST]>
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTP connection (5): iam.test.com
DEBUG:botocore.endpoint:ConnectionError received when sending HTTP request.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/botocore/endpoint.py", line 174, in _get_response
proxies=self.proxies, timeout=self.timeout)
File "/usr/lib/python2.7/site-packages/botocore/vendored/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/botocore/vendored/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(111, 'Connection refused'))
DEBUG:botocore.hooks:Event needs-retry.iam.ListUsers: calling handler <botocore.retryhandler.RetryHandler object at 0x7f23843a2790>

How to handle https traffic using libmproxy?

I want to implement a proxy server that intercepts both http and https requests. I came across libmproxy (http://mitmproxy.org/doc/scripting/libmproxy.html) that it is SSL-capable. I start with this simplest proxy that just prints the headers of all requests and responses, and forwards them to clients and servers normally.
#!/usr/bin/env python
from libmproxy import controller, proxy
import os
class Master(controller.Master):
def __init__(self, server):
controller.Master.__init__(self, server)
self.stickyhosts = {}
def run(self):
try:
return controller.Master.run(self)
except KeyboardInterrupt:
self.shutdown()
def handle_request(self, msg):
print "handle request.................................................."
print msg.headers
msg.reply()
def handle_response(self, msg):
print "handle response................................................."
print msg.headers
msg.reply()
config = proxy.ProxyConfig(
cacert = os.path.expanduser("~/.mitmproxy/mitmproxy-ca.pem")
)
server = proxy.ProxyServer(config, 1234)
m = Master(server)
m.run()
Then I configure http and ssl proxy in firefox to 127.0.0.1 port 1234. http seems to work fine as I can see all the headers are printed out. However, when the browser sends https requests, the proxy server does not print anything at all, and the browser displays "the connect was interrupted" error.
Further investigation reveals that the https requests go though the proxy server but not controller.Master. I see that proxy.ProxyHandler.establish_ssl() is being called when there is an https request, but the request does not go though controller.Master.handle_request(). Despite that establish_ssl() is called, the browser does not seem to get any response back. I test this with https://www.google.com.
First, how can I make proxy.ProxyHandler works properly with https requests/responses? Second, how can I modify controller.Master so that it can intercept https requests? I'm also open to other tools that I can build a custom http/https proxy server on top of.
You need to install the mitmproxy CA in the browser you are testing with.
Please see details here ("Installing the mitmproxy CA" section):
http://mitmproxy.org/doc/ssl.html
This solved the problem for me.