Tweepy: Trying to detect a university filter or firewall - ssl

I think my university recently changed a firewall or filtering protocol with regards to Twitter's streaming API. However, for me to provide them more information, I'm going to have to figure out the details.
Ok, below is the simplest possible program to collect data from Twitter's stream:
from tweepy import Stream
from tweepy import OAuthHandler
from tweepy.streaming import StreamListener
ckey="[OMITED]"
csecret="[OMITED]"
atoken="[OMITED]"
asecret="[OMITED]"
class listener(StreamListener):
def on_data(self, data):
print(data)
return(True)
def on_error(self, status):
print status
auth = OAuthHandler(ckey, csecret)
auth.set_access_token(atoken, asecret)
twitterStream = Stream(auth, listener())
twitterStream.filter(track=["car"])
When I run this program on my laptop/home desktop, it works fine. However, I have a university computer setup for data collection and it produces the following error:
Traceback (most recent call last):
File "test.py", line 25, in <module>
twitterStream.filter(track=["car"])
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 228, in filter
self._start(async)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 172, in _start
self._run()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tweepy/streaming.py", line 106, in _run
conn.connect()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1278, in connect
server_hostname=server_hostname)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 352, in wrap_socket
_context=self)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 579, in __init__
self.do_handshake()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 808, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
Now, the thing is, I can open up Safari and browse the web. Further, data collection like this use to work (as of a month ago). The error says the SSL cert is failing verification (but strangely the original OAuth seems to work -- it is the filter command kicking up the error). Is there a way to get it to spit out if that is because an intermediate cert has been put in place? Or maybe a port is being blocked? Basically, is there a way to get a lot more details so I can go the university and say: 'you are blocking X'?
Any suggestions?

I've resolved the situation. It was an error relating to SSL sites that were signed with Digicerts.

Related

Getting (insecure_transport) OAuth 2 MUST utilize https with CERT managed by Heroku. I have a subdomain attached pointed to Server

I'm trying to user google sheets API service, which requires an HTTPS connection. I'm getting the following error
Exception Type: InsecureTransportError at my_site/google/success/
Exception Value: (insecure_transport) OAuth 2 MUST utilize https.
I am using Heroku, and on my settings it says AMC Status: ok. I verified that I'm using HTTPS by running curl -vI https://my_site/google/success
which returned:
SSL certificate verify ok
From my perspective it seems that I am using HTTPS, but I am getting this error. What could I be doing wrong? Surely I have something misconfigured Anything else I need to provide from troubleshooting? Here is the Full traceback:
traceback (most recent call last):
File "/app/.heroku/python/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
File "/app/.heroku/python/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/app/google_api/views.py", line 56, in authorize_success
flow.fetch_token(authorization_response=code)
File "/app/.heroku/python/lib/python3.8/site-packages/google_auth_oauthlib/flow.py", line 286, in fetch_token
return self.oauth2session.fetch_token(self.client_config["token_uri"], **kwargs)
File "/app/.heroku/python/lib/python3.8/site-packages/requests_oauthlib/oauth2_session.py", line 239, in fetch_token
self._client.parse_request_uri_response(
File "/app/.heroku/python/lib/python3.8/site-packages/oauthlib/oauth2/rfc6749/clients/web_application.py", line 203, in parse_request_uri_response
response = parse_authorization_code_response(uri, state=state)
File "/app/.heroku/python/lib/python3.8/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 256, in parse_authorization_code_response
raise InsecureTransportError()
import os
os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = '1'

fail to create a connection with nessus server

I am trying to get a connection with the Nessus server with the bellow command in python but it failed with an error message can you tell me what can be the cause. I have checked my network connection it is fine.
requests.post( 'https://164.99.175.30:8834/'+ '/session',data={'username':'admin','password':'micro#123'},verify=False)```
error message
Traceback (most recent call last):
File "nessus.py", line 425, in <module>
login()
File "nessus.py", line 111, in login
res = requests.post(url + '/session',data={'username':username,'password':password},verify=verify)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='localhost', port=8834): Max retries exceeded with url: /session (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f46f2d6d410>: Failed to establish a new connection: [Errno 111] Connection refused',))
The nessus api is depreciated as of version 7.x, this is the best source I could find.
EDIT: I have found a better source directly from tenable
What has been removed from Nessus 7:
There is a restriction in scan API capabilities.
The ability to manage scans via API and CLI has been removed in v7. All Nessus Pro scanning operations must be done through the user interface.
So currently the ability of the Nessus API is as follows:
Removed the ability to run scans or reports and create new objects
The Read features, where the ability to pull scan data so GET /scan/scan ID now works again and this aids with some of the integration processes.
https://community.tenable.com/s/article/The-differences-between-Nessus-6-and-Nessus-7
This is only for Nessus pro versions

Is something weird happening to wrapdb.mesonbuild.com?

I am using meson build system, but this morning I have a problem with
wrapdb.mesonbuild.com.
With my computer and another one (for checking), when using a web browser like Chromium or Firefox, I get messages like:
Your connection is not private
Attackers might be trying to steal your information from wrapdb.mesonbuild.com
(for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_DATE_INVALID
(I have checked my computer time etc... with ntpdate... so I priori this is not the origin of the problem)
Also any attempt to use:
meson wrap list
fails, with output like:
SSL connection failed. Falling back to unencrypted connections.
Traceback (most recent call last):
File "/usr/lib/python3.7/urllib/request.py", line 1317, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/usr/lib/python3.7/http/client.py", line 1422, in connect
server_hostname=server_hostname)
File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
session=session
File "/usr/lib/python3.7/ssl.py", line 870, in _create
self.do_handshake()
File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)
Can you reproduce? I cannot find the source of the problem.
update: it seems that wrapdb.mesonbuild.com has already been down in the past:
https://github.com/mesonbuild/meson/issues/3497
maybe this is my current problem.
update 2: I did not have the reflex but with
https://www.sslshopper.com/ssl-checker.html#hostname=wrapdb.mesonbuild.com
one can see that the site's certificate expired yesterday...
Also thanks to #TomEsposito for pointing out this.
It appears their SSL Certificate expired yesterday, but they are still using HTTPS. Until they update their certificate, or move to an HTTP connection, your browser will continue to warn you of this. All modern browsers now warn users when a site is using HTTPS but there is no certificate

ncclient: connecting to a NETCONF server

I want use the python library ncclient 0.6.6 with Python 2.7.15 to connect to a NETCONF server (netopeer2) and read out the running config.
I tried to follow the example from the manual, running this code in the console:
with manager.connect(host="*the IP adress*", port=*the port*, timeout=None, username="*user*", password="*pwd*") as m:
c = m.get_config(source='running').data_xml
with open("%s.xml" % host, 'w') as f:
f.write(c)
As written in the manual, I try to disable public-key authentification with allow_agent and look_for_keys as False. Unfortunately, this does not work properly, because I get the error message:
File "<stdin>", line 1, in <module>
File "/home/sisc/.local/lib/python2.7/site-packages/ncclient/manager.py", line 177, in connect
return connect_ssh(*args, **kwds)
File "/home/sisc/.local/lib/python2.7/site-packages/ncclient/manager.py", line 143, in connect_ssh
session.connect(*args, **kwds)
File "/home/sisc/.local/lib/python2.7/site-packages/ncclient/transport/ssh.py", line 481, in connect
raise SSHUnknownHostError(known_hosts_lookup, fingerprint)
ncclient.transport.errors.SSHUnknownHostError: Unknown host key [e3:8d:35:a9:43:f9:3c:8a:f4:d3:88:5b:a9:36:93:59] for [[192.168.56.2]:1831]
I do not get why it still complains about the unknown host key, even though I explicitly disabled public-key authentification.
The netopeer NETCONF server is definitely running, for I get a "Hello" Message as soon as I try to SSH into it from out of the terminal.
Did I miss something?
m = manager.connect(host="172.17.0.2", port=830, username="netconf", password="netconf", hostkey_verify=False)
Did the trick. Hostkey_verify has to be false.

How to authenticate using Firebase with endpoints-proto-datastore?

I've been trying to set up my EndpointsV2 backend (billing has been enabled) with Firebase authentication but am always ending up with either an error when I attempt to generate the OpenAPI spec or a 401 Unauthorized when accessing the API.
My current configuration is as follows (errors out on OpenAPI generation):
main.py:
firebase_issuer = endpoints.Issuer(
issuer='https://securetoken.google.com/[GAE/FIREBASE_APP_ID]',
jwks_uri=('https://www.googleapis.com/service_accounts/v1/metadata/x509/'
'securetoken#system.gserviceaccount.com'))
issuers = {
"firebase": firebase_issuer
}
#endpoints.api("user", "v1", api_key_required=True,
issuers=issuers)
class UserApi(remote.Service):
#models.UserModel.query_method(
user_required=True,
path="info",
name="user.info",
http_method="GET")
def info(self, query):
return query.filter(
models.UserModel.owner == endpoints.get_current_user())
Error on OpenAPI Gen:
$ python lib/endpoints/endpointscfg.py get_openapi_spec main.UserApi --hostname [SERVICE-NAME].endpoints.[GAE_APP_ID].appspot.com
No handlers could be found for logger "endpoints.apiserving"
Traceback (most recent call last):
File "lib/endpoints/endpointscfg.py", line 633, in <module>
main(sys.argv)
File "lib/endpoints/endpointscfg.py", line 629, in main
args.callback(args)
File "lib/endpoints/endpointscfg.py", line 487, in _GenOpenApiSpecCallback
application_path=args.application)
File "lib/endpoints/endpointscfg.py", line 332, in _GenOpenApiSpec
application_path=application_path)
File "lib/endpoints/endpointscfg.py", line 214, in GenApiConfig
services, hostname=hostname))
File "D:\Users\Will\Documents\Projects\CardApp\backend\lib\endpoints\openapi_generator.py", line 973, in pretty_print_config_to_json
descriptor = self.get_openapi_dict(services, hostname)
File "D:\Users\Will\Documents\Projects\CardApp\backend\lib\endpoints\openapi_generator.py", line 959, in get_openapi_dict
return self.__api_openapi_descriptor(services, hostname=hostname)
File "D:\Users\Will\Documents\Projects\CardApp\backend\lib\endpoints\openapi_generator.py", line 872, in __api_openapi_descriptor
security_definitions)
File "D:\Users\Will\Documents\Projects\CardApp\backend\lib\endpoints\openapi_generator.py", line 691, in __method_descriptor
service.api_info.audiences, security_definitions)
File "D:\Users\Will\Documents\Projects\CardApp\backend\lib\endpoints\openapi_generator.py", line 722, in __x_security_descriptor
_INVALID_AUTH_ISSUER % default_auth_issuer)
api_exceptions.ApiConfigurationError: No auth issuer named google_id_token defined in this Endpoints API.
The error leads me to think that if I simply swap "firebase" for "google_id_token"in issuers that it may work but I have tried that and just gotten a 401.
I've also started to see this error in GAE Logging:
Cannot decode and verify the auth token. The backend will not be able to retrieve user info (/base/data/home/apps/s~[GAE/FIREBASE_PROJECT]/20170318t232908.399940418908006950/lib/google/api/control/wsgi.py:588)
Traceback (most recent call last):
File "/base/data/home/apps/s~[GAE/FIREBASE_PROJECT]/20170318t232908.399940418908006950/lib/google/api/control/wsgi.py", line 585, in __call__
service_name)
File "/base/data/home/apps/s~[GAE/FIREBASE_PROJECT]/20170318t232908.399940418908006950/lib/google/api/auth/tokens.py", line 83, in authenticate
"allow provider id: " + provider_id)
UnauthenticatedException: The requested method does not allow provider id: google_id_token
The idToken I am sending with my testing (done through Postman using the header Authorization: Bearer [idToken]) was generated using pyrebase and I checked jwt.io to confirm that it is in fact a valid token.
My initial thought is that I haven't set up the audiences correctly, but I'm not sure what it should be (though I have tried appending a ClientID from API Manager > Credentials).
Is it possible to authenticate with firebase using endpoints-proto-datastore on the endpointsv2 framework? If so, what do I need to do differently to get that working? If not, would the code that I have work right away without using endpoints-proto-datastore?
Edit:
Looks like this is an issue with the endpoints library itself
https://github.com/cloudendpoints/endpoints-python/issues/32