I am trying to connect to a REST endpoint via the GetHTTP Processor in NiFi 1.5.0.
The problem that I am faceing is, that the SSL certificate is issued to the domain but I only have direct access to the IP:Port address (company firewall).
With that I run into the problem that host name and certificate owners don't match up and the IP is not added as subject alternative name.
When I try to connect, I get this error message:
javax.net.ssl.SSLPeerUnverifiedException: Certificate for
<[IP-ADDRESS]> doesn't match any of the subject alternative names: []
Is there a way to bypass the host name verification?
I have found this NiFi Jira ticket but it doesn't seem to be addressed yet. Is there a workaround I could use?
You could try using InvokeHttp and use the "Trusted Hostname" property.
As the "Trusted Hostname" property is deprecated in recent versions of NiFi you can use the ExecuteScript processor with Ruby. The example is below. The body of the POST request must be in FlowFile contents. The body of the response will be in FlowFile contents after the processor.
require "uri"
require "net/http"
require "openssl"
java_import org.apache.commons.io.IOUtils
java_import java.nio.charset.StandardCharsets
java_import org.apache.nifi.processor.io.StreamCallback
# Define a subclass of StreamCallback for use in session.read()
class JRubyStreamCallback
include StreamCallback
def process(inputStream, outputStream)
text = IOUtils.toString(inputStream, 'utf-8')
url = URI("https://...")
https = Net::HTTP.new(url.host, url.port)
https.use_ssl = true
https.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Post.new(url)
request["Authorization"] = "Basic ..."
request["Content-Type"] = "application/json"
request.body = text
response = https.request(request)
outputStream.write((response.read_body).to_java.getBytes(StandardCharsets::UTF_8))
end
end
jrubyStreamCallback = JRubyStreamCallback.new
flowFile = session.get()
if flowFile != nil
flowFile = session.write(flowFile, jrubyStreamCallback)
session.transfer(flowFile, REL_SUCCESS)
end
Related
I have the below function in my code, which works perfectly fine when I'm not behind any proxy. In fact, without even mentioning the certifi default CA certificate, it works fine if I pass verify=TRUE, I guess, because it works in the same way.
def reverse_lookup(lat, long):
cafile=certifi.where()
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile)
#response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=True) <-- this works as well
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
When I run the same code from within my enterprise infrastructure (where I'm behind proxy), I make some minor changes in the code mentioning the proxy as parameter in requests method:
def reverse_lookup(lat, long):
cafile=certifi.where()
proxies = {"https" : "https://myproxy.com"}
params={'lat' : float(lat), 'lon' : float(long), 'format' : 'json',
'accept-language' : 'en', 'addressdetails' : 1}
response = requests.get("https://nominatim.openstreetmap.org/reverse", params=params, verify=cafile, proxies=proxies)
result = json.loads(response.text)
return result['address']['country'], result['address']['state'], result['address']['city']
But it gives me one out of these 3 SSL errors at different times, if I set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
Only time it works is when I completely bypass the SSL verification with verify=False
My questions are:
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
Any help is appreciated. Code tested in both Python 2.7.15 and 3.9
Since I'm sending the https request via proxy, is it ok if I bypass SSL verification ?
Do you need the protection offered by HTTPS, i.e. encryption of the application data (like passwords, but also the full URL) to protect against sniffing or modifications by a malicious man in the middle? If you don't need the protection, then you can bypass certificate validation.
How to make the default context of SSL verification work in this case, when I'm behind proxy ?
The proxy is doing SSL interception and when doing this issues a new certificate for this site based on an internal CA. If this is expected (i.e. not an attack) then you need to import the CA from the proxy as trusted with verify='proxy-ca.pem'. Your IT department should be able to provide you with the proxy CA.
But it gives me one out of these 3 SSL errors at different times, if I
set verify=True or verify=certifi.where():
CERTIFICATE_VERIFY_FAILED
UNKNOWN_PROTOCOL
WRONG_VERSION_NUMBER
It should only give you CERTIFICATE_VERIFY_FAILED. The two other errors indicate wrong proxy settings, typically setting https_proxy to https://... instead of http://... (which also can be seen in your code).
I am trying to proxy requests using lua-resty-http module. I am currently passing headers, body, request method, and url for http request which works fine. But when a file upload post request comes it is unable to do so, obviously i haven't configured it to do so. So I want proxy that type of request also. Here is a snap of my so far code, what changes should i make so that it extract file upload data and sends it using lua-resty-http module ->
local http = require "resty.http"
local cjson = require "cjson"
local httpc = http.new()
local path = ngx.var.request_uri
local passHeader = {["cookie"]=ngx.req.get_headers()["cookie"]}
passHeader["content-type"] = ngx.req.get_headers()["content-type"]
ngx.req.read_body();
body = ngx.req.get_body_data();
local original_req_uri = "https://" .. "fakehost.com" .. path
local req_method = ngx.req.get_method()
local res, err = httpc:request_uri(original_req_uri, {
method = req_method,
ssl_verify = false,
keepalive_timeout = 60000,
headers = passHeader,
keepalive_pool = 10,
body = body
})
Read the docs!
https://github.com/openresty/lua-nginx-module#ngxreqget_body_data
POST request may contain big body and nginx may write it to a disk file.
If the request body has been read into disk files, try calling the ngx.req.get_body_file function instead.
PS: IMO approach to proxy HTTP request by Lua is not optimal, because it is the full buffered way. For me it make sense only if we need to issue a subrequest(s). For the main path with most requests processed I recommend to use proxy_pass.
I was able to setup the CherryPy HTTPServer to require an SSL client certificate using the following code:
ssl_certificate = os.environ.get("SSL_CERTIFICATE")
ssl_adapter = BuiltinSSLAdapter(
certificate=ssl_certificate,
private_key=os.environ["SSL_PRIVATE_KEY"],
certificate_chain=os.environ.get("SSL_CERTIFICATE_CHAIN")
)
verify_mode = ssl.CERT_REQUIRED
ssl_adapter.context.verify_mode = verify_mode
HTTPServer.ssl_adapter = ssl_adapter
Now I am trying to get the SSL client certification information from my request handler, but I can't figure how. After reading https://github.com/cherrypy/cheroot/blob/master/cheroot/ssl/builtin.py#L419 it seems that the wsgi environment variables should be populated with SSL_CLIENT* variables. I could not find any method/property from the request objectwould allow me to fetch such information
How can I obtain this variables from a request handler ?
I have learned the answer from a conversation with CherryPy maintainers on Gitter.
An CherryPy request object may contain more attributes than those documented in the API source, such attributes are set dynamically once the request object is created as part of the WSGI handling:
https://github.com/cherrypy/cherrypy/blob/master/cherrypy/_cpwsgi.py#L319
...
request.multithread = self.environ['wsgi.multithread']
request.multiprocess = self.environ['wsgi.multiprocess']
request.wsgi_environ = self.environ
...
Knowing this, to obtain the WSGI environment which includes the SSL* variables, we just need to access it by importing the request object:
import cherrypy.request
...
print(cherrypy.request.wsgi_environ)
...
I'm trying to connect to a site, e.g www.yahoo.com via proxy using python requests library. So, I define proxy settings as:
HOST = 'host'
PORT = 1234 # random port I have used here
USER = 'user'
PASS = 'password'
PROXY = "%s:%s#%s:%d" % (USER, PASS, HOST, PORT)
PROXY_DICT = {
"http" : 'http://' + PROXY,
"https" : 'https://' + PROXY,
}
I use the following line of code:
requests.get('http://www.yahoo.com', proxies=proxy_dict)
This doesn't raise an exception but the response text is an error page from the proxy saying "Ensure you have installed the certificate". I have a certificate "certificate.crt", which runs fine when used with chrome browser. And the certificate is self-signed. I have tried a couple of things which raise errors.
When used the crt file as a verify param, following error:
SSLError: [Errno bad ca_certs: 'certificate.crt'] []
When used the crt file as a cert param, following error:
Error: [('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_certificate_file', 'PEM lib')]
I managed to get a .pem file(I'm not sure but, it might have been generated using a key and a crt file) as well. When using it with cert param, it doesn't throw error, but the response text is again having the text "...Ensure that the certificate is installed..."
When used .pem file with verify param, following error:
SSLError: [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]
Now when I refer to requests docs, I see I can use two parameters verify and cert. What shall I use here? And how?
I have a rake task that pulls and parses JSON data over an SSL connection from an external API.
I use a gem that wraps this external API and have no problems running locally, but the task fails when run on heroku with #<Curl::Err::SSLCaertBadFile: Curl::Err::SSLCaertBadFile>
I installed the piggyback SSL add-on, hoping it might fix it, but no dice.
Any ideas?
UPDATE
I managed to fix it by disabling ssl verification on the curl request previously set by the following two fields:
request.ssl_verify_peer
request.ssl_verify_host
I don't know enough about SSL to know exactly why the error was caused by these settings in a heroku environment or what the implications of disabling this are, aside from reduced security.
It is a bad idea to disable certificate checking. See http://www.rubyinside.com/how-to-cure-nethttps-risky-default-https-behavior-4010.html, http://jamesgolick.com/2011/2/15/verify-none..html and associated references for more on that topic.
The issue is that your HTTP client doesn't know where to find the CA certificates bundle on heroku.
You don't mention what client you are using, but here is an example for using net/https on heroku:
require "net/https"
require "uri"
root_ca_path = "/etc/ssl/certs"
url = URI.parse "https://example.com"
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = (url.scheme == "https")
if (File.directory?(root_ca_path) && http.use_ssl?)
http.ca_path = root_ca_path
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
http.verify_depth = 5
end
request = Net::HTTP::Get.new(url.path)
response = http.request(request)
Here is an example using Faraday:
Faraday.new "https://example.com", ssl: { ca_path: "/etc/ssl/certs" }
Good luck.