clojure http-kit failure on ssl - ssl

The trouble I am facing is hard to debug as it doesn't offer much of an explanation in the stack trace. What I am trying to do is retrieve an oAuth token using a key and a secret provided to me. When using postman it works just fine meaning the issue cannot be external. When I try it in Clojure it fails.
See sample code below.
(let [options {:keepalive 10000
:timeout 10000
:query-params {:grant_type "client_credentials"}
:headers {"Authorization" (str "Basic " (-> (str (:key config/oAuth) ":" (:secret config/oAuth)) .getBytes b64/encode String. ))}
}]
#(http/get "https://my-path.com/oauth/v1/generate" options))
the result I get is
1. Unhandled java.io.IOException
An existing connection was forcibly closed by the remote host
SocketDispatcher.java: -2 sun.nio.ch.SocketDispatcher/read0
SocketDispatcher.java: 43 sun.nio.ch.SocketDispatcher/read
IOUtil.java: 223 sun.nio.ch.IOUtil/readIntoNativeBuffer
IOUtil.java: 197 sun.nio.ch.IOUtil/read
SocketChannelImpl.java: 379 sun.nio.ch.SocketChannelImpl/read
HttpsRequest.java: 93 org.httpkit.client.HttpsRequest/doHandshake
HttpClient.java: 133 org.httpkit.client.HttpClient/doRead
HttpClient.java: 377 org.httpkit.client.HttpClient/run
Thread.java: 745 java.lang.Thread/run
IOException java.io.IOException: An existing connection was forcibly closed by the remote host
Below is the curl request sample generates
curl -X GET 'https://my-path.com/oauth/v1/generate' -H 'authorization: Basic Nasjdbajksdkasjdkey:secret==' -H 'cache-control: no-cache' -H 'postman-token: f2af6f4a-f197-20df-f9be-a5a9a7525e57'
I have obviously changed any sensitive information. Now the strange thing is that when I run that curl request it doesn't work despite postman returning expected results.
See curl error below
* Adding handle: conn: 0x1328a58
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x1328a58) send_pipe: 1, recv_pipe: 0
* About to connect() to my-path.com port 443 (#0)
* Trying xxx.xxx.xxx.xxx...
* Connected to my-path.com (xxx.xxx.xxx.xxx) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: E:\Git\bin\curl-ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to my-path.com:443
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to my-path.com:443
Why does it work on postman but not for the exact curl request postman generated?
Why does it not work for my Clojure app, any assistance will be greatly appreciated.

Related

Strange CURL issue with a particular website SSL certificate

I am trying to use CURL to get web pages from a paricualr website however it gives this error:
curl -q -v -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://www.saiglobal.com/ --output ./Downloads/test.html
....
* SSL certificate verify ok.
} [5 bytes data]
> GET / HTTP/1.1
> Host: www.saiglobal.com
> User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
> Accept: */*
>
0 0 0 0 0 0 0 0 --:--:-- 0:11:53 --:--:-- 0* OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
* stopped the pause stream!
0 0 0 0 0 0 0 0 --:--:-- 0:11:53 --:--:-- 0
* Closing connection 0
} [5 bytes data]
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
I am not sure what is going on. I can't find a lot of useful info regarding to the error message. On my Mac, the errorno is 60 instead of 104.
However, using Chrome on these machines can load the page without any issue. One of the machines' CURL version is 7.58.0.
Any help is appreciated.
The problem is not the certificate of this site. From the debug output it can be clearly seen that the TLS handshake is done successfully and outside this handshake the certificate does not matter.
But, it can be seen that the site www.saiglobal.com is CDN protected by Akamai CDN and Akamai features some kind of bot detection:
$ dig www.saiglobal.com
...
www.saiglobal.com. 45 IN CNAME www.saiglobal.com.edgekey.net.
www.saiglobal.com.edgekey.net. 62 IN CNAME e9158.a.akamaiedge.net.
This bot detection is known to use some heuristics in order to distinguish bots from normal browsers and detection of a bot might result in a status code 403 access denied or in a simple hang of the site - see Scraping attempts getting 403 error or Requests SSL connection timeout.
In this specific case it seems to currently help if some specific HTTP headers are added, specifically Accept-Encoding, Accept-Language, Connection with a value of keep-alive and User-Agent which matches somehow Mozilla. Failure to add these headers or having the wrong values will result in a hang.
The following works currently for me:
$ curl -q -v \
-H "Connection: keep-alive" \
-H "Accept-Encoding: identity" \
-H "Accept-Language: en-US" \
-H "User-Agent: Mozilla/5.0" \
https://www.saiglobal.com/
Note that this deliberately tries to bypass the bot detection. It might stop working if Akamai makes changes to the bot detection.
Please note also that the owner of the site has explicitly enable bot detection for a reason. This means that with deliberately bypassing the detection for your own gain (like providing some service based on scraped information) you might get into legal problems.

HTTPS requests to paypal sandbox fail

I can't currently use the PayPal Sandbox to test my shopping-cart integrations. When i access the sandbox in a browser or via curl, all i get is connection errors.
Here's an example curl-session:
curl -v "https://api-3t.sandbox.paypal.com/nvp?user=test"
* Trying 173.0.82.83...
* TCP_NODELAY set
* Connected to api-3t.sandbox.paypal.com (173.0.82.83) port 443 (#0)
* TLS 1.2 connection using TLS_RSA_WITH_AES_256_CBC_SHA256
* Server certificate: api-3t.sandbox.paypal.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> GET /nvp?user=test HTTP/1.1
> Host: api-3t.sandbox.paypal.com
> User-Agent: curl/7.51.0
> Accept: */*
>
* SSLRead() return error -9806
* Curl_http_done: called premature == 1
* Closing connection 0
curl: (56) SSLRead() return error -9806
Any hints if this is a serverside issue or something i can work around?
Thanks!
Looks like the NVP/SOAP integration method of Paypal Express Checkout is deprecated as of January 1, 2017. The live URL (https://api-3t.paypal.com/nvp) seems to work for now, but this might be why the sandbox doesn't work anymore.
I don't exactly understand why, but using POST instead of GET when submitting data to the PayPal Sandbox solves the problem nicely. No more weird SSL-errors.
Found below answer to curl 56 error . Hope that could help work out of the problem.
curl (56) Recv failure
PHP CURL Error - curl: (56) Recv failure: Connection reset by peer

Heroku ssl certficate not switched with custom domain

I have a heroku app (whatever-app-11123) and I want to use a custom domain through ssl (https://myapp.co).
I have bought the certificate and followed all the instructions provided by heroku.
Strange thing is that https://whatever-app-11123 is displaying the myapp.co certificate and https://myapp.co is displaying the *.herokuapp.com certificate.
I need this to be the other way around : myapp.co with the myapp.co certificat.
Any hints on how achieving this?
Results of different commands :
herok:certs
Endpoint Common Name(s) Expires Trusted
--------------------------------- ---------------------------------------- -------------------- -------
whatever-app-11123.herokuapp.com myapp.co, www.myapp.co 2017-02-12 23:59 UTC True
curl -kvI https://www.myapp.co
* Adding handle: conn: 0x7fbd03804000
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fbd03804000) send_pipe: 1, recv_pipe: 0
* About to connect() to www.myapp.co port 443 (#0)
* Trying 46.132.168.142...
* Connected to www.myapp.co (46.132.168.142) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: *.herokuapp.com
* Server certificate: DigiCert SHA2 High Assurance Server CA
* Server certificate: DigiCert High Assurance EV Root CA
> HEAD / HTTP/1.1
> User-Agent: curl/7.30.0
> Host: www.myapp.co
> Accept: */*
>
< HTTP/1.1 302 Found
HTTP/1.1 302 Found
* Server Cowboy is not blacklisted
<
* Connection #0 to host www.myapp.co left intact
Here's the answer: check your DNS redirection. Make sure it is a CNAME type and MOST OF ALL (what I did wrong) the redirection should be :
www.myapp.co -> whatever-app-11123.herokuapp.com
myapp.co -> whatever-app-11123.herokuapp.com
and not
www.myapp.co -> myapp.co
myapp.co -> whatever-app-11123.herokuapp.com

Office 365 REST API to create contacts using CURL gets HTTPCode 400 Bad Request

I am testing Office 365 REST API using CURL following this link:
Contacts REST API in Office 365 APIs Preview
I can obtain correctly one contact using curl command in Windows like this:
curl --no-sessionid --insecure --basic --user "user#domain.com:password" -H "Accept: application/json" "https://outlook.office365.com/EWS/OData/Me/Contacts?$orderby=DisplayName+asc&$top=1"
And following documentation on this link if I try to create one contact using CURL with minimum required options for testing:
curl -X POST -d "{\"#odata.type\": \"#Microsoft.Exchange.Services.OData.Model.Contact\",\"GivenName\": \"TestContact\",\"EmailAddress1\": \"test#test.com\",\"BusinessPhone1\": \"123-456-7890\"}" https://outlook.office365.com/ews/odata/Me/Contacts --header "Content-Type:application/json" --insecure --verbose --user "user#domain.com:password"
I receive following error:
* About to connect() to outlook.office365.com port 443 (#0)
* Trying 157.56.250.178...
* connected
* Connected to outlook.office365.com (157.56.250.178) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
...
* SSL connection using ECDHE-RSA-AES256-SHA
* Server certificate:
...
* Server auth using Basic with user 'user#domain.com'
> POST /ews/odata/Me/Contacts HTTP/1.1
...
> Content-Length: 157
>
* upload completely sent off: 157 out of 157 bytes
< HTTP/1.1 400 Bad Request
...
< Content-Length: 82
<
{"error":{"code":"ErrorInvalidRequest","message":"Cannot read the request body."}}* Connection #0 to host outlook.office365.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
I have searched the internet and in stackoverflow but similar questions have no answer I'm looking for.
How could I create on contact using CURL?, I have tested it on Linux too but have the same results.
The following request works when I try it from Fiddler. Can you please try this out using CURL?
POST https://outlook.office365.com/ews/odata/Me/Contacts HTTP/1.1
Authorization: Basic <XXXX>
Content-Type: application/json
{
"GivenName" : "John",
"EmailAddresses" : [
{ "Address": "John#contoso.com", "Name" : "John" }
],
"BusinessPhones" : [
"123-456-7890"
]
}
I made a few changes to your request. You don't need to specify the OData.type as we infer that you are adding a Contact because you are sending a POST to Contacts collection. We need to fix our documentation as it lists the entity type as required. We have updated our namespace to Microsoft.OutlookServices and hence the type definitions have changed. To make the API easier to use, we have replaced EmailAddress1, EmailAddress2 etc. with a collection of EmailAddresses. Similarly, we have also changed BusinessPhones, HomePhones etc. to collections as well.
As I just explained in another post, the issues you are seeing are from some changes being rolled out to our preview APIs and our documentation is in the process of being updated. The current set of changes include versioning support, and this won't be an issue going forward.
Please let me know if you have any questions or need more info.
Thanks,
Venkat
Using following CURL command worked perfect:
curl -X POST -d "{\"GivenName\":\"John\",\"EmailAddresses\":[{\"Address\":\"John#contoso.com\",\"Name\":\"John\"}],\"BusinessPhones\":[\"123-456-7890\"]}" https://outlook.office365.com/ews/odata/Me/Contacts --header "Content-Type:application/json" --insecure --verbose --user "user#domain.com:password"
Now we can export contacts (this is not allowed from OWA) and import it using simple utilities like curl.
Thanks for your help Venkat.
Your original post showed the following namespace:
Microsoft.Exchange.Services.OData.Model.Contact
That has been changed to:
Microsoft.Office365.OutlookServices.Contact
Since the namespace was wrong, it couldn't read the request body.

Vagrant is not port forwarding with VirtualBox and PuPHPet

Problem
I cannot connect to my virtual machine's environment, despite using an accepted private network and forwarded ports.
Description
System
MAC OSX 10.9.3
Vagrant 1.6.3
VirtualBox 4.3.12
Vagrantfile file
Puphpet file
Cisco AnyConnect VPN
Using Private Company Network
ifconfig results
Virtual Host file for Web Project.
Upon vagrant up, I vagrant ssh into my virtualbox. The following request work as expected:
[06:04 PM]-[vagrant#precise64]-[~]
$ curl -v 192.168.56.101
* About to connect() to 192.168.56.101 port 80 (#0)
* Trying 192.168.56.101... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: 192.168.56.101
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 09 Jul 2014 18:05:05 GMT
< Server: Apache/2.4.9 (Ubuntu)
< Vary: Accept-Encoding
< Content-Length: 481
< Connection: close
< Content-Type: text/html;charset=UTF-8
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
<head>
<title>Index of /</title>
</head>
<body>
<h1>Index of /</h1>
<table>
<tr><th valign="top"><img src="/icons/blank.gif" alt="[ICO]"></th><th>Name</th><th>Last modified</th><th>Size</th><th>Description</th></tr>
<tr><th colspan="5"><hr></th></tr>
<tr><th colspan="5"><hr></th></tr>
</table>
</body></html>
* Closing connection #0
[06:05 PM]-[vagrant#precise64]-[~]
$ curl -v playworldsystems.dev
* About to connect() to playworldsystems.dev port 80 (#0)
* Trying 192.168.56.101... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: playworldsystems.dev
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 09 Jul 2014 18:05:52 GMT
< Server: Apache/2.4.9 (Ubuntu)
< X-Powered-By: PHP/5.5.14-2+deb.sury.org~precise+1
< Vary: Accept-Encoding
< Content-Length: 122
< Connection: close
< Content-Type: text/html
<
<pre class='xdebug-var-dump' dir='ltr'><small>string</small> <font color='#cc0000'>'hello'</font> <i>(length=5)</i>
* Closing connection #0
</pre>
However, when I try both commands from within my HOST terminal, I receive the following error:
curl -v 192.168.56.101
* About to connect() to 192.168.56.101 port 80 (#0)
* Trying 192.168.56.101...
* Adding handle: conn: 0x7fc6f1000000
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fc6f1000000) send_pipe: 1, recv_pipe: 0
* Failed connect to 192.168.56.101:80; Operation timed out
* Closing connection 0
curl: (7) Failed connect to 192.168.56.101:80; Operation timed out
☁ ~ curl -v playworldsystems.dev
* Adding handle: conn: 0x7fdbe9803000
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fdbe9803000) send_pipe: 1, recv_pipe: 0
* About to connect() to playworldsystems.dev port 80 (#0)
* Trying 192.168.56.101...
* Failed connect to playworldsystems.dev:80; Operation timed out
* Closing connection 0
curl: (7) Failed connect to playworldsystems.dev:80; Operation timed out
Even after trying cURLing from both port 6969 and 8080, I still have no success. I've used random IP Addresses as well as Ports. I've tried altering my virtual host to other port numbers. Nothing seems to work. I will mention that when first starting out, I noticed that this vagrant setup has worked --both only twice. Each time it was working, i would vagrant suspend and vagrant up the next morning to find that my prior solution is no longer working.
Perhaps this fact is what makes this process so frustrating. I want to get of MAMP for my current projects but fear that it is either my work machine's settings that are interfering or perhaps some other application or network related issue. I'm unsure what steps to take and am looking forward to any and all solutions.
a few tips/pointers.
check that the firewall rules are not blocking the ports, if all else fails, just disable it for testing "iptables -F"
make sure ports are forwarded correctly https://docs.vagrantup.com/v2/networking/forwarded_ports.html
so what you want for http is host:8080 guest:80
sometimes your IP address changes on your host ( for example, acquiring new dhcp or joining new network home/work/vpn )
check that you can ping your VM guest from your host via the correct IP from the correct adapter. On the host run "ifconfig -a" (on the mac there should be some en0 for ethernet and en1 for airport) if you can ping the vm guest, then you can hit the web server via port 8080 ( http://mylocal.dev:8080/ ) provided of course that you edited your /etc/hosts to point your vhost to the above mentioned IP address.