using curl in sas: oauth request from api - api

I'm trying to use curl in SAS to download data from an api. I have a Consumer key,Consumer secret,OAuth Token, & OAuth Token Secret. It appears that I can connect to the api but I get an error stating that "HTTP Transport: Couldn't determine the content length". Does anyone have any insights or thoughts? Thanks so much.
Code:
dm 'clear log';
options;
%let consumer_key =;
%let consumer_secret=;
%let oauth_token=;
%let oauth_token_secret=;
filename curl pipe "curl -X POST -k https://api.tradeking.com/v1/market/ext/quotes.xml?symbols=aapl --verbose --header
'Authorization:OAuth oauth_nonce=4572616e48616d6d65724c61686176,oauth_timestamp=1359019570,oauth_version=1.0,
oauth_signature_method=HMAC-SHA1,oauth_consumer_key={&consumer_key.},oauth_consumer_key_secret={&consumer_secret.},
oauth_token={&oauth_token.},oauth_token_secret={&oauth_token_secret.}";
data _null_;
infile curl lrecl=32767;
input;
put _infile_;
run;
Log:
dm 'clear log';
options;
636
637 %let consumer_key =;
638 %let consumer_secret=;
639 %let oauth_token=;
640 %let oauth_token_secret=;
641
642 filename curl pipe "curl -X POST -k
642! https://api.tradeking.com/v1/market/ext/quotes.xml?symbols=aapl --verbose --header
643 'Authorization:OAuth
643! oauth_nonce=4572616e48616d6d65724c61686176,oauth_timestamp=1359019570,oauth_version=1.0,
WARNING: The quoted string currently being processed has become more than 262 characters long.
You might have unbalanced quotation marks.
644 oauth_signature_method=HMAC-SHA1,oauth_consumer_key={&consumer_key.},oauth_consumer_ke
644! y_secret={&consumer_secret.},
645 oauth_token={&oauth_token.},oauth_token_secret={&oauth_token_secret.}";
646
647 data _null_;
648 infile curl lrecl=32767;
649 input;
650 put _infile_;
651 run;
NOTE: The infile CURL is:
Unnamed Pipe Access Device,
PROCESS=curl -X POST -k https://api.tradeking.com/v1/market/ext/quotes.xml?symbols=aapl
--verbose --header 'Authorization:OAuth
oauth_nonce=4572616e48616d6d65724c61686176,oauth_timestamp=1359019570,oauth_version=1.0,
oauth_signature_method=HMAC-SHA1,oauth_consumer_key={
JXpV},oauth_consumer_key_secret={},
oauth_token={},oauth_token_secret={},
RECFM=V,LRECL=32767
Fault Name: HttpRequestReceiveError
Error Type: Default
Description: Http request received failed
Root Cause Code: -19013
Root Cause : HTTP Transport: Couldn't determine the content length
Binding State: CLIENT_CONNECTION_ESTABLISHED
Service: null
Endpoint: null
Operation (Client):
--_curl_--oauth_signature_method=HMAC-SHA1,oauth_consumer_key=
--_curl_--oauth_token=,oauth_token_secret=Q
Stderr output:
* Adding handle: conn: 0x1ea8850
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x1ea8850) send_pipe: 1, recv_pipe: 0
Total Received Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to
connect() to api.tradeking.com port 443 (#0)
* Trying 206.132.7.9...
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to
api.tradeking.com (206.132.7.9) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0* SSLv3, TLS
handshake, Finished (20):
{ [data not shown]
* SSL connection using RC4-MD5
* Server certificate:
* subject: OU=Domain Control Validated; OU=COMODO SSL Wildcard; CN=*.tradeking.com
* start date: 2013-06-09 00:00:00 GMT
* expire date: 2016-06-08 23:59:59 GMT
* issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO SSL CA
* SSL certificate verify result: self signed certificate in certificate chain (19),
continuing anyway.
> POST /v1/market/ext/quotes.xml?symbols=aapl HTTP/1.1
> User-Agent: curl/7.33.0
> Host: api.tradeking.com
> Accept: */*
> 'Authorization:OAuth
>
< HTTP/1.1 411 Length Required
< Connection: close
< Content-Length: 284
<
{ [data not shown]
100 284 100 284 0 0 178 0 0:00:01 0:00:01 --:--:-- 178
* Closing connection 0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]
* Rebuilt URL to:
oauth_nonce=4572616e48616d6d65724c61686176,oauth_timestamp=1359019570,oauth_version=1.0,/
* Adding handle: conn: 0x1f00560
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 1 (0x1f00560) send_pipe: 1, recv_pipe: 0
* Could not resolve host:
oauth_nonce=,oauth_timestamp=1359019570,oauth_version=1.0,
* Closing connection 1
curl: (6) Could not resolve host:
oauth_nonce=4572616e48616d6d65724c61686176,oauth_timestamp=1359019570,oauth_version=1.0,
[1/3]:
oauth_signature_method=HMAC-SHA1,oauth_consumer_key=
--> <stdout>
* Rebuilt URL to:
oauth_signature_method=HMAC-SHA1,oauth_consumer_key=
/
* Adding handle: conn: 0x1ef10e0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 2 (0x1ef10e0) send_pipe: 1, recv_pipe: 0
* Could not resolve host:
oauth_signature_method=HMAC-SHA1,oauth_consumer_key=
* Closing connection 2
curl: (6) Could not resolve host:
oauth_signature_method=HMAC-SHA1,oauth_consumer_key=
[1/2]:
oauth_token=,oauth_token_secret=
--> <stdout>
* Rebuilt URL to:
oauth_token=,oauth_token_secret=
/
* Adding handle: conn: 0x1ef56c0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 3 (0x1ef56c0) send_pipe: 1, recv_pipe: 0
* Could not resolve host:
oauth_token=,oauth_token_secret=
* Closing connection 3
curl: (6) Could not resolve host:
oauth_token=,oauth_token_secret=
NOTE: 12 records were read from the infile CURL.
The minimum record length was 0.
The maximum record length was 174.
NOTE: DATA statement used (Total process time):
real time 1.84 seconds
cpu time 0.23 seconds

The first thing to do is to try running your cURL statement from the command line. If you get a valid response it's a SAS issue, otherwise it's an issue with your cURL request.
Have you tried adding the -k option to your cURL statement seeing as you're making an https request?
Oh also, do any of the macro variables contain values that include ampersands, percent symbols, double quotes, or single quotes? These could all cause issues - especially ampersands as they need to be further masked with a caret symbol ^.

Related

pushbullet curl w/ access token doesn't work

I generated an access token at https://www.pushbullet.com/#settings/account but it doesn't seem to work:
$ curl --http1.1 --header "Access Token: $PUSHBULLET_API_TOKEN" https://api.pushbullet.com/v2/users/me | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 176 100 176 0 0 1093 0 --:--:-- --:--:-- --:--:-- 1086
{
"error": {
"code": "invalid_access_token",
"type": "invalid_request",
"message": "Access token is missing or invalid.",
"cat": "(=^・ω・^)y="
},
"error_code": "invalid_access_token"
}
Any suggestions for what might be going wrong? The only deviation from the example in the API docs is the http1.1 option, which i added because http2.0 support seems broken:
$ curl --header "Access Token: $PUSHBULLET_API_TOKEN" https://api.pushbullet.com/v2/users/me | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 1555 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

Open 443 port for ssl

I have installed ssl certificate on nginx using certbot, I am able to access https://URL from browser only if i stop iptables, So I checked if 443 port was open
I created rule in iptables to open 80 and 443 ports:
iptables -A INPUT -p tcp -m multiport --dports 80,443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp -m multiport --dports 80,443 -m conntrack --ctstate ESTABLISHED -j ACCEPT
service iptables save
service iptables restart
and checked by nginx:
$netstat -anltp
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 27432/nginx
...........
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 27432/nginx
in mysite.conf file
server {
listen 443 ssl;
}
and nginx
$ nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
and iptables..
iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
133 9875 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:53
3543 271K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:587
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:110
1496 131K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:143
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:110
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:143
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:993
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:995
2274 3696K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
1 56 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
45 2700 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:25
53 3288 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
180 32695 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F
0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:!0x17/0x02 state NEW
0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:25
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:465
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:110
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:995
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:143
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:993
0 0 DROP all -- * * 58.218.204.189 0.0.0.0/0
0 0 DROP all -- * * 58.218.204.189 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:53
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:53
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:443
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 ctstate NEW,ESTABLISHED
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443 ctstate NEW,ESTABLISHED
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT 6686 packets, 10M bytes)
pkts bytes target prot opt in out source destination
133 9875 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:53
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp spt:443 ctstate ESTABLISHED
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443 ctstate ESTABLISHED
but still I am getting connection refused in browser if iptables service is running, Nothing in nginx logs as it seems firewall issue.
nginx version: nginx/1.12.0
Centos 6, uname -a Linux server.name 2.6.32-696.1.1.el6.x86_64 #1 SMP Tue Apr 11 17:13:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Any help? thanks
I had issue with iptables I should have used -I instead of -A and SSL worked, as mentioned by Leader at Let's Encrypt Community

iptables recent needs a time-limited --set option, or how else can I solve this?

For a port-knocking scheme, I'm wondering how to make the iptables recent module temporarily (for just a few seconds) list-name matched source addresses. My intuition tells me that I need the --set function of the recent module to accept the --seconds option, making the list assignment temporary, but all I can see is the way to assign the list name to an address permanently and have another rule remove the list-name assignment only upon receipt of some later packet. The reason that doesn't fit into my mental concept is because the removal (--reap or --remove, I suppose) of the address from the list will only occur upon reception of a future packet, whereas my intuition wants the address removed at a certain time expiration regardless of whether any packets arrive to trigger said name removal. The only way I can see to do something vaguely similar to this is very non-intuitive to me, and therefore suspicious to me that I'm missing something about how it all works: I would need a recent module rule and --rcheck option to ensure the listed packet's name matches and that it had gotten assigned within the previous x seconds and remove it with a jump destination, the rule in that jump destination would be to assign the next list name to the source address. In the meantime, the length of the lists just keeps growing (don't they?), filling up with stray source addresses that never completed the knock sequence[s]. What a simple solution it would be for the recent module to accept the --seconds option with --set! Can anyone help me help me see this more clearly?
( I've looked at other knocking solutions using iptables, but they are limited to only using each port-protocol combination for one knock in the sequence, while a good knocking solution should, IMHO, allow for the same port-protocol combo to be used as many times in the knock sequence as the user wants it to be used. knockd had that same limitation, as well as exhibiting terrible non-robust operation. I tried to obtain the pknock module for iptables, but it appears that not all its components exist [specifically two shell scripts referred to in the documentation, knock.sh and knock-orig.sh, supposedly "found in doc/pknock/util", wherever that is...certainly not SF, Github, nor anywhere else I could see], making me very suspicious of using it.)
EDIT: I'm seeing that the ruleset needs to be even more complex than described - the knock steps numbered two and above all need to match the packet by name first, then jump it to their own chain that removes the name, determines whether the correct timing and port-protocol matches, then jump it to yet another chain to rename it, or don't jump it if timing or knock is off which falls into a drop rule. WHEW
My initial solution is shown below. The knocks in this example just happen to be unique, but non-unique knocks will function fine as well. As you can see, I make every knocking packet reap every list because I don't know if the lists self-limit the length of time they'll keep entries otherwise. It seems like the only way to ensure that no list can get too long.
$--> iptables -wnvL
Chain INPUT (policy DROP)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- $internal_net_interface * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 ctstate RELATED,ESTABLISHED /* extract ssh for knock testg frm private side in ssh */
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: authorized side: source mask: 255.255.255.255 ctstate NEW /* 1-packet pass: 1 chance to establish or then knock higher */
0 0 knockerstest all -- $internal_net_interface * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 200 name: knocker side: source mask: 255.255.255.255 /* for knock capability */
0 0 knockstage1 tcp -- $internal_net_interface * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1 flags:0x17/0x02 recent: SET name: knocker side: source mask: 255.255.255.255 /* for knock capability, 1st port */
0 0 knockers all -- $external_net_interface * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 200 name: knocker side: source mask: 255.255.255.255 ctstate NEW /* for port knock capability */
0 0 knockstage1 tcp -- $external_net_interface * 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:1 flags:0x17/0x02 recent: SET name: knocker side: source mask: 255.255.255.255 /* for port knock capability, 1st port */
Chain knockerreap (10 references)
pkts bytes target prot opt in out source destination
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 60 reap name: knocker side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage1 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage2 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage3 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage4 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage5 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage6 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage7 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage8 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage1 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage2 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage3 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage4 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage5 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage6 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage7 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 12 reap name: knockstage8 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage1 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage2 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage3 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage4 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage5 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage6 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage7 side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knockstage8 side: source mask: 255.255.255.255
Chain knockers (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0 ! ctstate NEW /* for port knock capability */
0 0 knockersort all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* for port knock capability */
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4 prefix "pktfail:knock|late|ctstate "
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 60 reap name: knocker side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knocker side: source mask: 255.255.255.255
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockersort (2 references)
pkts bytes target prot opt in out source destination
0 0 knockstage2 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2 flags:0x17/0x02 recent: CHECK seconds: 12 name: knockstage2 side: source mask: 255.255.255.255 /* knock to stage 2 successful */
0 0 knockstage3 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3 flags:0x17/0x02 recent: CHECK seconds: 12 name: knockstage3 side: source mask: 255.255.255.255 /* knock to stage 3 successful */
0 0 knockstage4 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:4 flags:0x17/0x02 recent: CHECK seconds: 12 name: knockstage4 side: source mask: 255.255.255.255 /* knock to stage 4 successful */
0 0 knockstage5 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5 flags:0x17/0x02 recent: CHECK seconds: 12 name: knockstage5 side: source mask: 255.255.255.255 /* knock to stage 5 successful */
0 0 knockstage6 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:6 flags:0x17/0x02 recent: CHECK seconds: 12 name: knockstage6 side: source mask: 255.255.255.255 /* knock to stage 6 successful */
0 0 knockstage7 tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:7 flags:0x17/0x02 recent: CHECK seconds: 12 name: knockstage7 side: source mask: 255.255.255.255 /* knock to stage 7 successful */
Chain knockerstest (1 references)
pkts bytes target prot opt in out source destination
0 0 knockersort all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4 prefix "knockertest fail "
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: CHECK seconds: 60 reap name: knocker side: source mask: 255.255.255.255
0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 recent: REMOVE name: knocker side: source mask: 255.255.255.255
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage1 (2 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage2 side: source mask: 255.255.255.255 /* Entry in log makes blacklisting get delayed until after knocking time window expires */ LOG flags 0 level 4 prefix "knocked: Stage1 "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage2 (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage3 side: source mask: 255.255.255.255 /* Entry in log makes blacklisting get delayed until after knocking time window expires */ LOG flags 0 level 4 prefix "knocked: Stage2 "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage3 (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage4 side: source mask: 255.255.255.255 /* Entry in log makes blacklisting get delayed until after knocking time window expires */ LOG flags 0 level 4 prefix "knocked: Stage3 "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage4 (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage5 side: source mask: 255.255.255.255 /* Entry in log makes blacklisting get delayed until after knocking time window expires */ LOG flags 0 level 4 prefix "knocked: Stage4 "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage5 (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage6 side: source mask: 255.255.255.255 /* Entry in log makes blacklisting get delayed until after knocking time window expires */ LOG flags 0 level 4 prefix "knocked: Stage5 "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage6 (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: knockstage7 side: source mask: 255.255.255.255 /* Entry in log makes blacklisting get delayed until after knocking time window expires */ LOG flags 0 level 4 prefix "knocked: Stage6 "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain knockstage7 (1 references)
pkts bytes target prot opt in out source destination
0 0 knockerreap all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 recent: SET name: authorized side: source mask: 255.255.255.255 /* allows time-limited access */ LOG flags 0 level 4 prefix "knock full success "
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0

Aerospike connection errors

We run Aerospike server 3.5.15-1 on Ubuntu 14.04 and periodically getting server connection errors from PHP clients ([-1]Unable to connect to server). PHP client version 3.4.1. We run PHP 5.3 clients from a separate server node. Connections created from php-fpm.
There are no any corresponding errors in the server logs and server didn't have to be restarted. So, the problem seem to be on the client side.
This application creates up to 400 simultaneous connections to Aerospike. We use r3.xlarge EC2 instance and server has plenty of available resources.
We followed Aerospike tuning documentation and tried updating proto-fd and recommended OS patameters on the server, but it didn't help
proto-fd-max 100000
proto-fd-idle-ms 15000
That's how we initialize and use Aerospike:
$opts = array(Aerospike::OPT_CONNECT_TIMEOUT => 1250,Aerospike::OPT_WRITE_TIMEOUT => 5000);
$this->db = new Aerospike($config, false, $opts);
//set key
$aero_key = $this->db->initKey($this->keyspace, $this->table, $key);
$aero_value = array("value" => $value);
$status = $this->db->put($aero_key, $aero_value, $ttl, $options);
//get key
$aero_key = $this->db->initKey($this->keyspace, $this->table, $key);
$status = $this->db->get($aero_key, $result);
Aerospike server stats before the disconnect:
Aug 27 2015 19:32:50 GMT: INFO (info): (thr_info.c::4828) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: bq 0 ::: iq 0 ::: dq 0 : fds - proto (237, 16073516, 16073279) : hb (0, 0, 0) : fab (16, 16, 0)
Aug 27 2015 19:33:00 GMT: INFO (info): (thr_info.c::4828) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: bq 0 ::: iq 0 ::: dq 0 : fds - proto (334, 16076516, 16076182) : hb (0, 0, 0) : fab (16, 16, 0)
Aug 27 2015 19:33:10 GMT: INFO (info): (thr_info.c::4828) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: bq 0 ::: iq 1 ::: dq 0 : fds - proto (288, 16079478, 16079190) : hb (0, 0, 0) : fab (16, 16, 0)
Aug 27 2015 19:33:20 GMT: INFO (info): (thr_info.c::4828) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: bq 0 ::: iq 0 ::: dq 0 : fds - proto (131, 16082477, 16082346) : hb (0, 0, 0) : fab (16, 16, 0)
Aug 27 2015 19:33:30 GMT: INFO (info): (thr_info.c::4828) trans_in_progress: wr 0 prox 0 wait 0 ::: q 0 ::: bq 0 ::: iq 0 ::: dq 0 : fds - proto (348, 16084665, 16084317) : hb (0, 0, 0)
From the log segment, we can see that there are around 300 client connections open on the node at any one time, well under the 100000 limit in proto-fd-max.
If you are using multicast for heartbeats (and I think you are), the heartbeats of 0 are fine.
I expect that you have already looked at this, but are you able to check network connectivity between the client and server at the time of the failure? I know that under normal conditions, the client and the server happily coexist, but at the time of the failure, do you see any basic connectivity problems?
Do you happen to have other applications installed on the client machine? Do they have any similar failures, possibly at the time of the Aerospike client problems?
Do you have the client installed on more than one server? Do you maybe only see the connectivity errors on one of the servers?
I know you have already been looking at this, so I apologize if I am covering topics that you have already reviewed.
Thank you for your time,
-DM

Verify errorcode = 20 : unable to get local issuer certificate

I have a certificate chain in server:
Certificate chain
0 s:/******/O=Foobar International BV/OU****
i:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/****
1 s:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/****
i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=**** - G5
2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=**** - G5
i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
And my local root CA certificate is:
s:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/****
i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=**** - G5
And I am using this snippet to verify the certificate:
//gcc -lssl -lcrypto -o certverify certverify.c
#include <openssl/ssl.h>
#include <openssl/asn1.h>
#include <openssl/bio.h>
#include <openssl/x509.h>
#include <openssl/x509_vfy.h>
#include <openssl/pem.h>
#include <openssl/x509v3.h>
#include <openssl/err.h>
#include <openssl/conf.h>
#include <string.h>
int main() {
const char ca_bundlestr[] = "./ca-bundle.pem";
const char cert_filestr[] = "./cert-file.pem";
BIO *certbio = NULL;
BIO *outbio = NULL;
X509 *error_cert = NULL;
X509 *cert = NULL;
X509_NAME *certsubject = NULL;
X509_STORE *store = NULL;
X509_STORE_CTX *vrfy_ctx = NULL;
int ret;
/* ---------------------------------------------------------- *
* These function calls initialize openssl for correct work. *
* ---------------------------------------------------------- */
OpenSSL_add_all_algorithms();
ERR_load_BIO_strings();
ERR_load_crypto_strings();
/* ---------------------------------------------------------- *
* Create the Input/Output BIO's. *
* ---------------------------------------------------------- */
certbio = BIO_new(BIO_s_file());
outbio = BIO_new_fp(stdout, BIO_NOCLOSE);
/* ---------------------------------------------------------- *
* Initialize the global certificate validation store object. *
* ---------------------------------------------------------- */
if (!(store=X509_STORE_new()))
BIO_printf(outbio, "Error creating X509_STORE_CTX object\n");
/* ---------------------------------------------------------- *
* Create the context structure for the validation operation. *
* ---------------------------------------------------------- */
vrfy_ctx = X509_STORE_CTX_new();
/* ---------------------------------------------------------- *
* Load the certificate and cacert chain from file (PEM). *
* ---------------------------------------------------------- */
ret = BIO_read_filename(certbio, cert_filestr);
if (! (cert = PEM_read_bio_X509(certbio, NULL, 0, NULL))) {
BIO_printf(outbio, "Error loading cert into memory\n");
exit(-1);
}
ret = X509_STORE_load_locations(store, ca_bundlestr, NULL);
if (ret != 1)
BIO_printf(outbio, "Error loading CA cert or chain file\n");
/* ---------------------------------------------------------- *
* Initialize the ctx structure for a verification operation: *
* Set the trusted cert store, the unvalidated cert, and any *
* potential certs that could be needed (here we set it NULL) *
* ---------------------------------------------------------- */
X509_STORE_CTX_init(vrfy_ctx, store, cert, NULL);
/* ---------------------------------------------------------- *
* Check the complete cert chain can be build and validated. *
* Returns 1 on success, 0 on verification failures, and -1 *
* for trouble with the ctx object (i.e. missing certificate) *
* ---------------------------------------------------------- */
ret = X509_verify_cert(vrfy_ctx);
BIO_printf(outbio, "Verification return code: %d\n", ret);
if(ret == 0 || ret == 1)
BIO_printf(outbio, "Verification result text: %s\n",
X509_verify_cert_error_string(vrfy_ctx->error));
/* ---------------------------------------------------------- *
* The error handling below shows how to get failure details *
* from the offending certificate. *
* ---------------------------------------------------------- */
if(ret == 0) {
/* get the offending certificate causing the failure */
error_cert = X509_STORE_CTX_get_current_cert(vrfy_ctx);
certsubject = X509_NAME_new();
certsubject = X509_get_subject_name(error_cert);
BIO_printf(outbio, "Verification failed cert:\n");
X509_NAME_print_ex(outbio, certsubject, 0, XN_FLAG_MULTILINE);
BIO_printf(outbio, "\n");
}
/* ---------------------------------------------------------- *
* Free up all structures *
* ---------------------------------------------------------- */
X509_STORE_CTX_free(vrfy_ctx);
X509_STORE_free(store);
X509_free(cert);
BIO_free_all(certbio);
BIO_free_all(outbio);
exit(0);
}
But this code return following output:
Verification return code: 0
Verification result text: unable to get issuer certificate
Verification failed cert:
countryName = US
organizationName = Symantec Corporation
organizationalUnitName = Symantec Trust Network
commonName = Symantec Class 3 Secure Server CA - G4
What's wrong here?
Your root CA uses probably the same public key as the first intermediate CA in chain (below the host certificate) and you have probably no root-CA which can be used to trust the last chain certificate. Such setups are not very common, but actually happen. Unfortunately OpenSSL has problems with this setup and will only try to verify the longest chain, even if a shorter chain provides already the necessary trust.
There is a bug entry for this OpenSSL problem, but nobody from the OpenSSL developers ever took care of it. You can also find a patch if you are looking for X509_V_FLAG_TRUSTED_FIRST. It looks like that OpenSSL 1.0.2 (not yet released) will have this option too.
From my understanding only OpenSSL has this kind of problem, i.e. neither NSS (Firefox, Chrome on Desktop) nor SChannel (Microsoft).
I think Steffen probably helped you solve the problem. But here's a small nitpick that may have side stepped the bug you are experiencing and improved your security posture.
const char ca_bundlestr[] = "./ca-bundle.pem";
You don't need the CA bundle. You only need Verisign's Class 3 Public Primary Certification Authority (G5). You can get the one CA cert needed from Verisign at Use of Root Certificates.
Its an improvement in your security posture because you're allowing any CA to certify the server's certificate (even wrong ones), and not using the one known to certify the server's certificate (Verisign).
And I am using this snippet to verify the certificate ...
If you want to see an example of a simple TLS client, then check out SSL/TLS Client on the OpenSSL wiki. It provides an example of fetching random numbers from random.org. It won't take much work to change it to example.com.
Note well: OpenSSL does not perform hostname matching during validation. You still need to do it yourself if you are using OpenSSL 1.0.2, 1.0.1, 1.0.0 and lesser versions. OpenSSL provides hostname matching in 1.1.0, but its not available yet.
The sample code to extract the hostnames from the Common Name (CN) and Subject Alt Names (SAN) in the X.509 certificate is provided in the SSL/TLS Client, but you will have to provided the actual matching code.
Based on the information in the comments, you need the certificate: "Symantec Class 3 Secure Server CA - G5". Below is what it looks like when providing the proper anchor - it ends in a Verify return code: 0 (ok) (and not the error 20).
The "Symantec Class 3 Secure Server CA - G5" is the one with the fingerprint 4e b6 d5 78 49 9b 1c cf 5f 58 1e ad 56 be 3d 9b 67 44 a5 e5. You can fetch it from Verisign's Use of Root Certificates.
The CAfile option being used by s_client (below) is set inside s_client.c with a call to SSL_CTX_load_verify_locations. Its set to the one CA needed to certify the server's certificate, and not the CA Zoo (i.e., cacerts.pem).
You can check the Subject Alternate Names (SAN) in the certificate with $ openssl s_client -connect www.smartbabymonitor.ugrow.example.com:443 | openssl x509 -text -noout. You will be OK because the host www.smartbabymonitor.ugrow.example.com is listed in the SAN. You could even add the -servername option to the command to use Server Name Indication (SNI).
$ openssl s_client -showcerts -connect www.smartbabymonitor.ugrow.example.com:443 -CAfile VeriSign-Class\ 3-Public-Primary-Certification-Authority-G5.pem
CONNECTED(00000003)
depth=3 C = US, O = "VeriSign, Inc.", OU = Class 3 Public Primary Certification Authority
verify return:1
depth=2 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5
verify return:1
depth=1 C = US, O = Symantec Corporation, OU = Symantec Trust Network, CN = Symantec Class 3 Secure Server CA - G4
verify return:1
depth=0 C = NL, ST = Netherlands, L = Eindhoven, O = Example International BV, OU = Consumer Lifestyle, CN = smartbabymonitor.ugrow.example.com
verify return:1
---
Certificate chain
0 s:/C=NL/ST=Netherlands/L=Eindhoven/O=Example International BV/OU=Consumer Lifestyle/CN=smartbabymonitor.ugrow.example.com
i:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
-----BEGIN CERTIFICATE-----
MIIF+DCCBOCgAwIBAgIQa0fyuH2bp1ucngiNHVoV4jANBgkqhkiG9w0BAQsFADB+
MQswCQYDVQQGEwJVUzEdMBsGA1UEChMUU3ltYW50ZWMgQ29ycG9yYXRpb24xHzAd
...
+eGxGqm8e1jgxB/fQePrh1vG4V40nr0cBKh6t52HmksBCfM0wOlMMJyUYiO0p44W
s4nxNrvMJS6e4bwdECI0UNhJznWr0tAu+ilFoTsfOlQpngCBDJEkZYr3mRjpIjX8
Sz4+hGzIhZVyjDvbcVCrsvCpM67cU2rQpJ2nkYM4ol/z6VDRs/G5aPiXe7o=
-----END CERTIFICATE-----
1 s:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
-----BEGIN CERTIFICATE-----
MIIFODCCBCCgAwIBAgIQUT+5dDhwtzRAQY0wkwaZ/zANBgkqhkiG9w0BAQsFADCB
yjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
...
QGX0InLNmfiIEfXzf+YzguaoxX7+0AjiJVgIcWjmzaLmFN5OUiQt/eV5E1PnXi8t
TRttQBVSK/eHiXgSgW7ZTaoteNTCLD0IX4eRnh8OsN4wUmSGiaqdZpwOdgyA8nTY
Kvi4Os7X1g8RvmurFPW9QaAiY4nxug9vKWNmLT+sjHLF+8fk1A/yO0+MKcc=
-----END CERTIFICATE-----
2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
-----BEGIN CERTIFICATE-----
MIIE0DCCBDmgAwIBAgIQJQzo4DBhLp8rifcFTXz4/TANBgkqhkiG9w0BAQUFADBf
MQswCQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsT
...
A4GBABMC3fjohgDyWvj4IAxZiGIHzs73Tvm7WaGY5eE43U68ZhjTresY8g3JbT5K
lCDDPLq9ZVTGr0SzEK0saz6r1we2uIFjxfleLuUqZ87NMwwq14lWAyMfs77oOghZ
tOxFNfeKW/9mz1Cvxm1XjRl4t7mi0VfqH5pLr7rJjhJ+xr3/
-----END CERTIFICATE-----
---
Server certificate
subject=/C=NL/ST=Netherlands/L=Eindhoven/O=Example International BV/OU=Consumer Lifestyle/CN=smartbabymonitor.ugrow.example.com
issuer=/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
---
No client certificate CA names sent
---
SSL handshake has read 4805 bytes and written 434 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: F1B9C9DFA3CFC6CB3F958FAD4ECBBAFA0E72EA8A86F6AC9601CF8204819DB0F0
Session-ID-ctx:
Master-Key: EC4C5B32E60B5A0458BC85CC02529EA18DE61AFB8583D85D275C2822AC84E0E5E0C5B5E2C3C2D90F8B6E0EBB518EAA99
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 27 e0 fb b8 dd c9 9f 29-31 85 2b 6c d1 5a b3 d1 '......)1.+l.Z..
0010 - 55 d6 e4 8a 4d f5 ef 2e-51 95 21 90 47 9d b6 0a U...M...Q.!.G...
0020 - df a5 d2 10 3d 03 e5 07-41 81 92 09 30 0e 08 3d ....=...A...0..=
0030 - fc ea 24 93 29 ed 60 9a-d0 d9 57 88 e4 4d 18 e3 ..$.).`...W..M..
0040 - ba aa 97 ee bf 39 9e 5b-76 5b 76 f7 81 c4 03 08 .....9.[v[v.....
0050 - fb b9 a3 4f 11 b0 99 4c-8c f2 a6 8a 9a e4 fe c6 ...O...L........
0060 - 0d 7b 6d a7 5b 53 b5 33-15 4f c4 ab 6b 29 7b 8f .{m.[S.3.O..k){.
0070 - ec 00 7f b2 6f 91 e4 ca-63 45 58 73 3a 78 8b 29 ....o...cEXs:x.)
0080 - 44 fc d5 e8 ad 4d dd 9c-22 df 50 eb d5 bf b9 90 D....M..".P.....
0090 - d8 6a 7d 6d bd 61 f2 63-07 75 8b d0 fc 40 64 76 .j}m.a.c.u...#dv
00a0 - 2b 97 53 aa 47 bc 3d d1-76 aa 8a 07 e1 60 14 d1 +.S.G.=.v....`..
00b0 - f7 88 8f f6 d9 b9 6b 0c-64 96 b5 f0 46 73 27 d6 ......k.d...Fs'.
Start Time: 1419835334
Timeout : 300 (sec)
Verify return code: 0 (ok)
---