curl does not have same result as browser - ssl

I am trying to download the following URL using from command line using curl. If the same URL is requested through the browser, it is able to fetch the image. But for curl, the server terminates the SSL handshake. Just to use exact same parameters; I tried the curl commands from 'developer tools' from google-chrome and firefox. But both fails with following error.
This question was asked before here here but there is no answer that works. I tried -http1.1 as suggested but did not work.
https://floridakeyswebcams.tv/sloppycam/camarchive/0807.jpg
(base) (15:39 test#testcomp ~) > curl -v 'https://floridakeyswebcams.tv/sloppycam/camarchive/0807.jpg' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.9,hi;q=0.8,mr;q=0.7' -H 'If-None-Match: "90cbf2d5a81d51:da782"' -H 'If-Modified-Since: Fri, 03 May 2019 12:07:53 GMT' --compressed
* Trying 74.209.245.140...
* TCP_NODELAY set
* Connected to floridakeyswebcams.tv (74.209.245.140) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /home/sagham/anaconda2/ssl/cacert.pem
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to floridakeyswebcams.tv:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to floridakeyswebcams.tv:443

it appears that floridakeyswebcams.tv requires TLS1.3 support, use the --tlsv1.3 argument,
curl --tlsv1.3 -v 'https://floridakeyswebcams.tv/sloppycam/camarchive/0807.jpg'
here is what i get when using --tlsv1.2:
$ ./CURL.EXE https://floridakeyswebcams.tv/sloppycam/camarchive/0807.jpg --tlsv1.2 -vv
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 74.209.245.140...
* TCP_NODELAY set
* Connected to floridakeyswebcams.tv (74.209.245.140) port 443 (#0)
* schannel: next InitializeSecurityContext failed: SEC_E_ALGORITHM_MISMATCH (0x80090331) - The client and server cannot communicate, because they do not possess a common algorit
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Closing connection 0
* schannel: shutting down SSL/TLS connection with floridakeyswebcams.tv port 443
curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_ALGORITHM_MISMATCH (0x80090331) - The client and server cannot communicate, because they do not possess a common algorit
and here is (roughly, it was printed in the wrong order for some reason) what i get if i use --tlsv1.3:
./CURL.EXE https://floridakeyswebcams.tv/sloppycam/camarchive/0807.jpg --tlsv1.3 -v
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
* TCP_NODELAY set
* Connected to floridakeyswebcams.tv (74.209.245.140) port 443 (#0)
> GET /sloppycam/camarchive/0807.jpg HTTP/1.1
> Host: floridakeyswebcams.tv
> User-Agent: curl/7.64.1
> Accept: */*
>
▒▒▒▒
\▒.▒:\▒.▒:▒▒
▒▒
%&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒
-truncated jpg binary-
question still remains about why this wasn't auto-negotiated tho, i'm not sure but i guess your TLS backend doesn't support tlv1.3, and that probably has something to do with why you just got a cryptic error..

Related

How to run curl --retry command in GitLab-ci.yml

i am trying to run a curl --retry command in GitLab-ci pipeline and it fails in the pipeline but when i try to run it somewhere else it's working fine.
I am trying to check if my url is up or not and this will run till my url is up and after that rest of my script run to configure my application
curl -Is -k --retry 50 --retry-delay 0 --retry-connrefused https://{URL} -vvv
Trying 127.0.1.1:8443...
TCP_NODELAY set
Connected to {url} (127.0.1.1) port 8443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to {url}
Closing connection 0
Cleaning up file based variables
ERROR: Job failed: exit status 1
I think what is happening here that when we run curl --retry for every retry it closes the connection and restart it and that's where my pipeline breaks and i am not able to figure out how to handle this.
Is there any workaround to do this in the pipeline.
you can use a simple bash script and check for curl return value:
while true; do if curl --location 'http://localhost:8080'; then break; else sleep 1; fi; done

CentOS6 - curl NSS error -5961 resolved by updating packages. What did it actually do?

I was running into the same issue as described in this question: cURL SSL connect error 35 with NSS error -5961
$ curl --verbose https://api.hostname.com
* About to connect() to api.hostname.com port 443 (#0)
* Trying 1.2.3.4... connected
* Connected to api.hostname.com (1.2.3.4) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* NSS error -5961
* Closing connection #0
* SSL connect error
curl: (35) SSL connect error
I followed user qingbo's advice after noticing that the server (CentOS 6.5 VM) had not been updated in a while. I ran this command:
$ yum update -y nss curl libcurl
and then re-ran the curl command and this time I received an expected HTTP/1.1 200 OK.
My question is - what did updating the nss, curl and libcurl packages do that fixed the connection issue?

Connect to Selenium within Docker-container

I'm developing tests with Selenium. Currently I'm using official selenium/standalone-chrome:3.11.0 image. I'm running only Selenium inside Docker-container. The project itself is compiled on the host machine (tests connect to the container's exposed port):
$ docker run -p 4444:4444 selenium/standalone-chrome:3.11.0
$ curl -v localhost:4444
* Rebuilt URL to: localhost:4444/
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 4444 (#0)
> GET / HTTP/1.1
> Host: localhost:4444
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
...
But I would like to compile and test the project entirely inside Docker-container. So I created my own image upon selenium/standalone-chrome:3.11.0. My (simplified) Dockerfile looks like this:
FROM selenium/standalone-chrome:3.11.0
RUN sudo apt-get --assume-yes --quiet update
RUN sudo apt-get --assume-yes --quiet install curl
CMD ["curl", "-v", "localhost:4444"]
As can be seen from the file, I'm trying to connect to port 4444 within container. When I run the image, e.g.:
docker build -t test . && docker run test
I get:
* Rebuilt URL to: localhost:4444/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 127.0.0.1...
* connect to 127.0.0.1 port 4444 failed: Connection refused
* Trying ::1...
* Immediate connect fail for ::1: Cannot assign requested address
* Trying ::1...
* Immediate connect fail for ::1: Cannot assign requested address
* Failed to connect to localhost port 4444: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 4444: Connection refused
Why I'm not able to connect to Selenium which is ran inside container from the same container?
I've found the solution at last (sorry for my stupidity).
Building an image upon selenium/standalone-chrome:3.11.0 is not sufficient. You need to start Selenium explicitly.
The Dockerfile:
FROM selenium/standalone-chrome:3.11.0
WORKDIR /app
COPY . /app
RUN sudo apt-get --assume-yes --quiet update
RUN sudo apt-get --assume-yes --quiet install curl
CMD ["./acceptance.sh"]
The acceptance.sh wrapper script:
#!/bin/bash
set -x
set -e
/opt/bin/entry_point.sh &
# It will be better to look into log and wait for
# record 'Selenium Server is up and running on port 4444'.
# But in this script simplified approach is used, just for
# the sake of brevity.
sleep 30
curl -v localhost:4444
The result:
...
+ set -e
+ sleep 30
+ /opt/bin/entry_point.sh
07:51:35.092 INFO [GridLauncherV3.launch] - Selenium build info: version: '3.11.0', revision: 'e59cfb3'
07:51:35.095 INFO [GridLauncherV3$1.launch] - Launching a standalone Selenium Server on port 4444
2018-05-15 07:51:35.661:INFO::main: Logging initialized #2436ms to org.seleniumhq.jetty9.util.log.StdErrLog
07:51:36.448 INFO [SeleniumServer.boot] - Welcome to Selenium for Workgroups....
07:51:36.450 INFO [SeleniumServer.boot] - Selenium Server is up and running on port 4444
^[[23;5~^[[23;5~+ curl -v localhost:4444
* Rebuilt URL to: localhost:4444/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 4444 (#0)
> GET / HTTP/1.1
> Host: localhost:4444
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
...
Cheers!

Docker: TLS handshake timeout

I've created my own private registry (private-registry) but I'm unable to push images to it.
Than I get the following error:
The push refers to a repository [private-registry:5000/ubuntu] (len: 1)
unable to ping registry endpoint https://private-registry:5000/v0/
v2 ping attempt failed with error: Get https://private-registry:5000/v2/: net/http: TLS handshake timeout
v1 ping attempt failed with error: Get https://private-registry:5000/v1/_ping: net/http: TLS handshake timeout
The logs of the running registry are showing the following:
time="2015-12-14T07:59:21Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.5.2 instance.id=a77e1955-3688-4fe3-a06e-0341787f8d0f version=v2.2.1
time="2015-12-14T07:59:21Z" level=info msg="redis not configured" go.version=go1.5.2 instance.id=a77e1955-3688-4fe3-a06e-0341787f8d0f version=v2.2.1
time="2015-12-14T07:59:21Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.5.2 instance.id=a77e1955-3688-4fe3-a06e-0341787f8d0f version=v2.2.1
time="2015-12-14T07:59:21Z" level=info msg="listening on [::]:5000, tls" go.version=go1.5.2 instance.id=a77e1955-3688-4fe3-a06e-0341787f8d0f version=v2.2.1
time="2015-12-14T07:59:21Z" level=info msg="Starting upload purge in 47m0s" go.version=go1.5.2 instance.id=a77e1955-3688-4fe3-a06e-0341787f8d0f version=v2.2.1
I'm unable to curl my registry (timeout).
This are the steps I performed:
First I've created selfsigned certificates:
mkdir -p certs && openssl req \
-newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key \
-x509 -days 365 -out certs/domain.crt
I've created my registry, which will use this certificates:
docker run -d -p 5000:5000 --restart=always --name private-registry \
-v `pwd`/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=certs/domain.crt \
-e REGISTRY_HTTP_TLS_KEY=certs/domain.key \
registry:2
I gave the right permissions:
chcon -Rt svirt_sandbox_file_t ~certs/
I've created: /etc/docker/etc.d/private-registry:5000/
And I copied my domain.crt in it.
I've edited my /etc/hosts and added:
10.0.0.X private-registry (my internal ip and the name of my registry)
I also restarted docker and my registry.
EDIT:
[centos# ~]$ curl -v private-registry:5000
* About to connect() to private-registry port 5000 (#0)
* Trying 10.0.0.xx...
* Connected to private-registry (10.0.0.xx) port 5000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: private-registry:5000
> Accept: */*
>
* Connection #0 to host private-registry left intact
[centos#~]$ curl -v https://private-registry:5000
* About to connect() to private-registry port 5000 (#0)
* Trying 10.0.0.xx...
* Connected to private-registry (10.0.0.xx) port 5000 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* NSS error -5990 (PR_IO_TIMEOUT_ERROR)
* I/O operation timed out
* Closing connection 0
curl: (35) I/O operation timed out
You may need to place the certificate into this directory.
/etc/docker/certs.d/private-registry.com:5000/ca.crt

Curl doesn't accept my ssl key

I am running the following command on unix using curl :
curl -v -d #SampleRequest_OFFSHORE.xml -E ./mypemfile.pem:nopass --cacert ./ifind.prod.ClientAuth.abc.com.CER https://world-service-dev.intra.abc.com:4414/worldservice/CLIC/CaseManagementService/V1
I am getting the following error:
* About to connect() to world-service-dev.intra.abc.com port 4414 (#0)
* Trying 148.171.176.115... connected
* Connected to world-service-dev.intra.abc.com (148.171.176.115) port 4414 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: ./ifind.prod.ClientAuth.abc.com.CER
CApath: none
* Unable to load client key -8178.
* NSS error -8178
* Closing connection #0
curl: (58) Unable to load client key -8178.
Are you sure your keys are in the right keystore?
Did you try and change the file permissions on the keystore?