Inserting client certificates into an HTTPS POST request - ssl

I'm building a hardware device which connects to the AWS IOT platform. According to the documentation the authentication with the aws iot platform is done with TLS. I have the Root CA, client key and client certificate files on the device that authorize the access. Is there a way to use these files in the HTTP header while making the POST request? If so, how? So far here is the code for the Energia IDE (based on the Arduino IDE) and using the WiFiClient methods.
if (client.sslConnect(aws_endpoint, 443))
{
Serial.println("\nConnected to AWS endpoint");
String PostData = "{\"value1\" : \"testValue\", \"value2\" : \"Hello\", \"value3\" : \"World!\" }";
request = "POST /things/";
request += thingname;
request += "/shadow";
request += " HTTP/1.1";
Serial.print("Request:\t"); Serial.println(request);
Serial.print("Post data:\t"); Serial.println(PostData);
client.println(request);
client.println("Host: ");
client.println(aws_endpoint);
client.println(":443");
client.println("User-Agent: Energia/1.1");
client.println("Connection: close");
client.println("Content-Type: application/json");
client.print("Content-Length: "); client.println(PostData.length());
client.println();
client.println(PostData);
client.println();
}
else
{
Serial.println("Connection failed");
}
Serial.println();
Serial.println("Server response:");
Serial.println();
// Capture response from the server. (10 second timeout)
long timeOut = 5000;
long lastTime = millis();
while((millis()-lastTime) < timeOut)
{ // Wait for incoming response from server
while (client.available())
{ // Characters incoming from the server
char c = client.read(); // Read characters
Serial.write(c);
}
}
This however, gives an authentication error:
HTTP/1.1 403 Forbidden
content-type: application/json
content-length: 91
date: Tue, 26 Jul 2016 11:46:59 GMT
x-amzn-RequestId: 4d5388a9-e3c4-460a-b674-c3f971f3330d
connection: Keep-Alive
x-amzn-ErrorType: ForbiddenException:
{"message":"Missing Authentication Token","traceId":"4d5388a9-e3c4-460a-b674-c3f971f3330d"}

The TLS client certificates would be sent/used as part of your client.sslConnect() call, not as part of the HTTP request. The TLS handshake (and exchange/validation of client and server certificates) happens before any HTTP message is sent.
This AWS forums post suggests that you may need to be using port 8443 (not port 443), for the shadow API. It looks like the use/requirement of TLS mutual authentication (via certificates), versus the use of AWS SIGv4 headers, is determined by AWS IOT based on the port used.
Hope this helps!

Related

Ktor Android Client Websocket Connection Failed

The WebSocket server is a online testing one
The Website
Something goes wrong And I don't know how to fix it.
val client = HttpClient(CIO) { install(WebSockets) }
GlobalScope.launch {
client.webSocket("ws://82.157.123.54:9010/ajaxchattest") {}
}
the error printStackTrace
java.lang.IllegalStateException: Failed to parse request body: request body length
should be specified,
chunked transfer encoding should be used or
keep-alive should be disabled (connection: close)
not knowing how to enable encoding or disable keep-alive or specify body length.
The 82.157.123.54:9010/ajaxchattest endpoint responds with 403 Forbidden instead of 101 Switching Protocols if the Origin header is absent or invalid. So to make it work just append the Origin header with a well-formed value:
val client = HttpClient(CIO) { install(WebSockets) }
client.webSocket("ws://82.157.123.54:9010/ajaxchattest", request = {
header(HttpHeaders.Origin, "http://example")
}) {}

Two sites same IIS server, RunImpersonation HttpClient passing empty NTLM credential

I have two sites on the same IIS instance. One uses HttpClient to request data from the other. They are both configured with Windows Authentication only. These are both ASPNET Core 3.1.
When I browse to the first site, it authenticates, but when it calls through to the other, it returns a 401. I validated that the user identity is correct.
var baseUri = new Uri(AppSettings.CurrentValue.MyBaseUrl);
var user = (WindowsIdentity)HttpContext.User.Identity;
WindowsIdentity.RunImpersonated(user.AccessToken, () =>
{
AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);
var credentialCache = new CredentialCache {{baseUri, "NTLM", CredentialCache.DefaultNetworkCredentials}};
var httpClientHandler = new HttpClientHandler {Credentials = credentialCache};
using HttpClient httpClient = new HttpClient(httpClientHandler) { BaseAddress = baseUri };
var response = httpClient.GetAsync("/home/test").Result;
var content = response.IsSuccessStatusCode
? response.Content.ReadAsStringAsync().Result
: response.StatusCode.ToString();
}
Wireshark shows the call into the first site, /home/test1, and shows the user credentials being passed correctly,
GET /home/test1 HTTP/1.1
HTTP/1.1 401 Unauthorized (text/html)
GET /home/test1 HTTP/1.1 , NTLMSSP_NEGOTIATE
HTTP/1.1 401 Unauthorized , NTLMSSP_CHALLENGE (text/html)
GET /home/test1 HTTP/1.1 , NTLMSSP_AUTH, User: COMPANY\fbloggs
The same trace shows the jump to the second site, /home/test2, and shows null domain and username,
GET /home/test2 HTTP/1.1
HTTP/1.1 401 Unauthorized (text/html)
GET /home/test2 HTTP/1.1 , NTLMSSP_NEGOTIATE
HTTP/1.1 401 Unauthorized , NTLMSSP_CHALLENGE (text/html)
GET /home/test2 HTTP/1.1 , NTLMSSP_AUTH, ** User: \ **
GET /home/test2 HTTP/1.1\r\n
[Expert Info (Chat/Sequence): GET /home/test2 HTTP/1.1\r\n]
[GET /home/test2 HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: GET
Request URI: /home/test2
Request Version: HTTP/1.1
Connection: Keep-Alive\r\n
Request-Id: |60fb71bd-482efe66c05094ec.1.\r\n
Host: testserver\r\n
Authorization: NTLM TlRMTVNTUAADAAAAAQDADEIAAAAAAAAAcwAAAAAAAABYAAAAAAAAAFgAAAAaABoAWAAzAAAABYqIogoAY0UAAAAPvaq0nk2I7YcqJmq01EbY20IASDTATGSAGVAALQBXAEUAQgAyADEAAA==\r\n
NTLM Secure Service Provider
NTLMSSP identifier: NTLMSSP
NTLM Message Type: NTLMSSP_AUTH (0x00000003)
Lan Manager Response: 00
NTLM Response: Empty
Domain name: NULL
User name: NULL
Host name: TESTSERVER
Session Key: Empty
Negotiate Flags: 0xa2888a05, Negotiate 56, Negotiate 128, Negotiate Version, Negotiate Target Info, Negotiate Extended Security, Negotiate Always Sign, Negotiate Anonymous, Negotiate NTLM key, Request Target, Negotiate UNICODE
Version 10.0 (Build 17763); NTLM Current Revision 15
Major Version: 10
Minor Version: 0
Build Number: 17763
NTLM Current Revision: 15
MIC: bdaab49e4d88ed872a266ab4d446d8db
HTTP/1.1 401 Unauthorized (text/html)

arduino gets response 301 when making a HTTPS GET request

I am making a simple arduino app that send a GET request to a HTTPS site. The code I'm using is exactly the same as the code in the MKRGSM library examples (GsmSslWebClient). But for whatever reason I always get the same response when connecting to a HTTPS site: "301 moved permanently". I kind of know what that means, I am aware that you are supposed to just make another request to the location specified in the header. But I don't know what I have to change to be able to address a https site. I'm sorry for my ignorance and I do know that in the example it clearly states that it connects to http://www.arduino.cc/asciilogo.txt but why is it then any different than the normal http example?
I would also point out here, that I have tried changing port to 80 and client settings, to work for unprotected http which works just fine. So its just the https that doesn't work.
this is the code:
/*
Web client
This sketch connects to a website using SSL through a MKR GSM 1400 board. Specifically,
this example downloads the URL "http://www.arduino.cc/asciilogo.txt" and
prints it to the Serial monitor.
Circuit:
* MKR GSM 1400 board
* Antenna
* SIM card with a data plan
created 8 Mar 2012
by Tom Igoe
*/
// libraries
#include <MKRGSM.h>
#include "arduino_secrets.h"
// Please enter your sensitive data in the Secret tab or arduino_secrets.h
// PIN Number
const char PINNUMBER[] = SECRET_PINNUMBER;
// APN data
const char GPRS_APN[] = SECRET_GPRS_APN;
const char GPRS_LOGIN[] = SECRET_GPRS_LOGIN;
const char GPRS_PASSWORD[] = SECRET_GPRS_PASSWORD;
// initialize the library instance
GSMSSLClient client;
GPRS gprs;
GSM gsmAccess;
// URL, path and port (for example: arduino.cc)
char server[] = "arduino.cc";
char path[] = "/asciilogo.txt";
int port = 443; // port 443 is the default for HTTPS
void setup() {
// initialize serial communications and wait for port to open:
Serial.begin(9600);
while (!Serial) {
; // wait for serial port to connect. Needed for native USB port only
}
Serial.println("Starting Arduino web client.");
// connection state
bool connected = false;
// After starting the modem with GSM.begin()
// attach the shield to the GPRS network with the APN, login and password
while (!connected) {
if ((gsmAccess.begin(PINNUMBER) == GSM_READY) &&
(gprs.attachGPRS(GPRS_APN, GPRS_LOGIN, GPRS_PASSWORD) == GPRS_READY)) {
connected = true;
} else {
Serial.println("Not connected");
delay(1000);
}
}
Serial.println("connecting...");
// if you get a connection, report back via serial:
if (client.connect(server, port)) {
Serial.println("connected");
// Make a HTTP request:
client.print("GET ");
client.print(path);
client.println(" HTTP/1.1");
client.print("Host: ");
client.println(server);
client.println("Connection: close");
client.println();
} else {
// if you didn't get a connection to the server:
Serial.println("connection failed");
}
}
void loop() {
// if there are incoming bytes available
// from the server, read them and print them:
if (client.available()) {
char c = client.read();
Serial.print(c);
}
// if the server's disconnected, stop the client:
if (!client.available() && !client.connected()) {
Serial.println();
Serial.println("disconnecting.");
client.stop();
// do nothing forevermore:
for (;;)
;
}
}
and this is the output:
Starting Arduino web client.
connecting...
connected
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Thu, 19 Nov 2020 20:24:07 GMT
Content-Type: text/html
Content-Length: 178
Connection: close
Location: https://www.arduino.cc/asciilogo.txt
Strict-Transport-Security: max-age=500; includeSubDomains
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
disconnecting.
It is also possible that I'm just stupid and the answer is clear as day, but if somebody could please tell me exactly what to do, maybe even put an example, that would be greatly appreciated.
I haven't really tried much except changing the setting to see if they work for normal http and some other libraries that worked even worse. Sadly that's all I could find on the internet since I'm not that skilled to be messing with libraries on my own. I am using arduino mkr gsm 1400.
Be sure to call me out if I missed to mention any detail that could help solving this issue
Thanks to anybody that can help me in advance.

SSL issue on Android 9 Google Pixel One

I am trying to perform HTTPS requests to a host 10.10.10.1 from Android host with 10.10.10.2 in network without Internet connection - only WiFi 2 peers AP and Android 9 Google Pixel One device.
I've created network_security_config.xml with my cert that is self-signed and has CN=10.10.10.1 and SAN= DNS: 10.10.10.1 PI: 10.10.10.1.
<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
<base-config cleartextTrafficPermitted="true">
<trust-anchors>
<certificates src="system" />
<certificates src="user" />
<certificates src="#raw/zone"/>
</trust-anchors>
</base-config>
</network-security-config>
I don't receive verification error and observe successful requests incoming to server - data are HTTP request, decrypted and shown on the server log. But the server can't send data back! It sends, but for some reason these data are not being accepted by the Android phone - just ignored.
I see packets are going from the server to the phone and the server repeatedly retries to shutdown SSL socket until error or success (I made such behavior intentionally during surveying) - here is Wireshark dump from WiFi air:
Here is my request from AsyncTask
protected String doInBackground(String... params) {
StringBuilder result = new StringBuilder();
try {
CertificateFactory cf = CertificateFactory.getInstance("X.509");
InputStream caInput = new BufferedInputStream(MainActivity.this.getResources().openRawResource(R.raw.zone));
Certificate ca = cf.generateCertificate(caInput);
String keyStoreType = KeyStore.getDefaultType();
KeyStore keyStore = KeyStore.getInstance(keyStoreType);
keyStore.load(null, null);
keyStore.setCertificateEntry("ca", ca);
String tmfAlgorithm = TrustManagerFactory.getDefaultAlgorithm();
TrustManagerFactory tmf = TrustManagerFactory.getInstance(tmfAlgorithm);
tmf.init(keyStore);
SSLContext ctx = SSLContext.getInstance("TLS");
ctx.init(null, tmf.getTrustManagers(), null);
URL url = new URL("https://10.10.10.1/connect");
HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
conn.setSSLSocketFactory(ctx.getSocketFactory());
conn.setRequestProperty("param1", params[0]);
conn.setRequestProperty("param2", params[1]);
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.setDoInput(true);
mInputStream = conn.getInputStream();
byte[] buffer = new byte[1024];
ByteArrayOutputStream _buf = new ByteArrayOutputStream();
int l;
BufferedInputStream bufin = new BufferedInputStream(mInputStream);
while ((l = bufin.read(buffer,0,1024)) != -1) {
_buf.write(buffer, 0, l);
String rec = _buf.toString("UTF-8");
Log.d("MAIN", "Read: " + rec);
result.append(rec);
}
Log.d("MAIN", "Read finished: " + result.toString());
} catch (Exception e) {
e.printStackTrace();
}
return result.toString();
}
I suspect that Android 9 Network Security does block traffic somehow. I tried to use SSLSockets, change port from 443 to e.g. 1234 - no luck.
In fact my app is being created with Qt and firstly I used Qt stuff, but having no luck - I made fallback to Android Java code within my MainActivity, that I call via JNI from Qt code. Result is the same and I have no ideas more...
Where to dig?
UPD1
When the self-signed certificate is generated with SAN containing DNS:10.10.10.1 only (without IP:10.10.10.1) SSL fails with warnings:
W System.err: javax.net.ssl.SSLPeerUnverifiedException: Hostname 10.10.10.1 not verified:
W System.err: certificate: sha1/gyr2GOhy5lA+ZAHEzh0E2SBEgx0=
W System.err: DN: CN=10.10.10.1,O=Some ltd.,L=Knoxville,ST=TN,C=US
W System.err: subjectAltNames: [10.10.10.1]
W System.err: at com.android.okhttp.internal.io.RealConnection.connectTls(RealConnection.java:201)
W System.err: at com.android.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:149)
W ...
And conversely, with SAN IP:10.10.10.1 (without DNS: 10.10.10.1) - works as before - session established, data transferred to server and decrypted, but responses from server to client just ignored by client.
UPD2
I've also tried to use domain name some.device for the 10.10.10.1 device and issued certificate with CN and SAN DNS = some.device. It's resolved by Android 9 client, data is being sent successfully but response is still not being accepting.
Looks like Android bug.
After making additional surveying:
1. Some set of Android devices (builds), including Pixel 1, does not accept TCP session that was not finalized by mutual [FIN,ACK] and received data is not delivered to upper level of stack. Also data may not be accepted if TCP stream was not solid, with many retransmissions and Seq changing.
2. In case of using Qt - Android Network Security Configuration does not affect on communications.
3. This is not TLS related issue.

How to connect http server websocket with ssl using vertx?

I have created two classes server and client. Server starting with ssl as below
HttpServer server =
vertx.createHttpServer(new HttpServerOptions().setSsl(true).setKeyStoreOptions(
new JksOptions().setPath("server-keystore.jks").setPassword("wibble")
));
Also one more i.e. client
vertx.createHttpClient(new HttpClientOptions().setSsl(true)).getNow(4443, "localhost", "/", resp -> {
System.out.println("Got response " + resp.statusCode());
resp.bodyHandler(body -> System.out.println("Got data " + body.toString("ISO-8859-1")));
});
While running both On client I am getting "Failed to create SSL connection". Is there any way to configure anything related to ssl?
To enable ssl in vertx you can use keystore.jks file
Then use following configuration :
HttpServerOptions secureOptions = new HttpServerOptions();
if (Configuration.SSL_enabled) {
LOG.debug("Secure Transport Protocol [ SSL/TLS ] has been enabled !!! ");
secureOptions.setSsl(true)
.setKeyStoreOptions(new JksOptions().setPath(Configuration.SSL_filename)
.setPassword(Configuration.SSL_password))
.setTrustStoreOptions(new JksOptions().setPath(Configuration.SSL_filename)
.setPassword(Configuration.SSL_password))
.addEnabledSecureTransportProtocol(Constants.TLS_VERSION_1)
.addEnabledSecureTransportProtocol(Constants.TLS_VERSION_2);
}
vertx.createHttpServer(secureOptions).requestHandler(router::accept).listen(Configuration.port);
I hope this will help you :)