SSL certificate verification on ESP8266 Wemos D1 Mini with MQTT broker - ssl

I have a raspberry pi 3 with raspbian stretch as its operating system. I have installed and fully configured a MQTT broker on the raspberry pi following this tutorial: https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-the-mosquitto-mqtt-messaging-broker-on-ubuntu-16-04
Everything works fine and well on the broker's side. The certificates get renewed after 60 days and you can only connect to port 1883 via the localhost and the other ports (8883 and 8083) are open but can only be accessed using TLS version 1.2 and for the latter also using websockets. Below you can find the code of my configuration of mosquitto (/etc/mosquitto/conf.d/default.conf).
allow_anonymous false
password_file /etc/mosquitto/passwd
listener 1883 localhost
listener 8883
certfile /etc/letsencrypt/live/home.kamidesigns.be/cert.pem
cafile /etc/letsencrypt/live/home.kamidesigns.be/chain.pem
keyfile /etc/letsencrypt/live/home.kamidesigns.be/privkey.pem
tls_version tlsv1.2
listener 8083
protocol websockets
certfile /etc/letsencrypt/live/home.kamidesigns.be/cert.pem
cafile /etc/letsencrypt/live/home.kamidesigns.be/chain.pem
keyfile /etc/letsencrypt/live/home.kamidesigns.be/privkey.pem
tls_version tlsv1.2
I also bought a ESP8266 Wemos D1 Mini to connect to this broker in a secure way. I used the pubsubclient library from this link: https: //github.com/knolleary/pubsubclient for my MQTT client.
I use the master branch of this link: https://github.com/esp8266/Arduino for my secure SSL connection. Below you see the code I used for programming my Wemos D1 Mini
#include <ESP8266WiFi.h>
#include <PubSubClient.h>
#include <time.h>
void callback(char* topic, byte* payload, unsigned int length) {
Serial.print("Message arrived [");
Serial.print(topic);
Serial.print("] ");
for (int i = 0; i < length; i++) {
Serial.print((char)payload[i]);
}
Serial.println();
}
const char* ssid = "ssid";
const char* password = "wifipassword";
const char* host = "home.kamidesigns.be";
const int port = 8883;
WiFiClientSecure espClient;
PubSubClient client(host, port, callback, espClient);
long lastMsg = 0;
char msg[50];
int value = 0;
void setup() {
Serial.begin(115200);
Serial.println();
Serial.print("connecting to ");
Serial.println(ssid);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.println("WiFi connected");
Serial.println("IP address: ");
Serial.println(WiFi.localIP());
// Synchronize time useing SNTP. This is necessary to verify that
// the TLS certificates offered by the server are currently valid.
Serial.print("Setting time using SNTP");
configTime(8 * 3600, 0, "pool.ntp.org", "time.nist.gov");
time_t now = time(nullptr);
while (now < 1000) {
delay(500);
Serial.print(".");
now = time(nullptr);
}
Serial.println("");
struct tm timeinfo;
gmtime_r(&now, &timeinfo);
Serial.print("Current time: ");
Serial.print(asctime(&timeinfo));
}
void reconnect() {
// Loop until we're reconnected
while (!client.connected()) {
Serial.print("Attempting MQTT connection...");
// Attempt to connect
if (client.connect("ESP8266LightController","username","password")) {
Serial.println("connected");
// Once connected, publish an announcement...
client.publish("outTopic", "hello world");
// ... and resubscribe
client.subscribe("inTopic");
} else {
Serial.print("failed, rc=");
Serial.print(client.state());
Serial.println(" try again in 5 seconds");
// Wait 5 seconds before retrying
delay(5000);
}
}
}
When I start my Wemos D1, the serial monitor says:
connecting to ssid
..
WiFi connected
IP address:
192.168.0.213
Setting time using SNTP.
Current time: Sat Oct 14 02:26:25 2017
Attempting MQTT connection...connected
This is good and it is exactly what I wanted but I'm confused by how my Wemos D1 is able to connect to port 8883 without it verifying the certificate chain of the server? Remember that I never uploaded a certificate to the Wemos D1 or implemented a certificate into the code, and still it can connect.

One of 2 options
The WiFiClientSecure includes a list of public CA certs and is verifying your certificate against this list
The WiFiClientSecure defaults to not verifying remote certs by default.
Looking at this issue it looks like option 2 is most likely as it implies you have to verify the cert yourself after the connection.

Related

arduino gets response 301 when making a HTTPS GET request

I am making a simple arduino app that send a GET request to a HTTPS site. The code I'm using is exactly the same as the code in the MKRGSM library examples (GsmSslWebClient). But for whatever reason I always get the same response when connecting to a HTTPS site: "301 moved permanently". I kind of know what that means, I am aware that you are supposed to just make another request to the location specified in the header. But I don't know what I have to change to be able to address a https site. I'm sorry for my ignorance and I do know that in the example it clearly states that it connects to http://www.arduino.cc/asciilogo.txt but why is it then any different than the normal http example?
I would also point out here, that I have tried changing port to 80 and client settings, to work for unprotected http which works just fine. So its just the https that doesn't work.
this is the code:
/*
Web client
This sketch connects to a website using SSL through a MKR GSM 1400 board. Specifically,
this example downloads the URL "http://www.arduino.cc/asciilogo.txt" and
prints it to the Serial monitor.
Circuit:
* MKR GSM 1400 board
* Antenna
* SIM card with a data plan
created 8 Mar 2012
by Tom Igoe
*/
// libraries
#include <MKRGSM.h>
#include "arduino_secrets.h"
// Please enter your sensitive data in the Secret tab or arduino_secrets.h
// PIN Number
const char PINNUMBER[] = SECRET_PINNUMBER;
// APN data
const char GPRS_APN[] = SECRET_GPRS_APN;
const char GPRS_LOGIN[] = SECRET_GPRS_LOGIN;
const char GPRS_PASSWORD[] = SECRET_GPRS_PASSWORD;
// initialize the library instance
GSMSSLClient client;
GPRS gprs;
GSM gsmAccess;
// URL, path and port (for example: arduino.cc)
char server[] = "arduino.cc";
char path[] = "/asciilogo.txt";
int port = 443; // port 443 is the default for HTTPS
void setup() {
// initialize serial communications and wait for port to open:
Serial.begin(9600);
while (!Serial) {
; // wait for serial port to connect. Needed for native USB port only
}
Serial.println("Starting Arduino web client.");
// connection state
bool connected = false;
// After starting the modem with GSM.begin()
// attach the shield to the GPRS network with the APN, login and password
while (!connected) {
if ((gsmAccess.begin(PINNUMBER) == GSM_READY) &&
(gprs.attachGPRS(GPRS_APN, GPRS_LOGIN, GPRS_PASSWORD) == GPRS_READY)) {
connected = true;
} else {
Serial.println("Not connected");
delay(1000);
}
}
Serial.println("connecting...");
// if you get a connection, report back via serial:
if (client.connect(server, port)) {
Serial.println("connected");
// Make a HTTP request:
client.print("GET ");
client.print(path);
client.println(" HTTP/1.1");
client.print("Host: ");
client.println(server);
client.println("Connection: close");
client.println();
} else {
// if you didn't get a connection to the server:
Serial.println("connection failed");
}
}
void loop() {
// if there are incoming bytes available
// from the server, read them and print them:
if (client.available()) {
char c = client.read();
Serial.print(c);
}
// if the server's disconnected, stop the client:
if (!client.available() && !client.connected()) {
Serial.println();
Serial.println("disconnecting.");
client.stop();
// do nothing forevermore:
for (;;)
;
}
}
and this is the output:
Starting Arduino web client.
connecting...
connected
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Thu, 19 Nov 2020 20:24:07 GMT
Content-Type: text/html
Content-Length: 178
Connection: close
Location: https://www.arduino.cc/asciilogo.txt
Strict-Transport-Security: max-age=500; includeSubDomains
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
disconnecting.
It is also possible that I'm just stupid and the answer is clear as day, but if somebody could please tell me exactly what to do, maybe even put an example, that would be greatly appreciated.
I haven't really tried much except changing the setting to see if they work for normal http and some other libraries that worked even worse. Sadly that's all I could find on the internet since I'm not that skilled to be messing with libraries on my own. I am using arduino mkr gsm 1400.
Be sure to call me out if I missed to mention any detail that could help solving this issue
Thanks to anybody that can help me in advance.

ESP32 MQTT TLS errors (self signed cert)

I cannot seem to make MQTT+TLS+self signed certs work on my ESP32.
I am using this "setup" in my home network without major hickups with paho-python and mosquitto_pub/sub. The latter works with the following bash command (note the insecure there):
mosquitto_pub -h 192.168.1.X -p 8883 --cafile /etc/mosquitto/ca_certificates/mosq_ca.crt --cert /etc/mosquitto/certs/mosq_client.crt --key /etc/mosquitto/certs/mosq_client.key --debug --insecure --topic "test/message" -m "off"
My broker's mosquitto config file contains these lines apart from cert paths and general settings:
tls_version tlsv1.3
require_certificate true
When I send the sketch to my ESP32, the result is that it is stuck in the reconnect() loop (see sketch below).
When I do not specify a client cert and key ( // for client verification rows are commented in setup()), I get the following error (I guess it is attributable to the config which requires certificate):
1594936874: New connection from 192.168.1.162 on port 8883.
1594936874: OpenSSL Error: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
1594936874: Socket error on client <unknown>, disconnecting.
When I do specify a client certificate, I get this:
1594936887: New connection from 192.168.1.162 on port 8883.
1594936888: Socket error on client <unknown>, disconnecting.
Not sure if it is a host name mismatch on the certificate or what. This would be my main tip, but I am uncertain. I also do not know if a similar option to --insecure can be specified for WiFiClientSecure class, so far I have not found anything like that. But I am a beginner, too.
My sketch if pretty unpolished yet, but it should be still readable:
#include <WiFiClientSecure.h>
#include <time.h>
#include <PubSubClient.h>
const char* ssid = "mySSID";
const char* password = "myPasswd";
const char* server = "192.168.1.X"; // Server URL
const char* MQTT_subscribe_topic = "test/esp32";
int timediff_hr = 2;
const char* root_ca = \
"-----BEGIN CERTIFICATE-----\n" \
"pem format broken into lines\n" \
"pem format broken into lines\n" \
"pem format broken into lines\n" \
"-----END CERTIFICATE-----\n" ;
// You can use x.509 client certificates if you want
const char* test_client_key = \
"-----BEGIN CERTIFICATE-----\n" \
"pem format broken into lines\n" \
"pem format broken into lines\n" \
"pem format broken into lines\n" \
"-----END CERTIFICATE-----\n" ;
const char* test_client_cert = \
"-----BEGIN CERTIFICATE-----\n" \
"pem format broken into lines\n" \
"pem format broken into lines\n" \
"pem format broken into lines\n" \
"-----END CERTIFICATE-----\n" ;
WiFiClientSecure wifiClient;
time_t now;
void msgReceived(char* topic, byte* payload, unsigned int len);
PubSubClient pubSubClient(server, 8883, msgReceived, wifiClient);
void setup() {
//Initialize serial and wait for port to open:
Serial.begin(115200);
delay(100);
Serial.print("Attempting to connect to SSID: ");
Serial.println(ssid);
WiFi.begin(ssid, password);
// attempt to connect to Wifi network:
while (WiFi.status() != WL_CONNECTED) {
Serial.print(".");
// wait 1 second for re-trying
delay(1000);
}
Serial.print("Connected to ");
Serial.println(ssid);
getTime();
wifiClient.setCACert(root_ca);
wifiClient.setCertificate(test_client_key); // for client verification
wifiClient.setPrivateKey(test_client_cert); // for client verification
}
unsigned long lastPublish;
int msgCount;
void loop() {
if (!pubSubClient.connected()) {
reconnect();
}
pubSubClient.loop();
if (millis() - lastPublish > 10000) {
String msg = String("Hello from ESP32: ") + ++msgCount;
boolean rc = pubSubClient.publish("outTopic", msg.c_str());
Serial.print("Published, rc="); Serial.print( (rc ? "OK: " : "FAILED: ") );
Serial.println(msg);
lastPublish = millis();
}
}
void msgReceived(char* topic, byte* payload, unsigned int length) {
Serial.print("Message received on "); Serial.print(topic); Serial.print(": ");
for (int i = 0; i < length; i++) {
Serial.print((char)payload[i]);
}
Serial.println();
}
void reconnect() {
Serial.print(F("PubSubClient attempting connection to: ")); Serial.println(server);
uint8_t i = 0;
while ( ! pubSubClient.connected()) {
// Attempt to connect
if (pubSubClient.connect("ESP32Client")) { //or "ESP8266Client"
Serial.println("connected");
// Subscribe
pubSubClient.subscribe(MQTT_subscribe_topic);
} else { //unsuccessful connect
Serial.print("failed, rc=");
Serial.print(pubSubClient.state());
Serial.print("\nNext try in 5 seconds: connection to "); Serial.println(server);
// Wait 5 seconds before retrying
delay(5000);
}
}
}
void getTime(){
// Synchronize time useing SNTP. This is necessary to verify that
// the TLS certificates offered by the server are currently valid.
Serial.print("Setting time using SNTP");
configTime(timediff_hr * 3600, 0, "de.pool.ntp.org");
time_t now = time(nullptr);
while (now < 1000) {
delay(500);
Serial.print(".");
now = time(nullptr);
}
Serial.println("");
struct tm timeinfo;
gmtime_r(&now, &timeinfo);
Serial.print("Current time: ");
Serial.print(asctime(&timeinfo));
}
P.S.: The base of this sketch is a test sketch of mine for MQTT:1883 unsecured, which also works fine. So the issue should definitely be in the TLS part.
Your test_client_key contains a certificate.
It would be helpful if you post the sample code with keys and certificates. Obviously not the ones you will be using later, but as you are self signing, generate some test key material that you can post here.
The configuration of your broker would also be useful.

Connecting Arduino to Heroku web connected to Django database

To describe my problem I was trying to connect my Arduino UNO to website created by me in Heroku.
Main purpose was to called rest api function in arduino conected to Internet and get a json data.
My Arduino code:
#include <ArduinoJson.h>
#include <Ethernet.h>
#include <SPI.h>
void setup() {
// Initialize Serial port
Serial.begin(9600);
while (!Serial) continue;
// Initialize Ethernet library
byte mac[] = {0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED};
Ethernet.init(8); // use pin 53 for Ethernet CS
if (!Ethernet.begin(mac)) {
Serial.println(F("Failed to configure Ethernet"));
return;
}
delay(1000);
Serial.println(F("Connecting..."));
// Connect to HTTP server
EthernetClient client;
client.setTimeout(10000);
if (!client.connect("https://salty-cliffs-06856.herokuapp.com", 80)) {
Serial.println(F("Connection failed"));
return;
}
Serial.println(F("Connected!"));
// Send HTTP request
client.println(F("GET /api/command/ HTTP/1.1"));
client.println(F("Host: https://salty-cliffs-06856.herokuapp.com"));
client.println(F("Connection: close"));
Serial.println(F("Done"));
if (client.println() == 0) {
Serial.println(F("Failed to send request"));
return;
}
// Check HTTP status
char status[32] = {0};
client.readBytesUntil('\r', status, sizeof(status));
Serial.println(status);
if (strcmp(status, "HTTP/1.1 200 OK") != 0) {
Serial.print(F("Unexpected response: "));
Serial.println(status);
return;
}
// Skip HTTP headers
char endOfHeaders[] = "\r\n\r\n";
if (!client.find(endOfHeaders)) {
Serial.println(F("Invalid response"));
return;
}
// Allocate JsonBuffer
// Use arduinojson.org/assistant to compute the capacity.
const size_t capacity = JSON_OBJECT_SIZE(3) + JSON_ARRAY_SIZE(2) + 60;
DynamicJsonBuffer jsonBuffer(capacity);
// Parse JSON object
JsonObject& root = jsonBuffer.parseObject(client);
if (!root.success()) {
Serial.println(F("Parsing failed!"));
return;
}
// Extract values
Serial.println(F("Response:"));
Serial.println(root["command"].as<char*>());
// Disconnect
client.stop();
Everything was working fine with that code when I was trying putting there non-secured HTTP address. After putting there my web powered by Heroku secrued by HTTPS I always got error.
The program annouced error when I was checking HTTP status and in my Arduino port terminal I got response:
Unexpected response: HTTP/1.1 400 Bad Request
I checked my heroku logs but there are not listed any request from Arduino. (Just to be sure I tried to called API from web browser and it works)
Could you help me where could be a problem ? I was thinking that it could be because of secured HTTPS. What do you think ?
Thanks for every help :)
First, change client.connect("https://salty-cliffs-06856.herokuapp.com", 80)
from
`https`
to
`http`
as port 80 is not https port and Ethernet shield does not support SSL.
Secondly, you have a wrong http header for Host. the HTTP 1.1 requires that only the domain name to use used without the protocol (i.e. http://) prefix. So change the line:
client.println(F("Host: https://salty-cliffs-06856.herokuapp.com"));
to:
client.println(F("Host: salty-cliffs-06856.herokuapp.com"));

Arduino MKR1000 not able to connect to AWS API Gateway

I tried modifying the basic Arduino code from here in order to send HTTP Requests to AWS API Gateway. While the example code from the link worked, I was not able to get a successful connection with AWS API Gateway.
I have tried a combination of things such as removing the https:// from server[], changing the port to 443 instead of 80, removing the /beta from server[], using client.connectSSL instead of client.connect, but none of these have worked so far.
The line:
int err = client.connect(server, 80);
returns me a value of 0.
There are no certificates set up with the AWS API Gateway, so I don't think it's a problem with that. Wifi works perfectly.
Any help would be greatly appreciated!
#include <SPI.h>
#include <WiFi101.h>
#include "arduino_secrets.h"
///////please enter your sensitive data in the Secret tab/arduino_secrets.h
char ssid[] = SECRET_SSID; // your network SSID (name)
char pass[] = SECRET_PASS; // your network password (use for WPA, or
use as key for WEP)
int keyIndex = 0; // your network key Index number (needed only for WEP)
int status = WL_IDLE_STATUS;
// if you don't want to use DNS (and reduce your sketch size)
// use the numeric IP instead of the name for the server:
char server[] = "https://**********.execute-api.us-west-2.amazonaws.com/beta"; // name address for Google (using DNS)
WiFiClient client;
void setup() {
//Initialize serial and wait for port to open:
Serial.begin(9600);
while (!Serial) {
; // wait for serial port to connect. Needed for native USB port only
}
// check for the presence of the shield:
if (WiFi.status() == WL_NO_SHIELD) {
Serial.println("WiFi shield not present");
// don't continue:
while (true);
}
// attempt to connect to WiFi network:
while (status != WL_CONNECTED) {
Serial.print("Attempting to connect to SSID: ");
Serial.println(ssid);
// Connect to WPA/WPA2 network. Change this line if using open or WEP network:
status = WiFi.begin(ssid, pass);
// wait 10 seconds for connection:
delay(10000);
}
Serial.println("Connected to wifi");
printWiFiStatus();
Serial.println("\nStarting connection to server...");
// if you get a connection, report back via serial:
int err = client.connect(server, 80);
Serial.println(err);
if (err) {
Serial.println("connected to server");
// Make a HTTP request:
client.println("GET /beta HTTP/1.1");
client.println("Host: https://**********.execute-api.us-west-2.amazonaws.com");
client.println("Connection: close");
client.println();
}
}
void loop() {
// if there are incoming bytes available
// from the server, read them and print them:
while (client.available()) {
char c = client.read();
Serial.write(c);
}
// if the server's disconnected, stop the client:
if (!client.connected()) {
Serial.println();
Serial.println("disconnecting from server.");
client.stop();
// do nothing forevermore:
while (true);
}
}
void printWiFiStatus() {
// print the SSID of the network you're attached to:
Serial.print("SSID: ");
Serial.println(WiFi.SSID());
// print your WiFi shield's IP address:
IPAddress ip = WiFi.localIP();
Serial.print("IP Address: ");
Serial.println(ip);
// print the received signal strength:
long rssi = WiFi.RSSI();
Serial.print("signal strength (RSSI):");
Serial.print(rssi);
Serial.println(" dBm");
}
Tips:
(1) client.connect(fqdn, port) is expecting an FQDN for the first parameter
The example in the docs is client.connect("Arduino.cc", 80) this seems to work well in my tests. The docs say "URL" but they mean FQDN.
(2) If you need SSL then you MUST load up your certs using the firmware updater
first. If you are using non-standard pins for the WiFi101 board then you MUST use wifi.setPins() to set the pins or the firmware updater will fail. Adafruit Feather M0 1500 owners will know what I am talking about here.
Reference: https://www.arduino.cc/en/Reference/WiFi101ClientConnect
I hope this helps.

Arduino Ethernet Shield - No response on GET request to server

I have a code for the Arduino Ethernet Shield that will send a GET request to a server and return a PHP echo statement.
However, most of the time it fails to connect to the server.
When it does connect, I keep getting 403 Forbidden error or it says bad header format for "Host:".
I have checked every forum and all StackOverflow links related to the topic, but none of their solutions worked. My code is attached below.
#include <SPI.h>
#include <Ethernet.h>
byte mac[] = {0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
char server[] = "kanishkaganguly.byethost5.com";
IPAddress ip(192,168,0,103);
EthernetClient client;
void setup() {
Serial.begin(9600);
while (!Serial) {
;
}
Ethernet.begin(mac, ip);
delay(1000);
Serial.println("connecting...");
if (client.connect(server, 80)) {
Serial.println("connected");
client.println("GET /test.php HTTP/1.1");
client.println("Host: www.arduino.cc");
client.println("User-Agent: arduino-ethernet");
//client.println("User-Agent: Mozilla/5.0");
//This supposedly fixed 403 error for another user
client.println("Connection: close");
client.println();
}else {
Serial.println("connection failed");
}
}
void loop(){
// if there are incoming bytes available
// from the server, read them and print them:
if (client.available()) {
char c = client.read();
Serial.print(c);
}
// if the server's disconnected, stop the client:
if (!client.connected()) {
Serial.println();
Serial.println("disconnecting.");
client.stop();
// do nothing forevermore:
for(;;) ;
}
}
I figured out the problem. The client.println() as a new line isn't working for some reason. So, here is the updated code
client.print("GET /checkcontrol.php HTTP/1.1\r\n");
client.print("Host: shahz.webatu.com\r\n");
client.print("User-Agent: arduino-ethernet\r\n");
client.print("Connection: close\r\n\r\n");
The \r\n is the right way to go about adding a new line for the server to recognize.
The Host header specifies the hostname of the site you're connecting to. In this case, you are trying to connect to the domain kanishkaganguly.byethost5.com, but your Host header is set to www.arduino.cc. Usually this is incorrect. The Host header should match the domain, so both should be kanishkaganguly.byethost5.com.
When a client connects to a domain, the client first resolves the domain name to an IP, then makes the connection to that IP. Because the connection is made to the IP, the server does not know what domain name was looked up by the client. Because IPs are a limited resource, many hosting providers host multiple domains on a single IP. Without the Host header, the server would not know which page to return to the client.
The println command sends the text followed by "\r\n" so instead of changing every println for print, you could have added CRLF to the close line.
Client.println("Connection: close\r\n")