Why the script doesn't receive data from mosquitto? - sql

Question

I have following code which receive data from different MQTT topics.
#!/usr/bin/python3
import paho.mqtt.client as mqtt
import pymysql
import json
import sys
import time
#-### Constantes ####
PUERTO_MQTT = ...
USUARIO = ...
CLAVE = ...
#DB_LOCAL = ...
DB_LOCAL = ...
RETARDO_SUSCRIP = 30 #s
#-### Variables ####
idsNgsConectados = set()
clienteMQTT = mqtt.Client("receptor_datos_ngs")
db_local = None
t1 = 0
t2 = 0
#-### Funciones ####
def on_connect(clienteMQTT, userdata, flags, rc):
print("Conexion establecida con el broker MQTT correctamente")
def on_message(clienteMQTT, userdata, msg):
msg_str = msg.payload.decode('utf-8')
# procesamiento de los mensajes MQTT
posPrimeraBarra = msg.topic.find('/')
print("topic: {}".format(msg.topic))
print("datos: {}".format(msg_str))
if posPrimeraBarra != -1:
id_ngs = int(msg.topic[:posPrimeraBarra])
ref_medicion = msg.topic[posPrimeraBarra+1:]
variables = json.loads(msg_str)
if type(variables) is dict:
if id_ngs in idsNgsConectados:
try:
cursor = db_local.cursor()
cursor.execute("INSERT INTO mediciones (id_ngs, referencia) VALUES ({}, '{}')".format(id_ngs, ref_medicion))
db_local.commit()
cursor.execute("SELECT id FROM mediciones WHERE (id_ngs={} AND referencia='{}') ORDER BY id DESC LIMIT 1".format(id_ngs, ref_medicion))
id_medicion = cursor.fetchone()[0]
for var in variables:
cursor.execute("INSERT INTO valores_mediciones (id_medicion, variable, valor) VALUES ({},'{}',{})".format(id_medicion, var, variables[var]))
db_local.commit()
cursor.close()
except Exception as e:
db_local.rollback()
print("Error 1: fallo el procesamiento de un mensaje MQTT: " + str(e), file=sys.stderr)
sys.exit(1)
else:
print("Advertencia: un ngs envio datos sin haberse presentado, por lo que se le indicara que salude primero", file=sys.stderr)
clienteMQTT.publish(str(id_ngs),"saludar",qos=1)
else:
print("Advertencia: un ngs envio datos con un formato incorrecto por lo que se le indicara que se reinicie", file=sys.stderr)
clienteMQTT.publish(str(id_ngs),"reiniciar",qos=1)
def sub_topics():
global clienteMQTT
cursor = db_local.cursor()
cursor.execute('SELECT id FROM ngs')
ids_ngs = cursor.fetchall()
cursor.close()
print("Topics suscriptos:")
for id_ngs in ids_ngs:
id_ngs = id_ngs[0]
idsNgsConectados.add(id_ngs)
topic = str(id_ngs) + '/#'
print("\t{}".format(topic))
clienteMQTT.subscribe(topic)
try:
db_local = pymysql.connect( unix_socket='/run/mysqld/mysqld.sock',
user=USUARIO,
password=CLAVE,
db=DB_LOCAL )
clienteMQTT.on_connect = on_connect
clienteMQTT.on_message = on_message
clienteMQTT.username_pw_set(username=USUARIO, password=CLAVE)
clienteMQTT.connect("localhost", PUERTO_MQTT)
clienteMQTT.loop_start()
# Bucle infinito
while True:
time.sleep(RETARDO_SUSCRIP)
# Suscripcion a los topics de todos los ngs
sub_topics()
except Exception as e:
db_local.close()
clienteMQTT.loop_stop()
clienteMQTT.disconnect()
print("Error 2: desconocido: " + str(e), file=sys.stderr)
sys.exit(2)
The script subscribe dinamically to the topics. If I run the script from the shell it works well, but if a setup it to be run at boot using systemd, it fails. I've setup the unit file to require and to be run after mosquitto, mariadb, dhcpcd and wpa_supplicant services. I'm sure the data is being publish from another device because I can receive it using "mosquitto_sub".
Which could it be the reason?
It runs in a Raspberry pi zero w and the Mosquitto version is 1.5.7.

Answer 1

I could find the cause of the problem and it wasn't because of mosquitto or paho-mqtt, it was because of pymysql. The fail in the code was that I wasn't using the method commit() in the sql transaction. That was causing the code to not being able to get the new data that were being introduced by another process in the database. That's because of the InnoDB's default isolation level REPEATABLE READ. You can read more about that here. The correct way to make that sql transaction is the following:
def sub_topics():
cursor = db_local.cursor()
cursor.execute('SELECT id FROM ngs')
ids_ngs = cursor.fetchall()
print("Topics suscriptos:")
for id_ngs in ids_ngs:
id_ngs = id_ngs[0]
idsNgsConectados.add(id_ngs)
topic = str(id_ngs) + '/#'
print("\t{}".format(topic))
clienteMQTT.subscribe(topic)
db_local.commit()
cursor.close()
That data is used by this code to determine which topics it has to subscribe to, and because of that the code wasn't subscribing to the expected topics and so it wasn't receiving the related data from mosquitto.
The pymysql documentation doesn't cover that topic, so I recommend to the newbies (I've realizaed that I'm 50% newbie 50% pro jaja) to read about ACID and the implementation details of the RDBMS that you're using.