Sending large message to RabbitMQ with Pika 0.9.5: message is silently dropped by Rabbit - rabbitmq

Question

I've got a bunch of celery tasks that take their results and post them to a RabbitMQ message queue. The results that get posted can become quite large (up to a few meg). Opinion is mixed as to whether putting large amounts of data in a RabbitMQ message is a good idea, but I've seen this work in other situations and as long as memory is kept under control, it seems to work.
However, for my current set of tasks, rabbit appears to be just dropping messages that seem to be too big. I've reduced it down to a fairly simple test case:
#!/usr/bin/env python
import string
import random
import pika
import os
qname='examplequeue'
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='mq.example.com'))
channel = connection.channel()
channel.queue_declare(queue=qname,durable=True)
N=100000
body = ''.join(random.choice(string.ascii_uppercase) for x in range(N))
promise = channel.basic_publish(exchange='', routing_key=qname, body=body, mandatory=0, immediate=0, properties=pika.BasicProperties(content_type="text/plain",delivery_mode=2))
print " [x] Sent 'Hello World!'"
connection.close()
I have a 3-node RabbitMQ cluster, and mq.example.com round-robins to each node. Client is using Pika 0.9.5 on Ubuntu 12.04 and the RabbitMQ cluster is running RabbitMQ 2.8.7 on Erlang R14B04.
Executing this script prints the print statement and exits without any exceptions being raised. The message never appears in RabbitMQ.
Changing N to 10000 makes it work as expected.
Why?

Answer 1

I suppose you have problem with tcp-backpressure mechanizm in RabbitMq. You can read about http://www.rabbitmq.com/memory.html.
I see two ways to solve this problem:
Add tcp-callback and make reconnect every tcp-call from rabbit
Use compressing messages before sending it to rabbit, It will make easier push to rabbit.
def compress(s):
return binascii.hexlify(zlib.compress(s))
def decompress(s):
return zlib.decompress(binascii.unhexlify(s))

Answer 2

This is what I do to send and receive packets. It is somewhat more efficient than hexlify, because base64 may use one byte where two bytes are needed by hexlify to represent one character.
import zlib
import base64
def hexpress(send: str):
print(f"send: {send}")
bsend = send.encode()
print(f"byte-encoded send: {bsend}")
zbsend = zlib.compress(bsend)
print(f"zipped-byte-encoded-send: {zbsend}")
hzbsend = base64.b64encode(zbsend)
print(f"hex-zip-byte-encoded-send: {hzbsend}")
shzbsend = hzbsend.decode()
print(f"string-hex-zip-byte-encoded-send: {shzbsend}")
return shzbsend
def hextract(recv: str):
print(f"string-hex-zip-byte-encoded-recv: {recv}")
zbrecv = base64.b64decode(recv)
print(f"zipped-byte-encoded-recv: {zbrecv}")
brecv = zlib.decompress(zbrecv)
print(f"byte-encoded-recv: {brecv}")
recv = brecv.decode()
print(f"recv: {recv}")
return recv
print("sending ...\n")
send = "hello this is dog"
packet = hexpress(send)
print("\nover the wire -------->>>>>\n")
print("receiving...\n")
recv = hextract(packet)