Aerospike: zlib/bz2 store and retrieve didnt worked - aerospike

I am compressing a string using zlib, then storing in Aerospike bin. On retrieval and decompressing, I am getting "zlib.error: Error -5 while decompressing data: incomplete or truncated stream"
When I compared original compressed data and retrieved compressed data, some thing is missing at the end in retrieved data.
I am using Aerospike 3.7.3 & python client 2.0.1
Please help
Thanks
Update: Tried using bz2. Throws ValueError: couldn't find end of stream while retrieve and decompress. Looks like aerospike is stripping of the last byte or something else from the blob.
Update: Posting the code
import aerospike
import bz2
config = {
'hosts': [
( '127.0.0.1', 3000 )
],
'policies': {
'timeout': 1000 # milliseconds
}
}
client = aerospike.client(config)
client.connect()
content = "An Aerospike Query"
content_bz2 = bz2.compress(content)
key = ('benchmark', 'myset', 55)
#client.put(key, {'bin0':content_bz2})
(key, meta, bins) = client.get(key)
print bz2.decompress(bins['bin0'])
Getting Following Error:
Traceback (most recent call last):
File "asread.py", line 22, in <module>
print bz2.decompress(bins['bin0'])
ValueError: couldn't find end of stream

The bz.compress method returns a string, and the client sees that type and tries to convert it to the server's as_str type. If it runs into a \0 in an unexpected position it will truncate the string, causing your error.
Instead, make sure to cast binary data to a bytearray, which the client converts to the server's as_bytes type. On the read operation, bz.decompress will work with the bytearray data and give you back the original string.
from __future__ import print_function
import aerospike
import bz2
config = {'hosts': [( '33.33.33.91', 3000 )]}
client = aerospike.client(config)
client.connect()
content = "An Aerospike Query"
content_bz2 = bytearray(bz2.compress(content))
key = ('test', 'bytesss', 1)
client.put(key, {'bin0':content_bz2})
(key, meta, bins) = client.get(key)
print(type(bins['bin0']))
bin0 = bz2.decompress(bins['bin0'])
print(type(bin0))
print(bin0)
Gives back
<type 'bytearray'>
<type 'str'>
An Aerospike Query

Related

How to fix UnicodeDecodeError for bytes from requests?

I have the following full working example code using selenium-wire to record all requests made.
import os
import sys
import json
from seleniumwire import webdriver
driver = webdriver.Chrome()
driver.get("http://www.google.com")
list_requests = []
for request in driver.requests:
req = {
"method": request.method,
"url": request.url,
"body": request.body.decode(), # to avoid json error
"headers": {k:str(v) for k,v in request.headers.__dict__.items()} # to avoid json error
}
if request.response:
resp = {
"status_code": request.response.status_code,
"reason": request.response.reason,
"body": request.response.body.decode(), # ???
"headers": {k:str(v) for k,v in request.response.headers.__dict__.items()} # to avoid json error
}
req["response"] = resp
list_requests.append(req)
with open(f"test.json", "w") as outfile:
json.dump(list_requests, outfile)
However, the decoding of the response body creates an error
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte
and without trying to decode the response body I get an error
TypeError: Object of type bytes is not JSON serializable
I do not care about the encoding, I just want to be able to write the 'body' to the json file in some way. If needed the byte/character in question can be removed, I do not care.
Any ideas how this problem can be solved?
I've used the next approach in order to extract some field (some_key) from json response:
from gzip import decompress
import json
some_key = None
for request in driver.requests:
if request.response:
if request.method == 'POST':
print(request.method + ' ' + request.url)
try:
# try to parse the json response to extract the data
data = json.loads(request.response.body)
print('parsed as json')
if 'some_key' in data:
some_key = data['some_key']
except UnicodeDecodeError:
try:
# decompress on UnicodeDecodeError and parse the json response to extract the data
data = json.loads(decompress(request.response.body))
print('decompressed and parsed as json')
if 'some_key' in data:
some_key = data['some_key']
except json.decoder.JSONDecodeError:
data = request.response.body
print('decompressed and not parsed')
print(data)
print(some_key)
gzip.decompress helped me with UnicodeDecodeError.
Hope this will be helpful.

Python3/Redis: redis.exceptions.ResponseError: unknown command 'JSON.SET'

I'm trying to run the sample program from this RedisLabs page.
I chose Option A - which was to set up the free Redis cloud server.
(Seems like if you install manually, then you have to add the JSON as a plugin.)
I'm able to connect and use other "set" commands, but getting error on JSON:
File "C:\Users\nwalt\.virtualenvs\TDAmeritradeGetQuotes\lib\site-packages\redis\client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "C:\Users\nwalt\.virtualenvs\TDAmeritradeGetQuotes\lib\site-packages\redis\client.py", line 915, in parse_response
response = connection.read_response()
File "C:\Users\nwalt\.virtualenvs\TDAmeritradeGetQuotes\lib\site-packages\redis\connection.py", line 756, in read_response
raise response
redis.exceptions.ResponseError: unknown command 'JSON.SET'
My Python test program (except put in the sample endpoint before posting):
import redis
import json
import pprint
host_info = "redis.us-east-1-1.ec2.cloud.redislabs.com"
redisObj = redis.Redis(host=host_info, port=18274, password='xxx')
print ("Normal call to Redis")
redisObj.set('foo', 'bar')
value = redisObj.get('foo')
print(value)
capitals = {
"Lebanon": "Beirut",
"Norway": "Oslo",
"France": "Paris"
}
print ("capitals - before call to Redis")
pprint.pprint(capitals)
print("JSON call to Redis")
redisObj.execute_command('JSON.SET', 'doc', '.', json.dumps(capitals))
print("Data Saved, now fetch data back from redis")
reply = json.loads(redisObj.execute_command('JSON.GET', 'doc'))
print("reply from Redis get")
pprint.pprint(reply)
This is the screen shot from their website where I created the database. I didn't see any option to enable JSON or add any modules.
Not sure this was available when I created the REDIS database, but it is now. When you create it on redislabs.com, you can turn on the modules, and pick one from the list.
Then use this library: "rejson" from https://pypi.org/project/rejson/ to get the method "jsonset" method, using such code such as this:
rj = Client(host=config_dict['REDIS_CONFIG_HOST'], port=config_dict['REDIS_CONFIG_PORT'], password=config_dict['REDIS_CONFIG_PASSWORD'], decode_responses=True)
out_doc = {}
out_doc['firstname'] = "John"
out_doc['lastname'] = "Doe"
rj.jsonset('config', Path.rootPath(), out_doc)
get_doc = rj.jsonget('config', Path.rootPath())
pprint.pprint(get_doc)
I'm not used the cloud redis, in my local the Python don't load the JSON.SET
I just so make done, in this sample https://onelinerhub.com/python-redis/save-json-to-redis

How do I write a BLOB, any BLOB to sqlite?

I need to write an Obj-C object (e.g. NSString in this case) to a sqlite database and store it in a BLOB column, using Python 2.7.
To that extent I wrote this demo code that fails with the below traceback.
from sqlite3 import connect
from Foundation import NSArchiver
conn = connect(':memory:')
create = "CREATE TABLE test(data BLOB)"
conn.execute(create)
conn.commit()
blob = NSArchiver.archivedDataWithRootObject_("Hello World").bytes()
print type(blob), blob
sql = "INSERT INTO test VALUES (?)"
data = [blob]
conn.execute(sql, data)
conn.commit()
This traces back with:
$ ./sqlite3_test.py
<type 'memoryview'> <memory at 0x104a5e218>
Traceback (most recent call last):
File "./sqlite3_test.py", line 16, in <module>
conn.execute(sql, data)
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.
No amount of magik using sqlite3.Binary (which is defined as Binary = buffer inside the sqlite3 module) or .tobytes() (from memoryview) made that work any better.
I also tried to create a buffer() object out of the NSArchiver blob, but a naïve approach:
b = buffer(blob, 0, len(blob))
traces back with TypeError: buffer object expected - arguably NSArchiver objects are not Python strings.
I got a working example!
On the Objective-C call bytes() I need to call the memoryview .tobytes() which can then be serialized to buffer().
#!/usr/bin/python -tt
from sqlite3 import connect, Binary
from Foundation import NSArchiver
conn = connect(':memory:')
create = "CREATE TABLE test(data BLOB)"
conn.execute(create)
conn.commit()
str = NSString.alloc().initWithString_("Hello World")
blob = NSArchiver.archivedDataWithRootObject_(str).bytes().tobytes()
print "Original (%s): %s" % (len(blob), blob)
sql = "INSERT INTO test VALUES (?)"
data = [Binary(blob)]
conn.execute(sql, data)
conn.commit()
cursor = conn.cursor()
cursor.execute("SELECT * FROM test")
rows = cursor.fetchall()
for r in rows:
print "In database (%s): %s" % (len(r[0]), r[0])
This gives:
$ ./sqlite3_test.py
Original (84):
streamtyped???#???OC_PythonString?NSString?NSObject??i?+
Hello World?
In database (84):
streamtyped???#???OC_PythonString?NSString?NSObject??i?+
Hello World?

read wsgi post data with unicode encoding

How can i read wsgi POST with unicode encoding,
this is part of my code :
....
request_body_size = int(environ.get('CONTENT_LENGTH', 0))
req = str(environ['wsgi.input'].read(request_body_size))
and from req i read my fileds,
this is what i posted :
کلمه
and this is what i read it from inside of py code:
b"%DA%A9%D9%84%D9%85%D9%87"
This is a byte string but i can't convert it or read it ,
I use encode and decode methods but none of these are not worked .
I use python3.4 and wsgi and mod_wsgi(apache2).
I use urllib module of python, with this code and worked :
fm = urllib.parse.parse_qs(request_body['family'].encode().decode(),True) # return a dictionary
familyvalue = str([k for k in fm.keys()][0]) # access to first item
is this a right way ?

ComplexModel not available on the client

I just started using Spyne and tried to use a ComplexModel as a parameter for one method. I mostly followed the user_manager example from the sources with spyne<2.99 but I always get a type error when doing the client.factory.create() call.
Example code that fails:
from spyne.application import Application
from spyne.decorator import rpc
from spyne.service import ServiceBase
from spyne.protocol.soap import Soap11
from spyne.model.primitive import String, Integer
from spyne.model.complex import ComplexModel
class DatosFac(ComplexModel):
__namespace__ = 'facturamanager.datosfac'
numero = String(pattern=r'[A-Z]/[0-9]+')
class FacturaService(ServiceBase):
#rpc(String, DatosFac, _returns=Integer)
def updateFacData(self, numero, data):
# do stuff
return 1
application = Application([FacturaService], 'facturaManager.service',
in_protocol=Soap11(validator='lxml'),
out_protocol=Soap11()
)
from spyne.server.null import NullServer
s = NullServer(application)
data = s.factory.create('DatosFac')
If you run this code you get:
Traceback (most recent call last):
File "spyner.py", line 25, in <module>
data = s.factory.create('DatosFac')
File "/Users/marc/.pyEnvs/default/lib/python2.7/site-packages/spyne/client/_base.py", line 30, in create
return self.__app.interface.get_class_instance(object_name)
File "/Users/marc/.pyEnvs/default/lib/python2.7/site-packages/spyne/interface/_base.py", line 114, in get_class_instance
return self.classes[key]()
KeyError: 'DatosFac'
(I used NullServer to make it easier to reproduce, but the same happens over Soap+Wsgi).
I amb pretty much stuck at this as I don't see what's essentialy different from this code and the user_manager examples.
What am I doing wrong?
thanks,
marc
Thanks for providing a fully working example.
The difference is that tns and the namespace of the DatosFac are different.
Either do:
data = s.factory.create('{facturamanager.datosfac}DatosFac')
or remove __namespace__ from DatosFac definition