Write in memory object to S3 via boto3 - amazon-s3

I am attempting to write files directly to S3 without creating a local file which is then uploaded.
I am using cStringIO to generate a file in memory, but I am having trouble figuring out the proper way to upload it in boto3.
def writetos3(sourcedata, filename, folderpath):
s3 = boto3.resource('s3')
data = open(sourcedata, 'rb')
s3.Bucket('bucketname').put_object(Key= folderpath + "/" + filename, Body=data)
Above is the standard boto3 method that I was using previously with the local file, it does not work without a local file, I get the following error: coercing to Unicode: need string or buffer, cStringIO.StringO found
.
Because the in memory file (I believe) is already considered open, I tried changing it to the code below, but it still does not work, no error is given the script simply hangs on the last line of the method.
def writetos3(sourcedata, filename, folderpath):
s3 = boto3.resource('s3')
s3.Bucket('bucketname').put_object(Key= folderpath + "/" + filename, Body=sourcedata)
Just for more info, the value I am attempting to write looks like this
(cStringIO.StringO object at 0x045DC540)
Does anyone have an idea of what I am doing wrong here?

It looks like you want this:
data = open(sourcedata, 'rb').decode()
It defaults to utf8. Also, I encourage you to run your code under python3, and to use appropriate language tags for your question.

Related

After building an App(Android) using Kivy, Reading and writing text file(authorization completed)

I am using Filechooser, and checked whether the correct file is loaded using the print function until the data path.
text file name: A.txt
self.data_name = str(self.list_data_load.selection[0])
self.fdata=self.data_name
check_error= 'Path' + self.fdata
f=open(self.fdata,"r")
/emulated/0/data/A.txt
#F=f.readline()
but, when I try to use readlines, the app crashes and shuts down in Andorid.
I'm not sure if I need to grant additional permissions to read text files in external storage.
I need some help please.

Using blockchain_parser to parse blk files stored on S3

I'm trying to use alecalve's bitcoin-parser pkg for python.
The problem is, my node is saved on an s3 bucket.
As the parser uses os.path.expanduser for the .dat files dir, expecting a filesystem, I can't just use my s3 path. The example from the documentation is:
import os
from blockchain_parser.blockchain import Blockchain
blockchain = Blockchain(os.path.expanduser('~/.bitcoin/blocks'))
for block in blockchain.get_ordered_blocks(os.path.expanduser('~/.bitcoin/blocks/index'), end=1000):
print("height=%d block=%s" % (block.height, block.hash))
And the error I'm getting is as follows:
'name' arg must be a byte string or a unicode string
Is there a way to use s3fs or any different s3-to-filesystem method to use the s3 paths as dirs for the parser to work as intended?

pandas.read_csv of a gzip file within a zipped directory

I would like to use pandas.read_csv to open a gzip file (.asc.gz) within a zipped directory (.zip). Is there an easy way to do this?
This code doesn't work:
csv = pd.read_csv(r'C:\folder.zip\file.asc.gz') // can't find the file
This code does work (however, it requires me to unzip the folder, which I want to avoid because my dataset currently contains thousands of zipped folders):
csv = pd.read_csv(r'C:\folder\file.asc.gz')
Is there an easy way to do this? I have tried using a combination of zipfile.Zipfile and read_csv, but have been unsuccessful (I think partly due to the fact that this is an ascii file as well)
Maybe the followings might help.
df = pd.read_csv('filename.gz', compression='gzip')
OR
import gzip
file=gzip.open('filename.gz','rb')
content=file.read()

Apache NiFi - Process content of a CSV S3Object

I am trying to process a CSV stored in a S3 Bucket with Apache NiFi.
For this aim I am using the following flow:
The thing is that I need to replace some text of the csv file, but what I get as a output of FetchS3Object is a file, not a text.
How can I access to the text of the S3Object?
Thanks!
Finally I found the way to fix the problem.
Using SplitText processor after FetchS3Object is possible to split line by line (count = 1) and process each line after that processor.

No such file or directory when opening file in memory with ZipRuby Zip::File

I'm consuming an api that replies with a zip file in the contents of the body of the http response. I'm able to unzip the file and write each file to disk using the example at the zip-ruby wiki (https://bitbucket.org/winebarrel/zip-ruby/wiki/Home):
Zip::Archive.open('filename.zip') do |ar| # except I'm opening from a string in memory
ar.each do |zf|
if zf.directory?
FileUtils.mkdir_p(zf.name)
else
dirname = File.dirname(zf.name)
FileUtils.mkdir_p(dirname) unless File.exist?(dirname)
open(zf.name, 'wb') do |f|
f << zf.read
end
end
end
end
However, I don't want to write the files to disk. Instead, I want create an active record object, and set a paperclip file attachment:
asset = Asset.create(:name => zf.name)
asset.file = open(zf.name, 'r')
asset.save
What's odd is the open statement in the first example that writes the file to disk works consistently. However, when I want to just open the zf (Zip::File) as a generic File in memory, I will sometimes get:
*** Errno::ENOENT Exception: No such file or directory - assets/somefilename.png
How can I assign the Zip::File zipruby creates to the paperclip file without getting this error?