How to pass uploaded file into amazon s3? - amazon-s3

I have this in my code:
size=128,128
#app.route('/',methods=['GET','POST'])
def upload():
print request.method
if request.method == 'POST':
file = request.files['image']
im = Image.open(file)
im.resize(size)
im.save("test.png","PNG")
f=open("test.png",'r')
conn = tinys3.Connection('AKIAI2GPQ','fAQxDLbvZcqhXvjd',tls=True)
conn.upload(im,f,"snappie.watermarks")
print "got file"
return redirect("https://www.google.com")
return render_template('index.html')
hopefully you can see that I am trying to handle the file upload from request.files, resize it, and then upload that to amazon s3. However right now its getting hung up on the conn.upload(im,f,"snappie.watermarks") line.
This is the error:
File "/home/alex/snappie/web/server.py", line 25, in upload
conn.upload(im,f,"snappie.watermarks")
File "/usr/local/lib/python2.7/dist-packages/tinys3/connection.py", line 152, in upload
return self.run(r)
File "/usr/local/lib/python2.7/dist-packages/tinys3/connection.py", line 233, in run
return self._handle_request(request)
File "/usr/local/lib/python2.7/dist-packages/tinys3/connection.py", line 255, in _handle_request
return request.run()
File "/usr/local/lib/python2.7/dist-packages/tinys3/request_factory.py", line 147, in run
headers['Content-Type'] = self.content_type or mimetypes.guess_type(self.key)[0] or 'application/octet-stream'
File "/usr/lib/python2.7/mimetypes.py", line 298, in guess_type
return _db.guess_type(url, strict)
File "/usr/lib/python2.7/mimetypes.py", line 114, in guess_type
scheme, url = urllib.splittype(url)
File "/usr/lib/python2.7/urllib.py", line 1074, in splittype
match = _typeprog.match(url)
TypeError: expected string or buffer
Apparently its having issues with one of those 3 arguments but I am not sure which one? I am also not sure I am handling the file correctly. Do I need to save the image and then re-open it in order to upload it to amazon s3? I do this because all the tinys3 examples do so, but my file is already open so perhaps its redundant?

It looks like your image argument might be mixed up. Here's the signature we want:
conn.upload(key, local_file, bucket)
tinys3 expects a string value for the key and an open file-like object for local_file. Try the following:
conn.upload("test.png", f, "snappie.watermarks")

Related

Sqlfluff: Cannot instantiate a templated file unsliced

I recently decided to add sqlfluff to my dbt project so i just followed https://docs.sqlfluff.com/en/stable/production.html and added sqlfluff with pre-commit to my project.
Following is how my very basic .sqlfluff configurations look like:
[sqlfluff]
dialect = postgres
templater = jinja
output_line_length = 80
ignore_templated_areas = True
runaway_limit = 100
[sqlfluff:rules]
tab_space_size = 2
max_line_length = 120
indent_unit = space
comma_style = trailing
[sqlfluff:rules:L014]
extended_capitalisation_policy = lower
[sqlfluff:templater:jinja]
apply_dbt_builtins = true
Here is .pre-commit-config.yaml:
repos:
- repo: https://github.com/sqlfluff/sqlfluff
rev: 1.0.0
hooks:
- id: sqlfluff-lint
name: sqlfluff-lint
entry: sqlfluff lint
language: python
description: 'Lints sql files with `SQLFluff`'
types: [sql]
require_serial: true
additional_dependencies: []
- id: sqlfluff-fix
name: sqlfluff-fix
# Needs to use "--force" to disable confirmation
# By default all the rules are applied
entry: sqlfluff fix --force
language: python
description: 'Fixes sql lint errors with `SQLFluff`'
types: [sql]
require_serial: true
additional_dependencies: []
Once I run pre-commit run --all-files I end up getting the following:
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/bin/sqlfluff", line 8, in <module>
sys.exit(cli())
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/cli/commands.py", line 769, in fix
result = lnt.lint_paths(
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/linter.py", line 1143, in lint_paths
self.lint_path(
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/linter.py", line 1095, in lint_path
for i, linted_file in enumerate(runner.run(fnames, fix), start=1):
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/runner.py", line 101, in run
for fname, partial in self.iter_partials(fnames, fix=fix):
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/runner.py", line 54, in iter_partials
for fname, rendered in self.iter_rendered(fnames):
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/runner.py", line 43, in iter_rendered
yield fname, self.linter.render_file(fname, self.config)
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/linter.py", line 816, in render_file
return self.render_string(raw_file, fname, config, encoding)
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/linter.py", line 787, in render_string
templated_file, templater_violations = self.templater.process(
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/templaters/jinja.py", line 413, in process
TemplatedFile(
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/templaters/base.py", line 102, in __init__
raise ValueError("Cannot instantiate a templated file unsliced!")
ValueError: Cannot instantiate a templated file unsliced!```
Not sure what breaking this
so first off your configuration has a bunch of unnecessary things in it -- let's pare that down (you also don't need both -lint and -fix -- they're redundant):
repos:
- repo: https://github.com/sqlfluff/sqlfluff
rev: 1.0.0
hooks:
- id: sqlfluff-fix
and with that configuration I can't reproduce your error -- you'll need to provide a sql file which triggers the particular error I suspect and your associated templates
disclaimer: I created pre-commit
I had the same issue, and making sure to exclude folders (macros and dbt_packages) having templated files solved this issue for me.
https://docs.sqlfluff.com/en/stable/configuration.html
Print screen of default .sqlfluffignore file on official site:
.sqlfluffignore default file
I hope this helps!

Python3/Redis: redis.exceptions.ResponseError: unknown command 'JSON.SET'

I'm trying to run the sample program from this RedisLabs page.
I chose Option A - which was to set up the free Redis cloud server.
(Seems like if you install manually, then you have to add the JSON as a plugin.)
I'm able to connect and use other "set" commands, but getting error on JSON:
File "C:\Users\nwalt\.virtualenvs\TDAmeritradeGetQuotes\lib\site-packages\redis\client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "C:\Users\nwalt\.virtualenvs\TDAmeritradeGetQuotes\lib\site-packages\redis\client.py", line 915, in parse_response
response = connection.read_response()
File "C:\Users\nwalt\.virtualenvs\TDAmeritradeGetQuotes\lib\site-packages\redis\connection.py", line 756, in read_response
raise response
redis.exceptions.ResponseError: unknown command 'JSON.SET'
My Python test program (except put in the sample endpoint before posting):
import redis
import json
import pprint
host_info = "redis.us-east-1-1.ec2.cloud.redislabs.com"
redisObj = redis.Redis(host=host_info, port=18274, password='xxx')
print ("Normal call to Redis")
redisObj.set('foo', 'bar')
value = redisObj.get('foo')
print(value)
capitals = {
"Lebanon": "Beirut",
"Norway": "Oslo",
"France": "Paris"
}
print ("capitals - before call to Redis")
pprint.pprint(capitals)
print("JSON call to Redis")
redisObj.execute_command('JSON.SET', 'doc', '.', json.dumps(capitals))
print("Data Saved, now fetch data back from redis")
reply = json.loads(redisObj.execute_command('JSON.GET', 'doc'))
print("reply from Redis get")
pprint.pprint(reply)
This is the screen shot from their website where I created the database. I didn't see any option to enable JSON or add any modules.
Not sure this was available when I created the REDIS database, but it is now. When you create it on redislabs.com, you can turn on the modules, and pick one from the list.
Then use this library: "rejson" from https://pypi.org/project/rejson/ to get the method "jsonset" method, using such code such as this:
rj = Client(host=config_dict['REDIS_CONFIG_HOST'], port=config_dict['REDIS_CONFIG_PORT'], password=config_dict['REDIS_CONFIG_PASSWORD'], decode_responses=True)
out_doc = {}
out_doc['firstname'] = "John"
out_doc['lastname'] = "Doe"
rj.jsonset('config', Path.rootPath(), out_doc)
get_doc = rj.jsonget('config', Path.rootPath())
pprint.pprint(get_doc)
I'm not used the cloud redis, in my local the Python don't load the JSON.SET
I just so make done, in this sample https://onelinerhub.com/python-redis/save-json-to-redis

aiogram.utils.exceptions.WrongFileIdentifier: Wrong file identifier/http url specified

I want to make a bot to which you send a link to the file, and it sends you the file, but I get an error from the header. I also tried to send InputFile object, but nothing was sent. Here's my code
from aiogram import Bot, types
from aiogram.dispatcher import Dispatcher
from aiogram.utils import executor
from aiogram.types.input_file import InputFile
bot = Bot(token = '')
dp = Dispatcher(bot)
#dp.message_handler()
async def send_playlist(message: types.Message):
print(message.text)
await bot.send_document(message.chat.id, message.text)
executor.start_polling(dp)
Here's full error text
future: <Task finished name='Task-14' coro=<Dispatcher._process_polling_updates() done, defined at B:\portable_soft\python\lib\site-packages\aiogram\dispatcher\dispatcher.py:331> exception=WrongFileIdentifier('Wrong file identifier/http url specified')>
Traceback (most recent call last):
File "B:\portable_soft\python\lib\site-packages\aiogram\dispatcher\dispatcher.py", line 339, in _process_polling_updates
for responses in itertools.chain.from_iterable(await self.process_updates(updates, fast)):
File "B:\portable_soft\python\lib\site-packages\aiogram\dispatcher\dispatcher.py", line 194, in process_updates
return await asyncio.gather(*tasks)
File "B:\portable_soft\python\lib\site-packages\aiogram\dispatcher\handler.py", line 117, in notify
response = await handler_obj.handler(*args, **partial_data)
File "B:\portable_soft\python\lib\site-packages\aiogram\dispatcher\dispatcher.py", line 214, in process_update
return await self.message_handlers.notify(update.message)
File "B:\portable_soft\python\lib\site-packages\aiogram\dispatcher\handler.py", line 117, in notify
response = await handler_obj.handler(*args, **partial_data)
File "B:\portable_soft\adb\file_bot.py", line 12, in send_playlist
await bot.send_document(message.chat.id, message.text)
File "B:\portable_soft\python\lib\site-packages\aiogram\bot\bot.py", line 402, in send_document
result = await self.request(api.Methods.SEND_DOCUMENT, payload, files)
File "B:\portable_soft\python\lib\site-packages\aiogram\bot\base.py", line 201, in request
return await api.make_request(self.session, self.__token, method, data, files,
File "B:\portable_soft\python\lib\site-packages\aiogram\bot\api.py", line 104, in make_request
return check_result(method, response.content_type, response.status, await response.text())
File "B:\portable_soft\python\lib\site-packages\aiogram\bot\api.py", line 78, in check_result
exceptions.BadRequest.detect(description)
File "B:\portable_soft\python\lib\site-packages\aiogram\utils\exceptions.py", line 136, in detect
raise err(cls.text or description)
aiogram.utils.exceptions.WrongFileIdentifier: Wrong file identifier/http url specified```
From your question, I don't really understand what you are trying to do. If you describe your use case a little better, I can probably suggest something more useful. What are you trying to do with playlists?
Here is one way to work with files:
#dp.message_handler(regexp='(^cat[s]?$|puss)')
async def cats(message: types.Message):
with open('data/cats.jpg', 'rb') as photo:
'''
# Old fashioned way:
await bot.send_photo(
message.chat.id,
photo,
caption='Cats are here 😺',
reply_to_message_id=message.message_id,
)
'''
await message.reply_photo(photo, caption='Cats are here 😺')
I actually prefer using FileIDs but I am not sure if that will help you.
Um, just write:
await bot.send_message(...)
but not:
await bot.send_DOCUMENT(...)
and if an error pops up:
from aiogram import md
await bot.send_message(msg.from_user.id, md.quote_html(msg.text))

What is the best way to get a content object's position in the parent? Plone 4

I am traversing folders with content items within them. I use the portal_catalog to get brains searched on certain paths. The brains have access to metadata, and brain.getObject() will return the actual object. I have gotten the parent for an object with brain.getObject().aq_parent. Now I want to get the object's position in the parent. At first I tried brain.getObject().getObjPositionInParent(), and afterwards, I realized that the getObjPositionInParent() is an attribute accessible from the index data.
idxData = catalog.getIndexDataForRID(brain.getRID())
sJson = json.dumps( idxData )
l = brain.getObject()
lUpdate = {'path': '/'.join( l.getPhysicalPath()), 'meta_type': l.meta_type, 'title':l.getRawTitle(), 'remoteUrl': l.getRemoteUrl(), 'json':sJson}
When I printed this out to the screen, I see all of the items within the dict that is returned from the catalog.getIndexDataForRID call. The problem is that for all of the objects, getObjPositionInParent() is an empty array ([]). On this page http://developer.plone.org/searching_and_indexing/query.html, it appears the value should be an integer. This made me wonder if I have to go create the index data, and, if so, then I might be reaching too far outward from the object to get data that must already be there (since the folders obviously know what position to put each child in). What is the best way to get a content object's position in the parent? Thanks in advance for any information?
More:
I am unsure why the adapter cannot be found, but it may have to do with lack of registering it. This is a script that I build the Zope environment to read the ZODB directly from the file as opposed to on top of the running Zope instance. Is it possible that I have to register the adapter with the GlobalSiteManager?
Thank you, Mathias. When I use the sort_on="getObjPositionInParent", I get the following error:
Traceback (most recent call last):
File "extractMenuStructure.py", line 459, in <module>
res = processFolder( home['childItems'], '/Sanford Guide Web Edition/' + appFolderNm + '', config['screens'] )
File "extractMenuStructure.py", line 390, in processFolder
results = portal_catalog(path={"query":currentPath, "depth":d},sort_on="getObjPositionInParent")
File "/Applications/Plone/buildout-cache/eggs/Products.CMFPlone-4.1.2-py2.6.egg/Products/CMFPlone/CatalogTool.py", line 427, in searchResults
return ZCatalog.searchResults(self, REQUEST, **kw)
File "/Applications/Plone/buildout-cache/eggs/Products.ZCatalog-2.13.20-py2.6.egg/Products/ZCatalog/ZCatalog.py", line 604, in searchResults
return self._catalog.searchResults(REQUEST, used, **kw)
File "/Applications/Plone/buildout-cache/eggs/Products.ZCatalog-2.13.20-py2.6.egg/Products/ZCatalog/Catalog.py", line 909, in searchResults
return self.search(args, sort_index, reverse, sort_limit, _merge)
File "/Applications/Plone/buildout-cache/eggs/Products.ZCatalog-2.13.20-py2.6.egg/Products/ZCatalog/Catalog.py", line 658, in search
b_size=b_size)
File "/Applications/Plone/buildout-cache/eggs/Products.ZCatalog-2.13.20-py2.6.egg/Products/ZCatalog/Catalog.py", line 678, in sortResults
index_key_map = sort_index.documentToKeyMap()
File "/Applications/Plone/buildout-cache/eggs/plone.app.folder-1.0.4-py2.6.egg/plone/app/folder/nogopip.py", line 91, in documentToKeyMap
ids = folder.getOrdering().idsInOrder()
File "/Applications/Plone/buildout-cache/eggs/plone.folder-1.0.1-py2.6.egg/plone/folder/ordered.py", line 41, in getOrdering
adapter = getAdapter(self, IOrdering)
File "/Applications/Plone/buildout-cache/eggs/zope.component-3.9.5-py2.6.egg/zope/component/_api.py", line 96, in getAdapter
raise ComponentLookupError(object, interface, name)
zope.component.interfaces.ComponentLookupError: (<ATFolder at /Sanford Guide Web Edition/amt>, <InterfaceClass plone.folder.interfaces.IOrdering>, u'')
The best way is to do like the index itself.
Code snipped based on the CatalogTool (Products.CMFPlone)
from Acquisition import aq_inner
from Acquisition import aq_parent
from OFS.interfaces import IOrderedContainer
obj = brain.getObject()
parent = aq_parent(aq_inner(obj))
ordered = IOrderedContainer(parent, None)
if ordered is not None:
return ordered.getObjectPosition(obj.getId())
return 0

How would I write the following applescript in Obj-C AppScript? ASTranslate was of no help =(

The translation tool isn't able to translate this working code. I copied it out of a working script.
set pathToTemp to (POSIX path of ((path to desktop) as string))
-- change jpg to pict
tell application "Image Events"
try
launch
set albumArt to open file (pathToTemp & "albumart.jpg")
save albumArt as PICT in file (pathToTemp & "albumart.pict")
--the first 512 bytes are the PICT header, so it reads from byte 513
--this is to allow the image to be added to an iTunes track later.
set albumArt to (read file (pathToTemp & "albumart.pict") from 513 as picture)
close
end try
end tell
The code is taking a jpg image, converting it to a PICT file, and then reading the file minus the header (the first 512 bytes). Later in the script, albumArt will be added to an iTunes track.
I tried translating the code (minus the comments), but ASTranslate froze for a good 2 minutes before giving me this:
Untranslated event 'earsffdr'
#import "IEGlue/IEGlue.h"
IEApplication *imageEvents = [IEApplication applicationWithName: #"Image Events"];
IELaunchCommand *cmd = [[imageEvents launch] ignoreReply];
id result = [cmd send];
#import "IEGlue/IEGlue.h"
IEApplication *imageEvents = [IEApplication applicationWithName: #"Image Events"];
IEReference *ref = [[imageEvents files] byName: #"/Users/Doom/Desktop/albumart.jpg"];
id result = [[ref open] send];
#import "IEGlue/IEGlue.h"
IEApplication *imageEvents = [IEApplication applicationWithName: #"Image Events"];
IEReference *ref = [[imageEvents images] byName: #"albumart.jpg"];
IESaveCommand *cmd = [[[ref save] in: [[imageEvents files] byName: #"/Users/Doom/Desktop/albumart.pict"]] as: [IEConstant PICT]];
id result = [cmd send];
'crdwrread'
Traceback (most recent call last):
File "objcrenderer.pyc", line 283, in renderCommand
KeyError: 'crdwrread'
'cascrgdut'
Traceback (most recent call last):
File "objcrenderer.pyc", line 283, in renderCommand
KeyError: 'cascrgdut'
'crdwrread'
Traceback (most recent call last):
File "objcrenderer.pyc", line 283, in renderCommand
KeyError: 'crdwrread'
Untranslated event 'rdwrread'
OK
I have no clue how to make sense of this.
Thanks for any and all help!
read is a Standard Additions command; ASTranslate doesn't support scripting additions, only scriptable applications, so you will have to translate it yourself.
Use ASDictionary to export the Standard Additions dictionary to HTML and create an objc-appscript glue (SAGlue). You can then use -[SAApplication init] to create a new SAApplication instance and send your read command to that.
Alternatively, you could skip Standard Additions completely and use NSData to read and slice the file and build the NSAppleEventDescriptor yourself.