I recently decided to add sqlfluff to my dbt project so i just followed https://docs.sqlfluff.com/en/stable/production.html and added sqlfluff with pre-commit to my project.
Following is how my very basic .sqlfluff configurations look like:
[sqlfluff]
dialect = postgres
templater = jinja
output_line_length = 80
ignore_templated_areas = True
runaway_limit = 100
[sqlfluff:rules]
tab_space_size = 2
max_line_length = 120
indent_unit = space
comma_style = trailing
[sqlfluff:rules:L014]
extended_capitalisation_policy = lower
[sqlfluff:templater:jinja]
apply_dbt_builtins = true
Here is .pre-commit-config.yaml:
repos:
- repo: https://github.com/sqlfluff/sqlfluff
rev: 1.0.0
hooks:
- id: sqlfluff-lint
name: sqlfluff-lint
entry: sqlfluff lint
language: python
description: 'Lints sql files with `SQLFluff`'
types: [sql]
require_serial: true
additional_dependencies: []
- id: sqlfluff-fix
name: sqlfluff-fix
# Needs to use "--force" to disable confirmation
# By default all the rules are applied
entry: sqlfluff fix --force
language: python
description: 'Fixes sql lint errors with `SQLFluff`'
types: [sql]
require_serial: true
additional_dependencies: []
Once I run pre-commit run --all-files I end up getting the following:
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/bin/sqlfluff", line 8, in <module>
sys.exit(cli())
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/cli/commands.py", line 769, in fix
result = lnt.lint_paths(
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/linter.py", line 1143, in lint_paths
self.lint_path(
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/linter.py", line 1095, in lint_path
for i, linted_file in enumerate(runner.run(fnames, fix), start=1):
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/runner.py", line 101, in run
for fname, partial in self.iter_partials(fnames, fix=fix):
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/runner.py", line 54, in iter_partials
for fname, rendered in self.iter_rendered(fnames):
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/runner.py", line 43, in iter_rendered
yield fname, self.linter.render_file(fname, self.config)
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/linter.py", line 816, in render_file
return self.render_string(raw_file, fname, config, encoding)
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/linter/linter.py", line 787, in render_string
templated_file, templater_violations = self.templater.process(
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/templaters/jinja.py", line 413, in process
TemplatedFile(
File "/root/.cache/pre-commit/repo20y2aa42/py_env-python3.8/lib/python3.8/site-packages/sqlfluff/core/templaters/base.py", line 102, in __init__
raise ValueError("Cannot instantiate a templated file unsliced!")
ValueError: Cannot instantiate a templated file unsliced!```
Not sure what breaking this
so first off your configuration has a bunch of unnecessary things in it -- let's pare that down (you also don't need both -lint and -fix -- they're redundant):
repos:
- repo: https://github.com/sqlfluff/sqlfluff
rev: 1.0.0
hooks:
- id: sqlfluff-fix
and with that configuration I can't reproduce your error -- you'll need to provide a sql file which triggers the particular error I suspect and your associated templates
disclaimer: I created pre-commit
I had the same issue, and making sure to exclude folders (macros and dbt_packages) having templated files solved this issue for me.
https://docs.sqlfluff.com/en/stable/configuration.html
Print screen of default .sqlfluffignore file on official site:
.sqlfluffignore default file
I hope this helps!
Related
I'm trying to run the sample program from this RedisLabs page.
I chose Option A - which was to set up the free Redis cloud server.
(Seems like if you install manually, then you have to add the JSON as a plugin.)
I'm able to connect and use other "set" commands, but getting error on JSON:
File "C:\Users\nwalt\.virtualenvs\TDAmeritradeGetQuotes\lib\site-packages\redis\client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "C:\Users\nwalt\.virtualenvs\TDAmeritradeGetQuotes\lib\site-packages\redis\client.py", line 915, in parse_response
response = connection.read_response()
File "C:\Users\nwalt\.virtualenvs\TDAmeritradeGetQuotes\lib\site-packages\redis\connection.py", line 756, in read_response
raise response
redis.exceptions.ResponseError: unknown command 'JSON.SET'
My Python test program (except put in the sample endpoint before posting):
import redis
import json
import pprint
host_info = "redis.us-east-1-1.ec2.cloud.redislabs.com"
redisObj = redis.Redis(host=host_info, port=18274, password='xxx')
print ("Normal call to Redis")
redisObj.set('foo', 'bar')
value = redisObj.get('foo')
print(value)
capitals = {
"Lebanon": "Beirut",
"Norway": "Oslo",
"France": "Paris"
}
print ("capitals - before call to Redis")
pprint.pprint(capitals)
print("JSON call to Redis")
redisObj.execute_command('JSON.SET', 'doc', '.', json.dumps(capitals))
print("Data Saved, now fetch data back from redis")
reply = json.loads(redisObj.execute_command('JSON.GET', 'doc'))
print("reply from Redis get")
pprint.pprint(reply)
This is the screen shot from their website where I created the database. I didn't see any option to enable JSON or add any modules.
Not sure this was available when I created the REDIS database, but it is now. When you create it on redislabs.com, you can turn on the modules, and pick one from the list.
Then use this library: "rejson" from https://pypi.org/project/rejson/ to get the method "jsonset" method, using such code such as this:
rj = Client(host=config_dict['REDIS_CONFIG_HOST'], port=config_dict['REDIS_CONFIG_PORT'], password=config_dict['REDIS_CONFIG_PASSWORD'], decode_responses=True)
out_doc = {}
out_doc['firstname'] = "John"
out_doc['lastname'] = "Doe"
rj.jsonset('config', Path.rootPath(), out_doc)
get_doc = rj.jsonget('config', Path.rootPath())
pprint.pprint(get_doc)
I'm not used the cloud redis, in my local the Python don't load the JSON.SET
I just so make done, in this sample https://onelinerhub.com/python-redis/save-json-to-redis
I want to make a bot to which you send a link to the file, and it sends you the file, but I get an error from the header. I also tried to send InputFile object, but nothing was sent. Here's my code
from aiogram import Bot, types
from aiogram.dispatcher import Dispatcher
from aiogram.utils import executor
from aiogram.types.input_file import InputFile
bot = Bot(token = '')
dp = Dispatcher(bot)
#dp.message_handler()
async def send_playlist(message: types.Message):
print(message.text)
await bot.send_document(message.chat.id, message.text)
executor.start_polling(dp)
Here's full error text
future: <Task finished name='Task-14' coro=<Dispatcher._process_polling_updates() done, defined at B:\portable_soft\python\lib\site-packages\aiogram\dispatcher\dispatcher.py:331> exception=WrongFileIdentifier('Wrong file identifier/http url specified')>
Traceback (most recent call last):
File "B:\portable_soft\python\lib\site-packages\aiogram\dispatcher\dispatcher.py", line 339, in _process_polling_updates
for responses in itertools.chain.from_iterable(await self.process_updates(updates, fast)):
File "B:\portable_soft\python\lib\site-packages\aiogram\dispatcher\dispatcher.py", line 194, in process_updates
return await asyncio.gather(*tasks)
File "B:\portable_soft\python\lib\site-packages\aiogram\dispatcher\handler.py", line 117, in notify
response = await handler_obj.handler(*args, **partial_data)
File "B:\portable_soft\python\lib\site-packages\aiogram\dispatcher\dispatcher.py", line 214, in process_update
return await self.message_handlers.notify(update.message)
File "B:\portable_soft\python\lib\site-packages\aiogram\dispatcher\handler.py", line 117, in notify
response = await handler_obj.handler(*args, **partial_data)
File "B:\portable_soft\adb\file_bot.py", line 12, in send_playlist
await bot.send_document(message.chat.id, message.text)
File "B:\portable_soft\python\lib\site-packages\aiogram\bot\bot.py", line 402, in send_document
result = await self.request(api.Methods.SEND_DOCUMENT, payload, files)
File "B:\portable_soft\python\lib\site-packages\aiogram\bot\base.py", line 201, in request
return await api.make_request(self.session, self.__token, method, data, files,
File "B:\portable_soft\python\lib\site-packages\aiogram\bot\api.py", line 104, in make_request
return check_result(method, response.content_type, response.status, await response.text())
File "B:\portable_soft\python\lib\site-packages\aiogram\bot\api.py", line 78, in check_result
exceptions.BadRequest.detect(description)
File "B:\portable_soft\python\lib\site-packages\aiogram\utils\exceptions.py", line 136, in detect
raise err(cls.text or description)
aiogram.utils.exceptions.WrongFileIdentifier: Wrong file identifier/http url specified```
From your question, I don't really understand what you are trying to do. If you describe your use case a little better, I can probably suggest something more useful. What are you trying to do with playlists?
Here is one way to work with files:
#dp.message_handler(regexp='(^cat[s]?$|puss)')
async def cats(message: types.Message):
with open('data/cats.jpg', 'rb') as photo:
'''
# Old fashioned way:
await bot.send_photo(
message.chat.id,
photo,
caption='Cats are here 😺',
reply_to_message_id=message.message_id,
)
'''
await message.reply_photo(photo, caption='Cats are here 😺')
I actually prefer using FileIDs but I am not sure if that will help you.
Um, just write:
await bot.send_message(...)
but not:
await bot.send_DOCUMENT(...)
and if an error pops up:
from aiogram import md
await bot.send_message(msg.from_user.id, md.quote_html(msg.text))
I am using the softLayer python API to change a username.
new_creds = {
'username': new_username,
'password': new_password
}
rc = client['User_Customer'].changeUsername(new_creds, id=user_id)
However I get the following exception:
...
File "/usr/local/lib/python2.7/dist-packages/SoftLayer/API.py", line 392, in call_handler
return self(name, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer/API.py", line 360, in call
return self.client.call(self.name, name, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer/API.py", line 263, in call
return self.transport(request)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer/transports.py", line 195, in __call__
raise _ex(ex.faultCode, ex.faultString)
SoftLayer.exceptions.SoftLayerAPIError: SoftLayerAPIError(Client): Function ("changeUsername") is not a valid method for this service
Is changeUsername not supported?
Currently the SoftLayer_User_Customer::changeUsername method is no longer supported. An issue is going to be reported in order to remove it from the SLDN documentation.
I have this in my code:
size=128,128
#app.route('/',methods=['GET','POST'])
def upload():
print request.method
if request.method == 'POST':
file = request.files['image']
im = Image.open(file)
im.resize(size)
im.save("test.png","PNG")
f=open("test.png",'r')
conn = tinys3.Connection('AKIAI2GPQ','fAQxDLbvZcqhXvjd',tls=True)
conn.upload(im,f,"snappie.watermarks")
print "got file"
return redirect("https://www.google.com")
return render_template('index.html')
hopefully you can see that I am trying to handle the file upload from request.files, resize it, and then upload that to amazon s3. However right now its getting hung up on the conn.upload(im,f,"snappie.watermarks") line.
This is the error:
File "/home/alex/snappie/web/server.py", line 25, in upload
conn.upload(im,f,"snappie.watermarks")
File "/usr/local/lib/python2.7/dist-packages/tinys3/connection.py", line 152, in upload
return self.run(r)
File "/usr/local/lib/python2.7/dist-packages/tinys3/connection.py", line 233, in run
return self._handle_request(request)
File "/usr/local/lib/python2.7/dist-packages/tinys3/connection.py", line 255, in _handle_request
return request.run()
File "/usr/local/lib/python2.7/dist-packages/tinys3/request_factory.py", line 147, in run
headers['Content-Type'] = self.content_type or mimetypes.guess_type(self.key)[0] or 'application/octet-stream'
File "/usr/lib/python2.7/mimetypes.py", line 298, in guess_type
return _db.guess_type(url, strict)
File "/usr/lib/python2.7/mimetypes.py", line 114, in guess_type
scheme, url = urllib.splittype(url)
File "/usr/lib/python2.7/urllib.py", line 1074, in splittype
match = _typeprog.match(url)
TypeError: expected string or buffer
Apparently its having issues with one of those 3 arguments but I am not sure which one? I am also not sure I am handling the file correctly. Do I need to save the image and then re-open it in order to upload it to amazon s3? I do this because all the tinys3 examples do so, but my file is already open so perhaps its redundant?
It looks like your image argument might be mixed up. Here's the signature we want:
conn.upload(key, local_file, bucket)
tinys3 expects a string value for the key and an open file-like object for local_file. Try the following:
conn.upload("test.png", f, "snappie.watermarks")
I am traversing folders with content items within them. I use the portal_catalog to get brains searched on certain paths. The brains have access to metadata, and brain.getObject() will return the actual object. I have gotten the parent for an object with brain.getObject().aq_parent. Now I want to get the object's position in the parent. At first I tried brain.getObject().getObjPositionInParent(), and afterwards, I realized that the getObjPositionInParent() is an attribute accessible from the index data.
idxData = catalog.getIndexDataForRID(brain.getRID())
sJson = json.dumps( idxData )
l = brain.getObject()
lUpdate = {'path': '/'.join( l.getPhysicalPath()), 'meta_type': l.meta_type, 'title':l.getRawTitle(), 'remoteUrl': l.getRemoteUrl(), 'json':sJson}
When I printed this out to the screen, I see all of the items within the dict that is returned from the catalog.getIndexDataForRID call. The problem is that for all of the objects, getObjPositionInParent() is an empty array ([]). On this page http://developer.plone.org/searching_and_indexing/query.html, it appears the value should be an integer. This made me wonder if I have to go create the index data, and, if so, then I might be reaching too far outward from the object to get data that must already be there (since the folders obviously know what position to put each child in). What is the best way to get a content object's position in the parent? Thanks in advance for any information?
More:
I am unsure why the adapter cannot be found, but it may have to do with lack of registering it. This is a script that I build the Zope environment to read the ZODB directly from the file as opposed to on top of the running Zope instance. Is it possible that I have to register the adapter with the GlobalSiteManager?
Thank you, Mathias. When I use the sort_on="getObjPositionInParent", I get the following error:
Traceback (most recent call last):
File "extractMenuStructure.py", line 459, in <module>
res = processFolder( home['childItems'], '/Sanford Guide Web Edition/' + appFolderNm + '', config['screens'] )
File "extractMenuStructure.py", line 390, in processFolder
results = portal_catalog(path={"query":currentPath, "depth":d},sort_on="getObjPositionInParent")
File "/Applications/Plone/buildout-cache/eggs/Products.CMFPlone-4.1.2-py2.6.egg/Products/CMFPlone/CatalogTool.py", line 427, in searchResults
return ZCatalog.searchResults(self, REQUEST, **kw)
File "/Applications/Plone/buildout-cache/eggs/Products.ZCatalog-2.13.20-py2.6.egg/Products/ZCatalog/ZCatalog.py", line 604, in searchResults
return self._catalog.searchResults(REQUEST, used, **kw)
File "/Applications/Plone/buildout-cache/eggs/Products.ZCatalog-2.13.20-py2.6.egg/Products/ZCatalog/Catalog.py", line 909, in searchResults
return self.search(args, sort_index, reverse, sort_limit, _merge)
File "/Applications/Plone/buildout-cache/eggs/Products.ZCatalog-2.13.20-py2.6.egg/Products/ZCatalog/Catalog.py", line 658, in search
b_size=b_size)
File "/Applications/Plone/buildout-cache/eggs/Products.ZCatalog-2.13.20-py2.6.egg/Products/ZCatalog/Catalog.py", line 678, in sortResults
index_key_map = sort_index.documentToKeyMap()
File "/Applications/Plone/buildout-cache/eggs/plone.app.folder-1.0.4-py2.6.egg/plone/app/folder/nogopip.py", line 91, in documentToKeyMap
ids = folder.getOrdering().idsInOrder()
File "/Applications/Plone/buildout-cache/eggs/plone.folder-1.0.1-py2.6.egg/plone/folder/ordered.py", line 41, in getOrdering
adapter = getAdapter(self, IOrdering)
File "/Applications/Plone/buildout-cache/eggs/zope.component-3.9.5-py2.6.egg/zope/component/_api.py", line 96, in getAdapter
raise ComponentLookupError(object, interface, name)
zope.component.interfaces.ComponentLookupError: (<ATFolder at /Sanford Guide Web Edition/amt>, <InterfaceClass plone.folder.interfaces.IOrdering>, u'')
The best way is to do like the index itself.
Code snipped based on the CatalogTool (Products.CMFPlone)
from Acquisition import aq_inner
from Acquisition import aq_parent
from OFS.interfaces import IOrderedContainer
obj = brain.getObject()
parent = aq_parent(aq_inner(obj))
ordered = IOrderedContainer(parent, None)
if ordered is not None:
return ordered.getObjectPosition(obj.getId())
return 0