There are many integrations for raven, including python logging. On the one side, twisted does not use python's logging. And on the other side, there is no direct integration for raven in twisted.
So what is the current best practice for using raven in a twisted based setup?
raven's captureException can only be called without arguments if there is an exception active, which is not always the case when a log observer is called. So, instead, pull the exception information out of the Failure that gets logged:
from twisted.python import log
from raven import Client
client = Client(dsn='twisted+http://YOUR_DSN_HERE')
def logToSentry(event):
if not event.get('isError') or 'failure' not in event:
return
f = event['failure']
client.captureException((f.type, f.value, f.getTracebackObject()))
log.addObserver(logToSentry)
user1252307 answer is a great start, but on the sentry side you get a relatively unhelpful dictionary and no stack trace.
If you are trying to view and track down unexpected exceptions try this small change to the log_sentry function:
from twisted.python import log
from raven import Client
client = Client(dsn='twisted+http://YOUR_DSN_HERE')
def log_sentry(dictionary):
if dictionary.get('isError'):
if 'failure' in dictionary:
client.captureException() # Send the current exception info to Sentry.
else:
#format the dictionary in whatever way you want
client.captureMessage(dictionary)
log.addObserver(log_sentry)
There maybe a better way to filter non exception based error message, and this might try to emit exception information that doesn't exist where there are failures which aren't exceptions.
from twisted.python import log
from raven import Client
client = Client(dsn='twisted+http://YOUR_DSN_HERE')
def log_sentry(dictionary):
if dictionary.get('isError'):
#format the dictionary in whatever way you want
client.captureMessage(dictionary)
log.addObserver(log_sentry)
Related
I have Apache Arrow data on the server (Python) and need to use it in the browser. It appears that Arrow Flight isn't implemented in JS. What are the best options for sending the data to the browser and using it there?
I don't even need it necessarily in Arrow format in the browser. This question hasn't received any responses, so I'm adding some additional criteria for what I'm looking for:
Self-describing: don't want to maintain separate schema definitions
Minimal overhead: For example, an array of float32s should transfer as something compact like a data type indicator, length value and sequence of 4-byte float values
Cross-platform: Able to be easily sent from Python and received and used in the browser in a straightforward way
Surely this is a solved problem? If it is I've been unable to find a solution. Please help!
Building off of the comments on your original post by David Li, you can implement a non-streaming version what you want without too much code using PyArrow on the server side and the Apache Arrow JS bindings on the client. The Arrow IPC format satisfies your requirements because it ships the schema with the data, is space-efficient and zero-copy, and is cross-platform.
Here's a toy example showing generating a record batch on server and receiving it on the client:
Server:
from io import BytesIO
from flask import Flask, send_file
from flask_cors import CORS
import pyarrow as pa
app = Flask(__name__)
CORS(app)
#app.get("/data")
def data():
data = [
pa.array([1, 2, 3, 4]),
pa.array(['foo', 'bar', 'baz', None]),
pa.array([True, None, False, True])
]
batch = pa.record_batch(data, names=['f0', 'f1', 'f2'])
sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, batch.schema) as writer:
writer.write_batch(batch)
return send_file(BytesIO(sink.getvalue().to_pybytes()), "data.arrow")
Client
const table = await tableFromIPC(fetch(URL));
// Do what you like with your data
Edit: I added a runnable example at https://github.com/amoeba/arrow-python-js-ipc-example.
How can I call some Python code when a new message is delivered from the Telethon API? I need to run the code all the day so that I can do my processing from Python code.
How to use this? #client.on(events.NewMessage(chats=channel, incoming=True))
Do I need run the scheduler to check this?
I am using history = client(GetHistoryRequest) method.
First Steps - Updates in the documentation greets you with the following code:
import logging
logging.basicConfig(format='[%(levelname) 5s/%(asctime)s] %(name)s: %(message)s',
level=logging.WARNING)
from telethon import TelegramClient, events
client = TelegramClient('anon', api_id, api_hash)
#client.on(events.NewMessage)
async def my_event_handler(event):
if 'hello' in event.raw_text:
await event.reply('hi!')
client.start()
client.run_until_disconnected()
Note you can "call" any Python code inside my_event_handler. It also shows how #client.on() is meant to be used. Note there is no need for a scheduler.
I am using history = client(GetHistoryRequest) method.
As a side note this is raw API which is discouraged if a friendly alternative, like client.get_messages, exists.
How to create a Persistent(or any for that matter) REST HANDLER for any given(inbuilt) SPLUNK REST API Endpoint? How to use PersistentServerConnectionApplication class ?
I have gone through https://gist.github.com/LukeMurphey/238004c8976804a8e79570d22721fd99 but cant figure out where to start and how to make one.
There was a great .conf presentation about REST Handlers by James Ervin from a few years ago, https://conf.splunk.com/files/2016/slides/extending-splunks-rest-api-for-fun-and-profit.pdf
Sample code is available from https://github.com/jrervin/splunk-rest-examples
James' echo example is quite straight forward. Make sure you also pay attention to the additions that are necessary in web.conf and restmap.conf.
import os
import sys
if sys.platform == "win32":
import msvcrt
# Binary mode is required for persistent mode on Windows.
msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
msvcrt.setmode(sys.stderr.fileno(), os.O_BINARY)
from splunk.persistconn.application import PersistentServerConnectionApplication
class EchoHandler(PersistentServerConnectionApplication):
def __init__(self, command_line, command_arg):
PersistentServerConnectionApplication.__init__(self)
def handle(self, in_string):
return {'payload': in_string, # Payload of the request.
'status': 200 # HTTP status code
}
Suggest you just get a copy of his app and deploy it, confirm it all works, then modify if for your particular use-case.
There are several similar questions on stack overflow, and I apologize in advance if I'm breaking etiquette by asking another one, but I just cannot seem to come up with the proper set of incantations to make this work.
I'm trying to use Flask + Flask-SQLAlchemy and then use pytest to manage the session such that when the function-scoped pytest fixture is torn down, the current transation is rolled back.
Some of the other questions seem to advocate using the db "drop all and create all" pytest fixture at the function scope, but I'm trying to use the joined session, and use rollbacks, since I have a LOT of tests. This would speed it up considerably.
http://alexmic.net/flask-sqlalchemy-pytest/ is where I found the original idea, and Isolating py.test DB sessions in Flask-SQLAlchemy is one of the questions recommending using function-level db re-creation.
I had also seen https://github.com/mitsuhiko/flask-sqlalchemy/pull/249 , but that appears to have been released with flask-sqlalchemy 2.1 (which I am using).
My current (very small, hopefully immediately understandable) repo is here:
https://github.com/hoopes/flask-pytest-example
There are two print statements - the first (in example/__init__.py) should have an Account object, and the second (in test/conftest.py) is where I expect the db to be cleared out after the transaction is rolled back.
If you pip install -r requirements.txt and run py.test -s from the test directory, you should see the two print statements.
I'm about at the end of my rope here - there must be something I'm missing, but for the life of me, I just can't seem to find it.
Help me, SO, you're my only hope!
You might want to give pytest-flask-sqlalchemy-transactions a try. It's a plugin that exposes a db_session fixture that accomplishes what you're looking for: allows you to run database updates that will get rolled back when the test exits. The plugin is based on Alex Michael's blog post, with some additional support for nested transactions that covers a wider array of user cases. There are also some configuration options for mocking out connectibles in your app so you can run arbitrary methods from your codebase, too.
For test_accounts.py, you could do something like this:
from example import db, Account
class TestAccounts(object):
def test_update_view(self, db_session):
test_acct = Account(username='abc')
db_session.add(test_acct)
db_session.commit()
resp = self.client.post('/update',
data={'a':1},
content_type='application/json')
assert resp.status_code == 200
The plugin needs access to your database through a _db fixture, but since you already have a db fixture defined in conftest.py, you can set up database access easily:
#pytest.fixture(scope='session')
def _db(db):
return db
You can find detail on how to setup and installation in the docs. Hope this helps!
I'm also having issues with the rollback, my code can be found here
After reading some documentation, it seems the begin() function should be called on the session.
So in your case I would update the session fixture to this:
#pytest.yield_fixture(scope='function', autouse=True)
def session(db, request):
"""Creates a new database session for a test."""
db.session.begin()
yield db.session
db.session.rollback()
db.session.remove()
I didn't test this code, but when I try it on my code I get the following error:
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "./venv/lib/python2.7/site-packages/_pytest/main.py", line 90, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
...
INTERNALERROR> File "./venv/lib/python2.7/site-packages/_pytest/python.py", line 59, in filter_traceback
INTERNALERROR> return entry.path != cutdir1 and not entry.path.relto(cutdir2)
INTERNALERROR> AttributeError: 'str' object has no attribute 'relto'
from sqlalchemy.orm import sessionmaker
from sqlalchemy import create_engine
from unittest import TestCase
# global application scope. create Session class, engine
Session = sessionmaker()
engine = create_engine('postgresql://...')
class SomeTest(TestCase):
def setUp(self):
# connect to the database
self.connection = engine.connect()
# begin a non-ORM transaction
self.trans = self.connection.begin()
# bind an individual Session to the connection
self.session = Session(bind=self.connection)
def test_something(self):
# use the session in tests.
self.session.add(Foo())
self.session.commit()
def tearDown(self):
self.session.close()
# rollback - everything that happened with the
# Session above (including calls to commit())
# is rolled back.
self.trans.rollback()
# return connection to the Engine
self.connection.close()
sqlalchemy doc has solution for the case
I have written a code in twisted .I need to write the log information in when we have call
d.addErrback(on_failure).
from twisted.python import log
log.startLogging(open('/home/crytek.etl/foo.log', 'w'))
def on_failure(failure):
log.msg(failure)
d.addErrback(on_failure)
Is this the correct way of implementing this.I don't get any values written to the file.Can someone suggest on how this can be implemented
You probably want to consider opening your log file in append mode. Otherwise, every time your application starts you'll wipe out all your old logs. This could make it appear as though the log messages you're expecting to see aren't being logged.
from twisted.python import log
log.startLogging(open('/home/crytek.etl/foo.log', 'a'))
You should also log failures using log.err instead of log.msg
def on_failure(failure):
log.err(failure)
And you can do this more easily since on_failure has exactly the same signature as log.err. Just write:
d.addErrback(log.err)
Also, I liked, log.err doesn't have exactly the same signature as on_failure. It is better, it accepts a 2nd argument which is used to present a header for the failure in the log file. You can use it like this:
d.addErrback(log.err, "Frobbing the widget failed")
This will present "Frobbing the widget failed" together with the failure in the log file.