Bad readable state with nanomsg surveyor socket - nanomsg

I am using the SURVEY scalability protocol with nanomsg in Python.
I have a bunch of respondent sockets, I connect a surveyor to them
in a loop; then, I wait for surveyor.send_fd to be readable. I assume it is connected to at least one respondent,
then I send the survey.
On the respondent side, I wait for respondent.recv_fd to be readable, and I call respondent.recv() to know about the
survey, then I send the answer.
Back to the surveyor, I wait for surveyor.recv_fd to be
readable and I call surveyor.recv() to get answers from respondents.
Sometimes I get an error when trying to read answers from respondents:
Traceback (most recent call last):
[...]
nanomsg.NanoMsgAPIError: Operation cannot be performed in this state
I simplify the code to show the issue:
test_respondent.py
import nanomsg
import select
resp = nanomsg.Socket(nanomsg.RESPONDENT)
resp.bind("tcp://*:6542")
print 'waiting for respondent to be readable'
select.select([resp.recv_fd],[],[])
print 'receiving survey'
print resp.recv(flags=nanomsg.DONTWAIT)
print 'sending reply to surveyor'
resp.send("reply")
test_surveyor.py
import select
import nanomsg
s = nanomsg.Socket(nanomsg.SURVEYOR)
s.connect("tcp://127.0.0.1:6542")
print 'waiting for surveyor to be connected'
select.select([s.send_fd],[],[])
s.send("hello", nanomsg.DONTWAIT)
print "waiting for surveyor to be readable"
select.select([s.recv_fd],[],[])
print 'receiving reply from respondent'
print s.recv(flags=nanomsg.DONTWAIT)
test.sh
python test_respondent.py &
python test_surveyor.py
Executing the test several times should lead to the exception described
above in some cases.
Any idea what I am doing wrong ? And how to fix the problem ?

Related

Celery Error Handling

I've built a fairly simple application linking Flask, Celery, and RabbitMQ using docker-compose by linking together a few solutions I saw online. I'm having some issues trying to update task states to reflect if a failure occurred. To keep error visibility at it's highest, I've had my custom class only raise expected errors, else the errors are handled at the celery app level as follows (in celery_app.py):
#celery_app.task(name='celery_worker.summary')
def async_summary(data):
"""Background summary processing"""
try:
logger.info('Summarizing text')
return BdsSummary(data, nlp=en_nlp).create_summary()
except Exception as e:
current_task.update_state(state='FAILURE', meta={'error_message': str(traceback.format_exc())})
logger.exception('Text Summary worker raised: %r'%e)
I've been doing some negative testing against my application, and when I pass it data that I know will throw an error (non-text data, for example), when i run r = requests.get('http://my.app.addr:8888/task/my-task-id') I get {'status': 'SUCCESS', 'result': None}. I'm vexed as to why this is happening. Based on my admittedly limited understanding of Celery's behavior, it should update the status to show a traceback and ExceptionClass, why would it not do this?
I am relatively new to Celery, so my understanding of the Canvas that they reference in the documentation is extremely basic. I'm just trying to provide some basic task failure information to the response/task. For context, when I give it proper input, I get back {'status': 'SUCCESS', 'result': {'summary': 'My Summary text here', 'num_sentences': 3, ...}}.
Any insight here would be much appreciated

Twisted deferreds block when URI is the same (multiple calls from the same browser)

I have the following code
# -*- coding: utf-8 -*-
# 好
##########################################
import time
from twisted.internet import reactor, threads
from twisted.web.server import Site, NOT_DONE_YET
from twisted.web.resource import Resource
##########################################
class Website(Resource):
def getChild(self, name, request):
return self
def render(self, request):
if request.path == "/sleep":
duration = 3
if 'duration' in request.args:
duration = int(request.args['duration'][0])
message = 'no message'
if 'message' in request.args:
message = request.args['message'][0]
#-------------------------------------
def deferred_activity():
print 'starting to wait', message
time.sleep(duration)
request.setHeader('Content-Type', 'text/plain; charset=UTF-8')
request.write(message)
print 'finished', message
request.finish()
#-------------------------------------
def responseFailed(err, deferred):
pass; print err.getErrorMessage()
deferred.cancel()
#-------------------------------------
def deferredFailed(err, deferred):
pass; # print err.getErrorMessage()
#-------------------------------------
deferred = threads.deferToThread(deferred_activity)
deferred.addErrback(deferredFailed, deferred) # will get called indirectly by responseFailed
request.notifyFinish().addErrback(responseFailed, deferred) # to handle client disconnects
#-------------------------------------
return NOT_DONE_YET
else:
return 'nothing at', request.path
##########################################
reactor.listenTCP(321, Site(Website()))
print 'starting to serve'
reactor.run()
##########################################
# http://localhost:321/sleep?duration=3&message=test1
# http://localhost:321/sleep?duration=3&message=test2
##########################################
My issue is the following:
When I open two tabs in the browser, point one at http://localhost:321/sleep?duration=3&message=test1 and the other at http://localhost:321/sleep?duration=3&message=test2 (the messages differ) and reload the first tab and then ASAP the second one, then the finish almost at the same time. The first tab about 3 seconds after hitting F5, the second tab finishes about half a second after the first tab.
This is expected, as each request got deferred into a thread, and they are sleeping in parallel.
But when I now change the URL of the second tab to be the same as the one of the first tab, that is to http://localhost:321/sleep?duration=3&message=test1, then all this becomes blocking. If I press F5 on the first tab and as quickly as possible F5 on the second one, the second tab finishes about 3 seconds after the first one. They don't get executed in parallel.
As long as the entire URI is the same in both tabs, this server starts to block. This is the same in Firefox as well as in Chrome. But when I start one in Chrome and another one in Firefox at the same time, then it is non-blocking again.
So it may not neccessarily be related to Twisted, but maybe because of some connection reusage or something like that.
Anyone knows what is happening here and how I can solve this issue?
Coincidentally, someone asked a related question over at the Tornado section. As you suspected, this is not an "issue" in Twisted but rather a "feature" of web browsers :). Tornado's FAQ page has a small section dedicated to this issue. The proposed solution is appending an arbitrary query string.
Quote of the day:
One dev's bug is another dev's undocumented feature!

what does greenthread.sleep do?

I'm pretty new with eventlet and have some questions on sleep()
I tested with a small piece of code.
At first I spawned 3 greenthreads and then called greenthread.sleep(0), then these 3 greenthreads all came to execute the functions in them. what's going on?
does sleep() mean execute all the greenthread spawned? what does the argument 0 we passed in mean?
Here is the code:
import eventlet
from eventlet import greenthread
from eventlet import event
evt = event.Event()
def func1():
print "starting func1"
evt.wait()
print "stopping func1"
def func2():
print "starting func2"
evt.wait()
print "stopping func2"
def func3():
evt.send()
gt1 = greenthread.spawn(func1)
gt2 = greenthread.spawn(func2)
gt3 = greenthread.spawn(func3)
greenthread.sleep(0)
That's a great question, it deserves a special place in Eventlet documentation.
eventlet.sleep(0) reschedules the calling greenthread to the end of run queue. If there were any other greenthreads waiting to run, they will execute now.
Current implementation detail of Eventlet has a certain guarantee that if you call sleep, the calling greenthread will not continue until all other greenthreads that are ready to execute are finished or came to similar wait state. Started as implementation detail, we keep it as a public API now: call sleep(0) to let others run.

Django traceback on queries

I want a traceback from every query executed during a request, so I can find where they're coming from and reduce the count/complexity.
I'm using this excellent snippet of middleware to list and time queries, but I don't know where in the they're coming from.
I've poked around in django/db/models/sql/compiler.py but apparent form getting a local version of django and editing that code I can't see how to latch on to queries. Is there a signal I can use? it seems like there isn't a signal on every query.
Is it possible to specify the default Manager?
(I know about django-toolbar, I'm hoping there's a solution without using it.)
An ugly but effective solution (eg. it prints the trace on all queries and only requires one edit) is to add the following to the bottom of settings.py:
import django.db.backends.utils as bakutils
import traceback
bakutils.CursorDebugWrapper_orig = bakutils.CursorWrapper
def print_stack_in_project():
stack = traceback.extract_stack()
for path, lineno, func, line in stack:
if 'lib/python' in path or 'settings.py' in path:
continue
print 'File "%s", line %d, in %s' % (path, lineno, func)
print ' %s' % line
class CursorDebugWrapperLoud(bakutils.CursorDebugWrapper_orig):
def execute(self, sql, params=None):
try:
return super(CursorDebugWrapperLoud, self).execute(sql, params)
finally:
print_stack_in_project()
print sql
print '\n\n\n'
def executemany(self, sql, param_list):
try:
return super(CursorDebugWrapperLoud, self).executemany(sql, param_list)
finally:
print_stack_in_project()
print sql
print '\n\n\n'
bakutils.CursorDebugWrapper = CursorDebugWrapperLoud
Still not sure if there is a more elegant way of doing this?
Django debug toolbar will tell you what you want with spectacular awesomeness.

Twisted IRC Bot connection lost repeatedly to localhost

I am trying to implement an IRC Bot on a local server. The bot that I am using is identical to the one found at Eric Florenzano's Blog. This is the simplified code (which should run)
import sys
import re
from twisted.internet import reactor
from twisted.words.protocols import irc
from twisted.internet import protocol
class MomBot(irc.IRCClient):
def _get_nickname(self):
return self.factory.nickname
nickname = property(_get_nickname)
def signedOn(self):
print "attempting to sign on"
self.join(self.factory.channel)
print "Signed on as %s." % (self.nickname,)
def joined(self, channel):
print "attempting to join"
print "Joined %s." % (channel,)
def privmsg(self, user, channel, msg):
if not user:
return
if self.nickname in msg:
msg = re.compile(self.nickname + "[:,]* ?", re.I).sub('', msg)
prefix = "%s: " % (user.split('!', 1)[0], )
else:
prefix = ''
self.msg(self.factory.channel, prefix + "hello there")
class MomBotFactory(protocol.ClientFactory):
protocol = MomBot
def __init__(self, channel, nickname='YourMomDotCom', chain_length=3,
chattiness=1.0, max_words=10000):
self.channel = channel
self.nickname = nickname
self.chain_length = chain_length
self.chattiness = chattiness
self.max_words = max_words
def startedConnecting(self, connector):
print "started connecting on {0}:{1}"
.format(str(connector.host),str(connector.port))
def clientConnectionLost(self, connector, reason):
print "Lost connection (%s), reconnecting." % (reason,)
connector.connect()
def clientConnectionFailed(self, connector, reason):
print "Could not connect: %s" % (reason,)
if __name__ == "__main__":
chan = sys.argv[1]
reactor.connectTCP("localhost", 6667, MomBotFactory('#' + chan,
'YourMomDotCom', 2, chattiness=0.05))
reactor.run()
I added the startedConnection method in the client factory, which it is reaching and printing out the proper address:host. It then disconnects and enters the clientConnectionLost and prints the error:
Lost connection ([Failure instance: Traceback (failure with no frames):
<class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.
]), reconnecting.
If working properly it should log into the appropriate channel, specified as the first arg in the command (e.g. python module2.py botwar. would be channel #botwar.). It should respond with "hello there" if any one in the channel sends anything.
I have NGIRC running on the server, and it works if I connect from mIRC or any other IRC client.
I am unable to find a resolution as to why it is continually disconnecting. Any help on why would be greatly appreciated. Thank you!
One thing you may want to do is make sure you will see any error output produced by the server when your bot connects to it. My hunch is that the problem has something to do with authentication, or perhaps an unexpected difference in how ngirc handles one of the login/authentication commands used by IRCClient.
One approach that almost always applies is to capture a traffic log. Use a tool like tcpdump or wireshark.
Another approach you can try is to enable logging inside the Twisted application itself. Use twisted.protocols.policies.TrafficLoggingFactory for this:
from twisted.protocols.policies import TrafficLoggingFactory
appFactory = MomBotFactory(...)
logFactory = TrafficLoggingFactory(appFactory, "irc-")
reactor.connectTCP(..., logFactory)
This will log output to files starting with "irc-" (a different file for each connection).
You can also hook directly into your protocol implementation, at any one of several levels. For example, to display any bytes received at all:
class MomBot(irc.IRCClient):
def dataReceived(self, bytes):
print "Got", repr(bytes)
# Make sure to up-call - otherwise all of the IRC logic is disabled!
return irc.IRCClient.dataReceived(self, bytes)
With one of those approaches in place, hopefully you'll see something like:
:irc.example.net 451 * :Connection not registered
which I think means... you need to authenticate? Even if you see something else, hopefully this will help you narrow in more closely on the precise cause of the connection being closed.
Also, you can use tcpdump or wireshark to capture the traffic log between ngirc and one of the working IRC clients (eg mIRC) and then compare the two logs. Whatever different commands mIRC is sending should make it clear what changes you need to make to your bot.