Writing a unit test for a twisted application. Trying to perform some asserts once deferred is resolved with a new(dummy) connection (instance of proto_helpers.StringTransport), however callback assert_cache_updated_on_connection is receiving None instead of the connection that was passed to <deferred>.callback(connection)
def test_send_to_new_connection(self):
# Given
peerAddr = ('10.22.22.190', 5060)
# If
self.tcp_transport.send_to('test', peerAddr)
# Then
assert peerAddr in self.tcp_transport._connections
assert True == isinstance(self.tcp_transport._connections[peerAddr], Deferred)
connection = _string_transport_connection(self.hostAddr, peerAddr, None, self.tcp_transport.connectionMade)
def assert_cache_updated_on_connection(connection):
print('--------- SUCCESS ----------')
peer = connection.transport.getPeer()
peerAddr = (peer.host, peer.port)
assert peerAddr in self.tcp_transport._connections
assert True == isinstance(self.tcp_transport._connections[peerAddr], Protocol)
def assert_fail(fail):
print('--------- FAIL ----------')
self.tcp_transport._connections[peerAddr].addCallback(assert_cache_updated_on_connection)
self.tcp_transport._connections[peerAddr].addErrback(assert_fail)
# Forcing deferred to fire with mock connection
self.tcp_transport._connections[peerAddr].callback(connection)
Is this the right way to manually fire deferred callback?
A callback receives as its first argument one of two things:
If it is the first callback, it receives the value passed to the callback method.
If it is a subsequent callback, it receives the return value of the callback before it.
So presumably the explanation is that assert_cache_updated_on_connection is not the first callback and that the callback before it is returning None.
Related
I'm currently working with:
Micronaut 3.7.3
RabbitMQ 3.11.2
Spock
Groovy / Java 17
I'm implementing a rabbitmq consumer for a simple demo project following the guidelines from micronaut project (https://micronaut-projects.github.io/micronaut-rabbitmq/3.1.0/guide/index.html)
I'm trying to mock a service that is a dependency of my rabbitmq consumer.
I've tried this approach that does not seem to work:
#MicronautTest
#Subject(SampleRequestConsumer)
class SampleRequestConsumerSpec extends Specification {
#Inject
ExternalWorkflowProducer externalWorkflowProducer
#Inject
SampleRequestConsumer sampleRequestConsumer
#Inject
SimpleService simpleService
#MockBean(SimpleService)
SimpleService simpleService() {
Mock(SimpleService)
}
def "It receives a sampleRequest message in the simple.request queue"() {
when:
externalWorkflowProducer.send(new SampleRequest(message: "Request1"))
then:
sleep(100)
1 * simpleService.handleSimpleRequest(_ as SampleRequest) >> { SampleRequest request ->
assert request.message != null
}
}
}
I get this error when running the integration test:
Too few invocations for:
1 * simpleService.handleSimpleRequest(_ as SampleRequest) >> { SampleRequest request ->
assert request.message != null
} (0 invocations)
Unmatched invocations (ordered by similarity):
None
See full source code on GitHub: https://github.com/art-dambrine/micronaut-rabbitmq-spock-mocking/blob/test-with-mq/src/test/groovy/com/company/microproject/amqp/consumer/SampleRequestConsumerSpec.groovy
Also notice that when I'm not reading from the queue and directly calling the method sampleRequestConsumer.receive([message: "Request1"])
mocking for the simpleService is working : https://github.com/art-dambrine/micronaut-rabbitmq-spock-mocking/blob/test-without-mq/src/test/groovy/com/company/microproject/amqp/consumer/SampleRequestConsumerSpec.groovy
Thanks in advance for your insight
IMPORTANT
Please use the branch test-with-mq. The branch test-without-mq's tests will succeed because it's not using rabbitMQ. This is an attempt to demonstrate that the issue lies in testing RabbitMQ consumers.
Moving the sleep() instruction to the when: block fixed the test.
Indeed, what is behind the then: block is not executed after the externalWorkflowProducer.send(), it is executed before by Spock.
This instruction:
1 * simpleService.handleSimpleRequest(_ as SampleRequest)
is creating an interaction in the scope of the feature method, and it is executed when the specification is configured.
Adding a sleep() instruction there, is not leaving the spec time for the consumer to receive the message. You need to add the sleep() after the send(). This is when your test needs to let the consumer time to execute.
Note: the closure itself:
{ SampleRequest request ->
assert request.message != null
}
would be executed afterwards, but only the closure. The sleep instruction was already executed when configuring the Spec. The Closure is not executed in this case, because the test finishes before the thread of the consumer can invoke the mock.
In summary:
This works:
def "It receives a sampleRequest message in the simple.request queue"() {
when:
externalWorkflowProducer.send(new SampleRequest(message: "Request1"))
sleep(100)
then:
1 * simpleService.handleSimpleRequest(_ as SampleRequest) >> { SampleRequest request ->
assert request.message != null
}
}
And this doesn't work:
def "It receives a sampleRequest message in the simple.request queue"() {
when:
externalWorkflowProducer.send(new SampleRequest(message: "Request1"))
then:
sleep(100)
1 * simpleService.handleSimpleRequest(_ as SampleRequest) >> { SampleRequest request ->
assert request.message != null
}
}
As #LuisMuñiz pointed out, the interactions declared in the then block are actually moved around. It creates an interaction scope that contains all the interactions, the setup of that happens immediately before the when block executes and the verification that all interactions had taken place happens before any other instruction in the then block.
That out of the way, I would advise against using any kind of sleeps for your code. At best you are just waiting uselessly, at worst you didn't wait long enough and you test breaks. It is preferable to use one or more CountDownLatch instances to synchronize your test.
def "It receives a sampleRequest message in the simple.request queue"() {
given:
def latch = new CountDownLatch(1)
when:
externalWorkflowProducer.send(new SampleRequest(message: "Request1"))
latch.await()
then:
1 * simpleService.handleSimpleRequest(_ as SampleRequest) >> { SampleRequest request ->
assert request.message != null
latch.countDown()
}
}
This way you test will wait until you mock was called, but then immediately finish.
You can also use latch.await(long timeout, TimeUnit unit) with a generous timeout, to guard against your test hanging indefinitely.
I'm actually confused on assembly time and subscription time. I know mono's are lazy and does not get executed until its subscribed. Below is a method.
public Mono<UserbaseEntityResponse> getUserbaseDetailsForEntityId(String id) {
GroupRequest request = ImmutableGroupRequest
.builder()
.cloudId(id)
.build();
Mono<List<GroupResponse>> response = ussClient.getGroups(request);
List<UserbaseEntityResponse.GroupPrincipal> groups = new CopyOnWriteArrayList<>();
response.flatMapIterable(elem -> elem)
.toIterable().iterator().forEachRemaining(
groupResponse -> {
groupResponse.getResources().iterator().forEachRemaining(
resource -> {
groups.add(ImmutableGroupPrincipal
.builder()
.groupId(resource.getId())
.name(resource.getDisplayName())
.addAllUsers(convertMemebersToUsers(resource))
.build());
}
);
}
);
log.debug("Response Object - " + groups.toString());
ImmutableUserbaseEntityResponse res = ImmutableUserbaseEntityResponse
.builder()
.userbaseId(id)
.addAllGroups(groups)
.build();
Flux<UserbaseEntityResponse.GroupPrincipal> f = Flux.fromIterable(res.getGroups())
.parallel()
.runOn(Schedulers.parallel())
.doOnNext(groupPrincipal -> getResourcesForGroup((ImmutableGroupPrincipal)groupPrincipal, res.getUserbaseId()))
.sequential();
return Mono.just(res);
}
This gets executed Mono<List<GroupResponse>> response = ussClient.getGroups(request); without calling subscribe, however below will not get executed unless I call subscribe on that.
Flux<UserbaseEntityResponse.GroupPrincipal> f = Flux.fromIterable(res.getGroups())
.parallel()
.runOn(Schedulers.parallel())
.doOnNext(groupPrincipal -> getResourcesForGroup((ImmutableGroupPrincipal)groupPrincipal, res.getUserbaseId()))
.sequential();
Can I get some more input on assembly time vs subscription?
"Nothing happens until you subscribe" isn't quite true in all cases. There's three scenarios in which a publisher (Mono or Flux) will be executed:
You subscribe;
You block;
The publisher is "hot".
Note that the above scenarios all apply to an entire reactive chain - i.e. if I subscribe to a publisher, everything upstream (dependent on that publisher) also executes. That's why frameworks can, and should call subscribe when they need to, causing the reactive chain defined in a controller to execute.
In your case it's actually the second of these - you're blocking, which is essentially a "subscribe and wait for the result(s)". Usually the methods that block are clearly labelled, but again that's not always the case - in your case it's the toIterable() method on Flux doing the blocking:
Transform this Flux into a lazy Iterable blocking on Iterator.next() calls.
But ah, you say, I'm not calling Iterator.next() - what gives?!
Well, implicitly you are by calling forEachRemaining():
The default implementation behaves as if:
while (hasNext())
action.accept(next());
...and as per the above rule, since ussClient.getGroups(request) is upstream of this blocking call, it gets executed.
I have a callback function in my program that I need to check success/failure. How can I do this?
In my example below, where does failure_code go?
My snippet:
def mq_callback(job_id, ch, method, body):
# Do some stuff. But if the stuff fails...
return failure_code
channel.basic_consume(
queue='some queue',
on_message_callback=lambda ch, method, properties, body: mq_callback(job_id, ch, method, body),
auto_ack=False
)
channel.start_consuming()
All in this link: https://www.rabbitmq.com/tutorials/tutorial-six-python.html
We create a class for calling requests then wait until the response id equal to the correlation_id that was sent before.
I´m using OpenTracing and I am trying to propagate a span through RabbitMQ. However I don't understand how I am supposed to inject the span and how to extract it later.
This is the code for sending a message
def send_message(self, message, tracer):
root_span = tracer.get_span()
with opentracing.tracer.start_span('Sending message to broker', child_of=root_span) as span:
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='default')
json_message = json.dumps(message)
channel.basic_publish(exchange='',
routing_key='default',
body=json_message)
connection.close()`
And I have a callback function for receiving the message
def _callback(self, ch, method, properties, body):
print(" [x] Received %r" % body)
So somewhere and somehow I want to inject the span and then extract it. Does anyone know or have any examples on how to do it?
I have tried to inject before calling basic_publish in sender like this
tracer.inject(span, Format.HTTP_HEADERS, headers)
But I have no idea which arguments are going to the inject method.
Then I tried to extract it like this in the callback
span_ctx = tracer.extract(Format.HTTP_HEADERS, {})
Again, I don't know which arguments are going into the extract method.
EDIT: solved, kinda
I solved it by sending the carrier into properties header. Then I could extract the span from the callback properties attribute
In the sender:
channel.basic_publish(exchange='',
routing_key='default',
properties=pika.BasicProperties(headers=carrier),
body=json_message)
In the callback, extract span:
def _callback(self, ch, method, properties, body):
span_ctx = tracer.extract(Format.TEXT_MAP, properties.headers)
I solved it by sending the carrier into properties header. Then I could extract the span from the callback properties attribute
In the sender:
channel.basic_publish(exchange='',
routing_key='default',
properties=pika.BasicProperties(headers=carrier),
body=json_message)
In the callback, extract span:
def _callback(self, ch, method, properties, body):
span_ctx = tracer.extract(Format.TEXT_MAP, properties.headers)
I have a class which has BaseScheduler as an attribute, nothing fancy, no frameworks etc.
class MyClass(object):
def __init__(self, ...):
...
self.Scheduler = BackgroundScheduler()
...
Later on I define methods that a) schedule jobs based on a schedule definition passed as kwargs, and b) handle the jobs when they are triggered:
def _schedule_Events(self, *args, **kwargs):
try:
Schedule_Def = kwargs.copy()
Schedule_Def['func'] = self._handle_scheduled_Event
job = self.Scheduler.add_job(**Schedule_Def)
self.Scheduled_Events.append(job)
except Exception as e:
self.Logger.exception('%s: exception in schedule_Events, details: %s' %(self.Name, e))
def _handle_scheduled_Event(self, Event_Action):
""" Callback function for scheduled jobs """
try:
.... do stuff ....
However, adding jobs with _schedule_Events fails with:
File "/usr/local/lib/python3.4/dist-packages/apscheduler/util.py", line 381, in check_callable_args
', '.join(unsatisfied_args))
ValueError: The following arguments have not been supplied: Event_Action
The reason is apparently that the 'func' argument must be globally callable, ie. not within a class instance scope. I also don't see how using a 'textual reference' as described in the documentation will help.
If I replace the 'func' callable with a function defined at the module level then it works, but I need to make it call a method within my instance object. Any ideas how to make this work ? Custom trigger ? Wrapping APS Scheduler inside another class and pass the callback ? Other ideas ?
Many thanks in advance.