My application uses Kafka and Hibernate. For Kafka a running docker image is required. If I run a Quarkus test for Hibernate, the test fails if Kafka is not running. In my IDE this is not a problem, but in Jenkins there is no Kafka server available and the test fails because it cannot resolve the Kafka server.
Is it possible to disable Kafka in Quarkus tests?
You could make use of Microprofile's Emitter for sending messages to Kafka channel:
#Inject
#Channel("hello")
Emitter<String> emitter;
By default, in case when there is no Kafka behind that emitter, it will create an in-memory message bus. So the docker image for Kafka would not be required.
Another solution would be to use KafkaContainer from TestContainers to create a throwaway Kafka container for each test run.
You could check both examples in Alex Soto's repository.
Look at CheckoutProcess class and corresponding component test and integration test.
Related
I intend to write integration tests with Cucumber for a GemFire cache client application using Spring Boot and deployed in an Apache Geode client/server topology. I referred to the question - How to start Spring Boot app without depending on Pivotal GemFire cache which was answered in 2018 and also referred to the integration test documentation here - Integration Testing with STDG.
The link to an example concrete client/server Integration Test extending STDG’s ForkingClientServerIntegrationTestsSupprt class appears to be broken.
The purpose of my integration tests would be to:
run an embedded locator and a server during the integration test phase
define the regions for the servers using cluster.xml
create, read, update and delete cache entries and verify the different use cases
Any help regarding the ideal approach to write integration tests (probably using an embedded GemFire locator and server) will be very helpful.
Tried an embedded GemFire CacheServer instance for integration tests using #CacheServerApplication annotation but not sure on how to create ClientCache objects to use the embedded GemFire or whether this is the right way to write the integration tests.
Edit: Also came across this - Is it possible to start a PIvotal GemFire Server, Locator and Client in one JVM? where it is mentioned as - In short, NO, you cannot have a peer Cache instance (with embedded Locator) and a ClientCache instance in the same JVM (or Java application process).
DISCLAIMER: I do not have experience with Apache Cucumber...
However, it is not difficult to spin up multiple GemFire or Geode server-side processes, such as 1 or more Locator and [multiple] CacheServers in a single test class. The Locators can be standalone JVM processes or embedded, as part of the servers.
In this typical test configuration arrangement the GemFire or Geode server-side processes are forked, yet coordinated, and the test class itself acts as the ClientCache instance.
You can see 1 such test configuration in the SBDG Multi-site Caching sample, here.
The key to this test configuration is the extension of the ForkingClientServerIntegrationTests class from STDG, as well as the forking of the 2 clusters (and specifically), in the test class setup method.
The configuration for each cluster is handled by Spring config and the coordination is all handled using GemFire/Geode properties (specifically) combined with some Spring Profiles (for example, then see here) to control which configuration gets applied for each GemFire/Geode JVM process.
Of course, this example and test configuration is quite complex given the fact that the test also employs GemFire/Geode's WAN capabilities, hence the "multi-site" caching reference, but serves to demonstrate that Spring and SBDG/SDG/STDG supports as complex or as simple of a setup as your testing needs require.
You can start any number of GemFire/Geode processes (Locators, CacheServers, etc). And, in nearly all cases, the test class (JVM) itself is the cache client (ClientCache instance).
Here are a couple more examples from the Spring Data for Apache Geode (SDG) codebase and test suite: here and here.
I am certain I have another test class or example (somewhere) that for a single Locator, then joined 2 CacheServer instances, and then the test (JVM process) proceeded as ClientCache instance, but I cannot seem to find it at the moment.
In any case, I hope this gives you some ideas.
In MassTransit documentation, I saw an example of testing a state machine saga with a bus, but there was no example of doing it with a Kafka rider. Whether you do it the same way, or should it be done differently?
There are no test harnesses for riders, only for the supported transports.
You can look at the state machine unit tests for Kafka in the unit tests project.
All,
I'm running a simple SpringBoot app in PCF using a Rabbit on-demand service. The auto reconfiguration of the ConnectionFactory for the internal Rabbit service works just fine.
However I need a list of all queues on the Rabbit host. AFAIK this is only available through a call to the Rabbit management plugin (a REST API), see RabbitManagementTemplate::getQueues. This class expects an http URI with credentials.
I know the URI+credentials are exposed through the vcap.service variables as "http_api_uri', but I wonder if there's a more elegant way to get an instance of RabbitManagentTemplate with Spring magic cloud connectors / auto reconfiguration instead of manually reading the env vars and writing custom bean config.
It seems the ConnectionFactory only knows about the AMQP interface, and cannot create a RabbitManagementTemplate?
Thanks!
Spring Cloud Connectors won't help you here. It doesn't support setting up RabbitManagementTemplate, only a ConnectionFactory.
You don't have to parse the env yourself, you can use the flattened properties that Boot provides such as vcap.services.rabbitmq.credentials.http_api_uri. But you'll need to configure a RabbitManagementTemplate yourself using those Boot properties.
For a large online application, use k8s to run it. The scale maybe daily activity user 500,000.
The application inside k8s need messaging feature - Pub/Sub, there are these options:
Kafka
RabbitMQ
Redis
Kafka
It needs zookeeper and good to run on os depends on disk I/O. So if install it into k8s cluster, how? The performance will be worse?
And, if keep Kafka outside of the k8s cluster, connect Kafka from application inside the k8s cluster, how about that performance? They are in the different layer, won't be slow?
RabbitMQ
It's slow than Kafka, but for a daily activity user 500,000 application, is it good enough? If so, maybe it's a good choice.
Redis
It's another option. Maybe the most simple one. But from the internet I got that it will lose message sometimes. If true, that's terrible.
So, the most important thing is, use Kafka(also with zookeeper) on k8s, good or not in this use case?
Yes, running Kafka on Kubernetes is great. Check out this example: https://github.com/Yolean/kubernetes-kafka. It includes ZooKeeper and Kafka as StatefulSets.
PS. Running any of the services in your question on Kubernetes will be pleasant. You can Google the name of the service and "kubernetes" and find example manifests. Many examples here: https://github.com/kubernetes/charts.
For Kafka, you can find some suggestion here. Kubernetes 1.7+ supports local persistent volume, which may be good for Kafka deployment.
You can also take a look to the following project :
https://github.com/EnMasseProject/barnabas
It's about running Kafka on Kubernetes and OpenShift as well. It provides deploying with StatefulSets with persistent volumes or just in memory (for developing or just testing purpose). It provides deploying for Kafka Connect and Prometheus metrics as well.
Another simple configuration of Kafka/Zookeeper on Kubernetes in DigitalOcean with external access:
https://github.com/StanislavKo/k8s_digitalocean_kafka
You can connect to Kafka from outside of AWS/DO/GCE by regular binary protocol. Connection is PLAINTEXT or SASL_PLAINTEXT (user/password).
Kafka cluster is StatefulSet, so you can scale cluster easily.
I have a Java API on my server, and I want it to create tasks and add them to Celery via RabbitMQ. I followed the following tutorial, http://www.rabbitmq.com/tutorials/tutorial-two-python.html, where I used java for the client (send.java) and python to receive (receive.py). In receive.py, where the callback method is invoked, I call a method that I've annotated with #celery.task so that the task is added to celery.
I'm wondering how all of this is deployed on a server though, specifically, why there is a receive.py file. Is receive.py a process that must continually run on the server? Is there a way to configure RabbitMQ so that it automatically routes java client tasks to celery?
Thanks!
RabbitMQ is just a message queue. Producers put messages and consumers get them on demand. You can only restrict access for specific queues via RabbitMQ's auth options.
As for deployment: yes, receive.py needs to continuously run. It is Celery's job to do that. See the Workers Guide for info on running a worker.