It seems that only one http client gets the stream of data, while the others do not.
Is it true that the Publisher is hot data, and that it should broadcast to all subscribers?
Please find more in Can I allow multiple http clients to consume a Flowable stream of data with resteasy-rxjava2 / quarkus?
package org.acme.kafka;
import javax.inject.Inject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import io.reactivex.Flowable;
import io.reactivex.Observable;
import org.jboss.resteasy.annotations.SseElementType;
import org.reactivestreams.Publisher;
import io.smallrye.reactive.messaging.annotations.Channel;
import java.util.Map;
import java.util.Objects;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import static io.reactivex.Flowable.fromIterable;
/**
* A simple resource retrieving the "in-memory" "my-data-stream" and sending the items to a server sent event.
*/
#Path("/migrations")
public class StreamingResource {
private volatile Map<String, String> counterBySystemDate = new ConcurrentHashMap<>();
#Inject
#Channel("migrations")
Flowable<String> counters;
#GET
#Path("/stream")
#Produces(MediaType.SERVER_SENT_EVENTS) // denotes that server side events (SSE) will be produced
#SseElementType("text/plain") // denotes that the contained data, within this SSE, is just regular text/plain data
public Publisher<String> stream() {
Flowable<String> mainStream = counters.doOnNext(dateSystemToCount -> {
String key = dateSystemToCount.substring(0, dateSystemToCount.lastIndexOf("_"));
counterBySystemDate.put(key, dateSystemToCount);
});
return fromIterable(counterBySystemDate.values().stream().sorted().collect(Collectors.toList()))
.concatWith(mainStream)
.onBackpressureLatest();
}
}
You may use Replay operator or ConnectableObservable
What I did then was to inject the messages coming on the Flowable into a PublishSubject pipe and apply a backpressure strategy, thus offering a broadcast downstream.
Related
The documentation from Micronaut is limited here.
We're using io.micronaut.http.client.HttpClient to access a 3rd party API.
All I can find is this https://docs.micronaut.io/latest/guide/configurationreference.html but it doesn't give examples.
How do we specify the version of TLS that we are prepared to interact with?
To specify the version of TLS that you are prepared to interact with for io.micronaut.http.client.HttpClient, you can use the sslContext property of the HttpClientConfiguration class.
import io.micronaut.context.annotation.Factory;
import io.micronaut.context.annotation.Value;
import io.micronaut.http.client.HttpClientConfiguration;
import io.micronaut.http.client.HttpClientFactory;
import io.micronaut.http.client.annotation.Client;
import javax.inject.Singleton;
import javax.net.ssl.SSLContext;
import java.security.NoSuchAlgorithmException;
#Factory
public class MyHttpClientFactory {
#Value("${my.http.client.url}")
String url;
#Singleton
#Client("myClient")
HttpClientConfiguration myClientConfig(HttpClientFactory factory) throws NoSuchAlgorithmException {
HttpClientConfiguration clientConfig = factory.createConfiguration(url);
SSLContext sslContext = SSLContext.getInstance("TLSv1.2");
sslContext.init(null, null, null);
clientConfig.setSslContext(sslContext);
return clientConfig;
}
}
In this code, we create a factory class named MyHttpClientFactory that defines a myClientConfig method. This method creates a new HttpClientConfiguration instance and sets its sslContext property to use TLS version 1.2.
You can then use this factory class to create instances of io.micronaut.http.client.HttpClient that are prepared to interact with servers using TLS version 1.2.
I have an Apache Apex application DAG which reads RabbitMQ message from a queue. Which Apache Apex Malhar operator should I use? There are several operators but it's not clear which one to use and how to use it.
Have you looked at https://github.com/apache/apex-malhar/tree/master/contrib/src/main/java/com/datatorrent/contrib/rabbitmq ? There are also tests in https://github.com/apache/apex-malhar/tree/master/contrib/src/test/java/com/datatorrent/contrib/rabbitmq that show how to use the operator
https://github.com/apache/apex-malhar/blob/master/contrib/src/main/java/com/datatorrent/contrib/rabbitmq/AbstractRabbitMQInputOperator.java
That is the main operator code where the tuple type is a generic parameter and emitTuple() is an abstract method that subclasses need to implement.
AbstractSinglePortRabbitMQInputOperator is a simple subclass that provides a single output port and implements emitTuple() using another abstract method getTuple() which needs an implementation in its subclasses.
The tests that Sanjay pointed to show how to use these classes.
I also had problems finding out how to read messages from RabbitMQ to Apache Apex. With the help of the provided links of Sanjay's answer (https://stackoverflow.com/a/42210636/2350644) I finally managed to get it running. Here's how it works all together:
1. Setup a RabbitMQ Server
There are lot of ways installing RabbitMQ that are described here: https://www.rabbitmq.com/download.html
The simplest way for me was using docker (See: https://store.docker.com/images/rabbitmq)
docker pull rabbitmq
docker run -d --hostname my-rabbit --name some-rabbit -p 5672:5672 -p 15672:15672 rabbitmq:3-management
To check if RabbitMQ is working, open a browser and navigate to: http://localhost:15672/. You should see the Management page of RabbitMQ.
2. Write a Producer program
To send messages to the queue you can write a simple JAVA program like this:
import com.rabbitmq.client.BuiltinExchangeType;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import java.util.ArrayList;
public class Send {
private final static String EXCHANGE = "myExchange";
public static void main(String[] args) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.exchangeDeclare(EXCHANGE, BuiltinExchangeType.FANOUT);
String queueName = channel.queueDeclare().getQueue();
channel.queueBind(queueName, EXCHANGE, "");
List<String> messages = Arrays.asList("Hello", "World", "!");
for (String msg : messages) {
channel.basicPublish(EXCHANGE, "", null, msg.getBytes("UTF-8"));
System.out.println(" [x] Sent '" + msg + "'");
}
channel.close();
connection.close();
}
}
If you execute the JAVA program you should see some outputs in the Management UI of RabbitMQ.
3. Implement a sample Apex Application
3.1 Bootstrap a sample apex application
Follow the official apex documentation http://docs.datatorrent.com/beginner/
3.2 Add additional dependencies to pom.xml
To use the classes provided by malhar add the following dependencies:
<dependency>
<groupId>org.apache.apex</groupId>
<artifactId>malhar-contrib</artifactId>
<version>3.7.0</version>
</dependency>
<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
<version>4.2.0</version>
</dependency>
3.3 Create a Consumer
We first need to create an InputOperator that consumes messages from RabbitMQ using available code from apex-malhar.
import com.datatorrent.contrib.rabbitmq.AbstractSinglePortRabbitMQInputOperator;
public class MyRabbitMQInputOperator extends AbstractSinglePortRabbitMQInputOperator<String> {
#Override
public String getTuple(byte[] message) {
return new String(message);
}
}
You only have to override the getTuple() method. In this case we simply return the message that was received from RabbitMQ.
3.4 Setup an Apex DAG
To test the application we simply add an InputOperator (MyRabbitMQInputOperator that we implemented before) that consumes data from RabbitMQ and a ConsoleOutputOperator that prints the received messages.
import com.rabbitmq.client.BuiltinExchangeType;
import org.apache.hadoop.conf.Configuration;
import com.datatorrent.api.annotation.ApplicationAnnotation;
import com.datatorrent.api.StreamingApplication;
import com.datatorrent.api.DAG;
import com.datatorrent.api.DAG.Locality;
import com.datatorrent.lib.io.ConsoleOutputOperator;
#ApplicationAnnotation(name="MyFirstApplication")
public class Application implements StreamingApplication
{
private final static String EXCHANGE = "myExchange";
#Override
public void populateDAG(DAG dag, Configuration conf)
{
MyRabbitMQInputOperator consumer = dag.addOperator("Consumer", new MyRabbitMQInputOperator());
consumer.setHost("localhost");
consumer.setExchange(EXCHANGE);
consumer.setExchangeType(BuiltinExchangeType.FANOUT.getType());
ConsoleOutputOperator cons = dag.addOperator("console", new ConsoleOutputOperator());
dag.addStream("myStream", consumer.outputPort, cons.input).setLocality(Locality.CONTAINER_LOCAL);
}
}
3.5 Test the Application
To simply test the created application we can write a UnitTest, so there is no need to setup a Hadoop/YARN cluster.
In the bootstrap application there is already a UnitTest namely ApplicationTest.java that we can use:
import java.io.IOException;
import javax.validation.ConstraintViolationException;
import org.junit.Assert;
import org.apache.hadoop.conf.Configuration;
import org.junit.Test;
import com.datatorrent.api.LocalMode;
/**
* Test the DAG declaration in local mode.
*/
public class ApplicationTest {
#Test
public void testApplication() throws IOException, Exception {
try {
LocalMode lma = LocalMode.newInstance();
Configuration conf = new Configuration(true);
//conf.addResource(this.getClass().getResourceAsStream("/META-INF/properties.xml"));
lma.prepareDAG(new Application(), conf);
LocalMode.Controller lc = lma.getController();
lc.run(10000); // runs for 10 seconds and quits
} catch (ConstraintViolationException e) {
Assert.fail("constraint violations: " + e.getConstraintViolations());
}
}
}
Since we don't need any properties for this application the only thing changed in this file is uncommenting the line:
conf.addResource(this.getClass().getResourceAsStream("/META-INF/properties.xml"));
If you execute the ApplicationTest.java and send messages to RabbitMQ using the Producer program as described in 2., the Test should output all the messages.
You might need to increase the time of the test to see all messages (It is set to 10sec currently).
i'm throwing a jax rs BadRequestException("message") and i always get the generic message back and not the one i put in there. why is that?
i'm using netbeans and postman chrome extension to test and see the response.
for the BadRequestException("message") to display when you request, you need an extra class that implements ExceptionMapper. learned this after i had learned to create my own custom exception classes. i must of skipped over the part they explained this in my book. anyway here is how it looks like.
import javax.ws.rs.BadRequestException;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;
import javax.ws.rs.ext.ExceptionMapper;
import javax.ws.rs.ext.Provider;
#Provider
public class BadRequestExceptionMapper implements ExceptionMapper < BadRequestException > {
#Override
public Response toResponse(BadRequestException exception) {
return Response.status(Status.BAD_REQUEST).type(MediaType.TEXT_PLAIN).entity(exception.getMessage()).build();
}
}
We had been using GWT-Dispatch to support the RPC calls using command patterns. We now need to move to GWTP since Dispatch has been absorbed into that project. Would seem to be all well and good. The problem is that we are unable to get a DispatchAsync object anymore. In Dispatch, it was extremely simple to get the default implementation:
private final DispatchAsync dispatchAsync = GWT.create(DefaultDispatchAsync.class);
This no longer works. DefaultDispatchAsync is deprecated, and when we use the suggested replacement for it (RpcDispatchAsync) it looks like this:
private final DispatchAsync dispatchAsync = GWT.create(RpcDispatchAsync.class);
we get the following error:
Rebind result 'com.gwtplatform.dispatch.rpc.client.RpcDispatchAsync' has no default (zero argument) constructors.
Does anyone have an idea about how to do this? I know if we rewrite all the pages to use the GWTP MVP pattern that it's available in the Presenter but moving things over to use full GWTP is a long process and if we can't get the RPC calls up and working quickly that will be a problem for the project.
Thanks in advance -- hopefully it's something easy.
DispatchAsync is no longer generated via deferred binding. Thus you can’t use GWT.create to instantiate it.
GWTP Dispatch is making heavy use of GIN/Guice. So I would recommend that you use this dependency injection framework to get GWTP Dispatch work.
Here is an example, which provides easy access to the DispatchAsync (without the need of rewriting all pages to use the GWTP MVP Pattern):
[Note: This example uses gwtp dispatch 1.0.3]
Client:
MyClientModule.java - configure injection-rules for DispatchAsync
import com.google.gwt.inject.client.AbstractGinModule;
import com.google.inject.Inject;
import com.google.inject.Provider;
import com.google.inject.Singleton;
import com.gwtplatform.dispatch.client.DefaultExceptionHandler;
import com.gwtplatform.dispatch.client.DefaultSecurityCookieAccessor;
import com.gwtplatform.dispatch.client.ExceptionHandler;
import com.gwtplatform.dispatch.client.RpcDispatchAsync;
import com.gwtplatform.dispatch.client.actionhandler.ClientActionHandlerRegistry;
import com.gwtplatform.dispatch.client.actionhandler.DefaultClientActionHandlerRegistry;
import com.gwtplatform.dispatch.shared.DispatchAsync;
import com.gwtplatform.dispatch.shared.SecurityCookie;
import com.gwtplatform.dispatch.shared.SecurityCookieAccessor;
public class MyClientModule extends AbstractGinModule {
private static final String COOKIE_NAME = "JSESSIONID";
#Override
protected void configure() {
bindConstant().annotatedWith(SecurityCookie.class).to(COOKIE_NAME);
bind(ExceptionHandler.class).to(DefaultExceptionHandler.class);
bind(SecurityCookieAccessor.class).to(DefaultSecurityCookieAccessor.class);
bind(ClientActionHandlerRegistry.class).to(DefaultClientActionHandlerRegistry.class);
bind(DispatchAsync.class).toProvider(DispatchAsyncProvider.class).in(Singleton.class);
}
public static class DispatchAsyncProvider implements Provider<DispatchAsync> {
private final DispatchAsync fDispatchAsync;
#Inject
public DispatchAsyncProvider(ExceptionHandler eh, SecurityCookieAccessor sca, ClientActionHandlerRegistry cahr) {
this.fDispatchAsync = new RpcDispatchAsync(eh, sca, cahr);
}
#Override
public DispatchAsync get() {
return fDispatchAsync;
}
}
}
MyClientInjector.java - injector provides access to DispatchAsync
import com.google.gwt.inject.client.GinModules;
import com.google.gwt.inject.client.Ginjector;
import com.gwtplatform.dispatch.shared.DispatchAsync;
#GinModules(MyClientModule.class)
public interface MyClientInjector extends Ginjector {
DispatchAsync getDispatchAsync();
}
Server:
MyGuiceServletContextListener.java - create injector for the servlet, which receives the commands and the servermodule, in which the bindings between (clientside) command and (serverside) handler are defined.
import com.google.inject.Guice;
import com.google.inject.Injector;
import com.google.inject.servlet.GuiceServletContextListener;
public class MyGuiceServletContextListener extends GuiceServletContextListener {
#Override
protected Injector getInjector() {
return Guice.createInjector(new ServerModule(), new DispatchServletModule());
}
}
DispatchServletModule.java - configures the servlet, which receives the commands
import com.google.inject.servlet.ServletModule;
import com.gwtplatform.dispatch.server.guice.DispatchServiceImpl;
import com.gwtplatform.dispatch.server.guice.HttpSessionSecurityCookieFilter;
import com.gwtplatform.dispatch.shared.Action;
import com.gwtplatform.dispatch.shared.SecurityCookie;
public class DispatchServletModule extends ServletModule {
#Override
public void configureServlets() {
bindConstant().annotatedWith(SecurityCookie.class).to("JSESSIONID");
filter("*").through(HttpSessionSecurityCookieFilter.class);
serve("/" + Action.DEFAULT_SERVICE_NAME + "*").with(DispatchServiceImpl.class);
}
}
ServerModule.java - bindings between (clientside) command and (serverside) handler
import com.gwtplatform.dispatch.server.guice.HandlerModule;
public class ServerModule extends HandlerModule {
#Override
protected void configureHandlers() {
bindHandler(YourCommand.class, YourHandler.class);
}
}
web.xml - tell the web-server to use MyGuiceServletContextListener
<filter>
<filter-name>guiceFilter</filter-name>
<filter-class>com.google.inject.servlet.GuiceFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>guiceFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
<!--
This Guice listener hijacks all further filters and servlets. Extra
filters and servlets have to be configured in your
ServletModule#configureServlets() by calling
serve(String).with(Class<? extends HttpServlet>) and
filter(String).through(Class<? extends Filter)
-->
<listener>
<listener-class>de.gwtpdispatch.server.MyGuiceServletContextListener</listener-class>
</listener>
Usage
Now you can create the injector with deferred binding and get access to the DispatchAsync-instance:
MyClientInjector injector = GWT.create(MyClientInjector.class);
injector.getDispatchAsync().execute(...YourCommand...)
(AND: Don't forget to include the jars of GIN and Guice in your project and add the gin-module to the project's gwt.xml)
I hope this explanation is detailed enough. Happy coding :)
I have exposed an api provided by a jetty server to a front-end application. I want to make sure that only the front-end application (from a certain domain) has access to that api - any other requests should be unauthorised.
What's the best best way of implementing this security feature?
Update: I have set up a CrossOriginFilter - however, I can still access the api via basic GET request from my browser.
Thanks!
Use the IPAccessHandler to setup whitelists and blacklists.
Example: this will allow 127.0.0.* and 192.168.1.* to access everything.
But 192.168.1.132 cannot access /home/* content.
package jetty.demo;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import org.eclipse.jetty.server.handler.IPAccessHandler;
import org.eclipse.jetty.servlet.DefaultServlet;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;
public class IpAccessExample
{
public static void main(String[] args)
{
System.setProperty("org.eclipse.jetty.servlet.LEVEL","DEBUG");
System.setProperty("org.eclipse.jetty.server.handler.LEVEL","DEBUG");
Server server = new Server();
ServerConnector connector = new ServerConnector(server);
connector.setPort(8080);
server.addConnector(connector);
// Setup IPAccessHandler
IPAccessHandler ipaccess = new IPAccessHandler();
ipaccess.addWhite("127.0.0.0-255|/*");
ipaccess.addWhite("192.168.1.1-255|/*");
ipaccess.addBlack("192.168.1.132|/home/*");
server.setHandler(ipaccess);
// Setup the basic application "context" for this application at "/"
// This is also known as the handler tree (in jetty speak)
ServletContextHandler context = new ServletContextHandler(ServletContextHandler.SESSIONS);
context.setContextPath("/");
// make context a subordinate of ipaccess
ipaccess.setHandler(context);
// The filesystem paths we will map
String homePath = System.getProperty("user.home");
String pwdPath = System.getProperty("user.dir");
// Fist, add special pathspec of "/home/" content mapped to the homePath
ServletHolder holderHome = new ServletHolder("static-home", DefaultServlet.class);
holderHome.setInitParameter("resourceBase",homePath);
holderHome.setInitParameter("dirAllowed","true");
holderHome.setInitParameter("pathInfoOnly","true");
context.addServlet(holderHome,"/home/*");
// Lastly, the default servlet for root content
// It is important that this is last.
ServletHolder holderPwd = new ServletHolder("default", DefaultServlet.class);
holderPwd.setInitParameter("resourceBase",pwdPath);
holderPwd.setInitParameter("dirAllowed","true");
context.addServlet(holderPwd,"/");
try
{
server.start();
server.join();
}
catch (Throwable t)
{
t.printStackTrace(System.err);
}
}
}
Or alternatively, write your own Handler to filter based on some other arbitrary rule.
Such as looking for a required request header, something that your specific front-end application provides, but a browser would not.
package jetty.demo;
import java.io.IOException;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.eclipse.jetty.http.HttpStatus;
import org.eclipse.jetty.server.Request;
import org.eclipse.jetty.server.handler.HandlerWrapper;
public class BanBrowserHandler extends HandlerWrapper
{
#Override
public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response) throws IOException, ServletException
{
String xfe = request.getHeader("X-FrontEnd");
if ((xfe == null) || (!xfe.startsWith("MagicApp-")))
{
// not your front-end
response.sendError(HttpStatus.FORBIDDEN_403);
baseRequest.setHandled(true);
return;
}
getHandler().handle(target,baseRequest,request,response);
}
}
The class IPAccessHandler is deprecated. The InetAccessHandler is recommended.
org.eclipse.jetty.server.Server server = ...;
InetAccessHandler ipaccess = new InetAccessHandler();
ipaccess.include(clientIP);
ipaccess.setHandler(server.getHandler());
server.setHandler(ipaccess);