Options for creating dynamic filters (xpath) in a Camel route - dynamic

I've the following static route that is loaded at my server startup. It listens for UDP messages on a port and pushes these messages to the seda queue defined in the route below.
from("mina:udp://hostipaddress:9998?sync=false").wireTap(
"seda:sometag?size=100&blockWhenFull=true&multipleConsumers=true");
Now I can have multiple clients that want to receive/subscribe to these messages. They also want to dynamically select which feeds they need.
Each client send a subscription request (REST) to the server (implemented using Spring-MVC, Jetty, Camel).
As soon as the server receives a request I create a new Camel route that looks like:
from("seda:sometag?multipleConsumers=true")
.routeId(RouteIdCreator.createRouteId(toIP, toPort, "sometag"))
.filter()
.xpath(this.xpathFilter).unmarshal().jaxb("sometag").marshal()
.json().wireTap("mina:udp://client_ip_address:20001?sync=false");
Once this route is deployed it will start to send UDP messages to the client_ip_address: 20001 (as specified in the dynamic route above.)
The client can send different filters to the server.
In case this server receives the new filter it does the following
1. checks if there is a route running (based on client ip and port)
2. If there is route running it stops that route and deletes this route with the older filter
3. It then recreates a new route which differs from the last route only in the xpathfilter.
My issue is that step 2 takes a lot of time (to stop and restart)
Is there is a way to resolve this issue?
Basically I want to change the XPath expression in the route without stops/migrating the route.
PS: I've also posted this on the official Camel mailing list.

You can try to store the xpath filter in a database (basically a simple table with the ip and the filter associated) when you receive a new subscription. Then you can read this filter from the database in the route, and use it as a filter.
from("seda:sometag?multipleConsumers=true")
.routeId(RouteIdCreator.createRouteId(toIP, toPort, "sometag"))
.setHeader("ip").constant(client_ip_adresse)
.filter().xpath(simple("${bean:xpathFilterComponent?methode=find}"))
.unmarshal().jaxb("sometag").marshal()
.json().wireTap("mina:udp://client_ip_address:20001?sync=false");
And your bean should look like
public class XpathFilterCompnent {
public void save(String ip, String filter){
//store a filter for an ip in database, when a subscription is received
}
public void find(#Header("ip") String ip){
String filter = ... //retreive filter from database
return filter;
}
}

Related

Using VRS for concurrent request

We need to use a single instace of VRS to support concurrent request.
We have a requirement where multiple different users should be able to create a route plan for different vehicles and locations same time. However, looking at VRS functionality, I am not able to understand how applications supports it. For demo, when I create a different route using different browser, it always merges first and second request and give one single result.
Just a little more elobration on the question:
We are aiming to convert requests as REST API endpoints which will be invoked by different uses same time for their usecase.
Eg. Request 1: Vehicle 1&2 with 50 locations. VRS can calculate route & give one message with all detailed calculations for request1.
Request 2: Vehicle 3 & 4 with 40 locations. So VRS can calculate route which later we can get as one message with all detailed calculations limited to request 2.
Both requests can be submitted same time & application should considered as separate requests without getting merged.
Is there a way to add request ID or any other paramaters to achive this?
For multi-tenant solving, the SolverManager API is ideal:
public class TimeTableService {
// tenantId is Long, but it can also be String or UUID
private SolverManager<TimeTable, Long> solverManager;
// Returns immediately, call it for every dataset
public void solveBatch(Long tenantId) {
solverManager.solve(tenantId,
// Called once, when solving starts
this::findById,
// Called once, when solving ends
this::save);
}
public TimeTable findById(Long tenantId) {...}
public void save(TimeTable timeTable) {...}
}

How to get the current TraceId and SpanId

This article, https://devblogs.microsoft.com/aspnet/improvements-in-net-core-3-0-for-troubleshooting-and-monitoring-distributed-apps/, tells me that the field TraceId is available as a correlation id, which is great!
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
=> ConnectionId:0HLR1BR0PL1CH
=> RequestPath:/weatherforecastproxy
RequestId:0HLR1BR0PL1CH:00000001,
SpanId:|363a800a-4cf070ad93fe3bd8.,
TraceId:363a800a-4cf070ad93fe3bd8,
ParentId: Executed endpoint 'FrontEndApp.Controllers.WeatherForecastProxyController.Get
(FrontEndApp)'
In fact, I can see that in our log sink this works as advertised: When web application A serves a request and in doing so invokes web application B, both of them write the same TraceId value to the log.
As far as I understand, any ASP.NET Core application that receives an incoming Request-Id header will attach the same header to outgoing requests, but if the header does not exist on the incoming request, an new value will be generated for the outgoing request.
We have been asked to add that value to the response from web application A, but it is (not surprisingly) not available on the incoming request.
I have been looking at the System.Diagnostics.Activity class, but accessing Activity.Current isn't giving me an instance with anything useful - the TraceID is just {} - i.e. empty.
My question is this: How can I access the TraceId value in the context of a web application?
-S
I had the same problem when I tried to add a header with TraceId value.
Doing some tests with ModelValidation, I saw then in this kind of error response the "traceId" value was correct, but I couldn't obtain this value from http context variable in any way.
Then I went to net core source code to see DefaultProblemDetailsFactory implementation and surprise! The "traceId" value is obtained doing this:
var traceId = Activity.Current?.Id ?? httpContext?.TraceIdentifier;
Yes, you can get THE traceId using Activity static variable.
You can get tracid and spanid in dictionary.
using var subject = _tracer.BuildSpan($"Operation").StartActive();
var spanContext = subject.Span.Context;
var dictionary = new Dictionary<string, string>();
_tracer.Inject(spanContext, BuiltinFormats.TextMap, new TextMapInjectAdapter(dictionary));

User destinations in a multi-server environment? (Spring WebSocket and RabbitMQ)

The documentation for Spring WebSockets states:
4.4.13. User Destinations
An application can send messages targeting a specific user, and Spring’s STOMP support recognizes destinations prefixed with "/user/" for this purpose. For example, a client might subscribe to the destination "/user/queue/position-updates". This destination will be handled by the UserDestinationMessageHandler and transformed into a destination unique to the user session, e.g. "/queue/position-updates-user123". This provides the convenience of subscribing to a generically named destination while at the same time ensuring no collisions with other users subscribing to the same destination so that each user can receive unique stock position updates.
Is this supposed to work in a multi-server environment with RabbitMQ as broker?
As far as I can tell, the queue name for a user is generated by appending the simpSessionId. When using the recommended client library stomp.js this results in the first user getting the queue name "/queue/position-updates-user0", the next gets "/queue/position-updates-user1" and so on.
This in turn means the first users to connect to different servers will subscribe to the same queue ("/queue/position-updates-user0").
The only reference to this I can find in the documentation is this:
In a multi-application server scenario a user destination may remain unresolved because the user is connected to a different server. In such cases you can configure a destination to broadcast unresolved messages to so that other servers have a chance to try. This can be done through the userDestinationBroadcast property of the MessageBrokerRegistry in Java config and the user-destination-broadcast attribute of the message-broker element in XML.
But this only makes the it possible to communicate with a user from a different server than the one where the web socket is established.
I feel I'm missing something? Is there anyway to configure Spring to be able to safely use MessagingTemplate.convertAndSendToUser(principal.getName(), destination, payload) in a multi-server environment?
If they need to be authenticated (I assume their credentials are stored in a database) you can always use their database unique user id to subscribe to.
What I do is when a user logs in they are automatically subscribed to two topics an account|system topic for system wide broadcasts and account|<userId> topic for specific broadcasts.
You could try something like notification|<userid> for each person to subscribe to then send messages to that topic and they will receive it.
Since user Ids are unique to each user you shouldn't have an issue within a clustered environment as long as each environment is hitting the same database information.
Here is my send method:
public static boolean send(Object msg, String topic) {
try {
String destination = topic;
String payload = toJson(msg); //jsonfiy the message
Message<byte[]> message = MessageBuilder.withPayload(payload.getBytes("UTF-8")).build();
template.send(destination, message);
return true;
} catch (Exception ex) {
logger.error(CommService.class.getName(), ex);
return false;
}
}
My destinations are preformatted so if i want to send a message to user with id of one the destinations looks something like /topic/account|1.
Ive created a ping pong controller that tests websockets for users who connect to see if their environment allows for websockets. I don't know if this will help you but this does work in my clustered environment.
/**
* Play ping pong between the client and server to see if web sockets work
* #param input the ping pong input
* #return the return data to check for connectivity
* #throws Exception exception
*/
#MessageMapping("/ping")
#SendToUser(value="/queue/pong", broadcast=false) // send only to the session that sent the request
public PingPong ping(PingPong input) throws Exception {
int receivedBytes = input.getData().length;
int pullBytes = input.getPull();
PingPong response = input;
if (pullBytes == 0) {
response.setData(new byte[0]);
} else if (pullBytes != receivedBytes) {
// create random byte array
byte[] data = randomService.nextBytes(pullBytes);
response.setData(data);
}
return response;
}

Apache Camel - Build both from and to endpoints dynamically

I have a camel route which processes a message from a process queue and sends it to upload queue.
from("activemq:queue:process" ).routeId("activemq_processqueue")
.process(exchange -> {
SomeImpl impl = new SomeImpl();
impl.process(exchange);
})
.to(ExchangePattern.InOnly, "activemq:queue:upload");
In impl.process I am populating an Id and destination server path. Now I need to define a new route which consumes messages from upload queue ,and copy a local folder (based on Id generated in previous route) and upload it to destination folder which is an ftp server (this is also populated in previous route)
So how to design a new route where both from and to endpoints are dynamic which would look something like below ?
from("activemq:queue:upload" )
.from("file:basePath/"+{idFromExchangeObject})
.to("ftp:"+{serverIpFromExchangeObject}+"/"+{pathFromExchangeObject});
I think there is a better alternative for your case, taking as granted that you are using a Camel version newer than 2.16.(alternatives for a previous version exist but the are more complicated and don't look elegant - ( e.g consumerTemplate & recipientList).
You can replace the first "dynamic from" with pollEnrich which enriches the message using a polling consumer and simple expression to build the dynamic file endpoint. For the second part, as already mentioned, a dynamic uri .toD will do the job. So your route would look like this:
from("activemq:queue:upload" )
.pollEnrich().simple("file:basePath/${header.idFromExchangeObject})
.aggregationStrategy(new ExampleAggregationStrategy()) // * see explanation
.timeout(2000) // the timeout is optional but recommended
.toD("ftp:${header.serverIpFromExchangeObject}/${header.pathFromExchangeObject}")
See content enricher section "Using dynamic uris"
http://camel.apache.org/content-enricher.html .
You will need an aggregation strategy, to combine the original exchange with the resource exchange in order to make sure that the headers serverIpFromExchangeObject, pathFromExchangeObject will be included in the aggregated exchange after the enrichment. If you don't include the custom strategy then Camel will by default use the body obtained from the resource. Have a look at the ExampleAggregationStrategy example in content-enricher.html to see how this works.
For the .toD() have a look at http://camel.apache.org/how-to-use-a-dynamic-uri-in-to.html
Adding a dynamic to endpoint in Camel (as noted in the comment) can be done with the .toD() which is described on this page on the Camel site.
I don't know of any fromD() equivalent. However, you could add a dynamic route by calling the addRoutes method on the CamelContext. This is described on this page on the Camel site.
Expanding slightly on the example from the Camel site here is something that should get you heading in the right direction.
public void process(Exchange exchange) throws Exception {
String idFromExchangeObject = ...
String serverIpFromExchangeObject = ...
String pathFromExchangeObject = ...
exchange.getContext().addRoutes(new RouteBuilder() {
public void configure() {
from("file:basePath/"+ idFromExchangeObject)
.to("ftp:"+ serverIpFromExchangeObject +"/"+pathFromExchangeObject);
}
});
}
There may be other options in Camel as well since this framework has an amazing number of EIP and capabilities.

WCF routing -- how to correctly add filter table programmatically

I am using the WCF 4 routing service, and need to configure the service programmatically (as opposed to via config). The examples I have seen of doing so, which are rare, create a MessageFilterTable as follows:
var filterTable=new MessageFilterTable<IEnumerable<ServiceEndpoint>>();
But, the generic parameter to that method is supposed to be TFilterData (the type of data you are filtering on)? I have my own custom filter that accepts a string -- can I still create the filter table this way?
If this will work...will the routing infrastructure create client endpoints out of the list I pass in?
I have created a WCF 4 routing service and configured it programmatically. My code is a bit more spaced out than it needs to be (maintainability for others being a concern, hence the comments), but it definitely works. This has two filters: one filters some specific Actions to a given endpoint, and the second sends the remaining actions to a generic endpoint.
// Create the message filter table used for routing messages
MessageFilterTable<IEnumerable<ServiceEndpoint>> filterTable = new MessageFilterTable<IEnumerable<ServiceEndpoint>>();
// If we're processing a subscribe or unsubscribe, send to the subscription endpoint
filterTable.Add(
new ActionMessageFilter(
"http://etcetcetc/ISubscription/Subscribe",
"http://etcetcetc/ISubscription/KeepAlive",
"http://etcetcetc/ISubscription/Unsubscribe"),
new List<ServiceEndpoint>()
{
new ServiceEndpoint(
new ContractDescription("ISubscription", "http://etcetcetc/"),
binding,
new EndpointAddress(String.Format("{0}{1}{2}", TCPPrefix, HostName, SubscriptionSuffix)))
},
HighRoutingPriority);
// Otherwise, send all other packets to the routing endpoint
MatchAllMessageFilter filter = new MatchAllMessageFilter();
filterTable.Add(
filter,
new List<ServiceEndpoint>()
{
new ServiceEndpoint(
new ContractDescription("IRouter", "http://etcetcetc/"),
binding,
new EndpointAddress(String.Format("{0}{1}{2}", TCPPrefix, HostName, RouterSuffix)))
},
LowRoutingPriority);
// Then attach the filter table as part of a RoutingBehaviour to the host
_routingHost.Description.Behaviors.Add(
new RoutingBehavior(new RoutingConfiguration(filterTable, false)));
You can find a good example on MSDN here: How To: Dynamic Update Routing Table
Note how they dont directly create an instance of the MessageFilterTable, but instead use the 'FilterTable' property provided by a new RoutingConfiguration instance.
If you have written a custom filter, then you will add it like this:
rc.FilterTable.Add(new CustomMessageFilter("customStringParameter"), new List<ServiceEndpoint> { physicalServiceEndpoint });
The CustomMessageFilter will be your filter, and the "customStringParameter" is the string that (I believe) you are talking about.
When the Router receives a connection request, it will attempt to map it via this table entry, if this is successful, then you are right, the router will create a client endpoint to talk to the ServiceEndpoint that you provided.