It seems MQTTUtils Only provide three methods,
def createStream(jssc: JavaStreamingContext, brokerUrl: String, topic: String, storageLevel: StorageLevel): JavaDStream[String]
Create an input stream that receives messages pushed by a MQTT publisher.
def createStream(jssc: JavaStreamingContext, brokerUrl: String, topic: String): JavaDStream[String]
Create an input stream that receives messages pushed by a MQTT publisher.
def createStream(ssc: StreamingContext, brokerUrl: String, topic: String, storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK_SER_2): DStream[String]
Create an input stream that receives messages pushed by a MQTT publisher.
But How can I provide username and password if the broker enabled authentication?
You could try including the username and password in the url:
mqtt://username:password#host:port
Please find this MQTT Scala Word Count Example.
Particular for your case run the publisher as
bin/run-example org.apache.spark.examples.streaming.MQTTPublisher mqtt://username:password#host:port foo
And Subscriber as
bin/run-example org.apache.spark.examples.streaming.MQTTWordCount mqtt://username:password#host:port foo
Before doing this ensure that you are started ActiveMQ broker.
example code
import org.apache.activemq.broker.{TransportConnector, BrokerService}
.
.
.
.
def startActiveMQMQTTBroker() {
broker = new BrokerService()
broker.setDataDirectoryFile(Utils.createTempDir())
connector = new TransportConnector()
connector.setName("mqtt")
connector.setUri(new URI("mqtt:" + brokerUri))
broker.addConnector(connector)
broker.start()
}
pom file
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-core</artifactId>
<version>5.7.0</version>
</dependency>
You can trying using the customized spark-streaming-mqtt-connector library available here - https://github.com/sathipal/spark-streaming-mqtt-with-security_2.10-1.3.0.
This library adds the following on top of the original library,
Added TLS v1.2 security such that the communication is always secured.
Stored topic along with the payload in the RDD.
So, use the following method to create the stream,
val lines = MQTTUtils.createStream(ssc, // Spark Streaming Context
"ssl://URL", // Broker URL
"<topic>", // MQTT topic
"MQTT client-ID", // Unique ID of the application
"Username",
"passowrd")
There are overloaded constructors that allows you to pass the RDD storage level as well. Hope this helps.
Related
I am moving a old app from Msmq to RabbitMQ. The App uses MassTransit 2.10 and I need a function that returns the number of messages in queue for a specific message type.
In the current implementation there is this line of code that returns the message types:
var messages = MsmqEndpointManagement.New(endpoint.Address).MessageTypes();
Is it possible to replace this instruction with something similar when using RabbitMQ ?
When moving to RabbitMQ, the management of queues is different. Since it's a broker (compared to MSMQ, which is a, well, different), it was designed with a separate management API and console. There are other libraries that can be used to get message counts, but not one that will get you the message types (since it would require reading every message to find the type - which is what that MSMQ method above is doing, btw).
I'd suggest looking at HareDu to manage your broker from the application/API.
With HareDu 2 Broker and Autofac APIs you can do the following:
var result = _container.Resolve<IBrokerObjectFactory>()
.Object<Queue>()
.GetAll()
.Select(x => x.Data)
.Select(x => new
{
QueueName = x.Name, x.TotalMessages
});
I have solved the issue using the following function, with EasyNetQ:
public static int GetMessageCount(string queueName)
{
IQueue queue;
IBus bus = getBusFromName(queueName);
if (queues.TryGetValue(queueName, out queue))
return (int)bus.Advanced.MessageCount(queue);
return 0;
}
the getBusFromName() it's a function that retrieve the IBus instance of the queue from a dictionary in which I store all the queues used by the software.
The documentation for Spring WebSockets states:
4.4.13. User Destinations
An application can send messages targeting a specific user, and Spring’s STOMP support recognizes destinations prefixed with "/user/" for this purpose. For example, a client might subscribe to the destination "/user/queue/position-updates". This destination will be handled by the UserDestinationMessageHandler and transformed into a destination unique to the user session, e.g. "/queue/position-updates-user123". This provides the convenience of subscribing to a generically named destination while at the same time ensuring no collisions with other users subscribing to the same destination so that each user can receive unique stock position updates.
Is this supposed to work in a multi-server environment with RabbitMQ as broker?
As far as I can tell, the queue name for a user is generated by appending the simpSessionId. When using the recommended client library stomp.js this results in the first user getting the queue name "/queue/position-updates-user0", the next gets "/queue/position-updates-user1" and so on.
This in turn means the first users to connect to different servers will subscribe to the same queue ("/queue/position-updates-user0").
The only reference to this I can find in the documentation is this:
In a multi-application server scenario a user destination may remain unresolved because the user is connected to a different server. In such cases you can configure a destination to broadcast unresolved messages to so that other servers have a chance to try. This can be done through the userDestinationBroadcast property of the MessageBrokerRegistry in Java config and the user-destination-broadcast attribute of the message-broker element in XML.
But this only makes the it possible to communicate with a user from a different server than the one where the web socket is established.
I feel I'm missing something? Is there anyway to configure Spring to be able to safely use MessagingTemplate.convertAndSendToUser(principal.getName(), destination, payload) in a multi-server environment?
If they need to be authenticated (I assume their credentials are stored in a database) you can always use their database unique user id to subscribe to.
What I do is when a user logs in they are automatically subscribed to two topics an account|system topic for system wide broadcasts and account|<userId> topic for specific broadcasts.
You could try something like notification|<userid> for each person to subscribe to then send messages to that topic and they will receive it.
Since user Ids are unique to each user you shouldn't have an issue within a clustered environment as long as each environment is hitting the same database information.
Here is my send method:
public static boolean send(Object msg, String topic) {
try {
String destination = topic;
String payload = toJson(msg); //jsonfiy the message
Message<byte[]> message = MessageBuilder.withPayload(payload.getBytes("UTF-8")).build();
template.send(destination, message);
return true;
} catch (Exception ex) {
logger.error(CommService.class.getName(), ex);
return false;
}
}
My destinations are preformatted so if i want to send a message to user with id of one the destinations looks something like /topic/account|1.
Ive created a ping pong controller that tests websockets for users who connect to see if their environment allows for websockets. I don't know if this will help you but this does work in my clustered environment.
/**
* Play ping pong between the client and server to see if web sockets work
* #param input the ping pong input
* #return the return data to check for connectivity
* #throws Exception exception
*/
#MessageMapping("/ping")
#SendToUser(value="/queue/pong", broadcast=false) // send only to the session that sent the request
public PingPong ping(PingPong input) throws Exception {
int receivedBytes = input.getData().length;
int pullBytes = input.getPull();
PingPong response = input;
if (pullBytes == 0) {
response.setData(new byte[0]);
} else if (pullBytes != receivedBytes) {
// create random byte array
byte[] data = randomService.nextBytes(pullBytes);
response.setData(data);
}
return response;
}
I was given an API url, and a method getUserPost() which returns the data needed for my data processing function. I am able to get the data by using Client from suds.client as follow:
from suds.client import Client
from suds.xsd.doctor import ImportDoctor, Import
url = 'url'
imp = Import('http://schemas.xmlsoap.org/soap/encoding/')
imp.filter.add('filter')
d = ImportDoctor(imp)
client = Client(url, doctor=d)
tempResult = client.service.getUserPosts(user_ids = '',date_from='2016-07-01 03:19:57', date_to='2016-08-01 03:19:57', limit=100, offset=0)
Now, each tempResult will contain 100 records. I want to stream the data from given API url to RDD for parallelized processing. However, after reading the pySpark.Streaming documentation I can't find a streaming method for customized data source. Could anyone give me an ideal how to do so?
Thank you.
After a while digging, I found out how to solve the problem. I employed the use of Kafka Streaming. Basically you need to create a producer from given API, specify topic and Port for communication. Then a consumer to listen to that specific topic and Port to start streaming the data.
Note that the Producer and Consumer must be working as different threads in order to archive real-time streaming.
I'm trying to add some custom logic to messages in ServiceStack and RabbitMQ.
It seems that the queues created by ServiceStack have some illegible characters prepended to the queue name and that makes it hard to reference them by name. For example (link from the RabbitMQ admin tool):
http://localhost:15672/#/queues/%2F/%E2%80%8E%E2%80%8Emq%3ATestRequest.inq
Note the %E2%80%8E%E2%80%8E prepended to the queue name. Although the queue looks like mq:TestRequest.inq it seems to have a different name. I also checked on another machine and the behaviour is consistent. I also suspect routing keys are affected in the same manner.
However, if I manually create a queue like this (and as far as I can see, ServiceStack does it in a similar way):
RabbitMqServer mqServer = new RabbitMqServer(connectionString: hostName, username: userName, password: password);
RabbitMqMessageFactory factory = (RabbitMqMessageFactory)MqServer.MessageFactory;
using (var mqClient = new RabbitMqProducer(factory))
{
var channel = mqClient.Channel;
string qName = new QueueNames(typeof(TestRequest)).In;
channel.QueueDeclare(qName, true, false, false, null);
}
The creted queue has a "normal" name without extra characters.
http://localhost:15672/#/queues/%2F/mq%3ATestRequest.inq
Also, it seems that the exchanges are created with names as expected.
My questions:
How to force ServiceStack to create queues without appending these characters?
OR
How to construct queue names containing these characters?
EDIT:
It seems that the inserted character is Left-to-right mark ( or \u200e). Prepending these characters to the queue name / routing key seems to get the job done. However, this looks rather hacky so I'd like to avoid doing this.
This might be inside the internals of RabbitMQ and may depend if you are using AMQP or STOMP. Here is an except from the full page:
If /, % or non-ascii bytes are in the queuename, exchange_name or routing_key, they are each replaced with the sequence %dd, where dd is the hexadecimal code for the byte.
RabbitMQ - Stomp - Destinations - AMQP Semantics
I am new to activeMQ, I have issues pushing messages to a queue defined by activeMQ from a message producer residing on another server.
I have a few queues in the application created on activeMQ using camel routes. I am trying to perform remote JNDI lookup on these queues from an application on another server. I have used the snippets from activemq documentation from http://activemq.apache.org/jndi-support.html page.
I could get connected to the activeMQ, but I couldn't look up the queues defined using camel routes.
The queue consumer is created through the camel route defined below.
from("jms:queue:APP.IF.JMS.OUTBOUND")
.... // This route does some processing.
But I don't see this queue in the lookup as performed below -
String destination = "APP.IF.JMS.OUTBOUND";
ConnectionFactory cf = null;
Destination dest = null;
Context ctx = null;
Properties params = new Properties();
readProperty(params, Context.INITIAL_CONTEXT_FACTORY, "org.apache.activemq.jndi.ActiveMQInitialContextFactory", false);
readProperty(params, Context.PROVIDER_URL, "tcp://localhost:61616", false);
readProperty(params, "queue.AS.IF.JMS.REQUEST",
"AS.IF.JMS.REQUEST", false);
ctx = new InitialContext(params);
cf = (ConnectionFactory) ctx.lookup("ConnectionFactory");
System.out.println(ctx.getEnvironment());
dest = (Destination) ctx.lookup(destination);
..............
The last line fails when lookup is done on this queue. I do see this on the console. Am I missing some configuration to expose this queue on JNDI?
Appreciate your response.
Change the word localhost in your PROVIDER_URL to your machineName.
Since it is remote operation, it needs your machine name.