Let me first explain what I am trying to achieve first and then I'll get into the details of the things I have tried already.
So, we have a VM that is on our premise and another VM that is on a customer's premise. The access to these VMs are only available to certain IP addresses. So, we could say that they are secure enough for our use-case.
Data from customer's environment flows through and into our VM through a mosquitto broker that is set-up on both these environments. This is done with the help of broker bridging that works fine. However, since this bridge is over the internet, we want to ensure that the data is encrypted and that no-one could intercept this over the internet and use this data in a malicious manner.
To achieve this we are making the use of SSL broker encryption. The first method I tried is to use PKS encryption method.
Here is the broker config at the customer environment.
listener 8883
connection bridgetest
address 147.1.20.1:8883
bridge_identity bridge1
bridge_psk 123456789
topic # both
And here is the broker config at our environment.
listener 8883
psk_hint SAAS Deployments
psk_file c:\DemoCompany\psk_file.txt
The contents of the psk_file.txt are very simple and same as the bridge identity and the bridge_psk provided in the config of customer environment.
The problem I am facing here is that even though I change the bridge_identity or the bridge_psk at customer's environment to something that is not in the psk_file.txt, I am still able to connect the 2 brokers over the bridge.
My understanding of this was that if I change the bridge_psk to some random hex code, the connection should get rejected. But that doesn't seem to happen. Am I doing something wrong or missing something over here?
The following config files work for me with v2.0.9 builds shipped from the mosquitto PPA on Ubuntu
Client broker:
listener 1889
connection bridge
address 127.0.0.1:1890
bridge_identity bridge1
bridge_psk 123456789987654321
topic # both 0
Bridge broker
listener 1890
psk_hint my test bridge
psk_file /temp/psk/psk_file.txt
use_identity_as_username true
The use_identity_as_username is required as from Mosquitto v2 onward allow_anonymous defaults to false
Related
Is kafka suitable for Internet-use?
More precisely, what I want is to expose kafka topics as "public interface", then external consumers (or producers) can connect to it. Is it possible?
I hear there are problems if I want to use the cluster in both internal and external networks, because it is then hard to configure advertised.host.name. Is that true?
And do I have to expose zookeeper as well? I think the new consumer/producer api no longer need that.
Kafka's wire protocol is TCP-based and works fine over the public internet. In the latest versions of Kafka you can configure multiple interfaces for both internal and external traffic. Examples of Kafka over the internet in production include several Kafka-as-a-Service offerings from Heroku, IBM MessageHub, and Confluent Cloud.
You do not need to expose zookeeper if the Kafka clients use the new consumer API.
You may also choose to expose a REST Proxy such as the open source Confluent REST Proxy as a more client firewall friendly interface since it runs over HTTP(S) and will not be blocked by most corporate or personal firewalls.
I would personally not expose the Kafka server directly to clients via TCP for these reasons, only to name a few:
If a bad client opens too many connections this may affect the stability of the Kafka platform and may affects other clients too
Too many open files on the Kafka server, HW/SW settings and OS tuning is needed to limit uncontrolled clients
If you need to add a Kafka server to increase scalability, you may need to go through a lot of low level configuration (firewall, IPs visibility, certificates, etc.) on both client and server side. Other product address these problems using gateways or proxies: Coherence uses extend proxy clients, tibco EMS uses routed destinations, other SW (many JMS servers) use Store&Forward mechanisms, etc.
Maintenance of the Kafka nodes, in case of clients attached to the Kafka servers, will have to consider also the needs of clients and the SLA (service level aggreement) that have been defined with the client (ex. 24*7*365)
If you use Kafka also as a back end service, a multi layered architecture should be taken into consideration: FE gateways and BE services, etc.
Other considerations require to understand what exacly you consider to be an external (over the internet) consumer/producer in your system. Is it a component of your system that needs to access the Kafka servers? Are they internal or external to your organization, etc.
...
Naturally all these considerations can be correctly addressed also using a TCP direct connection to the Kafka servers, but I would personally use a different solution.
HTTP proxies
Or at least I would use a dedicated FE Kafka server (or couple of servers for HA) dedicated for each client that forward the messages to the main Kafka group of servers
It is possible to expose Kafka over the internet (in fact, that's how managed Kafka providers such as Aiven and Instaclustr make their money) but you have to ensure that it is adequately secured. At minimum:
ZooKeeper nodes should reside in a private subnet and not be routable from outside. ZK's security is inadequate and, at any rate, it is no longer required to bootstrap Kafka clients with ZK address(es).
Limit access to the brokers at the network level. If all your clients connect from a trusted network, then set appropriate firewall rules. If in AWS, use VPC peering or Direct Connect if you are connecting cloud-to-cloud or cloud-to-ground. If most of your clients are on a trusted network but a relative minority are not, force the latter to go via a VPN tunnel. Finally, if you want to allow connectivity from arbitrary locations, you'll just have to allow * on port 9092 (or whichever port you configure the brokers to listen on); just make sure that the other ports are closed.
Enable TLS (SSL) for client-broker connections. This is easily configured with a self-signed CA. Depending on how you expose your listeners, you may need to disable SSL hostname verification on the client. (The certificate chain of trust breaks if the advertised host names don't match the certificate's common name.) The clients will need the CA certificate installed. (Same CA that signed the brokers' certs.)
Optionally, you may enable mutual TLS authentication; however, this is logistically more taxing, as it requires each client to have its own private key that is signed by a CA trusted by the broker.
Use SASL to authenticate the client to the broker and create individual users for each application and each person that is expected to access the cluster.
Issue minimally-sufficient cluster- and topic-level access privileges in the ACLs for each user, following the Principle of Least Privilege (PoLP).
One other thing to bear in mind: Not all tooling supports SASL/SSL connectivity and some tools actually require a connection to ZooKeeper nodes (which will not be reachable in the above setup). Make sure any tooling you rely on uses the 'new' style of connectivity directly to the Kafka brokers and does not require a Zookeeper connection.
Beyond configuring client TLS, brokers have to have public IPs which we try to avoid. Normally for other services we hide everything behind load balancers. Would this be possible with kafka?
I'm not sure the Confluent REST proxy hosted on a public server is a real option when you need the high performance batching of the java producer client.
I'm trying to get RabbitMQ configured behind an F5 load balancer. I have a working RabbitMQ node with the default node name of rabbit#%computername%. It's set to listen on all network interfaces (all IP addresses 0.0.0.0:5671 which is the AMQP SSL port), and it's working fine. However, all client applications that connect to it are currently using the specific node name e.g. "%computername%". In order to take advantage of the fault tolerance of the load balancer, I want to update all my client applications to use the load-balanced name instead of the specific node name e.g. connect using HostName = "balancedname.mycompany.com" instead of "%computername%". However, when I update my client applications to connect to the load-balanced name, the connection fails. How can I get this to work?
I'm a novice at F5, and I did notice that the pool's members' addresses are IP addresses...should these be the node names instead of the IPs? Is that even possible seeing as the node name can be completely arbitrary and doesn't necessarily map to anything that's network-resolveable? I'm in a hosting situation where I don't have write access to the F5, so trying these things out is a bit tricky.
I haven't found very much information at all on load balancing a RabbitMQ setup. I do understand that all RabbitMQ queues only really exist on one node, and I've set up the F5 in an active-passive mode so that traffic will always route to the primary node unless it goes down.
Update 1: It seems that this issue came back to bite me here. I'm using EXTERNAL authentication using an SSL certificate, and since clients were connecting using the load balance name instead of the node name, and the load balance name was NOT used to create the certificate, it was rejecting the connection. I ended up re-generating the certificate and using the load balance name, but that wasn't enough - I also had to add an entry in the Windows hosts file to map 127.0.0.1 and ::1 to the load balance DNS address.
Update 2: Update 1 solves connection problems only for running client applications on the app server that is part of the load balancer, but remote clients don't work. Inner exception says "The certificate chain was issued by an authority that is not trusted". RabbitMQ + SSL is hard. And adding load balancing makes it even harder.
I'm answering my own question in the hopes that it will save folks some time. In my scenario, I needed for clients to connect to a load balanced address like myrabbithost.mycompany.com, and for the F5 to direct traffic to one node as long as it's up and failover to the secondary node if it's down. I had already configured security and was authenticating to RabbitMQ using self-signed certificates. Those certificates had common names specific to each host which was the problem. In order to work with .NET, the common name on the certificate must match the server name being connected to (myrabbithost.mycompany.com in my case). I had to do the following:
Generate new server and client certificates on the RabbitMQ servers with common names of myrabbithost.mycompany.com
Generate a new certificates for the clients to use while connecting in order to use SSL authentication
Still on the RabbitMQ servers, I had to concatenate the multiple cacert.pem files used for the certificate authority so that clients can authenticate to any node using a client certificate generated by any node. When I modified rabbit.config to use the "all.pem" instead of "cacert.pem", clients were able to connect, but it broke the management UI, so I modified the rabbitmq_management settings in rabbit.config to specific the host-specific cacert.pem file and it started working again.
In order to set up high availability, I set up a RabbitMQ cluster, but ran into some problems there as well. In addition to copying the Erlang cookie from the primary node to the secondary node at C:\Windows and C:\users\myusername, I had to kill the epmd.exe process via task manager as the rabbitmqctl join_cluster command was failing with a "node down" error. The epmd.exe process survives RabbitMQ stoppages and it can cause rabbitmqctl.bat to report erroneous errors like "node down" even when it's not down.
I am new in using rabbitMQ and I am trying to sent an 'hello' message over internet,
I am implementing the example available in the rabbitMQ website Java RabbitMQ Hello world example, but in the example they use localhost, I try to change it to the IP address for the sender and receiver computer as explained at the website and put the sender code at a machine and receiver code on another, but it doesn't work.
My questions:
1) is rabbitMQ works over internet or it works just over local network?
2) in both cases, how to configure each computer and what each one should have?
3) Do I need to install rabbitMQ on both machines? or on one of them to run it a server?
Please if anyone can help me in configure them step-by-step, get me an answer with details.
It is a grant problem.
The user guest guest ( default for rabbitmq ) works only in localhost.
Please read this post:
Can't access RabbitMQ web management interface after fresh install
and also this:
RabbitMQ 3.3.1 can not login with guest/guest
To enable guest guest and/or create a new user.
The best practice is to create another user.
Let me answer your questions one by one
1) Yes. RabbitMQ should work over internet, you should be able to connect by giving the public ip of the RabbitMQ server. If you connecting to a server with username/password enabled then it should be provided while creating the connection.
ConnectionFactory factory = new ConnectionFactory();
factory.setUsername("username");
factory.setPassword("pwd");
2) One of the machines should have the RabbitMQ server(broker) installed and running. You can produce or consume messages from any of the machines using Java RabbitMQ client. If you had 3 machines then all three, the RabbitMQ server, message producer and message consumer could be on 3 machines.
3) You don't need to install RabbitMQ on both the machines. Install only on the machine which is running as server.
Not sure if i'm really up-to-date, but i'm looking in a way to convert my existing project to use HTML5 websockets.
Here's my situation :
- Client runs a modified java vnc applet with extra parameter (CONNECT).
- Modified stunnel listenin on webserver (with both public, private IP) port 443
- Client connects to 443 and sends (prior to RFB) a HTTP packet like :
'CONNECT 10.0.0.1:4001'
- Stunnel opens a new stream to 10.0.0.1:4001 using SSL wrapper
- VNC Server (#10.0.0.1:4001) responds, connection is established.
Now I want to get rid of the Java Applet and switch to Websocket using NoVNC.
I want to be able to :
- Open a single port on the webserver (HTTPS preferably)
- Have client connect using HTML5 only (no more java applet)
I cannot change :
- VNCServer will still be listening on private LAN only.
- VNCServer will still listen to a bunch of ports, each corresponding to
a virtual server
Questions are :
- How to give NoVNC the notion of target HOST:PORT ?
- Is stunnel still be usable ? Or should I change to websocket proxy ?
If anyone has a starting point, i'd really appreciate !
Disclaimer: I created noVNC so my answer may be heavily biased ;-)
I'll answer you second question first:
stunnel cannot be used directly by noVNC. The issue is that the WebSockets protocol has an HTTP-like initial handshake and the messages are framed. In addition, until binary payload support is added to WebSockets, the payload is base64 encoded by the websockets proxy (websockify). Adding the necessary support to stunnel would be non-trivial but certainly doable. In fact noVNC issue #37 is an aspirational feature to add this support to stunnel.
First question:
noVNC already has a concept of HOST:PORT via the RFB.connect(host, port, password) method. The file vnc_auto.html at the top level shows how to get noVNC to automatically connect on page load based on the host, port and password specified as URL query string parameters.
However, I think what you are really asking is how do you get noVNC to connect to alternate VNC server ports on the backend. This problem is not directly addressed by noVNC and websockify. There are several ways to solve this and it usually involves an out-of-band setup/authorization mechanism so that the proxy can't be used to launch attacks by arbitrary hosts. For example, at my company we have a web based management framework that integrates noVNC and when the user wants to connect to the console, an authenticated AJAX call is used to configure the proxy for that particular user and the system they want to connect to. Our web management interface is internal only.
Ganeti Web Manager uses a similar model and the source is available. They have a fork of VNCAuthProxy that has WebSockets support. They use a control channel from the web interface to the VNCAuthProxy to setup a temporary password associated with a specific VNC server host:port.
Also OpenStack (Nova) integrates noVNC uses a similar out-of-band token based model to allow access with their nova-vncproxy.
Some links:
Ganeti Web Manager
Wiki page about how noVNC works in Ganeti Web Manager
Ganeti Web Manager sources
Ganeti Web Manager VNCAUthProxy sources
Using noVNC in Nova/OpenStack
OpenStack fork of noVNC
Old nova-vnc-proxy code
Current nova vnc proxy code
I want to write a simple SMPP Server that basically forwards traffic to another SMPP server (C#, PHP). What are the things I need to know? How do I proceed?
With regards to Goran's comment, one possible solution would be a simple tcp proxy such as simpleproxy.
From the Ubuntu package description:
simpleproxy acts as a simple TCP proxy. It opens a listening socket on
the local machine and forwards any connection to a remote host. It can be
run as a daemon or through inetd.
Olaseni,
I've done something similar in the past, but i used perl. What i did was taking a port forwarding proxy which i downloaded from accordata.com. (port-proxy.pl)
I modified this to use the NET::SMPP module to validate PDU's when reading the incoming socket. Once the PDU was of type "Bind_request" i would validate against a dbase, replace credentials if validation was successfull and than forward or if credentials were not validated, issue a reject to the client and disconnect. Alternatively if the PDU contained anything else, i would forward using the logic that was already existing in port-proxy.pl.
You can write simple smpp lib and forward smpp traffic from many applications to the one smpp connection to the sms provider
I can advice you jsmpp lib, but it's for java. It's very simple and cool lib. Many low level things happen behind the scenes and you can focus on your business logic
Find more here
I have written exactly what you are asking for in vb.net
What i did was listen for inbound PDU (connect, bind, sms, and disconnect too) identifying each inbound connection uniquely - for the authentication bit,
then i forward the traffic onward to the delivery smsc.
Your SMPP service simply needs to listen for inbound PDU packets... as well as send heartbeat packets to the connected clients, if required.