I'm managing a rather widely distributed software application in a semi industrial environment. The software at its heart is based on SOA and employs OPC-UA to make communications between important processes (on local or LAN-based machines) possible. These processes are either a server (e.g. an outer network management server, hardware managers server etc.) or a client (customer panel) or both (servers talking to each other).
OPC-UA has the following problems:
Configuring and maintaining the configurations is a hard job (just Config file settings takes lots of time)
Security measures are too much detailed for my needs (certificate management and sudden invalidation of certifications on customer systems)
Modeling and networking overheads in the library make it hard to work with in my communications (high data rates usually ends in server and client disconnecting)
Unspecified and weird errors like UA Discovery Server stopping to work or respond, etc. which I have reported to OPC GitHub forum many times.
Troubleshooting in internal parts of OPC UA is nearly impossible.
Overall, for me its performance and stability are not reliable enough. I am willing to sacrifice features for better performance and reliability. I've even considered to write sockets from bottom-up for my inter-process (IPC) needs. This way I could at least trace errors to their core. Since I do not need its most advertised feature (aka PLC support) I'm desperate to find a good alternative for it. My main requirements are:
OPC-UA like Data Modeling support that enables me to provide a clean interface to customers and other teams (something like IDL).
Publish/Subscribe, Remote Commands, Update Notifications and Node Based Behavior.
Tough Security is not my concern as my network is closed.
High performance for data rates up to 1Gbps (this could mean UDP support).
I am entirely working in .NET framework. So C# support of OPC-UA is a great help for me.
I've looked at DDS (lacks commands and Update Notifications) and WCF (lacks cross platform support) and many more.
This link also notes about MQTT: Alternative to OPC-UA
What about Google's gRPC + protobufs?
https://grpc.io/
Related
I have just started out with learning WebRTC for implementing audio and video application and know there are various Public stun servers available for peer connection. But i am a bit confused can I use these Public servers for a commercial application?
Also I would like to know if there is any tutorial or guide available from where i can understand how to make and deploy my own stun or turn server if i want to create a commercial app?
Whether you can use public STUN servers for commercial applications depends entirely on the licensing/Terms of Service agreement of the operator of said servers. Peruse those if available. If not indicated otherwise, I wouldn't distinguish "commercial" use from any other use.
There are many many implementations of STUN/TURN servers available you can set up yourself on any machine you happen to have (in practice that probably means an instance on AWS, Azure or the like). Search for and pick one you like. STUN servers use relatively little resources, while TURN servers typically need powerful CPUs and fast internet connections to be useful (they must relay the entire video stream as quickly as possible).
Operating such a server yourself may become expensive, depending on your usage. Using a commercial provider for TURN servers may be the better option; personally I've had good experiences with Twilio in this regard, but do shop around for other offerings.
My only experience with SOA comes from working with NServiceBus. It's really well designed and has some nifty features that I really like:
Processing of messages is transactional (and taken care of for you), so it's "once and only once" (as opposed to "at least once")
Built-in saga support
Automatic retry support
Pub/sub support (but I assume all service/message busses support this)
All that stuff leaves me to concentrate exclusively on the app's logic, but I've only used it in .NET, and Windows is almost certainly a non-starter for the project I'm working on. So my question is...
What service bus technologies out there exist in the *nix ecosystem (including experiences running NSB on Mono) and how do they measure up to NServiceBus running on Windows?
Bonus: In addition to the features above, what languages can send/receive/process messages from the bus? Having the flexibility to write one service in language A and another in language B would be very advantageous for this project.
SonicESB / MQ
I can speak from experience with SonicESB on Linux. It's basically SonicMQ - a JMS compliant MOM - with a process mediation layer on top; that's the service bus.
Pros
Itinerary-based routing; messages contain their own workflow for what services to visit
Intra-container messaging; increases performance by routing messages within the ESB container, avoiding the broker entirely.
Web service endpoints; ESB service may be invoked over HTTP endpoints, bridging the gap between non-Java/JMS clients and the bus.
Good development tools in eclipse for developing and debugging esb services and processes in Java.
Administration tools aren't bad either.
Cons
Transactions don't span service boundaries (may have changed since version 7.5)
No message retry mechanism; we rolled our own using a utility service and message parameter manipulation (may have changed in 7.5)
Pricey; licensed by core, so expect to pay for it.
Client Support
In terms of what technologies can send and receive to/from SonicESB, anything Java JMS, that's for sure. Sonic also offers a library in .net/C#/VB for Windows technology, so you can send and receive messages from the bus. This was a huge plus for us since our development teams use RIA's built in C#.
You may want to check out MuleESB, an open source service bus.
Hope it helps,
It has been quite a while since I've seriously used NServiceBus (in fact, back then, it wasn't even called that - it was just replacement library Udi wrote, to an awful comm library we had to use on a joint project :) ) so I don't know about all NServiceBus features.
I would say, based on what I do know, that Apache Kafka (originally developed by LinkedIn) would be a good starting point. It has relatively strong ordering guarantees, supports publish/subscribe to pools of consumers and is supportive of workflow behaviours e.g. Samza which provides a streaming framework on top of it
Also regarding the client interfaces. You can find the current list here
Apologies if what I am going to ask is too generic and please feel free to mark it as community wiki or even close it but after getting a good answer please.. Here goes nothing.
I had a heated debate over the term application infrastructure with a consultant from a BIG company and hence I am here to see what the experts think as I myself need a good understanding as well. I am a puny software developer and was trying to have a chat about software architecture and stuff and then we kinda got stuck up at the term application infrastructure.
As per my rival's understanding, this is solely used re the back-end hardware components and whereas I thought he is mixing the contexts and from the context of software eng it should refer to stuff that supports the application such as logging, ORM, or even framework (.net) etc.
To what extent am I wrong and if I am please shed some light on the definition...it's really gonna help a lot.
Thanks
Here are a few links with a little description about the term Application Infrastructure extracted from them:
Application Infrastructure -- f5.com
Application infrastructure, comprised of application servers, web servers, and often database servers, is a core component in most network architectures. This part of the network infrastructure delivers high performance application services to the LAN as well as to employees, partners and customers on the WAN.
Some of the key functionality of application infrastructure includes transaction management, clustering, reliable application-to-application messaging, system management, advanced application development tools, proprietary access, and interoperability with legacy technologies.
Application Infrastructure -- networkmagazineindia.com
A (common/standard) platform is required to make different applications in an enterprise work across geographies or multiple locations, and to manage a large number of users and transactions taking place within an enterprise. This platform which ensures that different applications work with each other, is known as an application infrastructure.
Application Infrastructure -- bitpipe.com
A high performance, reliable, and secure integrated technology infrastructure for managing multiple hosted applications by Application Service Providers.
What he seem to be describing is the platform for software (I'll leave it open if that is the same thing as the software platform. I do include software service in the Application Infrastructure definition and also include connectivity/networking along with hardware and software as the three foundation corner stones of Infrastructure. Infrastructure is the general purpose commodity components as opposed to the domain specific components which will typically always be Software; (A possible exception would be an organisation uses custom hardware). AIH; IBM pitch WebSphere as Appliction Infrastructure and Microsoft Consider their Windows Server to the Application Infrasture both agree with you and I.
From what I can see it speaks to HW and SW (middleware and API management).
Gertner definition : Application infrastructure is software platforms for the delivery of business applications, including development and runtime enablers.
Interesting info from Gartner:
http://wso2.com/resources/analyst-reports/comparing-vendors-of-comprehensive-application-infrastructure-suites/
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
ZooKeeper is a highly available coordination service for data centers. It originated in the Hadoop project. One can implement locking, fail over, leader election, group membership and other coordination issues on top of it.
Are there any alternatives to ZooKeeper? (free software of course)
I've looked extensively at Zookeeper/Curator, Eureka, etcd, and consul. Zookeeper/Curator and Eureka are in many ways the most polished and easiest to integrate if you are in the Java world. Etcd is pretty cool and very flexible, but It is really just a HA key store so you would have to write a lot of code to turn it into an opinionated service discovery system.
Consul is (to me) the best of both worlds. It is an opinionated service discovery system written on top of serf, using raft for cluster consensus and gossip for communication. It exposes the discovery / registration endpoints with a well documented REST api, and also allows you to discover services with DNS SRV records, and register services with configuration (i.e. so you can register a database or application you can't integrate a client with, or if you just want to keep your service discovery decoupled from your app)
I've written a blog post about consul where you can learn more and walk through my "try it out" demo
I've also discussed service discovery with etcd & docker if you want to see more about what that custom code might look like.
One last thing! etcd & consul are written in go, so maintaining them is much easier then java solutions like zookeeper. All you need is the consul / etcd binary. no dependencies, no linked libraries, no jvm.
There's a very promising alternative to ZooKeeper called etcd (github.com/coreos/etcd), written by CoreOS team. Unlike Doozerd, etcd is being actively developed.
Just discovered Accord (C) and OpenReplica/ConCoord (Python) which may be interesting solutions
[EDIT] The Hashicorp crew, of Vagrant and Packer fame, are cooking "a decentralized solution for service discovery and orchestration" called Serf.
[EDIT2] Hashicorp strikes again ! They just released Consul, built on top of Serf. The pitch: "a solution for service discovery and configuration, completely distributed, highly available, scalable to thousands of nodes and services across multiple datacenters".
Yes, there is also Doozerd (https://github.com/ha/doozerd). Take a good look at it, it's a nice, single binary distributed coordination service developed by Heroku. With bindings/libraries for java/python/ruby/node. Very easy to get started with and play around.
Take a look at Serf. There is a comparison vs Zookeeper here.
OpenReplica from my research group is a highly available FOSS coordination service for data centers. It can be used for implementing locking, fail over, leader election, group membership and other coordination services. It differs from ZooKeeper in two critical ways:
It uses an object-oriented API. This makes it much easier to write coordination services. Synchronization code for OpenReplica looks exactly like its textbook counterpart; there is no need to master a file and upcall-based API like in ZooKeeper and Chubby.
It enables dynamic membership updates to the replica set. There is no need for static configuration files. The system is integrated into DNS (authoritative, slave for OpenReplica, or Amazon Route 53).
We actively support the system, do not hesitate to let us know if you have further questions.
There's a project called Noah on github that looks interesting, it says that it's "loosely based on Apache ZooKeeper" https://github.com/lusis/Noah with REST support being a key feature (ZK has this as a contrib/option rather than built in).
There are different tools that optimize for different engineering trade-offs.
ZooKeeper Scales marginally for reads; writes with many observers can be slow. It is proven and has a sizable community.
Accord Seems interesting for write-intensive uses, however typical use-cases already have domain-specific solutions (i.e., logging, telemetry).
The others are somewhat interesting but generally unproven. Don't get this wrong if intended for production usage.
I'd found this comparison of Zookeeper, etcd and Doozer:
http://devo.ps/blog/zookeeper-vs-doozer-vs-etcd/
Serf (serfdom.io) is also a nice solution as it is simple! But you must consider, SERF is just a cluster-manager which enables you to send custom events to all cluster nodes. Thats nice, but you have to write your own shell scripts (aka events).
See this example: "https://www.digitalocean.com/community/articles/how-to-set-up-a-serf-cluster-on-several-ubuntu-vps"
The advantage is, you're getting a very simple cluster-manager and you're able to combine this with your favorite configuration, deployment or continuous integration tool.
It seems Corosync is also like ZooKeeper.
I know this post is quite old, but someone who is looking at all possible alternatives I would also like to suggest JGroups library which is mature enough to be used in production environment. I have used it successfully in one of my projects mainly for distributed coordination and to share messages between cluster. It also support AWS support in addition to its flexible architecture where you can customize its stack to get what you need. I suggest you to have a look at it
Simplified, I have an application where data is intended to flow over the internet between two servers. Ideally, I'd like to test at what point the software ceases to function. At what lowerbound limit (bandwidth, latency, dropped packets) do things stop working to test the reliability of the software.
What I thought I would do was the following:
Setup up 3 machines (VMware instances)
Install the 2 applications on two of the servers.
Setup up the 3rd server to sit between the two machines by doing some sort of magic with Routing and Remote Access on Windows 2003
Install either Traffic Shaper XP or NetLimiter to limit the bandwidth
Run something like TMnetSim Network Simulator to simulate a bad connection.
Does this sound like a good idea or are there easier/better ways of doing this? I'm not that comfortable on Linux and my team mates are even less so.
WANem does exactly this. We have used it both in a virtual machine on the desktop and on a dedicated old pc and it worked great. It can simulate all sorts of broken connectivity.
FreeBSDs ipfw has provisions to simulate links with a given bandwith, latency or error rate. You could use that FreeBSD machine as your machine "in the middle" in your above setup.
You probably can also run at least one of the endpoints on the same machine if you want to reduce the amount of servers involved.
Someone actually packaged up the settings and whatnot necessary for the FreeBSD solution to this problem and they call it DUMMYNET.
It simulates/enforces queue and bandwidth limitations, delays, packet losses, and multipath effects. It also implements a variant of Weighted Fair Queueing called WF2Q+. It can be used on user's workstations, or on FreeBSD machines acting as routers or bridges.
It can simulate exactly what you want, and its free and will boot onto commodity hardware. They even have a canned install of it that is small enough to put on a floppy disk (!) that you can download at that link.
Maybe it is time to learn a bit about Linux because adding a 50ms delay on every outgoing packet can be done in typing just one line:
tc qdisc add dev eth0 root netem delay 50ms
For more see the Linux Traffic Control HOWTO
We had a similar requirement some ten years ago - I'll see if I can recall how we managed it.
If I remember, we wrote a socket proxy program which was controlled by inetd on a UNIX box. This socket would accept connections from a client and open equivalent sessions through to the server. It would then loop, passing messages in both directions.
The way we achieved WAN characteristics was to introduce random delays (with upper and lower limits) in both the connection establishment and the passing of data once the link was up.
It also had the feature to drop the link occasionally as WAN links were less reliable for us than local traffic.
I recall we had to make it threaded to stop the delays from affecting reverse traffic on the link.
There is a very good (and free) Microsoft solution for that, we use it for quite some time and it works great, it can very easily simulate every thing(packet loss, low bandwidth, disconnection, latency....)
This is the best solution i found for a windows environment
More information and a download link can be found here: MARCO blog post
this product has gone some evolution and it is now integrated into visual studio as part of the automation testing, but i found the use of the standalone(that is quite hard to find, so keep a local copy) to work much better. keep in mind that you need at least two computers(or VMs) since you need to pass through a network adapter in order for the application to work its magic.