We have written clients in different programming languages (Java, .NET/Silverlight, Flash, Javascript) that communicate with a server, as our target is to support various technologies on client side. The functionality they are supposed to perform is the same.
One of the main challenges we are having now is finding a simple and effective approach for testing this variety of client technologies against the server. Currently we use maven, hooked with many maven plugins such as JSTestDriver, Flexmojo, NPanday and others which we have developed by our own to do this. Is there any better approach?
Any help would be appreciated, whether it is recommendation for available frameworks/tools or innovative ideas to do this.
Thanks
What you need is a clean design, otherwise everything is a mess and you have to test everything together.
Your server should have an interface with other systems (Browsers, desktop applications, mobile apps) and then test thoroughly this API. You can do that by using the appropriate framework, depending on technology used for the server. This should be your main test effort and then try to keep API stable, so that for every new version of the server you just run a regression test.
Meanwhile you can test the client applications alone by creating a mock server that uses the same API.
Last one would be your integration test where you run a live version of your server and your client application and you run integration tests.
expect is a good framework for testing program-external text interfaces such as client-server interaction. It operates with tests formulated in Tcl on a purely black-box logic level.
Related
In my company we use Ruby to create a framework for API automation and I have heard we can automate using Postman or SoapUI. So why do we have to create a automation framework when we already have tools for it?
It is like a buying a suit at the clothes shop versus going to a tailor to get a suit that is measured especially to your needs.
Using an existing tool will require less initial setup, you will have access to a lot of commonly needed features, without reinventing the wheel. For instance, in Postman there are available test snippets that you can use with little or no programming knowledge. Tools such as ReadyAPI, Katalon Studio, Robot Framework, SoapUI, etc. usually don't have a too steep of a learning curve, compared to developing a customer automation framework from scratch.
Using tools is fine, especially if you understand how they work in the background and have analysed the testing needs for your particular project. For example, a tool like REST Assured makes writing tests for RESTful webservices very easy, but it's actually very complex in the background.
You would build an inhouse automation framework if after researching the existing solutions, you realize they don't fully provide all that you need. A well designed/architectured framework will be far more customizable than any other tool, although it will require more initial work and maintenance as well.
In terms of using a custom test automation framework your testers will generally have to be more technical, more like SDETs rather than typical testers, but does not always has to be the case - I have seen automation frameworks build by developers and the testers would only write tests inside it by re-using the methods in the framework.
Lastly I would advise you to do some experimentation, try one of the commercial or Open Sources tools for API testing and after doing some testing with it try doing the same with a more hands-on approach, like using Python's Request client Apache Http client for Java, but every language has it's equivalent.
We are in the process of transitioning towards SOA.
Our current goal is to try and ensure that more of the application is developed as "Services" (mainly to improve visibility of capability, re-use and de-risk change). Some of those services will be exposed as web services, but many (and probably the majority) will not, and be used for "internal" use only to help reap some of the benefits of SOA.
For those "internal" services we are currently intending on implementing them as OSGi bundles; however we are struggling to understand how best to test them. Our goal is to enable the current System Test team to test all types of services and we have been investigating tools like SoapUI and SOA Test; however it's becoming clearer that we may face some challenges in testing our services implemented as OSGi bundles using tools like these; and indeed asking the test team to do so.
So we're looking for some advice on how best to test aspects of our capability designed to act as a "service", but implemented as an OSGi bundle instead of a web service.
What tools would people recommend, and is this a type of testing that's traditionally done by a developer during unit test, or can it be done by a less technical tester, undertaking the same basic principles of testing interfaces (i.e. inputs, processing, outputs)?
You could theoretically use a Remote Service Admin implementation (like Aries RSA or Eclipse ECF) to expose your internal services to the outside during testing to access them using an external system test tool.
I would not recommend to let an external team test your OSGi services though. It is much better to test the services in your own build using an integration testing tool like pax exam. It allows to define which bundles and other config to install. Then it boots up an OSGi framework with your setup and runs modified junit tests against it. The advantage is that such tests are quite realistic and still quite simple.
See here for some pax exam tests in aries rsa or apache karaf.
The first example uses the pax exam forked container for a very fast test (<1s per Test) while the second example uses the apache karaf container (~10s per Test) for tests that are very near a production system.
So you get much faster feedback than with an external system test team that will always lag a bit behind your current development. It also allows you to establish the policy that each team member runs the tests locally before committing.
I have used LoadRunner a long time ago for web applications, now i wonder if i can use it for desktop application automation. Is it possible ? Can someone link to any guide or examples?
Thanks in advance.
LoadRunner is a tool for performance testing. Usually you want to test the performance of some server that serves several clients via some form of communication. This can work for both web and desktop (and mobile) applications.
I think what you are looking for is a functional testing tool - a tool that can use the application in an automatic way instead of the user. There is a complementary product called UFT (previously QTP) that does exactly that!
UFT - http://www8.hp.com/us/en/software-solutions/software.html?compURI=1172957#.UUlnERxTDRM
LoadRunner can test web and non web client-server applications for performance. Load all of the sample applications and you will find flight samples for Web, for database, for Winsock, for DCOM, etc... You can look to the VUGEN manuals and find all sorts of non-web based client-server protocols for use by thick client applications.
I am developing a Remote Software Provisioning system that should be able to handle all deployment, installation, un-installation and upgrades of software components. Software can be in any language (java, .net, c/c++ etc) and target side can be PC, embedded systems and smart phones.
I have found Apache ACE as good candidate for developing this system.
I want to know if there is any advantage/necessity of using OSGi at target side as Apache ACE can do software provisioning to non-OSGi targets as well.
Having a modular framework like OSGi at the client side is a huge advantage when doing remote management, because it gives you much insight into what's happening inside - installed bundles, dependencies, states of the bundles, available services etc. This helps a lot when you have to solve a problem remotely. Another advantage is that OSGi basically forces programmers to develop proper modular and dynamic systems, which makes (remote) updating much easier.
So, if you have to decide now what language and framework to use for the client side, I strongly recommend OSGi for the embedded and mobile clients. For the PCs (I guess you mean desktop PCs?) this is probably not the best choice - it depends a lot what you want to achieve there. If you want to install MS Office remotely OSGi won't bring you forward ;)
However, if you already have existing programs at the client side and are discussing whether to convert them to OSGi, I would recommend to investigate some time first to see whether they can be converted easily. Some software packages could give you a lot of trouble converting to OSGi, not because OSGi is complex, but because the program itself is not modular and has a lot of assumptions about the static nature of the environment (e.g. nothing ever disappears, parts of the system never get updated etc.). The irony in the matter is that these are exactly the programs which will give you most trouble later anyway no matter which remote provisioning system you chose.
If you have OSGi at some of the targets be sure to use a remote provisioning system which gives you access to the full OSGi functionality and not only the most basic and simple install and update functions. I haven't yet used Apache ACE, but I have experience with another provisioning system - mPower Remote Manager. Here are some snapshots from the documentation which can give you a feeling what is possible with OSGi as a base - you can draw your own conclusions whether it will be useful for your case or not.
I've given some examples in the other question you asked:
What are the non-osgi targets with which Apache ACE can work
You can write your own management agent that talks to the ACE server and installs artifacts. There actually are a couple of places where you could hook in your own code and protocol. Is there a concrete language/environment you're thinking of using, or are you just exploring the possibilities right now?
Well, the advantages of OSGi haven't changed, so for that I can refer you to the standard page.
To be a bit more constructive, I'll read the question as 'Should I bother converting my application to OSGi, as it is not necessary for ACE?'
I think that depends on what 'kind' of updating mechanism you're after. If you have a monolithical application (at least from the provisioning perspective) which you deploy and update only as a whole (Like an iOS app) then there isn't much to gain for provisioning purposes by using OSGi.
For the rest I can tell you the same as I tell anybody else: Converting an application to OSGi isn't hard, but modularizing code can be a nightmare, but something you'll need to face at some point, OSGi or not. If your code is modularized already, using OSGi should be a piece of cake.
I am looking for a tool capable of generating multiple Xmpp connections to load-test a XMPP server with a secure connection, especially starttls.
For a xmpp plain text authentication I had used jab_simul(followed this tutorial) and tsung both with success.
But I was unable to use the tolls above for the starttls,I peeked into the code of both tools and tried different configurations of the tools.
Another option I am pondering is using a xmpp library like eXmpp and make a specific load-testing tool myself with, instead of altering jab_simul (C software with comments in language i do not understand) or altering tsung(all purpose load-testing tool, so lots of place where you can go wrong).
short-story - I am looking for a tool or advice to stress-testing/load-testing a xmpp server.
We are facing exactly the same challenge right now. After deep consideration we found out that only especially build software can deliver the load we want to test. (Remember, you can configure ejabberd to something very specific :-)
For that we developed a small library called xmpp_talker https://github.com/burinov/xmpp_talker (Apache Licence) which is a kind of xmpp client made as a gen_server. I find it is a very nice starting point to build any kind of load simulation software. There is also echo_worker example included. So, you have good base to start. At the moment xmpp_talker is suited for exmpp 0.9.7. As far as I know in a few days will be out version 1.0.0. (or 0.9.9?) There are many bug fixes (trust me you don't want to know about them). On monday I will release xmpp_talker for exmpp 0.9.8 with proper service interruption handling.
In case you deside to go the same way xmpp_talker could be useful for you.
Added: Here is also great article that is realted to the topic: https://support.process-one.net/doc/display/EXMPP/Scalable+XMPP+bots+with+erlang+and+exmpp
There's also the recently started XMPP benchmarking project called xmppench which aims to be a high-performance benchmarking tool simulating some reasonable use cases of XMPP servers. It's written in C++, based on Swiften and boost.