I have started to use Karate to test our mobile app which is using GraphQL and it is working good so far, though a bit of a learning curve for me as I am not a programmer by trade, but I need to look further into the future and be sure to find a framework that will also support our need to automate tests for our custom web applications as well. I would think Selenium would be the way to go so I am looking for a testing framework where I can test not only the GraphQL queries which our micro services/APIs are written, but also our web applications and our mobile app. We are a MS shop but if need be, as with Karate, we can venture into a different stack. Thanks!
Disclaimer: dev of Karate here.
I don't know about mobile, but teams have been successful mixing Karate and Selenium / WebDriver into a single integrated script, which is possible because of Karate's Java inter-op.
This is a Stack Overflow answer recommending the above approach and this answer is an update from the same team reporting success.
One more reason you may want to consider Karate is that it may be the only framework that allows you to re-use functional test scripts as performance tests, via Gatling integration (still in development).
Karate in my opinion is a "safe" choice as a testing framework, because it is technically Gherkin and "language neutral". This may be a surprise for those new to Karate, but it actually leans towards using JavaScript for scripting instead of Java.
Of course, it is worth mentioning that I have yet to find a framework that makes testing GraphQL as easy as how Karate does.
Selenium and Rest assured integration framework is the answer to your question. I created a test automation framework to test an ionic application which is using GraphQL api along with Rest webservices. I have used Testng and cucumber in my framework to perform web app side validations and used Rest assured along with GSON to validate the GraphQL and rest services. You can easily integrate appium in your framework along with selenium and rest assured to cater the need of mobile testing.
Related
When you are doing e2e tests for your application, you want to test the whole application, not some portions of it like unit tests or integration testing.
But in some situations, people do mock APIs.
For example, when you have a massive microservice as your back-end, which makes your e2e tests very slow, or beside your own API, you rely on other third-party APIs, which makes your e2e tests fail occasionally.
So you only want to make sure that your front-end application works well, what should you do?
In my company, we have a massive system with a really heavy database which makes e2e testing very ineffective. Is it right to mock APIs in such a scenario?
My understanding here is that if you want to test only your front-end application (what is not E2E testing in my opinion) you can use unit tests instead. If you still want to test the user interface from the browser, then you can mock the APIs' responses, but still not being E2E testing.
In case you want to perform an end-to-end testing, then you shouldn't mock any database or API call.
The exception here is a third-party API that is not under your control. In that specific case you can mock it to have less external dependency in your tests, but if that third party changes and you are not aware of it, you wont't notice if it's mocked. Said that, if you mock third-party APIs be sure you have a fluent communication with the API provider to get alerts on changes before your app fails.
I have a scenario where I want to automate my functional tests from the UI for a web application. Had it been simple I would have used a karate/similar framework to get the work done. The main caveat is to surpass the biometric scan in order to complete the functional tests. I know appium supports such a case for the mobile-native/ hybrid applications. But i want to know what are all the other tools that i can use to automate the same for a plain web applications. Thanks in advance.
Give some examples for test automation tools used for testing an ESB?
Can Selenium be used for testing or are there any specific tools?
I believe Citrus is the standard in ESB automation testing. While I am not as familiar with Selenium, since it is browser automation, at a minimum it can be used for REST based service testing, as this can be done thru a typical browser without additional support.
Google Chrome Postman with jetpack can do both REST and SOAP.
I'm developing an Ember.js app with a Phoenix API. I have followed someone's advice to keep the UI and API as separate projects, and I know that I can use ember-cli-mirage to mock my API during development and testing. But this makes me really nervous. I'm used to having a suite of integration tests that tests the UI and API together. I know for a fact that one day me or another developer is going to make a breaking change in one of the projects, and we won't realise it until users start complaining.
On the other hand, I really like the idea of mocking the API in the client where Ember.js is running. It should make development and testing really fast.
Is there a way that I can extract a high-level description of my API end points, and use that as a sanity check to make sure that my mocked API fulfills all of the requirements? For example, if I add or remove a route in my API server, I want my Ember.js tests to fail immediately if the mocked API doesn't match those changes. Because I know that this is going to happen one day. It's especially concerning if I want to use continuous deployment after successful builds.
Or should I just start up a real API server on the CI machine, and run my tests against that?
I kind of like the idea of enforcing an API contract, though. I could also reuse the principle in any future mobile or desktop apps. You get some guarantee of consistency without having to install a ton of dependencies and spin up a real API.
Another idea: Maybe I write a set of API acceptance tests, but run them against both the real and the mocked API. And then I could include the mocked API code (ember-cli-mirage) inside my main API repo, and link it into the Ember.js repo as a submodule.
How are people currently approaching this issue?
Your Ember tests should focus on the behavior of the client application itself, not the implementation details of your API. While it is more trouble to maintain a separate mocked instance of your API logic in ember-cli-mirage, in reality you should only be adding Mirage endpoints and behavior as your Ember tests require it.
If you add a route to your API server, and no Ember code interacts with it, there should be no need to write an Ember test that consumes a mocked Mirage endpoint.
If you remove a route or change behavior in your API, it makes sense that you would want any affected Ember tests to immediately fail, but rewriting those Ember tests and Mirage endpoints to mirror the new behavior is just the cost of doing business.
It's more work in the long run, but I think your tests will be more meaningful if you treat the API and your mocked endpoints in Mirage as separate concerns. Your Ember tests shouldn't test whether your server works - they should only verify that your Ember code works given known constraints.
Perhaps one way to avoid omissions in your test suites is to enforce a (human) policy wherein any change to the API's behavior is specced out in a formal testing DSL like Gherkin, at least to document the requirements, and then use that as the single reference for writing the new tests and code that will implement the changes.
I'm trying to develop an automated testing suite for a REST API. I've been told that Swagger would be the right tool to use to do this.
The API is being developed using the Spring Framework.
I can't figure out how I would use Swagger to do this.
From my understanding Swagger is used to create nice API documentation.
I've looked at this Dreamfactory blog post and it looks like they've got some kind of in browser testing functionality, but I don't think its really a test suite.
I've also taken a look at ServiceStack( and cucumber) but since its another framework I don't think it would work.
This google group discussion didn't see to point anywhere in particular either.
So, does anyone know how to develop a RESTful API testbed using Swagger?
Thanks!
As the other answers have mentioned, Swagger provides a way to define and document your API endpoints, methods, responses, errors, and more. It does not do any sort of automated testing out of the box.
There are a few tools that can read a Swagger definition to created automated tests, though:
Assertible is a tool where you can import your Swagger spec to automatically create tests and assertions for every endpoint and method. You can then set up monitoring, post deployment testing, and alerts. (Read the blog Testing an API using Swagger). It has a free plan with options to upgrade for more tests.
swagger-test is an NPM package, if you're looking for something to integrate with your code. I haven't personally used it, but it does look active and useful.
Dredd is another really cool open-source tool that will automate testing your Swagger spec against a live backend. This is also a CLI too, and it works with API Blueprint in addition to Swagger.
There's others as well, because Swagger provides a good common language for API developers there's some great tools that are written on top of it.
Another answer mentioned to check out the Commercial Tools page on swagger.io, which has some more hosted services (free and paid).
Full disclosure - I'm one of the co-founders of Assertible and would love to hear your feedback if you get a chance to use it.
Swagger will let you nicely document your API, and will help you do manual, live tests with swagger-ui in that you can easily fill in arguments and see the responses through a web ui. Try the Swagger demo as an example of what I am referring to. http://petstore.swagger.io/
I have not tried this, but this might be of interest for more automated testing against Swagger definitions.
https://github.com/Maks3w/SwaggerAssertions
You can find a lot of software using Swagger specification for (automated) testing of your REST API on http://swagger.io/commercial-tools/. But they are not free.
Postman is great tool to test with Swagger apis. It can also store requests and responses. More details as follows
https://www.getpostman.com/docs/importing_swagger.
You might also want to look at the community tools according to languages at this page. http://swagger.io/open-source-integrations/
I have written an article on how you can generate all the test logic by using the Swagger or OpenAPI specifications. And then feed test data through excel sheets. That way your test data and test logic remain separated and in future if your Swagger spec changes then you will be able to even quickly re-sync the test logic.