Mobile QA Automation Tools – Robotium, Appium and Calabash – Comparative Study


Mobile sub banner

There are a lot of tools available for mobile application automated testing. In this blog post, we will try to compare among three of them – Robotium, Calabash, and Appium.  Let’s look at them individually and find out what are their capabilities concerning a set of parameters.

Robotium

Robotium supports both native and hybrid app through its Android test automation framework. It provides the test case developers an easier way to write functional, system and acceptance test scenarios, with minimal knowledge of the application under test in minimum time. The framework can handle multiple Android activities automatically and can integrate smoothly with Maven or Ant for continuous integration.  The test case execution is fast and more robust.

Advantages:

  • You can develop robust test cases, with minimal knowledge of the application under test.
  • The framework handles multiple Android activities automatically.
  • Minimal time needed to write solid test cases.
  • Readability of test cases is significantly improved, compared to standard instrumentation tests.
  • Blazing fast test case execution

Limitations

  • No support for flash & web applications
  • Test execution on one device at a time
  • No support for cross-platform testing

Appium

Appium provides an API based on Selenium’s WebDriver JSON wire protocol and supports platforms like Android, iOS, and FirefoxOS. Since the standard automation APIs work on all the platforms, there is no need to recompile the app. Also, we can use any WebDriver compatible language like PHP, Java, Ruby, Python, C#, etc. to write the tests.  It provides a true cross-platform mobile automation.

Advantages:

  • Appium is Open Source(Free)
  • Appium Supports many languages (Ruby, Python, Java, JavaScript, PHP, C#)
  • Appium supports automation of hybrid, native and mobile web applications.
  • Appium is Cross Platform.
  • Appium supports multiple platforms:
    • iOS
    • Android
    • Windows

Limitations:

  • Appium directly supports android version 17 and later, but is not compatible with Android versions older than 17.
  • Users cannot locate images with Appium. They had to work with screen coordinates.

Calabash

Calabash is a free open source project which can help you run tests on iOS, Android, and Simulator. It consists of two libraries – Calabash android and Calabash iOS, which helps in writing tests in domain specific languages. They enable test code to interact with the apps, wherein each of these interactions consists of some end-user actions like gestures, assertions or screenshots. You can even check the HTML 5 part of the application using the cucumber interface.

Advantages:

  • Calabash is available for Android and iOS platforms.
  • Calabash uses Gherkin Syntax (Cucumber style)
  • It is updated quite frequently.
  • It supports testing on the real devices.

Limitations:

  • Non-availability of an IDE or an Editor.
  • It supports only Ruby language.
  • You don’t get much online support.

Tool comparison:

Capture

How to pick the best Automation Tool

  • Identify the tests that need to be automated
  • Research and analyze the automation products that meet your automation needs
  • Do a pilot for two best tools as per your requirements and budget.
  • Based on the requirements, select the most suitable one or more tools
  • Discuss the chosen automation tools with other stakeholders, explain the choice, and get their approval
  • Proceed to test automation

Leave a Reply