Next Generation Test Automation

March 31, 2009

In this post I would like to present the past evolution of test automation, and try to draw the next stage.

First Generation – Control

The first generation of test automation (at least as I remember, I’m not so old after all) focused on GUI applications and tried to solve the control problem: how can we control the application? How should the tool identify UI components?

Adding to it the ‘Record and Replay’ approach and we get a total mess. We can create tests very fast but maintenance is nightmare. In most cases we will not be able to execute them on the next version of our application.

For example, let’s say that we want to test some kind of authentication system with web management interface. Our test will include commands as following:





Or, if we want to test a networking device, the same approach would yield the following test:

connect(deviceHost, username, password);

cliCommand("router bgp 200");
cliCommand("neighbor remote-as 300");

As you can seem we end up with extremely long scripts.

It was very hard to understand the test logic from the scrip itself; it affected the scale of the project, because usually only the persons who built the script were able to debug and maintain it.

We can still see tools that have this exact focus.

Second Generation – Business logic

In the second generation, the focus moved from controlling the application to modeling the business logic of the System under Test (SUT). In this methodology, we start by building a management object with logical operations representing the SUT business logic and then we use these operations in our test. It is the management object responsibility to control the SUT.

Note that now we can talk about SUT in general, not just GUI applications, since the control is no longer in the hurt of the platform, and we can think of many ways to control an application, not just it’s GUI.

With our networking device, we will have a management object with operations like the following:



And the test will be:


router.bgp.addNeighbor(“”, 300);

Note that the tests are much shorter, and much more readable. The most importing advantage of this approach is the ability to overcome changes in the application. If the syntax of a command in the SUT changes you only need to change a single point in the management object, not hundreds of tests.

Additional advantage that I used many time is in using the object oriented capabilities. Let’s say you have two types of routers in your setup (or two version of the save router). You can build 2 implementation of the save interface, one for every router. The same test will be executed without any changes. It’s highly important in large scale projects.

In many cases business logic automation testing will do fine. If we want to improve maintainability even more, the following capabilities can be added:

  • SUT parameterization – configuration file that describes the SUT. Holds setup data like host name, user name, password …

  • Test parameterization – support for user input per test and scenario.

  • Setup configuration management – manage different setup states (configurations).

You can go very far with this kind of methodology.

The main disadvantage of this methodology is that it requires programming skills from all the people involved in the project. If the complexity of the system you are testing is not very high, you can use frameworks that hide the code to some degree (but never completely).

So what if you have a complex system to test but not all the members in the testing team have programming skills?

Model base

The future belongs to Model base testing. It enables to move the complexity from the tests to the model. The tests can stay very simple and in the same time test very complex environment.

Let’s say I would like to test routing environment. A test for multi router environment can be very complex.

The logical model can be built as follow:

  • You have a lab that contains routers

  • Each router contains network interfaces and has properties like name and host. It can have more than one implementation.

  • Network interface contains IP interfaces, stations and have properties like slot, port and peer.

  • Station can have Linux and Windows implementations and have properties like host.

JSystem, an automation framework that supports model base test automation.

First I build a simple tree representation of the model using the “SUT Planner”:

Now I can build tests by using generic business logic statements:

Build a loop over 2 models (STAR and MESH and maybe more).

  1. First step will take the model and implement it on the setup

  2. Second step will test ping connectivity from every station to all the others.

The nice thing about it is I can use the same test on new models I will build.


Model base testing is the future of test automation. It enable team work were the automation experts build the model capabilities and testers can use it to build models and flows. This allows the testers to focus on designing tests rather than writing software.

The main advantage is by using the same flows on deferent models.


Low Cost Test Automation with Open Source

February 3, 2009

I want to start with two quotes from Gartner:

“Companies are on survival path”.

“Open source will quietly take over”.

Well besides the fact that Gartner says is already something to mention, I see a vivid connection between these two statements. Basically one is a partial solution for the other. We all see the cut budgets for development and testing yet we are required to deliver quality products and make sure that we meet competition. During these times, when every penny counts, we seek for cheaper solutions that will provide the same results. Well seek no more; open source solutions and specially in the test automation area provide exactly what we were looking for.

Before we dig into the automation using open source tools, I think it’s only fair to give some background on how I see open source tools. So what is an open source tool? What are the advantages and what are the disadvantages?

“Open source is a development method for software that harnesses the power of distributed peer review and transparency of process. The promise of open source is better quality, higher reliability, more flexibility, lower cost, and an end to predatory vendor lock-in.”

The above definition kind of speaks for itself so I will try to list the advantages of open source tools as I see them:

Price – you don’t need to pay money for the actual open source tool. To be honest in some cases you will require professional help from someone that is familiar with the product or maybe even developed it, but the actual tool is free of charge.

Quality – surprisingly (to some), open source projects maintain very high level of testing, quality of the releases and of course a  large number of dedicated tester that report anything that they don’t like. Usually these projects are being developed by a devoted team of individuals that participate purely due to their passion for the project. Not everyone can start contributing to the project and standards are very high. Don’t forget that open source means true transparency of the process, what you do is being evaluated by many others and this is the best motivator for delivering of high quality results.

Up to Date – open source projects greatly influenced from what people need. Basically once a project is released it’s being evaluated for usability, relevancy and quality all the time. If there are features that are missing or not usable, be sure that someone there will point it out. These inputs usually are taken into consideration by project developers not mentioning the fact that if something bothers you, one of the option is just adding it to the project.

Community – open source projects are all about the people involved in the project. You don’t have to be the developer to be involved. People can contribute by testing, documenting, helping in the technical forums, etc. Having a large community just reduces the risk of using the project – you can always find someone that already solved your problem.

So let’s say that you have decided to use an open source automation tool, but you don’t know how to decide upon which tool to use. Here are some tips that might help you.

Professional services – make sure that there is someone that provides professional services for this product. It’s true that in most cases you will find help in the community but when you are talking about tight deadlines, specific customization for you product you might need experts.

Companies behind the project – in many cases the initiator or the supporter of an open source project is a big commercial company. This sometimes means that the product is mature and stable or at least had a good basis for future development.

Community – I have talked already about this before. Just make sure that the community is active and large enough to get answers when you will need them.

Roadmap – it’s important to see that the tool that you are choosing going to directions that you might benefit from. If there is no roadmap whatsoever it’s not a good indication.

Needless to say that considerations that are valid for test automation open source tool are also true for any other open source tool that you are considering using.

In case you were wondering which test automation open source tools are available, here are a couple of names that you should know:

xUnit – one of the most popular tools for unit testing. Such frameworks are based on a design by Kent Beck and are part of the extreme programming methodology. We can find implementation of this approach for almost every language: JUnit (, NUnit (, etc.

JMeter – long running project that allows testers and developers creation of load tests. This tool support variety of protocol and is easy to use (

Selenium – one of the most popular tools for Web testing. This tool supports most of the currently used browsers, runs on different platforms and allows integration with almost any framework that you have (

JSystem – this is a complete testing environment that allows use development, execution and management of tests. The framework is Java based and contains many advanced feature and integration with tools like Selenium, Soap, SNMP and many others (

SoapUI – this is a very common tool that allows invoking, inspecting and developing web services. This tool can be easily integrated into load and functional testing framework (

I hope that after this short review you will be more open to adopting open source test automation tools in your organizations.

Web Test Automation

February 3, 2009

Web testing is not a new interface that requires automation; however lately is creates new exciting challenges for us. The main reason for these changes is the transition to Web 2.0 from the traditional Web 1.0. I think it’s only fair that we list a couple of differences between the two before we continue:

Web 1.0

Web 2.0

Static content Dynamic content
Company oriented Community oriented
Web forms Web applications
Explorer Firefox, Opera

The main difference for my opinion is the fact that Web 2.0 provide user richer experience. Basically Web 2.0 uses the browser as a container for rich GUI applications with different orientations like community, blogs, etc.

So after understanding what we are up against, let’s understand the new challenges better. In the old days with plain HTML that ruled the Web, tools like QTP, RFT and others, provided recording ability allowing test engineers to record and replay. This fast method usually provided the required test coverage. The recording was performed using Windows object identification method and was done inside the Internet Explorer only. This mechanism relays on hooking to objects and recording its states when a change occurs.

So why recording is not enough today? You might ask. Well the main problem is that not only that you have to support multiple number of browsers like Firefox and Opera (which are not supported by the recording mechanism), but also that recording doesn’t work when a AJAX operations is being performed in the browser.

In addition to the fact that browser environment has changed, we have to keep in mind that Web testing is not only about browsers, it’s about whole Web environment. To emphasize this let’s examine a typical Web environment.


We can see that usually besides Web browser we have a Web server of different kinds, we have some kind of a database and all these have to be controlled via same framework.

So if this is the typical Web environment then a typical flow would be something like this:

  1. Modify something through the browser
  2. Verify that the change took place in a database
  3. Verify no errors in the Web server log
  4. Generate some load and made sure that the Web server continues to run smoothly

Of course this is just a simplified example, but you get the picture…

So now let’s choose a set of tools that will be sufficient for testing a whole Web environment. We will start by finding a tool that solves a specific interface issue.

  1. Web interface should allow controlling different types of browsers and allow user operations emulation. For that interface we will be using Selenium. This is an open source tool that not only supports different browsers (Explorer, Firefox, Opera, Safari) but runs on most of the platforms (Windows, Linux, Solaris). It has a set of cool, advanced features including a recording IDE (via Firefox plugin) which results you can export to most of the used languages (Java, Python, Ruby, PHP, Perl).
  2. Database interface should allow controlling seamlessly any type of database without changing the actual test. For that role we would like to use the JDBC (java standard interface) that is supported by the  JSystem Test Automation Framework.
  3. Web server interface should allow operations like log manipulations (read, analyze), configuration modification, process analysis, etc. This can be done by connecting to the Web server machine using telnet connection. For that role we would like to use the JSystem Test Automation Framework.
  4. Web server load interface should allow generation of different types of load traffic like HTTP or maybe SOAP. For that role we would like to use the JMeter.

Now we need to understand that we need to combine these tools together using a framework. In this example the framework that will control the whole environment will be JSystem Test Automation Framework, allowing easy integration of tools like Selenium and JMeter. The framework also provide out of the box reporting system, advanced analysis tool and many other features that you can learn about by visiting their Web site.

So after defining which tools we will use let’s see how our Web environment will now look like:


Now you are ready to test your Web environment. Please notice that everything was done using open source tools! What a bonus!

Integration & Automation

January 28, 2009

”Integration is a ‘pay now or pay more later’ kind of activity. That is, if you integrate throughout the project in small amounts you will not find yourself trying to integrate the system for weeks at the project’s end while the deadline slips by” (Extreme Programming, Don Wells)

The Integration Hell

An automation project takes a lot of integration. By Nature, the automation process integrates different platforms and MUST run perfectly in an unattended manner where no human logic can be applied.

Running the test automation scenario on your desktop set up is not enough. Apart from developing a quality code, system implications must be considered. In simple words, you want the whole thing to run smoothly, in an unattended manner, with 100% reliability on the main set up.

The main set up (A.K.A “the BIG set up” or “the Regression set up”) is the closest to real life simulation the SUT (System Under Test) gets. The main set up is where all the devices and platforms are working together. It may combine different Operation systems, Testing Equipment, Simulators, and of course the SUT/s.

Leaving integration only to the final project stages of our test automation project, might lead us to what Xtreme programmers call “The Integration Hell” where everything brakes and no one understands why…

Continuous integration

So, what can be done to avoid this integration hell?

Stay synchronized

When developing code integrate it frequently. This can be done by releasing the code (Check IN) into the code repository whenever possible. If everyone will work with the latest version, the required code integration will be minimal.

Re-Run your test until it runs smoothly

You should frequently run the automatic test on your dedicated development setup, until it runs smoothly. Don’t leave it with errors or warnings.

Assemble tests into scenarios

After you’ve completed a single test, integrate it with other tests. That is, add it to its relevant scenario. This way you’ll be able to inspect Inter-test relationship and minimize the surprises.

Assemble multiple scenarios

When you have more than a single scenario (which includes 1 or more tests) combine it with another scenario. That will give you the opportunity to check their inter-scenario relationship.

Start Overnight Looping ASAP

You should start “looping” overnight scenarios ASAP. Start even with a single test, than add the second one and so on… That will save you a lot of time when you’ll reach the final project stage. It can also produce usable results from early project stages.

Dedicate integration time to each phase

While planning ahead, always keep integration in mind. Each and every phase should have a dedicated integration time slot.

Remember the complexity of the final Integration

Toward the end of the project there is the final integration stage.It is a special stage, all systems must work together, and anything less than perfect is not acceptable. Things that we have over looked or postponed must have an answer now. Everything, even the smallest details that we’ve dismissed few weeks ago, must be resolved. All scenarios must run smoothly on the main set up, unpredictable things will happen and “Murphy” will come and visit. Keep that in mind and plan your Gantt so you’ll have enough time and resources for this stage.

Burying in mind that test automation projects are very sensitive to the quality of Integration, and following the above ‘golden rules’ of continuous Integration will minimize the risk of not meeting your project deadline, it will also help you avoid the end-of- project integration hell where everything might brake.

Integrating Java and COM

January 20, 2009

The main challenge I face was to integrate Java with COM objects. I tried few java2com bridges but I was not satisfied with the solution. It was too completed and tend to break in the eages.

I decided to use vbscript. It is very simple to work with and it (obviously) has very good COM integration. But I should use vbscript as an interpreter. I need the flexibility to build the vbscript dynamically as the Java code is executed.

The first step was to build a vbscript interpreter. Following is the code I used:

do while true

set ln = Nothing
ln = wscript.stdin.readline
if lcase(trim(ln)) = “exit” then exit do
on error resume next
wscript.echo “stdout: <”
execute ln
wscript.echo “>” & vbCrLf
wscript.echo “errorCode: <”
wscript.echo err.number
wscript.echo “>” & vbCrLf
wscript.echo “errorInfo: <”
wscript.echo err.description
wscript.echo “>” & vbCrLf
wscript.stdout.write(“vbscript# “)


Now the problem is in new domain I can use java exec to execute the following command:

“cmd.exe”, “/C”, “cscript.exe”, “main.vbs”

Now it work like shell application. I can execute vbscript command one by one.

For every command I get the standard output, the error code and the error description, so I can use my java code to analyze it. and pass it to upper layers as exception.

In this link you can find the full java utility file

Following is a java code example:

public static void main(String[] args){
 VbShell vb = new VbShell(new File(System.getProperty("user.dir")));
 try {
 ShellCommand sc = new ShellCommand("Set Word = CreateObject(\"Word.Application\")", null);
 sc = new ShellCommand("Word.Visible = True", null);
 sc = new ShellCommand("Word.Documents.Add", null);
 sc = new ShellCommand("Word.Quit", null);
 } catch (Exception e) {
 // TODO Auto-generated catch block e.printStackTrace();