Posts Tagged Testing

WebDriver (Selenium 2.0): First Impressions

I’ve been using Selenium and Watij for a while now (more Watij than Selenium). I’ve written about my prior experiences here and here. Watij — there is no significant development happening there for the last year or two. While Selenium Remote Control was nice, its IE support (especially IE 7) was not that great when I last evaluated (it may have improved since then).

I guess on Twitter, I first heard about WebDriver, and accolades about its approach and its API design. So decided to give it a try. Best part is Selenium and WebDriver projects are joining forces with a merger in Selenium 2.0. Per my understanding, the plan is to provide both Selenium and WebDriver APIs under Selenium 2.0 umbrella. At the time of writing this post Selenium 2.0a1 (Alpha-release) was out.

So here are my first impressions:

  • Approach: At the outset WebDriver is similar to Selenium (Remote Control) and Watij — developer-focused and API-driven. But the best part, for me at least, WebDriver doesn’t take the approach of running as a Javascript application within the browser (like Selenium), rather it takes JNA approach interacting with the browser. That approach rules out same origin issue (a headache with Javascript approach).  Also this doesn’t require native libraries installed and registered on the client (e.g: DLLs need to installed on the client for Watij).
  • API: API is intuitive. For example, following code snippet finds the element (text box, in this case) by name and simulate the typing by the sendKeys() method.
    driver.findElement(By.name("full_name")).sendKeys("Joe Schmo");
    

    Similarly,

    driver.findElement(By.name("Details")).click();
    

    finds the element (button, in this case) and clicks the button.

  • Multiple drivers: Unlike Watij, which only supports IE at this time, WebDriver supports — IE, Firefox, Chrome, and it also has a HtmlUnit version to run headless.
  • Window/Frame handling: This is yet another area where WebDriver excels with a caveat (that I’ll get into in a moment). When an operation like a button-click or a hyperlink-click opens a new pop-up browser, WebDriver provides an excellent way to get a handle to the new browser. Following call returns a set of all the window Ids.
    driver.getWindowHandles()
    

    You can iterate over the Set to find which one is your new window and switch to that window for all future operations.

    driver.switchTo().window(windowName)
    

    Windows’ handling is quite solid compared to handling of the Frames. In most of the situations it worked except when there are iFrames which load dynamically the behavior is not that consistent.

  • Wait behavior: When a particular operation (like a button-click) is underway WebDriver waits for the operation to finish before it executes the next step of the test case. Although this sounds trivial, I’ve had issues with this while using other tools. In those cases I had to use Thread.sleep() to wait between the page load and the execution of the subsequent step.
  • Alert Dialog: Couldn’t find a way yet to handle alert dialogs. I have to dig a bit more into the API and forums to see if they are supported at this time.
  • Mouse Events: I’d like to see more mouse events — double click, hover over an element, page scrolling, right click, etc.

Are you using WebDriver? What are your thoughts?

driver.findElement(By.name("q"));

Tags: , ,

Joel vs UncleBob: Relevance of TDD and SOLID principles

This is really a clash of two titans, both of whom I greatly admire — Joel Spolsky (Joel on software) and Robert Martin (uncle Bob). I follow both of them for many years, their writings, on Twitter, podcasts and so on. Uncle Bob was not so happy about the discussion on Stackoverflow podcast #38 and he responded in kind.

My thoughts:

  • Unit testing is extremely important. Every minute that a developer spends writing a testing is well worth hundreds of minutes he would spend later in the maintenance phase. Having said that, if the concern is whether 100% code coverage is too much? Perhaps it is, determine what gives best bang for your buck and go with that. If the question is whether they are useful at all? Absolutely. Taking the same reason as cited in the podcast — you have a large code base and now you need to either modify or add new features. Once you make the change:
    • how do you know you are done? I run the unit tests that I wrote for the new functionality and pass them.
    • more importantly, how do I know whether I broke any existing functionality? I run all the tests that I have (assuming I have decent code coverage) and make sure all the tests are passed.
    • unit tests along with the acceptance tests, if run more often, can provide extremely useful information about any deviations from the requirements. Rapid feedback, isn’t it one of the core principles of agile?
  • You don’t have to follow TDD to the tee, or any methodology for that matter. But the spirit of unit testing, in my opinion, is extremely important to know the health of your code.
  • SOLID principles, as Uncle Bob said, are engineering principles. These principles guide you in building the software that is flexible, maintainable and reusable. This is even more important in agile environments where you develop software in tiny increments.
  • Enough emphasis must be given to code hygiene. With out that, it is just a matter of time that the system gets polluted, and not sure how you can keep the customer happy for long. I agree that the focus should be on the customer and just don’t architect or refactor just for the sake of it. Focusing on the customer means — designing flexible, extensible architecture (using design principles) so that you spend less time in constant-maintenance mode.
  • Another common argument against writing tests is time to market. Goal is to get the software out as quickly into the market as possible, fair enough, but at what cost? As software craftsmen we need to pledge for not shipping shit (must read).

I repeat, I have great regard and respect to both the individuals, and hope Joel can elaborate on some of the points that are discussed in his future podcasts and clear the air.

Tags: , , ,

Root Cause Analysis: Found a Bug, Now What?

A bug is identified, now what? Fix it. Of course that is an obvious answer. This post is intended towards a better approach of addressing a bug with long-term gains in mind. Addressing a bug is a given thing, but what you do as a follow up act is important. A bug can be reported at any stage of the development process, we will consider one that is reported after releasing the product to the client.

Assessing the root cause of the issue is the key. The metrics available from this exercise are so valuable that the time spent is an extremely valuable investment that an Organization can make. Wikipedia defines Root cause analysis as:

Root cause analysis (RCA) is a class of problem solving methods aimed at identifying the root causes of problems or events. The practice of RCA is predicated on the belief that problems are best solved by attempting to correct or eliminate root causes, as opposed to merely addressing the immediately obvious symptoms. By directing corrective measures at root causes, it is hoped that the likelihood of problem recurrence will be minimized.

How can we apply this concept to Software development? Extend the life-cycle of a bug. Once a bug is fixed don’t just close it in the tracking system. Move it to something like a Root cause analysis stage. Only after completing the analysis close the bug. Okay, so what consists of analysis? It actually depends on what phase of SDLC the bug is more associated with. More often than not a bug is associated with more than one phase — like an issue that is missed during design/implementation phase and also missed during the testing phase.

Requirements Phase

In general, a bug or defect is something that is not compliant to the agreed upon requirements. So if an issue is reported and the requirements explain the behavior to be something different, user-misunderstanding of the requirements could be the issue.

  • User misunderstanding: This is relatively easy one, but still requires some action from the teams involved. Leadership may have to verify if the user training material needs any update based on this experience.

Design/Implementation Phase

Whether developers or their managers like it or not, in many Organizations more “blame” is attributed to them if a bug arises. That is debatable, but what is more important is to pro-actively address as many issues as possible in the given time frame. They need to make good use of the tools available at their disposal. Perform static analysis, code reviews, maintain good code coverage with unit tests (More on each of these in separate posts). One most important aspect, unfortunately not given utmost priority, is automated functional/integration tests. There are many tools available, including open-source free options that can be used. The tools being too expensive to afford is no more an excuse. My thoughts on some of my experiences evaluating these tools are published here.

Coming back to RCA, when a defect surfaces the teams must identify which areas can further be strengthened. For example,

  • Code Review: Perhaps doing better code reviews could have addressed this issue long time ago. Static analysis can be automated with tools like PMD, FindBugs, etc. If you already have these processes in place then add an additional rule in PMD (or similar tool) related with the issue in question.
  • Unit Test: If you determine that a JUnit (or some XUnit framework) test case can better track the issue — the next action is to write a unit test and add it to your test suite. The assumption is that you run the unit tests more often in the spirit of Continuous Integration. If you already have a unit test that did not catch the issue, fix it.
  • Functional Tests (automated): With the advent of open-source tools developers can do themselves a favor by writing an automated test or ten. If they get into that mode, tests can be written considerably faster and saves significant time later on in the SDLC. The concept is very similar, if you determine that an automated functional test could have identified the issue sooner, by all means add one.

Testing Phase

Ideally, testing group should be the first line of defense when a bug arises. An introspection from the testers on how they could have identified the issue (during the testing phase) would certainly help.

  • Regression Suite: Regardless of the processes followed testing teams usually will have a set of test cases and test plans that they use to certify a build is good for release. When a defect is reported, regression suite is what you would examine first and make any necessary updates.
  • Acceptance tests: Ideal to automate as explained above, and perform the similar actions as described above. All that it requires from the leadership is to identify if this is the area that has to be strengthend for achieving maximum results.

Is it a Reactive Approach?

It appears that the approach is more reactive in nature, you are performing certain actions only after a bug is reported. Wouldn’t you like to find the issues sooner than later. An analysis of this sort, if done correctly, can provide you with certain patterns, trends and metrics. For example, while adding a test scenario you may identify a related scenario or another variation of the issue, and you address that by adding additional tests or adding another unit test. You may also identify a particular pattern emerging, say more issues are being reported in a specific module or certain kinds of design issues are emerging more often.

As explained in Wikipedia:

RCA, initially is a reactive method of problem solving. This means that the analysis is done after an event has occurred. By gaining expertise in RCA it becomes a pro-active method. This means that RCA is able to forecast the possibility of an event even before it could occur.

It would be intersting to know how many Organizations take this kind of long-term approach in their development process. The emphasis should be more on improving the processes than to “scapegoating” individuals by taking short-term band-aid approaches.

Tags: ,

Test Automation Revisited

More than an year ago I published my experiences with Selenium and Watij. At that time I was actually not sure whether Selenium and Watij were ready for prime usage, especially for testing on IE7 (a requirement for my app). So fast forward to now, I’m lot more satisfied with what I’m seeing ..

Both the tools have got more recent versions out compared to what I used earlier. Started to work on Selenium RC. I was glad a lot of stuff that was a bit shaky in my earlier experience is more polished now. Just when I was thinking that I’m proceeding quite well with Selenium RC, hit with an issue where Selenium loses handle to the main window if the (child) popup window posts data and the main window refreshes based on that post. Logged a critical issue at Selenium’s Jira, still awaiting for some action on that (I wish I had some time to look into that and provide a patch, if required).

After that, started to look for alternatives — having used Watij earlier, prompted me to give it another try. I was pleasantly surprised by the ease at which I could proceed this time, including the issues that I was having earlier with modal dialogs. Here are some of the comparisons:

  • Watij’s API is more Object-oriented than that of Selenium’s, which is perhaps more functional in nature.
  • While using Selenium I had to use waitForXXX calls to wait till the page gets loaded, or for the popup to get loaded, etc. But with Watij the tool is “smart” enough to figure when the page load completed so as to proceed with the subsequent calls. Having said that there are still some areas where some sort of wait is required, but I believe that is based on the application functionality that is under test.
  • Selenium injects Javascript into the pages of application under test. It also requires Selenium server running (only proxyInjection mode worked for me). Watij on the other hand doesn’t need a server. It requires a dll (yikes!, but that’s not that bad) to be placed in your System32 folder of Windows. I found Watij to be a bit more robust and found that behavior was consistent when compared with Selenium.
  • A drawback for Watij is it works only for IE. That works fine for me as that is our primary requirement. If you have to test under different browsers than Selenium is perhaps the way to go.
  • With Selenium IDE, the tests can be recorded using a Firefox plugin. There is no such recording capability with Watij.
  • Watij doesn’t have a Selenese type of “language” to write the test cases outside of using a programming language (like Java). Needless to say that this kind of notation would be useful for non-technical people. Having said that it is not that difficult to write your own version of Selenese kind of notation.

With all those observations, Watij was what I chose to proceed with. Combination of the robustness of Watij and the critical issue that I was hit with while using Selenium (described above) made me to move in the direction of Watij.

Tags: ,

Watij and Selenium: Are they ready for prime usage?

Note: Published an update here.

I have been looking out for some good open open source automated functional testing tools. After some bit of research and based on some recommendations I went on to try Watij and Selenium.

Watij:
Based on popular ruby based Watir, I was quite impressed by its API. It has got good little documentation on its site and on wiki, just enough for you get started and do some reasonable testing. You can write the tests using the popular JUnit framework, which is a big plus.

It all worked fine, in fact, I could automate quite a few screens in an application until I hit a roadblock :( Whenever it encounters modal dialog windows it just stops working. Some how it seems not recognize them, and waits for the screen to load even though it has already been loaded.

Wondering how other people working with Watij working around this limitation. It has been a while since they made a release after 3.1.0. I hope the support for modal dialog windows would be functional in the next release.

Selenium:
After that experience with Watij, I started looking at Selenium. For some reason, I did not have that much luck, not even as much compared with Watij.

After starting up the Selenium server and try to hit a web page I’ve received a 404 error on page. Based on the documentation on the site, I have started using *iehta instead of *iexplore and the 404 went away. The joy was short lived as I started getting windows dialog messages saying a “script is trying to run, do you want run it?” (not the exact verbage). I had to keep clicking Yes on it to move forward.

However, with bit more researching I found a way to resolve that. This time started the server in the mutiwindow mode (with -multiWindow option) the application started running on a different window than that of the Selenium browser.

That is as far as I can go with this at this point. As I start with a page on the application I see dozens of browser windows started popping for the same page. Each of this page shows up a confirmation popup saying that the web site you’re on is trying to close the window, do you want to close the window? Not sure why Selenium wants to close the browser as soon as it opens this very first page! This was working some time ago when I was on IE 6, now that I’m on IE 7 I find this issue.

Conclusion:
All in all, I found more success with Watij than Selenium (on IE 7). I’m sure there are many people out there using both these APIs. I see there are many open source projects using Selenium. I’d be interested in knowing what they think about these two APIs. Which one they liked better and why? Are they ready for prime usage?

Tags: , , ,