Tuesday, December 12, 2006

Redefining Exploratory Testing

Can an exploratory test become scripted? Is there really such a thing as a truly exploratory test? Or is it in fact just a scaled-down scripted test? With Exploratory tests becoming increasingly scripted in nature, I feel this is the right time to redefine the meaning of exploratory test and draw a line between exploratory and scripted test.

As any reference book or present-day tester would claim, the Methodology of modern software testing could then be divided into mainly two dominant categories: Exploratory and Scripted Testing. But what then is the difference between these two methodologies? After all, they share the same purpose: to find bugs and check functionality against requirements. Isn't it?

Some might suggest that the main difference is that Scripted testing is much more well-documented than Exploratory testing and thus weighs heavier as a formal proof of the code having been thoroughly tested. Others might say that it's really about automation. With Scripted tests, the code could easily be performance tested through automated tests that runs thousands of iterations and logs the program's every behaviour. Yet another group might claim that the Exploratory test approach applies solely to GUI-testing and should not be used for anything else. According to
James Bach, the plainest definition of exploratory testing is test design and test execution at the same time.

Let's take a closer look
Obviously, there is a great deal of confusion about both the structure(1) and the purpose(2) of Exploratory testing as well as when it should be conducted(3). A further examination of these three issues is therefore warranted:

1. The Structure of Exploratory Testing.
Unless the program to be tested is very small, most Exploratory tests have a written template, a guide to follow when performing the test. Alternatively the Tester is extremely familiar with the product and can go through its different functions from the top of their head. This is not a recommendable situation though as there is no contingency plan for the company if the tester suddenly becomes unavailable. Also; we're only human. On a typical Monday morning, anyone could forget an essential part of the routine even if it has been performed a trillion times before.

So let's assume that the Exploratory test is in fact documented and that copies of that document are being used every time the product is tested. In that case it is also likely that the tester, after completion of the tests, signs off his task with a reference to that very same document. The document hence becomes formally connected with the performed test and its certificate of adequate test coverage and robustness. Doesn't that sound awfully familiar to scripted testing?

2. Purpose of Exploratory Testing.
The purpose of any testing (including Exploratory testing) is to find bugs and check functionality against requirements. But if we again try to scrutinize the said difference between the two testing paradigms mentioned above, we'll find that Exploratory testing mainly focuses of finding bugs that would appear when the program is used correctly, that is; according to the specifications. If, for example, a certain function requires the in-values of: "$100 to 1000" and a specific time-duration expressed in "full weeks" to calculate a ROI, then the Exploratory test would try the function with exactly these figures, whereas a Scripted test would also try to run the function with maybe "-£500" and "44.7 years" as typical in-values to see how the software behaved with anomalies.

However, even though the scope of the two methodologies could be different, a scripted test could easily be written to cover exactly the same functions, with the same depth, as any typical Exploratory test. All you really have to do is to use the same type of user-behaviour and in-data as stipulated in the requirements. Again, a blurring of the definition.

3. When Exploratory testing should be conducted.
After having just received the first Alpha version of a new product, a stressed Test Manager might say: "Give it a quick try with a Exploratory test now and then again just before we release", meaning that everything else in between should be Scripted testing. This strategy might be reasonable since there often is a need of quickly showing the developers what faulting functions needs to be addressed first after the initial release. There is of course also the necessity of once again going over the whole system just prior to the final release to make sure that all the latest bugfixes hasn't caused new ones. Exploratory testing, being rather cost-effective and rapid, might seem a good option in those two cases.

However, with modern software testing becoming introduced on all levels of the development cycle, the initial "quick look" should not be necessary. The test team should have had enough time to write proper test cases and suites while the program was being coded had they already been present at the planning stages. A final overhaul of the whole system could perhaps be more justifiable, especially if there is very little time left before the deadline. However, this would probably prove to be unsatisfactory in regards of quality, since if there is anything that the program should really be tested for at that stage, it should be its robustness and performance, not just basic functionality.

Redefining Exploratory Testing
With these few exceptions accounted for, I feel that it is safe to say that Exploratory testing is becoming obsolete or at least less important. While many might want to defend the role of Exploratory testing as being an important part of their everyday work, the very same people would probably agree that it is their team’s overall goal to perform as much Scripted testing as possible. Likewise would most project managers like to see their products being formally tested on all levels before release, which again makes fumbling Exploratory tests a waste of time.

I would therefore suggest a re-definition of Exploratory testing. It should be an activity reserved for the two situations described above and for the case where a tester, who has been newly assigned to the test work, needs to get a quick understanding of how the product is meant to work. Exploratory testing, in the old meaning of the word, should be incorporated into what we normally refer to as Scripted testing. Perhaps on a very rudimentary level, at least in comparison with what Scripted tests normally do, but still in the same context as Scripted tests in terms of repeatability, formal reference and thoroughness. This is the way we are already heading, we now only have to also officially recognize it.
Any Comments ?

Friday, December 08, 2006

Top Ten Threats For 2007 - From McAfee

It's near the end of 2006, so lists will be a big part of the stories you read on virtually any topic from now through the New Year. McAfee has a list for the security-conscious. The list covers what their Avert Labs data indicated will be the top 10 security threats next year, presented in no particular order: \

1. The number of password-stealing Web sites will increase using fake sign-in pages for popular online services such as eBay
2. The volume of spam, particularly bandwidth-eating image spam, will continue to increase
3. The popularity of video sharing on the Web makes it inevitable that hackers will target MPEG files as a means to distribute malicious code
4. Mobile phone attacks will become more prevalent as mobile devices become "smarter" and more connected
5. Adware will go mainstream following the increase in commercial Potentially Unwanted Programs (PUPs)
6. Identity theft and data loss will continue to be a public issue - at the root of these crimes is often computer theft, loss of back-ups and compromised information systems
7. The use of bots, computer programs that perform automated tasks, will increase as a tool favored by hackers
8. Parasitic malware, or viruses that modify existing files on a disk, will make a comeback
9. The number of rootkits on 32-bit platforms will increase, but protection and remediation capabilities will increase as well
10. Vulnerabilities will continue to cause concern fueled by the underground market for vulnerabilities

The greater concern that should make these threats more of a priority for law enforcement around the globe is how so many of them will be perpetrated by criminals. With potential profits available through exploiting technology and the people who use it, the more organized and sophisticated criminal groups will continue to seek ways to garner those gains. McAfee said its researchers have seen "evidence of the rise of professional and organized crime in malware creation.":

(D)evelopment teams are creating malicious software, testing it and automating its production and release. Sophisticated techniques such as polymorphism, the recurrence of parasitic infectors, rootkits, and automated systems with cycling encryption releasing new builds are becoming more prevalent. Furthermore, threats are being packed or encrypted to disguise their malicious purpose on a more rapid and complex scale.

Threats to videos and mobiles could be a combined problem. The popular video sharing site YouTube recently inked a deal with Verizon Wireless that will make videos available to V Cast subscribers. If video on mobile finally starts to gain a greater userbase, that will make it a target for attackers.

Thursday, December 07, 2006

Getting started with Software Testing

Suvasis asks, "I am trying to get into software testing. But I never took a course on testing. What would u recommend for a beginner???"

How many times do you come across such question? My approach to such kind of question would be to follow three golden rules of learning !

1. Read it.
2. Practice it.
3. Build Network.
Unfortunately testing is not something which might be taught at most schools. So, undoubtedly most things in testing are new for you as a fresher. So start with reading a lot. You can start with good testing books, online articles, testing magazines, online forums and the list continues. There are many good books in the market. Some of the best sites to refer to would be
Reading through their articles would introduce one to the world of testing. If you have extra cash on you, attending a seminar/training sessions might be useful (I say might be since I have no on hand experience of attending one of these). I believe what could be taught could be very well learnt by you. Another major source (and a free one) is that of blogs. There are many testers blogging out there. One person I blindly follow is Michael Hunter (http://blogs.msdn.com/micahel). Just reading his blog should get you hooked onto testing. So yes, you need to start reading.
ut all books is not going to get you anywhere, unless you practice it. If your day job does not support testing, make it your night job. If you write code, test it more sincerely. One practice I used to follow was trying to break my friend’s code for some project while he tries to break mine. Working in pairs could help. Try getting a partner. Not only does it make your code robust during submissions, but it also gets you better grades if you are still in your final years.
Look up Mozilla (www.mozilla.org). Participate in the open source community. You could download nightly builds from Mozilla and try to break them. Want to know what a test case or a bug report would look like? You'll get those on Mozilla. Did you know Microsoft Vista Betas were being shipped free? Get one of those and try to probe into it. The more inquisitive you are about testing, the more questions you will have. And more the questions, more you will learn. Sounds positive?
Sign up as Beta tester. Microsoft has a Beta testing program. Acronys has a Beta testing program. Try to look up some test automation tools. Dig into Junit and the likes.
Lastly, network around. Participate in forum discussions, write articles, and offer suggestions. Not everything you do might turn up being right. But if you aint getting your fingers slapped once in a while, you aint pushing the boundaries far enough. If nothing else write a blog.
Start working on a pet project, test it and upload it to one of the communities. People will download it, run it, and test it for you. Vice-versa! Every time they report a bug, you know where your testing+coding skills need to be improvised. Pick project ideas off Codeproject, Planetsourcecode etc. Buddy up with someone and build it, break it. Break it to build it. Try to automate your tests. Every time you wonder how you could get a certain aspect of it done, look up Google, ask people.And of course, if you land up with a Software Testing job, it just makes life simple.
Reading up, implementing it and asking around are what helped me. There aint any formula to getting started. Just get started with one aspect of testing, it leads to others. How wide you spread your arms depends on your interest.Quick advice before you venture into any coding. Write a quick Spec. Getting a clear spec is the first step towards being a good tester. Not only will it help you design, it will help you catch bugs before you start coding. So get started into testing and get going.