Automate This

A big bucket of rambley nonsense about automating web testing

Okay, so we have been trying to grapple with FitNesse and Selenium to produce automated “acceptance” tests for our continuous integration development.

Before we brought ThoughtWorks in i had already been looking at automated test tools. There are lots of tools out there, some very simple (yet effective) and other very complicated.

Some of these tools profess to be web testing tools, while others claim to be able to test anything. Having said that any tool that requires me to purchase an add-on or plug-in to achieve automated testing is going to get shot down straight away. Testing is core to what i do I’m a tester, so far as i’m concerned testing is not an add on.

I thought i would get my thoughts on automation down, so should the wheels of the fabled bus ever get chance to put my mothers under-cracker warnings to the test my ramblings can be copy and pasted into someone else’ documentation.

During the usual testing activities we aim to undertake the following test activities:

1) Functionality Testing
2) Usability testing
3) GUIInterface testing
4) Browser Compatibility testing
5) Performance testing
6) Security testing

The list isn’t in any particular order and none is implied. But any seasoned tester will recognise that i’m talking about the upper right hand side of the V model, wheras FitNesse has traditionaly been used further down.

1) Functionality Testing:

It isn’t immediately obvious what functional testing means when talking about web testing. If you are sat in front of a static html page, that is merely a funnel into a web application you are fairly limited in the amount of functional testing you can do. All web pages can be functionally tested for working includes (css, images, etc.) testing all the links, forms, images,JavaScript and cookies. All the individual elements that are used to make up a web page. Lets take a more in depth look at this.

Link checking:

Once aspect of functional testing of web pages, checks links for broken or malformed types of:

  • Outgoing links
  • Internal links.
  • named anchors.
  • links used to send email (mailto).

If its possible (its not always possible to get access the the disc) test for orphaned objects. Note that i didn’t mention things like SEO within the context of functionality testing or links, because i see that as being non functional testing of a link. I also didn’t mention that the link was tagged appropriately for disabled users, because again this is covered later on. What should be covered is that the link is tagged appropriately for whichever analytic packages is being used by the website.

Test forms:
Forms were previously essential for submitting user information, but thanks to web 2.0 they are starting to disappear. Nevertheless, they still need to be tested for

  • input validations (where applied).
  • default values of fields (where set).
  • constraints (sting length, numerical, alpha etc).

I haven’t mentioned the error handling for the actual submission be cause that is the responsibility of the functionality within url the form points at, and not the form itself.

Testing Cookies, can be a right PITA. Thankfully firefox and its numerous wonderful plugins like web developer and firebug have made our life a lot easier. Cookie testing normally covers:

  • cookies enabled
  • cookies disabled
  • cookies deleted
  • cookies set
  • cookies read
  • cookies updated

We often visually inspect the cookie to make sure the developer hasn’t done something incredibly stupid like store the username and password in the cookie in plain text.

Database testing:
Data consistency is very important in web application. Check for data integrity and errors while you edit, delete, modify the forms or do any DB related functionality.
Check if all the database queries are executing correctly, data is retrieved correctly and also updated correctly. More on database testing could be load on DB, we will address this in web load or performance testing below.

2) Usability Testing:

Test for navigation:
Navigation means how the user surfs the web pages, different controls like buttons, boxes or how user using the links on the pages to surf different pages.
Usability testing includes:
Web site should be easy to use. Instructions should be provided clearly. Check if the provided instructions are correct means whether they satisfy purpose.
Main menu should be provided on each page. It should be consistent.

Content checking:
Content should be logical and easy to understand. Check for spelling errors. Use of dark colors annoys users and should not be used in site theme. You can follow some standards that are used for web page and content building. These are common accepted standards like as I mentioned above about annoying colours, fonts, frames etc.
Content should be meaningful. All the anchor text links should be working properly. Images should be placed properly with proper sizes.
These are some basic standards that should be followed in web development. Your task is to validate all for UI testing

Other user information for user help:
Like search option, sitemap, help files etc. Sitemap should be present with all the links in web sites with proper tree view of navigation. Check for all links on the sitemap.
“Search in the site” option will help users to find content pages they are looking for easily and quickly. These are all optional items and if present should be validated.

3) Interface Testing:
The main interfaces are:

Web server and application server interface
Application server and Database server interface.

Check if all the interactions between these servers are executed properly. Errors are handled properly. If database or web server returns any error message for any query by application server then application server should catch and display these error messages appropriately to users. Check what happens if user interrupts any transaction in-between? Check what happens if connection to web server is reset in between?

4) Compatibility Testing:
Compatibility of your web site is very important testing aspect. See which compatibility test to be executed:

  • Browser compatibility
  • Operating system compatibility
  • Mobile browsing
  • Printing options

Browser compatibility:
In my web-testing career I have experienced this as most influencing part on web site testing.
Some applications are very dependent on browsers. Different browsers have different configurations and settings that your web page should be compatible with. Your web site coding should be cross browser platform compatible. If you are using java scripts or AJAX calls for UI functionality, performing security checks or validations then give more stress on browser compatibility testing of your web application.
Test web application on different browsers like Internet explorer, Firefox, Netscape navigator, AOL, Safari, Opera browsers with different versions.

OS compatibility:
Some functionality in your web application is may not be compatible with all operating systems.
We Test our web application on different operating systems like Windows, MAC, Linux with different OS flavors. What can work in MSIE7 on Windows XP with service pack three, will fail onMSIE7 Windows Vista Home, but work on MSIE7 with Windows Vista Office Premium.

Mobile browsing:
This is a tough one. We get thousands of visitors a day using mobile devices, and even games consoles to view our site.

Printing options:
If you are giving page-printing options then make sure fonts, page alignment, page graphics getting printed properly. Pages should be fit to paper size or as per the size mentioned in printing option.

5) Performance testing:
Web application should sustain to heavy load. Web performance testing should include:
Web Load Testing
Web Stress Testing

Test application performance on different internet connection speed.
In web load testing test if many users are accessing or requesting the same page. Can system sustain in peak load times? Site should handle many simultaneous user requests, large input data from users, Simultaneous connection to DB, heavy load on specific pages etc.

Stress testing: Generally stress means stretching the system beyond its specification limits. Web stress testing is performed to break the site by giving stress and checked how system reacts to stress and how system recovers from crashes.
Stress is generally given on input fields, login and sign up areas.

In web performance testing web site functionality on different operating systems, different hardware platforms is checked for software, hardware memory leakage errors,

6) Security Testing:

Following are some test cases for web security testing:

  • Test by pasting internal url directly into browser address bar without login. Internal pages should not open.
  • If you are logged in using username and password and browsing internal pages then try changing url options directly. I.e. If you are checking some publisher site statistics with publisher site ID= 123. Try directly changing the url site ID parameter to different site ID which is not related to logged in user. Access should denied for this user to view others stats.
  • Try some invalid inputs in input fields like login username, password, input text boxes. Check the system reaction on all invalid inputs.
  • Web directories or files should not be accessible directly unless given download option.
  • Test the CAPTCHA for automates scripts logins.
  • Test if SSL is used for security measures. If used proper message should get displayed when user switch from non-secure http:// pages to secure https:// pages and vice versa.
  • All transactions, error messages, security breach attempts should get logged in log files somewhere on web server.

Please Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.