Software Testing Social Network

The theory and promise of a 24/7 Web site are always talked about, the reality for many company Web sites is that they need to be taken offline (at least occasionally) to install new releases or upgrade software or hardware. Occasionally you need to take your production site offline for an hour or two during the night to install a new release. A variety of people are involved with these releases and the pressure is high—everybody takes their turn working through their piece of the process, knowing that offline time means lost revenue and reduced customer service. In this race against the clock, quality assurance is designated the last position—the final checkpoint before the site is open to the public again.

In such a scenario a quality assurance person brings to the table a critical career skill that has nothing to do with the latest or coolest technology: the skill of planning. It’s important to be organized in a profession that often senses other people (such as the rest of the technical and management team) pacing and waiting for QA to finish their job. Quality assurance needs to be clear with the rest of the team about what needs to be done and the risks we take when corners are cut because we are short on staff or short on time.

When the magic hour approaches, the network staff begins by trying to close the site as gently as possible for customers—waiting for people to sign off and hoping more people aren’t logging in at three o’clock in the morning. Once the customer count is as low as possible,  a maintenance page is posted stating that our site will be offline temporarily, and an email will be sent when the site is available again. A connection outside of our firewall is used to verify that this maintenance page is what is being shown to the "outside world by the quality assurance guy." He enters his email address to receive notification when the site is live again. Then, the network staff makes any hardware or software upgrades as needed, followed by configuration changes.

Meanwhile, our database administrators (DBAs) begin making modifications to the production databases. Database tables may be added, changed, or deleted. Sometimes these database changes can only happen when the DBAs can gain exclusive access to a table. The only database changes that are made are those agreed upon in advance by development, quality assurance, and the DBAs. These database changes are well documented and executed according to the agreed-upon plan.

The network staff continues by installing the new software. They follow release notes that were prepared by quality assurance in advance. The developers review configuration settings, ensure content is moved to production, and check that the correct files are in place. Once the network, DBA, and developers are done, quality assurance steps in. QA ensures that the release is ready, and that the site is ready to go live again.

What does quality assurance do at four o’clock in the morning with a production Web site on hold? To begin with, there is no time to plan during offline time—you have to be ready to work. You should have a checklist that was prepared ahead of time. This reduces release stress and gives quality assurance the ability to ask for other team members to lend a hand in speeding up the process. At this stage, release cycle testing is the same as development cycle testing—planning is essential.

Guidelines to check the new release

Here are the proposed steps that should be followed by the quality assurance team to verify the correct release has been installed on each production server.

Log in to Each Server using specifically created Login Page.

Generally multiple production servers are used to support a Web site. A load-balancing software is used that pushes each new customer session onto the server with the lowest number of users. While this load balancer is an important element of our production site, it is not helpful when quality assurance or development needs/wants to log in to a particular production server. To work around this issue, the developers should build a simplified version of our login page that allows the quality assurance team to specify which production server they want their session to be run on. For middle-of-the-night installations, this page becomes an important tool as we use the simplified page to log in to each server and ensure each server is ready to go.

Verify Basics First

For each server, some basic elements are checked that have nothing to do with the new release. Always make sure you can access each server and log in using a customer account that has been created ahead of time. Also check that basic transactions relevant to our site can be performed (like searching for an item, browsing the aisles, putting an item in the shopping cart, submitting an order, etc.).

Verify SSL is Works on Every Server

SSL (secure sockets layer) is an imperative part of an e-commerce site, because it renders any credit card number entry online reasonably safe. Check SSL to make sure it’s working on each production server after every installation. There might be secure and nonsecure pages in a web site. There are secure pages exposed only to existing customers and nonsecure pages exposed to new customers. Check both, and check each server.

Like any assessment done by quality assurance, you have to assess risk, likelihood, and impact. How big of a risk would it be to your site if SSL weren’t working? How likely would it be for a customer to encounter the flaw? And, could you survive the impact of that flaw on your production site? When you consider these points, checking each secure page on each server is like maintaining your company’s own life insurance policy; one that you wouldn’t want to skip payment on.

Use a Checklist of Visual Features that Changed for the Release

While most releases contain some functionality change that is not visually noticeable, they may include some change to the user interface such as a newly revamped page or a new icon/graphic. To prepare for your middle-of-the-night releases, create a list of these quick visual checkpoints ahead of time. Log into each production server and use these checkpoints to ensure each server has the correct release in place. Remember, the entire release has already been through the quality assurance process. This is the release process, not the development process.

You can see how forcing a session on a particular server combined with knowing the visual changes of a release creates a reasonably fast, manual means of ensuring the new release is in place on each server. Logging in, checking the basics, checking SSL, and making a visual checkpoint of a few items have to be done on each server. Because you are working against the clock, log into each server rapidly so if a particular server is having an issue, you can call out that issue to the network staff immediately. Then go back through each server and check the basics, SSL, and visual checkpoints. Making sure the basic build is working before testing new features in depth is similar to the QA strategy used typically during the development process.