Software Testing Social Network

Check list for Installation Testing 

{mosgoogle left}  

§ Test the functionality and UI of the installer.

§ Test the functionality of the application that is installed and set up.

§ Test the known error conditions and error handling of the installer and uninstaller.

§ Test the impact that the installer and uninstaller have on existing system environments.

§ Test software and hardware compatibility.

§ Test the installer functionality on multiple server configurations.

§ Test the installer functionality using multiple installation options and conditions.

§ Test the configurations and modifications that the installer makes to existing files and registry entries.

§ Test the uninstall program to see that it removes all data files—including application executables and .DLLs—that are installed by the installer.

§ If your company markets multiple products with independent installers, test for installer compatibility between products. For example, can you install both products without conflicts? Can you uninstall individual products without affecting the others?

AREAS OF CONCERN THAT SHOULD BE CONSIDERED DURING INSTALL/UNINSTALL TESTING

§ The functionality of the installed application.

§ The functionality of the install and uninstall programs.

§ The error handling of the install and uninstall programs.

§ The UIs of the install and uninstall programs.

§ The environment conditions in which the install and uninstall programs (and, ultimately, the installed application) will operate. Test coverage should include application-specific and environment-specific variables (including both dynamic and static conditions).

§ Application-specific conditions: all supported user-setup options, all supported upgrade options, and all reinstallation conditions.

§ Environment-specific conditions: all supported software and hardware conditions (especially when the installer relies on the existing configuration in determining which setup options to run).

§ Does your product require administrative (Admin) privileges to install it? If so, is an explicit error message to this effect given if you try to install it without Admin rights?

{mosgoogle left} 

TEST SCENARIOS THAT SHOULD BE CONSIDERED

§ Installation under minimum configuration.

§ Installation and running of application on a clean system. A clean environment consists of only the required components of an operating system.

§ Installation and running of an application on a dirty system. A dirty environment consists of the operating system components and other commonly used software, such as various versions for browser, productivity applications, virus checkers, and so on.

§ Installation of upgrades that are targeted toward an operating system (e.g., Windows 98 to Windows 2000).

§ Installation of upgrades that are targeted toward new application functionality. Did the installer remove the dated files? Did any other applications depend on the dated files?

§ Installation of software over multiple operating systems.

§ Reducing the amount of free disk space during installation to see if the installer can respond gracefully to an unexpected lack of sufficient space after the installation has begun.

§ Canceling the installation midway through to see how well it restores the system to the base state.

§ If you change the default target installation path to a different drive, will all the files really be installed in the specified path? For example, changing C:\program files\targetdir to D:\program files\targetdir: some programs will still place some files in the C:\program files\ targetdir path without warning, thus spreading the installation between two or more drives.

FUNCTIONAL INSTALLATION TESTING CONSIDERATIONS

§ Test a mix of UI navigation and transition paths.

§ Look for user-level logic errors. For example, run the installer by following all on-screen instructions and user guide instructions; look for software-to-documentation mismatches.

§ Consider test cases for error detection and error handling.

§ Make sure that the installer does not prompt inaccurate or misleading error messages.

§ Consider whether the installer might obtain incorrect path information, and thereby install shared files in the wrong place or update registry keys with the wrong information.

§ Consider incorrect default path errors.

§ Check for incorrect file permissions. The install program may provide permissions based on the read/write/execute permissions of the install media.

§ Consider the file ownership for the installed files.

§ Test with full, compact, and custom installation options.

§ Test with various installation branches.

SOME COMMON INSTALLATION FUNCTIONALITY ERRORS

§ The main application does not successfully operate in all setup options.

§ The installer fails to operate under the minimum configuration.

§ The installer fails to operate under the maximum configuration. For example, if the size of the variable used to store the value of free disk space is too small for the actual amount of free disk space, that variable will be overflowed. This error often leads to a negative value reported for free disk space. In turn, it might prevent the installer from executing.

§ The installer assumes (via a hard-coded path) that some source files are on floppy drive A. Therefore, installation fails if the user installs from floppy drive B or over the network or from any other drive whose name is not A.

§ The installer fails to provide the user with default options.

§ The installer does not check for available disk space.

§ The installer fails to check whether certain key components (such as Internet Explorer or Acrobat) are already present on the user’s system. Instead, it installs a new version (which might be older than the copy on the user’s disk) and sets a path to that newly installed version.

§ The installer fails to inform the user of how much space the installation requires.

§ The installer fails to operate on a clean system.

§ The installed application fails to operate after the completion of an install on a clean system.

§ The installer fails to complete due to a forced reboot in the middle of the install script.

§ The uninstaller fails to remove all program files.

§ The uninstaller removes files that the user created without informing the user or offering an alternative.

§ The uninstaller moves user files stored in the user directory to a new location without informing the user or offering an alternative.

§ The uninstaller fails to remove empty directories left behind by the application.

USER INTERFACE INSTALLATION TESTING CONSIDERATIONS

§ Execute the test cases in Appendices D and E (the keyboard and mouse action matrices, respectively).

§ Test the default settings of the UI controls.

§ Test the default command control for each dialog and message box. Does it lead to a typical installation?

§ Check the behavior of common function keys such as ESC, Enter, F1, Shift-F1, WINDOWS, and so on.

§ Check for proper UI updating and refresh during dialog box interaction. Also check navigation between dialog boxes (using Back and Next buttons).

§ Test the default click path that is generated by clicking the Tab button repeatedly. Is the path intuitive?

§ Test the default click path that is generated by clicking the Tab button repeatedly while holding down the Shift button. Is the path intuitive?

§ Test the implementation of accelerator keys (underscores beneath letters of menu selection items). Are the keys functional? Have intuitive character selections been made (N for Next, B for Back, etc.)?

§ Are there conflicts between accelerator commands? If so, is the most commonly used command given preference?

§ If a common command is not given an accelerator shortcut, is a symbolic alternative offered (for example, Ctrl-X for Cut, and Ctrl-W for Close)?

§ Is a quick-key or accelerator key (one-handed) interface possible?

COMMON UI CONVENTION FOR DIALOG BOX COMMANDS

§ The X button in the top right corner of Windows means “close the current window” or “close the current window and cancel the current operation.”

§ Next means “go to the next dialog box and close the current dialog box.”

§ Back means “go to the previous dialog box and close the current dialog box.”

§ Cancel means “cancel the current operation and close the current dialog box.”

§ Resume means “resume the current application and close the current dialog box.”

§ Exit Setup means “exit the Setup program and close the current dialog box.”

§ Yes means “yes to the question being posed, and close the current dialog box.”

§ No means “I choose No to the question being posed, and close the current dialog box.”

§ Finish means “finish the installation and close the current dialog box.”

COMMON ERRORS IN MISINTERPRETATION OF COLLECTED INFORMATION (WINDOWS-SPECIFIC)

§ The installer misidentifies the existence (or nonexistence) of a certain application (e.g., a browser) or shared file (e.g., a DLL) because it refers to an unreliable source—for example, a wrong key in the registry database.

§ The installer misidentifies the software platform and configuration (OS, drivers, browsers, etc.).

§ The installer misidentifies the hardware configuration (CPU type, CPU clock speed, physical or virtual memory, audio or video player settings, etc.) because it misinterprets the return values of an API call.

COMMON INSTALLATION ERRORS RELATED TO OPERATING SYSTEM ISSUES (WINDOWS-SPECIFIC)

§ The installer fails to register basic information (per Microsoft logo guidelines) such as company name, application name, or version in the registry.

§ The installer copies files other than shared DLLs to \WINDOWS or \SYSTEM directories.

§ The installer fails to register OLE objects in the registry.

§ The installer places application fonts in a folder other than the Fonts folder.

§ The installer fails to use a progress indicator.

§ The installer fails to add shortcuts to the Start menu.

§ The installer fails to register document types.

§ The installer fails to support universal naming convention (UNC) paths.

§ The installer does not autorun from a CD.

§ The name of the installation program is not SETUP.EXE.

§ The installer fails to configure the Context menu.

§ The uninstaller fails to remove all information from the registry.

§ The uninstaller fails to remove shortcuts from the desktop.

§ NTFS compression: Some applications have problems and display erroneous I/0 error messages when they detect NTFS compression.

COMMON DLL-RELATED ERRORS (WINDOWS-SPECIFIC)

§ The installer fails to copy required DLLs (perhaps the files are not even included in distributed media).

§ The installer fails to install the correct versions of DLLs ( MFC DLLs, IFC DLLs, and other shared DLLs).

§ The installer fails to check for the existence of DLLs needed by the application.

§ The installer fails to correctly reference count-shareable DLLs in the registry. Shared DLLs that are to be installed in the Windows\System or Program Files\Common Files directories (that are not part of a clean install of Windows 9x) need to register, increment, and decrement the reference count in the registry.

§ The application fails to operate correctly due to the existence of several incompatible versions of DLLs that are shared by multiple applications.

§ The application fails to operate properly due to the existence of several incompatible versions of DLLs that are produced or supported by a specific vendor.

§ The installer fails to copy systemwide shared files (e.g., VBRUN40O. DLL) to the Windows\SYSTEM or WinNT\SYSTEM directories.

§ The uninstaller fails to correctly reference count-shareable DLLs in the registry.

§ After decrementing a DLL’s usage count that results in a usage count of zero, the uninstaller fails to display a message offering to delete the DLL or save it in case it might be needed later.

§ The uninstaller fails to completely remove files, including program folders (unless there is user data in them), LNK files, nonsystem-shared files (if no longer used), directories, and registry keys.

§ The uninstaller mistakenly removes system DLLs.

< Prev		Next >