Did I Remember To "Test Guide"

Accessibility

• Check the following MSAA property settings for every control on every dialog, command bar, and other UI feature: NAME, ROLE, STATE, VALUE, KBSHORTCUT, and DEFACTION
• Change values in edit boxes and verify those values are reflected in the MSAA property set
• Run in high contrast mode
• Run in large font mode
• Run with Sound Sentry
• Run with sticky keys and other mouse key settings
• Verify focus events are passed when controls receive focus
• Verify that all audio, video, and animations have supporting textual scripts, screen narration, and/or closed captioning
• Verify that text is rendered as text, not as bitmaps
• Verify tabbing order
• Verify shortcut keys exist for each control
• Verify that each actionable color item can have its color customized (e.g., red squiggly lines)
• Verify that all flashing objects flash to the system cursor blink rate

Alerts

• Verify each alert identifies the problem first and then a solution
• Verify alert text is not unnecessarily long (i.e., more than 3 lines)
• Verify alert text is consistently worded and consistent in style
• Verify alert text is correct and appropriate for the situation
• Verify alert text contains complete sentences with correct capitalization and punctuation
• Verify alert text avoids using abbreviations and acronyms
• Verify alert text does not use an accusatory tone but rather is polite and helpful
• Verify alert text uses the product name where necessary, not "we"
• Verify the buttons work correctly
• Verify the buttons have unique access keys
• Verify alert text is localized correctly
• Verify the correct icon (Information, Warning, Critical) is displayed on the left hand side
• Verify the title bar contains the name of the product (e.g., "Microsoft Bob")
• Verify the buttons are centered below the message text

Anti-Virus

• Test with and without anti-virus software installed

Application Interoperability 

• Verify clipboard operations (i.e., cut/copy/paste) between this and other features, between this and other applications
• Verify drag and drop operations between this and other features, between this and other applications

Boundary Cases 

• Text length (e.g., of a list entry, in a field)
• Extra-long filenames
• Spaces in the text (e.g., of a list entry, field name)
• Duplicate items
• Large selection (e.g., the entire document or list)
• Giant files
• 4096 x 4096 pixel image
• Maximum number of objects (e.g., shapes, columns)
• Maximum and minimum values (e.g., for margins, values in edit boxes)

CPU

• Multiple processors
• Multiple vendors CPU's (e.g., Intel, AMD)
• Laptop configurations

Dialogs

• Verify default edit focus and default button focus is correct
• Verify tab order
• Verify dialog and terms used therein are consistent with those in the rest of the application
• Verify ninched controls are shaded
• Verify mutually exclusive controls work correctly
• Verify any samples in the dialog reflect the actual formatting in the document
• Verify all commands that launch dialogs (e.g., F12 to Save As)
• Verify every control has a short-cut letter, that all short-cuts work, and that every action can be executed via the keyboard
• Verify every accelerator is unique within the dialog
• Verify the dialog's title is correct
• Verify the menu command to show dialog is followed with "..."
• Verify the sizing and spacing of all controls matches the design standard
• Verify all dialog states, especially including more or less controls being visible due to application state or settings
• Verify the correct system controls display on the title bar, and that they work correctly
• Verify help topics are correct
• Verify dialog tips (right-click on a control) are correct
• Verify status bar text is correct
• Verify the dialog displays and functions correctly with different color, font, and high contrast settings
• Verify the dialog displays and functions correctly in high DPI mode
• Verify all bitmaps being used with the text in the dialog are correctly localized
• Verify controls respond correctly to valid and invalid input
• Verify controls respond correctly to boundary case input (e.g., correct alerts are shown)
• Verify Escape cancels the dialog
• Verify the dialog is sticky (or not, as appropriate); i.e., it displays in same position as when last dismissed
• Verify settings in the dialg are saved (or not, as appropriate) the next time the dialog is invoked
• Verify the dialog is sized correctly in relation to its controls

File Names

• With and without the complete path
• With and without the extension
• Long file names
• Short file names
• Deep directory levels
• International characters
• Invalid characters (e.g., '?')
• Invalid filenames (e.g., COM1, COM1.txt)

Hardware

• Tablet PC
• Removable Media
• Pocket PC
• Run with Power Management settings enabled and disabled

International Sufficiency 

• 
  
• Look for geo-political sensitivity issues (e.g., borders may be disputed between neighboring countries)
• Run with Complex scripts and Right to Left (RTL) support
• Verify all controls function correctly with the East Asian (IME)
• Verify switching the keyboard to a different mapping works correctly
• Verify system locale, unicode, and non-active code page support functions correctly
• Verify ANSI, double-byte, and Unicode text are handled correctly for display, input, and elsewhere (e.g., double-byte characters will be split if treated as ANSI)
• Verify any language-dependant features work correctly
• Verify extended and non-standard characters are displayed and handled correctly
• Verify the executable and other core files function correctly on any code page and any supported OS language
• Verify the correct sorting order is used.
• Verify user locale or system locale are used as appropriate.

Localization 

• Look for geo-political sensitivity issues (e.g., borders may be disputed between neighboring countries)
• Verify you know which languages to test
• Verify controls in dialogs and elsewhere are aligned and sized correctly (e.g., text is not truncated)
• Verify any functionality affected by the localization process works correctly
• Verify data is ordered correctly
• Verify tab order is correct
• Verify no unlocalized strings remain
• Verify accelerators are localized
• Verify hot keys are localzied
• Verify each accelerator is unique and consistent
• Verify each hot key is unique and consistent

Menus and Command Bars 

• Verify all commands work from menus and command bars
• Verify keyboard shortcuts are correct on all platforms
• Verify built-in menu items on a custom commandbar work correctly
• Verify commands are visible and/or enabled only when appropriate
• Verify command captions are correct and consistent with similar terms used elsewhere
• Verify custom commands work when dragged onto built-in menus and toolbars
• Verify voice command works correctly
• Verify menu/command bar item context menus work correctly
• Verify status bar text is correct and not truncated

Miscellaneous -- Have you

• Completed the spec inspection has been completed
• Discussed the feature design and target users with your Program Manager
• Done real world testing (i.e., used the application the way the user will)
• Talked with your developer about the areas they think need special attention
• Brainstormed test cases with the rest of your team
• Discussed cross-feature implications with the rest of your team
• Discussed the feature/application with Product Support
• Perused bugs and other feedback from beta releases
• Considered backward compatibility problems resulting from something that has been moved or modified changing default values or breaking something in a previous version

Networking

• Verify correct handling of different levels of user access and permissions
• Verify correct handling of loss of network access
• Verify correct handling of user authentication requests
• Verify correct handling of file access via mapped drive (mapped root drive where supported)
• Verify correct handling of file access via UNC
• Verify correct handling of Print Preview when the network printer is disconnected
• Verify correct handling of long file name and long-file-name-to-short-file-name conversion correctly

Open/Save

• Open/Save from/to Sharepoint Document Library
• Context menu | Open, New, Print
• Drag-and-drop
• Double-click a shortcut on the Desktop
• Most Recently Used list
• Command line argument
• Read-only files, directories, floppies, drives
• Open from/save to over various types and speeds of network connections
• Open from/save to different types of prototols (e.g., FAT, NTFS, UNC, http://, ftp://, floppy)
• Open from/save to previous versions; roundtrip from/to previous versions
• Open/save documents created in a different language-version of the application
• Interrupted
• File/Path name (e.g., long filenames, spaces in name)
• Auto-save
• Save when no documents are dirty, one document is dirty, multiple documents are dirty
• Save when multiple documents are dirty but the user chooses to not save some of them
• Save As

Out Of Memory (OOM)/Stress Testing 

• Low memory
• Low disk space
• OOM caused via automation
• OOM caused via real world scenarios
• Minimum requirements machine
• Flaky network
• Too may users
• Working with floppies
• Aborting operations (e.g., user cancels, network cable pulled)
• Pen And Speech Input
• Speech Tips
• Handwriting Tips (non-TabletPC)
• Microsoft and non-Microsoft Input Method Editors (IMEs)
• Unaware application support enabled and disabled (Windows SP1 or later only)
• Handwriting/speech/IME on Tablet PC Windows

Performance

• Verify perf testing has been done
• Verify perf targets exist and are being met
• Verify the correct data is being benchmarked
• Verify performance with Clear Type turned on
• Verify performance optimizations actually have the intended effect
• Compare performance with previous versions
• Compare performance with similar applications

Platforms

• Run on every supported platform; at a minimum, probably, WinXP SP1, WinXP SP2, Windows Server 2003 Gold

Printing

• Verify changing orientation works properly
• Verify printing to a file works properly
• Verify collation works properly
• Verify "Number of Copies" works properly
• Verify color printing works properly
• Verify canceling printing works properly
• Verify scaling works properly
• Verify line or page numbering works properly (if supported)
• Print to both local and network printers
• Print to Adobe Acrobat to create PDF files

Registry / Configuration File Settings

• Verify settings that should modify behavior do
• Verify the appropriate settings are available in policy templates
• Verify settings roam with the user

Security 

• Consider what buffer overrun attacks might be possible
• Consider what denial of service attacks might be possible
• Consider what SQL injection attacks might be possible
• Consider what virus attacks might be possible
• Verify no NULL DACLS are created or used.
• Verify security for links and macros is sufficient and works correctly
• Verify filenames such as "..\..\..\file" are handled correctly
• Verify any temporary files are created in appropriate locations and have appropriate permissions
• Verify any user-created files have appropriate permissions
• Verify the application functions correctly under different user rights and roles
• Verify user-specified input is handled correctly (e.g., are SQL injection attacks possible)
• Verify every input is bounds-checked
• Verify user privacy is not violated (e.g., file paths are stripped, user aliases and other identifying data can be removed)

Setup - General

• Consider what problems could appear when installing as Minimum, Typical, Install on First Use, Run from Network, Custom, and Not Available.
• Verify registry items are created/modified appropriately, and that other registry items are not created or modified.
• Verify the application works correctly when portions of it have been disabled by an administrator
• Verify any files shared with another feature or application are handled correctly
• Verify installing and uninstalling the application or feature has the correct effect (which may be "nothing should change) on other features and applications
• Verify "normal" users can use the application when it is installed by an administrator
• Verify the correct files are installed when the application is installed via Install On First Use
• Verify mass deployment (e.g., via Systems Management Server) works correctly
• Verify the application works correctly on Terminal Server
• Verify the application works correctly on Virtual PC and Virtual Server

Setup - Local Caching 

• Verify the correct CABs are cached
• Verify any files shared with another feature or appliation are handled correctly by install/uninstall/reinstall
• Verify multiple SKUs share the cache correctly
• Verify deleting a CAB from the cache causes a reinstall and the CAB is recached

Setup - MSI Authoring

• Verify the MSI includes the correct components, files, and registry settings
• Verify custom actions, conditions, shortcut creation, and other special authoring works correctly
• Verify the default installation type (e.g., Typical, Minimum) is correct
• Verify the correct install states are available
• Verify the application works correctly in all possible install states

Setup - Multi-User

• Verify the application works correctly for User2 after User1 installs it
• Verify per-user features must be installed by User2 even after User1 has installed them
• Verify User2's per-user settings do not change when User1 changed them

Setup - Network Setup 

• Does your feature uninstall cleanly when installed from the network (Post-Admin install)
• Verify the correct files are installed when the application is installed via Run From Network
• Verify the application can be uninstalled

Special Modes and States 

Safe Mode (e.g., booting with /safe to disable some settings) works correctly (if supported)

Zoomed (e.g., 100%, 25%, 200%)

Send To

Sharing documents between multiple users/machines

Cut, copy, delete

MDI, SDI

No file open (just app shell)

Full screen view

No command bars visible

Application and document (e.g., minimized, maximized, sized really small)

OLE

Autosave on

System background color changed

Multiple selection

Undo/Redo 

Edit/Undo, Edit/Redo

Multi-level Undo/Redo

Menu command text strings

Upgrade and Migration - Application Upgrade

• Verify upgrading over a previous version replaces appropriate files but no others
• Verify side-by-side installs work correctly
• Verify the correct files (and versions thereof) exist after an upgrade
• Verify default settings are correct
• Verify previously existing settings and files are modified correctly by an upgrade
• Verify the application functions correctly when the previous and/or new version are set to Run From Network
• Verify any features or applications dependent on files or functionality affected by the upgrade work correctly after the upgrade

Upgrade and Migration - OS Upgrade 

• Verify upgrading over a previous version replaces appropriate files but no others
• Verify the application works correctly after the upgrade
• Verify any features dependent on OS files or functionality work correctly after the upgrade
• Verify the application works correctly after upgrading both the application and the OS

User Assistance/Help

• Verify each error message is accurate, easily understood, and actionable
• Verify input validation error messages refer to the correct problem
• Verify the tutorial is correct (e.g., the UI it depicts matches the actual UI, the steps it takes are correct)
• Verify all help topics have been reviewed for technical accuracy
• Verify all context sensitive help is correct

Video

• Multi-monitor environments (two, three and four monitors)
• Video resolutions (1024x768 through 1600x1200)
• color depths (8-, 16-, 24-, and 32-bit)
• Color themes

Window Interactions 

• IME
• Z-ordering, especially Help on top
• Modal, modeless dialogs
• Window and dialog focus
• Window size after a restore-from-minimize, restore-from-maximize
• Window size on first launch, on subsequent launches
• Window size after a manual resize
• Multiple windows (MDI)
• Multiple instances of the application
• Arrange Vertical, Horizontal, All, Tiled, Cascade (both MDI children and application windows on the Desktop)

Windows Logo Requirements

Run all applicable Application Verifier tests (especially check that NULL DACLs are not used or created)

Install and run under Application Verifier with the “High Version Lie” test enabled

Install and run with Fast User Switching enabled, while Fast User Switched

Install and run on Windows 2000, then upgrade to Windows XP

Install and run in a highly secure Windows environment (“%system%\security\templates\hisecwsinf)

Run as a limited user

Verify temp files are placed properly

Verify System File Protection files are not overwritten

Verify that My Documents is the initial default location for Open and Save

Verify all UI inherits Windows XP themes

Verify files outside of the application folder are associated with a particular file type (i.e., double-clicking does the right thing)

Verify all executable components have a (correct) digital signature

Verify all UI is legible and accessible at 133DPI (high DPI)

Year 2000 

Verify dates entered with a two-digit year are interpreted as:

 1/1/00 through 12/31/29 are interpreted as 1/1/2000 through 12/31/2029

 1/1/30 through 12/31/99 are interpreted as 1/1/1930 through 12/31/1999

Verify dates on leap years are correctly interpreted (e.g., 2/29/2000, 2/29/2004)

Verify dates at least to 2035 are supported

What's the Difference? Virtualization vs. Cloud Computing

What's the Difference? Virtualization vs. Cloud Computing.

Is virtualization right for your business? How about cloud computing? Don't know the difference? That's OK — most non-IT folks don't either. The word "cloud" is often thrown around as an umbrella term, while "virtualization" is often confused with cloud computing. Although the two technologies are similar, they are not interchangeable, and the difference is significant enough to affect your business decisions. Here is a guide to help demystify the tech behind the jargon.

What is virtualization?

In a nutshell, virtualization is software that separates physical infrastructures to create various dedicated resources. It is the fundamental technology that powers cloud computing. [Cloud Computing: A Small Business Guide]

"Virtualization software makes it possible to run multiple operating systems and multiple applications on the same server at the same time," said Mike Adams, director of product marketing at VMware, a pioneer in virtualization and cloud software and services. "It enables businesses to reduce IT costs while increasing the efficiency, utilization and flexibility of their existing computer hardware."

The technology behind virtualization is known as a virtual machine monitor (VMM) or virtual manager, which separates compute environments from the actual physical infrastructure.

Virtualization makes servers, workstations, storage and other systems independent of the physical hardware layer, said John Livesay, vice president of InfraNet, a network infrastructure services provider. "This is done by installing a Hypervisor on top of the hardware layer, where the systems are then installed."

How is virtualization different from cloud computing?

Essentially, virtualization differs from cloud computing because virtualization is software that manipulates hardware, while cloud computing refers to a service that results from that manipulation.

"Virtualization is a foundational element of cloud computing and helps deliver on the value of cloud computing," Adams said. "Cloud computing is the delivery of shared computing resources, software or data — as a service and on-demand through the Internet."

Most of the confusion occurs because virtualization and cloud computing work together to provide different types of services, as is the case with private clouds.

The cloud can, and most often does, include virtualization products to deliver the compute service, said Rick Philips, vice president of compute solutions at IT firm Weidenhammer. "The difference is that a true cloud provides self-service capability, elasticity, automated management, scalability and pay-as you go service that is not inherent in virtualization."

What are the advantages of a virtualized environment over the cloud?

To best understand the advantages of virtualization, consider the difference between private and public clouds.

"Private cloud computing means the client owns or leases the hardware and software that provides the consumption model," Livesay said. With public cloud computing, users pay for resources based on usage. "You pay for resources as you go, as you consume them, from a [vendor] that is providing such resources to multiple clients, often in a co-tenant scenario."

A private cloud, in its own virtualized environment, gives users the best of both worlds. It can give users more control and the flexibility of managing their own systems, while providing the consumption benefits of cloud computing, Livesay said.

On the other hand, a public cloud is an environment open to many users, built to serve multi-tenanted requirements, Philips said. "There are some risks associated here," he said, such as having bad neighbors and potential latency in performance.

In contrast, with virtualization, companies can maintain and secure their own "castle," Philips said. This "castle" provides the following benefits:

Maximize resources — Virtualization can reduce the number of physical systems you need to acquire, and you can get more value out of the servers.  Most traditionally built systems are underutilized. Virtualization allows maximum use of the hardware investment.
Multiple systems — With virtualization, you can also run multiple types of applications and even run different operating systems for those applications on the same physical hardware.
IT budget integration — When you use virtualization, management, administration and all the attendant requirements of managing your own infrastructure remain a direct cost of your IT operation. 

How do you know if your business needs a virtualization solution?

Determining whether or not virtualization is the best solution for a business requires an in-depth analysis of the organization's specific needs and requirements.

"Some of the items we discuss with customers when they are evaluating private cloud — virtualization — versus cloud computing include who is going to be providing the support and how challenging is integration with other systems," Livesay said.

You should also consider costs — total cost of expenditure (TCO), operational expenditures (OPEX) and capital expenditures (CAPEX) — how much management the business can and want to do, scalability requirements, security needs and how much feature development can be expected, Livesay said.

"Generally speaking, businesses who work more on an OPEX model that have less IT staff and fewer security concerns are more cloud oriented," Livesay said. "Businesses that need greater control for integration and security or who work more on a CAPEX model would lean towards virtualization."

How do businesses know if they should use a true cloud solution?

While virtualization is the best solution for some organizations, a cloud solution offers several benefits that are more suitable for other businesses.

Philips said cloud solutions are best for business with the following needs:

Outsourced IT — The day-to-day administration, care and feeding of supporting systems move away from you to the service provider. This could free up internal IT resources for higher-value business support and allow you to put IT budget dollars toward efforts that advance your business.
Quick setup — Cloud startup is relatively quick and easy. Plus, servers, appliances and software perpetual licenses go away when you use such a service.
Pay-as-you-go — An example could be found in Software-as-a-Service (SaaS) applications available today that allow the off-loading of basic IT requirements to cloud service providers.  You pay for what you need and use. But you do not have to continue to invest in many of the products used to support the network and systems, such as spam/anti-virus, encryption, data archiving, email services and off-site storage.
Scalability — By using the cloud, you can also temporarily scale your IT capacity by off-loading high-demand compute requirements to an outside provider. As a result, as mentioned above, you pay for only what you need and use, only at the time when you need it.
Keep in mind, however, that virtualization and cloud services are not end-all, be-all solutions. Like any other technology or service a business adopts, things can always change.

"While cloud computing and virtualization each have their own benefits, they are not competing approaches," Adams said. "We view cloud computing as an evolution of virtualization. Customers that virtualize their hardware servers may adopt cloud computing over time for increased self-service, scale, service delivery levels and agility."

What should businesses look for in a virtualization provider?

Businesses considering virtualization should think about the following questions, Adams said:

1. Is it a tried and tested solution? Research the vendor's track record of product innovation, success and customer adoption.
2. Is there a vision and public roadmap for the solution? You want to understand how the solution will advance and how it will help your business in the long run.
3. What type of ecosystem support exists for the solution? It's imperative that the vendor work with key business and industry-specific independent software vendors (ISVs), as well as a wide range of resellers, service providers and system integrators.
4. Does the solution support openness and choice? As your business grows, you want the flexibility to evolve your products and processes, and the ability to incorporate other technologies over time.

VIRTUALIZATION

You probably know a little about virtualization if you have ever divided your hard drive into different partitions. A partition is the logical division of a hard disk drive to create, in effect, two separate hard drives.

Operating system virtualization is the use of software to allow a piece of hardware to run multiple operating system images at the same time. The technology got its start on mainframes decades ago, allowing administrators to avoid wasting expensive processing power.

In 2005, virtualization software was adopted faster than anyone imagined, including the experts. There are three areas of IT where virtualization is making headroads, network virtualization, storage virtualization and server virtualization:

• Network virtualization is a method of combining the available resources in a network by splitting up the available bandwidth into channels, each of which is independent from the others, and each of which can be assigned (or reassigned) to a particular server or device in real time. The idea is that virtualization disguises the true complexity of the network by separating it into manageable parts, much like your partitioned hard drive makes it easier to manage your files.

• Storage virtualization is the pooling of physical storage from multiple network storage devices into what appears to be a single storage device that is managed from a central console. Storage virtualization is commonly used in storage area networks (SANs).

• Server virtualization is the masking of server resources (including the number and identity of individual physical servers, processors, and operating systems) from server users. The intention is to spare the user from having to understand and manage complicated details of server resources while increasing resource sharing and utilization and maintaining the capacity to expand later.

• Virtualization can be viewed as part of an overall trend in enterprise IT that includes autonomic computing, a scenario in which the IT environment will be able to manage itself based on perceived activity, and utility computing, in which computer processing power is seen as a utility that clients can pay for only as needed. The usual goal of virtualization is to centralize administrative tasks while improving scalability and work loads.