Measuring Quality Perception

Prior to QA testing cycle, making quality measurements is difficult. At this point, QA is focused on test plans, automation, and product definition, while the engineering team is focused on designing and building the product.  Getting an earlier quality snapshot of what is going on prior to any testing being done is useful because most of the quality problems are built in during definition and design.

A potential early quality indicator is QA team quality perception. This approach measures the QA team quality perception during the release the entire release cycle. This approach is effective on larger projects when the QA team is working alongside engineering during development to build up their test plans.

Quality perception requires a survey be sent out weekly to assess subjective opinions by the QA team.  A simple 1-5 ranking scheme works well.  5 would represent best quality and 1 the worst.  When the data is collected, calculate the minimum, maximum, and average values of what is reported.  The end results can be shown in ½ unit increments.  Perception can be an overall number or applied to specific projects.  

The easiest way to set up a survey is to create a web form that emails your when they submit. The form will have a text box for the submitters to include comments. Another approach is to email the request and have them reply.  I found that people responded better to a web-form than just the email.

A spreadsheet example for entering the data follows showing 6 entry columns and the min, max, average calculations are shown in Figure 1.  Data is entered in the Entries columns, one per person per week.  Min/max/average are calculated using =max( ), =Min( ), and =Average functions in Excel.   Rows of data can be added for each week. 

Figure 1 – Quality Perception Spreadsheet

	A	B	C	D	E	F	G	H	I	J
1	Date	Max	Min	Avg	Entries
2	1-Jun	4.0	3.0	3.5	3	4	3	4	3	4
3	8-Jun	5.0	3.5	4.17	5	3.5	3.5	4	5	4
4	15-Jun	4.5	3.0	3.75	4.5	3	4	3	4	4
5	22-Jun	5.0	3.5	4.17	4.5	4	3.5	4	4	5
6	29-Jun	5.0	4.0	4.33	4	4	4	4	5	5
7	6-Jul	5.0	4.0	4.75	5	4	4.5	5	5	5

A useful way of plotting this information is to use the format usually associated with the stock market.  Instead of low, high, and close, use minimum, average, maximum values reported. A sample plot of Quality Perception is shown in Figure 3.  Notice the dip during week 6/15. This would be a good week to ask the QA team what has changed.  The text note describing the change will assist in understanding what is going on. Although the average value is useful for spotting trends, investigate the low rankings as these may be localized problems on the project.

Figure 2 – Plot of Quality Perception

My experience is that QA teams do find this type of feedback empowering.  They like having their concerns heard and are more likely to point out problems early.