GV,
Below is the methodology used by the latest PC magazine customer service and reliability. It's based on users response, not magazine bias. This differs greatly from the magazines hardware tests where it's rare to see any DELL hardware "earn" less than 4 stars out of 5. For what its worth, in editorial columns from various magazines the general sentiment has been that there is no bias, only very similar machines from all the major players, and they now differentiate ratings by price, included hardware, and customer rated reliability, and in these areas, DELL usually shines. To most end users, it matters not that one machine might be able to open a word document a few tenths of a second faster than a different machine. Ultimately you are correct, there is no real significant difference.
Cheers
We ran this year's reader survey in April 2003. We e-mailed subscribers invitations and ran an invitation in the magazine. We accepted only one entry per subscriber—and only subscribers could participate. Equation Research hosted our survey and tabulated the results.
We asked respondents which types of equipment they had used in the past year (desktops, notebooks, servers, and printers); they then received only the pertinent sections of the survey. They could complete the forms for one or two qualifying products per section.
For each survey question, we compile the individual ratings and exclude any company for which we did not receive at least 50 responses, the minimum number necessary to produce statistically reliable results. Question by question, we compare each company's score (the mean rating) against the average of all the companies' scores in the same product category. We do not include clones and self-built machines in this calculation.
We use a t-test measure on each individual vendor's score on each question against the average of all the companies' scores and determine whether the scores are significantly different at a 95 percent confidence level. Note that in calculating our baseline average we give equal weight to every company's score, rather than averaging all the individual responses to the question (biases resulting from one company having a much larger number of responses than the rest of the companies evaluated are removed).
The t-test considers a company's score for each measure, the total number of responses for the company on each measure, and the variability of the responses in calculating the confidence interval for each score—the range into which we are 95 percent certain that the score belongs. When this range is entirely above or below the average of all companies' scores, then we can say that the score is significantly different from this average.
In the diagram (below), D and E have the same score, but E's larger confidence interval crosses the average line, and we can't say with 95 percent confidence that E's score is significantly better than average. On negative questions, such as "Solution offered did not work," a better-than-average score means that fewer people than average had this problem.
To calculate each Readers' Report Card grade, we look at four key measures: overall satisfaction, satisfaction with reliability, units needing repair (or, for servers, frequency of failure), and willingness to buy the same brand again. Each score that is significantly better than average gets 4 points; average gets 2, and significantly worse than average gets 0. If the average of these point values is greater than 3, the company gets an A. For a B, the company needs to average better than 2; for a C, exactly 2; for a D, 1 or better. Anything below 1 is an E.
Scores for satisfaction with technical support and with repair contribute to a plus or minus added to a grade. If the average of these scores' point values is greater than 2, the company gets a plus; if below 2, a minus. An A+ requires above-average scores on all six criteria. The top-ranking companies in each category receive Readers' Choice status. |
