John: It wasn't an academic point. These are some of the things that do happen: a questionnaire is given to firm juniors, even to outside telephone canvassers, who have no knowledge of the industry, and they are instructed to telephone as many on an industry list as they can reach on a Thursday afternoon. What you get is not a representative sample of the industry but a sample of those who were willing to talk on Thursday afternoon. The people they get to speak to at the respondent-company can range from people in PR, in the CFO office, or Sales, and the respondents may have a deeply informed or only superficial knowledge of the questions being asked, yet all responses are given equal weight in the survey. Bias in questions is not merely malicious bias, bias is when the question is based on assumptions which are not spelled out and which are not universally understood. Ambiguity is connected to assumptions. The same question can mean slightly different things to the questioner, the respondent and other respondents.
I can't agree with you that Mulonuvich simply asked simple questions and reported the simple answers. It is not a world of pure questions and brute facts. The answer is conditioned to a large extent by the question. ML wrote the questions based on ML understanding of what the questions should be, and ML has a definite thesis about the industry: it would be naive to assume that its questions did not reflect its thesis. And ML commented on all the answers. I thought some of ML comments were not supported by its own survey results, even if the responses had been unambiguous to me, and even if they had been representative.
(All of the above is by way of example. Another example might be a survey conducted by well-informed industry experts and dealt with by the heads of the appropriate departments in all the relevant companies or a scientifically chosen sample of them. The fact is we don't know in this case).
I have no difficulty with your idea that ML are trying to produce data and their data is necessarily flawed and that the output is only as good as the input. You seem to be agreeing with me that it was a superficial exercise even though you object to the word. So let's just say it was half-baked.
At the end of the day you will rely on Milonuvich or you won't. I won't. That's not to say I won't read his opinion and take it into consideration. |
