Michelle, to understand just what type of publicity game McAfee is playing, check out the response to McAfee's press release. Note all the info McAfee seems to have 'left out'.
This can be found at nha.com
Response:
NH&A responds to recent McAfee press release
September 18, 1996. NH&A specializes in providing anti-virus,
security and network management software. As such, we often find ourselves
talking to prospective customers about software of competing products.
Accordingly we strive to work closely with the companies we represent in
order to provide information that the customer needs to know to make an
informed decision. We have been "beta" testers for McAfee, Symantec
and other anti-virus products over the past 6 years.
In July, we were contacted by Symantec technical staff to do testing of 3
anti-virus products on the macro viruses that were "in-the-wild". No one
was doing macro testing and Symantec felt we were capable of doing such
a test. I had the thought that the technical staff wanted someone independent
to confirm that they had achieved a superior product. We thought of the
project as a potential business area that we could develop into a "niche" area
since there was no one doing this for macro viruses what Patricia Hoffman
and others were doing with file and boot sector viruses. At the time there
were only about a dozen commonly known macro viruses (plus one trojan)
but only 2 or 3 were on Joe Well's list of viruses "in-the-wild". We agreed
to do the testing and proceeded to put together a test set of viruses, get the
latest versions of the products to be tested and then conduct the test. The
results of our test were put on our web site under URL:
nha.com
Symantec took our results and issued a press release and I understand did some
additional marketing that referred to our test. We were flattered that Symantec
used our results but, as usual with anti-virus companies, were somewhat
dismayed by the way it was presented. We were comforted that the press release
referenced our web page where the user could see our results and draw their
own conclusions of the tests.
Today we were contacted by PC World and, after providing us
with a copy of a press release, we conducted an interview with PC World. Some
of our comments can be found on Real Audio at URL:
pcworld.com
Additionally we felt it necessary to answer to some of the statements that
were included in the McAfee press release that references our test.
McAfee made statements questioning the tests that we performed on the 10
macro viruses which we posted on our web site under What's New via URL:
nha.com on July 18th, 1996. We felt it was important to
answer the McAfee's statements that focused on our study, hence this posting.
Obviously NH&A has no control over the content of either the Symantec or the
McAfee press releases but we feel it important to answer to some allegations.
Additionally, as part of the McAfee press release, McAfee quotes another study
by Secure Computing which we, at NH&A, believe is inherently biased because
Secure Computing (despite their good reputation) receives viruses from CARO
members. This study and our response have been moved to the back of this
response for clarity and priority of issues.
> Symantec's Advertising Claims Rely upon Suspicious Virus Detection
> Study
>
> In its advertising and on its Web site, Symantec draws a series
> of conclusions from a supposedly "independent" study conducted by
> Norman Hirsch, who is actually a Symantec reseller.
NH&A (Norman Hirsch & Associates) is a reseller of McAfee, Symantec,
Dr. Solomon, Sophos, Trend, TBAV, Cybersoft, Cheyenne, et. al. plus other
anti-virus software. NH&A specializes in anti-virus software. We feel the
above statement is misleading because it implies prejudice toward Symantec,
which we have none.
> "Many in the industry suspect that the Hirsch study was actually
> commissioned by Symantec," continued Scott Gordon. "When one
> examines the testing methodology of the Hirsch study, it becomes
> immediately apparent that McAfee was put at a deliberate
> disadvantage."
The study WAS commissioned by Symantec. This was not a hidden fact
although it was not stated explicitly. The testing methodology and results
are completely repeatable if the current versions of the software at the time
are used. We stand behind our study. I expressed this to Scott Gordon
when he called me regarding this study. Two months ago, McAfee missed
some common viruses at the time. Trend's PC-Cillin missed fewer than
McAfee yet more than Symantec. They have all improved considerably
now. We hope our study helped all anti-virus developers to improve their
products.
> While Hirsch tested the July release of Norton AntiVirus, he
> tested the June version of VirusScan. McAfee's July release, which
> came out just 11 days later, would have scored significantly better.
We used the current versions of the products we tested at the time. If we'd
have waited 12 more days, we could have also used a later version
of Symantec! The Trend PC-cillin product release was the most recent--dated
only two days prior to our study. We obtained all of the latest versions from
the internet. McAfee releases have, in fact not been produced on a on a
regular basis. Indeed at the time, there was not even a later "beta"
version of McAfee available.
> Furthermore, Hirsch's peculiar choice of viruses, in which he
> completely omitted well-known Macro viruses, casts additional doubt
> upon the sponsorship and motivations behind the test.
At the time (2 months ago), there were only about 10-12 viruses that were being
talked about on multiple anti-virus vendor's Web sites and in the newsgroups.
We insured ourselves that the viruses we used were either talked about on at
least 2 vendors Web sites or that we had samples that backed up what at least
one anti-vendor described. Now there are considerably more viruses but still
not enough to base tests solely on in-the-wild criteria. Nevertheless we have
recently provided a newer and more comprehensive comparison test of all the
macro viruses we have and these results are posted on our web page
URL: nha.com
> Excluded from
> the Hirsch test were several well known Macro viruses which the
> Secure Computing test demonstrated Symantec could not detect.
The "Hirsch" tests in question were done 2 months ago. The Secure Computing
test was put on CompuServe yesterday for publication in the November issue
of Secure Computing. NH&A has also released a current test via
URL: nha.com which also has considerably more well
known viruses than existed 2 months ago.
> According to Ian Whalley, editor of Virus Bulletin, a
> well-respected anti-virus newsletter published in the U.K., "The
> scores in the Norman Hirsch test of July 18, 1996 are not in
> question, however the methodology and subsequent presentation of the
> results used in Symantec advertising is open to interpretation. Of
> the viruses used [in the test by Hirsch], only three - Concept,
> Impostor, and Wazu - are actually in the wild. Format.c, which was
> listed in Hirsch's test as a virus, is actually a trojan, not a
> virus. Therefore, Hirsch's claim that each sample was replicated to
> ensure that the virus was functional is simply not true."
We appreciate the interpretation of Ian Whalley.
On the subject of being "in-the-wild", we contacted Joe Wells at the time
because there were only a few viruses in the wild, yet many anti-virus
companies were talking about other viruses as if they were in the wild. We
could not do an "in the wild" test since there were only 2 or 3 at the time.
Joe's response was that anti-virus companies were simply not reporting
viruses in a timely manner. (We believe this is an important responsiblity
of anti-virus researchers and encourage them to do same.) We went with
the viruses that were being talked about by anti-virus vendors that were
believed to be "in-the-wild."
On the subject of the Format.C trojan, we of course did not replicate this
as it is not a virus, it is a trojan. We included it nevertheless because it
was mentioned on virtually every anti-virus companies anti-virus web
page despite the fact that it doesn't fit the definition of a true virus.
Format.C is also included in the Secure Computing study that McAfee
references. We believe that such common trojans and Excel viruses
should be addressed by anti-virus developers.
> "While many third party tests are conducted under reasonable
> conditions using appropriate standards and controls, many such tests
> are flawed in a variety of ways," said Peter Tippet, president of
> the NCSA (National Computer Security Association --
> ncsa.com). "Prior to accepting the results from such
> third-party testing, consumers should ask several questions:
> 1. Is the testing organization truly independent, or is there a
> hidden agenda which has motivated the test? Tests conducted by
> vendors, resellers, or others who have a vested interest in the
> outcome should be weighed carefully. 2. Does the tester use the
> latest version of each product included in the test? Side by side
> comparisons of outdated products are misleading and unfair. 3. Was
> the test suite accurate and comprehensive using a real-world suite of
> viruses? In order for the test results to reach an accurate
> conclusion about the anti-virus software's detection capabilities,
> tests should be conducted against viruses that are actually in the
> wild. The NCSA's product certification program carefully addresses
> each of the above three concerns."
We agree with the above questions and believe we can answer affirmatively
to each of the questions except at the time, there were not (and are still not)
enough macro viruses to consider ours or any test suite at the time as a
"real-world" suite. (The In-The-Wild list is only as good as the degree of
reporting that is being done to it.) Note: At the time we did the study, the
latest "In-The-Wild" list that NCSA had reference to was March, 1996 which
was at least 3 months old.
----------------------cut to separate issues-----------
McAfee press release stated:
>
> Bolstering
> McAfee's charge is a new independent Macro virus test released this
> week from Secure Computing Magazine, which shows that Symantec only
> detects 48% of known macro viruses.
With members in CARO, S&S's Secure Computing's "known macro viruses"
rely to a degree on viruses received through its CARO members. As a result,
this poses an inherent flaw in the Secure Computing Magazine test since the
same CARO people who send and receive samples from Secure Computing
also send and receive the same samples to McAfee via their CARO member.
Since Symantec does not have an employee on staff that is a member of
CARO, it therefore does not receive CARO viruses and this very fact
automatically biases the tests toward companies with CARO members in the
Secure Computing test and against companies that do not have CARO members.
> In the independent test
> published this week by Secure Computing, however, Norton AntiVirus
> was shown to detect only 48% of the 42 known macro viruses and was
> shown to remove only 46%. McAfee's VirusScan, which earned the
> highest score in the test, was shown to detect and remove 81% of the
> 42 known Macro viruses.
In addition to the flaw stated above, we have 9 samples of viruses that the
Secure Computing study claimed NAV could not detect that it, in fact does
detect and clean our samples. Additionally we found 2 viruses that the
study claimed McAfee did not detect, that it did in fact detect and clean.
This points out a flaw in macro virus tests in general that it depends on
which particular sample or variant of the "foo-bar" macro virus you have
and that it cannot be told Yes/No on detection across the board for any
given "named" virus. Taking these factors together, the Secure Computing
study flaw is multiplied due to the fact that the same CARO viruses were
used by Secure Computing that were received by McAfee (via CARO) and
NOT by Symantec or other companies not having a member of CARO on
staff. If there were enough "in-the-wild" macro viruses to make a test,
this would be the solution, but at this time there are only a few.
> Each vendor's product was the
> most current at the time and individual results were reviewed by each
> vendor prior to publication.
As was all NH&A studies. The following discrepancies are noted in the
Secure Computing study: Key: S=Symantec, M=McAfee Under
Discrepancy, find how our samples disagreed w/Secure Computing's.
> Macro Virus McAfee VirusScan Symantec Norton Anti-Virus Discrepancy
> v.2.05 v.95.0a
> Detect Repair Detect Repair
>
> Antidmv No No Yes Yes M Yes on one sample "Date"
> Atom No No No No S Yes on one sample NAV
> Birthday No No No No
> Boom Yes Yes Yes Yes
> Concept.e Yes Yes No No S Yes on sample Concept.f
> Concept.her Yes Yes No No
> Colors Yes Yes Yes Yes
> Colors.b Yes Yes Yes Yes
> Colors.c Yes Yes Yes Yes
> Colors.d Yes Yes Yes Yes
> Concept Yes Yes Yes Yes
> Concept.b Yes Yes Yes Yes
> Concept.c Yes Yes No No
> Concept.d Yes Yes No No
> Divini Yes Yes No No S Yes on Divina
> DMV Yes Yes Yes Yes
> Doggie Yes Yes No No
> Extra Yes Yes No No
> Formatc -
> (Trojan) No No Yes Yes
> Friendly Yes Yes No No S Yes
> Goldfish Yes Yes No No
> Hot Yes Yes Yes Yes
> Imposter Yes Yes Yes Yes
> Imposter.a Yes Yes Yes Yes
> Imposter.b Yes Yes Yes Yes
> Irish Yes Yes Yes Yes
> Killdll Yes Yes No No
> Laroux No No Yes
> MDMA Yes Yes Yes Yes
> NOP Yes Yes Yes Yes
> Npad Yes Yes No No
> Nuclear Yes Yes No No S Yes
> Nuclear.b Yes Yes No No S Yes
> Phantom Yes Yes No No S Yes
> Polite No No No No
> Reflex No No No No M Yes
> Telefoni Yes Yes No No S Yes
> Wazzu Yes Yes No No S Yes
> Wazzu.a Yes Yes Yes Yes
> Wazzu.b Yes Yes No No
> Wassu.c Yes Yes No No
> Xenixos Yes Yes Yes Yes
>
> Total Macro
> Viruses Detected/
> Removed 34 34 20 19
> Percentage 81% 81% 48% 46%
> *T
----------------
Final points:
1. CARO members provide a valuable service to the anti-virus community
by coordinating the naming of viruses and sharing information among
themselves. Let's hope CARO members will provide anti-virus companies
such as Symantec, Trend and others with virus samples of any in-the-wild
viruses on a timely basis and will report such viruses to Joe Wells. The
user community will benefit from the increased detection rate and increased
features that will come as a conseqence.
2. Competition improves products. Let's keep up the competition but
be more careful and considerate about how we present the information
about our products. Let's focus our attention on product features and
useability and let's discourage virus writing whenever possible.
--
Best regards,
Norman Hirsch Fax: 212-304-9759
NH&A BBS: 212-304-9759,,,,,,,3
577 Isham St. # 2-B CompuServe: 72115,661
New York, NY 10034 Internet: nhirsch@nha.com
Phone: 212-304-9660 URL: nha.com |
