Frank (first of two posts):
I been doing some reading on primary screening, and have come to the
conclusion that it is by no means certain that the AutoPap System is
going to be approved for primary screening in the near future. There
seems to be a lot of nagging questions within the community of
cytotechnologists. People are beginning to challenge the data being
put forth by the industry.
The American Society for Cytotechnology has started a new readers'
forum called "Truth in Advertising Forum". Here is what they have to
say about the new forum in the Oct 13, 1997 ASCT Journal of
Cytotechnology:
®With this issue, ASCT News begins a readers' forum focusing on
allegedly unproven, distorted claims by some of the advertisers of
automated cytology instruments. We intend to explore these claims and
examine what cytology professionals can do to counteract them. Barry
Staymates, an experienced and highly regarded California
cytotechnologist, has written the lead-off article in this series.
Although Mr. Staymates names no instrument by its actual name and uses
a light touch at times, he effectively makes his point. ASCT opens the
debate to others. Please write to ASCT News. Share your experience,
your opinion, and your ideas as they relate to advertising. Both pro
and con arguments are welcome and will be aired. Mail submissions to:
Jean Triol, "Truth in Advertising Forum", P.O. Box 367, Somers, MT
59932.¯
Here is the reference article:
® Truth in Advertising
I Can See Clearly Now!
by Barry Staymates
I don't like numbers! I survived mathematics rather than passed! I
still cringe each year, when the next child in line approaches with
that darn book and asks, "Dad, if Cliff and Maria live 100 miles
apart, they drive for two hours, one 10 mph faster than the
other......" my head starts to cloud, I hear the sounds of chalk
tap-dancing across a college blackboard, the professor turning to us
(but looking at me!), from the indecipherable mess of equations,
sleeves covered in white dust, delicately scratching his head and
saying, "As you can clearly see......."
So, now, here I sit staring at one of the reprint articles included in
my beautiful packet given me by the Pinochio 100 representative.
Because it is in the packet, it must be proof positive why the
Pinochio 200 is the best automatic screener in the whole world. Boom!
Right off the bat, first paragraph, I read that 50% of 10% is five
times better than the original ten percent. What? Next, there are data
tables, with not just a few numbers, but numbers next to numbers in
parentheses, over more numbers in brackets, all in columns headed by
decimal thresholds carried out to the thousandths' place. I am getting
dizzy, I am tempted to say I believe it, all these numbers can't be
wrong, I will give up my job. I quit! But I have promised myself to
see this through.
I understand the premise of the study. An independent laboratory which
has a significant annual volume of gynecologic cases (150,000)
selected random batches (how many?) of 150 cases from the case load
each day. A total of 3,577 cases were selected. Cases were excluded
from the study if they were: 1) initially recognized as abnormal (90)
or unsatisfactory, 2) multiple slide cases, or 3) "failed" processing
on the Pinochio 300...due to physical limitations (I think I would
like to know what percentage of the total "failed" and why). These
exclusions distilled the final number of normal (WNL) cases to 3,487.
This total was then rescreened both manually and by the Pinochio 400
(perhaps three times at different thresholds?) I am still with the
study, this is a side-by-side comparison between John Henry and the
steam hammer. This is good! The manual rescreen of the 3,487 cases by
Cytotechnologists identified an additional 106 cases which, upon
review were interpreted as not to be within normal limits. I think
this information alone is extremely significant! To me, a
non-statistician, false negative is a simple concept. It is the number
of cases which are initially reported as negative and upon review are
proven to be not negative. This review of a large (3,487) "negative"
series documents a "false negative" rate of 3%. This is considerably
less than the much publicized 5-25% seen in the media. But wait! These
106 cases were reviewed again by two panels of experts: one internal
to the facility and another outside the facility. These two panels
reduced the number of false negatives from 106 to 80 and 86
respectively, lowering further the "false negative rate" of this
excellent laboratory to 2.3% and 2.5%! This is really good!
The authors define "false negative" as satisfactory smears that do not
show an abnormality in the presence of a cervical lesion. Returning to
the 106 "false negative" cases, 51 were categorized as "ASCUS" while
an additional 13 were indicated as showing "AGUS." I could not find
that these 64 cases (60.3% of the false negatives) had tissue
confirmation of the presence of a "cervical lesion." Now this brings
us to a philosophical dilemma rather than a mathematical exercise.
Mathematics always supports philosophy not the reverse. I enjoy
philosophy. We must consider this question. Is a case reclassified as
"not normal", but which does not have a cervical lesion present, or in
which no attempt to document (tissue biopsy) a cervical lesion is
undertaken, or precipitates only a repeat cytology, which should have
occurred anyway, in an organized screening program, considered a
"false negative?" I do not believe these 64 can be labeled "false
negative" cases until proven to have cervical lesion. That leaves us
with 33 LGSIL and 9 HGSIL cases that I assume were confirmed by
subsequent tissue diagnosis. If one calculates the false negative rate
based upon cases called WNL in the presence of a cervical lesion (i.e,
SIL), that rate becomes 1.2% This is remarkable for a screening test
as complex as the Pap smear. This is indeed an excellent laboratory
and its personnel deserve congratulations.
Since the Yin of "false negative" (fn) is used very frequently,
perhaps its Yang, "false positive" could be applied to the data with
equal poetic license. As I read the table detailing the Pinochio 500
sensitivity and review rates, it appears that at the first threshold
(0.604) the machine identified 207 cases as "not normal" or positive.
Of these two-hundred seven positive cases, only 34 (16.4%) were
confirmed as being "not normal" or positive. This equates to a 83.6%
false positive rate. At the highest threshold setting, the absolute
number of not normal cases detected was increased to 50 (47.2%) of the
106, but the number of positive cases sorted by the machine more than
doubled, raising the false positive of this level to 89.3% (415/465).
If I am reading the table correctly, the Pinochio 600 missed
completely 21 of the 33 LGSIL at its lowest threshold setting and
still "failed" to detect 15 of the thirty-three at its most sensitive
setting. Of even greater concern is that of nine cases of high grade
SIL found in the manual rescreen; more than half were completely
missed by the machine regardless of the setting. So at the highest
threshold, the machine threw out 465 cases as positive, which did not
include; five of the 9 cases of High Grade SIL (55.6%), fifteen cases
of the 33 Low Grade SIL (45.5%), and 36 (56.3%) of the ASCUS/AGUS
group identified by the manual QC. Totaling these omissions, the
Pinochio 700 was unable to detect 56 (52.8%) of the so-called false
negative cases even though it looked at every one of them. This is
good?
Returning to the original total of cases selected before the exclusion
process with the QC review data, we can determine the absolute numbers
of SIL in the case population and compare side by side, the manual to
the Pinochio 800 in a simple detection rate. Both contestants
evaluated the slides in an initial setting and again in a 100% QC
review. "Geeves, the tables, if you please..."
Cases of SIL In Study Group
Type Total Cases LGSIL HGSIL Total SIL %SIL
Initial Scrn 3577 70 19 89 2.48
QC Review 3487 33 9 4 1.2
Abs. # SIL 3577 103 28 131 3.67
Now we can look at the ability of both modalities to perform the basic
function of gynecologic cytology--to identify the number of LGSIL and
HGSIL in the case population.
SIL Detection rate of absolute # of SIL in Case Study
LGSIL Detect Rt HGSIL Detect Rt Total SIL Detect Rt
TOTAL 103 - 28 - 131 -
Pinochio 88 85.4 23 82.1 111 84.7
Manual 103 100 28 100 131 100
It has been understood for years by those within the cytology
community that rescreening 10% of those cases initially interpreted as
normal was of little practical benefit. The dilemma has always been a
question of economics and practicality. How much expense and resources
should be added to a screening procedure to raise its predictive value
from 97%, as documented by the authors of this article, to what? 98%?
99%? As illustrated in this paper, the most accurate method of Quality
Control is total manual rescreen of all negatives. That will serve
patient care and confidence to the fullest possible extent. If the
cost estimates of an automated Pap smear being rumored ($45 to $65)
are close, then cost to the patient is no longer a concern. For sixty
dollars per pap smear I believe any laboratory could opt for 100%
manual rescreen of normal smears, assuring the patient of the most
accurate cancer screening test available. Of course, in this price
range access to the test will soon be restricted to those who can
afford it and, probably, need it least. This could easily result in an
increased incidence of cervical cancer because many women will opt not
to have a Pap smear. So to boldly print that "The Pinochio 900 is
superior to randon rescreening in detecting false negatives, dot, dot,
dot" is not a surprise within the profession. To the public, however,
the receivers and procurers of the Pap smear test, this statement is a
revelation and a presumed truth! The implication left unstated, is
that their current level of screening is flawed, less than adequate
and may put them at risk due to the false negative rates of manual
cytology.
Contemplating the data presented in the article, I am not sure at all
that I follow the logic or how the numbers support the conclusions. I
have stated that, like Mark Twain, I am not a statistician, but
depending upon point of view, I believe the headlines could, just as
easily, boldly indicate, "Pinochio 100 fails to detect 52.8% of
abnormal cases. Machine misses completely five of 9 HGSIL cases in
rescreen of negative Pap smears.....Manual review proven more
accurate...." The final resting place of the machine, of course, is
in the spot of primary screener (where many of us are now sitting),
not this kindly augmentation to Quality Control. That it will be
superior to manual evaluation and human decision making has yet to be
proven, but the public already believes that it is. Even my neighbors
have all read about it and ask me about the machine. "Does my
hospital have one yet? Can I send my smear to a lab that has one?" The
automated screener may, some day, turn out to be the most accurate
means of detection of gynecologic cancer and its precursors, and that
will be wonderful. But if it is only pretty good, or if the parameters
of information cytotechnologists now look for and must incorporate
into the microscopic evaluation are deleted to accommodate what the
machine cannot do, the public has been foiled and we have been
eliminated unjustly as a work force. A technology, no matter how
impressive, that decreases accuracy and increases expenses may not be
so much innovative as it is creative.
As one ponders the article, it is stated that one category of cases
excluded were those initially recognized as abnormal. I am unclear
whether the Pinochio also successfully analyzed the initial number of
3577 cases in a "primary" mode, recognizing these initial 90 cases of
SIL and invasive carcinoma or not. I believe that is implied. If that
is so, it follows that the Pinochio 100 also missed the 106 "false
negative" cases in the original evaluation, as did the
cytotechnologists. Then, in the 100% rescreen the machine could not
detect over half of the cases identified by manual rescreen as
false negative. Can it be presumed from this data that as a primary
screener the Pinochio 100 will itself leave 17.8% (5/28) of the high
grade SIL, 14.5% (15/103) of low grade SIL and even larger percentages
of ASCUS/AGUS in the WNL category as it did in this study? It would
appear that the potential problem with the Pinochio may be not only
the significant false positive to positive ratio, but rather with the
number of "not negative" cases left in the WNL category. What
percentage of the machine's WNL cases will have to be rescreened and
by whom?
Statements become facts in two ways: first if they are proven correct,
and second, if they are never challenged.¯
cytopathnet.org
Cisco |
