1972
Goodenough
DJ, Rossmann K, Lusted LB. Radiographic applications of signal
detection theory. Radiology 105: 199-200, 1972.
1973
Goodenough
DJ, Metz CE, Lusted LB. Caveat on use of the parameter d' for
evaluation of observer performance. Radiology 106: 565-566, 1973.
Metz CE,
Goodenough DJ. On failure to improve observer performance with scan
smoothing: a rebuttal (letter to the editor). J. Nucl. Med. 14:
873-876, 1973.
Metz CE,
Goodenough DJ, Rossmann K. Evaluation of receiver operating
characteristic curve data in terms of information theory, with
applications in radiography. Radiology 109: 297-303, 1973.
1974
Goodenough
DJ, Metz CE. Effect of listening interval on auditory detection
performance. J. Acoust. Soc. Am. 55: 111-116, 1974.
Goodenough
DJ, Rossmann K, Lusted LB. Radiographic applications of receiver
operating characteristic (ROC) analysis. Radiology 110: 89-95, 1974.
1975
Goodenough
DJ, Metz CE. Implications of a "noisy" observer to data processing
techniques. In: Information Processing in Scintigraphy (C Raynaud and
AE Todd-Pokropek, eds.). Orsay, France: Commissariat à l’Energie
Atomique, Département de Biologie, Service Hospitalier Frédéric
Joliot, 1975, pp. 400-419.
Metz CE,
Goodenough DJ. Quantitative evaluation of human visual detection
performance using empirical receiver operating characteristic curves.
In: Information Processing in Scintigraphy (CE Metz, SM Pizer, and GL
Brownell, eds.). Oak Ridge, Tennessee: USERDA Tech. Info. Center
(Publication CONF-73-0687), 1975, pp. 140-152.
Metz CE,
Starr SJ, Lusted LB, Rossmann K. Progress in evaluation of human
observer visual detection performance using the ROC curve approach.
In: Information Processing in Scintigraphy (C Raynaud and AE Todd-Pokropek,
eds.). Orsay, France: Commissariat à l’Energie Atomique, Département
de Biologie, Service Hospitalier Frédéric Joliot, 1975, pp. 420-439.
Starr SJ,
Metz CE, Lusted LB, Goodenough DJ. Visual detection and localization
of radiographic images. Radiology 116: 533-538, 1975.
1976
Metz CE,
Starr SJ, Lusted LB. Observer performance in detecting multiple
radiographic signals: prediction and analysis using a generalized ROC
approach. Radiology 121: 337-347, 1976.
1977
Metz CE.
An overview of some measures of image quality. Proc. SPIE 127: 4-5,
1977.
Metz CE.
Empirical evaluation of diagnostic medical images using Receiver
Operating Characteristic (ROC) analyses. In: Advances in the
Psychophysical and Visual Aspects of Image Evaluation (RP Dooley,
ed.). Washington, D.C.: SPSE, 1977, pp. 111-114.
Metz CE,
Starr SJ, Lusted LB. Quantitative evaluation of medical imaging. In:
Medical Radionuclide Imaging, Volume 1. Vienna: International Atomic
Energy Agency, 1977, pp. 491-504.
Metz CE,
Starr SJ, Lusted LB. Quantitative evaluation of visual detection
performance in medicine: ROC analysis and determination of diagnostic
benefit. In: Medical Images: Formation, Perception and Measurement (GA
Hay, ed.). London: John Wiley & Sons, 1977, pp. 220-241.
Starr SJ,
Metz CE, Lusted LB. Comments on generalization of Receiver Operating
Characteristic analysis to detection and localization tasks (letter to
the editor). Phys. Med. Biol. 22: 376-379, 1977.
1978
Metz CE.
Basic principles of ROC analysis. Sem. Nucl. Med. 8: 283-298, 1978.
1979
Metz CE.
Applications of ROC analysis in diagnostic image evaluation. In: The
Physics of Medical Imaging: Recording System Measurements and
Techniques (AG Haus, ed.). New York: American Institute of Physics,
1979, pp. 546-572.
Metz CE.
Evaluation of medical imaging in terms of receiver operating
characteristic (ROC) curves and decision analysis. Postepy Fizyki
Medyczney 14: 47-58, 1979.
1980
Metz CE.
Introduction to observer performance tests. In: Optimization of Chest
Radiography. Rockville, Maryland: Bureau of Radiological Health (HHS
Publication [FDA] 80-8124), 1980, pp. 8-15.
Metz CE,
Kronman HB. A test for the statistical significance of differences
between ROC curves. In: Information Processing in Medical Imaging (R
DiPaola and E Kahn, eds.). Paris: INSERM (Vol. 88), 1980, pp. 647-660.
Metz CE,
Kronman HB. Statistical significance tests for binormal ROC curves.
J. Math. Psych. 22: 218-243, 1980.
1984
Loo L-N,
Doi K, Metz CE. A comparison of image quality indices and observer
performance in the radiographic detection of nylon beads. Phys. Med.
Biol. 29: 837-856, 1984.
Metz CE,
Wang P-L, Kronman HB. A new approach for testing the significance of
differences between ROC curves measured from correlated data. In:
Information Processing in Medical Imaging (F Deconinck, ed.). The
Hague: Nijhoff, 1984, pp. 432-445.
1985
Metz CE.
Evaluation of image quality by ROC analysis: concepts, techniques, and
future possibilities. Jpn. J. Radiol. Technol. 4: 990-1002, 1985.
Loo L-N,
Doi K, Metz CE. Reply to "Importance of internal noise in models of
observer performance" (letter to the editor). Phys. Med. Biol. 30:
267-268, 1985.
Wagner RF,
Metz CE, Brown DG. Signal detection theory and medical image
assessment. In: Recent Developments in Digital Imaging (K Doi, L Lanzl,
and P-J Lin, eds). New York: American Institute of Physics, 1985, pp.
39-59.
1986
Metz CE.
ROC methodology in radiologic imaging. Invest. Radiol. 21: 720-733,
1986.
Metz CE.
Statistical analysis of ROC data in evaluating diagnostic performance.
In: Multiple Regression Analysis: Applications in the Health Sciences
(D Herbert and R Myers, eds.). New York: American Institute of
Physics, 1986, pp. 365-384.
1988
Metz CE.
Current problems in ROC analysis. In: Proceedings of the Chest Imaging
Conference 1987 (WW Peppler and AA Alter, eds). Madison, Wisconsin:
Department of Medical Physics, University of Wisconsin-Madison, 1988,
pp. 315-336.
1989
Metz CE.
Quality of the observed image. ICRU News 2: 20-23, 1989.
Metz CE.
Some practical issues of experimental design and data analysis in
radiological ROC studies. Invest. Radiol. 24: 234-245, 1989.
1990
Begg CB,
Metz CE. Consensus diagnoses and "gold standards" (letter to the
editor). Med. Decis. Making 10: 29-30, 1990. [Erratum: Med. Decis.
Making 10: 149, 1990.]
Gur D,
Rockette HE, Good W, Slasky BS, Cooperstein LA, Straub WH, Obuchowski
NA, Metz CE. Effect of observer instruction on ROC study of chest
images. Invest. Radiol. 25: 230-234, 1990.
Metz CE.
Some applications of ROC analysis in digital radiology. In: Jpn. J.
Med. Imaging and Info. Sci. 7: 14-16,
1990.
Rockette
HE, Gur D, Cooperstein LA, Obuchowski NA, King JL, Fuhrman CR, Tabor
EK, Metz CE. Effect of two rating formats in multi-disease ROC study
of chest images. Invest. Radiol. 25: 225-229, 1990.
Rockette
HE, Obuchowski N, Metz CE, Gur D. Statistical issues in ROC curve
analysis. Proc. SPIE 1234: 111-119,
1990.
Straub WH,
Rockette HE, King JL, Obuchowski NA, Good WA, Feist JH, Good BC, Metz
CE. Training observers for receiver operating characteristic (ROC)
studies. Proc. SPIE 1234: 126-130, 1990.
1991
King JL,
Gur D, Rockette HE, Curtin HD, Obuchowski NA, Thaete FL, Britton CA,
Metz CE. Radiologists’ confidence in detecting abnormalities on chest
images and their subjective judgments of image quality. Proc. SPIE
1446: 268-275, 1991.
Metz CE.
ROC methodology for the evaluation of PACS. In: A New Horizon in
Medical Physics and Biomedical Engineering (H. Abe, K. Atsumi, T.
Iinuma, M. Saito, and M. Inoue, eds.). Amsterdam: Elsevier Science
Publishers, 1991, pp. 89-97
.
1992
Dorfman
DD, Berbaum KS, Metz CE. Receiver operating characteristic rating
analysis: generalization to the population of readers and patients
with the jackknife method. Invest. Radiol. 27: 723-731, 1992.
Metz CE.
Evaluation of medical images. In: The Formation, Handling, and
Evaluation of Medical Images (AE Todd-Pokropek and MA Viergever,
eds.). Berlin: Springer-Verlag, 1992, pp. 277-302.
Metz CE,
Shen J-H. Gains in accuracy from replicated readings of diagnostic
images: prediction and assessment in terms of ROC analysis. Med. Decis.
Making 12: 60-75, 1992.
Rockette
HE, Gur D, Metz CE. The use of continuous and discrete confidence
judgments in receiver operating characteristic studies of diagnostic
imaging techniques. Invest. Radiol. 27: 169-172, 1992.
1993
Metz CE.
Quantification of failure to demonstrate statistical significance: the
usefulness of confidence intervals. Invest. Radiol. 28: 59-63, 1993.
1994
Metz CE.
Forward. In: Fundamentals and Applications of ROC Analysis. Kyoto:
Japanese Society of Radiological Technology, 1994.
Metz CE.
Recent progress in ROC analysis. Jpn. J. Radiol. Technol. 50:
1915-1922, 1994.
Nishikawa
RM, Giger ML, Doi K, Metz CE, Yin FF, Vyborny CJ, Schmidt RA. Effect
of case selection on the performance of computer-aided detection
schemes. Med. Phys. 21: 265-269, 1994.
1995
Dorfman
DD, Metz CE. Multi-reader multi-case ROC analysis: comments on Begg’s
commentary. Academic Radiol. 2 (Supplement 1): S76-S78, 1995.
Metz CE.
Evaluation of radiologic imaging systems by ROC analysis. Med. Imag.
Inform. Sci. 12: 113-121, 1995.
Metz CE,
Wagner RF, Doi K, Brown DG, Nishikawa RN, Myers KJ. Toward consensus
on quantitative assessment of medical imaging systems. Med. Phys. 22:
1057-1061, 1995.
1996
Halpern EJ,
Alpert M, Krieger AM, Metz CE, Maidment AD. Comparisons of ROC curves
on the basis of optimal operating points. Academic Radiology 3:
245-253, 1996.
Jiang Y,
Metz CE, Nishikawa RM. A receiver operating characterisitc partial
area index for highly sensitive diagnostic tests. Radiology 201:
745-750, 1996.
Metz CE.
Evaluation of digital mammography by ROC analysis. In: Digital
Mammography ‘96 (K Doi, ML Giger, RM Nishikawa, RA Schmidt, eds.).
Amsterdam: Elsevier Science (Excerpta Medica International Congress
Series, No. 1119), 1996, pp. 61-68.
1997
Dorfman
DD, Berbaum KS, Metz CE, Lenth RV, Hanley JA, Dagga HA. Proper ROC
analysis: the bigamma model. Academic Radiology 4: 138-149, 1997.
Metz CE.
Receiver operating characteristic (ROC) analysis in medical imaging.
ICRU News (June 1997 issue; no vol. #): 7-16, 1997.
Pan X,
Metz CE. Non-iterative methods and their noise characteristics in 2D
SPECT image reconstruction. IEEE Trans. Nucl. Sci. 44: 1388-1397,
1997.
Pan X,
Metz CE. The "proper" binormal model: parametric ROC curve estimation
with degenerate data. Academic Radiology 4: 380-389, 1997.
Roe CA,
Metz CE. The Dorfman-Berbaum-Metz method for statistical analysis of
multi-reader, multi-modality ROC data: validation by computer
simulation. Academic Radiology 4: 298-303, 1997.
1998
Metz CE,
Herman BA, Roe CA. Statistical comparison of two ROC-curve estimates
obtained from partially-paired datasets. Med. Decis. Making 18:
110-121, 1998.
Metz CE,
Herman BA, Shen J-H. Maximum-likel ihood estimation of ROC curves from
continuously-distributed data. Statist. Med. 17: 1033-1053, 1998.
Roe CA,
Metz CE. Variance-component modeling in the analysis of receiver
operating characteristic index estimates. Academic Radiology 4:
587-600, 1997.
1999
Engelmann
R, MacMahon H, Metz CE, Hoffmann K, Doi K. Implementation of a
technique for performing real-time ROC observer studies to examine the
efects of CAD schemes on the performance of radiologists. In:
Computer-Aided Diagnosis in Medical Imaging (K Doi, H MacMahon, ML
Giger and KR Hoffmann, eds.). Amsterdam: Elsevier Science (Excerpta
Medica International Congress Series, Vol. 1182), pp. 71-76, 1999.
Metz CE.
Evaluation of CAD methods. In: Computer-Aided Diagnosis in Medical
Imaging (K Doi, H MacMahon, ML Giger and KR Hoffmann, eds.).
Amsterdam: Elsevier Science (Excerpta Medica International Congress
Series, Vol. 1182), pp. 543-554, 1999.
Metz CE,
Pan X. "Proper" binormal ROC curves: theory and maximum-likelihood
estimation. J. Math. Psych. 43: 1-33, 1999.
2000
Metz CE.
Fundamental ROC analysis. In: Handbook of Medical Imaging, Vol. 1:
Physics and Psychophysics (J Beutel, H Kundel and R Van Metter, eds.).
Bellingham, WA; SPIE Press, 2000, pp. 751-769.
Metz CE.
Software for ROC Analysis. Comm. Imaging Jap. Soc. Radiol. Technol.
23: 2-9, 2000
2001
Beiden SV,
Wagner RF, Campbell G, Metz CE, Chan H-P, Nishikawa RM, Schnall MD,
Jiang Y. Analysis of variance components in multi-reader studies of
computer-aided diagnosis with different tasks. Proc. SPIE 4324:
167-176, 2001
Beiden SV,
Wagner RF, Campbell G, Metz CE, Jiang Y. Components-of-variance models
for random-effects ROC analysis: The case of unequal variance
structures across modalities. Academic Radiol. 8: 605-615, 2001
Beiden SV,
Wagner RF, Campbell G, Metz CE, Jiang Y, Chan H-P. Multiple-reader
studies, digital mammography, computer-aided diagnosis – and the Holy
Grail of imaging physics (II). Proc. SPIE 4320: 619-626, 2001
Jiang Y,
Metz CE. An optimal method for combining two correlated diagnostic
assessments with application to computer-aided diagnosis. Proc. SPIE
4324: 177-183, 2001
Wagner RF,
Beiden SV, Metz CE. Continuous vs. categorical data for ROC analysis:
Some quantitative considerations. Academic Radiol. 8: 328-334, 2001.
2002
Edwards DC, Kupinski MA, Metz CE,
Nishikawa RN. Maximum-likelihood fitting of FROC curves under an
initial-detection-and-candidate-analysis model. Med. Phys. 29:
2861-2870, 2002.
Wagner RF, Beiden SV, Campbell G, Metz CE,
Sachs WM. Assessment of medical imaging and computer-assist systems:
Lessons from recent experience. Academic Radiol. 8: 1264-1277, 2002.
2003
Edwards DC, Lan L, Metz CE, Giger ML,
Nishikawa RM. Bayesian ANN estimates of three-class ideal observer
decision variables for classification of mammographic masses. Proc.
SPIE 5034: 474-482, 2003.
2004
Edwards DC, Lan L, Metz CE, Giger ML,
Nishikawa RM. Estimating three-class ideal observer decision variables
for computerized detection and classification of mammographic mass
lesions. Med. Phys. 31: 81-90, 2004.
Edwards DC, Metz CE, Kupinski MA. Ideal
observers and optimal ROC hypersurfaces in N-class classification.
IEEE Trans. Med. Imaging 23: 891-895, 2004.
Edwards DC, Metz CE, Nishikawa RM,
Hypervolume under the ROC hypersurface of a "near-guessing" ideal
observer in a three-class classification task. Proc. SPIE 5372:
128-137, 2004.
2005
Edwards DC, Metz CE. Review of several
proposed three-class classification decision rules and their relation
to the ideal observer decision rule. Proc. SPIE 5749: 128-137,
2005.
Edwards DC, Metz CE. Evaluating Bayesian
ANN estimates of ideal observer decision variables by comparison with
identity functions. Proc. SPIE 5749: 174-182, 2005.
Jiang Y, Sacks W, Metz CE. Effect of
observer inattention in a detection task on ROC analysis. Proc SPIE
5749:114-117, 2005.
2006
Edwards DC, Metz CE. Optimization of an
ROC hypersurface constructed only from an observer's within-class
sensitivities. Proc. SPIE 6146: 61460A1-61460A7, 2006.
Edwards DC, Metz CE. Analysis of proposed
three-class classification decision rules in terms of the ideal
observer decision rule. J. Math. Psych. (in press), 2006.
He X, Metz CE, Tsui BMW, Links JM, Frey
EC.Three-class ROC analysis - A decision theoretic approach under the
ideal observer framework. IEEE TRANSACTIONS ON MEDICAL IMAGING 25 (5):
571-581, 2006
Metz CE. Receiver operating characteristic
(ROC) analysis: a tool for quantitative evaluation of observer
performance and imaging systems. JACR 3: 413-422, 2006
