AAMC 2nd Comments on OMB
Information Quality Guidelines
October 29, 2001
Brooke J. Dickson
Office of Information and Regulatory Affairs
Office of Management and Budget
Washington, DC 20503
By fax: 202-395-5167
Re: Guidelines for Ensuring and Maximizing the Quality,
Objectivity, Utility, and Integrity of Information Disseminated
by Federal Agencies, final guidelines with request for comments.
[66 FR 49718-25]
Dear Ms. Dickson:
The Association of American Medical Colleges (AAMC) has
closely reviewed the final guidelines on information quality
published in the Federal Register on October 1 and is pleased
to respond to the additional request for public comments concerning
the interim final standard, "capable of being substantially
reproduced." The AAMC commends the Office of Management and
Budget for its responsive efforts to address the scientific
community's deep concerns for the draft guidelines while fulfilling
its statutory obligations.
The OMB has wisely provided federal science and public health
agencies flexibility in adapting these standards. AAMC is
concerned that some remaining ambiguity in the guidelines
may permit interpretations that, we believe, are contrary
to OMB's intent for implementation. For example, the guidelines
clearly do not pertain to information published or disseminated
by scientists working on federal research grants at non-federal
institutions. The notice declares, "Our guidelines do not
seek to impose new standards on [scientific and statistical]
communities." Yet, the guidelines might be interpreted as
applying to scientists who are federal employees in research
programs within NIH, the CDC, Department of Veterans Affairs,
or other "intramural" research programs, and who publish and
communicate their research findings in the same manner as
do their colleagues in the private sector. It is also open
to interpretation whether the guidelines affect research communications
by non-federal scientists working on federal contracts. The
AAMC believes that none of these categories are intended to
fall under the guidelines and urges that these provisions
be clarified in a future issuance of the guidelines or in
plans for implementation by federal agencies.
Our following comments focus as requested on the standard
"capable of being substantially reproduced" detailed in paragraphs
V.3.B, V.9, and V.10 of the guidelines.
[V.3.B.i] If the results have been subject to formal, independent,
external peer review, the information can generally be considered
of acceptable objectivity.
The AAMC appreciates this language and strongly endorses
the inclusion of this standard, which allows NIH, CDC, FDA,
and other federal science and health agencies to continue
to align information dissemination practices with standards
of excellence internationally recognized by the scientific
community. We do not see any need to alter the standard of
paragraph V.3.B.i.
[V.3.B.ii] In those situations involving influential scientific
or statistical information, the results must be capable of
being substantially reproduced, if the original or supporting
data are independently analyzed using the same models….
The AAMC is troubled by this provision. The background discussion
of this section plainly states that this standard "is added
as a quality above and beyond some peer review quality standards"
[emphasis added]. The latter statement is ambiguous and we
are concerned that the word "some" suggests that OMB is implying
that other peer review quality standards do establish whether
results are capable of being substantially reproduced. In
fact, customary peer review practices never attempt to assert
or demonstrate that results are capable of being substantially
reproduced from original or supporting data.
The standard proposed here, above and beyond peer review,
is unnecessary. Consider that the most cited and elite journals
in medicine and science require no similar standard to supplement
peer review for assessing scientific merit in publication
although they often convey highly influential-even "life and
death"-information. In fact, in circumstances in which research
findings may have important and immediate implications for
public health or safety, medical journals or public health
agencies may choose to expedite the dissemination of findings
following peer review but prior to formal publication. In
other instances, Data Safety and Monitoring Boards may halt
clinical trials if data indicate that experimental treatments
are in fact maleficent or, conversely, are so clearly beneficial
as to make it unethical to deny treatment to the control population.
Federal health agencies often proceed without delay to communicate
such actions widely to health care providers, and often the
public, even prior to peer review or publication.
The imposition of the overarching standard in V.3.B.ii thus
could delay or obstruct the release of critical information
or otherwise alter long-standing practices for communicating
findings important to public health and safety, whether from
conclusive research or, as in the case above, from discontinued
studies. For these reasons, the AAMC recommends again that
the standard not be required of information that has already
met the peer review standard in section V.3.B.i. At the very
least, the standard must be tailored so as not to prohibit
release of vital public health information.
[V.3.B.iii] Making the data and models publicly available
will assist in determining whether analytical results are
capable of being substantially reproduced. However, these
guidelines do not alter the otherwise applicable standards
and procedures for determining when and how information is
disclosed. Thus, the objectivity standard does not override
other compelling interests, such as privacy, trade secret,
and other confidentiality protections.
Although we have no quarrel with this section as an observation
applicable to those research studies that involve large databases
and that may not be feasible to replicate, we do not wish
to see it interpreted as a general precondition of agency
dissemination of scientific information.
OMB is certainly correct to recognize the inviolability
of privacy and confidentiality protections within the guidelines.
If paragraph V.3.B.iii remains in the final guidelines, OMB
should explicitly recognize that the individuals, institutions,
or agencies accountable for securing privacy, confidentiality,
and other protections have the final authority to determine
that such information is kept secure from disclosure.
[Definitions, V.9] "Influential" when used in the phrase
"influential scientific or statistical information" means
the agency expects that information in the form of analytical
results will likely have an important effect on the development
of domestic or international government or private sector
policies or will likely have important consequences for specific
technologies, substances, products, or firms.
The AAMC fully appreciates that published research findings
often form the basis for regulation or other federal actions
with far-reaching impacts on commerce or society, and that
the government may, therefore, wish to require that such research
findings have been confirmed before driving regulatory or
other decisions. But confirmation of results is not in every
instance equivalent to "reproducing" results or asserting
"capability to reproduce" results, and the continued inclusion
of an overarching review standard is confusing and remains
troubling.
AAMC respects the authority of OMB to require agencies to
establish and make known the explicit criteria by which they
determine the quality of information used in performing their
duties. The AAMC further acknowledges the authority of OMB
to set guidelines that agencies must follow in establishing
their criteria. However, these guidelines should not prevent
the agencies from developing criteria that are most appropriate
to their areas of responsibility. The definition of paragraph
V.9 would trigger application of the superposed standard,
"capable of being substantially reproduced," to much of the
research generated by federal Public Health Service agencies.
The combination of V.3.B.ii and V.9 would prevent these agencies
from fulfilling their mission of timely dissemination of vital
public health information.
At best, guidelines governing information dissemination,
at least for scientific information, can effectively aspire
to standards of excellence no higher than those employed by
the scientific community.
Sincerely,
Jordan J. Cohen, M.D.
|
