AAMC Comments on OMB Information
Quality Guidelines
August 13, 2001
Ms. Brooke Dickson
Office of Information and
Regulatory Affairs
Office of Management and Budget
Washington, DC 20503
[66 FR 34489] Proposed Guidelines for Ensuring and Maximizing
the Quality, Objectivity, Utility, and Integrity of Information
Disseminated by Federal Agencies
Dear Ms. Dickson:
I am grateful for this opportunity to comment on behalf
of the Association of American Medical Colleges (AAMC) concerning
the proposed guidelines on information quality. As indicated
by the discussion section of the Federal Register notice,
the proposed guidelines are extremely broad in scope. The
AAMC's comments focus on the guidelines' implications for
medical and scientific research, based on our experience as
the national representative of all 125 U.S. medical schools,
more than 400 teaching hospitals, and 92 academic and professional
societies, comprising more than 91,000 faculty members. Our
institutions perform over half of the extramural research
sponsored by the National Institutes of Health (NIH), the
nation's leading agency supporting biomedical and basic science
research, as well as conduct research sponsored by other Public
Health Service and federal agencies.
From this vantage, the AAMC is deeply concerned that the
proposed guidelines would impede communication of valid research
findings from scientific and health research and, contrary
to federal intent, actually diminish the quality, objectivity,
utility, and integrity of federal information by imposing
inappropriate standards and layers of review that supersede
time-honored processes for scientific review and validation.
Background
Congress directed the Office of Management and Budget to
promulgate information quality guidelines in Section 515 of
the Treasury and General Government Appropriations Act for
Fiscal Year 2001 (P.L. 106-554). As described in the Federal
Register notice, Congress' intent in Section 515 is to ensure
and maximize the quality of information disseminated by the
federal government, particularly given reasonable concerns
that the use of the Internet has greatly increased the volume
of governmental information available to the public. Neither
Congress in writing this legislation nor the OMB in drafting
the Federal Register notice addressed any specific concerns
about the quality of scientific research sponsored, performed,
cited, or disseminated by the federal government.
The U.S. Public Health Service (PHS) is prominent both for
sponsoring and conducting biomedical and other health-related
research in order to promote the health of the public. The
information created, collected, and disseminated by the National
Institutes of Health, the Centers for Disease Control and
Prevention, and their sister PHS agencies is regarded as the
most highly influential and respected public health guidance
in the world. The federal government must take particular
caution that its new proposed guidelines, so broadly cast,
not impair the functioning of this remarkably beneficial and
effective system.
Peer Review
The statutory language in Section 515 requires OMB to provide
policy and procedural guidance to federal agencies for ensuring
and maximizing the quality, objectivity, utility, and integrity
of information disseminated by the Federal Government. The
proposed guidelines, which do not define these terms, stipulate
that the terms collectively require information to be presented
in an "accurate, clear, complete, and unbiased manner." In
contrast to the general and often equivocal language used
elsewhere in the notice, the proposed guidelines set a blunt
and highly prescriptive standard for "scientific research
information" within the section on definitions:
"….With respect to scientific research information, the
results must be substantially reproducible upon independent
analysis of the underlying data." (Section V.1, paragraph
B.ii.a)
This language, which resembles a regulation more than a
guideline, is seemingly reinforced in V.1.A: "…[W]hen reproducibility
and transparency of the information are relevant for assessing
the information's usefulness from the public's perspective,
the agency must take care to ensure that reproducibility and
transparency have been taken into account."
The AAMC strenuously objects to the proposed standard
for scientific research information and urges its removal
from the final guidelines. The gold standard for dissemination
of scientific and medical research information is publication
in a peer-reviewed journal. However, the process of peer review
does not and cannot itself ensure the "reproducibility" of
published data or its interpretation. Rather, that is the
task of the interested scientific community, adequately trained,
appropriately equipped and funded, and properly motivated
to undertake the demanding effort required to reproduce published
scientific findings. When data or findings from the peer-reviewed
literature are contested-and contested competently-the discordant
data and analyses are themselves submitted for peer review
and published as research articles, commentaries, or letters;
and so the process continues until the original findings and
interpretations have become established, modified, or passed
by and discarded. The tension created by this process of discovery,
interpretation, challenge, and rediscovery openly communicated
through the scientific and medical literature is inherent
in the scientific method and fundamental to scientific progress.
In contrast, for OMB to assert reproducibility as the threshold
criterion for "quality" of scientific information would set
a standard far beyond peer review and would essentially prevent
federal agencies from depending, as does the entire scientific
community, on the reliability of the peer reviewed literature.
A result less in the public interest is hard to imagine.
Scientists working in the same areas routinely attempt to
confirm the published findings of their colleagues, whenever
possible, by replicating them. To do so increasingly requires
access to, and proficiency in the use of, particular "research
tools," e.g., constructed cell lines, transgenic animals,
sophisticated instrumentation, or other specialized resources.
In some kinds of scientific studies, e.g., long-term epidemiology
studies, strict reproducibility of the experimental findings
may be limited or precluded by sheer impracticality; consider,
for example, the federally supported Framingham Heart Study
that has been ongoing for five decades. In no instance, however,
can reproducibility be assessed unless the scientific findings
are first published and made known.
In some instances, federal agencies may determine that it
is necessary for protection of public health and safety to
disseminate scientific or health information quickly, without
reliance on peer review or being able to wait for the ordinary,
time consuming scientific processes of substantiation or refutation
to take place. The AAMC believes that clear and transparent
procedures for assessing the quality of such information should
be established by the agencies themselves in close consultation
with the relevant scientific and health communities. In these
instances, coordinating guidance from OMB could be valuable
in ensuring consistency across agencies.
The standard for dissemination of scientific information
is deficient in other respects:
"Independent analyses": The guidelines imply that
there is a single independent and bias-free position from
which to assess the validity of scientific findings. The history
of science proves otherwise. It is axiomatic that each observation
and observer is potentially subject to bias, as is each reviewer
and commentator. A range of error and uncertainty affects
every recorded data element. These errors and uncertainties
are attributable to human fallibility, the physical limitations
of instruments, experimental methodologies, chance, and especially
in biology and medicine, the inherent nature of the subjects
under investigation. It is counterintuitive that federal agency
officials or "affected persons" (discussed below) would categorically
make the most appropriately independent reviewers or referees
of such information, as the proposed guidelines seem to imply.
Scientific peers, versed in the procedures, methods, and issues
under investigation (and who are sufficiently distant from
the researcher and findings under scrutiny), are better able
to provide balanced assessment about the quality and reliability
of reported findings from research.
"Underlying Data": The definition of underlying data,
noticeably absent from the guidelines, is highly problematic,
as was forcefully underscored during the prolonged process
of rule-making on Section __.36(c) of OMB Circular A-110 (referred
to as the Shelby provision) pursuant to P.L. 105-277. Definitions
of data may range from a select set of recorded observations
sufficient to establish the research findings to all records
and materials (raw data) accumulated in the course of investigation.
Most commonly, research data require annotation and formatting
to be useful and are comprehensible only to individuals expert
in the area of investigation. This fact belies the statement
(Section V.1.B.i), in so far as the guidelines relate to the
dissemination of data sources from scientific research: "…[T]he
agency needs to identify the sources of the disseminated information
(to the extent possible, consistent with confidentiality protections),
so that the public can assess for itself whether there may
be some reason to question the objectivity of the sources."
Numerous concerns, in addition to confidentiality, were raised
by the federal proposals pursuant to the Shelby provision
to identify or provide source data. It was after extensive
dialogue with the scientific community and careful consideration
of this matter that OMB wisely decided to limit the definition
of data, as described at length in 64 FR 43786, section III
A and in 64 FR 54926 section II A. The information quality
guidelines should not unnecessarily reopen that well-considered
decision.
Exceeding the Statutory Requirement
In asserting a standard for scientific communication, the
OMB has gone beyond the statutory requirement of Section 515.
Congress did not require OMB to set a standard for scientific
communication, but rather generally to:
"provide policy and procedural guidance to Federal agencies
for ensuring and maximizing the quality, objectivity, utility,
and integrity of information (including statistical information)
disseminated by Federal agencies." [emphasis added].
The specification of "statistical information" demonstrates
that Congress did in fact deliberate on specific types of
federal information for OMB to address and that scientific
research was not singled out for special attention. Nor is
OMB confronted by questions of 515's consistency with other
statutes or policies regarding scientific information that
need to be addressed or resolved in the proposed guidelines.
Neither the Paperwork Reduction Act (44 USC § 3501 et seq.)
nor OMB Circular A-130-upon which the proposed guidelines
build and are designed to be consistent-distinguish scientific
research apart from other types of information, nor do either
require scientific information to meet a separate federal
standard.
Further, the proposed guidelines promulgate a specific standard
for scientific communication that is not reflected in the
National Science Foundation Act, the Public Health Service
Act, nor, to our knowledge, in any other federal legislation
creating or controlling federal research or health programs.
The AAMC thinks it entirely unreasonable to believe that the
Congress, in an appropriations bill (H.R. 5658) not originally
pertaining to a single science or health agency, intended
to propose such a fundamental alteration in the federal policies
governing the communication of scientific research. It would
be far more appropriate to permit the federal agencies overseeing
science and health research programs to address standards
for communication of scientific information in their own guidelines
that are consistent with the totality of their complex statutory
and regulatory requirements.
Mechanisms for Review by "Affected Persons"
As directed by Congress, the proposed guidelines require
that:
As a matter of citizen review, agencies should establish
administrative mechanisms allowing affected persons to seek
and obtain correction of information maintained and disseminated
by the agency that does not comply with these OMB guidelines.
(III.3)
Information from scientific or health-related research that
is in the possession of or disseminated by federal agencies
should not be subject to correction or amendment without stringent
qualified scientific review. The perception that the legislation
and proposed guidelines could provide a mechanism for vested
interests or partisan activists to alter or control federal
dissemination of peer-reviewed information has understandably
alarmed the scientific community, as it should.
We fully concur with the principle put forward by OMB, that
"these administrative mechanisms [for public review] should
be consonant with established agency practice, flexible, and
appropriate to the nature of the disseminated information."
Any requests to amend or alter scientific and technical information
must be subject to a commensurately rigorous level of review
assuring the appropriate scientific expertise. The OMB final
guidelines should not be interpretable as precluding federal
agencies from establishing scientific peer review as the preeminent
mechanism for guaranteeing the quality of scientific information
disseminated by the federal government.
A major concern of the AAMC is that the guidelines include
a public review mechanism and the standard for dissemination
of scientific research information. The two items could appear
conjointly to provide grounds for interested parties to demand
access to underlying data, to compel the government to replicate
research findings (at great expense and with unnecessary delay),
or in other ways impede, discredit, harass or stymie research.
The AAMC does not wish the information quality guidelines
to be construed as an activist's instrument for circumventing
established scientific procedures and assailing research.
Administrative Burden
The AAMC is sensitive to the increasing administrative burden
tasked to PHS agencies. Various federal hiring freezes and
other restrictions have constrained the ability of these agencies
to acquire staff even for core missions of the agency. The
AAMC does not believe that the efforts of public servants
should be further attenuated by supervisory and reporting
tasks that are less than minimally contributive to the agencies'
effectiveness. While the notice conveys OMB's interest to
promulgate guidelines that are consistent with existing agency
practices, the notice provides no estimate of the actual burden
that would be imposed on agencies. Moreover, the administrative
"complaint mechanism" provided in the guidelines (at the direction
of the statute), is an open-ended task, because the number
of public complaints received will be determined as much by
interest group concerns as by the actual quality of the information
disseminated. The OMB should work with federal agencies to
estimate the administrative and cost impact of the information
quality guidelines. The information would be helpful to the
public and to Congress in assessing the merit and practicality
of the guidelines.
Scope of the Guidelines
The AAMC believes that the scope of the proposed guidelines
cannot reasonably be judged to extend to academic or other
research institutions that disseminate results of research
conducted under federal grants; but we are deeply concerned
by the ambiguities in the definitions of information, government
information, dissemination, and other non-delimiting language
in the guidelines. The statutory language of Section 515 builds
upon the authorities granted OMB by the Paperwork Reduction
Act, which established the Office of Information and Regulatory
Affairs. The express purpose of the PRA was to "minimize the
paperwork burden" for individuals and non-governmental entities,
including "educational and nonprofit institutions" (44 USC
§ 3501). Any attempted federal restriction on dissemination
of academic research would be wholly unacceptable in principle
and would likely raise First Amendment considerations. It
would moreover be a monstrous burden in practice for academic
institutions. The scope and definitions of the final information
quality guidelines should explicitly identify the federal
sphere to which the guidelines apply, and should specifically
exclude academic and other non-federal institutions performing
research under federal grants.
Conclusion
In closing, we wish to underscore that federal agencies
charged with protecting human health and safety occasionally
must act on the best information available and do not have
the luxury of waiting for "perfection" before crafting administrative
guidelines. The approach taken in this notice could seriously
threaten public health and safety by tying the hands of the
agencies with an unrealistic and impossible standard of "quality."
For these and other reasons, the final guidelines should
defer as much as possible to individual federal agencies to
develop standards and definitions for dissemination of agency
information. Federal science and public health agencies should
continue to enlist the advice of the research community in
establishment of such guidelines. The AAMC agrees with the
OMB in its perception that information quality guidelines
developed by federal agencies should also be made available
for public comment under the provisions of the Administrative
Procedures Act.
In light of these concerns, we recommend the following changes
to the proposed guidelines:
1. OMB should not exceed its statutory mandate by prescribing
quality standards for scientific research information. Reference
to such information, and in particular paragraph ii.a in section
V.1.B, should be deleted.
2. Consistent with our first recommendation, references
to "reproducibility" or "transparency" of information should
be omitted as they may inappropriately apply to dissemination
of peer-reviewed scientific research. Issues of the reproducibility
of results or the soundness of underlying data need to be
addressed in the peer-reviewed scientific literature, following
publication of original findings.
3. Section V.1.B.i. should be amended: "Also, the agency
needs to identify the sources of the disseminated information
(to the extent possible, consistent with confidentiality and
other established protections)." If references to research
"data" are retained in the final guidelines, the term should
be defined consistent with definitions in OMB Circular A-110,
Section __.36 (d)(2)(i) as revised per the Shelby provision.
4. The scope of the final guidelines and the definitions
of dissemination, information, and other key terms provided
in the statute must be made more explicit and consistent with
the existing authorities of OMB. The guidelines should not
extend to academic or other non-federal institutions receiving
federal grants.
5. OMB should estimate the administrative burden and cost
impact of the proposed guidelines on federal agencies.
Finally, we urge OMB to approach Congress and request an
extension of the legislated deadline beyond September 30,
2001 in order to provide for publication of a revised set
of guidelines and a second public comment period.
For further information or questions regarding AAMC's position,
please contact Steve Heinig, Division for Biomedical and Health
Sciences Research, 202-828-0488, sheinig@aamc.org.
Sincerely,
Jordan J. Cohen, M.D.
|
