ATOMS Project Technical Report:
N=1 Assistive Technology (AT) Outcomes Summit
May 3, 2006
ATOMS Project, R2D2 Center, University of Wisconsin - Milwaukee
Table of Contents
Summit Brief Report
A small group of assistive technology (AT) and other methodology leaders in small N research (list of names at the end of report) was invited to this meeting with the following objectives:
- To articulate the state of single subject (N=1) study design AT research methodology and potential for an N =1 database.
- To promote and advance an N=1 research agenda to document outcomes of AT service delivery.
- To identify obstacles to creating N=1 data collection system for AT outcomes.
- To propose specific actions and mechanisms to advance small N research agenda for AT outcomes.
The participants in the meeting agreed that there is both a place and a need for single subject design (N=1) studies and a need to develop an N=1 national database. The major recommendation reached at the meeting is the establishment of a national N=1 study design database to assist AT practitioners, researchers and policy-makers to make effective decisions.
The basis for this recommendation includes the observations that while funding agencies have become more assertive in requesting evidence to support reimbursement decisions, few organizations, whether consumer, practitioner, researcher or policy-maker, collect assistive technology outcome (ATO) data as a routine matter and that “one of the weaknesses of the assistive technology field is the lack of controlled outcomes studies”. However understandable this may be, the most obvious consequences are that the lack of an AT outcomes evidence base means that AT practitioners are 1) more limited in their ability to make the most appropriate intervention decisions for their clients, 2) more limited in making the most appropriate AT device selection, and 3) less able to assist consumers/clients to decide which is the best of what is often a large number of options.
The lack of evidence-based AT information requires resolution. The group agreed that the examination and the potential development of a national N=1 database should be recommended. The database should be designed to serve a variety of groups interested in AT, i.e., consumers, practitioners, researchers, product developers/manufacturers, administrators, and policy-makers. A national N=1 database would provide the opportunity to “facilitate the production and both the number and quality of N=1 studies in the AT field”. This database would provide 1) quality evidence-based data for improving clinical AT decision-making 2) a quality assurance mechanism for researchers who attempt to identify which AT options are most effective under which circumstances, 3) evidence for AT policy makers as they determine how best to support the provision of AT to the many clients who depend on AT and 4) AT information for consumers. This means that the database will need to be both sufficiently sophisticated for use in research and clinical decision-making but also simple enough to be comprehensible to consumers.
Outcomes that address a variety of heterogeneous populations with specific problems and needs, a common observation in the AT field, need to be identified. The outcomes in N=1 studies therefore need to be 1) interpretable by and important to consumers, 2) relevant to practitioners and claims personnel, 3) measurable by researchers, and 4) as comprehensible and standardized as possible. The group discussed the potential role that the AT industry might play in the development of an N=1 database and they agreed that buy-in from AT industry would be important, for example, in providing feedback about specific AT devices.
The group endorsed the generation of a series of research studies to design, develop, and deploy a pilot project to aggregate state-of-the-science N=1 research studies to document AT outcomes within individual contexts. The group encouraged examination of a methodology for consumers to directly enter data about theirAT experiences and outcomes. An international standards project was recommended to begin the dialogue, research, development, and collaboration required to establish the procedures necessary to reliably document and upload AT outcomes data from N=1 studies into a web-based system. In order to accomplish this, the group recommended extending development and validation efforts for new AT outcomes instruments that will be critical for the success of any N=1 database.
Summary List of Summit Nuggets
- As single subject (N=1) experimental designs may be considered to be complementary to RCTs, an N=1 AT outcomes database is a great idea and the WWW is the mechanism to build this system.
- A weakness of the AT field is the lack of controlled outcomes studies and a lack of information that is not biased.
- There is a need for quality N=1 studies, as opposed to case description studies, as a consequence of the wide range of complex heterogeneous AT problems.
- Consensus is needed of main elements to be included in an N=1 database and reports which would add value to a data repository.
- Outcomes should reflect efficacy, resource utilization, be of value to the user, and have social acceptance.
- Outcomes must be a) important to client; b) relevant to claims people; c) standardized as possible; d) comprehensible as possible.
- The database would advance clinical data collection to be more rigorous.
- Database for either individual use or to create clinical guidelines.
- Reporting of N=1 varies widely in terms of what is typically considered essential and a database would encourage more standardized reporting.
- Importance of consumer input mechanism into database which should accommodate wide range of users and complexity: e.g., consumers→ frontline clinicians→ scientists and simple→ sophisticated methodology.
- Database should serve a variety of users including clients, clinicians, researchers, policymakers and should facilitate a number of processes including claims judication and evidence summaries.
- Key features of summaries: create rules.
- Database development is first part of a 3 level process; a) database of studies, b) synthesis, c) guidelines.
- N=1 will complement conventional RCT studies for guidelines.
- Evaluate database and pilot syntheses during development.
- Potential fields should be related to taxonomies. Consider whether each field should be complemented with free text; that the creation of database should pilot both range of input & output users; that the pilot should explore presentation (output/report) formats; that the database should come with efficient search engine.
- Need incentive structures to collect data: continuing education, certification, administration.
- Seek commitments on organizational levels as well as clinicians.
- Set-up pre-registration of N=1 trials.
- Project should use SOA data collection technology, e.g., hand-held.
- Potential role of the VA?
- Standardizing protocols will help aggregate data.
- Communication mechanism for collaboration and dissemination: N=1 section in publication with “EB ATO case of the week.”
- Potential of database to generate an electronic medical record.
Summit Participants
Brian Cobb, Ph.D. Gordon Guyatt, M.D. Mark Johnston, Ph.D. |
Ralf Schlosser, Ph.D. Mike Jones, Ph.D. Ken Ottenbacher, Ph.D., OT |
From the R2D2 Center:
| Neil Oldridge, Ph.D. Senior Scientist, College of Health Sciences Professor Emeritus, Occupational Therapy |
Kathy Rust, M.S., OT Research Coordinator, R2D2 Center |
| Roger O. Smith, Ph.D., OT Director, R2D2 Center Professor, Occupational Therapy |
Pimjai Sudsawad S.C.D. |