Isolating the Impact of Interventions (I3) Instrumentation

Overview

This webpage is a resource for individuals seeking information regarding subjective outcome measures for assistive technology (AT). The individual who uses AT rarely uses it in isolation. For example, an individual may use a piece of equipment or device on a temporary basis as they are receiving therapy to ameliorate their symptoms. Or a person may have adaptive equipment to help them dress, but prefer to have their spouse help them when the spouse is available. The R₂D₂ Center has worked to find ways to delineate and tease out the issue of concurrent interventions for a number of years. Our I3 instrumentation recognizes and quantifies this performance from the perspective of AT consumers using a subjective estimation process.

The resources discussed below originated during the ATOMS Project (Assistive Technology Outcomes Measurement System), beginning in 2001 and extend through today's research. The ATOMS Project is based at the Rehabilitation Research Design & Disability (R₂D₂) Center at the University of Wisconsin-Milwaukee and involves a consortium of assistive technology experts and stakeholders across the country. The ATOMS Project was funded by The National Institute on Disability and Rehabilitation Research (NIDRR).

Theory

That a diverse set of rehabilitation intervention approaches is considered to improve activity and participation for people with disabilities (PWD) has been documented for decades in the literature. The rehabilitation and occupational therapy fields have emphasized these factors as constructs used for determining best treatment options. In 1991 both Smith and Christiansen developed models explaining the theoretical relationships surrounding individuals with disabilities and possible intervention approaches to improve function. Christiansen recognized five major intervention categories in occupational therapy. These interventions include 1) use of occupation as a therapeutic medium, 2) education and training strategies, 3) strategies for sensory and neuromotor remediation, 4) modification of the physical environment and 5) application of technological aids and devices (Christiansen, 1991). Concurrently, Smith had identified six primary intervention approaches in the field. The six defined by Smith are as follows: 1) reduce the impairment, 2) compensate for the impairment, 3) use assistive technology device and service, 4) redesign the activity, 5) redesign the environment and 6) use personal assistance. Each model was published and acted as the basis for instruments produced after their development such as the Individual Accommodations Model from the University of Kansas. In time this work evolved into the IMPACT2 Model during the ATOMS Project work.

IMPACT2

IMPACT2 stands for the Integrated Multi-intervention Paradigm for Assessments and Applications of Concurrent Treatments (Smith, 2005). The “Intervention Approaches” portion of the model is comprised of the six aforementioned approaches available to improve functional performance. The R2D2 Center has worked to find ways to delineate and tease out the issue of concurrent interventions for a number of years. A significant confound occurs when AT outcomes are not considered within the context of concurrent interventions. Our I3 instrumentation recognizes and quantifies this performance from the perspective of AT consumers using a subjective estimation process. Individual circumstances PWD require unique decision processes. Data elicitation principles from the decision sciences were employed to meet these measurement challenges.

View more about IMPACT2

MAU

There are many measurement and research methodologies that are not typically used in AT outcomes development that have the potential to contribute to this methodology. Examples include goal-attainment scaling (GAS), dynamic norming, subjective elicitation of data, multiattribute utility techniques (MAUT or MAU) and Bayes.

Frequently, decisions regarding rehabilitation services, including AT, must be made with atypical and complex sets of variables. Individual circumstances of people who have disabilities can require unique decision processes. Diagnostic populations may be small and function can be very idiosyncratic. Thus, data collection and questioning strategies need to be flexible, customized to the individual, and created on the spot with the client. This is especially true for people who use AT. In addition to their diverse circumstances, these individuals may use more than one device for more than one task in more than one environment.

Because most decisions have multiple objectives, decisions should assess as many objectives as are deemed important in the specific circumstance being analyzed. When the methodologies of the MAU method, Canadian Occupational Performance Measure, and GAS are compared side-by-side, similar characteristics are shown that can be used to improve decision-making in practice. GAS and the COPM have substantial quantitative theoretical support. Review of MAU literature suggests that MAU, also, may offer significant relevance to AT outcomes systems. The I3 instrumentation is guided by this thinking.

View more about MAU

Development

The ATOMS Project helped to fund the development of several prototype AT outcomes instruments. These assessments include: the Relative Advantage of Assistive Technology and Services (RAATS), Relative Impact Assistive Technology Survey (RIATS), Student Performance Profile (SPP),  and the ATOMS-Division of Vocational Rehabilitation Consumer Survey (A/D-CS). A subsequent grant funded the Got-it? Project. All are person-focused subjective perspectives of outcome. A cumulative knowledge base was acquired as these developed and the nomenclature of I3 is now used to describe the methodology that evolved.

Relative Impact Assistive Technology Survey (RIATS) & The Relative Advantage of Assistive Technology and Services (RAATS)

James Lenker, at the time a philosophy doctoral student in the Department of Industrial Engineering from the University at Buffalo, State University in New York worked with the ATOMS team to develop the first instrument we designed to identify the amount of contribution an AT device or service has on an outcome when obtained using an ecological approach outside of a laboratory controlled environment. The Relative Advantage of Assistive Technology and Services (RAATS) was designed to address the outcomes of any AT devices an individual may be using on a daily basis utilizing both subjective and objective data. The RAATS allowed the user to rate the overall impact of six interventions on a 7-point scale. The RAATS consists of eight open-ended questions, which were designed to be answered through an interview process. The eight questions were directly related to specific elements of rehabilitation including; assistance from others, changes in how you do tasks, attitudes and expectations of others, the physical environment, your physical and mental abilities, social services and public assistance, assistive technology, and AT training services. For example, how has assistance from others (family, friends, co-workers, aides) impacted your everyday performance related to achieving your work and school goals? These categories covered a variety of topics ranging from the individual's perspective of his or her health to the impact that certain training services have had on daily performance.

Furthering the development of this method of assessment, Lenker addressed the limitations of the RAATS and developed a second tool, The RIATS. It focused on vocational rehabilitation (VR) clients from the state of New York for addressing the needs of this population (Lenker, Scherer, Fuhrer, Jutai, & DeRuyter, 2005). The RIATTS involved a semi-structured interview regarding the individuals experience with computer-based AT devices.

Student Performance Profile (SPP)

The AT Infusion Project (ATIP) of the Ohio Department of Education collaborated with the ATOMS Project in the development of the Student Performance Profile (SPP), the next instrument to advance the I3 methodology. ATIP was funded from the United States Department of Education School Renovation, the Individuals with Disabilities Education Act (IDEA), and Technology Grants and as part of the work allocated $9.2 million dollars of support to purchase AT and measure the outcomes of students in terms of access and participation in the general curriculum.

Outcome measures were specially designed web-based instruments that assessed programs in general curriculum and Individual Education Plan (IEP) goals. The SPP was developed to meet the challenge of measuring AT outcomes when students received concurrent interventions specific to classrooms. The team revised and updated questions based on the original RAATS work as a portion of their instrument.

School teams completed the SPP prior to using devices and again as a follow-up measure eight months to one year later. The SPP provided a way to begin analyzing student outcomes resulting from the provision of AT in the schools. The profile was designed to examine important factors that impact the educational progress with AT. Since all funded technology must directly support the student's IEP goals, the rate of progress on the goals was documented on the SPP. The impact of AT on access and progress in general education was also assessed through the SPP.

During an early analysis, a report on 1,760 students found a significant difference between the amount that AT devices contributed to progress on the goals prior to the AT intervention and after using the technology for 8 months. The authors concluded that the results of this study provide confirmation that the method used to determine the relative contribution of a given intervention has potential, and should be researched more extensively (Fennema-Jansen et al., 2007).

View revised SPP (.doc)

View Technical Report - The Assistive Technology Infusion Project (ATIP) Database

View Technical Report - Design of the Data Collection System for the Ohio Assistive Technology Outcomes (OhioATO) Project

Subjective estimation, SPP

Intrinsic to all of the instruments that evolved from I3 methodology is the subjective report of the consumer. In all applications the individual identifies a goal and then rates their progress towards the goal for any of the concurrent interventions that they are participating in at the time.  Below is a sample screenshot from the Ohio SPP that demonstrates some of these questions.

An additional screenshot demonstrating subjective estimation questions is provided in the discussion of the Got-It Project (below)

SPP study #2

Ann Watson, a Doctoral Student in the Occupational Therapy Department at Nova Southeastern University, investigated the effectiveness of AT in the public school setting with students age 3-21 (Watson, 2007). Watson focused on a multidisciplinary approach in assigning AT to enhance a student's school performance. Watson examined the contribution of AT in supporting the student's IEP goals in comparison to other interventions the students received. Watson used the SPP as a study measure after modifying the instrument with the original developers. Additionally, Watson used a second instrument, the School Functional Assessment-Assistive Technology (SFA-AT). [This instrument was modified from the original SFA by the R2D2 Center, 2002.]

Results indicated that relative to other interventions, AT provided by a multidisciplinary team may have a significant effect on IEP goal improvement (t[12] = 5.54, p =.00). She found that the SPP required less time to administer, score, and analyze than the SFA-AT. A strength identified with the SPP was its ability to bypass irrelevant questions (Watson, Smith, & Andersen 2010; Watson & Smith, 2012).

SPP study #3

In 2006, Carol Hankins Olsen, a Doctoral Student in the Philosophy Department at the University of North Dakota, investigated the impact of AT devices and services for students with learning disabilities (LD) who have an academic need in writing. Olsen's study analyzed data collected as part of the 9.4 million dollar grantß for assistive technology to the state of Ohio (above). The factors explored included: the demographics of students in the study, the numbers of students with LD in need and receiving writing AT, the rate of student progress, the number of services received, and the rate of change in the entire process.

Olsen's final inquiry explored the changes in contributions of various external interventions before and after implementation of AT devices. The interventions explored included: student strategies of natural development and compensation for impairment; teacher strategies of adaptations of specific curricular tasks, redesign of instructional environment, performance expectation changed, and participation in general education instruction; special services interventions of related and support services and personal assistance; and AT interventions of AT devices used and AT services provided.

Olsen reached six general conclusions related to the impact of AT for students with LD and a need in a writing. 1) AT is an efficacious intervention to be used by educators and occupational therapists in the school setting to improve abilities and rate of progress of students with LD and a need in the academic area of writing. 2) Word processors and word processors in combination with writing software are particularly helpful in improving abilities of students with LD and a need in the academic area of writing. 3) Word prediction software is a common AT device request for students with LD and a need in the academic area of writing, however additional research demonstrating the efficacy of this software is needed. 4) AT services should be used in combination with AT devices to most significantly impact progress of students. 5) AT device implementation increased the contribution of 9 out of 10 parallel interventions, with AT devices and services the intervention noting the most significant improvement in contribution. 6) The Student Performance Profile Pre (SPP-Pre) and the Student Performance Profile Post (SPP-Post) are valid tools to measure student progress related to AT (Olsen, 2006).

ATOMS-Division of Vocational Rehabilitation Consumer Survey (A/D-CS)

Another iteration of the I3 instrumentation was created and studied by Johnson, a graduate student from the University of Wisconsin-Milwaukee. She developed the Impact of Concurrent Interventions (ICI) as part of her thesis (Johnson, 2006). Similar to the developers of the RAATS and RIATS, Johnson once again focused on DVR consumers. Unlike pervious outcomes measures, Johnson eliminated many of their limitations and reached consumers directly by developing the outcome tool into a phone interview. The instrument queried DVR consumers about the overall impact of the six concurrent intervention approaches from the IMPACT2 model on their employment goals. The instrument was created to isolate and assess a specific AT device and the impact on employment and ability to function in the work environment thus producing outcomes data. Among the findings, Johnson reported that A/D-CS was an effective instrument in isolating the contribution of concurrent interventions and to measure the impact of AT on employment goals.

[Note: The origin of I3 terminology evolved during the write up of Johnston's work. In earlier documents you may see that The Isolating the Impact of Interventions Instrument, abbreviated as I3, acted as the new title of Johnson's instrument. Subsequently the team identified that the term I3 is more properly the instrumentation methodology supporting multiple studies and now refer to Johnson’s work by its original title.]

Got-it?

A recent iteration of this methodology was developed to prototype a web-based version of these questions. The R2D2 Center developed a prototype website, the Got-it? Project to gather and provide consumer feedback on different AT products. As a pilot project the scope was limited to bathroom safety equipment. Primarily, testers rated products and then were invited to pilot the I3 application in this context. They were asked to detail their goals for bathroom safety and answer a series of questions asking how AT, as well as other interventions, affected the attainment of their goals. The project final report delineates recommendations for future work provided by participants, and input from the project team.

GOT-it? Final Report

Conclusion

The ATOMS Project worked to clarify the conceptual underpinnings of AT outcomes variables for developing measurement systems and delineate variables to increase an understanding of AT interventions as they are practiced in the natural environment. The R2D2 Center continues to advance these efforts. The I3 instrumentation is a demonstrated, effective approach to control for the confound of concurrent interventions when measuring AT outcomes.

---

 

Copyright © 2001-2015 ATOMS Project at the University of Wisconsin-Milwaukee