AJET 23(2) Freeman et al (2007) - educational innovation with in class electronic response systems

Australasian Journal of Educational Technology
2007, 23(2), 149-170.

Factors affecting educational innovation with in class electronic response systems

Mark Freeman, Amani Bell, Carole Comerton-Forde, Joanne Pickering and Paul Blayney
The University of Sydney

This paper reports the use of Rogers' diffusion of innovation perspective to understand the factors affecting educational innovation decisions, specifically in regard to in class electronic response systems. Despite decreasing costs and four decades of research showing strong student support, academic adoption is limited. Using data collected from academic users, non-adopters and other stakeholders reflecting on factors known to affect innovation diffusion, we find issues of cultural compatibility, complexity and relative advantage to be the most critical aspects affecting adoption decisions. These issues partially negate the benefits of increased in class interaction and student engagement. Suggestions for overcoming these issues are discussed.

Introduction

Although a number of seemingly worthy educational technologies have been adopted to facilitate learning in classroom contexts, audience electronic response systems have not been universally adopted. An electronic response system (ERS) can be introduced to promote learning because it allows students to answer in class questions quickly and anonymously, in small and large classes, and to gain feedback immediately on the right answer and how other students in the class answered. Academics can also use the immediate feedback to inform teaching decisions. In the last four decades electronic response systems have become cheaper, more widely available, more sophisticated, better integrated into existing hardware and software systems (including presentation programs), more aggressively marketed by textbook publishers, and more widely researched. Despite these improvements and the apparent student benefits (Cue, 1998; Judson & Sawada, 2002; Draper & Brown, 2004; Fies & Marshall, 2006), particularly when appropriately integrated, academics have not widely adopted this educational technology into their teaching repertoire. One possible exception is physics higher education (Crouch & Mazur, 2001).

Although some studies briefly allude to the impact of electronic response systems on academics (eg, Sharma, Khachan, Chan & O'Byrne 2005; Draper & Brown, 2004; Jones, Connolly, Gear & Read, 2001, Beatty, 2004; Burton, 2004), there is limited systematic research of academics' experiences and adoption decisions reported in mainstream learning technologies research or disciplinary based journals. Hence this paper focuses on the academic decision to adopt ERS.

Electronic response systems

Fondly referred to as clickers, electronic response systems (ERS) are also known as personal response systems, audience response systems, electronic voting systems, classroom communication systems, classroom response systems and classroom performance systems. ERS have been used in higher education for four decades. Academics might introduce ERS to promote learning since they allow improvements to the formative aspects of learning. Almost irrespective of class size, students can answer in class questions quickly using a handheld keypad to transmit their response. A receiver feeds the responses to computer software which interprets, records and aggregates such responses. Typically responses are displayed to a class so students may gain immediate feedback on the aggregate distribution of responses. If questions have a correct response, students can gauge if their thinking and answer is correct as well as gauge how other students in the class answered. In class discussion typically follows, allowing students to construct and clarify improved meaning and understanding which is important for those who were incorrect, and particularly important if some choices related to threshold learning concepts. ERS can be used also to gauge student opinions to spark discussion, simulate theories and make decisions to promote learning and the overall student learning experience (Draper & Brown, 2004). As well as using them in a similar fashion for summative purposes, academics can also use ERS data to gather feedback to inform teaching decisions (e.g. concepts poorly understood; areas for further revision; lecturing speed too fast/slow). Depending on the purchasing model, the keypads are brought to each class either by the academic or each student.

Judson and Sawada (2002) review four decades of research showing that students appreciate classroom use of ERS well beyond any novelty value, but that impact on actual student achievement is mixed. Consistent with the 'no significant difference' phenomenon (Russell, 2001), Draper and Brown (2004) demonstrate that student success arises when pedagogy, not ERS technology, is the focus. Judson and Sawada (2002) and Crouch and Mazur (2001) note the real gains to learning occur when ERS are used in tandem with knowledge construction activities and, in particular, interaction provided by reciprocal peer learning. However, this is not to say that there ERS does not afford opportunities that can be usefully incorporated to aid the student learning experience. Freeman, Blayney and Ginns (2006) show how anonymity afforded by ERS can improve student perceptions of the classroom experience. Ongoing recent research (e.g. Banks, 2006; Sharma, Khachan, Chan & O'Byrne 2005; Freeman & Blayney, 2005) continue to highlight the student benefits.

In contrast, the academic teaching or innovation experience with ERS has not been systematically investigated. This is despite the obvious benefits noted by Sharma, Khachan, Chan and O'Byrne (2005), Fies and Marshall (2006), Shapiro (1997), Mitchell (2001), Judson and Sawada (2002), Draper and Brown (2004), Jones, Connolly, Gear and Read (2001), Beatty (2004), Burton (2004) and Elliott (2003), which include improved classroom dynamics (eg, lecturing style, level of student preparation or fatigue), motivated students, enhanced information for teaching decisions (called contingent teaching by Draper and Brown, 2004) and efficiencies in teaching administration (e.g. attendance records, student evaluation of teaching). The main potential costs of ERS noted have been set up time and difficulties in learning how to use the technology (Cutts, 2006; Burton, 2006). This study aims to explore this gap by examining factors influencing the decision to adopt an ERS.

Literature review

A range of approaches can be used to understand academic decisions to adopt ERS. Lueddeke (2003) examines the academic's conception of teaching and found academics with a conceptual change/student focus teaching orientation were more interested in adopting interactive approaches in face to face contexts than academics without this conception. Similarly Bain, McNaught, Mills and Lueckenhausen (1998) investigated the impact of teachers' beliefs on the design of e-learning and proposed a more complex categorisation than the student centred versus teacher driven framework used by Lueddeke. Elton (2003) recommends taking a more holistic approach in examining decisions. Bennett (2001) surveys academics' attitudes to new teaching methods, including those involving educational technology. He notes larger classes, university pressure and student expectations as the major incentives triggering adoption, but inadequate university facilities, insufficient time and inadequate knowledge and training as the major barriers to adoption. Eley and Eley (1995) found general patterns of academics teaching themselves requisite IT related skills. Bennet (2001) finds that colleagues are the major influence in academic development and e-learning adoption decisions. While the advice and experiences of respected colleagues might provide useful anecdotal advice, he cautions that some are potentially relatively uninformed about the wider evidence for good practice innovation.

Rogers' 1962 classic theory updated most recently in 2003 is an alternative holistic, grounded and well-established model for studying diffusion of innovation. Rogers (2003) characterises adopters as innovators (2.5% of the population), early adopters (13.5% of the population), early majority (34% of the population), late majority (34% of the population) and laggards (16% of the population). Cumulative adoption over time would follow an S curve pattern, with small numbers of innovators and early adopters using the technology first, followed by the majority and lastly the small number of laggards taking the longest time to adopt. In this theory, it is not an expert's objective evaluation of the innovation but rather the potential adopters' perceptions of five factors that drive adoption decisions. These five factors are relative advantage, compatibility, complexity, trialability, and visibility. Relative advantage is described as the degree to which the innovation is perceived to be superior to the current approach. Compatibility is the degree to which the innovation is perceived to be compatible with existing values, beliefs, experiences and needs. Complexity is the degree to which an innovation is perceived to be relatively difficult to use or understand. Trialability is the degree to which the innovation is perceived to be able to be used on a trial basis before confirmation and adoption must occur. Finally, visibility is the degree to which the innovation is perceived to have results which are visible or observable to others.

The model has been applied, extended and critiqued. Geoghegan (1995) notes a deep and dividing chasm between the first two adopter types and the following three, arguing that the latter are more networked and influenced by disciplinary colleagues than the former. Anderson, Varnhagen and Campbell (1998) confirm the chasm in their study of a large, research intensive Canadian university. McLean (2005), Wilson and Stacey (2004) and McLoughlin (2000) use it to inform the practice of academic developers working with educational technologies. Van Slyke, Lou and Day (2002) apply it to understand groupware adoption decisions. Frambach and Schillewaert (2002) argue that the effect of external variables (e.g. supplier marketing efforts) and the characteristics of the organisation (e.g. size and organisational innovativeness) are additional factors to consider. The model is not without its critics, such as McMaster and Wastell (2005) who propose social constructivist models as more useful. Slappendel (1996; p. 109) further notes that innovation is an interactive and complex process achieved by the "interaction of structural influences and the actions of individuals." She notes that questions about process are best addressed by case studies. Bell and Bell (2005) use Rogers framework as one of three in a case study analysing the introduction of a virtual learning environment in several UK institutions.

This study uses a case study approach and a diffusion of innovation Rogers' perspective to explore the research gap, specifically relating to decisions about in class electronic response systems introduced to support student learning. It focuses on experiences of various types of staff who considered using or were involved in using or supporting ERS. As such, this paper is aimed at increasing our understanding of the diffusion and adoption of ERS and, therefore, represents an important addition to educational technology and specifically ERS research.

Research context

Although electronic response systems were first used in higher education in the mid-1960s (Judson & Sawada, 2002), ERS were new in the educational context of the Faculty of Economics and Business at the University of Sydney, Australia's oldest research intensive university. Fifteen academics attended a demonstration in November 2003 following a faculty wide invitation. About half of the attendees expressed an interest in using ERS in their classes. However, the first pilot involved a single academic and equipment borrowed from the vendor. It was adopted into the classroom teaching and learning experience in January 2004 during an intensive summer school course in undergraduate management accounting. It involved 30 handsets shared between two or three students to encourage peer interaction and learning. The academic was mentored through the innovation process with support from a faculty based academic developer and the equipment vendor.

The second pilot was undertaken during the two regular academic semesters in 2004 and first semester 2005. University teaching grant funding was used to purchase mobile ERS equipment such that up to 150 handsets could be used in either one or two classrooms at any time. Further funding enabled permanent installation in two lecture halls in 2005. In the final stage of the pilot when further equipment was installed in two lecture halls, technological and logistical assistance was provided to assist the academics to overcome difficulties encountered with the equipment and software.

A project team was established in the second pilot to overcome the limitations identified with higher education innovations, such as bottom-up innovation being pursued individually (Luddeke, 1999) and the passive dissemination of top-down innovation (Elton, 2003). The project team comprised four academics from various disciplines, an academic developer and a project manager who did the bulk of the liaison with technical staff and other support staff as well the vendor. Three of the four academics became users and discussed their experiences with other colleagues. Consistent with Kirkpatrick (2001), the project team structure supported the innovation by providing mentoring in pedagogical and technical issues as well as administrative support. Regular meetings were held for team members to reflect on their experiences. Meetings were also used to discuss the more general educational technology research that foregrounds the role of learning design over technology (Goodyear, 2005; Russell, 2001) as well as the specific ERS and peer learning research.

Research method

Fourteen people involved in the second ERS pilot were asked to reflect on the process of adopting an ERS using the five factors outlined by Rogers (2003). Five of the six project team members wrote their own reflections, while an additional nine reflections were collected through structured interviews with other people involved in the project. Details of the questions used to guide these reflections and which were paraphrased into the interviewers' own words are provided in the Appendix. Three of the additional nine were academics; one who used the technology in one class with the support of one of the project team members, and two who chose not to adopt the technology. The remaining six participants consisted of the equipment vendor, the University audiovisual manager who subsequently installed the equipment in the lecture halls, three Faculty IT staff who installed and supported the software on staff computers, and a casual assistant who provided logistical and basic software support in the latter stage of the pilot. In total, 6 academics provided data: 4 users and 2 non-users. Non-academics were included as we thought it useful to examine whether they had similar or different views of the ERS, and because it reflects the nature of our holistic approach. Bell and Bell (2005; p. 642) recommend the inclusion of administrative and other support staff since project success depends critically on their involvement and perceptions in that it "enables engagement of the whole programme team, highlights problems that may then be addressed in an institution-wide response, and aids the identification and dissemination of good practice".

Results

The following results are organised around Rogers' (2003) five factors, namely relative advantage, cultural compatibility, complexity, trialability and visibility (see Appendix). Where they differ to academic users and non-users, the views of other stakeholders are included.

Relative advantage

While several positive and negative aspects are detailed below, overall the academic users reflected that ERS provided the potential for significant advantages over their current approaches to teaching. However, they reflected that such opportunities were more likely to be experienced by those who already designed their face to face classes to be interactive. The relative advantage was less pronounced for those who needed to reorient to a more interactive teaching approach so that they could use the ERS.

First, academics reported that ERS provided improved opportunities for student engagement and to exploit in class teaching and learning efficiencies in a number of ways:

Ability to ascertain student demographics, and identify the level of student preparation and understanding of topic material meant academics could adapt and construct in class time more effectively.
ERS greatly assisted the setting of expectations in the first few classes because students were shown the aggregated responses and could therefore see that they were not alone in their prior or current experiences: "Used at the start of semester, the ERS can help set students' expectations. It can also help students feel more at ease, because they realise that they are similar to many of the other students in the class (e.g. typically students think they are the only student who has never studied business before, but usually they are in the majority)."
Having an accurate understanding of student preparation and knowledge in both early and later classes meant warnings of the impact of continued behaviour could be applied. It appeared therefore to encourage future preparation.
In contrast to other means by which students can reveal their responses and opinions (e.g. raising hands), academic staff thought that ERS provided students with anonymity when responding to in class questions, so students might have been less worried about class peer pressure and instructor pressure. As a result, more students responded to questions and appeared to be engaged in learning.
Initially the ERS has novelty value and is effective for getting the students interested, e.g. one academic "saw the use of ERS as an immediate way to gain engagement, involvement and interest from students in large classes..."
Greater opportunities for reciprocal peer learning appear possible with ERS. Instructors noted that students appeared more likely to engage and discuss their initial thoughts and answers with a neighbouring student when they know they can retain their anonymity to the instructor and the rest of the class. However, given prior experiences of passivity in lectures, there was some resistance initially to participate and interact. One academic found that: "There was strong resistance from students to interacting with one another. It took several weeks before they did this without repeatedly being reminded of it."

The ERS also offered academics opportunities for increased job satisfaction:

Without the ERS, academics felt it would be almost impossible to accurately ascertain students' understanding of class material and adjust their teaching appropriately.
Greater sense of student engagement, preparation and interest in class helped academics experience an increased sense of a job well done.
Academics found the ability to gauge the level of student understanding as a result of the ERS, an efficient tool to help them to be aware of and guide students who were at risk of failing.
Academics used the saved data to analyse student results and design future classes to address any problems identified.

The perceptions of relative advantage of the use of the ERS were not all positive. The main disadvantages expressed by academics related to time and risk. In regard to the former, there were several time consuming aspects:

Staff had to spend time learning how to use new hardware and software. "Class time was required for use of the ERS and my time for learning. It wasn't until the 4th class that I was able to set up the equipment without needing technical assistance. ... To run the pre-test took up most of my time for a week."
Time had to be spent on each set of new class questions and answers, even if classes were already designed for interactivity. Even when question templates were developed to assist the reuse of question types, it still took time to set particular parameters for each new question.
Although limited technical and administrative support was available, academic users found that the time to set up the mobile equipment for use and subsequent collection and return to base, was excessive.
Even when fixed receivers and software were installed in two lecture halls, staff continued to experience an excessive amount of time being used in collecting and distributing the handsets.
Last but not least, there is also a major cost to making a class interactive if it is not already. Using ERS forced the academics to think about their teaching style and apart from those whose teaching style included a large amount of student interaction, with well developed in class questions, significant time was required to adjust their teaching style and develop new material. With time pressures in the research intensive university context, this was perceived as a major barrier.

The second negative aspect that academics reported was the amount of risk involved in using ERS, especially due to technical problems (detailed later). Furthermore, several academic users who asked students questions about their study habits and prerequisite learning reported a sense of a lack of control as they became aware for the first time about such issues.

The academic participants who had chosen not to adopt ERS reported that while it appeared to add benefits, especially for engaging students, they considered it would take too much time to implement, particularly in regard to the development of questions for a more interactive style of teaching:

Mixed reaction. Better - quick, entertaining, something different, novel, catches the eye. Functionally it's quick and captures the information so I don't need to remember things. But not better. Difficult to start using it. I was trained in its use but then got very busy so it became a choice - I could get away without using it and there was no incentive to use it.

Interestingly, support staff were more positive than academics about the time efficiencies of this approach to interaction:

The keypads were certainly advantageous in terms of getting anonymous responses to questions and displaying the results... I would rate quality and effectiveness as high. It is more time consuming in one sense than other methods eg show of hands - but in terms of collating and displaying the results, there were vast time savings associated with the keypad system.

These findings are consistent with Bennett's (2001) UK academic survey where inadequate university facilities, insufficient time, and inadequate skills were the major barriers to adoption of new teaching technologies and methods.

Cultural compatibility

Academic users and non-users felt that the cultural compatibility of ERS was dependent on the individual's existing approach to teaching. It was easier to adopt ERS if the academic already had adopted an interactive approach to teaching, such as the use of in class multiple choice questions. A decision to adopt the technology requires academics to either have a fairly interactive approach to teaching, or a willingness to change their approach to meet this objective. One academic felt that "Lecturers that adopt a 'chalk and talk' approach would probably be reluctant to adopt ERS due to the set-up costs."

Another academic reflected that although ERS are "very compatible [for students] in today's technology driven world... [the technology] is less compatible to digital immigrant staff. The current staff and student culture is that lectures are passive, so redesigning [lectures] for interaction is the hardest step." While the academic developer agreed, noting that a switch from focussing on content coverage to interaction and engagement was a major paradigm shift for both staff and students, he saw the faculty culture becoming more favourable to innovation. As classes got larger and student cohorts more diverse, and as technology became more ubiquitous he opined, conscientious teachers who experienced less success with tried and true approaches would be more likely to try new approaches, providing potential relative value could be demonstrated. Related to this desire to respond to student preferences, he also noted the increasing emphasis upon quality assurance and accountabilities, not just arising from government funding but also due to higher student expectations and demands for better learning experiences as student fees increased. This is consistent with Bennett's (2001) survey of UK academics where larger classes, university management expectations and student expectations were noted as the three most significant drivers.

Academic users and non-users agreed that ERS are also easier to adopt if academics are already innovative and comfortable with taking risks with their teaching. However, academic culture is perceived as discouraging risk taking for four reasons: (1) other necessary classroom computer technology is not universally available nor of reliable quality; (2) the potential for embarrassment if the technology or teaching method does not work (content experts can have a very high expert image and status to protect); (3) the potential impact on student evaluations of teaching if failure is experienced; and, (4) rewards and recognition for teaching innovation are perceived to be valued less than research achievements in most departments and the university as a whole. One academic commented that "Novelty [is] quite acceptable to students but less acceptable to staff. Any innovation is viewed by staff as more work, so why do it when research beckons?" Only one academic was less concerned about failure with a teaching innovation and reflected that he sees risk taking more positively and sees himself as an innovator. One non-user thought appropriate rewards might overcome these disincentives.

While technical and administrative support staff comments largely mirrored those of academics, there was less awareness of the cultural gap. For example, one person noted a perception that most academics and students are familiar with presentation software, thus implying it was a relatively small step for academics to include ERS and that the attraction of facilitating interaction was obvious.

Complexity

Despite the apparent simplicity when initially observed, partly because it relied on popular standard presentation software (i.e. Microsoft PowerPoint), academic users reported a moderate to high level of complexity in learning and using the ERS software and hardware. Not only was it cumbersome to carry the equipment to class but there was considerable time and stress involved in setting up the portable hardware:

The equipment set-up appeared to be straightforward, but connections have to be done in very specific order or it doesn't work. I spent time each week practising the set up in my office prior to attempting it in class. Previous technology failures in class (computers, videos, projectors) have caused me to feel stress whenever I'm dealing with new technology... basically I'm just waiting for it to crash. So the manual set up of equipment was an added stress.

The AV manager agreed that "it's beyond reasonable expectation for academic staff to have to do the system set up." Even when question templates were provided and later when an administrative assistant was made available to assist the hardware set up in the rooms without fixed equipment, academics reported the process was still too complex.

The software was even more complex for those who were not familiar with PowerPoint. One academic first needed to learn how to use a laptop and then learn the presentation software before embarking on learning how to use the ERS software. An academic who was familiar with presentation software felt that "low level usage is moderately easy to learn [but] some aspects of even simple things could be seen as challenging. High level uses like demographics and decision making questions [are] somewhat more difficult to learn and use."

One of the academic non-users agreed that ERS seemed difficult to start using, as it is unfamiliar technology and requires time to learn: "If you're not an IT person, it's difficult to start up. Just dealing with computers can be difficult for the average person. ERS is something different from normal and requires time learning and practicing before you go to lectures with it."

Academic users also faced difficulties due to problematic software and one experienced severe software failure and data loss of student responses that would have been catastrophic if it had been a summative assessment. The majority of problems that academics reported to the IT support staff were related to software rather than hardware.

When software problems were experienced, the "Help" function in the software rarely was sufficiently helpful and there was no perceived instant support available. Not only were response times from the company slow, but the answers were sometimes inadequate. Some of the problems encountered, such as network usage rather than desktop usage, seemed too difficult to resolve immediately, with the only option being to wait for the next software version. The distributor's low understanding of the system's use in a higher education context meant that there were considerable hurdles to be overcome. Possible or actual equipment failure meant that there was a heightened risk of embarrassment in front of students or colleagues.

Interestingly the IT support staff not only held views consistent with academic users that it was less intuitive and thus more complex than initial impressions, but were even more outspoken. For example the technical support person noted that academics called him mid-lecture about the software, and he commented that "the software was a bit jittery and seemed to freak out over very small things". The IT manager noted that local support for overseas products was often a fundamental support issue in the Australian context.

Trialability

Academic users and non-users agreed that the ERS were easy to observe by watching a colleague in action. One academic non-user pointed out that "You can try it out with friends or colleagues but that's not the same as in lectures. Trialling is very different from actual performance." The ERS was time consuming to trial even for a single class, due to the need to learn how to use the hardware and software, let alone the time that needed to be invested to develop suitable interactive questions. One non-user commented that "If the way you use it is wrong then students don't react positively and it's less trialable. But this is the same with any innovation."

A further difficulty in piloting the systems was the need to carry the mobile equipment to class. Academics reflected that it would be far more attractive for a potential user to trial the system if they were given support in preparing the presentation slides and used the ERS in rooms with permanent installation. Both these suggestions were carried out in the latter stages of the pilot. Despite permanent installation in two lecture halls, timetable allocations to these specifically equipped rooms were not without problems - not a surprising result given the competition for rooms due to a large and expanding student population.

The academic developer commented that having the initial option of usage on loan from the distributor made it relatively easy to trial ERS before buying.

Visibility

The visibility of the results of the pilot usage of ERS was low, due to the private nature of teaching, and also low dissemination by project team members, due to the problems encountered. Few people outside the project team were aware of the existence of it, despite several faculty broadcasts and showcases, active marketing by book publishers (offering it as a free optional extra to staff if their book was recommended as the course textbook and students paid extra for a keypad included with the textbook) and a published research paper (Freeman & Blayney, 2005). Indeed, the AV manager said that "We had to deal with a member of your faculty who didn't know about the system but who wanted the same end result."

The academic users were reluctant to encourage other users to adopt it due to the difficulties they encountered. One academic said that she was "reluctant to promote the system and encourage other users to adopt it, due to the difficulties I have encountered with it. I am unwilling to widely promote its use until the issues have been resolved and until appropriate support is available."

While the equipment vendor was disappointed that no further sales had been made from the extra efforts to make the results visible, another non-academic user (i.e. an IT support person) commented that: "From the academics I had a bit of... pessimism from some of them. There seemed to be a perception that it was going to break down before they'd even tried it - phrases like 'the usual technology problems'.... My understanding was that it was word of mouth via the academics who were involved in the trial... [who] at various times threw up their hands."

Discussion

Several themes emerge from these results. The issues of cultural compatibility, complexity and relative advantage appeared the most critical aspects affecting adoption decisions. These issues negate the benefits of increased in class interaction and student engagement to such an extent that only one of the academic users continued usage (albeit very limited) following the 18 month trial. This is consistent with Bell and Bell's (2005) report that 70% of educational innovations fail.

Recognising that innovation is an interactive and complex organisational process (Slappendel, 1996), several actions can be taken to assist staff to adopt ERS and other innovations that enhance the student experience. First, the organisation can remove administrative and technical barriers such as the burden upon academics of carrying and distributing ERS equipment. At one level this might mean providing personal support, such as an administrative assistant, to undertake this support role. On another level and at relatively little cost it could mean permanently installing receivers in each room and providing all students with a keypad on admission to the university. Based on our experience however, fixed equipment is unlikely to be sufficient on its own to attract academic adopters. Yet another institutional response would be to integrate educational technology with student lifestyle decisions. One example of this would be to install Bluetooth receivers and use software that students can download onto their phones and therefore allow both multiple choice and qualitative responses to be communicated by mobile phone (cell phone) or SMS messaging. Prensky (2005) reports a trial using mobile phone messaging that takes advantage of the penetration of mobile phones amongst students and the availability of inexpensive question development kits.

Assuming universities can resolve the administrative and technical issues discussed above, a far more difficult issue to change arises, namely that of cultural compatibility. The introduction of ERS challenges some academics' conceptions of teaching. Those for whom teaching is more about transmitting important disciplinary information or content view the inclusion of interaction activities, which are fundamental to ERS, as tantamount to wasting time and getting in the way of covering all the content. In contrast, those who view teaching as helping students change their conceptions are more likely to view interaction activities as part of the important learning process that encourages a deep approach to learning (Prosser & Trigwell, 1999). In addition, there is a cultural perception, although counter to systemic measures to the contrary, that there is a lack of reward for teaching innovation in this research intensive environment. This is consistent with Frambach and Schillewaert's (2002) conceptual framework in which the organisation's characteristics - here a strategic focus on research - influence adoption decisions. Any innovative teaching practice that is culturally incompatible is likely to have a flow on effect to academics' evaluation of the relative advantage. This is consistent with Bennett's (2001) survey results where keeping abreast of latest teaching methods and technologies for future career advancement was rated second lowest amongst innovation incentives by UK academics.

But does this mean that ERS are more culturally compatible to physics academics since ERS are used more widely in physics than other disciplines and, if so, what can we learn from that? Individual change is linked to disciplinary focus and departmental practice. Academics hold a strong allegiance to their discipline and their cultural teaching practices (Becher, 1994; Hicks, 1999) where a community of practice is already established (Hutchings, 2000) and a favoured epistemology held (Healey, 2000). Lueddeke (2003) reports that discipline and conception of teaching most strongly impact scholarship of teaching or pedagogical research (over qualification, gender, years of teaching). Teachers who are student focused and more open minded to change and innovation to teaching are more likely to engage in designing learner centred activities. Lueddeke (2003) finds a significant difference between underlying discipline and conception of teaching. A significantly greater proportion of academics, for example in business at his institution, held the student centred conception than academics from the technology disciplines. It is therefore a surprise that physics is one discipline where ERS appear to have taken greater root in the US and to a lesser extent also in Australia. Compared to some other disciplines, innovative teaching practices that promote deep approaches to learning (such as in class interaction with or without educational technologies like ERS) have been widely disseminated within the physics discipline (Hake, 1998: Crouch and Mazur, 2001; Sharma et al, 2005; Lorenzo, Crouch & Mazur, 2006). Therefore universities that promote departmental leaders who are also student centred might find this an effective professional development and change strategy that improves the quality of learning, irrespective of whether it involved educational technologies.

The final theme that emerges relates to the role of project teams in educational innovation. Organisations can provide competitive funding for educational innovation where collaborative project teams are proposed since Bates (2000), Lueddeke (1999) and McKenzie et al (2005) note that innovation by lone rangers often fails to produce an end product or succeed in achieving embedded change beyond their own course and tenure. Given that teaching is still largely a private and at times secretive behaviour (Elton, 2003), it is not surprising that innovation and change are often attempted by individuals.

Project team membership is also important. Given Geoghegan's (1995) observation of a chasm between Rogers' categories of innovators and early adopters across to the majority, teams should contain several members, particularly from the early majority, since they are the opinion leaders and level headed individuals who require hard evidence, two critical ingredients for sustainable innovation. Members with different skill levels are important also for academic development. Boud (1999) argues strongly for strategies that support reciprocal peer learning at the coal face, as this is the most obvious place for localised, disciplinary specific academic development. Over the duration of the project, trusting relationships build and comparative expertise is shared, offering opportunities for professional development for all team members. This is useful since Martin and Ramsden (1994) found that good outcomes could be subsequently contradicted when staff returned to their local context following participation at centrally delivered academic development courses.

In two national studies, Alexander and McKenzie (1998) and McKenzie et al (2005) note successful innovation is more likely to occur where project teams: (1) are supported with a skilled project manager, (2) have adequate time to carry out their project responsibilities, which means buying out or using others to fulfil some of their other obligations, (3) were able to resolve conflict and (4) had support of department or faculty leaders. McKenzie et al (2005; p. 121) note the importance of personal contact and that adopters engaged with projects "because they perceived them to have aspects of relevance to their personal teaching and learning interests, their students and/or their teaching situations".

Our experience confirms these findings of localised project teams and the inclusion of those with student centred conceptions of teaching. Our experience also confirms that irrespective of whether the educational innovation is subsequently embedded or not, the teams that take such an evidence based approach to innovation appear to be more effective in changing conceptions of teaching than academic development seminars or forums. Even though, in this case, the project team approach did not ensure the ongoing adoption of the technology, it did provide an environment in which we could identify the barriers to adoption and in which we were willing to research and report on failure (Latchem 2005), as well as discuss non-technological influences on the student learning experience.

Conclusion

The objective of this paper was to apply Rogers' diffusion of innovation framework to understand factors influencing academics' decisions to adopt electronic response systems. Despite nearly four decades of positive student support for its usage, and recent reductions in cost combined with technology advances, there has been minimal penetration in higher education other than physics. Fourteen staff (6 academic users and non-users and 8 other stakeholders) reflected on the 18 month trial at the Faculty of Economics and Business in the University of Sydney using Rogers' (2003) diffusion of innovation factors. The main limitation of this research arises from the context and the qualitative nature of the study. Although the sample size was small, the number of academics who can potentially adopt an innovation being trialled within a single faculty context is by definition limited. However the rich qualitative data provides useful evidence of the potential pitfalls for academics seeking to trial ERS in their own contexts. By including three other non-adopter academics, as well as academic developers and support staff, this qualitative study was richer in a holistic sense in its analysis of why technology that is four decades old has made such little penetration. This is consistent with Slappendel (1996; p. 124) who notes that case studies generate new insights, particularly into complex organisational phenomena and permit generalisations of a theoretical nature.

Together with cultural compatibility and complexity challenges, academics report that the relative advantage of adopting the innovation is not positive. To paraphrase Draper, Cargill and Cutts (2002), to achieve improved learning experiences for students, we must first serve the teachers better. Various logistical and support solutions are needed to reduce the complexity and cultural compatibility barriers in order for academics to reverse the trend of poor uptake. While some of these solutions are relatively simple as long as resources are provided (e.g. permanent installations, template questions and on hand software and teaching training, and support for good practice usage), we believe the most significant barrier is conception of teaching. This is consistent with Luedekke (2003) who found that staff who view teaching as helping students change their conceptions are more likely to design learning contexts that result in deep approaches to learning, than those that view teaching as information transmission. The former are more likely to pursue interaction and engagement strategies both with and without technology.

We believe the results are of interest to both academics considering adopting this particular learning and teaching innovation, and academic developers interested in promoting innovative teaching practices, including educational technologies, that seek to improve the student learning experience. Based on the findings of this study, we posit that the greatest opportunity for improving the student learning experience through technological innovation in this and other contexts might be to promote those who hold change conception views of teaching to lead departments and to lead pilot innovation teams in localised projects, and to ensure that the majority of team members hold such views. A majority of team members holding student centred conceptions is essential. In addition, to optimise the chances of an innovation being trialled successfully, evaluated, and if appropriate, embedded successfully, project teams should include support staff and both academics who are in the early majority category as well as those who are primarily innovators or early adopters.

Future research should consider the application of the Rogers model to the uptake of ERS in other contexts. While some researchers have challenged the diffusion framework (McMaster & Wastell 2005), we have found it a useful model to explore the failure to adopt this technology. Future research with this and other models could also focus on the use of project team members selected on the basis of their adopter status, particularly if that was combined with a requirement for the majority of the project team to hold a change conception view of teaching.

Acknowledgments

This project was funded by a University of Sydney teaching grant.

References

Alexander, S. & McKenzie, J. with Geissinger, H. (1998). Evaluating IT projects for university learning: The CAUT experience. EIP study, Australian Government Publishing Service. [viewed 21 Aug 2006] http://www.dest.gov.au/archive/cutsd/publications/exsummary.html

Anderson, T., Varnhagen, S., & Campbell, K. (1998). Faculty adoption of teaching and learning technologies: Contrasting earlier adopters and mainstream faculty. The Canadian Journal of Higher Education, 28(2, 3), 71-98.

Bain, J. D., McNaught, C., Mills, C. & Lueckenhausen, G. (1998). Describing computer-facilitated learning environments in higher education. Learning Environments Research, 1, 163-180.

Bates, T. (2000). Managing technological change: Strategies for college and university leaders (1st ed.). San Francisco: Jossey-Bass.

Beatty, I. (2004). Transforming student learning with classroom communication systems. EDUCAUSE Research Bulletin, 3, 1-13. [viewed 15 Jan 2007] http://www.utexas.edu/academic/cit/services/cps/ECARCRS.pdf

Becher, T. (1994). The significance of disciplinary differences. Studies in Higher Education, 19(2), 151-161.

Bell, M. & Bell, W. (2005). It's installed... now get on with it! Looking beyond the software to the cultural change. British Journal of Educational Technology, 36(4), 643-656.

Bennett, R. (2001). Lecturers' attitudes to new teaching methods. International Journal of Management Education, 2(1), 42-58.

Boud, D. (1999). Situating academic development in professional work: Using peer learning. The Journal for Academic Development, 4(1), 3-10. [verified 6 Apr 2007] http://www.education.uts.edu.au/ostaff/staff/publications/db_24_ijad_99.pdf

Boyer, E. (1990). Scholarship reconsidered: Priorities of the professoriate. Princeton. NJ: The Carnegie Foundation for the Advancement of Learning.

Burton, K. (2004). Interactive PowerPoints: is there any point in giving power to students? E Law - Murdoch University Electronic Journal of Law, 11(4). [viewed 15 Jan 2007] http://www.murdoch.edu.au/elaw/indices/issue/v11n4.html

Burton, K. (2006). The trial of an audience response system to facilitate problem-based learning in legal education. In D. A. Banks (Ed), Audience response systems in higher education: Applications and cases. (pp 265-275). Hershey, USA: Information Science Publishing.

Crouch, C. H. & Mazur, E. (2001). Peer Instruction: Ten years of experience and results. American Journal of Physics, 69, 970-977.

Cutts, Q. (2006). Practical lessons from four years of using an ARS in every lecture of a large class. In D. A. Banks (Ed), Audience response systems in higher education: Applications and cases. (pp 65-79). Hershey, USA: Information Science Publishing.

Draper, S. W. & Brown, M. I. (2004). Increasing interactivity in lectures using an electronic voting system. Journal of Computer Assisted Learning, 20, 81-94.

Draper, S. W., Cargill, J. & Cutts, Q. (2002). Electronically enhanced classroom interaction. Australian Journal of Educational Technology, 18(1), 13-23. http://www.ascilite.org.au/ajet/ajet18/draper.html

Elliott, C. (2003). Using a personal response system in economics teaching, International Review of Economics Education, 1(1), 80-86. [verified 6 Apr 2007] http://www.economicsnetwork.ac.uk/iree/i1/elliott.htm

Elton, L. (2003). Dissemination of innovations in higher education: A change theory approach. Tertiary Education and Management, 9, 199-214.

Fies, C. & Marshall, J. (2006). Classroom response systems: A review of the literature. Journal of Science Education and Technology, 15(1), 101-109.

Frambach, R. T. & Schillewaert, N. (2002). Organisational innovation adoption: A multi-level framework of determinants and opportunities for future research. Journal of Business Research, 55, 163-176.

Freeman, M. & Blayney, P. (2005). Promoting interactive in-class learning environments: A comparison of an electronic response system with a traditional alternative. In S. Cheung (Ed.), Proceedings of the Eleventh Australasian Teaching Economics Conference: Innovation for Students Engaged in Economics, 23-34, University of Sydney. [verified 6 Apr 2007] http://ses.library.usyd.edu.au/bitstream/2123/199/1/03+Freeman+Blayney.pdf

Freeman, M., Blayney, P. & Ginns, P. (2006). Anonymity and in class learning: The case for electronic response systems. Australasian Journal of Educational Technology, 22(4), 568-580. http://www.ascilite.org.au/ajet/ajet22/freeman.html

Geoghegan, W. (1995). Stuck at the barricades: Can information technology really enter the mainstream of teaching and learning? Change, 27(2), 22-30.

Goodyear, P. (2005). Educational design and networked learning: Patterns, pattern languages and design practice. Australasian Journal of Educational Technology, 21(1), 82-101. http://www.ascilite.org.au/ajet/ajet21/goodyear.html

Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66(1), 64-74.

Healey, M. (2000). Developing the scholarship of teaching in higher education: A discipline-based approach. Higher Education Research and Development, 19, 169-189.

Hicks, O. (1999). Integration of central and departmental development - reflections from Australian Universities. The Journal for Academic Development, 4(1), 43-51.

Hutchings, P. (2000). Opening lines: Approaches to the scholarship of teaching and learning. Menlo Park, CA: The Carnegie Foundation for the Advancement of Teaching

Jones, C., Connolly, M., Gear, A. & Read, M. (2001). Group interactive learning with group process support technology. British Journal of Educational Technology, 32 (5), 571-586.

Judson, E & Sawada, D. (2002). Learning from past and present: Electronic response systems in college lecture halls. Journal of Computers in Mathematics and Science Teaching, 21 (2), 167-181.

Kirkpatrick, D. (2001). Staff development for flexible learning. The International Journal for Academic Development, 6(2), 168-176.

Latchem, C. (2005). Failure - the key to understanding success. British Journal of Educational Technology, 36(4), 665-667.

Lorenzo, M., Crouch, C. H. & Mazur, E. (2006). Reducing the gender gap in the physics classroom. American Journal of Physics, 74(2), 118-122.

Lueddeke, G. R. (1999). Toward a constructivist framework for guiding change and innovation in higher education. The Journal of Higher Education, 70(3), 235-260.

Lueddeke, G. R. (2003). Professionalising teaching practice in higher education: A study of disciplinary variation and 'teacher-scholarship'. Studies in Higher Education, 28(2), 213-228.

Martin, E. & Ramsden, P. (1994). Effectiveness and efficiency of courses in teaching methods for recently appointed academic staff. DEET Evaluation and Investigations Program. Canberra: Australian Government Publishing Service. [verified 6 Apr 2007, abstract only] http://www.dest.gov.au/sectors/higher_education/publications_resources/profiles/archives/ effectiveness_efficiency_courses_teaching_acaemic_staff.htm

McKenzie, J., Alexander, S., Harper, C., & Anderson, S. (2005). Dissemination, adoption and adaptation of project innovation in higher education. Carrick Institute of Learning and Teaching in Higher Education. [verified 6 Apr 2007] http://www.carrickinstitute.edu.au/carrick/webdav/site/carricksite/users/siteadmin/ public/Dissemination,%20Adoption%20and%20Adaption.pdf

McLoughlin, C. (2000). Creating partnerships for generative learning and systemic change: Redefining academic roles and relationships in support of learning. The International Journal for Academic Development, 5(2), 116-128.

McLean, J. (2005). Addressing faculty concerns about distance learning. Online Journal of Distance Learning Administration, VIII(IV). [viewed 21 Aug 2006] http://www.westga.edu/%7Edistance/ojdla/winter84/mclean84.htm

McMaster, T. & Wastell, D. (2005). Diffusion - or delusion? Challenging an IS tradition. Information Technology & People, 18(4), 383-404.

Mitchell, C. (2001). PRS Support System - A Summer Project. [viewed 21 Aug 2006] http://grumps.dcs.gla.ac.uk/papers/PRS_Support_System3.doc

Prensky, M. 2005. What can you learn from a cell phone? Almost anything! Innovate, 1(5). [viewed 21 Aug 2006] http://www.innovateonline.info/index.php?view=article&id=83

Prosser, M. & Trigwell, K. (1999). Understanding learning and teaching. UK: Open University Press.

Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: Free Press.

Russell, T. L. (2001). The 'No Significant Difference' Phenomenon. International Distance Education Certification Centre, 5th ed. [viewed 21 Aug 2006] http://www.nosignificantdifference.org/

Shapiro, J. (1997). Student response found feasible in large science lecture halls. Journal of College Science Teaching, 26(6), 408-412.

Sharma, M. D., Khachan, J., Chan, B. & O'Byrne, J. (2005). An investigation of the effectiveness of electronic classroom communication systems in large lecture classes. Australasian Journal of Educational Technology, 21(2), 137-154. [viewed 21 Aug 2006] http://www.ascilite.org.au/ajet/ajet21/sharma.html

Slappendel, C. (1996). Perspectives on innovation in organizations. Organization Studies, 17(1), 107-129.

Van Slyke, C., Lou, H. & Day, J. (2002). The impact of perceived innovation characteristics on intention to use groupware. Information Resources Management Journal, 15(1), 5-12.

Wilson, G. & Stacey, E. (2004). Online interaction impacts on learning: Teaching the teachers to teach online. Australasian Journal of Educational Technology, 20(1), 33-48. [viewed 15 Feb 2007] http://www.ascilite.org.au/ajet/ajet20/wilson.html

Winefield, A., Gillespie, N., Stough, C., Dua, J., Hapuarachchi, J. & Boyd, C. (2003). Occupational Stress in Australian Universities: A National Survey 2002. International Journal of Stress Management, 10, 51-63.

Appendix: Rogers' (2003) reflections

The relative advantage of the innovation over what it replaced (in time, cost, effectiveness, quality of results, etc). Is it better than the status quo?
The innovation's compatibility with existing practices, values, needs, culture, or conversely its disruptiveness. How does the innovation fit with people's past experiences and present needs? It if doesn't fit both well, it won't spread well. Does it require a change in existing values? If members of the culture feel as though they have to become very different people to adopt the innovation, they will be more resistant to it.
The complexity of the innovation: how difficult is it to learn, understand, and to use effectively? How difficult is the innovation to understand and apply? The more difficult, the slower the adoption process.
The innovation's trialability: how easy is it to experiment with the new way of doing things before making an adoption decision? Can people "try out" the innovation first? Or must they commit to it all at once? If the latter, people will be far more cautious about adopting it.
The observability or visibility to other potential adopters of the results achieved by using the innovation. How visible are the results of using it? If people adopt it, can the difference be discerned by others? If not, the innovation will spread more slowly.

Authors: Mark Freeman and Amani Bell are from the Office of Learning and Teaching in the Faculty of Economics and Business [http://www.econ.usyd.edu.au/], University of Sydney. Carole Comerton-Forde is in the Discipline of Finance in the same Faculty, and Joanne Pickering and Paul Blayney are in the Discipline of Accounting in the same Faculty.

Address for correspondence: Mark Freeman, Office of Learning and Teaching in Economics and Business, Merewether H04, University of Sydney, NSW 2006, Australia. Email: m.freeman@econ.usyd.edu.au

Please cite as: Freeman, M., Bell, A., Comerton-Forde, C., Pickering, J. and Blayney, P. (2007). Factors affecting educational innovation with in class electronic response systems. Australasian Journal of Educational Technology, 23(2), 149-170. http://www.ascilite.org.au/ajet/ajet23/freeman.html

Print version errata 22 May 2007: p.152. '... laggards (2.5% of the population)' should read '... laggards (16% of the population)'

[ AJET 23 ] [ AJET home ]
HTML Editor: Roger Atkinson [rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet23/freeman.html
Created 16 May 2007. Last correction 22 May 2007.
Page access count