With a history of innovative course material delivery mechanisms, the Information Management Department at Queen Margaret University College in Edinburgh, Scotland, decided to take advantage of funding from the University College to develop a Web site for teaching an undergraduate second year core module entitled Business Information Sources (BIS-Online). This site (http://jimmy.qmced.ac.uk/ usr/cimoff/temp/bis/) provides independent learning material supplementary to class lectures. We intended to:

Make students take responsibility for some of their own learning, since use of the Web site would be treated as a private study activity.

Minimize the potential for information overload in the subject area, since we would present only quality materials on the Web site.

Help students develop skills in evaluating electronic information services provision (bo in-house and exte through comparing and contrasting sites with similar subject area coverage.

Have students contribute publicly to the module over a mock intranet, for example, in providing the results of a company profile exercise online.

Test students' understanding of module content through the use of online multiple-choice quizzes developed with Perception software.

Encourage students' participation in an active online environment that replicates those used in industry for team working.

The Web site's established structure now provides an overview page giving the weekly components of the module with links to lecture notes, seminar exercises, and required reading; an index page with links to seminars, exercises, and self assessment tasks; and navigation and help pages.

This organization and delivery of material reflects, to a degree, the structure of the Navigating Business Information Sources: A Practical Guide for Information Managers (Burke and Hall, 1998), which is used as the core textbook for the module. The visual appearance of the site is intended to reflect that of the textbook. For example, the use of purple of icons on the Web site matches the color scheme of the book jacket.

EXPECTATIONS FOR WEB LEARNING

We expected that, over the duration of the module, students would visit the site at least once a week to extract material in preparation for the weekly lectures and seminars. The lecture outlines could be brought along to the classroom for annotation. Since the seminar preparation tasks would require additional offline work, some materials would need to be printed or copied.

From a broad perspective, we wanted the implementation of this project to match the philosophy of recent UK government initiatives designed to enhance the learning process by improving the quality, flexibility, and effectiveness of higher education. Having the students evaluate their BIS-Online course discovered the student view of the Web site materials and alternative learning methods (for example, in terms

of the accessibility, retrievability, and usefulness of the information presented, and the "enjoyment" of the learning experience); and compared student performance with that of previous years. Hopefully, our findings will improve the electronic delivery of learning materials in UK higher education.

The methodology adopted draws on the techniques for evaluating online and hypermedia courses as described by authors such as: Alavi (1994); Draper, et al. (1994); Hutchings, et al. (1993) and Reiterer & Opperman (1993). The study extends the tools employed by Gillham, et al. (1999) in their evaluation of a Media Studies site (http://jimmy.qmced.ac.uk/usr/ cibutt/ narrgenn/). The initial data collection tool was a questionnaire based on the one developed for evaluation of the Media Studies site, designed to elicit information about:

Access and use of the site

Evaluation of key criteria

General views about Web-based education

Use of other computer facilities and feelings about computing demographics

QUESTIONNAIRE RESPONSES AND DEMOGRAPHICS

The questionnaire was completed by 12 students, representing exactly half the population taking the module. All were in their second year of the integrated degree in Information Management. Of the five female and seven male students, three were direct entrants to the course at Level 2 (as opposed to continuing their studies from Level 1 of the Information Management degree course as the rest of the group had done), two were international students, and 75% were in the 18-25 age group. Threequarters had access to a computer at home, but Internet access from home was possible for only 25%.

Realizing the limitations of the earlier study, we extended this evaluation to in-depth qualitative data gathering through focus groups. This provided the opportunity to follow up on the findings from the questionnaire responses and permitted the triangulation of results. The focus group was composed of seven students who had volunteered to take part at the same time that they completed the questionnaire. There were three female and four male students, two of the males being international students. One researcher acted as facilitator for the group. The proceedings were recorded on audio and videotape.

To compare the performance of the first set of students who worked with the new Web materials with that of those who did not have access to such facilities, we analyzed similar examination marks back to the 1993/4 academic year (when the Business Information Sources module was first added to the Information Management course).

We gathered Web usage statistics from the automatic logging records of the host server to enable analysis of actual number of visits to pages of the site. These figures indicate all requests for the pages, but do not specify where or from whom the requests originate. They can, therefore, only be used as an approximation to the activities of the students studying the module between February and May 1999.

ACCESS FREQUENCY

Responses from the students' questionnaires indicated that they all had used the BIS-Online site. Frequency of access was "most days" (8) or "once a week" (4). The estimated average length of a typical visit was 14 minutes with reported lengths of sessions ranging from 2 to 90 minutes. In terms of frequency of access, the key part of the site seemed to be the lecture and seminar plans and notes. Other areas, such as links (to external sites), links to study skills materials, the module descriptor, and the contacts page were not accessed as frequently across the sample.

Analysis of request data logs generated automatically by the host server gave an indication of student activity. Care must be taken in interpreting these figures, however, as requests for the pages can be generated by people other than the students on the module: the server-provided statistics do not actually record who makes page requests. It is reasonable to assume, however, that the bulk of requests, totaling about 4,500 during the period under study, were from the students studying the module. Lecture and seminar plans and information pages were commonly accessed. The highest access was, however, for the main home page, which, according to the questionnaire returns, was visited only "occasionally." It seems likely that this page was visited as a means of getting to the more important work areas and not considered to be a real working visit by the respondents. Consequently, it was under-reported. Other casual visitors to the site could also have caused the high number of hits for the main home page.

The site was most frequently accessed during the first two months of the semester, February and March. It should be noted that the students' Easter holidays fell in March and April. This reduced the number of working days in each month. The server statistics do not report on pages where access is less than 20 requests in one month. This inevitably results in some under-reporting of requests for the less popular pages and other pages in the quieter times.

USE OF THE SITE

We asked an open question about the students' activities when visiting the site, with the option of giving up to three answers. Everybody responded that they had printed material from Web pages, but only one reported saving to disk. Five had also copied information by hand, proving that pencil and paper are not completely out of fashion.

Much of the material provided on the site in advance of classes could have been prepared and distributed on paper at the beginning of the module by the lecturer. However, students preferred extracting information for the module directly from the site by themselves. One commented, "It [the information] was alive on the Internet," referring to the attractive onscreen presentation, the method of navigation, and the promise of new and updated material, which added an element of excitement: "You wanted to look at the stuff because it was updated all the time" [referring to external sites linked to the BIS-Online site]. Some elements presented online, such as the interactive multiple choice quizzes and links to other sites, could not have been presented in an equivalent way on paper.

For the future provision of online information, we were interested in how the students work best with the material provided by the site. There are practical considerations, such as access to centrally-provided computers, time constraints, and costs. Focus group participants discussed the possibility of charging far printing. Not surprisingly, respondents were not keen on the idea. They suggested that this would reduce efficiency and possibly cause congestion: "I think it is going to create more queues in the IT centre...if you can't afford to print it, then you are going to have to sit and write down all the headings. Then you are going to spend a lot more time." The current practice of providing lecture outlines as well-spaced headings (to enable note-taking in class) would need to be reviewed if students were to be charged by the page. In sites like this where there is a possibility of charging for printing, provision should be made for students to download material quickly for printing elsewhere.

TIME AND LOCATION

Putting course material on the Web means that students must have access to an Internet-connected computer for this part of their work. The issue of access in the University College computer center was raised by focus group participants in terms of:

Finding a free computer at which to work

Managing time (taking into account the center's limited opening hours)

Lack of privacy in the center

Poor environment for study because of noise and other disturbances

The few who could access the Internet at home commented on its benefits based on avoiding the factors listed here. Those without home access were at a disadvantage. This is true for all work that requires the use of a computer, but becomes especially important when considerable access is needed for particular facilities and the home institution has dif faculty in meeting user demand.

Having the module program clearly shown on the site led participants to comment that this aided time management. One student said that this "allows you to be more efficient... because you know in advance what is happening." However, it can be argued that this is simply good module administration, not necessarily a feature of online module delivery. In later student discessions on the benefits of face-toface contact, the opposite view was expressed when it was suggested that in a purely online environment "you need to be very dedicated and manage your time."

PERCEIVED VALUE OF THE SITE

Evaluation of the site can be considered at a holistic level looking at the overall value to the students and also by examining components that make up the overall experience. Questionnaire respondents stated that the site was "essential" (9) or "useful" (3) in their studying of the module (no one said it was "of limited use" or "of no use"), and all but one stated that the site had assisted study "a lot". Most (11) thought the site was the right size with one respondent thinking it should be expanded.

Students were asked to rate various components of the site on a scale from 1 (excellent) to 5 (poor). All areas scored well. "Quality of information" and "usefulness of content" were ranked equal highest with a mean score of 1.33. Only one criterion was graded lower than 1. This was for "fun to use," indicating, perhaps, the serious nature of the material.

Some participants reported that they were happy reading on-screen, while others preferred to print information for reading. There is perhaps an age difference here, with one of the younger participants stating that "most of us are fairly used to reading off a computer screen...we have been exposed to it from a relatively young age." As more teaching staff adopt the practice of providing course material online, and especially with the future possible introduction of printing charges at Queen Margaret University College, there is likely to be more reading from the screen. It is, therefore, important for the success of online systems that screen design takes human-computer interaction principles and guidelines into account.

The design of the BIS-Online site was well-received: "very easy to read...with the...ease of navigation". When asked what feature of the site was most memorable, the participants commented affectionately about the purple color scheme, making the link with the course textbook. (The color purple was also readily associated with the lecturer who taught the module, since it was known to be her favorite color and she often wore a purple suit to class!)

VALUE OF PERSONAL CONTACT TO LEARNING AND MOTIVATION TO LEARN

Evaluation of Web sites such as this cannot fully be done in isolation of other aspects of course delivery. The BIS-Online site was just part of the overall experience of studying the module. The participants were clearly worried about removal or reduction of the amount of more traditional face-to-face contact session.

Lectures were appreciated for facilitating the linking of issues and relating of theory to real-life practice. As one student put it "the lectures [are] more traditional and a way [in which] we are more used to taking information...it's easier to take something in when someone is speaking to you... you can confirm it...you are sure you are not going off on the wrong tangent."

The seminars were also valued as an opportunity to reinforce learning with practical work and discussion: "What you learn in the lectures is all useful, but it's what goes on in the seminars that's like what you really have to learn...and you can spend more time discussing things."

From the focus group discussions, it became clear that the students taking part had strong views on the value of face-to-face contact as a major part of their learning experience. There is an argument that students attending full-time courses do so to get the benefit of face-to-face tuition from lecturers as well as the chance to learn from their peers. This was emphasized in the focus group discussions: "I made the decision to come full time as a student so, therefore, part of that reason was the face-to-face. That's important to me." One participant considered his experience on the Information Management course as favorable when compared with a previous course studied in Norway: "I prefer doing this course because I [am]... coached through the whole thing rather than having to do everything on my own."

As well as a means of facilitating better learning, face-to-face contact was deemed important for motivation and building working relationships: "You motivate each other or you support each other...whereas, if you are doing distance learning you feel very much out on your own and it's down to you to say `right come on you'd better get moving and doing something'.... You don't have someone saying `what did you do for this?' or `did you manage that?' or `what on earth did they say? I didn't understand a word'." This was seen as an important preparation for work: "[It's] about developing relationships and that's important because once you go to work anywhere you have to be able to relate to other people."

The focus group participants did not see the need for online discussion facilities and were, in fact, rather ambivalent to this idea. They expressed a need for good interpersonal skills and felt that class discussions were valuable forums for this: "if you are going to have a discussion I say it is preferable to have it face-to-face 'cause it is instantaneous when you come up with ideas, you can fire [them] off at each other". It should be remembered, however, that focus group volunteers are likely to be the more confident communicators from any set of people.

COMPARING PERFORMANCE

A comparison of the performance of the first set of students who worked with the new Web materials with that of those who did not have access to such facilities prior to 1998/9 reveals interesting results. We analyzed examination marks back to the 1993/4 academic year (when the Business Information Sources module was first added to the Information Management course). A marked improvement is shown for 1999. The mean mark for 1999 tested against the means for the previous five years indicates that marks for all years except 1995 were significantly different from those gained in 1999. Uniquely, there were no fails in 1999. Care must be taken in the interpretation of these data, as there may be factors other than the availability of the BIS-Online Web site that could account for the improvement in performance.

The feedback about the module in general, and regarding the Web site in particular, was universally favorable. It is difficult in a study of this kind to isolate the effect of a Web site from other aspects of the student experience on the module, and indeed from the influence of the particular class group. It was evident from the contributors to the focus group that the whole experience of studying the module was highly positive and that the Web site significantly contributed to this.

Other components that were identified as important to the successful student experience did not necessarily depend on the provision of a module Web site. The students identified these additional features as interpersonal characteristics of the staff and students, such as (a) the commitment and enthusiasm of the lecturer; (b) the ease of communication with the lecturer ("always email contactable... your answers would come back within one to three hours unless she was...out of the building"); and (c) the cohesion and co-operation within the student group ("for this research I think it is worth mentioning that we all-well the majority of us-at together at tea-time and lunch-time").

The emphasis on the vocational relevance of the module material and, by association, the degree program as a whole, encouraged the students. One of the seminar exercises required the students to analyze job advertisements for business information specialists. One of the focus group students referred to this when saying, "I think you could see the point-a lot of real life stuff: It's useful. We were all convinced we were well on our way to being able to achieve those [highsalary information jobs]....That is a great feeling half way through university and half way through a huge amount of debt for coming to university." The variety of resources for the learning environment made what might be regarded as a rather dry subject area more interesting to tackle, and the Web site simply enlarged the scope for diversity: "there was a lot of stuff in there, a lot of content...but you had so many resources to deal with them, which was wonderful....We had different exercises and you were reading about it from the book, and you could also look at the Web site that then reinforced the names for you.... It was not as if you were just reading the lecture notes or just reading a book, but the fact that you were doing a variety of things."

On the whole, this evaluation revealed that the initial hopes for the BIS-Online Web site were realized. The students used the Web site as an aid for private study. Certain aspects worked better than others. For example, the multiple choice quizzes were popular due to their interactive nature and ability to provide instant feedback and pointers to the correct answers, whereas the proposal for students to share material generated by a company-profiling exercise failed due to technical difficulties. Complaints about information overload that had characterized the module in its earlier manifestations were less common.

Although the student view of the use of Web site as an alternative tool for learning was positive, this evaluation raises a few critical issues:

Students value highly face-to-face contact with staff

Students value highly face-to-face contact with their peers

Convenience of access to Webbased resources is critical to their uptake

From a lecturer's point of view, the initial effort in setting up a Web-based resource and the subsequent commitment in keeping it current are also of importance. It is certain that BISOnline would not have been created without central funding to employ a contract Web developer who put the existing teaching resources online.

Perhaps the strongest message from this evaluation is to what extent the novelty of the method of information presentation enthuses students. Between 1994 and 1999, each course had been taught by the same lecturer using essentially the same material, albeit with content updates each year. The difference in 1999 was that virtually the entire module content was available online from the start of the semester. This "completeness" of presentation, linked in with the "traditional" aspects of the module, appears to have motivated the students to study hard and achieve exceptionally high results in the module assessment. As one focus group student explained, "No matter what the problem was, or what area it was in, I had a lecturer who I could contact in person or by email. I had lectures to go to, seminars to go to. I had classmates to work with and I had my book to read, and links on the Web site. It's wonderful."