ISM-6155: Enterprise Information Systems

Ism6155--Enterprise Information Systems--is the required capstone course in the MS in MIS program offered by the University of South Florida. Its specific objective is to help students, at the end of the program, synthesize the technical content of the program and place it in a management context. For the past four years, the course content has consisted of a blend of case discussions, debates on topics of interest to MIS professionals and research projects, all highly participative in nature. The course has been extremely well received by students--perceived by many to be the highlight of the program--and, in 2005, was the winner of the Decision Science Institute's Innovative Curriculum competition, against and international field of entrants.

Since spring 2003, the course has employed audience response technology (eInstruction's CPS system) as a tool for assessing student preparation (for case discussions and debates) and as a means of gauging student opinion. While the student reception to the technology has generally been positive, the instructor has had one major complaint with it: the multiple choice format required for responses tends to limit the types of questions that can be posed to students and--even worse--tends to emphasize factual preparation (which is relatively easy to test) as opposed to the degree that the student has actually tried to synthesize solutions (which is relatively hard to test with a pre-established set of answers). It was hoped, therefore, that by using Tablet PCs and Classroom Presenter a richer set of questions could be posed to students. This, in turn, would encourage them to place greater emphasis on analysis, rather than memorization, as they prepared for the discussion each week.

The class is currently almost entirely discussion-based (roughly 4 hours of lecture are included over the entire course of the semester), with preparation and additional out-of-class activities being central to course completion. As currently constructed, the course has three major components, each weighted equally in grading:

1. Case discussions. Students prepare and discuss over a dozen Harvard Business School cases each semester. Most discussions take place in the classroom, although one is held asynchronously (using a Blackboard discussion group) and one synchronously (using Elluminate software). During classroom sessions, students are tested on their knowledge of case facts using CPS (clicker) technology. The results of these tests are used in assessing the student's overall participation grade (1/3 of their total grade for the course).
2. Debates. 7 to 10 debates are conducted each semester (the exact number depends on class size), with roughly 1/3 of the class being assigned to a specific debate panel each week. Students get to sign up for specific topics (e.g., Within 50 years, computers will be able to perform nearly any routine task that would, today, require common sense) based on their interests, but have no choice as to whether they are assigned to the PRO side, the CON side, or as a moderator. At the beginning of each debate, CPS technology is used to survey the class regarding their position on the topic being debated (a survey that is repeated immediately after the the debate, so the impact of the debate on student opinion can be assessed). In addition, the entire class is quizzed on the briefing paper that has been prepared by the moderator. After that quiz, both PRO and CON sides present, after which the resolution is opened up for questions and comments by the class as a whole.
3. Strategic Systems Research Project. A long term project, continuing from semester to semester, that involves students determining the impact and the fate of about 100 widely publicized information systems developed in the 1980s.

Since the course had been working well, by nearly any standards, there was little or no motivation to make radical changes to the pedagogy as part on the project. What was of interest to the instructor, however, was seeing what would happen if the clicker technology were to be replaced with Tablet PC technology so as to support gathering richer responses from students. Of particular interest was seeing if Classroom Presenter technology could be used to foster greater depth and insights in case discussions.

Classroom Presenter (CP), delivered through a peer-to-peer network of Tablet PCs (since the university's general wireless network consistently failed to support the necessary multi-casting capabilities), has been the central technology of the project. It is being used in a variety of ways:

• At the outset of each case discussion, a general question, such as "What should XYZ's management do to respond to the problem presented in the case?" is posed to the students. They must reply using CP. The instructor then displays selected replies and uses them as a basis for choosing who will open the discussion.
• During the case, as students discuss the case, the instructor uses CP and his tablet in place of a chalkboard to write down key points and help keep the discussion organized.
• Near the end of each case discussion, the instuctor poses a question relating to "lessons learned" which is also answered by students using CP. These responses are then used as a tool for concluding the discussion.
• At the end of the class, the student responses (which include their names) are saved from CP. This provides a valuable tool for assessing the typical level of preparation for each student when it comes time to assign grades at the end of the semester
• For debates, the survey capability of the tool is used to assess class opinions--replacing the clicker system that was previously used for the same purpose.

The specific goal of the project is to determine if using Tablet PC technology to assess student preparation and to generate discussion points is more effective than using clicker technology. Success will be determined on the basis of:

• Student perceptions (compared with comparable reactions to clicker technology, which we've been gathering for 3 years).
• Instructor assessment--highly subjective, unfortunately--of discussion quality when tablets are employed.
• The perceived utility of student responses (i.e., saved in CP filed) is assessing participation and preparation when grades are prepared at the end of each semester.

As the instructor notes:

There are relatively few things that are harder to measure than the "learning" that occurs through case discussion. Such learning tends to be in the form of incremental improvements in judgment, which necessarily have a huge subjective component. What we can assess, however, is the degree to which the process we are using to encourage such learning seems consistent with our objectives. Clearly, asking factual questions about a case study would seem to emphasize the wrong type of preparation. With Tablet PCs and Classroom Presenter, we can pose the types of questions we really want our students to focus their attention on.

What we are particularly interested in seeing is how students react to Tablet PCs when compared to the CPS systems. In this context, there are 3 years of data regarding their perceptions of how the technology impacted their level of class preparation and interest in the course (using a Likert 1-5 Disagree-Agree scale). Because student evaluations of this course have historically been very high (typically ranging from 4.6 to 5.0 on a 1 to 5 scale), little evidence of impact from these measures was expected (unless large declines occurred).

• 50% of students felt that having clicker-based tests increased their level of case preparation
• 41% of students felt that having clicker-based tests and surveys increased their level of debpate preparation

During fall 2006, students indicated a strong preference for Tablet PCs over clickers. This was not a completely valid test, however, since tablet/CP responses were kept anonymous (while the instructor tried to determine effective ways of using the technology and learned to savor the intense excitement associated with actually getting the CP sessions to recognize each others' presence). With 11 of 12 students responding, reactions were as follows (none significant, owing to the small sample size):

• 64% felt the tablet use impacted their level of case preparation, whereas 55% felt the clicker impacted their preparation. This result was rather intriguing, in spite of the small sample, since the Tablet usage was anonymous, whereas the clickers were not--which would have led to the expectation that clicker impact should have been higher.
• 82% felt that tablet use was worth the class time required, whereas 73% felt that clicker use was similarly justified. Again, this result is interesting because tablet use required substantially more class time than clickers.

73% agreed that use of tablets added significantly to their learning. (No corresponding question was asked in relation to clickers).

In spring 2007, only tablet technology was used. This allowed a direct comparison between a pure clicker and a pure tablet class to be made, since using both technologies at the same time (as was done in the fall) would seem relatively unusual. Here the results were even more compelling. Contrasting spring 2006 (all clicker) to spring 2007 (all Tablet PC), we see:

• Whereas 44% of clicker users felt that using clickers has impacted their prepararation, the value was 69% for tablet users. The difference in means was significant art p>0.01 levels.
• Whereas 61% of clicker users felt the technology added value to the class, the number was 88% for tablet users. The difference in means was also significant.
• Highly significant differences in two overall benchmark questions were also observed. When asked to agree/disagree with the question (1-Strong disagree to 5-Strong agree) "I learned more in this course than in my typical MS-MIS course", the clicker means of 3.27 and tablet means of 4.31 were significant at the p ~ 0.001 level. On the question "I believe what I learned in this course will make me a better manager", the difference in means (3.69 vs. 4.75) was even more significant.

With respect to the last of these results, it should be noted that there was one other difference between spring 2007 and spring 2006. Whereas the earlier course involved performing research on the strategic systems project, the spring 2007 required students to design and build a system specifically to disseminate these results. This exercise proved to be highly popular--both with students and with the instructor--and may have cast a halo over other course perceptions.

Complte comments from student evaluations (Spring 2007):
This was one of the better course I have attended in this program.  The case method was excellent for learning objective of the class.

This is the best class in the MS/MIS program.  I learned a lot.  The case discussion concept is an excellent way of learning.

Good class.  Very valuable information for management of IT.

Best class and above and beyond the best professor  I’ve ever had.  It was so much more useful than ANY other class I’ve taken as a MIS student.

Excellent course.  Dr. Gill conducts this in a phenomenal way.  Shapes one to start thinking in an ‘out-of-box’ approach.  Thanks, Dr. Gill!

Great class.  Will use what I learned.  Thank you.

The case studies made the class more interesting by tying together technology and management information.

Dr. Gill is a very good professor and I really enjoyed his class.  Although I must say that I did agree with the students in the lst case that we read, but that’s basically because I have felt that all my other classes have not required enough. 

Every professor needs to follow the case scenario class discussions module.  If we are made to get ready to open cases like he did, everyone will learn something.  It could even be spontaneously asking questions in class.  Good job, Dr. Gill.

This is the best class I ever attended as a student.  Learned more than what I anticipated.

I have to admit, I had a huge challenge with the course structure on the first day, but after having concluded the course, I’m very glad I took it.  I learned a lot about technology and myself.  I think this class has helped me come out of my shell and not be so hesitant to express myself.  Thanks, Dr. Gill.  I wish you the best.

Excellent course, but always there’s space for improvement.  The course might include some material for preparation of case studies, strategy, tactic, management.  Debates should offer a format to follow.  Several debates were plain.  Debates people should generate some material to be shared with the class – notes / presentation and be posted in 66.   Thanks for everything.

The case studies are a great learning tool.  Also, the strategic system project provided a learning experience with practical implications.  I enjoyed the class.  This class is truly a capstone course.

Presentations

• AACSB Faculty Conference on Learning (June 2007)

Publications

• Decision Line 37(2)