SUMMARY STEMTEC EVALUATION REPORT
This summary report includes data derived from student surveys administered during the Spring 1999 semester. The report is divided into five parts: demographics, teaching methods, assessment methods, attitudes and beliefs, and findings in relation to STEMTEC goals. The full report is available from STEMTEC HQ.
A total of 956 surveys were returned from 31 courses taught by 31 instructors in seven of the collaborating colleges. There were almost equal number of women and men among the students who returned the surveys. This differs from previous semesters because of the relatively large number of students enrolled in one course, UMass Physics 151 for engineers (24% of returned surveys), and that of those students, 78^% were men. Nearly 2/3 of the respondents identified themselves as Caucasian/White, with the rest divided among Latino (7.7%), African American (3.3%), Asian/Pacific Islander (7.8%) and Native American (3.1%). As might be expected, the distribution of ethnic minorities was different in the community colleges from the four-year colleges. While the percentage of African Americans and Latinos in the community colleges was approximately the same as in the four-year colleges, the percentage of Asian/Pacific Islanders at the community colleges was about 1/10th that of the four-year colleges.
Students were asked to indicate on the surveys the frequency that they experienced certain teaching methods. The modified Likert-type scale provided the students with these options: "Never", "Rarely", "About half the time", "Almost every class", and "Every class". The list of teaching methods on the survey can be found in Table 1. The students' responses to this section of the survey were analyzed along several dimensions. We first looked at all the responses from the seven colleges. We then de-aggregated the data into individual colleges, content domain, ethnicity of students, and gender. The complete set of graphs is available from STEMTEC HQ.
The student surveys indicate that "Listening and taking notes during class" is the most common teaching method experienced by the students enrolled in these courses. However, several STEMTEC recommended teaching methods were reported as experienced "About half the time" or more. These include "Working in small groups and/or pairs," "Instructor use of educational technology," "In class problem solving and/or open-ended questions," and "Opportunities to give feedback to the instructor." Two others, while not at the level of "About half the time…" were reported to occur often enough to note: "Participating in class discussions" and "Hands on activities." While this data do not support the conclusion that there has been a revolution in college mathematics and science teaching among the STEMTEC participants, it does suggest that their pedagogy have been expanded beyond the traditional college lecture. In addition, a comparison with data from Fall 1998 suggests that the changes are enduring. It is also important to note that the data suggest that some activities that are integral to the STEMTEC project are not occurring. The most important is "Collaborating with K-12 teachers and/or students", which, on the average, has almost never been experienced by the students enrolled in these courses.
There appears to be little difference in teaching methods among the colleges. Smith College students did report higher incidences of participation in class discussions and instructor use of technology. They also reported a higher incidence of opportunities to work on long term projects. STCC stands out among the community colleges as being the most "STEMTECed" group. This can be seen in significantly higher than average reports of working in small groups, participating in class discussions, the use of hands on activities, and the interactions with K12 teachers and students.
The second de-aggregation is by content domain: Mathematics, Science, and Education. This is where we see the largest differences among groups, with the education courses appearing to be using the most STEMTEC recommended methods while the science courses are using them the least. Students indicated that the education courses make use all of the STEMTEC methods more than half the time. Science students report a much lower incidence of participating in class discussions, opportunities to give feedback to instructors or ask questions in class, opportunities to work on long-term projects, and discussions on how students learn. Math students reported the use of most STEMTEC methods more than half the time. However, they also reported the most time spent "Listening and taking notes."
The third de-aggregated analysis of teaching methods was by reported ethnicity. It is important to note that the number of respondents among underrepresented minorities is quite low and may have little statistical significance. With that caveat in mind, we note that there appears to be little difference between the way that minority students and Caucasian students experienced these STEMTEC courses.
Gender was also used as a way to understand students' reports of their experiences in STEMTEC courses. It appears that there were significant differences between the ways female and male students experienced STEMTEC courses. Men experienced the use of educational technology, did in-class problem solving, and had hands-on activities more frequently than the women did. Women reported more frequent listening and taking notes, and opportunities to work on long term projects.
The data were then separated into four-year college, University and community college students by gender. Women at the four-year colleges report a higher frequency of experiencing "Participating in class discussions", "Opportunities to give feedback to instructors", and "Opportunities to work on long term projects." These differences were not seen between UMass or community college women and men. This suggests that the differences seen in the four-year colleges may be due to the fact that two of the colleges are all women.
The list of assessment methods on the survey can be found in Table 2. Students were asked to respond in the following way: A. Never B. 1 to 2 times a semester C. 3 to 5 times a semester D. 5 or more times a semester E. Used in class but not for assessment.
The data suggest that most of these assessment methods are rarely if ever used in the four-year colleges or the University. Only lab reports, class participation, and exams and quizzes other than multiple choice average above two. The community college data suggest that two other forms of assessment are being used on a regular basis: essays or other writing samples and short term projects.
The data on assessment methods were also de-aggregated in different ways. Among the four-year colleges, students from Smith and Mt. Holyoke indicate more frequent use of essays and writing, as well as long-term projects. Smith students indicated a much higher than average use of class participation for assessment purposes. Mt. Holyoke students indicated a higher than average use of short-term projects. UMass students reported more frequent use of pyramid exams. The item that shows the greatest variation is lab reports. Those differences are due to differences in the number of students surveyed at the colleges who were enrolled in laboratory courses. We noted few significant differences among the community colleges.
The final part of the survey looked at students' beliefs and attitudes towards science and mathematics, and science and mathematics education. These items were derived from a series of studies done by J. Randy McGinnis, Tad Watanabe, Gilli Shama, and Anna Graeber at the University of Maryland as part of the Maryland Collaborative for Teacher Preparation (MCTP), another Collaborative for Excellence in Teacher Preparation (CETP) funded by the National Science Foundation (NSF).
STEMTEC student responses in Spring 1999 appear to be very similar to those of the McGinnis data, as was the case for the STEMTEC Fall 1998 data. However, STEMTEC students indicate that they are more looking forward to taking more science or math courses, they like science or math courses more, and they agree more with the statement that hands-on activities should come before the introduction of new vocabulary. The STEMTEC data also suggest that the students have beliefs that are in-line with STEMTEC goals. There is one item in particular that is troubling, 48, which asks students whether they are familiar with STEMTEC. The vast majority (72%) of them reported that they are not.
As with the data on teaching methods, we have looked for differences in attitudes and beliefs among students in the seven colleges, by ethnicity and gender, and by content domain. We now report on the findings from these analyses. There appears to be little difference in beliefs among the students at the four-year colleges and UMass, or among the community college students. For the four-year colleges and UMass, there are few items in which students differ by college and/or gender. UMass men and women agreed more with the statements about the reasons for studying math or science. Men at Amherst and women at MHC disagreed more with the statements that math or science consist of unrelated topics. The community college data show little differences among colleges or between genders in the attitude and belief data.
Significant descriptive statistical differences are evident in the attitudes and beliefs data among different ethnic groups. In particular the data suggest that the African American students who responded to the survey agree with STEMTEC recommended teaching methods but see them as primarily a way to promote learning through concrete means such as the manipulation of materials, repetitive problem solving, and immediate rewards such as the acknowledgement that one has gotten the correct answer.
What do the findings from the student surveys tell us about STEMTEC's progress towards meeting its goals? It is important to note that the student surveys were not designed to provide evidence that can be used to evaluate progress towards goals 6 (support of new science/math teachers) and 7 (document and disseminate the program nationally). Therefore we will look only at the first five goals in this concluding section of the report.
Goal 1: Assist college faculty in learning and adopting new pedagogic approaches
Evidence from the student surveys suggests that STEMTEC faculty continue to use the teaching methods promoted by STEMTEC. While lecturing remains the most common form of instruction, other student-centered and student-active methods are increasingly being used.
Goal 2: Improve preparation of elementary teachers to teach science and math.
Goal 3: Improve preparation of middle school science/math teachers and elementary science/math specialists.
The student surveys provide little direct evidence that these goals are being met. However, there is evidence that STEMTEC is having an effect on the way that prospective teachers learn science. This can be seen in the use of STEMTEC recommended pedagogy by the college faculty. It can also be seen in the students' attitudes and beliefs about science and mathematics, and science and mathematics teaching.
Goal 4: Expose science, math, engineering majors to teaching experiences and careers.
Data from the student surveys suggest that there is still much to be done for STEMTEC to reach this goal through the incorporation of K12 teaching experiences in undergraduate mathematics and science classes. While the surveys show that few courses incorporate these types of experiences, data from other sources indicate that the courses that do have provided quality and worthwhile experiences for the undergraduates and K12 students.
Goal 5: Increase the number of women and minorities -- high school, college, returning students -- preparing to be science/math teachers, especially in inner cities and in poor rural areas.
The current data suggest that women are experiencing the same types of pedagogy in their math and science courses as men. This is different from previous data that indicated that Caucasian men were experiencing the courses in ways that encouraged them to continue in math or science while women were reporting the opposite. The current data also suggest that African American students are experiencing courses differently than majority students. Additional inquiry is needed to understand why.
Table I: Teaching methods in section II of Spring 1999 student survey
10. Working in small groups and/or pairs
11. Listening and taking notes during class
12. Participating in class discussions
13. Instructor use of educational technology (computers, videodisks, VCR’s, etc.)
14. In class problem solving and/or open-ended questions
15. Hands on activities
16. Opportunities to give feedback to the instructor
17. Encouraged to ask questions in class
18. Opportunities to work on long term projects
19. Discussions on how we, as students, learn best or why certain teaching approaches are used in the course
20. Collaborating with K-12 teachers and/or students
II: Assessment methods in section III of Spring 1999 student survey
21. Exams and quizzes other than multiple choice
22. Pyramid exams
23. Essays or other writing samples
24. Journal writing
25. Lab reports
26. Short term projects
27. Long term projects
29. Class participation
Table III: Items in Attitudes and Beliefs section of Spring 1999 Student Survey
30. The teaching methods used in this class helped me learn the course material.
31. This course has increased my interest in this subject.
32. This course has encouraged me to think about my own learning.
33. I am looking forward to taking more science [mathematics] courses.
34. The use of technology (e.g., calculators, computers, etc.) in the study of science [mathematics] will improve students’ understanding of science.
35. Getting the correct answer to a problem in the science [mathematics] classroom is more important than investigating the problem in a scientific [mathematical] manner.
36. Truly understanding science in the science [mathematics] courses requires special abilities that only some people possess.
37. Students should be given regular opportunities in class to think about what they are learning in the science [mathematics] course.
38. Science [Mathematics] is a constantly expanding field.
The primary reason for learning science is to learn how to apply mathematics.
[The primary reason for learning mathematics is to learn skills for doing science]
40. To understand science [mathematics], students must solve many problems following examples provided.
41. I like science [mathematics].
42. I enjoy using technology (e.g., calculators, computers, etc.) in science [mathematics] courses.
43. The use of technology (e. g., calculators, computers, etc.) in science [mathematics] is an aid primarily for slow learners.
44. Hands-on activities should come before the introduction of new vocabulary in science [mathematics] courses.
Science consists of unrelated topics like biology, chemistry, geology, and
[Mathematics consists of unrelated topics (e.g., algebra, arithmetic, calculus and geometry).]
46. Small group activity should be a regular part of the science [mathematics] class.
47. This course has increased my interest in becoming a teacher.
48. I am familiar with the STEMTEC program. a. yes b. no
 No data were received from Hampshire College because there were no STEMTEC-funded courses taught there in Spring 1999
 There is some question as to whether this data accurately reflects the involvement of college undergraduates in K12 situations as part of STEMTEC activities. The evaluators are seeking other ways to measure this effect.