Excerpts From the Fourth Report of the NSF National Visiting Committee on STEMTEC

Introduction-- The National Visiting Committee held its fifth on-site meeting with the STEMTEC project PI's, staff, and evaluators on October 29 and 30, 1999. Members present were: Angelo Collins (Vanderbilt University), Julia Cothron (Mathematics and Science Center, Richmond), Leo Hickey (Yale University), Paul Irish (Champlain Valley Union High School), John Layman (University of Maryland), Arnold Ostebee (St. Olaf College), and Barbara Tewksbury (Hamilton College). Jack Hehn and Katylee Hoover represented the NSF at the meeting.

The NVC had three items of especial concern that it asked the PI's to be prepared to comment on at the meeting: First was a progress report on the evaluation process. Second was a report on the STEMTEC summer workshop, and the third was a discussion by the PI's on the steps they had taken to increase the number of math and science teachers.

Overview and Recommendations-- As a result of our discussions with the STEMTEC staff and PI's and with their evaluation team, the NVC feels that the STEMTEC program has turned a corner and is now well on its way to the satisfactory completion of its stated goals. While there is no doubt that early missteps on the evaluation front cost the program valuable time, these problems are now in the process of being resolved and the program, while still behind in this area, is making rapid progress here. There has also been a commendable clarification of STEMTEC's goals and definitions of key terms. Another major improvement in STEMTEC's organization has been the assignment of specific areas of responsibility to the PI's and an augmentation and rationalization of the staff and the specific responsibilities of its members. Overall, it is the judgement of the NVC that STEMTEC is now a maturing program that has in place the requisite mechanism for identifying and implementing necessary changes.

The last section of this report consists of the NVC's specific comments on STEMTEC's performance in achieving its stated list of seven goals and the NVC's recommendations for improvement.

Goal #1: Establish a functional educational collaborative

In the committee's opinion this has been achieved. The original collaborative has been expanded to a total of eight, beyond the original five colleges. In addition, reports from faculty drawn from the two-year college partners indicate that these participants feel that they are being treated as equals with those from the original core of five colleges.

Goal #2: Redesign the science and math curricula...

This goal, and especially the aspects of it that depend on the presence of a well formulated mechanism for evaluation, has been the source for most of the criticism that the NVC has leveled at the STEMTEC program in the past. However, the NVC feels that the process of course redesign is now going well and that the PI's are meeting their goals in this area. In particular, we feel that the PI's have refined and sharpened their goals and expectations for STEMTEC courses and that they made very effective use of the STEMTEC summer workshop to convey these aspects of their program to the participants and to receive their feedback. The NVC regards the application questionnaire and the report procedure as a very positive development and commends the PI's and staff for them.

One of the ongoing concerns of the NVC has been with the definition and identification of inquiry as it is used in various STEMTEC documents. After careful consideration the committee does recognize that inquiry is one of the overarching expectations of the STEMTEC program.

The difficulties have lain in the projectís failure to define STEMTEC inquiry in terms that honor the broader use of inquiry within the national standards, in the NSF publication "Shaping the Future", and within the science-education reform community. The NVC therefore urges the PI's to more broadly define inquiry in terms used by other educators, and to describe measurable student actions that indicate that inquiry is taking place in the classroom.

Justification for the projectís commitment to the use of inquiry can be based on statements contained in two prominent national documents. From "Shaping the Future":

The goal--indeed the imperative--deriving from our review is that:

All students have access to supportive, excellent undergraduate education in science, mathematics, engineering, and technology, and all students learn these subjects by direct experience with the methods and processes of inquiry.

And from NSES:

TEACHING STANDARD A: describing the skill needed by teachers to "plan an inquiry-based science program for their students."

TEACHING STANDARD B: Teachers of science guide and facilitate learning which further describes how to support inquiry, especially in support of student diversity.

Again, while the NVC recognizes that in STEMTECís "Application for Inclusion of Course Document" the first characteristic of a STEMTEC course is, "It incorporates inquiry-based teaching methods", the centrality of this goal is not always apparent in the programís expository or evaluation documents. In addition, from the NVC's readings of STEMTEC materials, discussions with PI's and staff, and classroom visits by some NVC members (see appended reports), STEMTECís attention still seems to be focused on what experiences the students have had rather than what they have gained by those experiences.

STEMTEC could chose to justify its use of inquiry with quotations from NSES and "Shaping the Future", then provide a general description of the characteristics of inquiry that would apply to all aspects of the programís courses. Under this scheme, inquiry would move from being the third and "Essential Characteristic of a STEMTEC Course," [Charlene DíAvanzoís document of February 23, 1999] to a general characteristic to be associated with what would become at least seven, "Essential Characteristics of STEMTEC Courses" as enumerated below:

Assessment of each of these elements, when they are part of a course, would produce evidence that some of the inquiry procedures as operationally defined in the general description were present. Such an evaluation would document that the STEMTEC program was producing a broader set of pedagogical changes than just the four presently highlighted in the Application document, only one of which is identified directly with inquiry. Further investigation might well reveal additional changes in addition to the seven, listed above, that the NVC has recognized.

Goal 3- Improve the preparation of future K-12 teachers

While from anecdotal evidence this initiative appears to be going well, the NVC notes that not enough concrete data have been collected to provide the necessary documentation for this judgement. What is needed is a clearer articulation of the numbers of STEMTEC students involved in different categories of experience and of the types of activities engaged in. For example, the numbers of STEMTEC students involved in specific activities such as class sessions in a K-12 setting, or the numbers of students involved in peer teaching sessions, are just two of the categories of data that would allow a detailed assessment of the impact of the STEMTEC program. These data should also be organized by school and by individual STEMTEC course where possible. In addition, the NVC strongly recommends that the STEMTEC PI's push their respective school administrations to provide some form of tracking for those students who indicated an interest in teaching on STEMTEC questionnaires.

Despite the lack of concrete, refined data regarding teacher recruitment and preparation, the NVC has the impression that the recruitment of faculty and students on the mathematics side of the program seems to be lagging somewhat behind that of the science side at the five colleges. We also have the impression that the richness of the math experience for STEMTEC students, especially with regard to the use of technology such as graphing calculators, is less rich than it is in the science area. The NVC therefore recommends that the PI's devote more effort to recruiting math faculty and students, and that they specifically identify math oriented elements of the STEMTEC program. The relative proportion of mathematics faculty in the program and their use of appropriate technology should be explicitly documented. In addition, during the course of its visits, the NVC has perceived a degree of isolation between the science and math parts of the STEMTEC program. A means to decrease this might be to have the math faculty suggest some specifically math-oriented experiences that members of the science faculty could incorporate into their programs.

Finally, the NVC urges the STEMTEC PI's and staff to make sure to make explicit connections between its courses and the STEMTEC program in publications and presentations whenever this is feasible. For example, the summer-fall educational certification program was not presented as being a STEMTEC initiative at first.

Goal #4: Recruit and train promising students into the teaching profession, with special attention to underrepresented groups

The NVC feels that STEMTEC's efforts in this area have been commendable and urge that they be continued at the same level of intensity. Of especial value are STEMTEC's outreach to the junior colleges in the area and its efforts to put role models from underrepresented groups in front of classes.

Goal #5: Develop a program to support new science and mathematics teachers...

The NVC is pleased with STEMTEC's initiatives in this area as well. In particular, we commend the free web site that they have produced, as well as their list of mentors, and their program of support for new teachers. We simply remind the PI's of their need to document these important support elements for the upcoming Reverse Site Visit.

Goal #6: Establish dissemination mechanisms

The NVC is also very favorably impressed with STEMTEC's actions regarding dissemination. We feel that it has now achieved the status of a statewide, rather than regional, program. The publication of the book, a series of articles, and proactive representation at meetings of educators have all helped to do this. The PI's do need to insure that all such activities are carefully documented in terms of specific numbers and dates of presentations, titles and numbers of articles about the program, and the numbers of copies of any brochures distributed. The NVC also recommends that the STEMTEC logo be a prominent part of any presentations or publications that result from the program. The NVC feels that there are clear advantages to thus "trade marking" the package of innovations resulting from the STEMTEC program.

Goal #7: Conduct strong programs of evaluation and assessment

The NVC observes that the STEMTEC program now has a very powerful set evaluators in place and that the PI's seem to be making effective use of them. As usual, the case studies project is going very nicely and appears to have benefited considerably from the clarification in goals and methods that has taken place over the past year in the STEMTEC program as a whole.

A question regarding the evaluation process concerned the amount of emphasis that should be devoted to assessing whether students improve in terms of scientific literacy as a result of taking STEMTEC courses. There is no doubt that increase in scientific literacy and critical thinking ability should be ultimate benefits of STEMTEC for all of its students, whether they become teachers or not. The NVC is concerned, however, about the fact that the current evaluation plan calls for a considerable amount of effort to be expended on assessing whether students improve in terms of scientific literacy as a result of taking STEMTEC courses. Neither the improvement of scientific literacy nor the means by which it can be achieved were a part of the original STEMTEC charge. As a result, none of the programís courses have been designed specifically to address the issue of improving scientific literacy. Rather, the focus on course redesign has been on making students better at scientific inquiry and scientific thinking. The NVC believes that the great majority of the resource available for evaluation should be devoted to determining STEMTECís performance in achieving its stated goals and the effect that this is having, rather than in pursuing research on deeper issues of scientific understanding. We therefore urge the PIís and the evaluators to continue to focus on evaluating outcomes directly related to STEMTECís charge, for example whether students are, indeed, better at scientific inquiry a result of their STEMTEC experiences. As a final note on the evaluation process, the NVC suggests that there should probably be a greater emphasis than at present on face-to-face interviews, even though this is obviously more resource intensive than questionnaires.


One final recommendation that the NVC wishes to make is that the PI's turn their attention to assembling the data needed to support their presentation at the upcoming Reverse Site Visit. Much of this should be in the form of concise tables and graphs that sort the data into categories that are relevant to demonstrating that STEMTEC is achieving its stated goals.

Leo Hickey

For the National Visiting Committee

December 17, 1999

BACK TO National Visiting Committee

Created June 2000
Author: STEMTEC Webmaster