Excerpts from the Third Report of the NSF National Visiting Committee on STEMTEC

Introduction—The National Visiting Committee held its third on site meeting with the STEMTEC project PI’s on October 2 and 3, 1998. Members present were: Leo J. Hickey (Yale University), Paul Irish (Champlain Valley Union High School), John W. Layman (University of Maryland), Arnold Ostebee (St. Olaf College), and Barbara Tewksbury (Hamilton College). Joan Prival and Katylee Hoover attended the meeting as representatives of the NSF. In the course of its visit the NVC attended a working luncheon with the STEMTEC Board, had detailed discussions with the STEMTEC evaluation team, and met with small groups of college and K-12 teachers involved with the project.

Once again, NVC members felt that this was a very positive visit with frank and forthright discussions taking place between the committee and STEMTEC PI’s, staff and participants. In general the NVC feels that the project is making good to excellent progress on most fronts and has effectively dealt with a number of concerns that the Committee raised in earlier meetings. In particular the changes made from Cycle I to Cycle II appear to have been positive; reports from the participants were generally very supportive. The PI’s are to be commended, as well, on establishing a new staff position that addresses scholarships and the improvement of minority participation.

However, the NVC’s focus in this third meeting was on the STEMTEC evaluation process, which the Committee found to be inadequate in its scope, planning, and execution. Much of the remainder of this report deals with the findings and recommendations of the NVC on the pressing issue of developing an effective evaluation program for the project.

Overview and Recommendations—After extended discussions with the evaluators and the PI’s, the NVC feels that STEMTEC does not have an adequate evaluation procedure nor appropriate advice on developing one… The NVC therefore recommends that it is imperative to develop evaluation instruments before the start of the Spring 1999 semester… The NVC also feels that it is urgent that some evaluation be done of a sample of current STEMTEC courses before the end of the present semester in order to develop a baseline for evaluating this phase of STEMTEC curricular development and for comparing its further evolution…

With regard to the whole issue of short and long term evaluation, the NVC notes that the goals statement needed as the basis for such evaluation has already been developed by the PI’s in their statement entitled, "What is a STEMTEC course?", written in response to the second NVC report. The NVC also requests copies of the relevant evaluation instruments as they are developed so that we may monitor the progress of this critical component of the STEMTEC program.

The NVC also feels that STEMTEC must achieve a broader context in its description of itself and its goals if it is to be a credible model for faculty development programs within the state and nationally. The feedback that the NVC received from the summer workshop participants, while almost universally enthusiastic, still indicates that many of them are unclear in their vision of successful teaching strategies.

With this in mind, the NVC urges that the general description of the STEMTEC project and its courses should be reformulated to conform with the terminology and context of the relevant documents describing the need for reform. These are "Shaping the Future" downloadable from the NSF website and the National Science Education Standards" available through the National Academy of Sciences website. Many of the concepts and approaches embodied in the STEMTEC program documents have already been defined and formulated in a national context and it is important that the STEMTEC literature make this available to a national audience by rendering it in the standard parlance. When persons read of the STEMTEC project in literature that uses the standard terminology they will credit the project with some of the insights expressed in these national documents even though they are not repeated in the STEMTEC material. This will also be important when it comes time to disseminate the results of the STEMTEC project on a national basis.

In general the NVC felt that, while the case study component of the project was progressing well, it needs to evaluate more than just attitudinal responses on the students’ part…

With regard to the STEMTEC board, the NVC suggests that, while encompassing, it appears to be too large to be used effectively as a body. We therefore recommend that it be restructured to form specialized committees; that the business members be cultivated, especially for their suggestions and possible support for a scholarship program; and that a few more people with minority/diversity perspectives be added.

The NVC also has some concerns on the issue of staffing for the STEMTEC project. If there are major elements that require staff they should be brought in, rather than adding to the PI’s already prodigious workload. The PI’s need the time and the space to carefully think through the aspects of institutionalizing, disseminating, and sustaining the STEMTEC initiative. The PI’s should be free to pursue the political, scholarly, and social details of their project as it matures and enters the dissemination phases.

In regard to the issue of the sustained impact of the STEMTEC program, the NVC feels at some disadvantage in not having before it a more clearly defined education component of the project that would guarantee formal and continuous contact with the STEMTEC students as they finish their programs. How are former STEMTEC students to be tracked? Will or might there be separate STEMTEC student gatherings, at the very least for those receiving scholarships?

Continuing our thoughts on the definition of the education component of the project: the NVC notes that the project and course descriptions that we have read often have a "home grown" terminology of their own. Project and course descriptions as well as faculty development plans should, as we recommend above, be based on national documents or make reference to external models. Some infusion from a project such as the NSF funded CESAME IMPACT Project, which is preparing curriculum trainers who will impart professional development and teacher support, would be advisable (p.hickman@nunet.neu.edu) so that the STEMTEC training and dissemination materials can be read and understood in a national context.

The NVC feels that it could render some important advice and council to the STEMTEC PI’s by a return visit in late March or early April of 1999. This would be a streamlined, one-day visit that would focus on looking at the revamped evaluation process through discussions with the new evaluation team. We expect that we will receive the necessary documentation for this visit at least two weeks in advance, to allow for the committee to formulate its questions. The Chairman will begin consultations with the STEMTEC PI’s and the Committee regarding suitable dates for this meeting and will have the discretion of canceling the meeting in late January or early February if he feels that it is not needed. Such a visit would allow the NVC to provide effective advice well in advance of the third year reverse site visit required by NSF. The NVC would also like to see some additional emphasis by STEMTEC on the issue of teacher preparation and will also be addressing this item at the spring, 1999, meeting.

Leo J. Hickey

For the National Visiting Committee

12 November 1998