Comprehensive STEMTEC EVALUATION PLAN

Submitted by: Audrey B. Champagne, Eileen O'Connor on January 17, 1999

(Scroll through the document to see the entire Comprehensive Evaluation Plan or click to the right on an entry to move directly to that area. When a footnote symbol appears within the documentation, you may click this item to move to the footnote.)

Introduction

Click here

Goal-by-Goal Evaluation Plan

 

 

Goal 1: Establish a functional educational collaborative

Click here

 

Goal 2: Redesign the science and math curricula on the campuses of the Collaborative to incorporate new pedagogies and establish mechanisms for supporting faculty in their course redesign



Click here

 

Goal 3: Improve the preparation of future K-12 teachers of mathematics and science

Click here

 

Goal 4: Recruit and retain promising students into the teaching profession, with special attention to underrepresented groups

Click here

 

Goal 5: Develop a program to support new science and math teachers in their first year in the classroom

Click here

 

Goal 6: Establish dissemination mechanisms

Click here

 

Goal 7: Conduct strong programs of evaluation and assessment

Click here

Appendix A: Interim Evaluation Plan

Click here

Appendix B. Populations taking the content-and-pedagogy survey

Click here

Appendix C. Faculty populations to be evaluated

Click here

STEMTEC EVALUATION PLAN

Introduction

This STEMTEC evaluation is goal based. Central to its conception are (1) the fourth element of the National Science Foundation (NSF) Outcome Goal, which is "Improved achievement in mathematics and science skills needed by all Americans" and (2) the seven goals of the STEMTEC Project contained in the proposal to the National Science Foundation. The questions the evaluation seeks to answer regarding the STEMTEC Project's progress toward meeting its goals and it achievement of those goals are posed in light of the NSF Outcome Goal.

The evaluation has four basic purposes: (1) to provide information to STEMTEC project managers, the National Visiting Committee (NVC) and the National Science Foundation about progress being made toward achieving project goals; (2) to provide evidence to the funding agency that the STEMTEC goals were met; (3) to provide information to the National Science Foundation that the Foundation can use to provide evidence to the US Congress about the Foundations achievement of its Outcome Goal; and (4) to provide information about what works to NSF and other institutions involved in systemic reform of science education. Toward meeting the first and second purposes, the STEMTEC evaluation will gather evidence that the STEMTEC is making progress toward achieving its goals. This evidence will provide STEMTEC Project Managers with information that can be used for planning and information that the National Visiting Committee and the National Science Foundation can use for their oversight responsibilities. Toward achieving the second purpose, the STEMTEC evaluation team will measure the degree to which students in collaborating institutions have achieved the science and mathematical skills needed by all Americans. This information can be used by the National Science Foundation to provide evidence to the US Congress about the Foundations achievement of its Outcome Goal. Toward meeting the third purpose of the evaluation, the STEMTEC evaluation will gather evidence about the effectiveness of STEMTEC Courses, that is, the degree to which students in STEMTEC Courses reach an acceptable level of science and mathematical skills needed by all Americans. The evaluation will attempt to and correlate student achievement with faculty understanding and use of innovative pedagogical processes. This information will provide NSF and other institutions involved in systemic reform of science education about what worked educationally.

The Evaluation Plan contains three types of Evaluation Questions: formative, summative, and effects. Certain of the formative and summative questions focus on the STEMTEC Project's stated goals. Formative questions serve to monitor the Project's progress toward meeting its goals and summative questions provide evidence of the Project's attainment of its goals. Other formative and summative questions address the intentions of the goals. By intention we mean that which the goal means to accomplish. For instance in the case of Goal 2, the intention is to improve students' understanding of mathematics and science. Consequently the formative and summative evaluation of Goal 2 ask questions about student learning. Effects questions evaluate the assumptions underlying the Project Goals. For instance, incorporation of recommended pedagogical strategies in science and mathematics courses is assumed to result in improved learning. An effects question would examine the correlation of the use of recommended pedagogical strategies with student achievement.

All of the STEMTEC Goals are enabling in respect to the NSF Outcome Goal element four. That is, the STEMTEC Project was designed with the assumption that if the enabling goals were met, improved achievement in mathematics and science skills of students in the collaborating institutions should be observed. The summative evaluation will measure the extent to which the enabling goals were met and the degree to which students in the collaborating institutions have gained the "mathematics and science skills needed by all Americans" and the skills and knowledge needed by science and mathematics teachers. The formative evaluation will measure the project's progress in meeting the enabling goals and the opportunity afforded students to gain the science skills needed by all Americans. The National Science Education Standards' Content Standards and the National Council of Teachers of Mathematics Content Standards will be the yard sticks(1)

against which students opportunity to achieve mathematics and science skills and their achievement of those skills will be measured. Yard sticks for the measurement of science and mathematics pedagogical content knowledge will be the national science and mathematics standards. (The National Visiting Committee may suggest that other standards such as those set forth by the National Board for Professional Teaching Standards (NBPTS), the AETS and the National Science Teacher Association (NSTA) be used for this purpose.)


Goal-by-Goal Evaluation Plan

Goal 1: Establish a functional educational collaborative

The document titled Strategic Plan for STEMTEC reviews past accomplishment among the collaborative members and provides the management structure of the STEMTEC collaborative. The STEMTEC Strategic Plan: Executive Summary asserts that Goal 1 "has largely been accomplished." Thus the evaluation issues central to the collaboration's continuation center around maintenance of an effective collaboration and a cost benefit analysis of the educational collaborative.

The formative and summative evaluation questions for Goal 1 address STEMTEC's progress toward meeting promises made in its proposal to NSF. The effects questions explore issues surrounding the assumption that collaboration among institutions is more effective than an individual institution efforts in improving achievement in mathematics and science.

The proposed formative evaluation strategy depends largely on gathering evidence contained in STEMTEC documents. Information contained in these documents can be used to verify the functional participation of collaborating institutions in STEMTEC work. Essential to evaluation of Goal 1 is an operational, that is measurable, definition of a functional collaborative. Such definition is necessary to guide the collection of acceptable evidence of functional collaboration.(2) For the present moment, we propose that attendance of responsible individuals at meetings and contributions of resources (personnel, fiscal, space) to the collaborative are indicators of a functional collaborative. Within the summative evaluation, it will also be necessary to evaluate and document how the collaboration has become a part of the institutional, administrative and/or academic structure of the participating schools. For any reform to become systemic, "innovations" must eventually become the norm.

With regard to maintenance and effects of the collaborative, the evaluation strategy will depend largely on interviews of responsible individuals to gather evidence about strategies that are being employed to maintain active participation and to understand the participant's perceptions about the costs and benefits of collaboration.

Progress Thus Far on the Evaluation of Goal 1

Although STEMTEC has established the foundation of a functional collaborative, documents should be assembled to substantiate participation among the various members and to indicate when members may need additional support. Periodically, interviews of key responsible parties within the collaborative should be conducted to capture the central elements that have made the STEMTEC collaborative so productive.

In addition, operational definitions of functional collaborations must be developed, that is, consensus must be reached on indicators of functional collaboration.

Formative Questions

What progress has been made toward meeting Goal 1?

Are all institutions in the collaborative continuing their participation in and contributions to the effective functioning of the collaborative?

What strategies are being employed to maintain the effective functioning of the collaborative?

Data Sources:

Documents including meeting agenda and notes, budgets..

Interviews of Key Responsible Parties

(Principal Investigators, Project Manager, Faculty and Staff from collaborating institutions who have STEMTEC Responsibilities, Faculty and Staff from collaborating institutions who do not have direct STEMTEC Responsibilities, administrators from collaborating institutions)

Summative Questions

Did the STEMTEC Project achieve Goal 1?

Will the Collaboration be sustained? Will funds be forthcoming from the various schools to support this endeavor once outside funding is no longer available?

Data Sources:

Interview of Individuals Responsible for each institutions' participation in STEMTEC

- How has the collaborative's goals become incorporated into their institution's structure so that outside funding is not required?

Effects Questions

Were the benefits of the collaboration commensurate with the costs, assuming no outside funding had been forthcoming?

Are there institutional and communications issues that must be addressed by any organizations that are considering similar broad-ranging education collaboration initiatives?

What role do the collaborative members ascribed to the NSF funding? That is, what would be different about the ultimate learning outcome, improving science and mathematics skills for all Americans, had the funds not been available?

Data Sources:

Interviews with STEMTEC Faculty and Staff from collaborating institutions, administrators from collaborating institutions

- How did the NSF funds facilitate a productive education collaborative? Would your institution have been willing to participate in the STEMTEC education initiative had outside funding not been available?


 

Goal 2: Redesign the science and math curricula on the campuses of the Collaborative to incorporate new pedagogies and establish mechanisms for supporting faculty in their course redesign.

The STEMTEC Strategic Plan: Executive Summary delineates the primary strategies that provide college faculty with funds, information, support, and evaluation tools, thereby facilitating their incorporation of innovative pedagogical practices into their courses. The Strategic Plan for STEMTEC further elaborates on workshops, seminars, and the like that have been made available to faculty. It appears that the evaluation tools, as explained in Appendix D of the Strategic Plan for STEMEC, explored primarily student attitude toward the course and student perception of the course workload and requirements. Therefore, for purposes of the meeting the National Science Foundation's Outcome Goal (number four), it is necessary to expand the breadth of the evaluation activities.

Two matters of interest in the evaluation of STEMTEC Goal 2 go beyond the exposure and perceptions of faculty and the attitudes of the students. One matter is faculty members' familiarity with what national standards say about the science and mathematics needed by all Americans and the faculty members' understanding and implementation of pedagogical practices recommended in national standards.(3) Consequently, faculty knowledge of national standards, understanding of recommended pedagogical practices, and implementation of these practices will be a focus of the formative and summative evaluation.

The second matter is the degree to which students in STEMTEC courses are exposed to and learn the science and mathematics needed by all Americans. Consequently, the evaluation of Goal 2 will also gather evidence about students' opportunity to learn science and mathematics and the degree to which they learn the needed content.

Two constructs, the science and mathematics skills needed by all Americans and recommended pedagogical practices require operational definition for this evaluation. Our discussion below, assumes that the National Research Council's National Science Education Standards (NSES) for Content will be the standard for this evaluation.(4) Recommended Pedagogical Strategies that have been emphasized in STEMTEC document are cooperative learning, critical thinking skills, problem-based learning, hands-on activities, educational technology, and new assessment methods. Consensus on the essential attributes of these strategies must be reached before further evaluation planning can proceed.(5)

Course redesign

Figure 1-Goal 2:Course Redesign illustrates the thinking behind how STEMTEC Goal 2 will contribute to improved achievement in mathematics and science skills needed by all Americans. The redesign goal is predicated on the assumption that the recommended pedagogical practices should result in better understanding of science and mathematics for all students and that students who sense that they understand are more likely to continue studying mathematics and science and to chose careers in science and mathematics. The diagram distinguishes all post-secondary mathematics and science students from those students who intend to become teachers. While in the overall NSF strategy both populations are equally important, the STEMTEC mission focuses on students who intend to become teachers. However, because most freshmen and sophomores have not chosen their professions, both populations must be given equal attention in the redesign of introductory courses. The assumption is that well designed courses, that is courses that apply recommended pedagogical practices, will serve as powerful motivators for further study of science and the choice of careers in science including science teaching. For these reasons, the achievement of needed science and mathematics of all students in STEMTEC courses will be evaluated. In addition, all students will be exposed to recommended pedagogical practices. It is assumed that this exposure will influence teaching practices of those who have chosen or will chose careers in science. Again, because the population of future teachers is not well defined, the formative evaluation of STEMTEC courses will consider all students' understanding of why the recommended pedagogical practices are effective.

Progress Thus Far on the Evaluation of Goal 2

In Strategic Plan for STEMTEC, considerable information is offered about the workshops, funds, seminars, and supports. This report also provides information about faculty perceptions of their experiences and the degree to which they implemented the recommended pedagogical practices. This information is valuable to STEMTEC managers as they continue providing faculty with experiences designed to develop their understanding of the recommended pedagogical practices and standards for science and mathematics content that are necessary for all Americans.

As explained above, it will also be necessary to evaluate how faculty integrate the needed mathematics and science content, as expressed in the national standards, into their course and how faculty employ the recommended pedagogies. Furthermore, it will be necessary to evaluate if students have learned the needed science and mathematics content and if students have understood the recommended pedagogies. Since it is not possible to designate which students will become science teachers, all students should be considered in the population of "student."

The two surveys of students mentioned below under "data sources" - the learning outcome surveys, so to speak - will be referred to collectively as the content-and-pedagogy survey since the evaluation would most likely be conducted as one survey. This survey could be used in many areas throughout the evaluation to examine an individual's understanding of science and mathematics content and of the recommended pedagogies. Using a common evaluation tool can facilitate the comparison of findings through the project. Throughout this evaluation plan, a number of applications of such a survey will be posited. An overview of these various populations is presented in the figure in Appendix B.

As mentioned below under "data sources," the evaluation of the courses taught by STEMTEC faculty should involve gathering evidence in several forms: course materials and syllabi, a pedagogy usage-and-understanding survey, and selected observations of classes. This assembled information will be referred to as the faculty-teaching analysis. In a manner to the content-and-pedagogy survey, this analysis would have a number of applications, thereby simplifying the process of gathering information and the later comparison of courses for their use of recommended pedagogies. An overview of the various teachers that can be analyzed using this methodology are presented in the figure in Appendix C.

Formative Questions

Is the science and mathematics content taught in STEMTEC courses congruent with the science and mathematics content needed by all Americans, that is, the science and mathematics content in the national science and mathematics standards?

Are the recommended teaching practices used in STEMTEC science and mathematics courses congruent with these practices as they are presented in national science and mathematics standards or other national science and mathematics teacher education and certification documents?

Data Sources:

Documents STEMTEC Course Syllabi and Tests

- Is the science and mathematics content taught in STEMTEC courses congruent with the science and mathematics content needed by all Americans, that is, the science and mathematics content in the national science and mathematics standards?

Student Performance on tests

- Have students in STEMTEC courses learned science and mathematics content needed by all Americans, that is, the science and mathematics content in the national science and mathematics standards?

Surveys of STEMTEC Faculty

- Is the understanding of recommended teaching practices held by science and mathematics faculty congruent with these practices as they are presented in national science and mathematics standards or other national science and mathematics teacher education and certification documents?

Surveys and Interviews of Students

- Is the understanding of recommended teaching practices held by science and mathematics students congruent with these practices as they are presented in national science and mathematics standards and science and mathematics teacher education and certification documents?

Observation of STEMTEC Classes

- Is the implementation of the recommended teaching practices by science and mathematics faculty congruent with these practices as they are presented in national science and mathematics standards and science and mathematics teacher education and certification documents?

Summative Questions

Has the Project adequately met Goal 2?

Will faculty members continue using the innovations? What are faculty members evaluation of the effectiveness of the innovations?

Has there been an improvement in the students' learning of the needed science and mathematics contents since the beginning of the STEMTEC project?

Data Sources:

Surveys and Interviews of STEMTEC Faculty

- What are the costs and benefits of implementing the recommended pedagogical strategies in college and university classrooms?

- Based on individual faculty member's cost/benefit analyses, will faculty continue using the innovation?

Effects Questions

Analysis of Implementation and Student Performance Data

- What correlations exist between faculty implementation of the recommended pedagogical practices and the level of student achievement?

Survey of Students who participated in STEMTEC Courses offered in the previous year


 

Goal 3: Improve the preparation of future K-12 teachers of mathematics and science

In Strategic Plan for STEMTEC, the preparation of science-education teachers is presented in terms of improving the pedagogy in science courses (as presented in goal 2), encouraging future elementary school teacher to take more science courses, integrating educational technology within the courses, creating a streamlined Science Major to encourage more middle and elementary science specialist, and reviewing the coursework required for graduation.

Evaluation of the specific course-taking goals can be accomplished by looking at the trends in course taking; evaluation of the student learning when students take these courses can be accomplished with the survey methods devised for goal 2. (See the chart in Appendix B.)

However, the Strategic Plan provides little information about the total process by which STEMTEC's future teachers will learn their craft. There appears to be no discussion of the field experiences where these students will first apply their understanding of content and first test their use of the recommended pedagogies. Nor is there any mention of the Method's Course where prospective teacher will learn the theories supporting effective practice or may be required to observe and participate in school settings. Preparing new teachers, much less improving this preparation, requires effective, coordinated, and appropriate instruction, supervision, and field work. Evaluating teacher preparation is a complex activity, where a range of activities should be considered:

graduating science-education students should be evaluated for their understanding of needed science and mathematics content and recommended pedagogies using methods devised for goal 2.

cooperating teachers, that is teachers who sponsor student teachers in their classroom, should also be evaluated for their understanding of needed science and mathematics content and recommended pedagogies using methods devised for goal 2. These teachers will play a critical role in modeling teaching practices to new teachers; it is important to evaluate the practices these teachers employ.

science-education courses should be evaluated for their use of recommended pedagogies and for their instruction in the theories; science education instructors and supervisors should be evaluated for their use of recommended pedagogies.

The improvement in teacher preparation may be difficult to measure in this context without sufficient baseline information about the quality of former teacher preparation. The effectiveness of STEMTEC teacher preparation, however, could be considered in terms of how the STEMTEC teacher education program meets national standards.

Progress Thus Far on the Evaluation of Goal 3

STEMTEC's Annual Report: Year 1 discusses several areas of teacher preparation not mentioned previously. The report notes that some "faculty . . . have not yet fully bought into the teacher enhancement mission of STEMTEC" (p. 7), apparently on the level of incorporating K12 teaching into their coursework. This document notes that reform measures included a practicum with student-active learning and the placement of student teachers STEMTEC K12 faculty to "help alleviate the dissonance between the ways in which the STEMTEC preservice teachers are taught to teach and the practices that they see in schools."

The evaluation questions for the present state of teacher education reforms should address the full range of activities mentioned above, that is, evaluating graduating students and the cooperating teachers for their understanding of science and mathematics content and recommended pedagogies and evaluating the science education coursework using the methods devised for the faculty in goal 2 (see Appendix B for a chart of faculty who are evaluated with these methods.)

Formative Questions

Are more prospective elementary and middle school teachers taking the revised Science Major than had taken the previous Science Major? Are elementary education students taking more science courses than in the past?

Are students who intend to become teachers learning the needed science and mathematics content and are they developing an understanding of recommended pedagogical practices congruent with these practices as they are presented in national science and mathematics standards?

Are students having their field experiences with teachers who use these recommended pedagogies? Assuming the cooperating teachers are modeling the recommended pedagogies, how has this exposure facilitated students teachers learning of pedagogy?

Are science education faculty incorporating the recommended pedagogies into their teaching practice?

Data Sources:

Documentation

- Do the enrollment data indicate more students are taking the revised Science Major? Do the enrollment data indicate that elementary-education students are taking science course, particularly STEMTEC courses?

Student Performance on STEMTEC Course Tests

- Have students intending to become teachers learned the science and mathematics content needed by all Americans, that is, the science and mathematics content in the national science and mathematics standards? (Evaluate with the methods devised for goal 2.)

Surveys and Interviews of Students

- Is the understanding of the recommended teaching practices held by science and mathematics students congruent with these practices as they are presented in national science and mathematics standards and science and mathematics teacher education and certification documents?

Surveys and Interviews of Cooperating Teachers

- Is the understanding of the recommended teaching practices held by cooperating teachers congruent with these practices as they are presented in national science and mathematics standards and science and mathematics teacher education and certification documents?

Summative Questions

At what level of performance vis a vis national standards are STEMTEC students who enter the teaching profession at the end of year 4 of the project?

Can STEMTEC students who plan to teach use the recommended pedagogical processes in ways congruent with these practices as they are presented in national science and mathematics standards and science and mathematics teacher education and certification documents?

Did modifying the Science Major bring about the intended increase in elementary and middle school science specialists?

Data Sources:

Documentation

- Have the course registration and graduation data indicated an increase in elementary and middle school specialist over the course of the STEMTEC project?

Tests of Content Understanding

- such as, tests written by STEMTEC Project Managers and Evaluators, tests developed using National Association for Educational Progress (NAEP) and Third International Mathematics and Science Study (TIMSS) released items, or the Massachusetts State Content Tests

Video tapes of lessons

- Do the videotapes of student teacher lessons effectively demonstrate that students can incorporate the recommended pedagogical practices into classroom practice?

Effects Questions

Overall, do graduates of a comprehensive STEMTEC teacher-education program appear to be better able to implement the recommended pedagogies? Do the videotapes STEMTEC education program graduates demonstrate effective implementation of the new pedagogies?

Data Sources:

Interviews of Science Education Faculty

- What changes in teacher preparation appear to be attributed to the STEMTEC changes in science courses? Would a school with traditional science courses be able to achieve these results, say if they only modified the teacher education component?

- How has placing student teachers with K12 teachers who employ the recommended pedagogies appeared to effect teacher education, particularly compared to situations where student teachers might be placed with teachers who employed traditional pedagogies?


 

Goal 4: Recruit and retain promising students into the teaching profession, with special attention to underrepresented groups

Toward the goal of increasing participation in teacher education, particularly among underrepresented groups, the STEMTEC Strategic Plan reviews the two-fold initiative presently underway. To increase the percent of the underrepresented students who will enter K12 education, STEMTEC is working with standing organizations within the Collaborative campuses that serve these populations. STEMTEC tailors the projects to the particular mission of each organization. In addition, efforts are underway to secure various financial supports to enable the recruitment and retention of minority students. To increase the number of science, mathematics, and engineering majors who will change to K12 education, STEMTEC cites efforts at improving science, mathematics and engineering courses (as proposed within Goal 2) and at having these students teach in college and K12 settings.

To evaluate the diverse programs within goal 4, first it is necessary to establish the evaluation tasks embedded within the goal and then consider appropriate evaluation questions. Broadly speaking, goal 4 can be considered as having both instructional aspects and administrative/logistical aspects:

As presented in greater detail with goal 2, a basic assumption underlying the STEMTEC project is that all students in STEMTEC courses (future teachers, underrepresented populations and science, mathematics, and engineering majors) have the opportunity to learn the science and mathematics content needed by all Americans and that all students be exposed to the recommended teaching practices. Therefore, the instructional situations within goal 4 that involve these students -- underrepresented students and science, mathematics, and engineering majors -- should be evaluated in a manner consistent with methods devised for goal 2. (See Appendix B for a chart of the different populations studied in this manner.) The learning outcomes of these students, regarding science and mathematics content and the recommended pedagogies, should be evaluated in a manner consistent with the evaluation in goal 2 as well.

This administrative/logistical components of the goal 4 should be evaluated by asking: Are these STEMTEC programs effectively reaching the intended students and are these STEMTEC programs providing these students with the resources and opportunities they need to remain within or change to science education?

Under the present discussion of the goal 4 evaluation, suffice it to say that, the courses students take and the teaching situations students become involved in will be evaluated with the procedures developed for goal 2. The details of their evaluation strategies will not be repeated. However, it should be noted that when these data are gathered, information should be recorded that allows the findings to be assigned to the appropriate student population, be it underrepresented students or science, mathematics, and engineering majors who are teaching in college and K12 schools.

Progress thus far on the Evaluation of Goal 4

To gather baseline information on the underrepresented students, the STEMTEC program managers have assembled data about the presence of such populations within the STEMTEC faculty, STEMTEC K12 teacher, and the STEMTEC students on some schools in the Collaborative. Efforts are underway to extend this data gathering to other institutions within the Collaborative. These data will help STEMTEC determine if their program measures are instrumental in achieving goal 4. Program managers are also seeking scholarship funding and developing recruitment programs to engage more underrepresented students. As underrepresented students become involved, asking these students (through surveys and interviews) what factors helped them join and stay in science education can further effective recruitment and retention efforts.

STEMTEC also needs to establish if science, mathematics, and engineering majors are changing to science education as a result of either STEMTEC courses or facilitated teaching experiences. Therefore, it will be necessary to assemble any historical or comparative information. Absent such information, STEMTEC will need to begin recording such data to substantiate that STEMTEC efforts have enabled progress toward this goal. As students change majors, asking these students (through surveys and interviews) what factors encouraged them to consider teaching can further enlighten efforts in this area.

Underrepresented Students / Recruitment and Retention

Formative Questions

As a result STEMTEC's recruitment and support efforts are underrepresented students entering and remaining in science education?

Data Sources:

Documentation

- What efforts have been made to locate and support underrepresented students with an interest in science? What efforts have been made to encourage such students to consider careers in education? How successful have these efforts been?

- What campus organizations have been most effective in encouraging underrepresented students to consider careers in science education? What aspects of these programs appear to be encouraging these students?

- What changes in the number of underrepresented students is evident in course enrollment and in student teacher participation?

Survey of Underrepresented Students/ Interviews

(to establish factors that enabled their choosing and continuing in science education)

- Did STEMTEC program factors, as evident in the student responses, appear to encourage underrepresented students to choose and remain in science education?

- Did campus organization, again as evident in student responses, appear to influence these students choosing and remaining in science education? Assuming influence is established, what aspects of these organization accounted for the success?

Summative Questions

Review the data gathered during the formative evaluations, asking first if the statistics indicate a positive trend in recruitment and retention among underrepresented students over the time of the project. Assuming a positive trend, analyze the past data asking what program components seemed most instrumental in bringing about the desired change.

Effects Questions

Did the STEMTEC efforts (scholarships and recruitment) appear to encourage underrepresented populations to become teachers?

Data Sources:

Documentation

- Do the enrollment and retention data, when compared to other institutions with similar underrepresented populations, suggest a correlation between STEMTEC program efforts and increase enrollment and retention?

Interviews of Program Managers Responsible for Recruitment and Retention

- How essential was outside funding to the recruitment and retention of underrepresented students?

- Assuming that no funds were available from outside sources, what efforts could be employed to attain the greatest gain in recruitment and retention of underrepresented students?

- What segment of the retained population would not be in the program had these efforts not been made? Why would this segment be absent without these efforts?

Campus Leaders involved in endeavors for underrepresented students

- (the evaluation question for program managers would be phrased to reflect the particular population with whom these people work)

Underrepresented Students / Instruction

The courses taken and learning achievement of these students will be evaluated by the methods devised for Goal 2. (The chart in Appendix B shows the various populations to use the content-and-pedagogy survey.) Information requested at that time will identify these students among the data gathered. If these students are placed in teaching situations, the congruence of these teaching opportunities with the national science and mathematics standards should be evaluated by the methods devised for goal 2 as well.

Science, Mathematics, and Engineering Majors / Recruitment and Retention

Formative Questions

As a result of STEMTEC's efforts at improving science course and in placing science, mathematics, and engineering majors in teaching situations, are these students changing their major to science education?

Data Sources:

Documentation

- What efforts have been made to locate science, mathematics, and engineering students to have them participate in K12 teaching? What efforts have been made to encourage these students to consider careers in education? How successful have these efforts been in having students change majors?

- What efforts are being made to assemble data that will demonstrate a rate change in the number of science, mathematics and engineering majors who change to science education?

Survey of Students Who Changed Majors / Interviews of Selected Students

(to establish factors that enabled the choice of science education)

- What STEMTEC program factors or aspects of K12 teaching appear to have contributed to these students choosing science education?

- Was there a common factor that appeared to encourage these students to change to majors?

Summative Questions

Review the data assembled over the years of the STEMTEC project, asking:

Did the number of students who change from science, mathematics, and engineering majors to science and mathematics education increase during the course of the STEMTEC project? If yes, how much of the increase could likely be attributed to STEMTEC efforts?

What aspects either of students actually teaching or of the STEMTEC efforts appear to have been most successful in having science, mathematics, and engineering students change to science education? Were the efforts expended in arranging teaching situation commensurate with the gain in education majors or with the improvement in science learning (through the process of teaching)?

Effects Questions

Review the data from the student interviews and surveys asking:

How central a role did teaching K12 or college appear to play in a student changing his or her major to science education?

How central a role did witnessing the recommended pedagogies in college courses appear to play in a student changing his or her major to science education?

Science, Mathematics, and Engineering Majors / Instruction:

The courses taken and learning achievements of science, mathematics, and engineering majors who change to science or mathematics education will be evaluated with the methods devised for goal 2. (See Appendix B for the total population evaluated by these methods.) Information requested at that time will identify these students among the data gathered. If these students are placed in teaching situations, the congruence of these teaching opportunities with the national science and mathematics standards should be evaluated with the methods devised for goal 2 as well.


 

Goal 5: Develop a program to support new science and math teachers in their first year in the classroom

Improving the retention of new teachers is STEMTEC's expressed purpose for goal 5. To this purpose, STEMTEC's Strategic Plan offers supports to ease the "trying experiences" of these new teachers. STEMTEC proposes arranging K12-teacher mentors for new teachers, creating science education seminars that focus on the theory and techniques for the recommended pedagogy, providing conveniently-scheduled science courses, and offering supportive services (email and information assembly).

An underlying project assumption dictates the value of these STEMTEC services. In K12 schools, STEMTEC's ultimate concern is to further the mathematics and science skills of K12 pupils. Therefore, the evaluation of progress toward goal 5 must determine more than simply the retention of new teachers. Progress must also be assessed on evaluating:

The efficacy of the mentoring relationships and the support these relationship offer new teachers in incorporating the recommended pedagogies

The utility of the science education seminars in helping new teachers reflect upon the value, role, and execution of the recommended pedagogies - the same evaluation of teacher preparation as proposed for goal 3. (See Appendix C for the total population of faculty examined by this method.)

The congruence of the science course content in the graduate courses with the science and mathematics content needed by all Americans -- the same evaluation of science faculty courses as proposed for goal 2.

An additional factor will necessitate further evaluation activities. The new teachers will not necessarily be STEMTEC graduates. Therefore, it will be necessary to ascertain whether all these new teachers have learned the science and mathematics content needed by all Americans (as evaluated in goal 2) and whether their teaching practices are consistent with the practices presented in national science and mathematics standards (as evaluated in goal 2 as well). (See Appendix B for the total populations to be evaluated by this method.)

Progress Thus Far on Evaluation of Goal 5

As of the first annual report, STEMTEC had made progress towards finding and preparing mentors, matching mentors with new teachers, and facilitating several meetings between the mentors and the new teachers. Whether the hoped-for changes in science education seminar offerings and in graduate science course time and delivery came to fruition is not known at the time of this writing.

STEMTEC needs to evaluate these services for their impact on both the retention of new teachers and the appropriateness of the mathematics and science content and use of the recommended pedagogies evident in these teacher's classrooms. To evaluate retention, historical data must be assembled on the rate of retention of new K12 science teachers within the collaborative, or on the state or national level. To evaluate the appropriate use of content and recommended pedagogies, the new teacher's understanding should be evaluated by the methods devised for student learning in goal 2. To evaluate the mentoring relationships as well as the graduate science-education and science courses, these K12 and higher-education faculty should be evaluated by the methods devised for goal 2 as well.

Formative Questions

Are the STEMTEC efforts at supporting new K12 teachers helping to retain these teachers? Are the appropriate science and mathematics content and recommended pedagogies evident in the classrooms of these new teachers? Are these teachers usage of the recommended pedagogies being strengthened reflectively and in-practice through the graduate science education offerings and the mentoring relationships? Are these recommended pedagogies being modeled in the graduate science courses?

Data Sources:

Documentation

- Does the percent of STEMTEC-supported new teachers still teaching after a year show a increase over statewide, national, or earlier-collaborative statistics?

Survey STEMTEC-Participating New Teachers

- Has the STEMTEC support furthered these teachers' implementation of the recommended pedagogies as evident in their survey responses? (Use the survey of content-and-pedagogy developed for goal 2 at the beginning and at the end of the first year of teaching)

Interview a Selection of STEMTEC-Participating New Teachers

- Do interviews suggest if, how, and what aspects of STEMTEC support contributed to their remaining in teaching?

- Similarly, do interviews suggest what contributed most to successful application of the recommended pedagogies? Which STEMTEC services are most cited positively by the interviewed new teachers?

Survey of K12 Mentors

- Is the understanding of the recommended teaching practices held by the K12 mentors congruent with these practices as they are presented in national science and mathematics standards and science and mathematics teacher education and certification documents? (Use the content-and-pedagogy survey developed for goal 2 to evaluate these teachers' understanding.)

Survey of Science Faculty and Science Education Faculty / Documentation of their Course and Assessment Materials

- Under the present discussion of the goal 5 evaluation, suffice it to say that, the graduate science and science education courses (content of syllabi and assessment, science and mathematics content and pedagogy congruency, observations) will be evaluated using the approach for all STEMTEC courses in goal 2. These evaluation questions will not be repeated at this point. However, it should be noted that when these data are gathered, information should be solicited that allows the findings to be assigned to the appropriate program goal, that is, the graduate offerings for these new science teachers.

Summative Questions

The information assembled during the project should be reviewed, asking:

Has there been a increased in the long-term retention of these new teachers compared to the state and national averages? If there is an increase, explain whether this increase be attributed to the STEMTEC support services?

Have the new K12 teachers increased their use of the recommended pedagogies as their experience in teaching grows? If, over time, these teachers used more the recommended pedagogies, explain whether this increase be attributed to the STEMTEC supports?

Effects Questions

Review the information assembled during the project, asking:

Which of these STEMTEC services appears to have contributed most effectively toward retaining teachers? What were the apparent reasons that these particular services proved most successful?

Which of these STEMTEC services appears to have contributed most effectively toward teachers improving their science and mathematics content?

Which of these STEMTEC services appears to have contributed most effectively toward teachers applying the recommended pedagogies?

Do some STEMTEC services appear to offer greater improvement-value than others? Why are these services more effective?

Data Sources:

Interviews of Graduate Science Faculty

- As in Goal 2, ask the graduate science faculty about their evaluation of the effectiveness of the recommended pedagogies in furthering their goals as teachers? Do these higher education faculty intend to continue using these innovations in their science instruction? Assuming most faculty consider these the recommended pedagogies beneficial, do they consider the "costs" of implementation commensurate with the gain in student learning?

Interviews of Graduate Education Faculty


 

Goal 6: Establish dissemination mechanisms

As reviewed in the Strategic Plan for STEMTEC, STEMTEC information is being disseminated internally (to the faculty who did not participate in STEMTEC, to students who might take STEMTEC courses, to academic administrators) and externally (to other schools, in national conferences and, in the final year, to an international audience). STEMTEC program managers and the participating faculty have initiated a wide-reaching array of dissemination efforts.

The value of disseminating information is to inform the wider education audience about the merits of the STEMTEC program itself as well as the individual course outcomes. Efforts should be made to ensure examination and publication of the all significant areas within the project, including non-course activities, such as forming productive collaboratives, recruiting and retaining underrepresented students, reforming science education (course work and student teaching), engaging reluctant science faculty in reform, and encouraging more students to consider science education.

Progress Thus Far on the Evaluation of Goal 6

The STEMTEC Annual Report: Year 1 enumerates the many dissemination efforts underway.

To evaluate the effectiveness of the dissemination effort, it will be necessary to consider the breadth of the efforts, the nature of the information being disseminated (will it ultimately be addressing all the aspects of the STEMTEC project), and the response from other faculty within the collaborative, faculty beyond the collaborative and other institutions.

Formative Questions

Is information about STEMTEC achievements reaching the intended internal and external audiences? Are efforts being made to establish, summarize, and publicize all efficacious aspects of the STEMTEC process to help other institutions replicate these efforts? Have STEMTEC science faculty reach a consensus about the most useful aspects of the recommended pedagogies and are these universal aspects being disseminated? Are these dissemination efforts reaching to broader science, education, and administration audiences over time?

Data Sources:

Documentation (Responses to publications, workshop, conferences and the like)

- Are these dissemination documents reaching intended audiences?

- Internally, are more faculty within the collaborative becoming interested in learning about the recommended pedagogies? Are more students registering for STEMTEC course?

- Externally, are requests for additional information forthcoming from those who read the publications or attend the seminars? Are the publications being cited in other publications?

- Are other science faculty trying to bring these reforms into their courses? Have these faculty discussed their efforts with STEMTEC participants?

- Are other institutions initiating these reforms? Have these institutions sought advice from STEMTEC participants?

Survey of New STEMTEC Faculty

- What information or evidence encouraged you to become STEMTEC faculty? What aspects of the STEMTEC project appeared most appropriate to your interests?

Interview of STEMTEC program managers involved in the international conference (conducted later in the project timespan)

- How are the principal outcomes of the STEMTEC project being selected, that is, what outcomes will most deserve this wider audience? Are efforts being made to disseminate information about the process of education reform as well as the educational effects within a particular course?

- What channels are being chosen to reach the appropriate international audience?

Summative Questions

Examine the data gathered during the prior years about the dissemination efforts. Have the publication and dissemination efforts effectively determined and promoted the important contributions of the STEMTEC project? Have the STEMTEC contributions to science education been announced to science faculty, to education faculty, and to institutions interested in systemic learning reform?

Effects Questions

The effects questions below should be asked during interviews of the STEMTEC participant (faculty, administrator, or program manager) who would have most likely communicated with the party outside the STEMTEC project.

Have faculty members within the collaborative who were previously uninterested in the STEMTEC reform become involved because of the dissemination efforts?

Have faculty from other colleges and K12 school inquired about the STEMTEC project because of the disseminated information?

Have other institutions inquired about the STEMTEC model based on information distributed through the various dissemination channels? Has this information appear to have engendered an interest in STEMTEC-like reform at other institutions?

Has international interest in education reform been an outcome of the international conference?

What channels of dissemination appear to be the most fruitful? (This question should be asked of the STEMTEC Project Managers.)


 

Goal 7: Conduct strong programs of evaluation and assessment

The revised evaluation plan (as of January 1999) is based upon the principal of aligning evaluation questions and tasks with the stated and intended goals of the STEMTEC project. To this purpose, the National Science Foundation's Outcome Goal 4 (all students must learn the needed mathematics and science content) has been a guiding strategy for this evaluation plan, as discussed in greater detail at the opening of this document. The various aspects of the evaluation (formative, summative, and effects) and the role of these components are also discussed in the opening sections of this document.

Progress Thus Far

Since November 1998, ABC/EOC have worked toward creating an interim evaluation report and a draft evaluation plan. They have also provided several survey instruments to help STEMTEC program managers gather baseline data for the fall semester of 1998. This draft evaluation plan, as stated in the coverletter, is intended to present the evaluation strategy; to raise issues that must be resolved before further progress can be made, including operational definitions of innovative pedagogies, needed science and mathematics content, and educational collaboratives and an agreement on priorities and responsibilities; and to document concerns about baseline data, particularly regarding present education performance. By reaching consensus at the outset on strategic and conceptual issues, a fruitful relationship between the evaluation team and the STEMTEC program managers should be forthcoming.

The questions posed below will be appropriate from January 1999 forward; previous evaluation efforts were beyond the purview of ABC/EOC.

Formative Questions

Is the evaluation information providing direction and guidance to the STEMTEC project managers in terms of data and documents that should be assembled? Are the evaluation tasks consistent with the STEMTEC goals (stated and intended) and with the requirements of the National Science Foundation (NSF) and the National Visiting Committee (NVC)?

Has the evaluation information pointed out areas that needed adjustment? Has this feedback helped the project managers make corrections so that the program has functioned more effectively?

Data Sources:

Documents

- Within the areas of assigned evaluation responsibilities, have the evaluator's documents been timely and useful?

- Have timely and appropriate evaluation information been provided to NSF and the NVC?

Interviews with STEMTEC Program Managers

- Have the evaluation information and findings served the stated and intended goals of the STEMTEC project?

Correspondence from NSF and the NVC

- Has NSF and the NVC been satisfied with the evaluation information?

Summative Questions

Have the final evaluation reports adequately summarized and evaluated the achievements and "lessons learned" during the STEMTEC project? Did the final evaluation report serve to appropriately evaluate the processes of: grant implementation, collaborative development, systematic incorporation of the STEMTEC project goal into the institutional structure of the collaborative, and dissemination of both the process of creating an academic structure to improve education and the process of improving the mathematics and science skills of all students?

Data Sources:

Documents

- Did the final evaluation reports convey clearly and adequately the appropriate information (stated in the previous paragraph) about the comprehensive STEMTEC project?

- Has the information presented in the final evaluation report been presented in such a manner that it can serve other institutions who may be considering projects of this nature?

Correspondence from NSF and the NVC

- Has NSF and the NVC been satisfied with the evaluation information?

Effects Questions

Have the evaluation questions and the evaluation process served to make the STEMTEC program managers more productive and better informed?

Data Sources:

Interviews of STEMTEC Project Managers

- How has the evaluation process informed the process and outcome of the STEMTEC project? If funding had not available for evaluation planning, how would operations had differed? What be the essential benefit to evaluation? What are the problems associated with evaluation?



 

Appendix A: Interim Evaluation Plan

DRAFT: STEMTEC

INTERIM EVALUATION PLAN

(1 December 1998- 28 February 1999)

Develop Evaluation Plan for the duration of the STEMTEC Project

Gather Baseline Data for Goals 2 and 3

Gather Baseline Data for Goal 2

(With Feldman and Thrasher)

1.Collect end-of-semester (EOS) data on STEMTEC faculty use of innovative pedagogical

practices in the Fall 1998 Academic Term.

2. Collect syllabi, course information, assignments, and assessments along with scoring guides used to collect information for grading.

3. Design a survey to measure STEMTEC faculty members' understanding and use of innovative pedagogical practices. Survey will be administered 6 February 1999.

4. Collect observations from Round Table discussions that reveal Faculty understanding of innovative pedagogical and assessment strategies.


Draft of Letter to Faculty for EOS Information

Dear Professor < >

We, Audrey Champagne and Eileen O'Connor, have agreed to be the external evaluators for the STEMTEC Project. Over the course of the next three years we will be asking you regularly for information that we can use for reporting the progress of the STEMTEC project to the National Science Foundation. Continuation of funding for the project will be contingent on the information we are able to provide the National Science Foundation about the project's progress.

This letter contains a request for information about the STEMTEC, < put in the name of the course >, you taught this term. We request that you send to the Project Manager copies of the following:

The course syllabi and any other documents that you use to explain the course requirements and processes

All test and any scoring guides for those tests

Copies of other methods you used to gather information to be used for grading students in the course

Included with this letter is a short survey about your teaching practices this term. Please send your response to the survey along with the information requested to the Project Manager.

An End of Semester Survey of Teaching Practices

Please take a few minutes to respond to the following questions about your teaching practices this semester in <Insert STEMTEC Course Name and Number> and then send the survey to the Project Manager.

Circle the response that most accurately describes your teaching and assessment practices in <Insert STEMTEC Course and Number>.

1. About how often this semester did you have your students work in pairs or small groups?

Never Once or Twice/Semester About Eight Times/Semester Weekly

2. About how often this semester did you speak to individual students or the class about teaching as a career?

Never Once or Twice/Semester About Eight Times/Semester Weekly

3. About how often this semester did you assign a project requiring two or more weeks in preparation?wt

Never Once or Twice/Semester About Eight Times/Semester Weekly

4. About how often this semester did you have your students engage in peer tutoring?

Never Once or Twice/Semester About Eight Times/Semester Weekly

5. About how often this semester did students teach a portion of your course?

Never Once or Twice/Semester About Eight Times/Semester Weekly

6. About how often this semester did you use a form of assessment for grading other than a conventional paper and pencil test?

Never Once or Twice/Semester About Eight Times/Semester Weekly

7. About what percent of the conventional paper and pencil tests you used in this course were extended response items?

I did not give any conventional tests 0% 25% 50% 75% 100%

8. Which of the following forms of alternative assessment did you use this semester?

I did not use any form of alternative assessment

Reports of Library-type research: Research requiring no empirical work

Reports of Laboratory Investigations

Oral Presentations

Other alternative forms of assessment (Please describe briefly)

__________________________________________________________

__________________________________________________________

Thank you for completing this survey. If you have any other comments about your experiences with non-conventional forms of assessment or innovative pedagogy, please write them below.

Return the survey to the Project M


Goal 3

(With Thrasher and Feldman)

Collect Baseline data from:

A. Practicing (in service) teachers involved with the Five College/Public School Partnership STEMTEC Activities

B. Elementary and secondary students in UMass methods courses and other pre-service teachers that can be identified at the other institutions

Collect Baseline data on in-service teachers'

Understanding of:

Science and mathematics

Concepts, principles, theories

Practice of science and mathematics

Nature of science and mathematics inquiry

Innovative pedagogical practices for science and mathematics

Classroom practices to engage students from populations under represented in science and mathematics

Classroom practices effect in engaging disabled students in science and mathematics and

Use of innovative practices in their classrooms

Collect baseline data on pre-service teachers'

Understanding of:

Science and mathematics

Concepts, principles, theories

Practice of science and mathematics

Nature of science and mathematics inquiry

Innovative pedagogical practices for science and mathematics

Classroom practices to engage students from populations under represented in science and mathematics

Classroom practices effect in engaging disabled students in science and mathematics and

The opportunities they have had to practice the innovations in their student teaching experiences.


Proposed Strategies to get Baseline Data

Inservice Teachers

Understanding of Innovative Pedagogical Strategies

Set the context for the activity, which is to design professional development activities introducing their colleagues to innovative pedagogical and assessment strategies.

After setting the context, ask individuals to respond in writing to each of the questions that follow.

Then for each of the questions, once participants have responded in writing, ask them to work in groups of 3 or so to develop a consensus response to the question. (Individuals may write dissenting responses if consensus can not be reached.)

Finally, working as a full group, develop a consensus response.

Context:

As a teacher leader you have the responsibility for planning and presenting an workshop for your colleagues-- the science or mathematics teachers in your school about*:

student interactive learning

alternative assessment

constructivist teaching

project based learning

authentic assessment

use of educational technology

new goals for teaching and learning

action research

engaging students in teaching experiences

students from under represented populations

students with disabilities

*The list of pedagogical strategies STEMTEC advocates is long B some selection must be made for the STEMTEC evaluation process. For the base line data to be collected between December and February we recommending focusing on the first three.

Questions:

1. In your introduction to the workshop on X, how would you describe/define X to your colleagues.

2. Based on what you know about how learning occurs, how would you explain to your colleagues why it is that X should improve student learning? (or what theories of human learning explain why it is that X should improve student learningB or what is the mechanism that makes X an effective in producing learning?)

3. Which of your experiences with X would you share with the individuals in your workshop?

4. What would you tell your colleagues about how often you use strategy X in your classes?

5. What would you tell your colleagues about which goals of science education are best achieved using strategy X?

Understanding of science concepts, principles, and theories

What is your interpretation of the word, understanding, as it applies to science concepts, principles and theories?

Write a response to each of the following problems that you believe that should characterize responses of science and math literate adults.

Life Science

The Plant-in-a-Jar Task

Some moist soil is placed inside a clear glass jar. A healthy green plant is planted in the soil. The cover is screwed on tightly. The jar is located in a window where it receives sunlight. Its temperature is maintained between 60o and 80oF. How long do you predict the plant will live? Write an explanation supporting your prediction.

Use ideas from the life, physical, and earth sciences to make a prediction. If you are unsure of a prediction, your explanation should state that, and tell what information you would need to make a better prediction. There is not a single right prediction.

Earth Science

Phases of the Moon

Draw a diagram and use it to explain the phases of the moon.

Physical Science

Mechanics

The diagram below shows a stick standing upright on the floor. The stick is marked with colored strips. Imagine that you drop a ball from the red mark.

How does the speed of the ball at the orange mark compare with its speed at the red mark. Is it about the same, a whole lot more, or a little bit more? Explain your answer.

How does the speed of the ball at yellow mark compare with its speed at the red mark? Is it about the same, a whole lot more, or a little bit more? Explain your answer.

The diagram below shows a moveable pulley with a string over it. At one end of the string is block, and at the other a ball. How does the weight of the block compare with the weight of the ball. Do the block and the ball have the same weigh or does the block weigh more or less than the ball? Explain your answer.

Technology

Paint Brush Problem

A paintbrush has just been used and the owner wishes to clean it. After the brush has been scraped against the side of the paint can, it still contains 4 fluid ounces of paint. The owner dips it into a quart of clean solvent and stirs well until the diluted paint solution is uniform. After draining, the brush still holds 4 fluid ounces, part of which is paint and part solvent, since the diluted solution is uniform. The process is repeated with a fresh quart of solvent.

Propose a mathematical model of this process.

Understanding the nature of science

What is science inquiry?

What is your understanding of what is meant by the phrase, nature of science, as it used in the National Science Education Standards document?


 

The Proposed Strategies to get Baseline Data

Pre-service Teachers in U Mass methods courses.

[If at all possible these data should be collected at the end of this term, Fall 1998]

1. Describe how would you explain each of the following innovative pedagogical practices to a friend who has never taken an education course but has just learned that she has a position teaching science.

student interactive learning

alternative assessment

constructivist teaching

project based learning

authentic assessment

use of educational technology

new goals for teaching and learning

action research

engaging students in teaching experiences

students from under represented populations

students with disabilities

*The list of pedagogical strategies that follows is long B some selection must be made for the STEMTEC evaluation process. For the base line data to be collected between December and February focus on the first three.

2. Based on what you know about how learning occurs, how would you explain to your friend why it is that X should improve student learning? (or what theories of human learning explain why it is that X should improve student learning B or what is the mechanism that makes X an effective in producing learning?)

3. Which of your experiences using X with students would you share with your friend?

4. What would you tell your friend about which goals of science education (for instance understanding of concepts and principles, developing the ability to inquire) are best achieved using strategy X?

Understanding of science concepts, principles, and theories

What is your understanding of the word, understanding, as it applies to science concepts, principles and theories?

Write a response to each of the following problems that you believe that should characterize responses of science literate adults.

Life Science

The Plant-in-a-Jar Task

Some moist soil is placed inside a clear glass jar. A healthy green plant is planted in the soil. The cover is screwed on tightly. The jar is located in a window where it receives sunlight. Its temperature is maintained between 60o and 80oF. How long do you predict the plant will live ? Write an explanation supporting your prediction.

Use ideas from the life, physical, and earth sciences to make a prediction. If you are unsure of a prediction, your explanation should state that, and tell what information you would need to make a better prediction. There is not a single right prediction.

 

Earth Science

Phases of the Moon

Draw a diagram and use it to explain the phases of the moon.

Physical Science

Mechanics

The diagram below shows a stick standing upright on the floor. The stick is marked with colored strips. Imagine that you drop a ball from the red mark.

How does the speed of the ball at the orange mark compare with its speed at the red mark. Is it about the same, a whole lot more, or a little bit more? Explain your answer.

How does the speed of the ball at yellow mark compare with its speed at the red mark? Is it about the same, a whole lot more, or a little bit more? Explain your answer

The diagram below shows a moveable pulley with a string over it. At one end of the string is block, and at the other a ball. How does the weight of the block compare with the weight of the ball. Do the block and the ball have the same weigh or does the block weigh more or less than the ball? Explain your answer.



Technology

Paint Brush Problem

A paintbrush has just been used and the owner wishes to clean it. After the brush has been scraped against the side of the paint can, it still contains 4 fluid ounces of paint. The owner dips it into a quart of clean solvent and stirs well until the diluted paint solution is uniform. After draining, the brush still holds 4 fluid ounces, part of which is paint and part solvent, since the diluted solution is uniform. The process is repeated with a fresh quart of solvent.

Propose a mathematical model of this process.

Understanding the nature of science

What is science inquiry?

What is your understanding of what is meant by the phrase nature of science as it used in the National Science Education Standards document?



 

Appendix B. Populations taking the content-and-pedagogy survey

Figure 2. Populations taking the content-and-pedagogy survey

The populations shown above are the primary individuals who receive STEMTEC services, the K12 teachers who support the faculty endeavors, or the support teachers who facilitate the field work of student teachers and new teachers. It is important to determine these individual's understanding of content and pedagogy to evaluate the STEMTEC endeavor.

Populations may overlap; gather information to store by program involved in, STEMTEC courses taken, teaching situation, and such ; present survey online; gather responses into a database



 

Appendix C. Faculty populations to be evaluated

Figure 3. Populations Evaluated Using the Faculty-Teaching Analysis

The populations cited above are the primary agents in providing the appropriate science and mathematics content and in modeling or instruction about the recommended pedagogies. Therefore, the content of the courses and their understanding of content and pedagogy are critical to the success of the STEMTEC endeavor.

Since the faculty-teaching analysis is more comprehensive (and time consuming) than the survey of content-and-pedagogy, it should be used only when the individual has a significant teaching responsibility. If the teacher serves more of a supportive role, this teacher's understanding of content and pedagogy could be appropriately evaluated with the content-and-pedagogy survey.

Populations may overlap; gather information to store by course taught, course number, discipline, STEMTEC workshops / seminars attended, grants received, and such.

Present the survey component online; gather responses into a database.

1. If the evaluation is to address the achievement of engineering and technology understanding, appropriate yardsticks for these measures must be identified as well.

2. Consensus must be reached on what constitutes indicators of collaboration. For instance is representation at all meetings of each of the collaborating institutions an indicator? What are others? NVC would be helpful in generating indicators of successful collaborations of the type attempted here.

3. Many of the "innovative" practices, as they are referred to within STEMTEC documentation, have been used and tested within education enterprises. Although, for purposes of this evaluation, consensus must still be reached regarding operational definitions of the essential elements of such innovative practices, the term recommended pedagogies or practices will be used extensively throughout this evaluation report. Thus language consistency with national standards will be maintained.

4. NAEP and TIMSS released items or New Standards Project Items might also serve as operational definition of the "needed" content. The issue of the yard stick to be used in the STEMTEC evaluation is a matter on which the NVC might be willing to provide advice.

5. The NVC has noted that the descriptions of recommended practices are not consistent in STEMTEC documents. The evaluators have noted that inconsistency is a serious issue in the instrumentation used for observations of STEMTEC classes. The evaluators also note that the definitions provided in national standards are very broad. Consequently we recommend that the STEMTEC Management Team, the Evaluation Team and the NVC collaborate in the development of observable indicators of these practices that should be used in the STEMTEC Evaluation.