What a Difference Data-In-A-Day Makes

by Verilette A. Hinkle, Kerry T. Hinkle, and David M. Monetti
Valdosta State University

Abstract

With the recent emphasis placed on school accountability, educators around the country have an enormous opportunity to gather and analyze data to understand current school practices and to guide school improvement. One school improvement strategy entitled Data-In-A-Day (DIAD), developed by the Northwest Regional Educational Laboratory in Portland, Oregon, was selected and implemented as part of a school improvement initiative by the faculty of a moderately-sized middle school in rural South Georgia. DIAD is an action research approach, which utilizes teams of educational stakeholders (business members, community members, parents, students, and faculty) to gather pertinent data about instructional practices. Specifically, DIAD provides a structure for assessing the degree to which varied instructional strategies are being implemented within the classroom. Data from DIAD reflect both the process and progress of planning and implementing interventions aimed at improving student achievement. The DIAD process provided a “snapshot” of classroom instruction with the goal of “improving” what teachers do, and minimizing the concern over “proving” what they do. The focus of this paper is to describe the design, implementation and impact of DIAD and the influence it had on school improvement in one rural middle-school setting.

What a Difference Data-In-A-Day Makes

With the recent emphasis placed on school accountability, educators around the country have an enormous opportunity to gather and analyze data to understand current school practices and to guide school improvement. In order to provide support for teachers as they engage in the school improvement process, several effective school improvement models have been developed. One model, Onward to Excellence II (OTE II), developed by the Northwest Regional Educational Laboratory in Portland, Oregon, was selected and implemented by the faculty of Lowndes Middle School (LMS), a rural and racially diverse school in South Georgia.

The Data-In-A-Day (DIAD) methodology is a component of the OTE II process that was conducted at LMS for the purpose of gathering data to improve student achievement. DIAD was considered an effective data collection activity by LMS because it was simple, practical, and had formative value without the use of complex statistical tools. In addition, DIAD engaged community members in the school’s efforts to improve student achievement. The development, implementation, and evaluation of DIAD at LMS exemplify how many schools are using data to make instructional changes and to assess the effectiveness of school programs.

What is DIAD?

DIAD is an action research approach, which utilizes teams of educational stakeholders (business members, community members, parents, students, and faculty from post-secondary institutions) to gather pertinent data about instructional practices with the intent of improving student achievement in a particular school. Action Research is a research paradigm intended to inform and improve educational practice (Calhoun, 2002; Mills, 2003), and as such it is a viable approach for practitioners to gain information about their particular schools for school improvement purposes. As noted by Gross (2002), action research methods are used primarily by practitioners to solve, analyze, and enhance current practices. At LMS, the DIAD process was guided by the school-wide goal of utilizing instructional strategies that increased students’ reading comprehension. Towards this end, the School Improvement Plan addressed specific instructional strategies that would be implemented across all content areas and classrooms. DIAD provided a structure for assessing the degree to which varied instructional strategies were being implemented in the LMS classrooms.

During the DIAD process, community members observed classroom instruction to record the use of instructional strategies as outlined in the LMS Improvement Plan. These results were then shared with the DIAD participants and the classroom teachers. The entire DIAD process was completed within a single day. The LMS DIAD activity provided a “snapshot” of classroom instruction with the goal of “improving” what teachers do, and minimizing the impression of “proving” what they do.

DIAD results reflected the process and progress of planning and implementing effective teaching strategies aimed at improving student achievement. Information gained by this process was used to direct and retain efforts toward school improvement through the planning of future professional development workshops.

Development of DIAD

As a data-based decision making process, DIAD requires “buy in” from the entire school community. Teachers are an integral part of the school community; their involvement in the DIAD process is essential. According to the OTE II School Improvement Model, teachers begin with the “buy in” to data-based decision making during the planning phase of the school improvement process as they and other community members work collectively towards school change. Over time, many teachers not only witness student improvement as a result of their involvement in school change, but they also develop a belief in their power to enhance student learning. A sense of collective efficacy is developed.

According to Goddard (2001), collective efficacy in education reflects the perception of teachers that they can take actions which produce positive effects on students. For example, research by Goddard, Hoy, and Woolfolk-Hoy (2000) found that math and reading achievement were positively related to collective efficacy. Through involvement in the development of instructional goals, the investigation and incorporation of scientifically valid interventions, and the sharing of data useful in monitoring and measuring faculty and student progress, the collective efficacy of the entire school community is heightened.

LMS teachers and community members were involved in numerous school improvement activities prior to the DIAD project. Activities included the development of a school profile, selection of school-wide goals, alignment of curricula with state standards and assessments, review of the research literature on effective educational practices, and the development of a school improvement plan. Each of these activities required that members of the school community work together for improved student achievement. By the time DIAD was developed and implemented, teachers and community members valued working cooperatively and had accepted the importance of data-based decision making in school improvement.

School improvement is an ongoing change process in which formative data are critical for decision-making purposes. Quantitative test scores of student achievement, while useful as summative outcome data, are not always helpful for evaluating the process of teaching. Similarly, qualitative data gained through discussion and interview often lack the precision needed to measure the impact of specific teaching strategies on student achievement. In addition to the selection of an appropriate research methodology, program evaluators should assess needs through a process of instrument development that allows the participants to determine the specific data to be collected rather than having instrumentation prescribe the data (Lusky & Hayes, 2001).

The OTE II School Improvement Model incorporates both formative and summative data to determine the effectiveness of change within a school. DIAD is a formative approach for gathering data about the particular interests or needs of a school. During DIAD, educational stakeholders observe classroom instruction with a focus on a school’s selected interest or need. DIAD participants record their observations on an observation checklist. Data are collected and analyzed. Then, analyzed data are used to direct future plans for continued school improvement.

Data collection instrument

The LMS teacher-facilitator for school improvement partnered with professors from a local university to develop the data collection instrument. This instrument serves as an observation checklist reflecting the selected interests and needs of a school’s improvement plan. The LMS Improvement Plan included the use of five specific instructional strategies to bolster reading achievement. The strategies were: experiential learning, cooperative learning, active listening, critical thinking, and computer-assisted instruction. These strategies were the focus of the observation checklist.

The Observation Checklist consisted of two sections. The first section contained demographic questions pertaining to such things as grade level and subject area observed. The second section focused on the five instructional strategies of the LMS Improvement Plan and included operational definitions, observable characteristics, and five-point Likert-scaled questions ranging from strongly disagree to strongly agree. For example, under the section on experiential learning, three characteristics were listed: (a) manipulation of materials, (b) peer interactions, and (c) emphasis on discussion as opposed to lecture. The DIAD participants were asked to record whether or not they observed the designated characteristics in the classrooms. Then, the observers were asked to make an overall reflection of the use of experiential learning in the classroom. For example, “Overall, the classroom environment seemed to reflect the definition and characteristics of experiential learning.”

                                           ------------------------------

                                           See Figure 1 below article

                                           -------------------------------

For simplicity and efficiency in reporting the data, the participants recorded their observations onto Scantron forms. For analysis, results were converted to percentages and shared with the community participants prior to their departure from the school. At the end of the school day, the DIAD results were shared with the teachers.

Selection of participants

The OTE II Model recognizes the importance of community involvement in school improvement. DIAD was developed on the premise that the school community (teachers, students, administrators, business leaders, and community members) needs to work together for improved student learning.

            For DIAD at LMS, 60 members of the school community were asked to participate. DIAD participants were selected by the LMS External Study Team, a group of LMS community members, formed at the beginning of the school improvement process to assist with data collection and evaluation.

            The 60 DIAD participants reflected a diverse group of students, parents, and other community representatives. Community representatives included business leaders and community educators. The business leaders consisted primarily of owners and managers of local businesses. Community educators included professors and students from a nearby university and a technical college.

            Of the 60 DIAD participants, a select number of each participant group was invited: 10 parents, 20 business leaders, 20 community educators, and 10 students. An effort to select a diverse population by gender and race for each participant group was made. In addition, the student group included representation from the sixth-, seventh- and eighth-grade classrooms.

-------------------------------

See Table 1 below article.

-------------------------------

Selection of classrooms for observations

Twenty classrooms were selected for observational purposes for DIAD at LMS. A random selection process was used. Classrooms for visitor observation during the DIAD activity included: reading, mathematics, social studies, art, computer science, physical education, health, science, and English. Teachers were not informed of the selected classrooms prior to DIAD.      

Implementation of DIAD

All of the invited participants (n=60) attended DIAD. Participants began the DIAD activity by attending a brief presentation conducted by the LMS teacher-facilitator and professors from a local university. The presentation focused on the importance of classroom observations for school improvement purposes and on the procedures for conducting observations in classrooms. An observation checklist and Scantron sheet were distributed to each participant. Educational terms on the checklist were defined, and examples of the instructional practices, as listed on the observation sheet, were provided. Participants were shown how to record observations onto the Scantron sheets. At the conclusion of the training session, participants were informed of the need to maintain the confidentiality of their observations of students and teachers within the school.

DIAD participants were placed into groups of three. Each observation group included: a parent or a student, a community member, and a business member. A LMS student was assigned to each group to guide them from classroom to classroom. Two 20-minute classroom observations were scheduled for each group. Participants observed classroom instruction for 20-30 minute periods. Participants completed an observation checklist for each classroom visited. Then, the groups returned to a designated site and the observation checklists were collected.

Evaluation of DIAD

DIAD involves the school community in data collection and analysis for school improvement purposes. Parents, teachers, students, business leaders, and community members work jointly for improving student achievement. DIAD results are utilized for planning educational interventions.

At the beginning of the OTE II process, the LMS teachers recognized the need to vary instruction in their classrooms after analyzing the LMS school profile and research literature on effective instructional practices. Although the faculty acquired some new knowledge about effective instructional practices through research literature, they expressed a need for professional development workshops to gain more knowledge and skills, so that they could implement those practices in their classrooms. The instructional practices included: experiential learning, cooperative learning, active listening, critical thinking, and computer-assisted instruction.

The purpose of LMS DIAD was to collect baseline data to determine the degree to which the five aforementioned instructional practices were currently being implemented in the classrooms and to use that baseline data to guide professional development for teachers. The percentages of DIAD participant observations of the aforementioned instructional practices in the classrooms became the baseline data. Those results were used to select the order and depth of topics for teacher professional development workshops throughout the year.

At the conclusion of the DIAD activity, results for the five “overall” statements on the observation checklist, pertaining to experiential learning, cooperative learning, active listening, critical learning, and computer-assisted learning were calculated. Because Strongly Agree and Agree indicated the presence of instructional practices in the classrooms, those categories were collapsed for a single percentage score. The categories of Strongly Disagree and Disagree were also collapsed.

The most frequently cited instructional practice on the observation sheet was identified as experiential learning. For experiential learning, DIAD participants agreed that 75% of the classroom observations included that particular instructional strategy; whereas, 16% of the classroom observations did not include that particular instructional practice. Nine percent of the observations were identified as neutral. The remaining instructional practices, as observed in the order of highest percentages of being observed in classrooms, were: active listening, critical thinking, cooperative learning and computer-assisted instruction.

                                                ------------------------------

                                                SeeTable 2 below article

                                                -------------------------------

As evidenced from the DIAD observations, the following percentages of classrooms in which the teacher-selected instructional strategies were observed, included: experiential learning 75%, cooperative learning 56%, active listening 66%, critical thinking 61%, and computer-assisted instruction 45%. The instructional strategy that was least observed in the classroom, computer-assisted instruction, became the topic for the first professional development workshop; whereas, experiential learning became the last topic to be addressed for teacher professional development. As a result of the data from DIAD, the school system for LMS became more financially supportive of the school’s needs by providing additional funding for the purchase of resources related to computer-assisted instruction.

The DIAD results were quite surprising for the school community, especially to the teachers. Despite the fact that the teachers had not attended professional development workshops on the five instructional practices prior to the DIAD process, the DIAD results provided evidence that the teacher-selected instructional practices were already being used to some degree in many of the classrooms. Thus, the teachers were a step ahead of the OTE II school improvement process…they already had assumed responsibility for doing what was necessary to enhance student learning prior to the professional development workshops. In spite of this fact, teachers still requested additional professional development opportunities to increase the variety and quality of their classroom instruction.

The DIAD activity is an action research methodology for gathering formative data about teaching and learning within a school. Participants of DIAD are involved in observations NOT evaluations; and observational data are used to “improve” teacher/student performances and not to “prove” that teachers/students are performing well in the classroom. Data guide the development of school programs so that specific needs or interests of a school are addressed. DIAD provides an opportunity for a school community to work together for improved student achievement. As evidenced in the description of the LMS improvement process, DIAD is an opportunity for a school community to enhance student achievement and to celebrate the school improvement efforts and academic successes together.

References

Calhoun, E. F. (2002). Action research for school improvement. Educational Leadership,   59(6), 18-24.

Goddard, R. D. (2001). Collective efficacy: A neglected construct in the study of schools    and      student achievement. Journal of Educational Psychology, 93(3), 467-476.

Goddard, R. D., Hoy, W. K., & Woolfolk-Hoy, A. (2000). Collective efficacy: Its   meaning,          measure, and impact on student achievement. American Education Research Journal, 37,             479-507.

Gross, R. R. (2002). Research-driven school improvement. Principle Leadership, 2, 35-40.

Lusky, N, B., & Hayes, R. L. (2001). Collaborative consultation and program evaluation.

            Journal of Counseling and Development, 79, 26-38.

Mills, G. E. (2003). Action research: A guide for the teacher researcher (2^nd ed.). Upper Saddle    River, NJ: Merrill.

Table 1

                                                     Gender                         Race                      Grade Level

Parents                                    10          40          60                  80          20                -        -        -

Business Leaders        20          25          75                  80          20                -         -        -

Community Leader      20          50          50                  85          15                -         -        -

LMS Students             10          50          50                  50           50               30      30     40

Note: Dashes indicate that participants were not represented by the designated group.

Table 2

Percentages of Classroom Observations Having Teacher-Selected Instructional Practices

(n=120)

_______________________________________________________________________

Instructional Practice                                       Disagree          Neutral                        Agree
______________________________________________________________________

Experiential Learning                                      16%                     9%                  75%

 Cooperative Learning                                      24%                     20%                 56%

 Active Listening                                              16%                     18%                 66%

 Critical Thinking                                             20%                     19%                 61%                                                    

Computer-Assisted Learning                          45%                     10%                 45%

Part II: “Data-In-A-Day” Observation Form 

Experiential Learning: Instructional activities that actively engage students in the learning process.

4. Overall, the classroom environment seemed to reflect the definition and examples of experiential learning.

                        A                      B                      C                      D                      E

            Strongly Disagree         Disagree               Neutral                  Agree     Strongly Agree

Cooperative Learning: Working in small groups to accomplish shared learning goals.

8. Overall, the classroom environment seemed to reflect the definition and examples of cooperative learning.

                        A                      B                      C                      D                      E

            Strongly Disagree         Disagree               Neutral                  Agree     Strongly Agree

Active Listening: A way of listening and responding that improves student understanding.

12. Overall, the classroom environment seemed to reflect the definition and examples of active listening.

                        A                      B                      C                      D                      E

            Strongly Disagree         Disagree               Neutral                  Agree     Strongly Agree

Critical Thinking: Systematic thought that asks questions and evaluates evidence in order to make decisions.

16. Overall, the classroom environment seemed to reflect the definition and examples of critical thinking.

                        A                      B                      C                      D                      E

            Strongly Disagree         Disagree               Neutral                  Agree     Strongly Agree

Computer-Assisted Learning: The use of computers to enhance achievement.

16. Overall, the classroom environment seemed to reflect the definition and examples of computer-assisted learning.

                        A                      B                      C                      D                      E

            Strongly Disagree         Disagree               Neutral                  Agree     Strongly Agree

Figure 1. Part II: “Data-In-A-Day” Observation Form

The curriculum and program exemplars showcased here have been contributed by educators in the field. Many were originally featured in Community Works Journal, or in Connecting Service-Learning to the Curriculum. We thank our contributing educators and their students for making their work available to us. Please contact us if you would like to share and 'exemplar" or reflection of your own. CWI EXEMPLARS: Exemplars Main l K-8 Exemplars l 9-12 Exemplars l Higher Ed Exemplars l Community Based Exemplars l Community Works Journal