Why it Matters
Building Evidence Through Evaluations is one of six strategies that LEA leaders can use to improve outcomes for all through procurement processes. Ongoing evaluation helps LEAs understand how well an education program is working, for whom and under what circumstances. Evaluation will help an LEA answer the question: “Are our investments making the impact we seek?”
Also referred to as research agendas, learning agendas are a set of prioritized research questions that guide evidence-building activities and decisions.
Learning agendas are valuable because they help LEA leaders make decisions about when and how to use limited resources to engage in data collection, analysis and evaluation activities.
Through learning activities designed to answer the questions on a learning agenda (e.g., literature reviews, simple data analyses, evaluation), LEA leaders will be better equipped to engage in continuous improvement throughout their programs’ life cycles and, ultimately, speak to whether their programs are effectively meeting their strategic goals.
Sample Learning Agenda for a High-Impact Tutoring Program for Reading
Learning Questions
Questions about Program Implementation or Impact |
Learning Activities
Activities to Address Your Questions |
---|---|
What are the design characteristics of highly effective tutoring programs? |
Literature review to learn what research says about the characteristics of highly effective tutoring programs. |
Are the students who are in greatest need of tutoring the students who are participating in the tutoring program? |
Data analysis to determine what percentage of students who meet the district’s thresholds for tutoring are participating in the program, disaggregated by student characteristics. |
Are students, teachers and tutors experiencing the tutoring program in a positive way? |
Student, teacher and tutor surveys asking questions about perceptions of and experiences with the tutoring program. |
Are the students participating in the tutoring program receiving sufficient, consistent tutoring support through the program, as defined by the characteristics of an evidence-based high-impact tutoring program? |
Descriptive data analysis to determine if students’ experience with tutoring (e.g. hours served per week, number of tutors with which the student engages, number of weeks the student is in the program) aligns with the characteristics of an evidence-based tutoring program. |
How is participation in the tutoring program affecting participants’ performance in reading? |
Impact evaluation to determine if students participating in the tutoring program have significant improvement in their reading performance as a result of the tutoring program. |
Learning agendas often include additional information that goes beyond the learning questions and activities themselves.
Depending on (1) how the LEA intends to use the learning agenda; and (2) the primary audiences for the learning agenda, an LEA may want to include all or some of the following pieces of information:
Alignment with the LEA’s strategic plan or strategic priorities.
Types and timing of activities intended to address the high priority questions.
How the LEA worked with stakeholders to develop the learning agenda and how the LEA will continue to work with stakeholders to address the high priority questions and disseminate findings.
How the LEA will use the results of studies carried out in service of the learning agenda; and
Contact information for researchers and other stakeholders interested in learning more about the learning agenda.
How LEAs organize the information on their learning agenda will vary. In general, an LEA should consider the following guidelines:
- Consider your goals in creating and sharing the learning agenda.
Are you seeking support from external researchers or does the LEA already have research activities planned (and the capacity to carry them out)? The format of the learning agenda can signal to researchers how they might interact with the LEA in addressing questions.
To what degree is the intended audience familiar with the structure of the LEA or the LEA’s strategic priorities? How will the format of the learning agenda help them find what they’re looking for quickly?
If the questions included in the learning agenda vary in terms of priority, the LEA can signal this through the design of the learning agenda.
Limit jargon to the degree possible, especially in versions of the learning agenda intended for the public;
Plan ahead for how often you will want to - or realistically be able to - update the learning agenda. Your anticipated update cycle may inform what kind of information you include in your learning agenda. You may also want to note the anticipated update cycle in the learning agenda itself.
Local Education Agency Learning Agendas
LAUSD has a series of research questions, by district office, that serve as their collective district research agenda.
Chicago Public Schools has arguably the strongest research agenda in the country
Orange County (FL), the ninth largest school district in the country, updates their one-page research agenda annually.
State Education Agency Learning Agendas
- The California Department of Education articulates its research priorities in several key areas to help external researchers understand the types of studies the Department will consider supporting.
- The Massachusetts Department of Elementary and Secondary Education (DESE) publishes their research agenda here (see link towards bottom of the page). DESE organizes their research agenda by office and topic and is meant to capture program offices’ high priority questions. In addition to the questions themselves, the research agenda has an introduction that includes:
- Guidelines for external researchers;
- Contact information for the Chief Strategy and Research Officer.
The Michigan Department of Education publishes its research agenda in partnership with the Michigan Education Research Institute and invites researchers to propose projects meant to further the strategic priorities outlined on the agenda.
The North Carolina Department of Public Instruction’s Office of Learning Recovery publishes its Research and Evaluation Plan and Logic Model.
The Pennsylvania’s Department of Education publishes a research agenda
Washington State’s learning agenda (expressed as research priorities)
Wisconsin’s annual learning agenda which was developed around the theme of using data to understand and advance equity for students.
Other State Agencies
RFA's State Government Standard of Excellence reviewed and compiled the strongest examples of agencies with evaluation policies, plans, and research/learning agendas. Among them:
California’s Air Resources Board produces three-year strategic research plans
California’s Workforce Development Board state plan includes priority research questions, evaluation activities, major findings, and how these findings are tied to major questions.
Kentucky's Center for Education and Workforce Statistics' research agenda
Virginia developed agency specific learning agendas for a number of their agencies that are worth looking at.
Federal Learning Agendas
The Foundations for Evidence-Based Policymaking Act of 2018 ("Evidence Act") emphasizes collaboration and coordination to advance data and evidence-building functions in the Federal Government by statutorily mandating Federal evidence-building activities, open government data, and confidential information protection and statistical efficiency.
Title I of the Evidence Act requires the 24 CFO-Act agencies to:
designate Evaluation Officers,
develop evidence-building plans (Learning Agendas), Annual Evaluation Plans, and agency evaluation policies, and
conduct capacity assessments to further evidence-building activities.
All federal agency Research Agendas are available here.
The federal agency research agendas tend to be extensive and complex. There is no single format used by all federal agencies in designing their Research Agendas, although it seems they were working off of similar guidance on what kinds of information to include in their agendas.
The U.S. Department of Education’s Research Agenda is available here and includes the following information:
Focus areas;
Background context;
Priority questions associated with each focus area;
Alignment between the priority questions and the agency’s strategic plan;
Short-term activities meant to address the priority questions; and
Long-term activities meant to address the priority questions;
Anticipated challenges and proposed solutions.
Additional information included in some of the other federal agency research agendas include:
Timing of activities;
Potential data, tools, methods and analytic approaches meant to address the priority questions; and
How results will be used.
Using Statewide Learning Agendas to Foster Equity and Success in Postsecondary Education (MRDC)
- Evidence Toolkit: Learning Agendas (Urban Institute)
Making Federal Agencies Evidence-Based: Learning Agendas (IBM Center for the Business of Government)
USAID has resources on establishing a learning agenda that includes a Learning Agenda template that includes space for the following information:
- Context
Institutional Relevance
Developmental Hypotheses/Theories of Change
- Learning Questions & Learning Activities
Learning Questions
Timing/Key Decision Points
Learning Activities
Resources
- W.T. Grant Foundation’s Research Practice Partnership Resources
The W.T. Grant Foundation developed a number of resources on building strong research practice partnerships. These resources include guidance on building joint research agendas and, in particular, includes an example of the Consortium on Chicago School Research at the University of Chicago’s Research and Public Informing Agenda. While the format of this agenda is primarily narrative and, as a result, possibly lacks some of the clarity of the other research agenda examples shared here, the resource is notable because of the intentionality behind pairing a research agenda and a plan to keep the public informed of the working happening as a result of the research and the results of the research.