How do I know what type of RCE report to run?

Updated by Mary Styers

You might have landed here because you have data and you want to use that data to learn more about an edtech product. But where do you start? Specifically, how do you decide which RCE report is most meaningful to you? Let’s start with your goal (your end game) and work backward from there.

Why are you running a RCE report?

A research goal describes the purpose of the RCE report. In other words, what are you trying to achieve through this research? You may have more than one goal. Determining your research goals will drive the RCE report and impact data collection, analysis, and conclusions. By selecting your research goals, you’ve narrowed the focus of your research as well as the scope of the analysis. Education administrators often generate research goals to help them determine how an edtech product is being used; how users are following recommended usage guidelines; whether greater use of an edtech product relates to higher learning outcomes; whether students who use an edtech product outperform other students; and/or potential cost savings. By selecting one or more of these goals, we’re able to guide you through the research process and provide you with the most accurate RCE report to achieve your goals.

Which research goal is right for you?

There are a number of factors that could impact which goal(s) are right for your research interests, so let’s take a closer look at each one.

Usage Analysis Goals

Goal 1: Understand how my edtech product is being used

Are you interested in learning new insights into edtech usage at your school or district? There is a lot of variability in the implementation of products as a result of use and access. Being able to gain insights into the extent to which a product is being used and by whom is an important first step in understanding edtech effectiveness in your setting. Depending on when data is made available for your edtech products, you can run these reports monthly or every few months to understand how a particular edtech product is being used. 

Were users supposed to engage with an edtech product in a particular way and you want to know whether they did or not? A product provider or education administrator might set expectations for users to use an edtech product (e.g., five hours per week). If you want to know how many users reached that recommended benchmark, this research goal is right for you. Similar to the first goal, these reports can be run monthly or every few months depending on how often data is made available.

Outcome Analysis Goals

Goal 3: Understand if greater use of my product relates to higher learning outcomes

Do you want to know whether or not increased usage of an edtech product is related to improved learning outcomes? What about if there are optimal levels of usage for your users? For this type of RCE report, you will only include users who received the edtech product or intervention (e.g., outcomes without a comparison group) to examine the relationships between edtech usage and an outcome measure (e.g., mid-year test scores). You will be able to examine how, if at all, learning changed over a period of time and whether usage relates to that change. This goal coincides with one of our most common RCE reports, and can be run as often as you have outcomes available. 

Goal 4: Understand if students who use my product outperform other students

Do you want to know if there is a difference in outcomes between users who use an edtech product when compared to users who do not receive the same product or intervention (e.g., outcomes with a comparison group)?  What about if there are optimal levels of usage for your users who used the edtech product? In this type of RCE report, you will include users who receive an edtech product or intervention (e.g., treatment/intervention group) and compare their outcomes to users who do not receive the same product or intervention (e.g., comparison group) to determine if there were differences between the groups. This is the most rigorous approach to understanding edtech product effectiveness, so we recommend building up to this type of RCE report by examining prior goals first. You can run this type of RCE report as often as you have outcomes available. 

The outcome refers to the education outcome of interest, which should be a measure that the intervention could impact (e.g., examining literacy achievement when studying a literacy intervention). Typically, the outcome should be a quantitative measure with high reliability and validity, such as standardized testing scores. For outcomes analysis RCE reports, the IMPACT analysis includes covariates that account for extraneous data points that might explain your outcome (e.g., pretest achievement).

Cost Analysis Goal

Goal 5: Understand potential cost savings for my product

Are you interested in understanding potential cost savings for your product? Once you’ve learned more about the usage of your edtech product over time, you can uncover potential cost savings based on learning when products continue to have low or limited use.

RCE reports can be performed to learn more about the overall population in your school or district or for various subpopulations (e.g., grade level, gender). The population is the group of participants (e.g., students) included in your analyses. The population of interest can be very broad or it can be specific to a particular subgroup of participants, which may include students within a given school or grade, students of differing performance levels, or students who identify as a given gender, ethnicity, or special needs status. For RCE reports, you may be able to create student groups based on your product implementation and the context for use in your organization. Your RCE Manager can help you determine what those groups may be.

How did we do?

Powered by HelpDocs (opens in a new tab)