Official SAT Practice on Khan Academy
Student dashboard
Primary practice area
Reading & writing practice questions
Overview and history
In spring 2014, Khan Academy and the College Board created a partnership to provide free, official practice for the SAT. The College Board, creators of the SAT, were launching a new version of the test, aligned to the Common Core, eliminating penalties for guessing, and returning to the traditional 1600 top score.
In the fall of 2014, I joined the project as lead designer (solo until spring 2015, when Natalie Fitzgerald began contributing primarily as an illustrator). We launched our beta in June 2015.
Our big launch came in January 2016, after students took the new PSAT and then received their score results, which our system imports to give each student a personalized practice plan focused on what they missed on the PSAT.
I continued as the primary designer through June 2017, expanding the product to include tools for teachers, essay practice, and driving the Khan Academy marketing effort for SAT.
Impact and efficacy
Since we launched Official SAT Practice on Khan Academy, it has become the de facto way to practice for the SAT, leveling the playing field for students whose families cannot afford expensive test prep.
-
Of students who prepare for the SAT, over 50% of them use Official SAT Practice on Khan Academy.
-
After the rollout of our free, official product, there was a 10%-20% drop in the number of students using paid test prep.
Our analytics teams also found incredible efficacy results:
-
Students who spend 20 hours practicing with us almost double their score gain between the PSAT and SAT, adding 115 points to their total vs. the expected 60.
-
Score gains are independent of ethnicity, gender, or family income—the determining factors are simply what their starting score was and the amount of time they invest.
With our tools for classroom teachers, Official SAT Practice is now a key offering that school district officials across the US are partnering with Khan Academy to deploy to their high schools.
Today, we continue to receive testimonials from students and parents about the difference that Official SAT Practice has made for them. The impact that we've been able to have is the highlight of my professional career.
Key concept innovations
In creating Official SAT Practice, we were not restricted to using our existing practice infrastructure or progress mechanics. With this freedom, I was able to design a product tailored to the needs of students preparing for the SAT, often with very short time horizons:
-
Students want to improve their score, not master learning concepts.
-
Students don't know what their specific strengths and weaknesses are.
-
Students don't know what the most common skill areas are on the SAT and how often they appear on the test.
-
Students are generally averse to doing the painful work of simulating a full, timed practice test.
Given these requirements, I designed the following properties of the system:
-
Our regular learning product focuses students on practicing the same skill until they master it and can always answer it correctly. This isn't an appropriate approach for SAT prep; instead, it makes sense to do bounded amounts of practice to try and make marginal skill improvements (i.e. pick the low-hanging learning fruit), then switch to improving the next skill with the greatest potential to increase their score.
-
We import students' PSAT results, with their right/wrong answers coded to individual skills. This enables us to prioritize their skill practice for maximum score impact by combining their individual information with our internal knowledge about how frequently the specific skills appear on the test. If the student has not taken the PSAT, we use an 80-question diagnostic to place them.
-
We provide and encourage taking full, timed practice tests; however, because it's so burdensome (as well as not targeted to the specific skills students need to improve), our primary focus for students is to use our personalized practice system.
One other very important innovation of this system is the introduction of content skill levels:
-
In our regular learning product, the different questions within a skill are designed to have a similar level of difficulty. However, on the SAT, there can be a large variance in the difficulty of questions in a particular skill.
-
For the question content our team created (and yes, our content team created all of the practice question content, though not the questions for the full practice exams), we calibrated it by difficulty, categorizing the easiest questions as "Level 2" and the hardest questions (or reading passages) as "Level 4". Students typically start at "Level 2" in a skill.
-
"Level 1" questions are special—they are questions from the prerequisites needed to understand the main skill. We draw these questions from other prerequisite SAT skills, or from the Khan Academy library of math practice.
Product onboarding
After arriving at our marketing landing page (originally designed by Natalie Fitzgerald) or signing up for Khan Academy, I created a set of onboarding screens that visually mirrored the experience students would have in answering multiple choice questions in our product.
We also immediately route them to connecting in their PSAT results, which bootstraps their entire practice experience. Illustrations designed by Natalie.
Product dashboard
After students have imported their PSAT scores, we have a wizard that walks them through creating a practice schedule, setting up their practice test dates as well as their weekly schedule.
Depending on what is scheduled for the day, students will see different CTAs in the main banner:
-
Practice reading & writing
-
Practice math
-
Take a practice test
-
Free day—no practice scheduled, do what you like
-
Actual test day—good luck!
-
After test day—congratulations, and check out our college admissions tutorials
Here's one of the screens from the original scheduling wizard. The amount of weekly practice we recommended was a function of the amount of time they had until their exam date. If they had more time, we'd recommend a lighter weekly schedule; less time and we'd recommend they study more intensively:
When we originally surveyed students, the respondents indicated that they wanted longer sessions, fewer times a week. However, in practice, this was a level of intensity that most students have difficulty achieving.
We analyzed activity data and A/B tested experiments to recommended different practice schedules. In the end, the recommended schedule that yielded the best actual results and consistency for students was practicing 15 minutes (which equates to 10 questions) every school day.
Question interfaces
For the SAT product, I did a visual reskinning of our existing question UI, based on work that designers at the College Board had done for a prototype version of an online SAT. The royal blue that appears throughout the product is the College Board's primary brand color.
The SAT has three main parts: math, reading & writing, and the essay.
In the math section, questions are typically standalone questions. In the footer, I created a series of boxes to help the student track their progress through the practice.
In the reading and writing section, all questions appear in the context of a passage; it does not have any standalone questions.
To address the challenge of wanting to be able to see both the passage and the questions together, I created a dual-column system, with the passage on the left and a scrollable list of questions on the right.
Having the questions be a scrollable list rather than requiring them to click through each better simulates the real test experience and also facilitates students being able to skim the questions before they read the passage, a common test-taking strategy.
To make it clear that the right column contained more than one question (on small screens, there might only be enough height to display one questions and its answers), I both labeled the question as "Question 1 of 11" and also added a bright blue "Scroll for more questions" indicator.
For essay writing, we actually created two different systems. For essay practice, we embedded a third-party system, "Revision Assistant," into an iFrame. It was a system designed to give students line-specific writing feedback.
For simulating the essay in the practice tests, however, I adapted the two-column passage UI:
Skill practice and timed mini-sections
In the daily practice areas for Math and Reading & Writing, the student goes through a specific cycle of practice.
I used different colors to represent Math and Reading & Writing so the otherwise similar interfaces would be visually distinct. The dark blue for math was the coded "math" color on Khan Academy; the purple for reading & writing was the coded "test prep" color on Khan Academy. I considered using our coded color for "humanities," but it was bright red and I felt it could be too closely tied to "red = wrong".
For math:
-
Students do three sets of targeted skill practice. The skills are chosen by an algorithm we developed that's based on the frequency that the skills appear on the test, plus whether the student recently missed questions in those skills (or, when first starting with the product, missed those questions on the PSAT).
-
After doing three sets, the student unlocks a "timed mini-section," which is essentially a shortened version of a full test section, designed to be completed in less than 15 minutes. If the student misses questions on the timed mini-section, our algorithm promotes those skills when it then generates a new set of three skills to practice.
-
This cycle enables our system to dynamically identify the handful of math skills that will make the biggest score increase if the student improves them.
-
Our algorithm also tracks the questions that students answer on practice tests. When a student misses questions from a particular skill two times in a row when doing timed mini-sections or practice tests, we downgrade that skill level (or upgrade it if they got them right).
For reading & writing, we use the same basic structure of three untimed practices, followed by a timed mini-section.
However, because the practice questions are bundled together with specific passages, it's not possible for us to target skill practice the way we do with math. Additionally, the College Board's research indicates that the difficulty of the reading or writing passage, rather than any particular skill proficiency, is the primary determinant of how a student performs.
Because of this, instead of dynamically choosing the reading and writing passages for students to practice (slots 1 and 2), we serve up preset passage categories (e.g. Reading: Literature) based on the frequency they appear on the test. However, the difficulty of the specific passage we choose is based on the student's current skill level.
For slot 3, we have the student practice a particular grammar skill, which is dynamically determined in the same way we prioritize math skills.
For essay practice, which uses the embedded third-party system "Revision Assistant", we have a fixed set of essay prompts (not the ones I fabricated for the mock below) that students can practice:
Open practice library
Below our practice recommendations, we list the full set of skills or passage types that students can practice, as well as links to lesson videos. If the student does this practice, they get questions at their current level. These questions also count toward satisfying their daily practice goal.
Practice exams
We recommend that students take practice exams on Saturday mornings, to best simulate what their experience taking the real Saturday SAT will be like.
The overview UI for the practice exams shows the (standard) order in which the sections will come. Each block is vertically sized based on the relative time it will take, as each section takes a different amount of time.
Starting a section takes the student through the standard instructions and pages they would encounter on the real test, then starts the timer, which appears in small type in the page header. It updates every minute, then starts counting down the seconds when two minutes remain.
Students are not able to pause the test section, as they would not be able to do so under real conditions. They are able to navigate between questions, and when they reach the end, the system guides them back to any questions they may have skipped.
Once the student finishes the test, they receive their score and can review their answers. (I didn't do so hot on my practice tests.)
Once they finish their test, we congratulate them on a job well done:
Onboarding and reminder emails
I designed a companion email onboarding series to shepherd students in getting started, as well as triggered reminder emails for scheduled practice days.
Email marketing campaigns
I also led Khan Academy's SAT email marketing, which was structured around the run-up to SAT test dates. This included partnering with the College Board marketing team to weave in our product to their marketing to students registered for the SAT.
I created "Official SAT Practice Day," the Saturday two weeks before every SAT test date, as a date for students to take a full practice test before their real test. The College Board also incorporated their event into their email series. When creating their schedule in-product, this day was always one of the dates recommended for taking a practice test.
Tools for classroom teachers
College Board identified an opportunity to expand Official SAT Practice from a direct-to-student product into a school- or teacher-driven product. Their research team assembled a proposal for our team to create tools for teachers; on their end, they would be working with teachers to develop instructional material to include in the product.
There are many ways high schools and teachers work to help their students prepare for the SAT:
-
After school programs
-
Specialized SAT practice class periods
-
Math and humanities teachers incorporating SAT prep into their curricula
My product manager Anju Khetan and I recruited and interviewed teachers who were already leveraging the product with their students despite the lack of reporting for teachers. Key insights from our interviews:
-
Teachers want to leverage the power of the personalized system, they're looking for ways to augment it.
-
Teachers want to be able to hold their students accountable to the schedule and goals they create.
-
Teachers want to be able to provide targeted small-group or whole-class instruction.
Working in a feedback cycle with these teachers, I designed an SAT dashboard for them that prioritized communicating:
-
The highest-priority Math and Reading & Writing skill areas our system is recommending to their students, with a histogram of their students' current levels in those areas, and a button to browse the lesson material associated with each skill area.
-
The overall activity of their students, including whether they'd linked their College Board account, and whether they were keeping up with their practice schedule.
Main dashboard:
Pressing the "View details and browse lessons" CTA for a specific skill area opens a modal that contains instructional material and a more detailed breakdown of student levels:
From the student table in the main dashboard, teachers are able to get a detailed view of what the student has been working on and also review how they answered questions.
We required that students opt-in to sharing their SAT practice information with their teachers, and we do not share the students' actual PSAT or SAT results with their teachers, only their practice test scores.
Even for practice test scores, we put additional messaging around this information, emphasizing that these scores should not be used for grading.
We specifically limited the reporting to covering the two most recent weeks because we wanted the tools to be used as a way for teachers to check that their students are practicing rather than to be used for grading purposes of some kind, for example if the teacher's class was specifically an SAT prep course.
Conclusion
Official SAT Practice has been Khan Academy's most successful product to date. It is a key offering for our new district product. It also opened a business opportunity for us to expand beyond the SAT and design a generalized test prep system, which I led the research and initial product development for. The generalized system is now used for our LSAT and Praxis (a teacher credential test) prep products.
I'll put in a little plug for the product here—if you know someone who has a high school junior, be sure to tell them about Official SAT Practice. It's free and personalized to them. It's the best way to study for the SAT, so don't miss out.