Project Background

This case study is based on the redesign of AutoGradr that was launched in 2019. In the 3 years we had heard a lot of feedback, and worked closely with instructors to understand how AutoGradr fits into their workflow.

Immediate problems

• Instructors spend tens of hours every week compiling and debugging submissions

• Students don’t get valuable feedback and often receive inaccurate grades 

• Current solutions are fragmented, outdated, and largely internal to universities

Larger problems

• Programming courses are in high demand in public schools in the US and abroad.

  There aren’t enough teachers who can teach programming, because they don’t code.

  Teachers who want to teach, don’t have access to quality materials to kickstart their curriculum.

AutoGradr is on it’s third version, having evolved from a lightweight solution that worked for high-touch customers to one that can be a plug-and-play solution for instructors all over the world who speak and teach in any language, in any format of classrooms.

Standard workflows

For Instructors:

Instructors currently download, run, and debug student solutions in order to grade it accurately. As student body sizes grow, this is becoming increasingly unsustainable, and finally a blocker to expanding course sizes.

Manual grading is extremely inefficient and not scalable. Any instructor in the world cannot be certain whether the code will run for a number of test cases in a reasonable amount of time without actually running the code itself. As a result, when the instructor has to grade hundreds of such pieces of code.

Feedback for Students:

Every student is unique, and can require help in different areas to help them grow, but when instructors are busy they cannot assist students or provide specific feedback on where to improve.

When tens or hundreds of students submit their code to be graded, any one of them who has worked hard expects to get an accurate grade and most likely a better grade than students whose solutions are not nearly as good. A classroom without AutoGradr is unable to differentiate these students.

Instructor interface from v2

• We were receiving a lot of support requests from instructors who were struggling to get started with AG

• In attempting to design a flexible interface that works for everyone, we did not have any set pathways for new instructors to follow

• Instructors were not able to test out how the students experienced the product

Student interface from v2

• Students would contact support about not understanding the feedback from their test cases

• Instructors wanted students to be able to code larger projects on AG itself

• Students wanted the ability to keep track of previous attempts

Guided on-boarding

When an instructor signs up, they are given five concrete steps to go through in order to get their course up and running. These steps were designed based on behavior we noticed in v2, and having talked to instructors asking them what they would need to successfully migrate.

Instructors also receive a series of emails after signing up that reinforce next steps after their first session.

Special emphasis on help content that is persistent across the experience in the right rail. Help content on complex screens is based on support requests received in v2

Console based test case creation

The IDE

Integrated development environment

Students further needed an IDE that would more accurately reflect real-world coding, and their own local environments if they wanted to switch back and forth between coding on their own devices, and bringing their workspace to AG to make attempts and edit based on feedback.

Keeping our focus on help content, the default state provides guidance on what students are able to do in the IDE, and settings that can help them feel comfortable with. Once students make attempts, an overview of their attempts appears in the sidebar, with access to previous attempts as they continue to make changes to their code. The console’s default state is to show the test case so students know what they are working towards, and after every attempt, the test cases are then juxtaposed with their own code’s output to provide students with context and a view of where their code might have failed the tests.