The app is open in the browser. Sample exams are already loaded. No install. For the teacher side, sign in with teacher / dropouts210.

Open the app
Socrates

Stanford CS210A/B · with NVIDIA

Working code is no longer the proof.

Coding agents make a finished project easy to produce, so a submission by itself says less about what a student understands. Instructors at CMU and Stanford described the same shift: ask students to explain their reasoning, defend a design choice, and answer a follow-up out loud. Socrates is the multimodal oral exam we built for that conversation, with NVIDIA.

50+ students Piloted in Stanford CS109, Probability for Computer Scientists.
89 sessions Real class use, then another pass on follow-ups, questions, capture, and speed.
600+ questions Generated for the course, grounded in the material, and asked across those sessions.
150+ chats Students kept talking with the system after the question, inside the same session.

A team of four — Alfred Yu, Shirley Yu, Aya Aburous, and Jad Bitar — in Stanford CS210A/B, Industry Innovation Lab, with NVIDIA. NVIDIA’s award at the end of the quarter. Promo video.

What instructors actually asked for

A useful oral exam remembers the conversation.

The early conversations were with professors and instructors at CMU and Stanford. The recurring theme was oral assessment: one student, one course, many turns.

  1. Remember what the student already said, and treat the session as one conversation.
  2. Ask a follow-up when the explanation is incomplete, in the moment.
  3. Stay on the course material, using the lectures and exams as the source.
  4. Leave a structured record a teacher can use later, tied to the transcript and to what the camera saw.

The first design, and why it changed

Everything fit on the Jetson until we ran it together.

We started on an NVIDIA Jetson Orin Nano so the exam could stay private, feel responsive, and keep going when the network was poor. Speech, vision, and a local language model each worked on their own.

On the device

The student-facing loop: hear the answer, speak the next question, and watch for a moment worth saving.

  1. Speech in and speech out, next to the student.
  2. A camera for the diagram, the equation, or the place they point.
  3. A local model for the turns that have to feel immediate.

What broke

Voice, video, and model inference competed for the same memory and compute. End to end, the pause was too long for a natural conversation.

  1. Each piece was fine alone.
  2. Together, the Orin Nano could not keep the dialogue moving.
  3. Forcing the heavy work onto the device made the exam feel stalled.

The redesign

Keep the conversation on the device. Send the heavy work to the cloud.

We split the system. The parts a student waits on stay on the Jetson. Heavier inference runs in the cloud, asynchronously, so a slow call does not freeze the turn they are in. A FastAPI layer routes that work and holds session state across the whole exam.

  • On the Jetson. Speech, the camera, and the turn the student is in right now.
  • In the cloud. The heavier language-model work, off the critical path of the live loop.
  • One session record. FastAPI tracks the multi-turn exam so each answer still belongs to the same conversation.
  • About 30% lower interactive latency after the split, compared with running the full stack on the Orin Nano.

Seeing the board without watching every frame

Students say “this part.” They rarely say “take a picture.”

In an oral exam someone draws a diagram, writes an equation, or points while they talk. Running vision on every frame would have spent the budget we had just moved off the device. The speech stream listens for those cues. A trigger grabs one frame, stores it with that turn, and later analysis can read the picture next to the transcript.

01Student says “here” or “this part” 02The cue is caught in the speech stream 03One frame is saved with that turn 04Transcript and image stay together

Questions that belong to the course

Retrieval, a fixed output shape, and memory of the session.

Follow-ups are only useful if they know the course and what this student already said. Course PDFs are chunked and indexed with FAISS. Generation is constrained to a schema, so a question arrives as structured fields the exam can ask and a teacher can review. Session state carries the conversation forward, so the next question is a response to the last answer.

01

FAISS over the course

Lectures and exams for CS109 are retrieved by topic. The question is grounded in that material, and the pipeline checkpoints each stage so a failed generation does not re-parse the course.

02

Schema-constrained generation

The model fills a known shape: the question, what a good answer covers, and the fields a later review can score. The output is a record, and a quoted line has to appear in the student’s answer.

03

State across turns

Each answer updates the session. The next follow-up can press on what was thin, and the teacher still sees one exam at the end.

Stanford CS109

Fifty students, then another pass on what they found.

We ran Socrates in CS109, Probability for Computer Scientists: 50+ students, 89 sessions, 600+ questions, and 150+ chat interactions. Students and instructors told us where follow-ups wandered, where a question was off, where the capture missed the page, and where the system still felt slow.

That feedback went back into the follow-up behavior, the question quality, the moment the camera fires, and the responsiveness of the live loop.

The hard part was getting speech, vision, retrieval, session state, the Jetson, cloud inference, and a real student in the same conversation, inside the memory and latency we actually had. Generating a question was the small piece of that.

Where to open the code

Start in these files.

PathWhat it is
jetson_runtime/proctor.py The live exam: speak, listen, follow up, and capture a frame when the student points.
backend/ FastAPI orchestration, session state, and the teacher review.
question_bank/pipeline.py Course PDFs to a FAISS index to questions the exam can ask.
integration/evidence_packet/ Structured scoring. A quoted line has to appear in the answer.
web/src/pages/ The student and teacher app. The review copy is linked at the top of this page.