# LEARN

> SWAYAM · Lecture segmentation & course discovery

- **Role:** Associate ML Scientist I
- **Company:** Wadhwani AI · Education
- **Period:** 2024 – 2025
- **Partners:** Ministry of Education India, SWAYAM, Bill & Melinda Gates Foundation

Hour-long STEM lectures are hard to navigate. On **LEARN**, my primary Wadhwani focus, I helped ship lecture segmentation and course search on **SWAYAM**, turning videos into chapter clips and making courses easier to find, in a national pilot with IIT Kanpur and the Ministry of Education.

## Problem

**SWAYAM** hosts thousands of hour-long STEM lectures. Without chapter markers, students scrub blindly and drop off. **LEARN** needed to split videos into topic clips and ship them on the Ministry of Education platform.

**Take home:** **Lecture segmentation toolkit, deployed nationally on SWAYAM.**

![LEARN program on SWAYAM, Wadhwani AI](https://yashwardhan.space/assets/images/work/learn/learn-hero.webp)

*source: Wadhwani AI*

## What I did

### AVLectures research stack

- Owned the segmentation research line for LEARN: replicated [Unsupervised Audio-Visual Lecture Segmentation](https://arxiv.org/abs/2210.16644) (WACV 2023) on SWAYAM lecture corpora, clip-level **visual** (frame CNN + 3D CNN), **transcript** (BERT), and **slide OCR** cues aligned via self-supervised narration–video matching.
- Fine-tuned embeddings on Indian STEM lectures, then clustered with **TW-FINCH** for temporally contiguous chapter boundaries. Lecture-domain fine-tuning on AVLectures beat zero-shot pretraining on generic video corpora.
- Evaluation on educator-labeled **Courses-with-Segmentation (CwS)** subsets: **F1** and **IoU** for segment overlap, **BS@30** for boundary accuracy within 30 seconds, plus **NMI** and mean-over-frames for clustering quality.

### IIT Kanpur intern squad

- Managed a parallel literature track: IIT Kanpur interns implemented **LGSS** (CVPR 2020) and **NeighborNet** on MovieNet + AVLectures, reporting mAP / mIoU against our shared benchmark suite. Owned scoping, experiment review, and synthesis.
- First sponsored field visit outside Delhi **for me**, campus interviews, squad coordination, and direct alignment with IIT Kanpur stakeholders on segmentation deliverables.

### Production toolkit (ASR + LLM)

- Shipped the national deployment path: **ASR transcript** → **LLM-proposed logical topic boundaries** → boundary validation against manually annotated chapters → chapter clips indexed into LEARN navigation.
- Packaged as an end-to-end **lecture segmentation toolkit** for SWAYAM ingestion. Ran head-to-head evals of frontier LLMs vs. AVLectures-derived heuristics on the same held-out labeled set.
- **National pilot: 10 SWAYAM courses** with chapterized lecture navigation from the toolkit output.

### Hybrid search index

- Built production **hybrid retrieval** for course discovery: Elasticsearch **BM25** + dense embeddings with **reciprocal rank fusion** over titles, descriptions, and syllabi, live on SWAYAM's public discovery surface.
- Led metadata ingestion pipelines and recall@k / human relevance evals as the MoE catalog expanded.

![Lecture segmentation pipeline: AVLectures research, intern literature track, ASR plus LLM production](https://yashwardhan.space/assets/images/work/learn/segmentation-pipeline.svg?v=2)

*Research track (AVLectures, my implementation) and production track (ASR + LLM, nationally deployed). Intern squad ran LGSS / NeighborNet in parallel.*

## Deployment

The segmentation toolkit went from AVLectures prototypes to a **10-course national pilot** on **SWAYAM**. Chapter clips fed LEARN navigation and search. The course discovery engine shipped on the same platform stack.

- **10** pilot courses
- **National Pilot** SWAYAM deployment
- **Live** course discovery engine

## Team

- [Arvind Balachandrasekaran](https://www.linkedin.com/in/arvind-balachandrasekaran-5653303) (Machine Learning Scientist · Reporting manager)
- [Makarand Tapaswi](https://www.linkedin.com/in/makarand-tapaswi-9a058532) (Principal ML Scientist · Project manager)

## Outcomes

- Built a **lecture segmentation toolkit** for a **10-course national pilot** on SWAYAM (AVLectures research → ASR + LLM production)
- Shipped the **course discovery engine** (BM25 + embeddings + RRF) for LEARN search
- Managed **IIT Kanpur intern squad** on LGSS / NeighborNet literature benchmarks under shared evaluation harness

## Sources

- [Wadhwani AI, LEARN](https://www.wadhwaniai.org/impact/education-solutions/learn/)
- [AVLectures (WACV 2023)](https://arxiv.org/abs/2210.16644)
- [AVLectures project page](https://cvit.iiit.ac.in/research/projects/cvit-projects/avlectures)

---

[Back to portfolio](https://yashwardhan.space/) · [HTML case study](https://yashwardhan.space/work.html?p=learn)