# FedEm · Project Seshu

> Decentralized foundation-model training

- **Role:** Core contributor
- **Company:** The Fifth Elephant · Open Source AI Hackathon
- **Period:** Jan – Apr 2024
- **Hackathon:** Meta (prize sponsor), The Fifth Elephant
- **Award:** National winner · ₹1,00,000

**FedEm** lets communities fine-tune language models without centralizing all the data. Adapters train locally, pass through a hub, and merge into shared checkpoints. **National winner** at **The Fifth Elephant** Open Source AI Hackathon (**₹1,00,000**, Meta-sponsored).

[FedEm demo · Project Seshu at The Fifth Elephant Open Source AI Hackathon](https://yashwardhan.space/assets/videos/fedem/fedem-demo-web.mp4)

## Problem

Training foundational models still concentrates GPU load in a few datacenters. Most contributors cannot pretrain at scale, yet open models like LLaMA 2 sparked demand for **community-built, multilingual alternatives**, especially for Indic languages where tokenization and compute costs bite harder.

**Take home:** Category: **AI for multilingual** · Demo Day **12 Apr 2024** · **national winner** and **₹1,00,000** prize (Meta-sponsored) at [The Fifth Elephant Open Source AI Hackathon](https://hasgeek.com/hack5/open-source-ai-hackathon-2024/), one of five winning teams from 40 applications.

![The Fifth Elephant Open Source AI Hackathon 2024 winners, Project Seshu](https://yashwardhan.space/assets/images/work/fedem/fifth-elephant-hackathon.png)

*The Fifth Elephant Open Source AI Hackathon 2024 · Meta-sponsored ₹1,00,000 prizes · Project Seshu national winner*

## What I did

### Continuous Relay Finetuning (CRF)

- Clients train **LoRA adapters** locally, push to a HuggingFace hub, and the next client continues finetuning, relay-style instead of one centralized run.
- Adapter **locking scheduler** so two clients never finetune the same adapter at once; hub checks every five minutes.
- Merged adapters feed back into a foundational checkpoint with transparent version tracking.

### Seshu, multilingual SLM track

- Explored **Mamba**-based pretraining for compute-efficient sequence modeling on Indic corpora (e.g. Samantar).
- Server-side continuous pretraining releases checkpoints; community adapters refine against each refresh.
- Client scripts for local pretrain → SFT → adapter push on the HuggingFace **mlsquare** org.

### Open source

- Shipped end-to-end pipeline in the **[fedem](https://github.com/mlsquare/fedem)** PyPI package (`pip install fedem`).
- Presented live demo at Hasura, Bangalore, 10 finalist teams from 40 applications.

![Continuous Relay Finetuning diagram](https://yashwardhan.space/assets/images/work/fedem/crf.png)

*Continuous Relay Finetuning (CRF) · from the FedEm repo*

## Outcomes

FedEm remains open source under MIT license. Initial contributors: [Soma Dhavala](https://github.com/dhavala), [Sai Nikhilesh Reddy](https://github.com/SaiNikhileshReddy), and me, community experiment to train models **by the people, for the people**.

- **₹1L** national winner prize
- **Top 5** of 40 teams
- **PyPI** fedem package

## Team

- [Soma Dhavala](https://www.linkedin.com/in/somasdhavala) (Advisor · mlsquare)
- [Sai Nikhilesh Reddy](https://github.com/SaiNikhileshReddy) (Core contributor · FedEm)
- [Yashwardhan Chaudhuri](https://www.linkedin.com/in/yashwardhan-chaudhuri) (Core contributor · FedEm)

## Impact

Hackathon demo, federated adapter training across volunteer GPUs; checkpoints on HuggingFace/mlsquare.

- National winner, The Fifth Elephant Open Source AI Hackathon 2024, ₹1,00,000 (Meta-sponsored)
- FedEm library on [GitHub](https://github.com/mlsquare/fedem) and PyPI
- Multilingual Mamba + LoRA roadmap for community SLM training

## Sources

- [FedEm on GitHub](https://github.com/mlsquare/fedem)
- [FedEm docs](https://mlsquare.github.io/fedem/)
- [Full demo video](https://drive.google.com/file/d/1cknBGBbAuwkPNMwy88yZ6hylL14ErVsd/view?usp=sharing)
- [Fifth Elephant hackathon](https://hasgeek.com/hack5/open-source-ai-hackathon-2024/)
- [Project Seshu submission](https://hasgeek.com/hack5/open-source-ai-hackathon-2024/sub/project-seshu-SrAMqmj7TXQrLrP16kvWNT)

---

[Back to portfolio](https://yashwardhan.space/) · [HTML case study](https://yashwardhan.space/work.html?p=seshu)