Amherst, Massachusetts

RobbieWalmsley

A computer scientist working at the orchestration layer of AI systems — building agents, retrieval pipelines, and tool-use into real products, across full-stack and real-time applications. Previously at Farsight AI. Open-source contributor to leading LLM agent and retrieval frameworks.

No. 01

About

I'm a computer science student at the University of Massachusetts Amherst, studying through the Commonwealth Honors College. My focus is the orchestration layer of AI systems — wiring language models into agents, retrieval pipelines, and tool-use, and deploying them as low-latency services on AWS. I contribute to the open-source agent and retrieval frameworks I build on.

Before UMass, I spent two years at UWC Atlantic College on the Welsh coast, completing the International Baccalaureate, leading development of an AI martial-arts coaching app, and placing in the top 1,000 in the UK in the UKMT — placing me into the British Mathematical Olympiad. Before that I spent my time between the United States and Argentina.

Languages

Python · TypeScript · JavaScript · SQL · C++ · Bash

Agents & Orchestration

RAG · Agents · Tool-use · LangChain · Instructor · Pinecone

Machine Learning

PyTorch · TensorFlow · HuggingFace · scikit-learn

Infrastructure

AWS · Docker · FastAPI · Postgres · WebSockets · Git / CI

No. 02

Experience

Software Engineer, Intern (Returning)

  • Re-architected the multi-agent deck-generation system off the Microsoft PowerPoint API onto an AI-native document format — a canonical JSON model, deterministic HTML rendering, and typed schema-validated operations as the agents' only write surface — driven end to end by Farsight's production task pane.
  • Closed the loop on agent output: slides render and screenshot in roughly 40 ms through headless Chromium, so agents visually verify their own work instead of editing blind through a remote API — 4× fewer vision calls at 3–4× lower cost, like for like.
  • Built the PPTX interop layer that lets agents safely edit real client decks: only agent-edited slides regenerate while every untouched slide stays byte-identical, validated with lossless round-trips on real 351-slide banking decks.

Software Engineer, Intern

  • Built an SEC-filings RAG feature using OpenAI APIs, Pinecone, and a sliding-window chunker for long-document queries; improved tabular QA accuracy by 29%.
  • Evaluated 6 embedding models and tuned retrieval, metadata filtering, and reranking strategies; reduced average query latency by 22%.
  • Prototyped schema-constrained outputs with Instructor to reliably return structured financial insights.

Open-Source Contributor

  • Implemented KaSA (ICLR 2025) as a fine-tuning method in HuggingFace PEFT (21k stars) via the library's LoRA-variant framework, which I extended to support it; found and fixed 3 latent bugs through adversarial self-review, and scored the lowest forgetting (0.013) of any LoRA experiment in the maintainer's benchmark.
  • 12 maintainer-merged pull requests across 10 widely-used AI/ML and LLM-agent libraries with 70k+ combined GitHub stars — the agent and retrieval orchestration layer: an MCP auth-header transport fix merged into Microsoft's agent-framework, a backend refactor in DocsGPT, and a contribution to Auto-GPT-Plugins.
  • Merged crash fixes to TransformerLens — a bfloat16 conversion crash and a sample_logits out-of-vocab crash, adding the function's first test coverage — plus a training-correctness fix in EleutherAI sparsify and merged fixes in circuit-tracer, pypdf, and SAELens.
No. 03

Selected Work

I.

Self-Play RL PvP Agent

2026·Python · PyTorch · PPO · Self-Play RL · NumPy

A Minecraft 1.8 sword-duelling agent trained with self-play PPO that plays the real game through pixels — screen capture in, synthetic mouse and keyboard out, no mod or game API in the loop. A vectorised NumPy rebuild of 1.8 movement and melee runs 2.44M environment steps per second across 4,096 parallel duels, training the policy for 99B steps. At runtime a CNN reads the screen and a GRU policy decides in 0.5 ms, inside the game's 50 ms tick. Built and trained on a single laptop.

Visit the project
II.

Real-time Motion Analytics

2025·Python · PyTorch · MediaPipe · AWS Lambda · WebSocket

Backend and ML pipeline for a real-time dance-analysis product at a stealth-stage UK startup. A causal Transformer on 3D pose data scores movement per frame and powers live coaching feedback at 93%+ accuracy, deployed through MediaPipe, WebSockets, and AWS Lambda for real-time delivery.

My contributionDesigned and built the ML pipeline and backend services — from the causal Transformer architecture and pose representation through the serverless inference path and WebSocket delivery layer.

III.

AutoRecycle

2023·Python · PyTorch · Vision Transformers · Raspberry Pi

A physical trash can with a built-in camera that classifies and sorts waste in real time using a DINO-pretrained ViT ensembled with a Swin Transformer, trained on the RealWaste dataset across nine waste categories. Achieved 95.5% multi-class and 98.7% binary recyclability accuracy. Built for a high-school NeurIPS competition.

My contributionBuilt the entire AI and software stack — the ViT + Swin Transformer ensemble, training pipeline, real-time classification, and control software. Teammates handled the physical Raspberry Pi build, camera rig, servo, and ultrasonic sensor.

Visit the project
IV.

MAIF Backtesting Framework

2025·Python · Pandas · NumPy · Optuna · yfinance · Alpaca · CCXT

A standardised plug-and-play framework for the Minutemen Alternative Investment Fund at UMass. Provides a unified pipeline for testing trading strategies across historical data, Monte Carlo simulations, and multiple data providers, catching overfitting through GBM and GAN-generated stress tests. Dual backtesting engines (bar-based and event-driven), Optuna parameter optimisation, and standardised HTML scorecards.

My contributionShipped the initial framework — the bar backtester, four-page scorecard generator, synthetic-data stress-test module, Optuna optimisation layer, and all ten data-provider integrations (Yahoo, Alpaca, Polygon, Tiingo, Finnhub, Alpha Vantage, Twelve Data, MarketStack, Stooq, FMP).

Visit the project
V.

Razorbill

2026·C++ · Zig · LLM Tool-Calling · Jolt Physics · Python

An AI-native 3D creation engine that lets anyone go from idea to playable experience in minutes. You describe a mechanic, a level, or a character behaviour, and an LLM agent builds the working project — scenes, scripts, prefabs, and interactive logic — by driving the engine through a tool interface.

My contributionBuilt the LLM orchestration layer that turns a prompt into a working project — a custom CLI exposed to the model as its tool interface, with context caching and model routing across calls. Also built the text-to-3D asset generator, procedural terrain, the animation and particle systems, Jolt physics tuning, and combat with navmesh-pathfinding enemy AI.

Visit the project
VI.

Health + Wealth

2025·Swift · SwiftUI · llama.cpp · Metal · Gemma 3 1B

A solo-built app pairing fitness and personal-finance tracking with an on-device AI coach — fully offline, zero network calls, all user data stays on the phone. It runs a custom on-device LLM serving stack for Gemma 3 1B (4-bit quantized GGUF) on llama.cpp + Metal — prompt templating, per-task sampling, a safety pre-screen, and thermal-aware token budgeting — and a structured workout generator and cross-domain health/finance coach grounded in the user's own data. Free, no subscriptions, no ads. Live on the App Store.

Visit the project
No. 04

Education

University of Massachusetts Amherst

Manning College of Information & Computer Sciences. Junior Quantitative Analyst with the Minutemen Alternative Investment Fund. Coursework in data structures, computer systems, reasoning under uncertainty, data management, artificial intelligence, and statistics.

UWC Atlantic College

Lead developer of Setup Sensei, an AI martial-arts coaching app. Mathematics Extended Essay. UKMT top 1,000 in the UK, qualifying for the British Mathematical Olympiad. Tech Representative and Codeventure CAS co-leader.

No. 05

Contact

Open to internships, research collaborations, or a conversation about an interesting problem. The fastest route is by email.