My favourite way to learn is to pick up a problem I care about and keep pushing the limits of my understanding. This allows me to learn new things fast. I turned some of these into products (CalWise, KidCompass, SaveToExo, AdvocateDesk), while others remain as my indulgences (Personal health assistant, IPL prediction agents). In that series, badminton was inevitable.
I play 6-7 hours a week, and I wanted to know whether AI could watch me play and give me useful feedback to improve. Signing up for formal coaching is impractical for most folks due to time, location or cost constraints. Even if you do, it does not simulate a match environment and most do not end up getting personal attention unless you are a pro. Hence, I thought a short coaching clip, made from self-shot videos, would be cool to build.
There are some existing purpose-built models for racket sports built on broadcast quality camera angles or motion sensors. However, in my case, the videos will typically be shot on mobile phones without a tripod. Worse, these videos are usually sent over WhatsApp (so compressed into low-res), the shuttle is a fast moving dot, the nearest player hides half the court and there will be background noise from nearby courts.
I was curious to see if vision models would solve this out-of the box. I started with Gemini 3.6 Flash, one of Google’s strongest video-capable models. I uploaded a 5-min clip and asked it to coach me. It was hilarious. The model confused me with my partner. It described non-existent shots during the break between rallies. It gave actions at uniform intervals - 2s, 4s, 6s, 8s (totally made up). Net net, garbage output. Tried all major models and pretty much the same. I could not find any prior implementation that worked reliably on footage like mine.
I did not have the training data or compute or the time to train a badminton model, so this was ruled out. I then attempted to solve the sub-components individually - (1) track the movement of each of the four players (2) track the flight of the shuttle and identify the shot type (lift, smash, drop, etc) and rally windows (3) catch the exact instance when each player hits the shuttle (4) make sense of what happened before and after the shot and (5) compare this against ideal singles and doubles techniques and formation and come up with actionable advice.
If any step in this flow was wrong, it would give wrong advice. So, I needed a source of truth for models to refer to. I took 5 clips that are ~15 minutes long and marked the players, rally windows, visible contacts, hitter and shot type. I also labelled what accurate and useful coaching advice would look like. I hated practically every moment of this, and the project was stalled for a couple of weeks due to this. No wonder data labelling is a billion dollar industry.
I then tried a bunch of open-source models. Experimented with YOLO for identifying players, ByteTrack for following the same player across frames, RTMPose for estimated body positions and TrackNetV3 for following the shuttle. But they weren't effective primarily due to the footage quality or angle not being good enough. To work around this, I assigned a single objective to each model. I made YOLO keep a box around the player who wanted coaching. I used TrackNet to only flag moments where a contact might have happened. This narrowed five minutes of video into short segments that I could send to the vision model.
Another interesting thing that I discovered was that my phone had recorded 30 frames per second (fps), but in my setup Gemini had 1 fps as default. Hence, Gemini said three out of three times that a smash that was actually in the video was not visible. I sent the same moment at 10 frames per second (more expensive call), and suddenly it could see it. So, it was literally not seeing most of the video earlier.
Once I fixed this, I ran a few more checks. Is this the right player to track? Is a rally actually happening or is it a break between rallies? Is the observation visible? Once these tests passed, the system would then pick an instruction from a small coaching library I had curated (my small version of RAG). I also put the development into an agent loop. I gave Claude Code and Codex a fixed goal to find at least two accurate and useful coaching moments within the preset cost thresholds. The loop had the flexibility to inspect a failed run, change the mix of tools, models and frame rates and keep running the eval again until it found a way out.
Five labelled clips from earlier became useful here. Unlike public video benchmarks which were general purpose, my eval told me whether it could follow one player in a compressed WhatsApp clip, catch a split-second action and give a sensible advice. It has now become my personal test whenever a new model claims to be better at video.
By the way, these were quite expensive experiments and cost me lower five figures (in ₹). Video costs add up quickly when every experiment involves multiple clips, models and frame rates (and an agent happily running the loop again). But it eventually led me to get Qwen3-VL to match the same level of performance as Gemini 3.6 flash at one-seventh the cost (!!!), which was a huge unlock. And for all the other stuff, I am using DeepSeek v4 flash as it has pretty insane price-performance.
So what do I have now? I have a system that can find two or three coaching themes from a 2-3 min clip, plays the relevant action and freezes on the exact point where the correction needs to be made. It works with reasonably good accuracy, but still fails now and then (1 in 10 times roughly). And even with all the optimizations, costs roughly ₹8 to ₹10 per minute of video uploaded.
This is why I have not launched it, yet. I am optimistic that this will eventually work, though. The models kept improving while I was building this over the last couple of months, and the repeated runs became more affordable thanks to Qwen. When the next model claims to understand video, I now have the player tracking, the video pipeline and my own eval to find out if it really does.
Meanwhile, if you have ideas to make this better or want to talk about it, hit me up.