The PM's Guide to AI Training Data: How Much Do You Really Need?
Your weekly deep-dive into practical AI product management, delivered with actionable insights
TL;DR:
Different fine-tuning approaches require vastly different amounts of data
SFT needs thousands of examples, while RFT can work with as few as 50
The key isn't always more data—it's having the right data and clear success metrics
Pre-training requires billions, but fine-tuning can be surprisingly efficient
The Data Dilemma: A PM's Perspective
As product managers diving into AI, we often face a critical question: How much data do we actually need? Whether you're building a customer service bot or a content recommendation engine, the answer isn't always "more is better." Let's break down the reality of data requirements for different AI training approaches.
Watch this short video by Miqdad below 👇.
Understanding the Training Trilogy
1. Supervised Fine-Tuning (SFT): The Foundation Layer
Think of SFT as teaching by example. Just as you wouldn't expect a junior PM to learn from just a handful of user interviews, your AI model needs substantial exposure to get things right.
Real-world example: When Notion trained their AI writing assistant, they used thousands of high-quality writing samples to nail the tone and style their users expected. The result? A tool that feels remarkably in tune with Notion's user base.
Key considerations for PMs:
Budget for collecting 3,000-5,000 input-output pairs minimum
Focus on quality and diversity in your training examples
Consider your data collection timeline in your product roadmap
2. Preference-Based Fine-Tuning (PFT): The User Voice
PFT is essentially A/B testing for AI outputs. Remember how we test different feature designs with users? PFT applies the same principle to model responses.
Case Study: GitHub Copilot's evolution showcases PFT in action. By collecting developer preferences on code suggestions, they continuously refined their model's output quality.
Strategic approach:
Start with 500-1,000 preference pairs
Design clear preference collection mechanisms
Build feedback loops into your product architecture
3. Reinforcement Fine-Tuning (RFT): The Efficiency Champion
Here's where it gets interesting. RFT can deliver impressive results with surprisingly small datasets, provided you have a solid reward model.
Success Story: OpenAI's ChatGPT used RFT to dramatically improve output quality with relatively few examples. The key? Clear metrics for what constituted "good" responses.
PM Implementation Guide:
Begin with 50-100 high-quality examples
Define clear, measurable success criteria
Implement robust monitoring systems
Making the Right Choice: A Decision Framework
Choosing the right AI training approach demands a strategic blend of resource management and performance optimization. Surprisingly, more data doesn't always equal better results. Product managers must navigate a complex landscape of constraints—balancing time-to-market pressures, data collection resources, and budget limitations. The most effective strategies often emerge from scarcity: identifying the minimal dataset that delivers maximum impact. Your training approach should be a precision instrument, where quality trumps quantity, and each training example is carefully selected to drive meaningful improvements in model performance.
The Bottom Line
The "right" amount of training data depends on your specific use case, but don't let perfect be the enemy of good. Start with what you have, focus on quality over quantity, and build robust feedback mechanisms.
What's Next?
Do you have questions for Miqdad on this? feel free to hit reply and he’ll answer them for you.
P.S. AI Product Manager is the new Product Manager. Do you want to Acquire the skills & confidently build AI products and future proof your career? Join our #1 Rated AI PM course on Maven where you'll create an actual LLM-powered solution for your workplace. We’re offering $500 off (our biggest discount of the year) to the next 3 people who sign up using this link. If the link works, the promo is still valid. Looking forward to building something amazing together!
 | A guest post by
|