5Y View｜How to Build AI Products That Are Actually Useful? How Generative AI Is Rewriting the Rules of Startup Building
Building AI products demands an entirely new playbook.
Recommender
Steven Shi, 5Y Capital
When I started focusing on AI investing five years ago, I hoped to find a genius product manager who could bridge "AI's limited capabilities" with "elegant, closed-loop user experience." Two years after ChatGPT's launch, the genius product manager for the AGI era has yet to appear. There has been no sudden emergence of a Steve Jobs, Allen Zhang, or Evan Spiegel to teach us how to build AI products.
Granola may not represent a viable standalone business niche in China, but in an AI note-taking market that's already extremely crowded even in the US, it has achieved the best product experience by word of mouth. I believe the product methodology Chris has developed is particularly instructive for all 2C / prosumer products. If the product you're building meets 3 out of 4 of the criteria below, feel free to reach out: stevenshi@5ycap.com
How to Build a Truly Useful AI Product
Generative AI Breaks the Old Startup Playbook
By Chris Pedregal
Co-founder and CEO of Granola. He previously co-founded Socratic, an edtech company acquired by Google.
First published: December 2024
If building a startup is like playing a very difficult video game, building in generative AI is like playing that game at 2x speed.
When you're building at the "application layer" — meaning you rely on AI models provided by companies like OpenAI or Anthropic — you're building on top of a technology that's advancing faster and more unpredictably than anything before it: at least two major model updates per year, with leaps in capability that are extraordinary. If you're not careful, you might spend weeks building a feature that the next model release automates away. And because everyone has access to excellent APIs and cutting-edge large language models (LLMs), a product idea you thought was unique can likely be built by anyone.
Of course, AI does open up genuinely new possibilities — code generation, research assistance, and other product capabilities were simply impossible before — but you need to make sure you're surfing the wave of technological progress rather than getting wiped out by it.
This is exactly why we need a "new playbook."
For the past two years, I've been building Granola — an intelligent notebook that uses transcription and AI to improve your meeting notes. This experience has convinced me that generative AI is a fundamentally different domain, where the traditional "laws of startup physics" don't fully apply. Rules like "solve your users' biggest pain point first" or "the more users you have, the lower your cost to serve" don't necessarily hold in AI.
If your intuition was trained by conventional startup experience, you need to rebuild it for AI. Through two years of trial and error, I've distilled four key principles for AI founders that I believe everyone building application-layer AI should understand.
1. Don't Waste Time on Problems About to Disappear
LLMs are undergoing one of the fastest technological developments in human history. Two years ago, ChatGPT couldn't process images, solve complex math problems, or generate complex code — all of which are now trivial. The capability landscape two years from now will likely be unrecognizable again.
For application-layer founders, it's easy to fall into the trap of "solving the wrong problem": you grind away at some immediate issue, only for the next GPT release to make it vanish. So don't waste time on problems about to disappear.
Easier said than done — because you have to predict the future, which sounds deeply uncomfortable. You need to find ways to anticipate what GPT-X+1 will be capable of, then build your product roadmap and strategy around those predictions.
Here's an example: Granola's first version couldn't record meetings longer than 30 minutes. The strongest model at the time was OpenAI's DaVinci, with a context window of only 4,000 tokens, so it could only handle short meetings. By conventional logic, we should have immediately prioritized fixing this — how could anyone use a note-taking tool that only works for short meetings? But we had a hypothesis: LLMs would soon become more powerful, faster, cheaper, and have longer context windows. So we decided to spend zero time optimizing for context window limitations, and instead focused all our energy on improving note quality.
During this period, we even had to consciously ignore user complaints about "too short a time limit." But our hypothesis was right: within months, LLM context windows had grown large enough to handle longer meetings. If we had initially invested in solving the "recording duration" problem, all that work would have been wasted. And the time we spent on "note quality" during that period remains one of the reasons users love Granola to this day.
2. Marginal Cost Is Opportunity
In the past, one of the most defining characteristics of software was that the marginal cost of adding one more user was essentially zero. If your product could serve 10,000 users, supporting 1 million wouldn't cost much more.
But this rule no longer applies in AI. Every additional user still carries marginal cost, and running cutting-edge AI models is extraordinarily expensive. For example, sending 30 minutes of meeting audio to OpenAI's flagship voice model GPT-4o costs roughly $4 per session. At thousands of daily users, this becomes substantial.
Moreover, your startup cannot freely scale to millions of users. Even with unlimited budget, OpenAI and Anthropic (the makers of Claude) don't have enough compute to serve millions of users simultaneously with frontier models. So for the first time in history, a new possibility emerges: providing a better product experience for a small number of users can be more viable than serving tens of millions.
But this isn't an obstacle — it's a massive opportunity for founders: large companies actually can't compete with you, because there simply isn't enough compute in the world for them to offer the most advanced AI experience to millions of users.
As a startup, you can absolutely give every user a "Ferrari-level" product experience:
- Use the most expensive, most advanced models without restraint
- Don't prioritize cost optimization
- If making 5 extra API calls (requests sent to your LLM provider) meaningfully improves the experience, do it
Yes, the per-user cost may be high, but you won't have many users in the early days. And remember: even a company like Google can only offer users a "Honda-level" experience.
You might worry: what if my "Ferrari experience" attracts too many users? Won't I face the same problem as big companies, unable to maintain quality?
The answer: don't worry. Even if your user count grows exponentially, AI inference costs are falling exponentially too.
Today's frontier models will be dirt cheap in a year or two. Today's Ferrari is tomorrow's Volkswagen. Drive the Ferrari while you still can.
3. Context Is King
When we first wrote prompts for Granola to generate meeting notes, we quickly realized that listing out step-by-step instructions didn't work well in practice.
The real world is messy; you can't pre-write rules for every situation. Even if you somehow covered all scenarios, the rules would easily conflict with each other.
Then we realized something crucial: rather than treating the AI model as a "tool that executes commands," treat it like an intern on their first day. An intern is smart but lacks context — they don't know what to do or how to do it. The key to making an intern successful is giving them enough of your "context."
This is the prompting strategy we now use at Granola: instead of just telling the model how to do something, we provide carefully curated context so it can think like you.
For Granola, its task is to generate high-quality meeting notes; and the context it needs is: who attended the meeting? What's the background? What do you hope to achieve in this meeting? What's your long-term goal? How does this meeting serve that goal?
We find this crucial information from the web and various sources, letting the model grasp your intent and ultimately write something genuinely useful. The "art" of this process lies in: choosing which contextual information is most valuable, and how to package it.
No matter how powerful the model becomes, the context it receives will always matter profoundly.
I believe "context window selection" will become a defining concept of our era, with significance far beyond AI itself. During the Industrial Revolution, people described brain function in mechanical terms — "letting off steam" was one such metaphor. In the computer age, we began using words like "bandwidth" and "storage capacity" to describe the brain. And next, we may use "context window" to understand how the mind works. This concept will eventually permeate society far beyond technology.
4. Narrower and Deeper
One of the most interesting challenges in building AI products today is that you're competing with general-purpose AI assistants like ChatGPT and Claude. They're already pretty good at many things. So how do you build something that makes users abandon the "Swiss Army knife" and choose your product?
There's only one answer: be "narrow" enough — truly, extremely narrow. Pick one very specific scenario, then go as deep as possible within it.
The startup iron law of "build something people really want" still holds in the AI era, but the bar has become higher.
But here's the interesting part: delivering exceptional experience in a specific scenario often has little to do with AI itself.
We've spent countless hours optimizing note quality in Granola, but we've spent just as much time polishing non-AI features — whether meeting reminders work smoothly, whether echo cancellation is good enough (the experience needs to be perfect whether or not you're wearing headphones). The "wrapper" around the AI is often what determines whether the user experience is "amazing" or "mediocre."
At the same time, the narrower your focus, the easier it is to optimize the AI portion of the experience too.
When AI gets it right, the experience feels genuinely magical; but when it gets it wrong, that failure tends to break the spell or even unsettle you — you suddenly realize you're talking to an algorithm, not a person. This "falling into the uncanny valley" product experience can easily drive users away for good.
But if your focus is narrow enough, it becomes much easier to identify AI's most common failure modes, so you can effectively avoid them, or at least ensure it "fails gracefully."
The Fundamentals Haven't Changed
Building a startup in generative AI is like running on a treadmill while traditional tech is still out for a leisurely stroll. This difference in pace affects the technical problems you face, your product refinement path, even your expansion rhythm.
While this "accelerated state" does demand new strategies, one thing remains unchanged: you still have to build something people genuinely want. There are no shortcuts.
You still need to obsess over details, patiently polish the product experience. The most critical question is still that deceptively simple yet deeply insightful one:
"How does this product make me feel?"
Original text below:
If building a startup is like playing a tough video game, building a startup in generative AI is like playing that video game at 2x speed.
When you're building at the application layer—your startup uses an AI model provided by companies like OpenAI and Anthropic—you're relying on technology that is improving at an unpredictable and unprecedented rate, with major model releases happening at least twice a year. If you're not careful, you might spend weeks on a feature, only to find that the next AI model release automates it. And because everyone has access to great APIs and frontier large language models, your incredible product idea can be built by anyone.
Many opportunities are being unlocked—LLMs have opened up product abilities like code generation and research assistance that were impossible before—but you need to make sure you are surfing the wave of AI progress, not getting tumbled by it.
That's why we need a new playbook.
Having spent the last two years building Granola, a notepad that takes your meeting notes and enhances them using transcription and AI, I've come to believe that generative AI is a unique space. The traditional laws of "startup physics"—like solving the biggest pain points first or that supporting users gets cheaper at scale—don't fully apply here. And if your intuitions were trained on regular startup physics, you'll need to develop some new ones in AI. After developing these intuitions over the last two years, I have a set of four principles for building in AI that I believe every app-layer founder needs to know.
1. Don't solve problems that won't be problems soon
LLMs are undergoing one of the fastest technical developments in history. Two years ago, ChatGPT couldn't process images, handle complex math, or generate sophisticated code—tasks that are easy for today's LLMs. And two years from now, this picture will look very different.
If you're building at the app layer, it's easy to spend time on the wrong problems—those that will go away when the next version of GPT comes out. Don't spend any time working on problems that will go away. It sounds simple, but doing this is hard because it feels wrong.
Predicting the future is now part of your job (uncomfortable, right?). To know what problems will stick around, you'll need to predict what GPT-X-plus-one will be capable of, and that can feel like staring into a crystal ball. And once you have your predictions, you have to base your product roadmap and strategy around them.
For example, the first version of Granola didn't work for meetings longer than 30 minutes. The best model at the time, OpenAI's DaVinci, only had a 4,000-token context window, which limited how long meetings could be.
Normally, lengthening this time frame would have been our top priority. How can you expect people to use a notetaker that only works for short meetings? But we had a hypothesis that LLMs were going to be much better: They'd get smarter, faster, cheaper, and have longer context windows. We decided not to spend any time fixing the context window limitation. Instead, we spent our time improving note quality.
For a while, we had to actively ignore users who complained about the duration limit. But our hypothesis was right: After a couple of months, context windows got big enough to handle longer meetings. Any work we would have done on that would have been wasted. Meanwhile, the work we did on note quality is one of the main reasons users say they love Granola today.
2. Your marginal cost is my opportunity
Historically, a defining characteristic of software was that the marginal cost of supporting an additional user was close to zero. If you had a product that worked for 10,000 users, it wouldn't cost that much more to support 1 million users.
This is not true when it comes to AI. The marginal cost of every additional user remains the same, and cutting-edge AI models are really expensive to run. For example, sending the audio of a half-hour meeting to OpenAI's flagship GPT-4o audio model costs about $4. Imagine that cost scaled across thousands of users, every day. There's also a limit to the number of users your startup can onboard. Even if you had all the money in the world, OpenAI and Anthropic (which makes Claude) don't have enough compute to support cutting-edge models for millions of users.
For the first time, it's possible to provide a better product experience for a small number of users than for millions of users. But this isn't an obstacle — it's a big opportunity for startups. Big companies with millions of users literally can't compete with you because there isn't enough compute available in the world to provide a cutting-edge experience at scale.
As a startup, you can give each of your users a Ferrari-level product experience. Use the most expensive, cutting-edge models. Don't worry about optimizing for cost. If doing five additional API calls (server requests to your LLM provider of choice) makes the product experience better, go for it. It might be expensive on a per-user basis, but you probably won't have many users at first. And remember: At best, companies like Google can provide their users with a Honda-level product experience.
You might be wondering what happens when users come flocking to your Ferrari product experience. Won't you end up in the same position as the big tech companies of today, unable to provide high-quality, cutting-edge services to your users?
The beauty is that even if your user base is growing exponentially, the cost of AI inference is decreasing exponentially. Today's cutting-edge models will be affordable commodities in a year or two. Today's Ferraris are tomorrow's Hondas. Be a Ferrari while you can.
3. Context is king
When we first started writing prompts for Granola to generate meeting notes, we quickly realized that providing a set of step-by-step instructions doesn't work well in practice. The real world is messy, and it's nearly impossible to anticipate and write rules for every situation an LLM might encounter. Even if you could cover every scenario, you'd inevitably have conflicting guidance.
We had an insight: Instead of treating AI models as something that just follows instructions, we should treat them like interns on their first day. An intern is smart but lacks context on what to do and how to do it. The key to an intern's success is to give them the context they need to think like you.
That's how we approach prompting at Granola now. We provide the model with curated context to guide its thinking. For Granola, the use case is writing great notes from a meeting. The context is understanding who is in the meeting and why it's being discussed. Our work is to find that information — from the web and other sources — and then get the model to think like you (What are you trying to get out of this meeting? What are your long term goals and how is this meeting in service of that?) and put only the relevant information in the notes. The art is in selecting which context to provide and how to frame it — because no matter how good models get, the context you give them will always matter.
I believe "context window selection" will be one of the defining ideas of our time, with implications far beyond AI. During the Industrial Revolution, the brain was described in terms of mechanical machines — blowing off steam, for example. When computers emerged, we started to use terms like "bandwidth" and "storage capacity." I think we will start describing how the brain works in terms of "context window selection." This idea will permeate well beyond tech.
4. Go narrow, go deep
One fascinating challenge with building AI products today is that you're competing with general-purpose AI assistants like ChatGPT and Claude. They're pretty good at most things. How do you build something good enough that users will choose you over these Swiss Army knives?
The only answer is to go narrow — really narrow. Pick a very specific use case and become exceptional at it. The cardinal rule of startups — building something people want — remains consistent in AI, but the bar is higher.
But here's the plot twist: Exceptional experiences for narrow use cases often have little to do with AI. We spend endless hours on note quality at Granola, but we spend just as much time on features like seamless meeting notifications and great echo cancellation (so our tool works whether you're using headphones or not). The "wrapper" around the AI is often the difference between a delightful experience and a great demo that is disappointing to actually use.
Going narrow also makes it easier to improve the AI part of your product. When AI gets a response right, it's magical. But when it gets it wrong, it does so in ways that can feel weird and disconcerting. It becomes obvious that you're not talking with a human, but with an algorithm. Product experiences that fall into the uncanny valley can push users away from your product for good. When you go narrow, it's much easier to identify the most common AI failure cases, and either mitigate them or try to fail more gracefully.
The fundamentals are the same
Building in generative AI is like running on a treadmill while traditional tech moves at walking speed. This speed impacts everything from the technical problems you tackle to your timeline for reaching scale. While this acceleration should change your strategy, it doesn't change the fundamentals of building a good product. You need to build something people want.
There are no shortcuts. You still have to sweat the details. And the most clarifying questions remain deceptively simple: How does this product make me feel when I use it?
5Y Capital seeks out, supports, and inspires entrepreneurs who walk a lonely path, providing everything from emotional backing to hands-on operational support. We believe that if the world starts believing in the "crazy" you that others doubt, things will get interesting.
BEIJING · SHANGHAI · SHENZHEN · HONG KONG
