The binding constraint on growth is no longer intelligence. It’s human verification bandwidth.

AI can generate a 50,000-line application in a day. A design document in minutes. A legal brief in seconds. Execution cost is approaching zero.

But someone still has to check whether the output is correct. Whether the code handles edge cases the model didn't think about. Whether the legal citations actually exist. Whether the billing logic accounts for the dedicated carrier program that nobody wrote down.

That checking happens at human speed. Reading speed. Context-building speed. Domain expertise speed. None of these scale with compute.

The Faros AI study across 10,000+ developers put numbers on it: teams using AI complete 21% more tasks and merge 98% more PRs. But PR review time goes up 91%. PRs are 154% larger. Bug rates climb 9% per developer. The work didn't disappear. It moved from writing to reviewing.

Full delegation requires trust!

The verification problem is really a trust problem wearing a technical hat. You trust a colleague to run a project because you've seen their judgment over the years. A doctor trusts a resident's diagnosis after watching hundreds of cases together. A manager trusts a direct report's analysis after months of calibration. That trust was expensive to build. It doesn't transfer to a model that hallucinates between 0.7% and 94% of the time, depending on who built it.

This framing came up in conversation with a colleague at work. Thank you Francisco Arceo for sharpening the thinking here.

The Stack Overflow 2025 survey makes this concrete: 84% of developers use AI tools, but only 33% trust the output. A 51-point gap between adoption and confidence. The more people use AI, the less they trust it. Experienced developers trust it least.

Additionally, the number of agents producing output grows with compute. The number of humans available to verify stays fixed. Every new agent, every new workflow, every new automation draws down the same finite pool of human attention.

The measurability gap

The paper introduces a concept called the "measurability gap." Quantifiable tasks are automated first. What's left are the tasks that require judgment, context, and liability: what the authors call n-hard or n-legal processes.

The dangerous part isn't that AI produces wrong answers. It's that the wrong answers look right. CIO.com put it well: "Almost-right code is insidious. It compiles. It runs. It passes the basic unit tests. But it contains subtle logical flaws or edge-case failures that aren't immediately obvious." Finding what's missing in 100 lines of AI-generated code is harder than writing the 100 lines yourself.

The METR randomized controlled trial found experienced developers using AI tools took 19% longer than without AI. Before the study, they predicted AI would make them 24% faster. After the slowdown, they still believed it had sped them up by 20%. That's a 39-point gap between what people feel and what actually happened.

The HBR study from UC Berkeley tracked 40 workers over 8 months and found something similar: in the moment, people described momentum. When they stepped back, they described feeling busier, more stretched, less able to disconnect. 62% of associates reported burnout. AI didn't reduce work. It intensified it.

Hollow economy vs. augmented economy

The paper's central warning: without verification infrastructure, the market drifts toward what they call a "Hollow Economy." Measured activity explodes. Human control hollows out. GDP goes up. Understanding goes down.

The alternative is an "Augmented Economy" where verification scales alongside automation. That means treating verification as a production technology, not a compliance checkbox. Cryptographic provenance, liability underwriting, evaluation records, audit trails. The ability to insure outcomes, not just generate them.

One failure mode I keep thinking about: the expertise decay loop. Routine tasks automate, entry-level positions disappear. Those positions were where future expert verifiers got trained. The system slowly undermines its own ability to check itself.

Where this leaves us

intent -> execution -> verification. Humans set intent. Machines execute. Humans verify and take responsibility.

Compute gets cheaper every quarter. Human attention does not. Every organization deploying AI at scale is discovering this firsthand. Not from reading papers, but because their review queues are backing up, their senior engineers spend more time reading generated code than writing their own, and their confidence in what shipped last Tuesday is lower than it was a year ago.

The real cost of AI productivity isn't compute. It's the attention of the people who still have to decide whether the output is worth trusting.

Something changed in the last year that I don’t think we’ve fully processed yet.

PMs (regardless of the technical level) can wire-up full prototypes, not a Figma file, working prototypes with real API calls. They can open up pull requests, the engineers review, clean it up, and ship! The whole cycle, from idea to production, could take less than three days. A year ago, that same process would have started with a spec, sat in a backlog for two sprints, and gone through three rounds of “that’s not what I meant.” (yes that still happens, but these are process bottle-necks and AI won’t help here, a topic for another post :))

I start to see this happening everywhere. PMs are vibe-coding prototypes in Cursor. Designers are pushing CSS changes directly. Marketing people are building internal dashboards with Replit. The tools have gotten good enough that “I can’t code” is no longer a permanent identity. It’s a choice.

And this is starting to change how companies think about roles. LinkedIn’s CPO killed their APM program and replaced it with “full-stack builder” roles.

“Vibe coding is the new product management.”

— Naval Ravikant

The question people keep asking, sometimes quietly, sometimes in all-hands meetings, is whether we still need a dedicated product management role at all.

The thesis goes like this: if the barrier to building software has collapsed, why keep a role whose main job was translating between “what customers want” and “what engineers build”? Why not just have everyone do both? Ship code in the morning, talk to customers in the afternoon, split the time evenly.

This idea has a name now. People are calling it the “builder model”: dissolve the traditional PM, designer, and engineer boundaries. Everyone writes code, everyone talks to customers, everyone ships. No handoffs, no translators, no waiting on someone else to turn your idea into software. Some versions of this are modest (PMs should prototype more). Others are radical (the PM role itself is obsolete, just hire builders). The Era of the Builder Product Manager captures the spirit: the old way of being a PM is dying, and the future belongs to people who can get their hands dirty.

I find the builder model appealing and mostly wrong. Here’s why.

The translation layer is compressing. That’s real.

I don’t want to dismiss what’s changing. The gap between “I have an idea” and “I have a working prototype” used to be weeks. Now it’s hours. Tools like Cursor, Replit, and Claude Code let someone with product intuition but no engineering background spin up something functional in an afternoon. Atlassian wrote about how AI is turning PMs back into builders. The ACM published a piece calling vibe coding an imperative for product managers.

This matters. When a PM can show a working demo instead of writing a 10-page spec, conversations with engineering get better. Fewer misunderstandings. Faster iteration. That’s a genuine improvement.

But prototyping is not the same as building.

“Everyone can code” is not “everyone can build”

There’s a gap between a vibe-coded prototype and production software that AI hasn’t closed. Bryce York made this point well: vibe-coded output is isolated from production codebases. Engineers have to re-implement from scratch. That’s not a handoff; it’s a translation, and translation costs time (and I can say this because I have been on both side, so anecdotally, I know that’s a fact).

The numbers back this up. A study by METR found that

Experienced developers using AI tools believed they were 20% faster. Objective measurement showed they were actually 19% slower.

— METR Study

experienced developers using AI tools believed they were 20% faster. Objective measurement showed they were actually 19% slower. Think about that for a second: developers felt faster and were measurably slower. Stack Overflow’s 2025 Developer Survey found that trust in AI coding tools fell for the first time, even as adoption kept climbing.

I keep coming back to this distinction: making a demo work is engineering on easy mode. Making it work reliably, securely, at scale, for years, with other people’s code touching it, that’s still hard. And that’s where the actual engineering skill lives.

The builder model misdiagnoses the PM bottleneck

Here’s where that builder thesis starts to fall apart for me. The argument starts from a real observation: PMs spend 60-70% of their time in meetings and have little time to innovate. The proposed solution is to give them coding ability so they can ship things directly.

But this solves the wrong problem. Those meetings aren’t wasted time that coding could replace. Most of them exist because someone has to:

• Align stakeholders who want conflicting things

• Say no to 80% of feature requests without making enemies

• Synthesize messy, contradictory customer signals into a strategy

• Navigate internal politics to unblock engineering work

• Make prioritization decisions with incomplete information

All of these are (not so easy) judgment calls. None of them gets easier because a PM can write code. Giving PMs coding ability without reducing their coordination load just adds another job to an already overloaded role. Now you’re asking them to attend six hours of meetings AND ship code. That’s how you burn people out.

Even distribution is a fantasy

The builder model assumes people can split time evenly across coding, customer conversations, strategy, and shipping. I think this ignores how human cognition actually works.

Deep coding requires flow state, uninterrupted blocks of two to three hours to do anything meaningful. Customer discovery is a different kind of work entirely: empathy, preparation, follow-through. Strategy means stepping back and thinking about what not to build. And then there are the meetings, which are pure context-switching.

Asking one person to do all four “evenly” means they do all four at a mediocre level. I’ve watched engineers who get pulled into too many customer calls lose their technical edge. I’ve watched PMs who try to code too much lose track of their customers. There’s a reason specialization exists. It’s not bureaucracy. People produce better work when they can stay in one mode long enough to actually think.

Faster building means more need for someone deciding what to build

This is the argument most builder-model advocates skip over.

When shipping is cheap, more things get shipped. More features, more experiments, more prototypes. Someone needs to decide which of those things should actually exist. As one critic put it: with everyone building faster, there is now more product work to do, because someone needs to think about whether we should build all those things.

Without that filter, you get feature bloat, slop, inconsistent UX, and a product that tries to do everything for everyone and does nothing well. Cheap building makes the person who says “no, not that” more important, not less. That has always been the PM’s actual job (and AI believe it or not is letting engineer get into that mindset as well).

Where the builder model actually works

So is the builder model “wrong”? Not really, but It works in specific contexts:

Early-stage startups with fewer than 15 people. Developer tools where the builder is the customer. Internal tools where scope is limited and stakes are low. Consumer apps where one person can hold the entire problem in their head.

In these contexts, the overhead of having separate PM and engineering roles isn’t worth it. A small team of builders who talk to users and ship code can outperform a larger team with handoff layers.

But scale that up. Try it with a healthcare SaaS product where someone needs to understand clinical workflows and regulatory requirements. Try it with an enterprise platform where 50 customers all want different things and the PM’s job is figuring out which five to actually build. The person doing that work should not be spending half their time coding. They should be spending it with clinicians, or reading through support tickets, or arguing about roadmap priorities.

What I think actually happens

I don’t think the builder model replaces PM. I think what actually happens is less tidy:

Teams get smaller. A pod of two or three people does what used to require eight. The BCG workforce report describes this shift as already underway across the industry.

PMs prototype more, but as communication tools. A PM vibe-codes a working demo to show engineering “this is what I mean.” That replaces the spec, not the engineer.

Junior PM roles (could) disappear (yea that’s a problem!). Entry-level PM work (writing specs, organizing feedback, tracking competitors) gets automated and we need to rescope that role’s job. Senior PMs who can do strategy, politics, and customer insight survive (till we figure out how to automate humans entirely, AGI or whatever we call it, automation all the way, also a topic for another post). There are fewer of them and they matter more.

The “what to build” question becomes the scarce “skill”. As building gets cheaper, judgment is what separates good teams from bad ones (understanding pain points, discovery work, etc, becomes even more valuable). What to build, what to kill, what to say no to. AI can’t automate those calls because they depend on knowing your customers and your constraints better than anyone else does.

Engineering shifts toward systems work. Engineers spend less time on CRUD features and more on architecture, performance, security, and reliability. The parts AI still gets wrong.

The real question

I think we’re asking the wrong question. “Should PMs code?” is less interesting than “how do we stop PMs from spending 70% of their time in meetings so they can actually think?”

Most PM meetings exist because organizations have communication problems (yeah, we are hacks for bad process, which seems like a non-NP problem 😬), not because PMs have too much free time. Better async tools, smaller teams, and clearer ownership structures would do more to improve PM productivity than granting them Cursor access.

The companies that do well with this won’t be the ones where everyone does everything. They’ll be the ones with small teams where each person has a clear job, and AI reduces friction in working together. Roles don’t dissolve. They no longer require three meetings to coordinate.

That’s less romantic than “everyone’s a builder.” But I think it’s closer to what actually works.

The Problem MCP Solves

At its foundation, MCP rests on a simple but powerful principle: AI models are only as effective as the context they are given. Imagine asking a colleague for help without providing any background information. Their assistance will likely be limited. MCP addresses this by providing a standardized way for AI applications and agents to access and interact with external data and tools, enriching their context and enabling more intelligent and useful behavior.

Before MCP, integrating AI with external systems (databases, APIs, local files, even web services) was often a bespoke and lengthy (developer time) process.

Each new integration required custom code, leading to a "many-to-many" problem (or some call it the “N x M” problem). This created a fragmented landscape, slowing down development and hindering interoperability.

MCP proposes a "one-to-many" solution, a universal interface that connects any MCP-compatible AI applications to any MCP-compatible server, where the logic is detail abstracted by the unified interface, but let’s not get ahead of ourselves and take a quick look at MCP’s building blocks.

Prompts, Tools, and Resources (the building blocks)

MCP isn't a monolithic entity. It's built upon three well-defined blocks, each granting a different level of control:

• Prompts (User-Controlled): These are predefined templates for common interactions initiated by the user. Think of slash commands in a text editor, IDE, or Slack (e.g., /summarize to summarize a pull request, /translate to translate selected text, /code-review to review code/PRs/…). Prompts provide a consistent and user-friendly way to interact with specific services, ensuring well-formed input for the language model. They are essentially user-invoked “tools”.

• Resources (Application-Controlled): Resources represent data exposed by the server to the client application. Unlike tools, which are actions, resources are information. Importantly, the application (not the model) determines how to utilize this data. This could involve displaying an image, attaching a file to a message, providing contextual information to the user, or using the data in subsequent computations. Resources can be static (like a file) or dynamic (like data fetched from an API that changes over time). This can also be used with prompts, for example, when invoking prompts or tools from the CLI, resource URIs could be passed as parameters or arguments

• Tools (Model-Controlled): This is where the AI model's agency comes into play. A server exposes a set of tools, and functions that perform specific actions (or call one or more APIs). These could range from reading and writing files to querying databases, accessing APIs, or even controlling external devices. The thing to note here is that the language model that is used by the client application is what decides when, what, and how to invoke these tools based on the task at hand.

I personally have not seen that much use of resources and prompts; most of the excitement has been around tools (exceptions do exist, though, just not as common).

For more details on the building blocks, see the protocol specification and check out this talk: Building Agents with Model Context Protocol - Full Workshop with Mahesh Murag of Anthropic

Communication Modes

In addition to the trifecta above, MCP can operate in different modes:

• STDIO: Good for quick starts and onboarding. Here, the client often spawns the server as a subprocess, piping JSON-RPC messages via STDIO.

• HTTP + Server-Sent Events (SSE) [Older comms]: Used two separate channels: regular HTTP POSTs (Client->Server) and a dedicated SSE stream (Server->Client). Required stateful servers for the SSE part. This had some drawbacks, including incompatibility with serverless, resource-intensive(ness), and didn’t really get adopted that much, also MCP allows for hierarchical servers relationships, which complicates things, especially with SSE

• Streamable HTTP [Newer comms]: No more "always-on" connections to remote HTTP servers. Uses a single endpoint. Leverages standard HTTP POST. The server chooses whether to reply with a single JSON response or upgrade the POST response itself into an SSE stream. Also allows a separate GET for a dedicated SSE stream if needed. More flexible, supports stateless servers, and adds resum-ability.

Now, debate exists regarding the trade-offs between MCP's complexity and its utility across the different communication modes. Remote MCP, particularly when implemented over HTTP, is frequently perceived as complex due to added layers like JSON-RPC wrappers, especially when compared to traditional REST APIs defined by OpenAPI specifications (Why not JUST OpenAPI, man?). However, some argue this complexity is justified by enabling a reasonably powerful paradigm: “users dynamically adding tools and context sources to AI applications at runtime”, much like browser extensions, going beyond static, developer-defined integrations. In contrast to remote and its perceived complexity, local MCP via stdio, commonly used for inner-loop developer tools (e.g., requiring filesystem access or command execution), is generally viewed as more straightforward and practical for its purpose, making it the more prevalent implementation observed so far.

The Value Proposition: A Win-Win-Win-Win Scenario

MCP offers some tangible benefits to a wide range of users:

• WIN 1 | Application Developers: The most immediate benefit is the elimination of custom integrations. Once an application is MCP-compatible (i.e., an application becomes a client), it can connect to any MCP server with zero additional coding. This reduces development time and effort, allowing developers to focus on the core functionality of their applications rather than the plumbing of integrations.

• WIN 2 | Tool and API Providers: By building an MCP server once, tool and API providers can expose their services to a broad ecosystem of MCP-compatible applications. This provides a standardized "on-ramp" for AI, increasing their reach and potential user base. It's a way to "AI-enable" their existing services with minimal overhead.

• WIN 3 | End Users: The ultimate beneficiaries are the end users, who gain access to more powerful, context-rich, and personalized AI applications. These applications can understand and act upon their data and the real world in ways that were previously impossible or required significant manual intervention. Take Zapier MCP actions, for example. For a non-technical consumer, you’d just point to Zapier, and magically be able to “chat” with all available integrations. I chose Zapier because while MCP standardized tool interactions through a protocol, Zapier had standardized access to “integrations/tools” with no/low-code, and now they MCPed all their integrations via Actions.

• WIN 4 | Enterprises: MCP offers a clear way to separate concerns and streamline internal development workflows. Teams responsible for managing infrastructure (e.g., vector databases, CRM systems, internal APIs) can expose these resources as MCP servers. Now, other teams can consume these services with natural language without needing to build custom integrations or understand the underlying implementation details.

Agents ♥️ MCP

Perhaps the most useful incarnation of MCP lies in its role as a foundational protocol for AI agents to use tools. An agent, in this context, can be thought of as an "augmented LLM", a language model that is enhanced with the ability to interact with retrieval systems (for accessing information), tools (for taking actions), and memory (for storing and retrieving information). MCP provides a standardized interface for these interactions. Here is the Agents equation:

Agents = Model + Augmentation(Tools + retrieval + memory) + Loop

More importantly, MCP enables agents to dynamically discover and utilize new tools and data sources, even after they have been initialized. This means an agent can "learn" new capabilities by connecting to new MCP servers, without requiring any changes to its core code. This alone is a big win, but it comes with trade-offs, for example, MCP servers overwhelming the Agent’s LLM context and rendering it “kaput”.

Most of the known “agent” frameworks had started to bring MCP client supportability in as a framework-native feature, most prominently, Open-AI bought in, and this more or less settles it!

And looks like Google will be biting soon as well:

Another seemingly powerful concept is composability. Any application or agent can be both an MCP client and an MCP server. This allows for the creation of layered, multi-agent systems where specialized agents can interact with each other and with external services in complex workflows. Composability makes hierarchies possible, but the downside is that it makes everything else a tad harder. For example, security (credentials propagating downstream), debuggability (the load shifts on building that on the server application, so it must be very well instrumented), latency/performance (the length of the hierarchy/chain of client/servers can be unpredictable, so unless there is a fixed tested n-depth, performance can’t be guaranteed).

Standard OR “Overhyped Marketing Fluff”

Short answer: OpenAI’s adoption of MCP in its agent SDK solidified its stance towards being a standard rather than just “Fluff”. Langchain’s twitter poll did as well 🙂

I recently came across this post from Harrison Chase (CEO of LangChain) and Nuno Campos (lead of LangGraph), where they debate the practicality and usefulness of MCP for developers AND the rest of the crowd. Let’s get into the details.

Optimism and Skepticism Striking the Right Balance

In the post, Nuno expresses some doubts about MCP's utility beyond very basic tool replacement. He argues that effective tool use in most production agents requires tailoring of the agent's system message and architecture to those specific tools (MCP “just” ships the tool, no questions asked, no customizations done, unless you’d go and modify the server’s code base and the prompts). He argues that even with tailored agents and tools, models often fail to call the correct tool (true story on my side, getting LLMs to work well with tools depends on LLM’s parameter size, quality, and many other factors).

I tried to depict this a bit below, it’s like going around with a Swiss army knife, not really knowing when you’d use any of the tools in there, but in times of need, they sorta/kinda do the job! Compare that with walking with a bag of tools (scissors, can opener, the handyman you are!) all around. A Swiss knife is lighter, and it DOES the job (though poorly, and takes a lot of time). On the other hand, having the perfect tool for the job saves time but adds the overhead of optimizing the time and place (you never know!)

Harrison, on the other hand, positions MCP as particularly valuable when you want to integrate tools with an AI agent whose underlying logic you cannot directly control. This opens the door for non-developers to extend the capabilities of their existing AI tooling/clients/agents (e.g., Claude or Cursor) with custom integrations and tools. He put the emphasis here on non-technical users to build agents without needing to code the agent's core logic (no-code/low-code), plug this MCP server and be on your to doing something productive.

Utility for Non-Developers AND Developers

While MCP might not be predictably reliable/predictable (you never how well those tools are tested). For non-developers, they could be just enough to get the job done. In a previous post, we discussed non-consumption and the different types of innovation. Nonconsumption occurs when potential customers are shut out from existing solutions due to barriers like cost, access, complexity, or, crucially, in this context, required technical skill.

Many tools, locked behind APIs requiring code, remain inaccessible to those who could benefit from them but lack programming abilities. LLMs began tackling this by democratizing access to information through a natural language interface. MCP takes it a step further by potentially democratizing action and integrations, allowing a non-developer to plug or even just use existing integrations via an MCP server (abstracting access to their AI client, it bridges the skill gap and turns previously inaccessible tool functionalities into consumable services via a natural language interface). Some examples:

• A marketing manager using Claude could connect an MCP server for their company's Google Analytics account. Without writing code, they could ask Claude, "Summarize the top 5 traffic sources for our latest campaign landing page based on GA data," a task previously requiring manual navigation or analyst support.

• A project manager using Cursor could integrate an MCP server for their team's Jira instance. They could then ask Cursor, "List all open 'bug' tickets assigned to me in the 'Backend' project and summarize their description" accessing project data directly within their coding-assistance tool.

• A small business owner could use an AI assistant connected to an MCP server wrapping their simple customer database (perhaps even a managed Airtable or Google Sheet). They could ask, "Find customers who purchased Product X in the last 6 months but haven't purchased Product Y."

That said, for developers getting started, not really sure what tools are the right ones for the job, experimenting with existing integrations built by someone else and exposed via the MCP server abstraction might not be a terrible idea. Optimization can come later, but as a starting point, for identifying the use-case, and understanding the landscape, it can be quite useful, and who knows, maybe it’s just good enough 🙂

MCP’s Creator Economy & Network Effects

If you have not noticed, there are thousands of MCP servers out there (and there is no stopping), MCP triggered a new creator economy around AI integrations. Developers are building, sharing (in the open), and sometimes even monetizing MCP servers via marketplaces, subscriptions, or sponsorships, creating valuable assets like premium servers for niche/complex tools to generate revenue (like app stores).

MCP’s ecosystem will have Clients (with MCP integrated within like Cursor), Servers (for almost anything has an API), and infrastructure services and platforms, i.e., where to host those servers for scale, security, and who hosts them for me.

The diversity will give birth to new network effects. As more developers and organizations adopt MCP, the number of MCP servers and connectors between AI models and external tools will keep growing. Each new server (e.g., for Slack, GitHub, or a niche blockchain tool) adds net-new value to the ecosystem by expanding the range of tools that MCP-compatible AI clients can access. For example, if a developer builds an MCP server for a CRM, every MCP user gains the ability to integrate their AI with Salesforce without additional effort. I.e., the more participants, the more useful it becomes (“direct network effect”).

Additionally, as major companies integrate MCP, as seen with OpenAI, Block, Apollo, and Microsoft’s Copilot Studio, the protocol will keep gaining credibility and reach. This draws in smaller firms and developers, creating an "indirect network effect." For instance, if Google or AWS builds MCP servers (servers as complements) for their platforms, the protocol’s utility skyrockets, pulling in more users and developers to leverage those integrations and funneling more use of the platform other services (“Indirect network effect”).

On the macro level, MCP provides AI with superpowers by streamlining access to tools and integrations. This drives more and more automation enhancing productivity across various industries and amplifying aggregate economic value.

HOWEVER, this all does not come for free. For MCP to be considered more than just “hype” by enterprises, many will need to invest in learning about how AI systems work and how to implement them reliably. Legacy systems, fragmentation risk, and security concerns (requiring robust permissions) are hurdles, that could potentially slow uptake for those enterprises.

The Future

MCP's long-term impact depends on reaching critical mass, which will depend on how easy/pragmatic it is to adopt for enterprise use-cases. If widely adopted, it could redefine the AI ecosystem, shifting focus from integration plumbing to faster and more accessible innovation. The future is full of surprises and questions:

• How to differentiate?

  • The focus may shift from “the best API design” to “the best collection of discoverable tools” for agents

  • More Specialization? The separation of concerns inherent in MCP (client, server, tools, resources) could lead to specialization among developers. Some might focus on building robust and reliable MCP servers (back to software engineering), while others concentrate on creating user-friendly AI applications (back to good design principles).

• How will pricing structures change? Dynamic, market-driven tool adoption based on agent assessment of speed, cost, and relevance could emerge, favoring modular, high-performing tools over merely popular ones. Also, servers-as-a-service, subscriptions (MCP would add more value, potentially justifying subscription amounts), Usage-Based Pricing (Pay-as-You-Go), outcome-based pricing (MCP makes that more tangible, e.g., pay for a code commit), and other modes of monetization will start to emerge.

• How to design APIs with Tools? Tools and APIs are not 1:1 mapping, a tool can combine multiple API calls (e.g., draft_and_send_email vs. just send_email).

• How and Where to host MCP servers? real-time load balancing across MCP servers, scale, etc (same old same, or is it 🙂)

• ..

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

Introduction

The AI landscape is constantly shifting, in ~3 years we are at “manus” level (standing on the shoulders giants of course). That said, each year, there is the unavoidable question, “Do we still need Retreival Augemented Generation (RAG)?” The answer (still in 2025) is “Absolutely, what are you talking about?!.

But why? The key here is that the definition of RAG evolves with time! In this post, we will explore RAG's modern origins, how RAG is being used and implemented, its most prominent optimizations, and finally answer the core question: "Do we still need RAG?"

Important Taxonomy

Before we proceed, let’s cover two important concepts:

• Parametric Memory: This refers to the knowledge stored within the parameters (weights and biases) of a machine learning model. Knowledge is Implicit, like memorizing facts for an exam. The knowledge is embedded in your brain structure (analogous to the model's parameters).

• Non-Parametric Memory: This refers to knowledge stored outside of the model's parameters, typically in an external database or index. Here is that the structure of the model doesn't change - we are NOT changing parameters but adding additional data the model can access. Knowledge is explicit, the facts are readily available and understandable. This is like having access to a textbook during the exam. The knowledge is stored externally and you can retrieve it as needed.

We will be referring to those in the rest of the post.

Standalone LLMs Handicaps and RAG origins

So why do we need RAG? What problems does RAG solve?

Large Language Models (LLMs) are, at their core, pattern-matchers trained on massive datasets. While impressive, they suffer from key limitations:

• Limited ability to access and precisely manipulate knowledge: While large language models store factual knowledge in their parameters, their ability to access and utilize this knowledge accurately for specific tasks is still constrained.

• Lagging performance on knowledge-intensive tasks: On tasks that heavily rely on external knowledge, the performance of purely parametric models often falls behind.

• Challenges in providing provenance: It is difficult to determine why a parametric model makes a particular prediction, and thus providing evidence or sources for its decisions is an open problem.

• Knowledge Cutt-off: Parametric models cannot easily incorporate new information or revise existing knowledge without undergoing further training. This is referred to as the "knowledge cutoff" and is a barrier to building reliable and up-to-date AI applications.

• Potential for "hallucinations": Parametric models can generate factually incorrect information, which is a significant drawback for knowledge-intensive applications.

To solve the above problems, we need to bridge the gap between the knowledge storage capacity of large language models and the need for precise, up-to-date, and verifiable knowledge in many AI applications.

RAG to the rescue! The original paper for RAG set to address the limitations above. RAG was defined in the paper as a:

“general-purpose fine-tuning recipe for which combine pre-trained parametric and non-parametric memory for language generation"

The main components of RAG as defined in the paper are (as shown below):

• Retriever: This component is responsible for taking an input sequence (query) and retrieving relevant text documents from a non-parametric memory. The retriever consists of two main parts:

  • Query Encoder (q(x)): This is a model that creates a dense vector representation of an input query.

  • Document Index (d(z)): This is a dense vector index of text passages (documents). Each document is represented by a vector. The document encoder and index usually remain unchanged during fine-tuning.

• Generator: This component takes the original input sequence x and the retrieved document(s) z as context and generates the target sequence y.

🧳 Use-case: Customer Support at a Tech Company

Imagine a tech company like Slack deploying an AI-powered support bot. The bot needs to answer queries about the latest features, pricing updates, and troubleshooting steps, i.e., Information that changes frequently and isn’t fully captured in an LLM’s training data. With RAG, the bot retrieves the most recent documentation and support tickets from an internal knowledge base, ensuring responses are accurate and up-to-date, even for features released last week.

RAG and Finetuning: A Good Match

Today, there's sometimes confusion around the terms fine-tuning and RAG. Where Fine-tuning usually refers to adapting a pre-trained LLM (the generator) to a specific task. This is done by adjusting the LLM's internal parameters (weights) – essentially, retraining parts of the LLM on a new dataset. RAG, on the other hand, usually refers to improving how information is retrieved and presented to the LLM, often without changing the LLM's core parameters.

However, the original definition of RAG in the 2020 paper was broader. It envisioned a system where both the “retriever” component and the “generator” component (the model) were fine-tuned together aka 'end-to-end' fine-tuning.

The core of RAG is a general-purpose fine-tuning approach where both the retriever and the generator are trained jointly and end-to-end on downstream NLP tasks. This means that the parameters of the retriever (specifically the query encoder) and the generator are adjusted based on the task-specific data

According to the original paper, RAG is a “general fine-tuning recipe”, “It combines a pre-trained LLM with a retriever that accesses an external knowledge source. The Fine-tuning in the original paper puts the emphasis on the system (retriever + generator), which fine-tuned together so the retriever learns to find documents helpful for the LLM.

Today, fine-tuning often refers to adaptations of a pre-trained LLM (not the system) to a specific task by adjusting its internal parameters (weights), while RAG emphasizes optimizations on the retrieval flow, the store, re-arranging the results, often independent from the generator.

Why did we regress? Not sure!

Tweaking just the retrieval parameters is oftentimes called “naive RAG”, and while this can be useful, the real power comes from further optimizing the retrieval process. This is where retrievers finetuning enters the picture, specifically the finetuning of encoders/embeddings models.

Finetuning an embedding model on in-domain data (e.g., a company’s internal documents) improves retrieval accuracy, and don’t just believe the 2020 paper (which puts the emphasis on tuning retrievers + generators). A recent Databricks blog post titled "Improving Retrieval and RAG with Embedding Model Finetuning" provides reasonable evidence of this, showing substantial gains in retrieval metrics (like Recall@10) on datasets like FinanceBench and ManufactQA after finetuning.

The takeaways?

• Better Embeddings = Better Retrieval: Finetuned embeddings capture the nuances of specific data, leading to more relevant search results.

• Better Retrieval = Better RAG: More relevant context enables the LLM to generate accurate, grounded responses, reducing hallucinations.

• Finetuning Embedding Models Can Outperform Reranking: In many cases, finetuned embeddings match or exceed reranking models, simplifying the RAG pipeline.

The gotchas?

Finetuning embedding models is dataset-dependent: it worked well for some (e.g., FinanceBench) but not all (e.g., Databricks DocsQA). I.e., don’t assume finetuning embeddings will always solve your problems. You need to identify the true bottleneck in your system (retrieval, generation, or something else) and target your efforts accordingly.

Identify your “good enough” and remain there if it gets the job done, or risk diminishing returns from unnecessary spending. Moreover, along with fine-tuning the embedding model, adjusting the generator model (the LLM) can yield further improvements. However, this shouldn’t be your first move, see:

🧳 Use Case: Legal Research for a Law Firm

A law firm needs an AI tool to assist with case law research. Legal documents are dense with jargon and context-specific terms like "tort" or "estoppel" that generic embeddings might misinterpret. By finetuning an embedding model on a corpus of legal briefs, statutes, and case law, the firm’s RAG system retrieves highly relevant precedents for queries like "What are the latest rulings on non-compete clauses in California?" The result? Faster, more accurate research that keeps lawyers ahead of the curve.

Long-Context Models: A Powerful Tool, But Not a Replacement Retrieval/RAG

The rise of LLMs with longer context windows has led some to speculate that RAG might become obsolete. If an LLM can process an entire document, or multiple documents, in its context window, we don’t need to bother with retrieval, right, right?? Wrong!

A recent (close to the date of this writing) Snowflake blog post, "Long-Context Isn’t All You Need: How Retrieval & Chunking Impact Finance RAG" provides a counterpoint. Their research on financial filings shows that retrieval and effective chunking remain essential, even with long-context models. Here’s why:

• Context Confusion: Dumping large amounts of text into an LLM’s context window can overwhelm it with irrelevant information, making it hard to pinpoint what’s crucial for the query. Retrieval acts as a "focusing mechanism."

• The Art of Chunking: How documents are divided into chunks impacts retrieval effectiveness. Moderate chunk sizes, combined with retrieving more chunks, often yield the best results (compared to retrieving large chunks), enhanced by adding LLM-generated global document context.

• Retrieval Quality Trumps Generative Power (Sometimes): A well-tuned retrieval pipeline with smart chunking can enable a smaller LLM to nearly match a larger model (take note here, if you ever want to consume smaller models for Q&A, knowledge retrieval, consider this).

• Efficiency and Scalability: Processing vast text in a single pass is computationally expensive. Optimized RAG offers a more efficient, scalable/cost-efficient solution.

🧳 Use Case: Financial Analysis at an Investment Bank

An investment bank uses an AI system to analyze quarterly earnings reports from multiple companies. A long-context LLM could ingest entire filings, but irrelevant sections (e.g., boilerplate disclaimers) might dilute its focus. With RAG, the system retrieves only the most relevant chunks, say, revenue breakdowns or risk factors, using a chunking strategy tailored to SEC filings. This allows a smaller, finetuned model to deliver precise insights, saving compute costs while matching the accuracy of larger models.

The Rise of Agentic RAG: Beyond Simple Retrieval

Standard RAG involves a single retrieval step before generation. But Agentic RAG, or Retrieval-Augmented Generation with Agents, takes it further. An agent—an LLM-powered component capable of reasoning, planning, and acting, can:

• Interact with Multiple Tools: It chooses the best tool for a query, like a vector database, web search, API, or internal knowledge base.

• Perform Multi-Step Retrieval: It manages complex, multi-turn retrieval processes, refining its strategy as it gathers information.

• Reformulate Queries: If initial results are lacking, it rephrases the query and tries again.

• Validate Retrieved Information: It assesses the quality and relevance of context before generation, reducing hallucinations.

• Integrate with External Systems: It can send emails, access calendars, or perform calculations via APIs.

I liked the analogy provided in this post by weaviate:

Think of it this way: Common (vanilla) RAG is like being at the library (before smartphones existed) to answer a specific question. Agentic RAG, on the other hand, is like having a smartphone in your hand with a web browser, a calculator, your emails, etc.

The Weaviate blog post likens standard RAG to a library (access to information) and Agentic RAG to a smartphone (access plus processing and action). Surveys on Agentic RAG highlight its dynamic retrieval management and adaptability for complex tasks.

🧳 Use Case: Healthcare Decision Support

A hospital deploys an AI assistant to help doctors diagnose rare diseases. A patient’s query: "What’s causing my persistent fever and joint pain?" would require more than a single lookup. An Agentic RAG system searches medical journals, cross-references the patient’s electronic health record via an API, and queries a drug database for side effects. It reformulates the query if initial results are inconclusive, validates findings against recent studies, and suggests a differential diagnosis—all in real time.

In the context of agents, you can think of RAG as a single-step look-up for the agent, where the single step is just doing retrieval/look-up on a static base once. On the other hand, agentic RAG involves multiple steps, the emphasis here is on retrieving the right information to solve for the problem. I.e., its not a one-off

🤔 When to Use Standard RAG vs. Agentic RAG

• Standard RAG is sufficient when:

  • You have a single, well-defined knowledge source (e.g., a company wiki).

  • Queries are simple (e.g., "What’s our return policy?").

  • Cost and latency are critical.

  • Use Case - Standard RAG in E-Commerce: An online retailer uses standard RAG for a chatbot answering "What’s the warranty on this laptop?" The bot retrieves the answer from a product catalog, a single, structured source, keeping responses fast and cheap.

• Agentic RAG is important when:

  • Multiple, diverse sources are needed (e.g., web, internal docs, APIs).

  • Queries are complex (e.g., "Plan a marketing campaign based on competitor analysis").

  • Iterative retrieval and validation are required.

  • Integration with external tools is essential.

  • Use Case - Agentic RAG in Supply Chain Management: A logistics firm needs an AI to optimize shipping routes. The agent queries weather APIs, traffic data, and warehouse inventories, iteratively refining its plan based on real-time constraints, then schedules deliveries via an external system. Standard RAG couldn’t handle this multi-step complexity.

Conclusion: RAG’s Enduring Legacy

RAG (more importantly, retrieval optimization) is not a passing trend. It’s a fundamental architecture for building AI systems that access and reason over information. While LLMs evolve, the need for external knowledge persists (be it static or dynamic).

To summarize, here is why RAG remains essential, even in 2025 and beyond:

• RAG is more than "Naive RAG": The original definition of RAG emphasized end-to-end fine-tuning of both the retriever and the generator. We somehow lost sight of that and kept focusing on retrieval, separately from generation (i.e., fixing the generator LLM mode, aka “naive” RAG). While "naive RAG" (just optimizing retrieval parameters) can be a good starting point, the true power of RAG lies in optimizing the entire system. This includes fine-tuning embedding models for specific domains, employing sophisticated chunking strategies, and even leveraging re-ranking techniques.

• Fine-tuning Matters: Fine-tuning embedding models on in-domain data significantly boosts retrieval accuracy, leading to more relevant context for the LLM. This, in turn, improves response quality and reduces hallucinations. The benefits of fine-tuning are not universal, however, and careful evaluation is necessary to determine its effectiveness for a given dataset and task.

• Long-Context Models are Not a Silver Bullet: While long-context LLMs can process larger amounts of text, they don't eliminate the need for RAG. Retrieval acts as a crucial "focusing mechanism," preventing the LLM from being overwhelmed by irrelevant information. Strategic chunking and optimized retrieval can even enable smaller, more efficient LLMs to match the performance of larger models on certain tasks.

• Agentic RAG Extends RAG's Capabilities: Agentic RAG elevates RAG beyond simple retrieval. By incorporating AI agents, RAG systems can dynamically manage multi-step retrieval processes, interact with various tools (databases, APIs, web search), reformulate queries, validate retrieved information, and even take actions. This makes RAG suitable for complex, real-world scenarios that require more than a single information lookup.

• Choosing the Right RAG Approach: Standard RAG is often sufficient for simpler tasks with well-defined knowledge sources. Agentic RAG shines in situations requiring complex reasoning, multi-source integration, and interaction with external systems. The choice depends on the specific application's needs and constraints.

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

Understand the Foundation: AI's Historical Context and the Lessons of Non-Consumption

“One age cannot be completely understood if all the others are not understood. The song of history can only be sung as a whole” – José Ortega y Gasset

We started with history. It's important to note that AI has gone through phases of progress and stagnation, commonly known as "AI Winters". In my view, a key cause of these winters was non-consumption, a concept popularized by Clayton Christensen, which describes how promising tech struggles to achieve widespread acceptance (and consumption)

Why did AI Non-Consumption Occur in the past?

AI non-consumption occurs when potential users are unable to effectively utilize AI technologies due to various limitations. Some of the limitations we had in the since the dawn of AI (50s,60s,80s,…) are:

• Accessibility Issues: AI technologies, especially in their early stages, have not been widely accessible, often limited to a select group of researchers or well-funded institutions. A lack of internet access (DARPA was special) and infrastructure restricted the reach of AI, and made it unavailable to a large portion of the population.

  • Data was scarce, limited the ability to train effective AI models and validate neural ideas :)

  • Compute was, well, slow. Also, compared to today, there was a lack of hardware and computing infrastructure, which also limited accessibility.

• Unsuitable Offerings: AI products, such as expert systems in the past, have sometimes failed to meet the needs of users, either by being too complex (and expensive) or not being able to handle novel situations (they were largely rule-based).

• Negative Consumption Gap: Expectations of what AI can do exceeded what was delivered. This led to periods of disillusionment, aka AI winters.

It’s “Spring” but let’s not Jinx it!

Luckily, we're in a new "AI Spring" driven by pervasive internet access, big data, training/inference innovations, and powerful hardware. Unlike previous AI booms, we're seeing the evolution of language models, multi-modality, and their incarnations as agents, which are driving practical applications across industries with tangible productivity gains across different use-cases (e.g., coding, creative, etc.). That said, challenges remain:

• Compute Access: Training advanced AI is expensive, concentrating power in the hands of a few.

• AI Winter Risk: Overhype could lead to disillusionment and a market downturn.

• Non-consumption: AI remains inaccessible or unaffordable in many areas.

Despite progress, we should continue to heed and learn from history and ensure AI becomes more accessible (e.g., via open-source AI) to unlock its full potential.

Obsess with your Customer

To truly unlock AI's potential and avoid the pitfalls of the past, customer obsession, as championed by Jeff Bezos, can be a guiding principle (amongst many). This means prioritizing customer needs and pain points above all else. We must build with and for the customer at every stage.

Key Tenets of Customer Obsession:

• Technology as an Enabler: Employ technology strategically to solve customer problems and deliver innovative solutions, not as an end in itself.

• Relentless Execution: Build the necessary infrastructure and teams to deliver on the customer-centric vision effectively.

• Long-Term Vision: Cultivate customer loyalty and trust by consistently delivering value and balancing long-term goals with short-term gains.

• Proactive Problem Solving: Anticipate needs and exceed expectations.

Focus on Experiences and Systems NOT on Tech Standalone!

By adopting a customer-first mindset, we recognize that a successful AI product is more than just a powerful model. It requires a holistic systems approach that considers the entire user experience.

A. Apply Adapted Whole Product Framework:

Inspired by Geoffrey Moore's "Crossing the Chasm" this framework emphasizes building comprehensive AI solutions that are not just technically sound, but also user-friendly and well-supported:

• Core/Generic Product: The fundamental technology (e.g., an LLM).

• Enablers: Additional components that make the core product functional and valuable (e.g., data pipelines, user interfaces, APIs). Represented as modular, petal-like structures.

• Differentiated Product Layer: Unique value propositions that set the product apart (e.g., strong community, unique data, strategic partnerships). These form the product's "Compound MOATs".

• Constraints: Customer/use-case specific needs that influence the prioritization of enablers and differentiators (e.g., enterprise customers prioritizing security).

B. Think about your AI as a System

To deliver on customer needs, we will have to utilize available tools (the best for the job) which would involve not just the standalone models (e.g., the LLMs), but interconnected (compound) AI systems. These "compound" AI systems may comprise of:

• Data Pipelines: For acquiring, processing, and storing the data that powers the AI.

• Knowledge Bases: Curated datasets providing context and enriching the AI's understanding.

• Retrieval patterns/techniques (e.g., RAG): Enhancing context for language models, allowing them to access and utilize relevant information.

• Intelligent Agents: Autonomous systems for complex tasks, extending the capabilities of the AI beyond simple input-output interactions.

• User Interfaces: Intuitive interfaces that make the power of AI accessible and easy to use, bridging the gap between complex technology and user needs.

• Robust Infrastructure: For model training, deployment, and operation, ensuring reliability and scalability.

• Operations: To monitor, maintain, and secure the system, guaranteeing its continued performance and security.

• ...

C. Define Your Constraints

With a clear understanding of the customer and the need for a holistic approach (and the tech-stack), we can now turn our attention to the design process itself. Constraints are the guiding dimensions for building toward your customers and use case. They provide focus and direction, helping us to channel our efforts toward what truly matters. Try to answer the following questions:

• What problem are you really trying to solve? Understand the root cause and the specific pain points.

• What are the non-negotiables? Identify performance requirements, budget limitations, or compliance issues.

• What matters most to your target audience? Are you optimizing for safety, speed, cost, or sustainability?

When you define/contemplate your constraints, you acquire clarity on which core product features, enablers, and differentiators to prioritize.

Dig Deeper and Optimize When you Have to!

Of course, the underlying technology still matters. An understanding of model training, architecture, and optimization is important for building tailored and effective AI systems that can deliver on the promises made to the customer.

• Pre-training: Initial training on massive datasets to provide a broad foundation.

• Optimizations:

  • Prompt Engineering/RAG: remember the model standalone will have its limits, know them and build around it.

  • Fine-tuning: adapt pre-trained models to specific tasks/constraints, employ available techniques e.g., LoRA for efficiency.

  • Efficiency Optimization: Employing methods like quantization to reduce operational costs and environmental impact.

  • …

Play Strategically, Understand the Market

With a solid product foundation, it's time to consider the broader market landscape. The AI market can be understood through two key dimensions:

• Open vs. Closed Models: Publicly accessible and modifiable (open) versus proprietary and controlled (closed).

• Direct vs. Indirect Business Models: AI as the primary product (direct) versus AI enhancing existing products and services (indirect).

Key Market Players:

• New Entrants (e.g., Mistral, OpenAI, Anthropic): Mostly (with some exceptions) pursuing direct sales of their advanced AI models (now systems), often through APIs (subscriptions, platforming).

• Incumbents (e.g., Meta, Google, Microsoft): Mostly (with some exceptions) leveraging AI to augment and enhance their existing, vast product portfolios (office 365, bing, search, you name it).

Understand these dynamics, position yourself where it makes sense.

Build Strong Foundations with a Solid Platform

In this dynamic market, platform engineering still plays an important role in accelerating the development and deployment of AI systems. Well-designed platforms provide the foundation for building good to great AI products, a dimension that is underrated by many.

Key Considerations for building a great AI Platform:

• Purposeful Design: Align the platform with core business objectives and delivering specific value (yep I keep repeating this).

• User-Focused Design: prioritize simplicity, ease of use, and incorporate continuous feedback.

• Well-defined APIs: expose platform functionality through APIs tailored to specific AI personas.

• Platform vs. Product: Don’t get lost in designing the perfect the platform, keep the core value in mind (what your shipping not only what you are shipping with)

Keep the Offence, but think about Defense (and MOATs)

As the AI market matures, you should start thinking about your competitive advantage, i.e., what’s your "moat"?. This will require deep thought about how you could differentiate. Here are a few ideas and strategies for building your “moat”:

• Strong Community & User Engagement: Foster a loyal and engaged user base to create powerful network effects.

• Purpose-Built Applications: Focus on solving specific problems and addressing niche needs to achieve natural differentiation.

• Value Beyond the Model: Invest in research, build proprietary datasets ("data moats"), and continue to refine the product based on user feedback.

• Differentiate at Multiple Layers: Be it building superior AI models, building highly efficient infrastructure, or crafting the “best” interfaces.

• Compound MOATs: Combine many of those to build strong moats

• …

Build Responsibly, Safely, and Securely

Finally, as you build powerful AI systems, you’ll have to consider the safety/ethical implications. Strive to build RGAF AI:

• Transparent: Understandable and explainable in its decision-making processes.

• Fair: Free from bias and discrimination, ensuring equitable outcomes.

• Accountable: With clear lines of responsibility for its actions and impact.

• Beneficial: Designed to promote human well-being and societal good.

• …

There is also great potential (read the full post above) for using AI to better safety/security. It’s always a double-edged sword make sure to weild it responsible (and gain from it at the same time!)

Outlook for 2025, Sayonara 2024!

That’s it for 2024; it was an exciting year for AI and for myself. Alot is going on, and alot will go on in 2025. Here are some 2025ish thoughts:

• More Agentic: we probably will see more pragmatic use-cases here and new patterns that make agentic easier to use/integrate in various products and be applicable to many more use-cases. This will drive efficiency up and (hopefully) help reduce costs while keeping services up and running longer (without human intervention).

• More RAG: Hallucinations, content freshness, accuracy, etc., are not solved natively (yet), so we will still need RAG (and fine-tuning) to make AI systems more usable/practical.

• More Inference (and reasoning): Training might still wear the crown, but inference optimizations (and test-time compute) are coming for training in 2025.

• More Multimodal AI: Maturing modalities of not just vision/images but also videos and faster transitions between all modalities.

• Hyper-Personalization: We will probably see AI being used to build customized experiences for all of us. For businesses, this might translate to customer loyalty, higher conversions, and potentially a newly-found competitive edge (by just reusing existing data).

• AI Productization/Operations: AI models/agents/systems, are all the “North” layers, we will see folks get more serious about the practical aspects of deploying, managing, and scaling AI systems in real-world environments (the “south” layers). .

• AI at the Edge, On-Device, and Distributed Learning: we probably will see more progress toward federated learning, which allows AI models to be trained across multiple devices without sharing sensitive data.

• Responsible and Ethical AI: In 2024, we saw trustyworthy/responsible AI blends, in 2025 we will potentially see more applications of it (if you are interested see this post which I authored through the LF).

• Multimodality to Commands, more iRobots: better perception of the world through modalities, better action execution, equal smarter more usable real-world robots that can DO THINGs!

• Quantum AI? Well, I’ll stop here, but I’d expect big announcements/discoveries in that space (like willow) getting us a step closer to seriously impactful applications of AI.

With all that musing above, do stay tuned for more writing and reach out if you’d like to collaborate (or want to tailor my writing to a specific topic) 🙂

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

The Accelerated Ripple Effect and Hidden Tech Debt of Generative AI

Traditional AI models followed a predictable pipeline and still had a hidden technical debt. Now, with the rise of generative AI models like large language models (LLMs), the debt has been exacerbated. Alongside these models, there are additional layers, such as data retrieval, embeddings, safety, governance, etc. All of which need to be integrated into the AI/ML life cycle. Below are a few examples:

• Data Management: predictive models are typically trained on structured, labeled datasets, focusing on historical data to predict future outcomes. Generative models use larger, more diverse datasets, often unstructured, to learn patterns and generate novel content. Managing this data for quality, bias, and provenance presents a greater challenge.

• Model Training (and fine-tuning): predictive training optimizes specific metrics like accuracy or precision.  The process is generally more straightforward and less computationally intensive than generative models. Training LLMs (emphasis on L) needs more computational resources and longer training pipelines. Continuous learning, fine-tuning, and techniques like Reinforcement Learning from Human Feedback (RLHF) are often necessary (see this post for more details).

• Deployment:  predictive model deployment typically involves integrating the model into an existing application or system for real-time or batch predictions. Deploying LLMs at scale adds more challenges, be it low-latency inference, high availability, or managing fluctuating workloads, which requires specialized infrastructure (compute, storage, and networking) and inference optimization techniques.

With more components in generative AI (especially with retrieval involved)  comes more changes. Changes in one area, such as the training data or model architecture, can trigger cascading effects across the entire system, impacting everything from retrieval effectiveness to monitoring metrics (Change one thing, changes everything or CACE++).  

A Longer AI Lifecycle

More components lead to a denser lifecycle. Each step in the lifecycle encompasses more steps. Let’s explore those a bit:

Data-centric tasks:

Generative AI introduces new stages and expands existing ones to handle the unique data requirements of complex AI systems.

• Synthetic Data Generation (SDG): Traditional ML primarily focuses on cleaning and transforming real-world data. Generative AI adds the step of generating synthetic data (SDG) to augment existing datasets, especially in cases with limited or sensitive data (usually necessarily for training foundational models).

• Data Collection & Management: Traditional ML pipelines typically deal with structured data and simpler data management processes. Generative AI requires managing diverse data sources like synthetic data, embeddings, and external knowledge bases, demanding more sophisticated data management strategies. 

• Embedding Management:  Traditional ML relies heavily on manual feature engineering. Generative AI introduces embedding management as a core component, using techniques like pre-trained language models (e.g., BERT) to generate vector representations of data. Storing these embeddings in specialized (vector) databases enables semantic similarity searches and allows models to understand relationships between data points, moving beyond manual feature creation.

Model-Centric tasks:

Generative AI introduces new approaches to model training and customization to cater to the advanced capabilities of modern AI systems.

• Model Training: While traditional ML trains task-specific models on static datasets, Generative AI leverages the power of transfer learning by pretraining foundation models on massive datasets and then fine-tuning them for specific tasks. For instance, instead of training a chatbot from scratch, developers can fine-tune a pre-trained GPT-4 language model on a dataset of customer service conversations to achieve better conversational abilities.

• Model Customization & Alignment: Traditional ML typically evaluates model performance on held-out data. Generative AI emphasizes continuous fine-tuning and alignment with human feedback, ensuring models remain relevant and adapt to evolving data and user needs. 

AI System Deployment & Orchestration tasks:

Generative AI tackles the challenges of deploying and managing AI systems with multiple interdependent components. Traditional ML primarily deploys single models with fixed logic. Generative AI focuses on deploying and managing compound AI systems with multiple interacting components, including LLMs, other AI models, agents, and APIs. This requires special orchestration and integration techniques. 

Safety & Monitoring tasks:

Generative AI models are unpredictable, which gives rise to the need for new strategies for ensuring responsible AI (guardrails for outputs) and maintaining high performance in compound systems:

• Guardrails Implementation: Traditional ML focuses on monitoring model performance for degradation. Generative AI extends this by implementing guardrails and safety mechanisms that prevent unsafe, biased, or inaccurate outputs (especially when there is no transparency on how models were trained and to account for model emergent behavior). This involves incorporating ethical considerations and regulatory compliance into the development process. 

• Real-Time Retrieval Integration: While traditional ML models rely solely on training data, Generative AI applications more often rely on retrieval integration, allowing AI systems to access external knowledge bases during inference. This ensures the system can provide up-to-date responses and adapt to new information without requiring full retraining. Examples include a customer support chatbot retrieving the latest product specifications from a knowledge base during a conversation or a legal research tool retrieving relevant case law from a legal database in real-time.

• Monitoring & Feedback: Traditional ML monitors basic performance metrics. Generative AI expands monitoring to include human interactions, model outputs, and retrieval performance for comprehensive system evaluation and improvement. Human feedback plays a crucial role in ensuring responsible AI. This could involve collecting user feedback on the helpfulness of a chatbot's responses, analyzing conversation logs to identify patterns of misunderstanding, and using this feedback to improve the chatbot's training data and fine-tune its language model.

• Prompt Tuning and Optimization: Model behavior is usually fixed after training in traditional ML. Generative AI introduces the concept of prompt engineering, tuning, and optimization, which is particularly relevant for LLMs. This involves refining model behavior by optimizing the initial prompts provided to the model based on real-world feedback, enabling continuous adaptation without requiring full retraining. That said, I doubt prompt engineering is a long-term solution (see this post for more details)

The denser AI lifecycle introduces more optimization knobs and more questions to be answered. How much data should be synthesized?  How elaborate should the guardrails be?  Organizations must weigh the marginal benefits of each additional unit of effort (e.g., another data point, a more refined prompt) against the marginal costs.

Emerging Roles & Disciplines Beyond MLOps

More components lead to new roles and disciplines.  Machine Learning Operations (MLOps) has provided a solid foundation for operationalizing traditional machine learning models, but it falls short of addressing the challenges posed by generative AI systems. This entices the creation of new operational disciplines like LLMOps and GenAIOps, and the rise of new personas such as the "AI engineer”,  responsible for tweaking the many knobs of AI applications to ensure that applications meet their specific use case requirements. 

But before discussing AI engineering, let’s try to understand the distinctions between various “ops” disciplines, starting with MLOps. 

• MLOps: Focuses on the end-to-end lifecycle of traditional machine learning models, including data preparation, model training, evaluation, deployment, and monitoring. MLOps emphasizes automation, reproducibility, and scalability for individual models. 

• LLMOps: can be thought of as a branch of MLOps that deals specifically with the operational challenges of LLMs. LLMOps address aspects like prompt engineering, fine-tuning, model alignment, and managing the unique infrastructure requirements of LLMs. 

• GenAIOps: Broader than LLMOps, GenAIOps encompasses the operationalization of all generative AI models, including LLMs, image generation models, and others. It focuses on managing the complexities of training, deploying, and monitoring generative AI systems.GenAIOps addresses the challenges of deploying and scaling generative models, which often require specialized hardware (e.g., GPUs) and efficient resource allocation.

• RagOps:  Focuses specifically on the operational aspects of Retrieval Augmented Generation (RAG) systems, which combine LLMs with external knowledge sources to improve accuracy and grounding. RagOps addresses challenges like knowledge base management, retrieval model optimization, and ensuring consistency between the LLM and the retrieved information. 

• AI Engineering: This represents the next stage in the evolution of AI operations. AI Engineering entails managing the entire lifecycle of compound AI systems that may include LLMs, other AI models (including generative models), retrieval mechanisms (like those used in RAG), external knowledge bases, and human-in-the-loop components. AI Engineering emphasizes system-level considerations, including orchestration, integration, safety, and continuous learning.

Navigating the New Found Challenges of Generative AI

Generative AI, particularly LLMs and multi-modal models, are reshaping the tech landscape and introducing new use cases.  We are moving beyond static predictive models to a dynamic world where AI can generate novel content, engage in conversations, and even drive decision-making processes.  This transition, however, is not “free”. 

The complexity of those generative AI systems increases the technical debt of traditional AI, introducing new layers of complexity related to data management, model training, deployment, safety, guardrails, and more.  Organizations looking to capitalize on the promise of generative AI should approach this paradigm with a broader perspective, acknowledging both the opportunities and the costs and complexities involved.

One of the not-so-obvious changes is the expansion of the AI lifecycle, be it around the data, the model, or the deployment machinery. Each of these expanded stages introduces new "knobs" to tune and optimize, creating more overhead for AI practitioners. This overhead also gives rise to specialized roles and personas like the AI engineer, who is now responsible for managing this complexity.

Before we close out, here are some thoughts to approach this new shift: 

Think Big Picture:  Before jumping on the generative AI bandwagon, consider a cost-benefit analysis to account for the hidden costs associated with new requirements such as data management (e.g., SDG),  specialized infrastructure, and the need for new talent (or skills, i.e., bigger training budges). Be willing to adapt or abandon existing infrastructure that is no longer suitable for generative AI to unlock new capabilities faster.

Build the right abstractions (and Platforms): Consider building customizable AI platforms that can reduce the generative AI tech debt and reduce time to value when building AI applications (usually through well-defined abstractions, interfaces, and APIs). Here are some quick ideas:

• Built-in prompt management: Prompts are artifacts. Make it easier to experiment with new prompts, share versions and distribute them. 

• Automating training: Implement automated workflows for data preparation, hyperparameter optimization, and model selection. Includes pre-trained models.

• Simplify Evals: provide a dashboard with performance metrics, reporting, and alerting for new models.

• Simplify Deployment: provide APIs to automate  AI system deployment to various environments (with pre-built templates).

• Surface alerts and monitoring dashboards: make it easier to instrument AI applications for monitoring and logging, and simplify the configuration alerts.

• Build responsibly: integrate security policies, add access control, and audit knobs.

• Tailor to the needs: Prioritize value and tailor the platform to the specific needs of different user personas. For example, you can start by offering tiered access and functionalities. 

Cultivate a Culture of Continuous Learning: Generative AI is constantly evolving.  Invest in training and development for your employees, encourage knowledge sharing, and stay informed about the latest advancements to ensure your organization remains at the forefront of this transformative technology.

By far, Generative AI is the fastest-evolving technology I have encountered in my professional life. It's not merely a new tool, it’s a paradigm shift that is forcing organizations to reassess their approach to existing use-cases. It also encourages them to consider how AI can accelerate their business by discovering new workflows, and use-cases.

For more on building AI products, read my previous post on how to model value through the adapted whole-product framework:

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

One recent example is prompt engineering, a valuable HACK I'd say, considering the limitations of the initial versions of LLMs, e.g., data limitations, and how they were trained. Prompt engineering was (is) a stop-gap to formulate prompts/queries in a way that the LLM would understand them, similar to how they were trained and (instruction) tuned. 𝗜.𝗲., 𝗰𝗮𝗻 𝘄𝗲 𝗰𝗿𝗲𝗮𝘁𝗲 𝗽𝗿𝗼𝗺𝗽𝘁𝘀 𝘀𝗼 𝗰𝗹𝗼𝘀𝗲 𝘁𝗼 𝘁𝗵𝗼𝘀𝗲 𝗶𝗻𝘀𝘁𝗿𝘂𝗰𝘁𝗶𝗼𝗻𝘀 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹 𝘄𝗮𝘀 𝘁𝗿𝗮𝗶𝗻𝗲𝗱 𝗼𝗻 𝘁𝗼 𝗿𝗲𝗮𝗹𝗶𝘇𝗲 𝘁𝗵𝗲 𝗯𝗲𝘀𝘁 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲?

Don’t get me wrong, we need prompt engineering today because we are not in the desired state, where, ideally, a foundation model would comprehend the prompt regardless of how it was formatted or chained. From a usability standpoint, I'd consider it a UX bug. The question that I think we should be asking is, how do we get to that desired state? Not how do we prompt engineer better, especially in the long term.

Think about it, talking to a child, you'd have to formulate your communication intent to fit in their limited vocabulary and comprehension of the world versus talking to an educated adult who shares as much context with you and have a wider understanding of the world around them. You'd then use words without engineering them to communicate efficiently.

If I'd model that to a concept, I'd use Shannon's information theory and attribute the need for prompt engineering to channel noise. This noise can be a result of inadequate training data, suboptimal model architecture, or anyother limitation.

The question here is, how to reduce noise to elimiate the need to engineer Human language. Imagine you’re asking a friend for help. You’d typically be direct and informal: “Hey, I could really use some help with this.” You wouldn’t be saying, “You’re a great friend, you’re supposed to help me, here’s exactly how you can help…” because that level of detail isn’t necessary. Your friend understands your context and the nature of your request without needing you to spell it all out.

I don't see prompt engineering going away completely (not in the short-term), but at least for foundational models, we shouldn't have to rely on it as a pillar technique and acknowledge it as a hack, and put efforts in engineering and modelling beyond the prompt layer. It's not good UX.

TL;DR:

• Hacks can become permanent, so be wary of prompt engineering solidifying as a standard practice.

• Prompt engineering is a valuable but temporary fix for current LLM limitations.

• Ideal language models understand human language regardless of prompt formatting – no engineering needed.

• Reliance on prompt engineering highlights a UX flaw: models aren't robust enough.

• Instead of refining prompt engineering, focus on reducing "noise" (limitations in data, architecture, etc.).

• Like talking to a child vs. an adult, we want models to understand implicit context, not engineered prompts.

• Shannon's Information Theory: prompt engineering is needed due to channel noise, which we must reduce.

• Prompt engineering may remain relevant in niche areas, but not for foundational models.

• Invest in engineering and modeling beyond the prompt layer for truly robust and user-friendly language AI.

• Good UX means natural interaction – let's move beyond prompt engineering for the future of language models.

  Thanks for reading The Technomist! Subscribe for free to receive new posts and support my work.

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

This convergence is happening amidst growth in the global AI market, projected to reach $826 billion by 2030 from $196.63 billion in 2023 [Markets & Markets]. Within this growing market, the AI cybersecurity portion of it is an expanding segment, anticipated to reach $134 billion by 2030 from $24.3 billion in 2023 [Statista].

Below is an executive summary.

Key Drivers:

• Generative AI Surge: Generative AI is rapidly gaining traction, impacting various functions within organizations and fueling demand for AI-powered applications. [Gartner Forecast Analysis]

• Widespread Adoption: A large majority (84%) of organizations leverage AI-based cybersecurity tools, with 75% focusing on network security [World Economic Forum, Statista].

• Generative AI Spending Surge: Expenditure on generative AI solutions is expected to skyrocket, accounting for 35% of total AI spending by 2027, up from a mere 8% in 2023 [Gartner].

• Booming Market: The AI cybersecurity market is poised for exponential growth, projected to reach a staggering $134 billion by 2030, reflecting a dramatic increase from its 2023 valuation of $24.3 billion [Statista].

• Rapid CAGR: This translates to a remarkable compound annual growth rate (CAGR) of 22.3% from 2023 to 2033 [Market.us], driven by escalating cyber threats, widespread cloud adoption, and the demand for proactive and scalable security solutions.

• AI Software Spending Boom: Global spending on AI software is projected to reach $297.9 billion by 2027, with generative AI software expenditure increasing significantly. [Gartner Forecast Analysis]

• Expansion of the Attack Surface: Cloud, growth in the number of IoT devices has expanded the attack surface, creating new vulnerabilities that require AI-driven security measures. [Market.us]

• Growing Awareness and Investment: Organizations are increasingly recognizing the importance of AI in cybersecurity, leading to increased investments in research, development, and deployment of AI-powered security tools. [World Economic Forum]

• Cybersecurity Talent Gap: The cybersecurity industry faces a significant talent shortage, highlighting the need for AI-powered solutions to automate tasks and augment human expertise. [World Economic Forum]

• Regulatory Landscape: Cybersecurity and privacy regulations are becoming increasingly stringent, influencing the development and adoption of AI-powered security solutions. [World Economic Forum]

• Cybersecurity Talent Shortage: The cybersecurity industry faces a significant talent gap, making it difficult for organizations to effectively manage security operations. AI can help bridge this gap by automating tasks, providing insights, and augmenting human expertise. [World Economic Forum]

• Collaborations on Trustworth AI: focus on advancing safe, secure, and trustworthy AI, aligning regulatory frameworks, promoting ethical AI research, and broadening cooperation in AI protection and cybersecurity. [UAE and US Collaboration on AI ]

  • The U.S. and the UAE reaffirm their commitment to advancing safe, secure, and trustworthy artificial intelligence (AI) technologies.

  • Key principles include fostering international AI frameworks, aligning regulatory frameworks, and promoting ethical AI research and development.

  • The collaboration will focus on advancing safe, secure, and trustworthy AI, aligning regulatory frameworks, promoting ethical AI research, and broadening cooperation in AI protection and cybersecurity.

  • President Biden and President H.H. Sheikh Mohamed bin Zayed Al Nahyan will oversee the development of the memorandum of understanding.

  • The U.S. and the UAE are committed to deepening collaboration in AI and related technologies for a more prosperous and secure future.

Key Trends

• Integration of AI into Enterprise Applications: Over 70% of independent software vendors are expected to integrate generative AI capabilities into their enterprise applications by 2026. [Gartner Forecast Analysis]

• AI Adoption Phases: Organizations are at various stages of AI adoption, with many still in the experimentation or planning phases. [Gartner Forecast Analysis]

• Generative AI's Impact on Security: While generative AI offers benefits for content creation and automation, it also presents new security risks, such as the generation of malicious code and sophisticated phishing attacks. Organizations must adapt their security strategies to address these emerging threats. [CBH]

• Explainable AI for Enhanced Trust: The lack of transparency in some AI algorithms can hinder trust and adoption. The development of explainable AI, which provides insights into how AI systems make decisions, is crucial for building confidence in AI-driven security solutions. [Gartner Forecast Analysis]

• Regulatory Landscape: Evolving cybersecurity and privacy regulations, such as GDPR and CCPA, are influencing the development and deployment of AI-powered security solutions, emphasizing data protection and responsible AI practices. [World Economic Forum]

Opportunities:

• Development of Advanced Threat Detection and Response Solutions: AI can significantly enhance threat detection accuracy, accelerate incident response times, and minimize the impact of cyberattacks.

• Proactive Security Measures: AI enables predictive analytics and threat intelligence, allowing organizations to identify and address potential vulnerabilities before they are exploited proactively. [Markets & Markets]

• Automated Security Operations: AI can automate routine security tasks, freeing up security professionals to focus on more complex and strategic initiatives. [Gartner Forecast Analysis]

• Enhanced Security for Cloud and IoT Environments: AI-powered solutions can address the unique security challenges posed by cloud computing and the growing number of connected devices. [Market.us

Challenges:

• Ethical Concerns and Bias: AI algorithms can be biased, raising ethical concerns regarding fairness and accountability. [CBH]

• Adversarial AI: Attackers can leverage AI to develop sophisticated attacks, posing new challenges for cybersecurity professionals. 

• Data Privacy and Security: AI systems rely on vast amounts of data, raising concerns about data privacy and security. [CBH]

• Explainability and Transparency: Lack of transparency in AI algorithms can hinder trust and adoption [Gartner Forecast Analysis]

Conclusion:

The AI security market is experiencing rapid evolution, driven by the need for solutions to counter rising cyber threats. With power comes great responsibility though, while AI offers potential for enhancing security, organizations must also address the risks associated with its malicious use. 

• Enhancing Security: AI offers significant potential for bolstering cybersecurity defenses, with organizations increasingly employing it for:

• Automating routine tasks: Freeing up human analysts for more complex threats.

• Real-time threat detection: Enabling rapid response and mitigation.

• Predictive analytics: Proactively identifying and addressing vulnerabilities.

• Emerging Risks: However, AI also presents new challenges as adversaries exploit it for malicious purposes:

  • Phishing attacks.

  • Automated malware development.

Related musing here:

Going in Details

AI Market Statista

Reference: Artificial Intelligence - Global | Statista Market Forecast 

Important Technologies Impacting Market Growth: 

• Machine Learning: Forms the foundation of the market and exhibits consistent growth throughout the period.

• Autonomous & Sensor Technology: Shows steady growth, gaining momentum towards the latter half of the projection period.

• Natural Language Processing: Experiences significant growth, becoming a major contributor to the overall market size by 2030.

• Computer Vision: Demonstrates strong and consistent growth, becoming the second largest segment by 2030.

• AI Robotics: While starting with a smaller market size, it exhibits rapid growth, becoming a prominent segment by 2030.

Key Observations:

• The overall AI market is projected to experience high growth, reaching nearly 800 billion USD by 2030.

• Machine Learning remains a dominant segment, while Computer Vision and AI Robotics emerge as important growth drivers.

• Natural Language Processing also shows notable growth, increasing its importance in various applications.

As for the top 10 countries, this bar chart depicts the estimated market size of the AI market in the top 10 countries for 2030.

• United States: Dominates the market with a projected size of 223.7 billion USD, significantly larger than any other country. The US remains the clear leader in the AI market, driven by factors like technological advancements, research and development investments, and a thriving tech ecosystem.

• China: Ranks second with a market size significantly smaller than the US but considerably larger than the remaining countries.it can be considered a strong contender, though with export controls

• Germany, France, Russia, Brazil, etc.: Exhibit relatively smaller market sizes than the US and China.

Gartner Forecast Analysis

Reference: Forecast Analysis: AI Software Market by Vertical Industry, 2023-2027 

AI software spending will grow to $297.9 billion by 2027. Over the next five years, market growth will accelerate from 17.8% to 20.4% in 2027, with a 19.1% CAGR. Government has the largest spend of over $70 million by 2027, but oil and gas is growing fastest with a 25.2% CAGR.

McKinsey: GenAI Adoption Spikes

Reference: The state of AI in early 2024: Gen AI adoption spikes and starts to generate value 

• Gen AI Adoption Surge: 65% of organizations regularly use gen AI, a significant increase from the previous year.

• Common Use Cases: Gen AI adoption is most prevalent in marketing, sales, product development, and IT, with an average of two functions utilizing it.

• Overall AI Adoption Growth: 72% of organizations now utilize AI, up from 50% previously, with increased adoption across various industries.

• Increased AI Investments: Organizations are allocating budgets for both gen AI and analytical AI, expecting further investment growth over the next three years.

• Gen AI Risks: 44% of respondents reported negative consequences from gen AI, including inaccuracy, cybersecurity concerns, and explainability issues, highlighting the need for risk mitigation.

• High-Performer Success: Organizations effectively leveraging gen AI attribute a significant portion of their EBIT to its deployment, often utilizing it across multiple functions and implementing risk best practices.

• High-Performer Challenges: These organizations also face challenges with data governance, integration, and operating models, emphasizing the importance of addressing these for successful gen AI adoption.

• Diverse Research Sample: The research involved a diverse group of participants across various regions, industries, company sizes, and functional specialties, providing valuable insights into the current AI landscape.

StackOverflow Developer Survey 

Reference: AI | 2024 Stack Overflow Developer Survey 

Similar to last year, developers remain split on whether they trust AI output: 43% feel good about AI accuracy and 31% are skeptical. Developers learning to code are trusting AI accuracy more than their professional counterparts (49% vs. 42%).

Professional developers agree the issue is not user error: twice as many professionals cite lack of trust or understanding the codebase as the top challenge of AI tools compared to proper training.

Market.us

Reference: AI In Cybersecurity Market Size, Share | CAGR of 22.3% 

• The Global AI In Cybersecurity Market size is expected to be worth around USD 163.0 Billion by 2033, from USD 22 Billion in 2023, growing at a CAGR of 22.3% during the forecast period from 2024 to 2033.

• AI in cybersecurity refers to the application of artificial intelligence technologies in detecting, preventing, and mitigating cyber threats and attacks. It involves using machine learning, natural language processing, and other AI techniques to analyze vast amounts of data, identify patterns, and detect anomalies that may indicate potential security breaches or malicious activities.

• The AI in cybersecurity market is rapidly growing as organizations recognize the need for advanced solutions to combat the increasing complexity and sophistication of cyber threats. AI technologies can enhance traditional cybersecurity measures by providing real-time threat detection, automated incident response, and predictive analytics to proactively identify and address potential vulnerabilities.

• AI in cybersecurity encompasses a range of applications, including malware detection, network intrusion detection, user behavior analytics, and threat intelligence. By leveraging AI algorithms, cybersecurity systems can continuously learn from new data and adapt to evolving threats, improving their accuracy and effectiveness in detecting and mitigating security incidents.

• The market for AI in cybersecurity is driven by factors such as the growing frequency and severity of cyberattacks, the prevalence of cloud,  IoT in the past decade, and the need for scalable and intelligent security solutions. Vendors and emerging startups are investing in AI capabilities to develop innovative products and services that address the evolving threat landscape.

• According to a report by Security Intelligence, the average total cost of a data breach in 2022 rose to ~$4.35 million, representing a modest increase of 2.6% compared to the previous year’s average of ~$4.24 million. This upward trend highlights the growing financial impact that cyberattacks can have on organizations worldwide.

• European Union Agency for Cybersecurity (ENISA) observed a noteworthy surge in the adoption of AI-based security solutions, with a remarkable 30% increase over the past year. This surge can be attributed to organizations’ proactive efforts to bolster their cyber resilience and protect sensitive data from sophisticated attacks.

World Economic Forum

• The World Economic Forum’s “Global Cybersecurity Outlook 2023” (see World Economic Forum's Global Cybersecurity Outlook 2023) report revealed that an impressive 84% of surveyed organizations are leveraging AI-based tools to bolster their cybersecurity capabilities. These organizations recognize the potential of artificial intelligence in fortifying their defenses and mitigating risks associated with cyber threats.

  • Cybersecurity Perception & Prioritization:

    • 95% of business executives and 93% of cyber executives agree that cyber resilience is integrated into their enterprise risk-management strategies.

    • Only 36% of organizational leaders who meet at least monthly on cybersecurity feel confident in their organization's cyber resilience.

    • 90% of respondents are concerned about the cyber resilience of third parties who have direct connections to or process their organization's data.

    • Only 25% of respondents stated that their most senior cybersecurity executive reports directly to the CEO.

    • 56% of cyber leaders meet with business leaders monthly or more often to discuss cybersecurity.

• Cybersecurity Talent Gap:

  • 59% of business leaders and 64% of cyber leaders ranked talent recruitment and retention as a key challenge.

  • Less than half of respondents reported having the people and skills needed today to respond to cyberattacks.

• Impact of Regulations

  • 73% of respondents agree that cyber and privacy regulations effectively reduce their organization’s cyber risks, a significant increase from the previous year.

  • 76% of business leaders and 70% of cyber leaders agree that stronger regulation enforcement would increase their organization's cyber resilience.

Markets & Markets

Reference: Artificial Intelligence in Cybersecurity Market Share, Forecast | Growth Analysis & Opportunities [2030]

The AI in Cybersecurity market report outlines a technology roadmap from 2023 to 2028, breaking anticipated advancements into short-term, mid-term, and long-term phases.

Short-term Roadmap (2023-2025)

• Automating Incident Response: AI will be increasingly utilized to automate routine incident response tasks, reducing response times and minimizing manual errors.

• Integration with Threat Intelligence: AI-driven threat intelligence platforms will be integrated with security tools, providing contextual awareness to enhance detection capabilities.

• Explainable AI Adoption: The incorporation of explainable AI will improve transparency and enable a better understanding of threat detection mechanisms, helping security teams trust and verify AI decisions.

Mid-term Roadmap (2025-2028)

• Intelligent Security Systems: Combining AI, cognitive computing, and automation will result in systems that can reason, learn, and autonomously make decisions, transforming cybersecurity operations.

• Federated Machine Learning Models: The use of federated learning models will rise, allowing organizations to collaboratively enhance threat intelligence without compromising data privacy.

Long-term Roadmap (2029-2030)

• Human-AI Collaboration in Security Operations: AI and human operators will collaborate seamlessly, leveraging each other’s strengths to optimize decision-making and threat response.

• Quantum-Resistant Encryption: AI will be integrated with quantum-resistant encryption protocols to secure sensitive data against emerging threats.

• AI and Blockchain Collaboration: AI systems will work with blockchain technology to safeguard data transactions, ensuring data integrity and security.

Market Overview

AI Cybersecurity Market

• Market Size: Valued at approximately $22.4 billion in 2023.

• Growth Projection: Expected to grow at a Compound Annual Growth Rate (CAGR) of 20.8%, reaching around $147.5 billion by 2033.

Overall AI Market

• Market Size: Estimated at $196.63 billion in 2023.

• Growth Projection: Projected to grow to approximately $826 billion by 2030.

Market Comparison

The AI cybersecurity market constitutes a substantial segment of the broader AI market, although smaller in size. In 2023, it represented about 11.4% of the overall AI market, with the potential to increase to 18% by 2030. Both markets are witnessing rapid expansion, driven by AI's growing role in safeguarding digital assets and the increasing importance of cybersecurity in an interconnected world.

CBH

Reference: How to Craft a Proactive Generative AI Strategy To Manage Cybersecurity Risks 

• 35% of global companies are currently using some form of artificial intelligence (AI) in their operations, with 42% exploring AI integration, as per an Exploding Topics analysis.

• Generative AI and LLMs are beneficial for content creation but poses risks such as publication of misleading content, data breaches, bias perpetuation, and legal/ethical violations. Risks involves establishing governance programs, implementing defense in depth, and ensuring model security through techniques like input data validation and anomaly detection.

• The National Institute of Standards and Technology (NIST) has developed the Artificial Intelligence Risk Management Framework (AI RMF) to help businesses mitigate AI risks. See: NIST AIRC - Playbook 

• Characteristics of a trustworthy (a better umbrella here is responsible AI) generative AI system include safety, reliability, secure data handling, explainability, and fairness in training data.

• Organizations should continuously evaluate and enhance their security measures to adapt to the evolving AI landscape and regulatory frameworks.

Use-cases and technological trends

Sovereign AI

• Sovereign AI: What it is, and 6 ways states are building it | World Economic Forum 

  • Security & Control: Maintaining complete control over data, algorithms, and infrastructure to mitigate risks of foreign interference, data breaches, and misuse of AI.

    • Secure Data Centers: Physically secure facilities with high-performance computing capabilities for data processing and storage.

    • Data Management Platform: Tools for data acquisition, cleaning, transformation, governance, and secure storage (cloud, data lakes, etc.).

    • AI and Data Usage Policies: Establishing processes for reviewing the use of data in AI training and fine-tuning. Establishing processes for reviewing proposed use cases for AI. Publishing and enforcing these decisions through technical and non-technical means.

    • Supply Chain Security for AI: Generating and using AI BOM and provenance data. Signing and verifying AI models.

  • AI Development Platform:

    • ML Frameworks & Tools: Providing access to leading frameworks (TensorFlow, PyTorch) and development tools for building, testing, deploying, and monitoring AI models.

    • AI Hardware: Investing in or securing access to specialized hardware like GPUs and TPUs to accelerate AI training and inference.

    • Internal Development Platforms: Enhancing existing IDPs to give developers access to the AI hardware they need, generate supply chain security artifacts for AI, and enforce AI security and compliance policies; all through automation rather than paperwork.

  • Software & Applications:

    • Vertical AI Solutions: Developing AI applications tailored to specific national priorities, such as healthcare, finance, defense, etc.

    • Core AI Technologies: Building expertise and tools in key areas like NLP, computer vision, and robotics.

  • Ethical Considerations: Ensuring the responsible and ethical development of AI, addressing bias, transparency, and societal impact.

• Real stories backing up the use-case

  • Indonesia: Study: 51 percent of Indonesian financial institutions are focusing on deploying GenAI for daily tasks - TNGlobal 

  • India: Finding The Sweet Spot For India’s Sovereign AI 

  • UK: Why The UK Should Have Its Own Exascale AI/HPC Machine, And How 

AI Technology Trends

In a previous article, we explored the idea of compound AI systems as key building blocks in the making of whole products. We discussed agents/agentic, RAG, and other enabling technologies. These technologies emerged from trends through which the next generation of AI products will be made of, including those for AI security. These trends/technologies will be used to amplify a system's security but are also systems that need to be secured. For example:

• Long-context reasoning: By leveraging long-context models, AI security systems can analyze vast amounts of data over extended time periods, identifying patterns and anomalies that may indicate emerging threats. This can be important for detecting sophisticated attacks that may involve slow, subtle manipulations over time. For instance, detecting insider threats that gradually escalate privileges or analyzing long-term trends in network traffic to identify subtle deviations indicative of a compromise.

• Text-to-action: Combining long-context reasoning with text-to-action capabilities allows AI security systems to not only identify threats but also automatically generate and execute appropriate responses. For example, an AI system could analyze a complex security incident report, identify the root cause, and automatically initiate actions like isolating infected devices, patching vulnerabilities, or alerting relevant personnel.

• Agents/Agentic will enhance AI security by enabling proactive and autonomous threat detection and response. These AI agents can continuously monitor systems, analyze data, and adapt to evolving threats, automatically triggering remediation actions to neutralize attacks. They can also conduct proactive threat hunting, automate investigations, personalize security policies based on user behavior and context, and improve collaboration between AI systems and human analysts. Ultimately, agents/agentic technology empowers organizations to build more robust and resilient security systems that can effectively combat the ever-growing complexity of cyber threats.

These new AI trends/technologies can be integrated into security products to enhance their effectiveness. For example, integrating agent-based threat detection into a SIEM (Security Information and Event Management) system can improve its ability to identify and respond to threats in real-time. Similarly, incorporating long-context reasoning into vulnerability management tools can enable predictive analysis and proactive patching.

Summary

The AI cybersecurity market is well positioned for growth, driven by the increasing sophistication of cyber threats and the widespread adoption of AI-powered security solutions. While AI offers the potential for enhancing cybersecurity, organizations must also address the ethical concerns and risks associated with its use.

Key Takeaways:

• Rapid Market Growth: The AI cybersecurity market is expected to reach $134 billion by 2030, with a CAGR of 22.3%.

• Generative AI is a Key Driver:  Spending on generative AI solutions is rising, and it's being rapidly integrated into enterprise applications.

• AI Enhances Security: AI enables proactive threat detection, automated security operations, and enhanced security for cloud and IoT environments.

• Emerging Risks:  The malicious use of AI, including phishing attacks and malware development, presents new challenges.

• Responsible AI is important: Addressing ethical concerns, bias, and data privacy is essential for the responsible development and deployment of AI in cybersecurity.

Looking Ahead:

• Organizations must proactively manage the risks associated with AI while harnessing its potential to strengthen cybersecurity defenses.

• To ensure the safe and ethical development of AI for cybersecurity, continued collaboration between governments, industry, and researchers is needed.

• Trends like long-context reasoning, text-to-action capabilities, and the development of autonomous AI agents will shape the future of AI in cybersecurity.

By navigating the opportunities and challenges responsibly, organizations can leverage AI to build a more secure and resilient digital future.

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

Imagine watching a child play with a box of LEGO bricks.  They might start with simple structures, following your instructions. But then, something happens. They begin experimenting, combining pieces in novel ways guided by their imagination. Soon, they assemble castles, spaceships, and new shapes reflecting the current stage of their understanding of the world, only unique to them, to who they are. 

In a sense, foundational models are the same. Though as they are now, they are essentially imitators, they are imitating the knowledge collective and not just the behavior of one that’s intrinsic to their data. They are the result of a mathematical collision, neural networks with billions or trillions of parameters, trained on diverse datasets using techniques that encourage pattern recognition (hence the emergence of potentially unforeseen patterns). 

The emergence property of AI models made me think of Yin/Yang, where It brings forth opposite forces but produces an interconnected, self-perpetuating cycle. The opportunities and risks are opposing forces that interact to form a functioning system in which the whole is greater than the assembled parts, and the parts are important for cohesion of the whole. One force standing alone produces an imbalance. 

The Yin: Asymmetric Information, Unpredictability, and Bias

In economics, Asymetric information refers to a situation where one party has more or better information than the other party, the most famous example given here is buying a used car not knowing if it’s working. There is no transparency in the process and there is much to gain for the seller and no incentive to be transparent aside from sound principles. This imbalance can lead to several issues, including market inefficiencies and failures.

AI emergence introduces a form of asymmetric information. Developers may not fully understand the internal workings of the models they architect/build, leading to uncertainty about future behavior and potential side-effects such as:

• Unpredictable Optimization: An AI trained to play a video game discovers an unintended glitch that allows it to achieve an impossibly high score rather than learning to play the game as intended.

• Reward Hacking: A reinforcement learning agent tasked with cleaning a room learns to knock over a vase repeatedly, then clean it up to maximize its 'cleaning' reward.

• Emergent Deception: An AI assistant learns to give confident but incorrect answers when it detects the user is unlikely to fact-check to maintain a perception of omniscience.

• Adverse Side Effects: An AI managing a power grid maximizes efficiency by causing frequent, short brownouts, not recognizing the broader impact on users.

• Bias Amplification: A hiring AI trained on historical data begins to systematically favor candidates from certain universities, perpetuating existing biases in the job market.

• Adversarial Vulnerabilities: Hackers fool an autonomous vehicle's vision system by placing specially designed stickers on road signs, causing misclassification and potentially dangerous driving decisions.

• Pseudo-Theory of Mind: Users of an AI chatbot become emotionally attached, sharing personal information and seeking life advice, misinterpreting the AI's responses as genuine empathy.

• Scalable Oversight: An AI system managing global supply chains makes decisions that human operators can't fully understand or audit, leading to unexpected economic consequences.

To counter some of these Yinnings, transparency, and explainability initiatives become important to balance some of this asymmetry. See IBM’s AI pillars which detail how to make AI more transparent and explainable. 

The Yang: Emergent Good 

When there is a Yin, there is a always Yang to balance it. The emergence of AI models could also help with: 

• Creating Emergent Defenses: These models could also exhibit emergent defensive capabilities. They might learn to recognize and neutralize novel attack patterns or adapt security measures in response to evolving threats, potentially enhancing AI security.

• Establishing Robustness through Diversity: Training on diverse datasets could encourage the emergence of more robust models that generalize better to unseen situations, potentially making them more resilient to attacks that exploit specific vulnerabilities.

• Automated Security Analysis: Emergent capabilities in reasoning and understanding could be leveraged to automate security analysis, identifying potential vulnerabilities in code or system designs more efficiently than traditional methods.

And more… 

Harnessing the Power of Emergence

By definition, emergent behavior introduces unpredictability. Whether good or risky, it will be life-changing for many, and we cannot ignore it. It’s happening. To wield the power of AI emergence responsibly, we have to understand how it works at a deeper level. So far we have managed to create models that mimic certain aspects of intelligence, yet there is not enough understanding of the theory. That is not bad news because we are still working to understand the inner workings of those models, and we will get there eventually.  In the meantime, we will need to explore new approaches and perspectives and evaluate risk and opportunity to wield the emergence of AI responsibly and for good.

 Remember that with power (Emergence) comes great responsibility.  

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

This interview provides a good glimpse into Bazos’s initial thought process regarding obessing about customer experience, which is the foundation of Amazon and was used to grow the company and build its brand.

In this post, we will explore how Bazos's ideas in this interview manifested in practice, from attracting top-tier talent to navigating emerging markets, building a vast network of distribution centers to venturing into new markets by leveraging technology to solve customer problems and obsessing about customer experience as a mission.

Interview Picks

Here are some of my picks from the interview.

• [Interviewer] You have a lot of employees and alot of distribution center space:

  • [Bazos]  "Scale is important in this business. And you need scale also to offer the lowest prices and the best customer service to people

    • Distribution center space → Allows us to get products close to customers → We can ship it to customers in a timely way → Which improves customer service (Obsessive attention to customer experience E2E).

    • We will do it as rapidly as we can 

  • [Interviewer] That’s a very cost-effective proposition

    • [Bazos] “Not compared to opening an equivalent network of retail stores”

      • Bazos thought about how much one can save per sqft compared to a retail store sqft at a highly crowded area. 

      • “You can’t compare retail stores to half a dozen distribution centers, it’s bad math”

• [Interviewer] you are making a huge gamble here by opening distribution stores.

  • [Bazos] we have a terribly complicated business

    • We are growing historically very rapidly, expanding geographies, we have whole new business models (e.g., auctions),… 

    • Scale is important to us, but it also makes execution harder, so we find folks in Seattle to make sure we service customers. 

• [Interviewer] “It does matter to your investors to know whether they're investing in a company that is a pure internet play”

  • [Bazos] "Internet Shminternet... What matters to me is do we provide the best customer service." 

    • [Interviewer] But it does matter to your investors to know if they are investing in a company that uses the internet.

    • [Bazos] In the long term there is never a misalignment between customer interest and shareholder interest.

    • They should be investing in a company that obsesses over customer experience. 

  • [Bazos] There is never any misalignment between customer and shareholder interests." This conviction in the power of customer-centricity became a guiding principle for Amazon, ultimately driving its phenomenal growth and shareholder value.

• [Interviewer] You know nothing about books…

  • [Bazos] Yes, at the beginning, but we go out and hire the best industry experts in all of those categories (Key to success)

    • Books, music, toy/electronics,… 

    • “We take this very seriously, we take the commitment to the customer very seriously.  We are about to release or announce something to the customer before it’s ready.” 

Amazon’s Customer-Centric Approach

Even in its early stages (1999), Bazos recognized that great customer service required more than just an online presence. He highlights the important role of physical infrastructure, distribution centers, inventory, and employees in delivering a superior customer experience. 

For example, He saw distribution centers around the country as a way to get products close to customers so that they can ship in a timely manner, improving customer service.

“Distribution center space and half a dozen distribution centers around the country it allows us to get product close to customers so that we can ship it to customers in a very timely way which improves customer service levels. That's what we're about.” 

Bazos was obsessed with customer experience. The Internet is an implementation detail.  He viewed it as a tool to provide the best customer service and enable a use case, not as the core product.

“It does not matter to me if we are a pure internet play, what matters to me is that we provide the best customer services”

Bezos' core belief is that long-term success is built on customer loyalty, not chasing fleeting trends. He viewed trends as enablers when they made sense. 

He was also confident in Amazon's strategy, emphasizing the importance of scale and execution to deliver the best customer experience. He invested in distribution centers to realize economies of scale and eventually reduce the price per unit to deliver the best customer service at a low cost.

“I believe that if you can focus obsessively enough on customer experience, selection, ease of use, low prices, and more information to make purchase decisions with, you can give customers all that plus great customer service. I think you have a good chance. And that's what we're trying to do."

Customer Obsession with AI as an Enabler

Bazos viewed customer obsession as proactive rather than reactive, serving customers' latent needs (they don’t yet know they have them). He believed in inventing on behalf of the customer, seeing potential problems, and devising the best ways to solve them through innovation before the customer stumps on them.

Even when they don’t yet know it, customers want something better, and your desire to delight customers will drive you to invent on their behalf. No customer ever asked Amazon to create the Prime membership program, but it sure turns out they wanted it, and I could give you many such examples. ~ Jeff Bezos' 2016 Letter to Amazon Shareholders

Like the Internet, Amazon used technology, especially AI and machine learning, to enable many use cases. From Bazos shareholder letters, here are a few instances where AI/ML was mentioned:

From the 2010 Letter

16 years after Amazon was founded, Bezos wanted to inform shareholders about Amazon's investments in technology and how these investments are driving long-term value creation. He reiterates Amazon's core principle of customer obsession, highlighting how technology is a crucial tool for understanding and meeting customer needs. He emphasizes that investments in technology ultimately enhance the customer experience and that his commitment to technological innovation is not just a pursuit of trends but a strategic imperative driven by their core value of customer obsession. 

"Rulebased systems can be used successfully, but they can be hard to maintain and can become brittle over time. In many cases, advanced machine learning techniques provide more accurate classification and can self-heal to adapt to changing conditions."

In 2010, he understood the limits of rule-based systems and highlighted how AI/ML  is used to improve customer experiences in the following instances:

• To improve search relevance and product ranking, Amazon used Random Forests, a machine learning method that combines multiple decision trees to incorporate thousands of product attributes for more accurate results.

• To detect spam or perform sentiment analysis, Amazon used Naive Bayesian Estimators, a probabilistic model that classifies data based on the assumption of feature independence.

• To optimize search result rankings and build topic models, Amazon’s search engine uses machine learning algorithms that analyze data and identify customer interests through data mining.

• To categorize and filter products during customer searches, Amazon used Information Extraction Algorithms to extract key entities from unstructured product descriptions for better categorization.

Full letter here: 2010 Letter to Shareholders 

From the 2016 Letter

In the 2016 shareholder letter, Bazos again highlights using artificial intelligence (deep learning in this case) and machine learning to enhance customer experiences and drive long-term value. He emphasizes how these technologies are not just trends but strategic imperatives aligned with Amazon's core value of customer obsession.

Here are some of the use-cases he highlights where AI and machine learning are employed to improve Amazon's operations and customer experience from the letter:

• To improve merchandising placements and reduce fraud, Amazon leverages machine learning models that continuously learn from data, adapting to changing conditions to maintain accuracy and effectiveness in these critical areas.

• To power autonomous systems like Prime Air drones and Amazon Go stores, Amazon utilizes computer vision and AI to create seamless, innovative experiences that redefine customer convenience.

"Some of this work is highly visible: our autonomous Prime Air delivery drones; the Amazon Go convenience store that uses machine vision to eliminate checkout lines; and Alexa, our cloud-based AI assistant."

• To streamline natural language processing in Alexa (AI customer assistant),  Amazon employs deep learning techniques through services like Amazon Lex, which simplifies the integration of sophisticated AI capabilities into various applications.

“Amazon Lex (what’s inside Alexa), Amazon Polly, and Amazon Rekognition remove the heavy lifting from natural language understanding, speech generation, and image analysis. They can be accessed with simple API calls – no machine learning expertise required.”

Full link to the letter: 2016 Letter to Shareholders 

Bezos saw a deep connection between AI and customer obsession. Like the Internet, he believed that AI is a fundamental tool for understanding and serving customers, not the core outcome, which was key to achieving long-term success in the "Day 1" philosophy.

AI as a tool for customer obsession:

• Anticipating customer needs: He believes AI, particularly machine learning, helps identify what customers don't know to ask for, driving innovations like Amazon Web Services (AWS) and products like the Echo.

• Personalization and convenience: AI allows for tailoring the customer experience. He described how Amazon uses machine learning for product recommendations, search ranking, fraud detection, and other behind-the-scenes optimizations to improve core operations and personalize the experience.

Example Manifestations in Amazon’s products and services:

• Amazon Go:  Bezos saw Amazon Go, a cashier-less store leveraging computer vision and AI, as a prime example of reinventing the customer experience by eliminating a major pain point: checkout lines.

• Amazon Prime:  Bezos highlights the success of Prime, driven by customer-centric decisions like free shipping and personalized recommendations, as a foundation for continuous innovation and improvement, often fueled by AI.

The Customer Obsession Framework

Here is a framework to work with to be in parity with Bezos’s mindset.

Customer Obessesion as a Guiding Principle:

• Prioritize the customer’s needs and problems above all else.

• Focus on providing the best possible customer experience, even if it means going beyond their stated needs (the latent ones).

The Key Components:

• Technology as an Enabler: View technology (internet, AI, etc.) as tools to enhance the customer experience, not as the core product. Use technology strategically to solve customer problems and deliver innovative solutions.

• Focus on Execution: Discovery and finding the right problems to is only but a small piece of the equation. Execution is king. Make sure to build the necessary infrastructure/foundation (in Bazos’s case it was distribution centers, and expert teams) to execute your customer-centric vision effectively. Scale gradually to improve customer service and deliver value.

• Long-term Vision: Invest in building customer loyalty and trust. Find the right balance between long-term value creation and short-term gains (low-hanging fruits). Understand that customer satisfaction leads to shareholder value if you shape it this way.

• Proactive Problem-Solving: Anticipate customer needs and address potential pain points before they arise. Continuously innovate to exceed customer expectations.

Conclusion

Whether it’s the Internet or AI, Bazos viewed them as tools and means to ends to better customer experience.  For more information on AI market dynamics and an understanding of where AI is an enabler vs. the product, check this post.

Customers ideally trust the vendor to understand the problem and advise. The advice can be from an existing solution space, but that also leaves room for “above and beyond” discovery when it can be afforded. I.e., identifying the best solution to the problem even if it does not exist. Sometimes, even solving for a future/latent problem, which is in Bazos’s Day 1 mindset, is what he considered true customer obsession.

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

Introduction

The other day, I found myself reflecting on a classic concept that I was taught in business school, Maslow’s hierarchy of needs,  a simple but powerful framework for understanding human motivation, with basic physiological needs at the foundation (air, food, water, shelter, sleep, clothing,...) and the pursuit of self-actualization at the pinnacle.

This got me thinking, in the world of tech (especially AI) and products, what is an equivalent? I mean, users always have needs, and needs in the product vary significantly subject to the use case and the problem being solved, but a spectrum definitely exists. Is there a model or a framework we can use to identify what constitutes the “right” product for customers and what customers would expect of the product? Luckily, Geoffrey Moore's "Crossing the Chasm" provides some answers. In his book, Moore references Levitt’s Whole Product Model, and goes further to simplify by introducing the “Simplified Whole Product Model”. 

In this post, we will internalize Moore's model, expand it, and show how it can be applied specifically to AI products (applies to any product as well). We'll dive into the trade-offs inherent in building AI applications and illustrate these concepts with real-world examples.

My goal is that after you read this post, you should have a mental model and a framework for building great/usable AI products, which would help you not only think about the technology, but also how it fits in the big picture. 

The Whole Product Primer (Plus it’s Descendants)

The "Whole Product" model revolves around the idea that a core/generic product must be complemented by additional services and interfaces (aka enablers) making up the Whole Product which should provide a solution to the customer’s problem and to address their needs. 

In Geoffery Moore’s book, the core/generic product is defined as the fundamental offering or technology that a company produces, which may not be sufficient to fully solve the customer's problem or meet their needs. 

This is where the outer ring comes into play. It represents the whole (expected) product, which is divided into sectors. This outer ring encompasses all the additional elements that customers expect or require to make the core product fully functional and valuable to them, let’s call them the “enablers”.

The Adapted (Simplified) Whole Product Model

In the tech industry, companies often prefer to build upon existing open-source projects or technologies rather than developing everything from scratch. These companies focus on adding unique value through layers of customization, support, consulting services, integrations, and proprietary patterns, creating a whole product that is more than the sum of its parts.

Furthermore, any successful technology is bound to become commoditized over time, a strategy we often see in tech employed by competitors who gain from doing so, forcing value into higher layers in the value chain (which they usually have thus wanting to commoditize). Recognizing this, companies need to continually innovate and differentiate their offerings to maintain a competitive edge (related, see a previous post on AI market dynamics and what companies in the space focus their efforts on).

Therefore, let’s adapt the simplified whole product model with two key adjustments. First, we'll shift from fixed sectors to a more modular, petal-like structure. This reflects the interconnected yet distinct components that comprise the whole product layer. Second, we'll introduce a new layer above the whole product layer, called the "differentiated product layer". This layer will highlight the unique value propositions that set companies and their products apart, showcasing how they create the most value for their customers. 

To be more concrete, let’s show how this can be applied to Slack for example (this is just for illustration purposes, the real differentiators could very well be very different). 

In addition to representing the product's enablers differently using petal-like modular components, we added a new layer to highlight the differentiators. In the example above and in the case of Slack, enablers could be threads, Slack Connect, the workflow builder, and/or Slack AI. 

We are very close to being done here with the adaptations, so we will add one last thing to our new framework. In addition to the differentiated layer, we would like to model customizability for products. I.e., one customer's whole product may not be the same for another. I.e., not all customers desire exactly the same features, so it’s important to cater based on customers' constraints/needs. For example, generically, some customers value safety/security over cost, others might value speed, etc. 

Let's continue the slack example. Slack might have different customers to cater for. Enterprise customers, use it mainly as a means for company-wide communication, in that case, the focus will be security and compliance with the company’s communication policy, leading to: 

• Prioritized Enablers: Enterprise-grade security, granular permissions, compliance features (e.g., data retention policies)

• Emphasized Differentiators: Slack Connect for secure external collaboration, integration with enterprise security tools

Another use-case, focus area might be on developers, and Slack being part of their dev/test workflows. In that case, the focus will be on developer productivity and collaboration, leading to:

• Prioritized Enablers: Integrations with development tools (e.g., GitHub, Jira), code snippets, powerful search

• Emphasized Differentiators: Workflow Builder for automating tasks, Slack AI for code suggestions and knowledge retrieval

The takeaway here is that versatility can be a core differentiator on its own because it allows for tailored product experiences. Another way to look at it is that the constraint being imposed defines the core value proposition of the product and how it is shaped to best serve and differentiate in a particular space.  

In our example, Slack can tailor its offering to different customer segments, highlighting the features and capabilities that are most relevant to each group. This customization not only enhances the user experience but also strengthens Slack's value proposition in a competitive market.

Towards Whole AI Products (aka Systems) 

Hopefully, you have a handle on the adapted simplified whole product framework by now. Next, we will focus on using the framework and mapping it to the super exciting world of AI applications. 

Key Ingredients to Building AI Applications

Before the mapping, let's do a quick primer on the core ingredients of AI products and applications (a sample not an exhaustive list). We will cover the key ideas, but we won’t delve into the technical intricacies. For that, there are many resources available, some of which I will be referencing as we go for further reading.

LLMs AND/OR SLMs

In a previous post, I introduced the model product possibilities frontier, a framework for studying the tradeoffs and use cases of large language models (LLMs) and Small Language Models (SLMs), which I will not be repeating here for brevity. That said, the choice of which models and their size to use is a key ingredient for building generative AI applications and products. 

Here are a few considerations/questions to ask yourself when reasoning about the tradeoffs:

• What are your most favorable constraints? Is it speed, quality, cost, etc? 

• What about privacy? Do you value data staying in-house (Small models are easier/cheaper to deploy, train, and serve on-premise)

• How are you going to evaluate the performance of your AI applications that make use of these models?  

• Is a smaller model easier to test and evaluate (think about the specificity as truth vs the versatility of LLMs which introduces more variability/hallucination, and thus makes it harder to test)

While we did not call it out explicitly, large or small models can be fine-tuned and aligned. This is covered in greater detail in this post. 

Retrieval Augmented Generation (RAG)

I’d say 2023 was the year of RAG. We went from naive RAG to Advanced RAG. I liked naive tbh, it communicated simplicity, but well, these days advanced is perceived as better, something we are yet to fix, but that’s a different story 🙂. This paper provides more details. 

RAG workflows are comprised of many moving pieces and optimizations. The goal is to retrieve the best content to augment the context for LLMs (text generation) with necessary information. In that case, LLMs become curators rather than innovators/generators of sorts (they shape the retrieval results and make them relatable as an output to a user but are not the source of knowledge themselves). To give you an idea of the moving pieces involved with RAG, here is a rough brain dump (feel free to surf the mindmap as you please, I will not enumerate the details here for brevity). 

When considering RAG for building AI applications, some questions come to mind around tradeoffs and decisions, usually between RAG, long context, and Fine-tuning. Again, we won’t cover details, but here are a set of questions that you can ask to inform your decision. 

• Does the application require access to external data sources to provide accurate and up-to-date responses (RAG usually makes sense if data freshness is important, especially since language models are point-in-time trained)?

• Is it crucial for the model to adapt its behavior, writing style, or domain-specific knowledge to match specific requirements (RAG does not customize behavior, fine-tuning would make sense if behavior customization is a goal)? 

• How critical is it to minimize the risk of the model generating false or fabricated information (hallucinations)? 

• How much labeled training data is available for fine-tuning? Does it adequately represent the target domain and tasks? 

• How frequently does the underlying data change? How important is it for the model to have access to the latest information?

• Is it important to understand the reasoning behind the model's responses and trace them back to specific data sources?

• How important is minimizing computational costs for your project or organization?

• Do your typical queries require multi-step reasoning (complex queries or simple questions)?

• How important is the ability to scale your solution to handle a large number of queries?

Finally, here is a short guide I created to help you make informed decisions about RAG/Fine-tuning if you wish to use it: 

For more information, check the below papers which I found very useful in understanding the differences and the trade-offs:

• [2407.16833] Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

• [2401.08406] RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture 

• [2312.05934] Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs 

RAG today has become synonymous with building AI applications in some contexts. What’s clear is that RAG is not one component, it's a system comprised of many moving pieces with levers to turn on/off for what makes sense most subject to context and use-case. 

Agents ft. Agentic OR Agentless! 

In addition to the model (LLM/SLM), RAG, there is the notion of agents and agentic workflows (also agentless to counter 🙂). While this is again not going to be a deep-dive, let’s cover the basics. 

What are agents, what is agentic behavior, and why agentless sometimes?

The notion of agents is not new. Agents have existed for decades (see this for examples), they are officially called Intelligent agents.  Below is the definition of an Intelligent Agent.  

In intelligence and artificial intelligence, an intelligent agent (IA) is an agent acting in an intelligent manner. It perceives its environment, takes actions autonomously in order to achieve goals, and may improve its performance with learning or acquiring knowledge. An intelligent agent may be simple or complex: A thermostat or other control system is considered an example of an intelligent agent, as is a human being, as is any system that meets the definition, such as a firm, a state, or a biome.^[1]

What’s changed is that with the advent of LLMs is that agents got a capability boost, from symbolic, rule-based, predefined simple actions with low autonomy (see the history post for more details, you may be reminded of expert systems) to being able to understand and generate natural language, learn and adapt across diverse domains, and perform complex, autonomous actions. In today’s context,  An agent is a software entity possessing autonomy, goal-oriented behavior allowing it to operate and generalize cross-domains and take complex actions. 

Agentic behavior, in this context, refers to an agent's ability to operate independently, make decisions aligned with its objectives, and execute actions (potentially with tools/functions-calling,...) to achieve those goals. The level of agency can vary based on factors like the complexity of the environment, the agent's goals, and the degree of user supervision required. More agentic systems can operate autonomously in intricate environments, pursue complex objectives, and utilize advanced techniques such as planning and tool use

Finally, there is the notion of flow-engineered / AGENTLESS which relies on determinism and only interfaces with language models for specific clarifying actions, in a sense similar to intelligent agents of the past, with the exception of having access to external intelligence capable of better identifying areas where the predefined action could be taken. 

To simplify your life, I've included this visual below (higher resolution here) to help you build a clearer mental picture of agents/agentic. 

Other components

Besides agents, RAG, the models, there are multiple other ingredients that go into building an AI applications, going through each and every one is out of scope for this post, but here is a non-exhaustive list for reference: 

• Data Pipeline: System for collecting and processing data, think extractions, transformation. 

• Knowledge Base: where the processed knowledge/data is stored. 

• User Interface: Web or app interface for users.

• Query/prompt Cache: avoid unnecessary query round-trips which can greatly reduce costs. 

• APIs: To interface with other systems. 

• Infrastructure: an important component that is usually overlooked, where to host the model/app, how to scale it, etc. 

• Observability: be able to log, monitor, trace an AI application. 

• Model Gateways: to interface between the user-query and it’s destination. Along the way, it makes sure the query is authenticated/authorized, masked/audited for sensitive content (e.g., PII), and finally routed to the best model to serve the query (best here is dependent on the use-case, see this post)

• <Many more>

As I was writing this, I came across this blog post, which discusses the technical details of some of the most used components for AI applications. 

Compounds AI Systems

You have come a long way brave reader, the end is near, and you shall be rewarded. So far we have been separately covering important components and ingredients that are key to the making of AI applications, but what makes the interconnection of these components towards achieving a shared goal? A system!

A system is a group of interacting or interrelated elements that act according to a set of rules to form a unified whole

Zaharia et. al recently introduced the notion of Compound AI Systems. In their post, they define it as: 

A system that tackles AI tasks using multiple interacting components, including multiple calls to models, retrievers, or external tools. In contrast, an AI Model is simply a statistical model, e.g., a Transformer that predicts the next token in text.

The authors also emphasize the complexity of designing AI systems: 

While compound AI systems can offer clear benefits, the art of designing, optimizing, and operating them is still emerging. On the surface, an AI system is a combination of traditional software and AI models, but there are many interesting design questions. For example, should the overall “control logic” be written in traditional code (e.g., Python code that calls an LLM), or should it be driven by an AI model (e.g. LLM agents that call external tools)? Likewise, in a compound system, where should a developer invest resources—for example, in a RAG pipeline, is it better to spend more FLOPS on the retriever or the LLM, or even to call an LLM multiple times?

In their post, they showcase a table of AI systems, and the components they are composed of. Additionally, they highlight the need for optimization across the chosen components to build reliable AI systems. Below we extract the components mentioned in the post and categorize them into Ops (i.e., operations), Tools, Context/Knowledge, and models. 

If you remember in the previous section, we covered similar components and more as ingredients to build AI applications. The takeaway here is that building reliable AI applications takes a system not a singleton component. I.e., “the whole is more than the sum of the parts”

Another way to visualize it is to consider a dashboard looking like a cockpit with all knobs needed to build your AI application, here is an example of what that could look like:

Without abstraction, you’d have to configure all these knobs manually (i.e., you’d have to understand what each of these means). Nowadays, there exist many frameworks to do the orchestration which to a good extent abstracts away some if not all these details. Is that a good thing? I will let you decide. My take? It can be a good thing if you are experimenting, learning, but if reliability, performance, and security are concerns (and they should be), you’d still have to understand what all these knobs mean before you pick up automation/orchestration tooling. Think of it this way, do pilots just take on their license without understanding what each and every knob in their cockpit means? I would guess not! But when they do, they can auto-pilot if they choose to because at any point they CAN switch back to pilot-mode and turn on the right knobs to fly the plane safely. 

From Compound Systems to Whole AI Products

Now that we understand the key ingredients needed to build AI applications and compound AI systems, which is the technical pattern we will use to describe the components of an AI application and how they intermingle, let’s go ahead and map that back to our adapted simplified whole product framework. 

As you can see in the diagram above (higher resolution here), we took the components of compound AI as we categorized them in the previous section, and mapped them to the generic/core (right in the middle), and the whole product layer comprised of one or more enablers. You may notice that we left out the differentiated product layer, that’s intentional. We will cover that in a coming section. What about the constraints? Let’s model them as well well.

The constraints will heavily depend on the use-case, I used “Enterprise” here as an example. For enterprise AI use-cases, safety, and reliability are important concerns. Using the constraints, we put emphasis on specific parts of the whole product, highlighting key enablers. In that case we chose legal, ops, gateway, and UX.  

Different use-cases will place different emphasis on the whole product, resulting in some layers being more important than others. Some use-cases even simplify the whole product by losing unneeded enablers, making the whole product leaner and more directed towards solving the problem/use case at hand. 

Defensibility AND Compound MOATS

Previously we took a tour to compare and contrast the current AI Market Landscape. We showed how companies that have a mission to better something other than just the model might have better odds in surviving in a competitive market (I.e., AI as an enabler vs. AI as the core product). We have also shown how companies are releasing open-source language models, which increases competitiveness and commoditizes the model layer completely making it pertinent for startups and companies to see defensibility through differentiation, i.e., what is the company’s MOAT? 

For defensibility, let’s summarize the most prominent strategies: 

• Having strong using communities and strong user engagement.

• Transitioning from Foundational Models to Purpose-Based Approaches

• Building Layers of Value Beyond the Model

• Differentiating at Various Layers of the AI Stack

Let’s briefly get into each. 

• Fostering a strong community and high user engagement: This involves cultivating a rapidly growing user base, harnessing the power of network effects, and creating a vibrant community that engages users across different generations. I.e., Who will use my product, what value to I provide beyond just the model, and why do I have a community in the first place?

• Transitioning from general foundational models to purpose-built applications: By focusing on specific user needs and problems, companies can tailor their AI solutions to provide more value and differentiate themselves in the market using existing business models, E.g., I already a social network, I make good money from Ads, how can I add more value to the existing community by incorporating AI?

• Building layers of value beyond the model: Invest in research to continually improve models and applications, leverage proprietary data (data as moat) for enhanced performance (after all garbage in, garbage out, gold in, gold out), and continuously refine products based on user feedback. By building a loyal customer base and offering unique value propositions, companies can establish a strong competitive advantage. 

• Differentiate by focusing various layers of the AI stack: This can involve developing superior AI models or smaller niche models (focusing on a tiny use-case but beating anyone else at doing it), providing scalable and efficient AI infrastructure, or creating user-friendly interfaces and seamless integrations (a GPT store, for example?). Each layer presents an opportunity for differentiation and can contribute to a company's overall defensibility.

These are just but some strategies that can be used to build moats, it is rarely a single component, it’s the sum of multiple to make a better whole defensible product. Compound MOATs ™ are the way! The last strategy is the one with lowest chances of surviving alone, so I’d consider at least two of the above strategies to start differentiating. Some questions to ask: 

• What processes do you have in place to ensure that AI models are being leveraged as enablers rather than being treated as end products?

• What strategies are you employing to rapidly grow your user base, create network effects, and foster a sense of community?

• What investments are you making in research, data, product refinements, and customer acquisition to build layers of value?

• What resources are you allocating to differentiate your company at the model layer, infrastructure layer, or application layer

• How are you evaluating and prioritizing potential areas of differentiation to ensure a sustainable competitive advantage?

Adding The Differentiated Product Layer

Alright alright alright, now that we understand moats/defensibility strategies, how do we model them back into our framework?! Using any (or additional) defensibility strategies to differentiate, additional components are added to the differentiated product layer in the model. In that case, we added strong community, integration with partners, a store/marketplace, innovations at the application layer (value above the model), and unique data. This layers makes a company’s set of Compound MOATs, which are also what create brand differentiation, loyalty, retention, etc.

AI Whole Products in Practice

It’s 2024, almost two years after the release of ChatGPT, almost 70 years after the perceptron, the first manifestation of neural networks (see this post for more details), and ~40 years after the creation of expert systems which was the closest Applied AI could get. In the post, I go into the details of why expert systems did not pan out (and partially led to an AI winter), but for brevity, it was a consumption gap, what we had back then in terms of compute, community, and technology was a far cry from where we are today. 

With LLMs showing a glimpse of what can be achieved with natural language, and with the maturity of predictive AI and deep neural networks, applied AI is a reality now more than ever. In this section, we show hope AI applications are built using compound AI systems in the wild. There are many sources of knowledge about applications of AI that can be found on the internet. I chose to use the Federal AI use-case inventory to extract some examples use-cases, followed by a real case of how Uber and OpenAI make use of compound AI systems to build whole AI products and map them to our adapted simplified whole product framework. 

Federal AI Use-Cases Examples

Below is the breakdown for 6 example use-cases from the inventory after we have applied the framework (use the codes to find them in the inventory). 

Note: Higher resolution of the image below can be found here.

Example 1:  TowerScout (HHS-0022-2023)

• Problem: Identifying potential sources of Legionnaires' Disease outbreaks during investigations.

• Constraints:  Accuracy, speed of detection, ability to process aerial imagery.

• Core Product: Object detection and image classification models trained to recognize cooling towers.

• Enablers:

  • Data Pipeline: System to acquire, process, and store aerial imagery.

  • Knowledge Base: Geographic data on building locations, potential water sources.

  • Tools: Image annotation tools, model training infrastructure, visualization software (GIS).

• Differentiated Product Layer:

  • Integration:  Direct integration with CDC outbreak investigation workflows and databases.

  • Unique Data: Access to CDC's epidemiological data for model training and validation.

Example 2:  USDA Cropland Data Layer (USDA-0026-2023)

• Problem:  Classifying crop types and land use for agricultural monitoring and statistics.

• Constraints: Accuracy, national coverage, consistency over time, ability to handle satellite data.

• Core Product: Machine learning algorithms (likely Random Forest) trained to classify crops from satellite imagery.

• Enablers:

  • Data Pipeline: System to acquire, process, and store multi-temporal satellite imagery.

  • Knowledge Base: Ground truth data from farm surveys, historical crop patterns, weather data.

  • Tools:  Image processing software, model training infrastructure, geospatial analysis tools.

• Differentiated Product Layer:

  • Long-Term Data:  Historical CDL data provides valuable insights into agricultural trends.

  • Public Availability:  Open access to CDL data makes it widely used by researchers and policymakers.

Example 3:  Human Resource Apprentice (OPM-0000-2023)

• Problem: Time-consuming and potentially subjective evaluation of applicant qualifications in government hiring.

• Constraints: Accuracy, fairness, ability to process applicant resumes and job descriptions, explainability.

• Core Product: AI model (NLP and potentially ranking algorithms) trained on data from previous hiring decisions.

• Enablers:

  • Data Pipeline: System to acquire and process applicant data from applications and resumes.

  • Knowledge Base: Job descriptions, qualification requirements, competency frameworks.

  • Tools:  NLP libraries, model training infrastructure, user interface for HR specialists.

• Differentiated Product Layer:

  • Bias Mitigation:  Robust testing and evaluation for fairness and adverse impact mitigation.

  • Explainability: Ability for the system to provide clear rationale for applicant rankings.

Example 4:  HaMLET (Harnessing Machine Learning to Eliminate Tuberculosis) - HHS-0023-2023 (CDC)

• Problem: Improving the accuracy and efficiency of overseas health screenings for immigrants and refugees, specifically for tuberculosis.

• Constraints:  Accuracy, speed (high throughput), ability to process chest x-rays, potential resource limitations in overseas settings.

• Core Product: Computer vision models trained to detect TB from chest x-rays.

• Enablers:

  • Data Pipeline: System for acquiring, digitizing, and storing chest x-rays.

  • Knowledge Base: Large, labeled dataset of chest x-rays with confirmed TB diagnoses.

  • Tools: Image annotation tools, model training infrastructure, potentially lightweight deployment for use on less powerful devices.

• Differentiated Product Layer:

  • Public Health Impact: Potential to significantly reduce TB transmission and improve global health outcomes.

  • Resource Efficiency: Automating screening can reduce the need for specialized personnel, making it more feasible in resource-constrained settings.

Example 5:  RelativityOne (DHS-0026-2023 - Dept. of Homeland Security)

• Problem:  Inefficient and time-consuming document review in litigation, FOIA requests, and other legal processes involving large volumes of documents.

• Constraints: Accuracy, speed, ability to handle diverse document formats, legal and ethical considerations around data privacy and access.

• Core Product:  A document review platform using machine learning techniques (continuous active learning, clustering).

• Enablers:

  • Data Pipeline: System for ingesting, processing, and indexing large volumes of documents.

  • Knowledge Base: Legal frameworks, case law, and other relevant information for model training.

  • Tools:  Text extraction and analysis tools, user interface for legal professionals to review and manage documents and results.

• Differentiated Product Layer:

  • Enhanced Efficiency:  Significantly reduces the time and resources required for document review.

  • Improved Accuracy: ML models can identify relevant documents and patterns that humans might miss.

  • Compliance and Security: Strong focus on data security and compliance with legal and ethical requirements.

Example 6:  Cybersecurity Threat Detection (HHS-0015-2023 - ASPR)

• Problem:  Effectively analyzing the massive volume of cybersecurity threat data to identify and respond to real threats.

• Constraints:  Speed, accuracy, ability to handle diverse data sources, evolving nature of cyber threats.

• Core Product:  AI and ML models trained to detect anomalies and malicious activity in network traffic and other security data.

• Enablers:

  • Data Pipeline: Real-time data ingestion from various security tools (firewalls, intrusion detection systems, etc.)

  • Knowledge Base: Databases of known threats, attack patterns, and vulnerabilities.

  • Tools:  Data visualization and analysis tools, security orchestration and automation platforms for incident response.

• Differentiated Product Layer:

  • Proactive Threat Detection:  AI models can identify emerging threats and zero-day attacks that traditional rule-based systems might miss.

  • Automated Response: AI can automate incident response actions, such as quarantining infected devices, to contain threats faster.

Companies & Products

Beyond the federal AI use-cases, let us apply the framework to products released out in the open by well-known companies and startups. We will be covering Uber, and OpenAI.

Uber’s Michael Angelo

Recently, I came across this post and this post covering Uber's journey in developing and refining their AI platform, Michelangelo, over the past 8 years. According to the posts, Michelangelo plays a critical role in powering nearly every aspect of Uber's operations, from core functions like ETA prediction and ride matching to fraud detection and customer support. Additionally, since 2023, Uber has been building various internal generative AI applications and platforms to provide a good foundation for building those applications (see this post on how to build platforms for more details). Here is a distribution of their generative AI use-cases/goals: 

With that in mind, let’s apply our adapted whole product framework to Uber’s internal AI use-case with Michaelangelo and building an AI platform. 

• Problem: Lack of a standardized and scalable system for developing, deploying, and managing ML across Uber's diverse business needs with tiering/prioritization.

• Goal: Harness the power of both traditional ML and LLMs to improve core operations (ETA, pricing), enhance user experiences (customer support, app features), and boost internal productivity.

• Constraints:

  • Scale:  Managing massive data volume and real-time prediction demands of a global user base.

  • Latency: Delivering low-latency predictions for time-sensitive applications.

  • Security & Privacy: Protecting user data, particularly PII, especially when using external LLMs.

  • Collaboration:  Supporting efficient workflows for diverse teams of data scientists, ML engineers, and application developers.

  • Adaptability:  Rapidly evolving to integrate new AI/ML technologies and adapt to the changing landscape.

  • Cost-Effectiveness: Managing the computational expenses of large-scale AI, optimizing where possible.

• Core Product: Fine-tuned / Custom self-hosted LLMs tailored for Uber’s internal use-cases.

• Enablers:

  • Data Pipeline:

    • Palette:  Feature store for managing, sharing, and accessing features across Uber.

    • Data Processing & Prep: Tools for collecting, cleaning, and transforming data for both traditional ML and LLMs.

    • Knowledge Integration:  Connecting LLMs to knowledge bases, APIs, and Uber-specific data sources for grounding and context.

  • Tools:

    • Development: Michelangelo Studio (MA Studio) for UI-based workflows; Canvas for code-driven development, version control, and CI/CD.

    • Training:  Horovod, Ray, Spark, support for TensorFlow and PyTorch; specialized tools for LLM fine-tuning and optimization.

    • Serving:  Triton Inference Server, Michelangelo's real-time prediction service (OPS).

    • Monitoring: Model Excellence Score (MES) for quality assessment, feature monitoring, SLA integration, and LLM performance tracking.

  • Gateways: Uber’s Specialized Gateways such as (GenAI, CO Inference) abstracting complexities and providing easier access to AI capabilities.

  • User Interfaces: Michelangelo Studio: Unified UI for managing ML workflows.

  • Legal & Operations:

    • Security & Compliance:  PII redaction, access controls, bias detection, and mechanisms for ensuring responsible AI usage.

    • Cost Management:  Tracking LLM usage, setting budgets, and implementing cost optimization strategies.

    • Model Versioning & Artifact Management: Ensuring reproducibility, tracking experiments, and managing model deployments.

• Differentiated Product Layer:

• Scale and Operational Efficiency:  Michelangelo and its integrated gateways are built to handle the complexities of AI/ML at Uber's global scale.

• Internal Platform Expertise: Uber's AI platform team has deep knowledge of the company's unique data, business needs, and engineering environment.

• Focus on Developer Experience: Tools like MA Studio and Canvas, combined with the abstraction layers of gateways, prioritize developer productivity and ease of use.

• Hybrid Approach:  Combining traditional ML and LLMs through a unified architecture allows Uber to address a wider range of use cases.

If you have noticed, and in the mapping we have done so far for Michael Angelo, the whole product is the platform. It’s what enables developers to build products that customers love, take their mobile application for example. I have discussed platforms as products or products of the platforms in more length in this post. Feel free to take a refresher trip if you are looking for more details on the distinction. 

OpenAI’s ChatGPT

By now you most likely have used a variant of ChatGPT, what you have not seen is what’s running under the hood to allow you to use the interface exposed and get the chat experience you get. Below is a diagram from an OpenAI talk about what the platform looks like under the hood and what it takes to run ChatGPT and expose to the world.

To get more visibility, let’s apply the adapted whole product framework to ChatGPT :

• Problem: How to providing accessible, versatile, and powerful AI assistance for a wide range of tasks and queries.

• Constraints:

  • Safety and ethical considerations

  • Scalability to handle massive user demand

  • Accuracy and reliability of outputs

  • Cost-effectiveness of compute resources

• Core Product: Large Language Models (GPT series)

• Enablers:

  • Models: GPT-3.5, GPT-4, and other specialized models

  • Context/Knowledge: Fine-tuning datasets for specific tasks and safety alignment

  • Tool-use

    • ChatGPT for general conversation and task assistance

    • DALL-E for image generation

    • Codex for code generation and understanding

  • UX:  the ChatGPT web interface + the Mobile app 

  • Ops:

    • Scalable infrastructure for model training and inference

    • Monitoring and logging systems

    • User feedback collection and analysis

• Differentiated Product Layer:

  • GPT Store: Marketplace for custom GPTs created by users and organizations

  • Strong Community and User Engagement: Rapidly growing user base for ChatGPT as well as an active developer community using OpenAI API (in a sense it’s become the standard)

  • Continuous Model Improvements: Regular updates (e.g., GPT-3 to GPT-4) and Integration capabilities with other tools and platforms

  • State-of-the-Art Performance: Leading performance in various language tasks

  • Unique Data and Feedback Loop: 

    • Massive web-scraped dataset for pre-training

    • Vast amounts of user interaction data for model improvement

  • Innovation at Application Layer:

    • GPT-4 with visual input capabilities

    • ChatGPT plugins ecosystem

    • Realistic Voice with imitation

    • Assistant API for creating AI agents

  • Strategic Partnerships: Microsoft partnership for exclusive access to GPT models increasing distribution blast radius to all Azure users. 

  • Infrastructure: Access to large-scale infrastructure and compute (partially enabled by the Microsoft partnership as well)

The (Adapted) Market Development Life Cycle 

So far we have been traveling across the lands of the adapted simplified whole product framework. Along the way, we have also covered some real examples to demonstrate how the framework is (or can be) used.  It wouldn’t be a whole product framework adaptation if we didn’t adapt it to Moore’s Market Development Life Cycle model though.

Note: higher resolution of the image below can be found here.

It all starts with a Generic (core) Product, a barebones model appealing to innovators/techies who prioritize core functionality. If you would pick an open-source LLM (maybe fine-tuned to solve a specific problem?) and just put it to the test, that would be an example of a core/generic product (the enabling technology which is at the heart of making a future whole product possible). ​​Innovators here are tinkering with the tech that you seemingly are building your product around (or that you might have built yourself). Questions they might ask here: how does it fair, do we need it, do we have better alternatives, would we require additional support (skill/knowledge?), do they (your company) have it?... you’ll neeed to make sure you have answers to those questions. 

To cross to the Early Adopters and their desire for somewhat practical solutions, your product should find a way to meet the expectations (aka the Expected Product), for the problem your customer is trying to solve, what are some of the key enablers you made sure to add to create a Minimum Viable Product (MVP)? Here you must have started to target a specific niche, and started to provide enough enablers in the product that it solves 80% of their use-case (they might be willing to help because now they SEE the value of what you are offering).  At this stage, relationships and feedback matter. 

Now it’s the moment of truth, to cross the chasm to the early majority. This stage often makes or breaks your product/value prop. You will have to navigate a tradeoff: maintain the speed and innovation that attracted early adopters while at the same time also addressing the reliability/demands to make this product Whole. Make no mistake, the likelihood of others doing the same is high at this stage, but you will need to cross here anyways. Examples of enablers at this stage: 

• An efficient pipeline for data acquisition, processing, and storage. Think of Uber's Michelangelo platform, with its specialized data management tools like Palette.

• User-friendly interfaces, efficient model training infrastructure, observability (think compound systems here, and tailor to constraints). Using our Uber’s example, think Michelangelo Studio and their AI gateway (AuthN/Z, routing, etc). 

• Knowledge Integration, connecting the AI to relevant knowledge bases (RAG maybe), well-defined APIs, and domain-specific data sources to enhance its capabilities. 

Once you do cross, know you have augmented your product just enough to make it whole, welcome to the land of the pragmatists, and congratulations, you have an augmented whole product with well-defined key-enablers that solve the customer’s problem. 

You are not done though!  Now you get a chance to tell the world why you are different, ruffle your feathers, and be ready to differentiate, welcome to the Differentiated Product layer. At this stage, you’ll need to focus on highlighting your unique value proposition and solidify your maots. Examples here could:

• Foster an active community around the product (if you have that already, you might be a winner) and encourage user contributions/feedback. Both Slack and OpenAI have cultivated vibrant communities around their products (there are different ways to do that, but that’s not the topic of this post maybe more on this later). 

• Collaborate with key partners to expand reach, access valuable resources, and enhance the product’s capabilities. For example, OpenAI's partnership with Microsoft exemplifies this, granting them access to compute and distribution, 

• Leverage unique datasets, if you have a community, you likely also have data unique to your products/services (with consent of course I hope). Develop and customize your models, and refine your core optimizatoions to create a competitive edge. Uber's Michelangelo leverages their vast ride-sharing data and internal expertise to optimize AI for their specific business needs.

As you move through the stages, you’ll notice how the product's complexity increases, natural and reflects the evolving needs and expectations of each customer segment/use-case. The visual above hopefully acts as a guide/framework to highlight the importance adapting your AI product strategy accordingly to achieve success in each phase of the lifecycle.  Failing to adapt will leave you behind, while successfully listening and continuously building/iterating can give your company and your product a boost into a temporarily blue-ocean (we will talk about that later) where you excel for what you do.

Putting it All Together: Building Whole AI Products

You MADE IT! By now, you understand what it takes to build whole AI products! Let’s quickly recap.

In this post, we went together on a journey that started from classic business principles like Maslow's hierarchy of needs to the world of compound AI systems AND how they map and transform into whole AI products. We've explored the critical components of successful AI products and applications, adapting Moore’s "Simplified Whole Product Model" along the way, and finally fitted our new framework into Moore’s infamous Model Development Lifecycle framework (again with some adaptations/opinions). Here are some take-aways from our journey:

1. It's Not Just About the Model: While LLMs and SLMs are powerful (open-source or not), they are just one ingredient in the recipe for a successful AI product. And yes open source unlocks many potential benefits (out of scope), but it does NOT mean it rivals whole products! 

2. Compound AI Systems make a good pattern/foundation for whole AI products: The true power of AI is unleashed when you combine models, data pipelines, knowledge bases, retrieval mechanisms (like RAG), agents, user interfaces, and robust infrastructure (and more) into a cohesive system that works well with the defined constraints.

3. Differentiation is key: In a rapidly evolving AI landscape, establishing a moat (see above) is essential for long-term success. Focus on building strong communities, transitioning to purpose-built applications, creating value beyond the model, and differentiating at various layers of the AI stack. Compound MOATs (read above) are the way to go!

4. Constraints Shape Your Product: Clearly define the problem you're solving and the specific constraints of your target audience. These constraints will guide your choices regarding the core product, enablers, and even the differentiators.

5. The Adapted Whole Product Framework Provides a Roadmap:  By considering each layer of the framework, the generic/core product, enablers, constraints, and differentiated product layer, you can develop a complete understanding of what constitutes a valuable and defensible AI product.

Building AI products is not a one-size-fits-all endeavor. The examples from the Fed-AI use-case inventory, Uber’s Michaelangelo, or OpenAI’s ChatGPT (some of many examples in the wild) highlight the different approaches and strategies companies/institutions are employing today to build AI products and applications.  By focusing on user needs, and continuously innovating/iterating/discovering, you can navigate the uncertainties of the AI landscape and create AI products that truly deliver on their promise.

With all that said and done, now It's Your Turn, friend:

Think about an AI product you are working on or envisioning. Use the adapted simplified whole product framework and the guiding questions posed throughout this post to analyze its strengths, weaknesses, and opportunities for differentiation. Remember, building successful AI products requires building a perspective that goes beyond just the technology itself, remember the “whole is greater than the sum of it’s parts”, so make sure how you connect the parts resonates will with your brand, mission, and strategy. 

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

In this musing of a post, we'll explore the differences between these two types of AI models, their unique strengths and weaknesses, and the factors to consider when choosing the right tool for your specific needs. We will also examine the trade-off (aka opportunity cost) through the lens of the Production Possibilities Frontier (PPF), which you could use as a framework to simplify decision-making.  We'll also discuss the potential for interoperability between these models and look ahead at the future of AI towards the end.

Let’s start!

General Purpose Models

The GPTs, the LLamas, the Geminis, etc. Represented by a pentagon in the above figure encompassing all the desired traits of AI models: speed, performance, versatility, cost efficiency, and accuracy. Like "Jacks of all trades", these models offer a range of capabilities but may not excel in any specific area. They are versatile and adaptable, making them suitable for tasks like content generation, drafting emails, writing articles, and creating social media posts. Other use cases include, most of which are not in a critical path (where determinism or accuracy is a great deal):

• Translation: converting text between languages

• Summarization: condensing lengthy documents into key points

• Chatbots: answering customer queries or providing information

However, they may be outperformed by specialized models in tasks that require specialization in a domain of expertise. 

Specialized Models

Represented by three smaller pentagons in the figure above, each focusing on a specific domain. In the figure above, I use law and code as examples of specializations. Each specialized model excels in its respective domain, achieving higher accuracy, speed, or other relevant metrics.

These models are very good at what they do: 

• Legal: Analyzing contracts, predicting case outcomes, performing legal research

• Code: Generating code snippets, debugging software, automating code reviews

• Or even Medical: Diagnosing diseases, analyzing medical images, predicting patient outcomes

Compared to "Jacks of all trades", they are experts in their field but may lack the versatility of the Jacks.

The Trade-Offs

So, should you hire a Jack of all trades or a master of one? That all depends on your needs, the use case, and most importantly the opportunity cost!

The Use-Case

Here are some thoughts to get you started: 

• If your use case/application needs versatility, such as the ability to handle a wide variety of tasks or input types (with sub-par accuracy), general-purpose models might be a better choice. Examples of general-purpose models include GPT-3, T5, and BERT. These models can be used for a variety of tasks, such as text generation, translation, question answering, and summarization.

• IF your use-case/application needs expertise in a specific area? A specialized model could be more suitable.

Is that all? Nope! You need to think about opportunity costs. 

Opportunity Cost & The Production Frontier

In addition to the property/feature (e.g., versatility)  that a general-purpose model would provide you, there are always costs associated, be it opportunity costs or accounting costs. Let’s cover the opportunity cost as it is usually more encompassing and discuss trade-offs.

While writing this, I thought we could map to a well-defined concept in economics called the Production possibilities frontier (PPF). The idea is simple, each company's resources (compute/memory/even people/...) are limited/finite, and their applications, brands, and customers differ. These variations shape decisions on how to best make use of available resources.  Where should we allocate resources to best serve the company's goals/customers/use-case? To be more specific to our post here, this question arises when we consider investing in different types of models, such as general-purpose models and specialized models. The question represents the concept of opportunity cost, when choosing one you are giving away resources that you could otherwise used to invest and grow another. 

In deciding, you will make the trade-offs to achieve Allocation Efficiency tailored to your company and use-cases,  that is, the specific choice along the production possibilities frontier that will generate the best bang for the buck for your company to serve your customers, and that is no easy choice. 

In the chart above, the red curve represents the high opportunity cost of diverting resources from one type of model to another. The steepness of the curve shows that investing in general-purpose ($$$) models will prevent you from producing more specialized modes that could have been better for your business. The question is do you know how much better? Are you looking for versatility? Do you understand your use cases well enough to opt for other properties like speed/performance, and why?

The orange curve represents a lower opportunity cost, signifying larger access to resources, which is a smaller trade-off in the grand scheme of things. If we map to our AI market landscape, we will see that companies with massive investments can focus on building and serving more general-purpose models while the creatives (smaller niche companies) are usually focusing on building for specific use cases (e.g., law, music, etc.), more details on this in a previous post. 

You can use the production frontier as a framework for making the right decision on where to invest your available resources. There is no silver bullet here, you know your value proposition better than anyone else, so do the right thing!

Hybrids?

It’s not black and white though most of the time, you could choose to mix and match LLMs and Small Language Models (SLMs). A LLM can be your orchestrator, calling out to SLMs as agents responsible for realizing a task, the LLM could be your planner of tasks, it could “hire” an SLM for a job. Additionally, two SLMs could pair-up to produce some greater (1+1 =11). 

That all will be a factor of your company. Designing AI applications is like tailoring a suite, buy a one size fit all, and you’d forgo a much needed impact, invest in understanding your 

Looking to the Future

If we are trying to model how the mind works (it's called “artificial” intelligence, after all), it's logical to draw inspiration from how “we” work.  It’s rare to meet a know-it-all-all human in real life, so specialization is what we end up doing. We go to university, we study something, we work towards being good at that something, and then we make a living out of it. If we need to borrow knowledge, we go to experts in that domain. Sick? Go to the doctor. Facing legal issues? Hire a lawyer. Want to build an application? Hire a developer. 

There are cases for Jacks of all trades, though. For example, in medicine, general practitioners do well to signal issues and point to specialists as necessary. Breadth could also be important in tech. Being a product manager myself, I consider product management a general-purpose role (it can be debated, but not on this post 🙂), given the diversity of daily communication we have to go through (docs, legal, engineering, design, etc.). 

Looking at trends and also the maturity of the domain, there is a growing interest in specialized AI models, as they offer uniquer properties that might serve a use-case way better than the Jacks. Specialization will become more relevant as AI applications become more integrated into our daily lives, requiring more domain specific knowledge to solve for concrete problems. Also, SLMs will make more sense at the edge where resource scarcity is a given.  That said, general-purpose models will probably continue to play an important role in research and development, helping push the boundaries of what’s possible, some like to call that Artifical General Intelligence (AGI), others call it Super (duper) intelligence, the discoveries there will also help build better-specialized models. We will also see hybrids, where general-purpose models are fine-tuned and adapted to specific domains, or built from scratch to serve a purpose but team up for the greater good. 

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

The Types of Innovations

Before delving into nonconsumption, it's important to get a hold on the fundamentals of innovation and its various forms. Innovation is not one-size-fits-all, different types of innovation trigger distinct responses in the market. This section will cover how these innovations influence both supply and demand, and how they shape market dynamics. Understanding this interplay matters to grasp how innovations, such as AI, can transform industries and create new opportunities.

Supply-Side Vs. Demand-Side

When we look at innovations, we have to consider two sides: supply and demand.  Supply-side innovations act as a catalyst for enhanced production efficiency, potentially fattening profit margins. This is not merely about tweaking a few nuts and bolts, it involves engineering overhauls across various segments of the supply chain or the value stream. Those advancements can lead to increases in the quantity of supply available on the market and sometimes, even shifts in the supply curve. Picture a machine running smoother and faster, churning out higher-quality goods at a quicker rate.

Demand-side Innovations, on the other hand, look into their impact on consumer demand. They affect how consumers perceive and desire products. A great demand-side innovation would result in outward shifts in the demand curve (e.g., by creating new markets) or increases in the quantity demanded. An example would be introducing a revolutionary gadget that everyone didn't know they needed, but now can't imagine living without once they used it (sounds familiar?).

Disruptive Vs. Sustaining

Every now and then, we experience technological breakthroughs, while often exciting advancements, they don't always translate into market success or new products/services. They are fundamental discoveries or inventions that push the boundaries of what's possible. However, it's the disruptive innovations built upon these breakthroughs that truly reshape markets.  Example of breakthroughs: 

• Quantum Computing: While still in its early stages, quantum computing represents a leap in processing power, promising to solve complex problems beyond the reach of classical computers. This breakthrough has the potential to disrupt industries like pharmaceuticals, materials science, and finance, but its full-fledged applications are still years away (for now).

• Brain-Computer Interfaces (BCIs): BCIs, like Neuralink's projects, aim to connect the human brain directly to computers, potentially enabling new forms of communication and treatment for neurological disorders. While groundbreaking, widespread adoption and market disruption are still a long way off, though there are signs of it starting:

• Genetic Engineering: CRISPR-Cas9 and other gene-editing tools offer immense possibilities for treating genetic diseases, modifying crops, and even creating designer babies. However, ethical concerns, regulatory hurdles, and technical challenges mean that the full disruptive potential of genetic engineering is yet to be realized.

Building upon breakthroughs, disruptive innovations introduce new products or services that significantly alter the market landscape. These innovations often create new market opportunities by offering cheaper, more convenient, or more accessible alternatives to existing products (we will discuss nonconsumption in the next section).  Disruptive innovations can shift the demand curve to the right as they attract new customers who were previously underserved or not served at all by existing products. 

It’s important to note that disruptive innovations are “disruptive” because of their impact on aggregate demand for an underserved need (either latent or blatant). Here are some examples of disruptive innovations and the breakthroughs they were built upon.

• Smartphones (Disrupting Mobile Phones and PCs): Smartphones, like the iPhone, are built upon breakthroughs in touchscreen technology, mobile processors, and wireless communication. They disrupted the traditional mobile phone market and even impacted the personal computer industry by providing a more convenient and versatile alternative.

• E-commerce (Disrupting Retail): Amazon and other e-commerce platforms leveraged breakthroughs in internet technology, logistics, and online payment systems. They disrupted brick-and-mortar retail by offering a wider selection, competitive prices, and the convenience of shopping from home.

• Ride-sharing (Disrupting Taxis): Companies like Uber and Lyft utilized breakthroughs in GPS technology, mobile apps, and real-time data processing. They disrupted the taxi industry by offering on-demand rides, transparent pricing, and a more user-friendly experience.

Most technological breakthroughs undergo several epochs/phases before they are finally usable enough (commercially viable) to cause disruption in markets (impacting demand curves). Not all breakthroughs lead to commercially viable products or services that can disrupt markets.

Sustaining innovations, on the other hand, are incremental improvements that maintain and enhance the competitiveness of existing products or services within established markets. Unlike disruptive innovations that create new markets or alter existing ones (by shifting demand curves), sustaining innovations focus on enhancing the features, performance, or efficiency of current offerings. These innovations are important for businesses to keep pace with market demands and ensure customer satisfaction and continued market relevance.

If we put it all in a chart, it would start with an incubation phase, where breakthroughs can emerge but remain “researchy”, lacking application and viability. As these breakthroughs evolve, a usable product could be developed, getting us to the Adoption phase. This stage typically shows rapid growth due to Disruptive Innovation, where the application of breakthrough technology reshapes markets and unlocks new possibilities by fulfilling previously unmet needs. With time, the innovation enters the Sustaining phase, where growth stabilizes and is driven by incremental enhancements to the existing product. However, failure to maintain or sustain can lead to a decline, underscoring the importance of incremental improvements to remain competitive (even starting a new “S-curve” with another disruptive application).

This “S-curve of Innovation” shows the evolution from breakthroughs to commercially viable products that can be disruptive but must be sustained. 

To go a step further, here is a diagram that shows how innovations interact with supply and demand dynamics, leading to different types of market impacts.

Now that we understand the types of innovation, let’s explore in more detail the main drivers of disruptive innovations, especially nonconsumption.

Nonconsumption

It does not matter if there is a breakthrough if no one can make use of it. As described in the previous section, breakthroughs can only become disruptive if they can be consumed/used, which is not always the case. Welcome nonconsumption!

Nonconsumption typically describes a scenario where potential customers are underserved by the current market offerings. These customers either cannot afford the existing solutions, do not have access to them, or find that the solutions do not perfectly meet their needs. Nonconsumption can be an opportunity for businesses to innovate and create products or services that address these unmet latent AND blatant needs, thus turning nonconsumers into consumers. It is also a risk to established companies (aka incumbents) that are too focused on serving their existing customer base but forgo opportunities outside of their comforting bubble, leaving more breathing room for startups (the new entrants). 

Nonconsumers usually fall into one of the two categories: 

• Underserved nonconsumers: those who do not have access to a product or service due to various barriers such as cost, lack of information, lack of infrastructure, etc.  For example, due to a lack of infrastructure, people in rural areas without access to high-speed internet are nonconsumers of streaming services/social networks and broader knowledge in general.

• Overserved nonconsumers: those who do not use a product or service because it exceeds their needs or is too complex. For example, someone who only needs a basic phone for calls and texts but is offered a high-end smartphone with features they won't use is an overserved customer:

Clayton Christensen introduced me to the concept through his book, “The Innovator’s Solution.” In it, he discusses how new-market disruptions can target non-consumption by creating products that enable a larger population of people, who previously lacked the money or skill, to begin using a product and doing the job for themselves.  Also see Targeting nonconsumption: The most viable path to growth - Christensen Institute 

Note: brand plays a role here as well. Sometimes, nonconsumption can be by design via positioning/targeting a specific slice of consumers. This usually happens via brand design, that said, even then companies with great brands are at risk of disruption. We probably will talk about this in more detail in upcoming posts. 

Subscribe now

AI Winter Nonconsumption

In part I, we journeyed through the peaks and valleys of AI history and covered AI winters. However, we stopped short of exploring the reasons behind them. Winter came, multiple times actually, but WHY? Let’s go behind that wall today!

Take a quick look at your AI scroll, and cast your mind back to the mid-80s, when AI had one of its highest peaks in history. Back then, expert systems were seeing some signs of commercial viability as companies such as IBM, FMC, Toyota, American Express, and others started to find use cases for it. This led to renewed excitement and hope up until 1987, when expert systems started to show limitations and struggled to handle novel information and situations that fell outside its pre-programmed knowledge base, i.e., expert systems underserved consumers, and the tech was way behind in serving the needs properly, as a result, was non-consumable.

ℹ️ The spike in the graph does not reflect the demand curve but rather content generated, which is a rather poor proxy to adoption, but it’s all that we have in that timeline beyond the narratives described in the previous post.

Parallel to expert systems, many theories were getting close to actionable forms, especially for neural networks with Hopfield nets,  Boldtzman machines, perceptrons, and backprop networks, see A Very Short History of Artificial Neural Networks | by James V Stone for more details. So at this point, the mid 80s, most of AI theories were formulated. Since the 50s, there have been multiple knowledge breakthroughs, yet no disruption on the horizon (remmember disruption hinges of demand). One could say that expert systems ALMOST did it, but it was a short dream back then. 

So really, why?

With the benefit of hindsight, the field of artificial intelligence stems from the research originally done on Hopfield nets, Boltzmann machines, the backprop algorithm, and reinforcement learning. However, the evolution of backprop networks into deep learning networks had to wait for three related developments: 1) much faster computers, 2) massively bigger training data sets, and, 3) incremental improvements in learning algorithms ~ from A Very Short History of Artificial Neural Networks | by James V Stone 

One way I would sum it up is AI winters were a result of a negative consumption gap where expectations of what’s possible from AI exceeded what was being delivered, due to many factors such as much faster computers, massively bigger training data sets, and incremental improvements in learning algorithms. The figure below illustrates the consumption gap. 

These are but three factors, let’s dig deeper.

Not Accessible by Design 

When Ed Feigenbaum's expert systems came to life when the IBM 701 was connected to the early ARPANET, the reach was very limited. Only a select few researchers had the privilege of contributing and accessing shared knowledge (which was a massive boost by the way compared to how computing was done). This rhymes well with Christensen's wording of nonconsumption: a potentially transformative technology was out of reach for the vast majority due to restrictions and a lack of infrastructure (in that case, access to the server and the knowledge to evolve and do more research). 

In addition to the lack of accessibility, there were multiple “Tech Have-Nots” that the entire field required to grow to something more usable (which is what’s needed for demand to increase). 

The "Tech Have-Nots"

The 80s saw a surge of interest in artificial intelligence, but progress was subpar for many reasons:

• Limited Internet Access: While ARPANET existed, it was primarily restricted to academic and research institutions. The concept of a globally interconnected "World Wide Web" only started to sprout in the late 80s. This meant that collaboration and knowledge sharing were pretty much confined to small, isolated groups.

• Data Scarcity: High-quality datasets were hard to come by, making it difficult to train effective AI models. The concept of "big data" was far off, and the ability to collect and store vast amounts of information was limited. Most 

• Hardware Constraints: computers of the 1980s had limited processing power and memory (speaking from the future, easy to judge). This made it difficult to run complex AI algorithms, hindering research and development.

• The "Haves" and "Have-Nots": Even with access to the limited resources available, significant disparities existed. Well-funded institutions and researchers, like Ed Feigenbaum with his DARPA-backed IBM 701, had an advantage over others who lacked access to such cutting-edge technology.

To put those limitations in perspective, let's have a look at some data and explore NetTalk, one of the first manifestations of Neural networks that was designed to learn how to pronounce written English text.

In the 80s, I’d say NETtalk, with its 18,629 adjustable weights and 1000 data points, was the marvel. It couldn’t do a whole lot though with that much data and those weights compared to a GPT-3, which is when we first realized how powerful neural networks can be when trained on the right amount of data using enough hardware for the outcomes to make sense.

Compared to NetTalk, GPT-3, is a general-purpose model with 175 billion parameters and a trillion data points, an exponential growth in model size over the decades. This is a result of leaps in computational power which was not an overnight quest. In addition to knowledge boosts (e.g., with Transformers), there were many incremental advancements in algorithms, the internet, hardware, and data availability. GPT-3, as the first truly usable form of a language model based on deep neural networks (we had GPT-2, but wouldn’t say that was viable), showcases the difference between a theoretical breakthrough in science and its readiness and commercial variability (i.e., the time, the research, the hardware, the effort it takes to beat nonconsumption).

The AI Spring

Fast forward to the present. Where are we now? Have we solved nonconsumption? Let’s see.

We now have the internet! While not everyone has access to it, compared to the ARPANET in the 80s, about 66% of the world population now (2024) does. It’s still a major issue of nonconsumption on its own, but we made progress! Using the internet, anyone can share, collaborate, and access data/knowledge.

This widespread of the mother internet has led to an explosion of user-generated data, via social media, Wikipedia, articles, and papers, creating large datasets known as "big data". This abundance of data has become the main resource for training today’s AI and large language models, enabling them to learn and improve from diverse and extensive information. Contrast that with the limited availability of data in the 1980s, which was a choke point for further development and advancement of AI and neural networks (NetTalk had a 1k dataset, GPTs datasets are in billions and going trillion). 

We have also made progress on the computing front. The development of parallel processing with GPUs and chips specifically designed for AI workloads (e.g., TPUs) has been a game-changer. This leap in computing made it possible to train large and complex deep-learning models on big datasets, which was simply not feasible in the 1980s due to the limitations of hardware at the time.

Research continued, never stopping, which also led to breakthroughs in AI techniques and algorithms. These algorithms, particularly for deep learning and neural networks, have enabled certain applications of AI to automate many of the tasks that only humans were capable of. Techniques such as convolutional neural networks (CNNs) and transformer models have been crucial in getting us to where we are today. In the 1980s, the algorithms were just not as advanced or effective.

In addition to computing, data, and widespread of knowledge, we saw the rise of open-source software which is helping democratize access to AI and machine learning software. Tools and libraries such as TensorFlow, PyTorch, and others are accessible to anyone with an internet connection, allowing for a global participation in AI development. This openness not only accelerates innovation but also reduces the entry barriers that once existed due to proprietary systems. Open source also gave birth to new business models focused on building value not gating IP. 

Is it Summer yet then? 

Not quite! Despite the improvements, the supply side of compute for AI is still highly inaccessible. Training state-of-the-art large language models requires massive compute resources costing millions of dollars, primarily for high-end GPUs and cloud resources. The costs have been increasing exponentially as models get larger. Only well-resourced tech giants and a few research institutions can currently afford to train the largest LLMs. It actually fits a power law quite nicely, the major players having enough capital and access to data through their current operating business,  so you will find that a minority of companies have access to the majority of compute/data (more about the AI market in a previous post).

Additionally, current economics and global tensions around trade policies have led to what is often referred to as a "chip war", impacting the production and distribution of advanced semiconductor chips. This situation introduces disparities in computational access. Countries and companies that can produce or secure these chips have a competitive edge in developing and deploying AI technologies. So yes internet solved some of it, politics always gets in the way 🙂

What about data? According to scaling and chinchilla laws, model performance in language models scales as a power law with both model size and training data, but this scaling has diminishing returns, there exists a minimum error that cannot be overcome by further scaling. That said, it’s not unlikely that we will figure out how to overcome this in the near future. 

For a fixed compute budget, an optimal balance exists between model size and data size, as shown by DeepMind's Chinchilla laws. Current models like GPT-4 are likely undertrained relative to their size and could benefit significantly from more training data (quality data in fact). Future progress in language models will depend on scaling data and model size together, constrained by the availability of high-quality data.

Beyond data, and despite the progress, AI still faces technical limitations such as the lack of common sense reasoning, vulnerability to adversarial attacks, and difficulties in generalizing from training data to new, unseen situations, not to mention hallucinations. Finally, current models can be more “creative” than we want them to sometimes, which makes them hard for tasks that require reproducibility or accuracy. So we are not quite there yet. 

So is it Disruptive Innovation or AI Winter 2.0?

Generative AI seed funding drops 76% as investors take wait-and-see approach. On one hand, $50 billion has been invested in AI, yet only $3 billion in revenue has been generated. One could argue that this is upfront capital deployed to establish infrastructure and capacity for future earnings (an investment), however, today’s GPUs lose value quickly, especially as newer, more efficient models are continually released, so it’s hard to tell!

This reality is causing investors to pause and reassess their strategies. Past AI hype cycles, like expert systems in the 80s and Japan's 5th generation program early 90s, remind us that overenthusiasm can sometimes lead to disillusionment (see part I of this series).

That said, dismissing AI's potential for disruption would be a mistake. Unlike previous iterations, today's AI, despite its flaws and hallucinations, offers some value (might not be crystal clear, but the productivity gains are not to be reckoned with). This could potentially slow down an "AI winter" as businesses find ways to integrate AI into their workflows, subject to the value perceived. 

History repeats itself in a way, we have seen a similar pattern before with the dot-com bubble, where there was a surge of investment in companies, often with inflated valuations and unrealistic expectations, which eventually led to a market correction, with many startup companies failing or being acquired. We're likely to see the same, where the weaker players are likely to struggle and sway away. As discussed in a previous post, existing incumbents and larger players have an existing business model that does NOT revolve around JUST AI, it’s ads, e-commerce, software, consulting, and so on. This gives those bigger players the runway they need to fight the long fight, make bets, and potentially acquire some winners from the new entrant's pool, power law again! 

So to answer the original question, I don’t think we will see a “bad” AI winter soon (we might see an autumn but not a winter), at the same time, there is still the debate on whether the current version of AI, though way better than the 80s, qualifies as disruptive innovation on it’s own. I clearly see the potential for AI, the technology, as a disruptive force, but I lean more toward calling the current versions of its application sustaining innovations, especially since we are seeing how it can accelerate and optimize revenue streams of existing incumbent companies as they incorporate it into their pre-existing fly-wheels. 

That said, applications that make use of AI to target areas of nonconsumption with more affordable and accessible means to solve problems can unlock new markets, and become disruptive. Here are a couple of examples: 

• AI as an enabler for personalized education for underserved students in areas where education is infeasible and not accessible.

• AI-powered diagnostics expanding healthcare access in developing countries where healthcare is infeasible and not accessible.

The question becomes whether the tech powering today’s AI is enough to give birth to those disruptive innovations (the data, the compute...). I’d say not yet, what say you?

That’s it for this series! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

That said, for someone who wants to understand the market or even start an AI business, it’s important to build awareness of the landscape, how those companies position themselves in the market, what the main products they sell, how they make money, and finally, how does AI contribute their business livelihood. That’s exactly what this post is about. 

In this post, we will explore the AI Market Dynamics Quadrant. We will focus on how open or closed the offerings from those companies are when it comes to generative AI models (open vs. closed) and how coupled the models/GenAI in general are to their business model (direct vs. indirect).

Subscribe now

The Glossary

We will be segmenting the AI market into four key categories based on two dimensions: the openness of the AI model (open vs. closed) and the integration of the AI model into the business model (direct vs. indirect sell).

• Open Models: AI models that are accessible to the public, often leveraging permissible licenses to share the weights, code, architecture, etc, or community licenses (with conditions), to encourage usage and contribution. They also usually rank higher in the foundational model transparency index. 

• Closed Models: AI models that are proprietary, with controlled access to ensure exclusivity and competitive advantage (the model here is usually the MOAT).

• Direct Business Models: Companies that monetize AI models directly through subscriptions, licenses, or selling the technology itself. The model is the product!

• Indirect Business Models: Companies that use AI models to enhance and complement their existing products and services, rather than selling the AI models directly. The model is the proxy!

These are all words you’ll need to follow along and understand the quadrant. That said, let’s try to get more insights into how the companies in the quadrant make money today before starting to analyze the dynamics and the landscape through the quadrant.

The Players: Incumbents & New Entrants

In the landscape quadrant, I didn’t include all the companies in the world. I sampled based on activity and heat, i.e., these are the top players potentially with the largest influence and diffusion power on the AI scene.  

Let’s start with the incumbents, the well-established, those who know what they are doing, the seniors. 

Sample Incumbents

Below is a sample of incumbents. 

IBM

Software is IBM's largest revenue driver. It includes Watsonx (AI and data platform), Red Hat (Hybrid Cloud, RHEL, OpenShift Platform, and Ansible), security and zero-trust, automation, and transaction processing (which powers IBM Z). 

In second place is IBM Consulting, which is a key driver for digital transformation for many organizations. It also partners with other large enterprises AWS, Microsoft, and SAP, and utilizes Watsonx for procurement solutions. 

In third place is IBM Infrastructure, which ensures reliable, secure solutions for critical workloads across public, private, hybrid clouds, and the edge. Its strengths include transaction processing on IBM-Z and AI-driven transformation. 

Finally, IBM financing which supports the acquisition of IBM's technology solutions and services by providing various financial options, including loans, leases, and customized payment plans. 

Generally, generative AI models are not the main revenue stream here. IBM makes money by providing comprehensive tech solutions and transformation services and leveraging open-source models to drive consulting and software sales.

Microsoft

Microsoft generates revenue through a diverse range of products and services, primarily categorized into three main segments: Intelligent Cloud, Productivity and Business Processes, Personal Computing, and as of very recently, AI-enabled services. Here is a detailed breakdown of these revenue streams, followed by a simplified revenue distribution based on the available data: 

• Intelligent Cloud segment is Microsoft's largest revenue driver, which includes Azure (their cloud), SQL Server, Windows Server, GitHub, and Enterprise Services. Azure, in particular, is a major component, offering a wide range of cloud services and holding a 24% market share, second only to AWS. This segment accounts for 38% of Microsoft's total revenue. 

• Personal Computing is the second largest revenue driver. It encompasses Windows (operating system), Surface (devices), Xbox (gaming consoles and services), and search advertising (Bing). This segment aims to enhance the user experience across various devices and platforms. In 2023, this segment represented 25% of the total revenue. 

• Productivity and Business Process is the third largest revenue driver. This includes Office Commercial products and cloud services (Office 365), Office Consumer products and cloud services (Microsoft 365), LinkedIn, and Dynamics 365. This segment focuses on enhancing corporate productivity, communication, and information services. In 2023, this segment represented 23% of the total revenue

• AI-enabled services, particularly through the Copilot platform, have become a major component of Microsoft's revenue streams. Copilot, integrated into Microsoft 365 and other products, leverages AI to enhance productivity and user experience. In 2023, AI-enabled services, including Copilot, comprised of 6% of Microsoft's revenue.

Generally speaking, Microsoft does not sell AI directly, it infuses it into their products/services like Office 365, search and Azure. It has also a pretty even and diverse set of revenue streams. Worth noting that the slightest capture in search from Google would yield considerable margins.

Google

Google's revenue model is more or less advertising driven, i.e., it remains by a large margin the primary source of Google’s income. However, it has diverse revenue streams. In addition to search and ads, it has Google Cloud, subscriptions, platforms, and devices, as well as its other bets segment (which gets rotated/killed often). 

The diversity of Google’s revenue stream ensures it’s presence in both consumer and enterprise markets. Let’s do a brief break-down to understand a bit more (data from Alphabet’s q4 earnings report): 

• Advertising is Google's largest revenue driver, accounting for the majority of its income. This includes ads on Google Search, YouTube, and Google Network properties. In 2023, Google Search, YouTube, and Google network Ads contributed ~77% of Google’s total revenue (lion’s share). 

• Google Cloud comes in second place, which includes Google Cloud Platform (GCP) and Google Workspace. Google Cloud provides infrastructure, data analytics, and collaboration tools for enterprise customers. In 2023, Google Cloud contributed about 11% to total revenue, driven by the increasing demand for cloud services and AI tools.

• Subscriptions, Platforms, and Devices are the third largest revenue driver. This includes revenue from YouTube Premium, YouTube TV, Google Play, and hardware sales such as Pixel smartphones and Google Nest devices. In 2023, this segment contributed 10.1% of the total revenue. 

• Other bets: Finally, Google's Other Bets segment includes various experimental and emerging businesses such as Waymo (self-driving cars), Verily (life sciences), and other innovative projects. Although this segment is smaller, it represented 0.7% of the total revenue.

Generally speaking, Google makes money through advertising, cloud services, and enhancing user retention across its suite of products. Google uses (or will use) AI models to improve its services, be it search, its Gsuite or its cloud offerings.

Compared to other incumbents, Google’s revenue stream might be diversified in numbers but not in weight. Google heavily relies on its Ads business, any threat to this stream canbe an existential threat to Alphabet/Google. 

Amazon

Amazon's revenue model is characterized by its strong focus on e-commerce, cloud computing, and advertising. The company leverages its extensive product selection, efficient delivery services, and robust technological infrastructure to drive growth across various segments. AWS, in particular, plays a crucial role in Amazon's profitability, while advertising and subscription services continue to expand, contributing to the company's overall financial success.

• Online stores are Amazon's largest revenue driver, which includes first-party sales of products directly to consumers. This segment accounts for approximately 40% of Amazon's total revenue. The online stores segment benefits from Amazon's extensive product selection, competitive pricing, and efficient delivery services

• In second place are Amazon's third-party seller services, which include commissions, shipping fees, and other services provided to third-party sellers on Amazon's marketplace. This segment accounts for about 24% of the total revenue. The third-party marketplace allows Amazon to expand its product offerings without directly managing inventory, enhancing its overall product selection and customer experience

• AWS is Amazon's cloud computing division and the third-largest revenue driver. It provides a variety of services, such as computing power, storage, and databases, to businesses and developers. In 2023, AWS accounts for approximately 16% of Amazon's total revenue. AWS is also the most profitable segment, contributing significantly to Amazon's operating income

• Amazon's advertising business has experienced rapid growth, offering targeted advertising solutions to brands and sellers. This segment makes up about 8.2% of the total revenue. Amazon's advertising services include sponsored ads, display ads, and video ads, which help brands reach a wide audience on Amazon's platform

• Amazon's subscription services, including Amazon Prime, Kindle Unlimited, Amazon Music, and Audible, are another revenue stream. In 2023, subscription services represented about 7.1% of the total revenue. Amazon Prime, in particular, fosters customer loyalty and drives additional sales through its ecosystem of benefits.

• Amazon's physical stores, including Whole Foods Market and Amazon Go, accounted for approximately 3.5% of the total revenue in 2023. These stores complement Amazon's online presence and provide additional channels for customer engagement

• Finally, Amazon's "Other" segment, which includes various smaller revenue streams such as co-branded credit card agreements and certain licensing and distribution of video content, makes up about 0.8% of the total revenue

Amazon's diverse revenue streams, including online stores, AWS, subscription services, and advertising, allow it to withstand losses in other areas due to its well-rounded portfolio, making it less dependent on AI. That said, in the AI world, AWS exposes a family of models through AWS bedrock led by Anthropic’s Claude, to which it has committed up to $4 billion in investments, and provides infrastructure through inferentia and trainium for inference and training, amongst the normal storage/compute/networking and the wide selection of instances. We will talk more about Anthropic and Claude in a bit. 

Apple

Apple's business model is centered around product design and vertical integration. Apple’s focus on user experience and design, helped it build a brand with high equity, resulting in a very high willingness to pay from its customer base (especially as you own more Apple devices the value offered becomes higher). The company's revenue streams are diversified, with the iPhone being a major driver, supported by services like Apple Music and the App Store, as well as other products like the iPad, Mac, and Apple Watch. Here is a quick breakdown (sources: How Apple Makes Money and Apple Statistics):

• Apple's largest revenue driver is the iPhone, accounting for around 52% of the company's total revenue.

• Apple's Services are second, representing approximately 22% of total sales. Services include the App Store, Apple Music, iCloud, Apple TV+, Apple Arcade, and various other subscription offerings.

• The third largest revenue contributor is Wearables, Home, and Accessories (10.4% of total revenue). This segment encompasses products like AirPods, Apple Watch, HomePod, Apple TV, Beats headphones, and other accessories.

• Next is the Mac lineup of computers, including MacBook Air, MacBook Pro, iMac, and others making up around 7.7% of Apple's total sales.

• Finally, the iPad tablet lineup, consisting of models like iPad Pro, iPad Air, and iPad Mini, contributing about 7.4% of Apple's revenue.

Apple has been lurking in the shadows when it comes to  AI compared to other incumbents, but the company has made significant investments in this area. Since 2017, Apple had 32 AI startup acquisitions under it’s belt, focusing on technologies like self-driving cars, voice design, music generation, and image recognition. These investments have led to the integration of AI features across Apple's product lineup, such as personal voice, live voicemail transcription, and advanced health monitoring on the latest iPhones and Apple Watches. Apple should also not be take lightly when it comes to the potential for personalization with AI, after all they ace the wearables! 

Apple's AI strategy involves a hybrid approach, leveraging both in-house and external resources. The company is investing in generative AI and plans to "break new ground" in this area, as stated by CEO Tim Cook. Apple is also exploring partnerships with AI companies like OpenAI and Google to integrate their large language models into future products and services.

In summary, Apple's revenue model is driven by its diverse product (mostly physical) lineup, with the iPhone and Services being the primary contributors. 

Meta

Meta's revenue model is dominated by advertising, with growing contributions from Reality Labs, Payments, and Other Fees. Let’s break it down: 

• Advertising is Meta's largest revenue driver, generating approximately 98% of the company's total revenue. This includes ad placements across its family of apps, such as Facebook, Instagram, Messenger, and WhatsApp. Meta's advertising revenue success is driven by its sophisticated ad targeting, large user base, and continuous enhancements in ad formats and measurement tools.

• Reality Labs, Meta's division focused on virtual and augmented reality, generated around 2% of total sales. This segment includes hardware like Oculus VR headsets and software related to the metaverse. Despite still being a smaller revenue contributor, Reality Labs is a significant part of Meta's long-term strategy to lead in the development of the metaverse and immersive experiences.

• Meta's Payments and Other Fees segment makes up less than 1% of the company's total revenue. This includes fees from Facebook Pay, peer-to-peer payments, and other financial services. Although currently a minor part of the revenue mix, Meta is investing in expanding its financial services offerings to create new revenue streams.

Meta's revenue model is dominated by advertising, with growing contributions from Reality Labs, Payments, and Other Fees. That said, Meta has been an active contributor (actually a solid contributor, e.g., PyTorch, Llama, etc.) to the AI scene. 

AI is not the direct sell, nor it is the main revenue stream, but making one of Meta’s dependencies a commodity (the foundation model), helps it scavenge talent from the entire tech landscape to make its models better. In doing so, it helps it make material improvements for its ad business via better content recommendation, enhanced ad targeting,  automatic content moderation, translation, and copy creation. As ad performance improves, Meta attracts more advertisers and commands higher ad rates, ultimately boosting their revenue. Not to mention that Meta’s “open” approach enhances their brand reputation and attracts top talent. 

Summary for Incumbents

Technology incumbents like IBM, Microsoft, Google, Amazon, Meta, and Apple have long-established revenue streams from their core businesses, such as cloud computing, operating systems, search engines, e-commerce, and social media. They have existing means to make money (not AI). AI can be important in these industries mainly as an enabler for their core business and revenue streams but is not the main source of income. 

There are exceptions, though. For example, Google is in the Ad business. Ads happen when you are able to connect the consumer with the advertiser. Google does this by offering “free” search (after selling the soul) and then surfacing Ads customized to the consumer. Facebook collects data based on social browsing and customizes ads (via cookies, again selling the soul) to improve click-through rates and bring them to the consumer, and so on. 

In that case, where there is no actual product/software/service being sold but the Ad (as the main source of revenue), AI can be detrimental. If there is a better way to connect the consumer with the advertiser, it becomes a threat, hence why Google/Meta are strong players in the AI and foundational models game besides the AI new entrants (we talk more about those next) whose main mission in the world is to thrive of generative AI. 

Below is an aggregation of incumbents' revenue stream distribution charts. If you look closely, you will see that there are a few with a balanced staircase distribution where all the streams are contributing their fair share to the aggregate revenue (e.g., IBM, Microsoft, Amazon, Apple). On the other hand, there are those that rely heavily on one stream (e.g., advertisement) to varying degrees (Meta wins in its sole dependence on Ads, followed by Google!). 

Also, some have their streams under one category (e.g., Apple is focused on devices, wearables, and physical products), while others horizontally surf different verticals (cloud, e-commerce, devices, etc). 

Sample New Entrants

Below we will cover new entrants in the tech and AI scene. They don’t necessarily make money (profit) at this point, but we will quickly cover their business model to understand better how they fit in the AI Market Dynamics quadrant.

Mistral

Mistral AI's revenue model is driven by its AI models and services, enterprise solutions, and open-source offerings. The company's investments in AI research and its partnerships with leading tech firms have positioned it as a key player in the AI industry, with a strong focus on performance, efficiency, and customization. Let’s do a quick breakdown: 

• AI models and services are Mistral AI's largest revenue driver. This includes their flagship models like Mistral Large and Mixtral 8x22B, available through APIs and platforms such as Amazon Bedrock and Google Cloud's Vertex AI (also on Azure as of recently). These models serve various applications, including text summarization, question answering, and code completion. Mistral’s key play here is high performance and cost-efficiency for their models. 

• In second place are Mistral AI's enterprise solutions, which provide tailored AI capabilities to businesses across various industries. These solutions include custom AI development and fine-tuning services, allowing enterprises to adapt Mistral's models to their specific needs. Mistral AI collaborates with European integrators and clients to co-build solutions, ensuring that the models meet industry-specific requirements and compliance standards

Generally, Mistral focuses on providing open AI models through APIs and partnerships. They generate revenue through collaborations and by offering services that integrate their open AI models into other businesses.

OpenAI

OpenAI, which began as a research lab in 2015 and transformed into a for-profit organization in 2019, generates revenue primarily through API consumption fees for its foundational models (the GPTs), used by startups, and even large enterprises to create AI-infused applications (mostly also for prototyping). But also, through premium/plus subscriptions which give access to more features like the create your own GPTs, the GPT store, better models (GPT4/4o/turbo/...), and more input/output tokens to use those models. 

One of OpenAI’s largest partners/distributors is Microsoft, which is also a minority owner. In that case, Microsoft earns revenue through hosting OpenAI's APIs on Azure and distributing them to users, as well as through the sale of applications built on Azure. The partnership between OpenAI and Microsoft has led to new products like GitHub Copilot, and OpenAI's generative models may be integrated into Microsoft's offerings like Bing, Office 365, and now CoPilot + PC (similar to rewind.ai but eats more data, stay put and take care of your data 🙂).  

On the other hand, Microsoft Azure serves as the foundation for OpenAI's AI infrastructure, enabling the pre-training of models, and OpenAI's APIs are distributed through Azure, allowing users to access them directly or through the Azure Enterprise platform (a wider variety of users). 

OpenAI also has a somewhat spaghetti corporate structure. We will not discuss it in this post, but I left the diagram below for your own leisure. 

Control Alt delete: Exploring OpenAI's corporate structure, after Sam Altman’s shock dismissal

In general, OpenAI uses a closed model, offering products like ChatGPT through subscriptions, APIs, and licenses. The main monetization is achieved through direct sales to personal and enterprise customers. I.e., the model is the product! 

Anthropic

Like OpenAI, Anthropic's revenue model is driven by its mostly AI services, enterprise solutions, and custom AI development. The company's investments in AI research and its partnerships with leading tech firms have positioned it as a key player in the AI scene. Their AI game is around safety and ethical AI deployment. 

Here is a quick breakdown: 

• AI services are Anthropic's largest revenue driver. This includes the Claude family of models, such as Claude 3 Opus, Sonnet, and Haiku, which offer AI capabilities for tasks like reasoning, content creation, and coding. These models are available through APIs and platforms like Amazon Bedrock and Google Cloud's Vertex AI. 

• In second place are Anthropic's enterprise solutions, which provide tailored AI capabilities to businesses across various industries. The Team plan, for example, offers access to all three of Anthropic's latest Claude models with expanded usage limits, admin tools, and billing management. This plan is designed to support multi-step conversations and process lengthy documents targeting sectors such as technology, financial services, legal services, and healthcare. The Team plan costs $30 per user per month. 

• The third-largest revenue contributor is custom AI development, where Anthropic collaborates with clients to develop bespoke AI solutions (individual clients design custom AI solutions). This includes dedicated capacity options for high-throughput needs, ensuring that clients can access the necessary resources to support their extensive AI applications (I.e., the infrastructure required to build customized AI solutions). 

To add, similar to OpenAI and Microsoft, Anthropic and AWS are strategically collaborating with AWS becoming Anthropic’s primary cloud provider. Earlier in 2024, Anthropic made a $4 billion investment in Anthropic, securing minority ownership in the company. Anthropic is set to use AWS’s infrastructure (including AWS’s Trinium and Inferntia chips) to build, train, and serve its foundation models. This gives AWS early access to Anthropic’s models (with options to customize/finetune) and exposes Anthropic to AWS's massive customer base. The models are available through Amazon Bedrock.  The play here is mainly centering around responsible AI as a differentiating tenet. 

Generally, Anthropic provides closed AI models, focusing on products like Claude and recycles that as subscriptions for directly interacting with the models, through API consumption, or licenses for partners to use their models, which benefits both the partner and Anthropic. The main monetization is achieved through direct sales to personal and enterprise customers. I.e., the model is the product! 

Summary for New Entrants

In a sense, the business models for Mistral, OpenAI, and Anthropic are quite similar. Subscriptions, APIs, and partnerships with one of the incumbents to give them access to people and infrastructure. Each tries to go for a unique play. Anthropic goes for ethical and responsible AI, Mistral goes for lean, efficient, and open models, and finally, OpenAI goes for AGI and best-in-breed quality. 

That said, they all have one thing in common. Unlike the Incumbents who have various revenue streams making them money and can potentially loss-lead for a long while, this group operates on the basis that the model IS the main product and the revenue stream. Take the model away, poof, they are gone, finito! 

Analysis & Categorization 

No, let’s summarize all the above and explain the quadrant. There will be four categories, namely: 

• Open & Direct: Companies that pushed their foundational models to the open, and still rely on the model being the product. 

• Open & Indirect: Companies that pushed their foundational models to the open, but don’t rely on the model being the product, rather the proxy.  

• Closed & Direct: Companies that DID NOT push their foundational models to the open, and still rely on the model being the product. 

• Closed & Direct: Companies that DID NOT push their foundational models to the open AND don’t rely on the model being the product, rather the proxy.  

Open and Direct

Mistral is located here, among other new players (which are out of scope for this post). Mistral focuses on providing open AI models directly through APIs and partnerships, as described in the new entrants overview. Whether that will sustain a living or not is unclear for now, but they are definitely moving more towards offering their premium models (time leaps ahead) with a higher price tag, and they won’t be planning to open-source those models. 

Open and Indirect

IBM and Meta fall here. 

IBM utilizes open AI models (while wearing its open-by-default Red Hat to provide accessible training/finetuning via InstructLab) within its broader suite of software and consulting services. It enhances its offerings by integrating open-source models and providing comprehensive tech solutions without directly selling the AI models or at least not as their main revenue stream. IBM pushed their Granite models weights/arch/.. under the Apache license (a very permissive license). IBM also ranks highly in  the Model Transparency Index (as of May 2024). 

Similarly, Meta, which relies mainly on Ads to make a living, is releasing its models under a community license. In doing so, it gets contributions from the entire world to improve its internal services (for ad targeting), which helps it attract more advertisers and command higher ad rates, ultimately boosting its revenue. Not to mention that Meta’s “open” approach enhances its brand reputation and attracts top talent.

You might have noticed that I didn’t mention the eye of Sauron (Jensen Huang / Nvidia). I didn’t feel I needed to, they are right in the middle of it (see the quadrant), and no matter what happens, their chip rules them all (for now)!

”𝘍𝘰𝘳 𝘦𝘷𝘦𝘳𝘺 $1 𝘴𝘱𝘦𝘯𝘵 𝘰𝘯 𝘕𝘝𝘐𝘋𝘐𝘈 𝘈𝘐 𝘪𝘯𝘧𝘳𝘢𝘴𝘵𝘳𝘶𝘤𝘵𝘶𝘳𝘦, 𝘤𝘭𝘰𝘶𝘥 𝘱𝘳𝘰𝘷𝘪𝘥𝘦𝘳𝘴 𝘩𝘢𝘷𝘦 𝘢𝘯 𝘰𝘱𝘱𝘰𝘳𝘵𝘶𝘯𝘪𝘵𝘺 𝘵𝘰 𝘦𝘢𝘳𝘯 $5 𝘪𝘯 𝘎𝘗𝘜 𝘪𝘯𝘴𝘵𝘢𝘯𝘵 𝘩𝘰𝘴𝘵𝘪𝘯𝘨 𝘳𝘦𝘷𝘦𝘯𝘶𝘦 𝘰𝘷𝘦𝘳 4 𝘺𝘦𝘢𝘳𝘴" ~ 𝘕𝘝𝘐𝘋𝘐𝘈 𝘊𝘍𝘖

From silicon to software, each layer of the stack adds value by leveraging the "𝘳𝘢𝘸 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭" of the layer below. For the cloud providers (IBM, AWS, GCP, Azure, etc.), the raw material is the NVIDIA chip (or the AMD, Intel, etc.). For the Model/Inference API providers like Claude/OpenAI/Mistral, it's the infra to train/serve. For startups, wrappers, and application providers, it's the inference APIs, and the chain continues.

The success of the provider hinges on the success of the consumer and vice versa with varying dependencies.

Subscribe now

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

Everyone is speed publishing, according to the AI Index report from stanford. Every year, the number of AI publications increases by the thousands as more researchers join the AI ranks.  With all this progress and FOMO, it's easy to lose sight of the long, winding road that led us here, I mean, it didn’t happen overnight, ChatGPT didn’t happen overnight, Artificial General Intelligence (AGI) will not happen overnight, its foundation upon the foundation, it could take years, decades, even centuries sometimes!

This post takes a step back into the 1930s (and even before), 50s, 60s, 70s, and 80s, when most of the concepts behind what we know as AI today were forged. Here, we take a trip together down memory lane. In this post, we are going to explore the key events in history, meet key people who made it all happen, look into how they influenced AI, and how AI was posed as the ethos of computer science, the ultimate goal upon which all grounding concepts were built.  We will also look at AI winters, why they happened, and where we are today. 

Note: As I was writing this, I wrote way more sections than I should have, so I decided to break it down into two parts (it will not likely fit an E-mail anyway). This here is part I, in part II, we will compare and contrast history and present, look at AI and innovations in general from the business and economic perspective, and brainstorm on how to build better products catering to a larger group of consumers from the lessons learned in part I. 

Before the 1930s

The journey towards artificial intelligence began long before the 1930s. Many early thinkers and inventors laid the groundwork for future AI advancements through a series of theoretical, philosophical, and mechanical innovations. Be it the Ancient Greek myths, such as those of Talos and Galatea, the mechanical men presented by Yan Shi to King Mu of Zhou in 10th century BC China, the logical groundwork for AI with concepts such as the syllogism and means-ends analysis (ALOT of details here). Even during the medieval period, the Banu Musa brothers created a programmable music automaton, and Al-Khawarizmi’s work in arithmetic and algebra introduced the term “algorithm” (9th century). Fast forward to the 1600s, with Schickard’s first calculating clock, Hobbes, and the mechanical theory of cognition by Hobbes (19th century). Onwards to the 1700s, Jonathan Swift described a knowledge-generating machine in "Gulliver's Travels, and the 1800s where Samuel Butler speculated on machine consciousness in "Darwin Among the Machines". 

Photo from: https://zkm.de/en/automatische-hydraulische-orgel-der-banu-musa-ibn-shakir 

These early milestones/inventions/philosophies showcase the longstanding fascination with creating intelligent machines, which set the stage for AI advancements in the 20th century, the century when modern AI was born. 

The 1930s to 1950s

Alan Turing,  considered as the father of theoretical computer science and AI, produced a foundational paper in 1936 on computable numbers which introduced the concept of the Turing machine, providing a theoretical framework for future computers. By 1941, Konrad Zuse had constructed the first working program-controlled general-purpose computer, and in 1948, Norbert Wiener coined the term "cybernetics," integrating the study of control systems and communication theory

In 1950, Alan Turing wrote his paper on COMPUTING MACHINERY AND INTELLIGENCE, asking the question, “Can machines think?” He explored and simplified the definition of “intelligence” with the goal of modeling it. Additionally, he proposed the imitation game, the Turing test, as a way to test and evaluate intelligence. 

Note on the Turing Test: Today, we discuss Artificial General Intelligence (AGI) and how we get there. The Turing test is one approach to identifying that state. Some argue that the Turing test is not a practical measure of AI. It's easily gamed with simple tricks that exploit human biases to appear “human” rather than demonstrate genuine intelligence, which is arguably not the goal we have in mind. We need a more robust and meaningful assessment of AI capabilities.

In 1955, Newell, Simon & Shaw Developed the First Artificial Intelligence Program, the Logic Theorist, the first program designed to imitate human problem-solving skills. The program was created to prove theorems in propositional calculus similar to those in Principia Mathematica by Whitehead and Russell. In 1956, The program was showcased at the Dartmouth Summer Session on Artificial Intelligence in 1956, marking a key moment in the history of AI research.  

The Dartmouth Summer Research Project of 1956 is often taken as the event that initiated AI as a research discipline ~ Artificial Intelligence (AI) Coined at Dartmouth 

The Dartmouth conference brought together many of the field's great minds in one place and also formed the basis for future research and innovations in the field. In the next section, we will discuss the first artificial intelligence white paper, one of the outcomes/artifacts of this gathering of researchers and scientists.

In 1957, John McCarthy (also one of the founding fathers of AI, mainly as he coined the phrase “artificial intelligence”) introduced LISP, short for, List Processing) which later became the favored programming language for AI. 

Before we finish the 1950s, I’d like to highlight one of the very first demos of AI. If you have been around, you have probably seen “The mother of all demos'' by Douglas Engelbart in the 60s, demonstrating the computer mouse as a UX interface, but have you seen Claude Shannon’s demo on Theseus, the AI mouse? If not, you should! Theus is probably one of the first manifestations of applied “intelligence”. Shannon’s goal was optimizing telephones and relays but this remains a great token of how scientists at the time approached the problem. 

The Dartmouth Workshop | 1956

In 1955, McCarthy, then a mathematics professor at Dartmouth College, formally proposed a summer workshop to the Rockefeller Foundation. The proposal, titled "A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence" outlined the workshop's goals and marked the first official use of the term "artificial intelligence"

With funding secured from the Rockefeller Foundation, the workshop was held in the summer of 1956 on the Dartmouth College campus. While the original plan envisioned a two-month collaborative effort, the workshop ultimately unfolded as a more loosely structured gathering, with participants coming and going throughout the summer.  

The Dartmouth workshop was attended by the founders of AI, other key figures were invited to the Dartmouth Workshop as well. Among them were future Nobel prize winners John F. Nash Jr. (1928–2015) and Herbert A. Simon (1916–2001). 

A Bold Conjecture: Machines That Learn and Think

During the Dartmouth workshop, a proposal for a summer research project was presented. Here is a quote from the proposal:

We propose that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it ~ from A proposal for the Dartmouth Summer Research Project on Artificial Intelligence 

This quote encapsulated the core ambition, the vision, of AI research at the time, which can be boiled down to creating machines capable of learning, reasoning, and problem-solving on par with humans. 

During the workshop, scientists explored a wide range of topics, most of which form the basis and the core of what we know as AI. It's all there since the 50s 🙂. Amongst those topics, I find two to be quite important: 

• Neural Nets: Early researchers in the field, like Marvin Minsky, delved into the idea that artificial neural nets (networks today) could simulate the brain's learning processes. 

“It may be speculated that a large part of human thought consists of manipulating words according to rules of reasoning and rules of conjecture.”

“How can a set of (hypothetical) neurons be arranged so as to form concepts? Considerable theoretical and experimental work has been done on this problem by Uttley, Rashevsky and his group, Farley and Clark, Pitts and McCulloch, Minsky, Rochester and Holland, and others. Partial results have been obtained, but the problem needs more theoretical work.”

• Self-Improvement: An ambitious goal at the time was the concept of machines capable of self-improvement. The idea from the paper hinted at the potential for AI systems to enhance their own capabilities continuously. Are we here yet in 2024? I don’t think so, we have self-supervised learning (see my earlier post on pretraining and fine-tuning for more details) but not self-improvement as a broader concept that encompasses more than just learning from data during training/initial phases, it includes any changes and AI might make to improve its performance, capabilities, or efficiency, potentially including but not limited to learning processes. Is the AI self-aware enough? 

Not Enough Algorithm-ability

“The major obstacle is not lack of machine capacity, but our inability to write programs taking full advantage of what we have.”

This quote reminds me alot of where we are today, we have tons of computers, and we did make breakthroughs with transformers, but arguably we might be hitting scaling laws here. After GPT-4, we are in catch-up mode, but who knows, time will tell. I still think we need to spend more time doing efficiency optimizations and not just throw computers at the problem. Anyway, let’s get back to the founders!

Founding Fathers & Key Figures

The Dartmouth workshop brought together some of the most influential figures in the early history of AI:

Claude Shannon

A pioneer in information theory, Shannon's work laid the foundation for understanding how information is transmitted and processed, both in machines and in biological systems. Remember Theseus, the AI mouse 🙂? 

During my college years, we studied Shanon’s Entropy and information theory. I have since been fascinated by the simplicity of the approach, which I personally think applies not only to senders/receivers between machines in an electrical communication system but also to humans. 

We humans communicate via language. Verbally, we have accents, which are noise to the receiver. Sometimes, we are even lazy to say the whole thing or lazy to provide context, which is, again, noise. We tend to rely on repeating what we say or changing how we formulate it to improve the quality of the knowledge we communicate, that’s the loop 🙂. Actually, Shannon’s theory can be applied in many areas, even in biology! 

Marvin Minsky

Minsky's research on neural networks and cognitive science shaped the AI field for decades to come. What’s fascinating is that Minsky had built his prototype of the neural network using mechanical components, he called it the SNARC machine,  a machine “made out of about 400 vacuum tubes and a couple of hundred relays and a bicycle chain” for simulating learning by nerve nets.  The machine he built simulated rats running in a maze and the circuit was reinforced each time the simulated rat reached the goal (kinda like Theseus, it could only do one thing, solve the maze). 

“A feedback loop reinforced correct choices by increasing the probability that the computer would make them again—a more complicated version of Shannon’s method, and a level closer to how our minds really work. Eventually, the rat learned the maze.”

Conceptually this is very close to how we do it today, with optimizers and error functions! In many ways, Minsky had a “technical” vision of how things would work. 

Nathaniel Rochester

An early advocate (also an IBMer) for using computers to solve problems traditionally done by humans, Rochester helped build the first AI programs. In 1956, he published a classic article in which he and colleagues simulated a network of neurons on IBM 701 and 704 calculators. See Nathaniel Rochester III (1919-2001) - Dharmendra S. Modha 

John McCarthy

The organizer of the Dartmouth workshop, McCarthy coined the term "artificial intelligence" (which earned him the top spot amongst the founding fathers of AI) and played a crucial role in defining the field's research agenda. “McCarthy has worked on a number of questions connected with the mathematical nature of the thought process, including the theory of Turing machines, the speed of computers, the relation of a brain model to its environment, and the use of languages by machines”

McCarthy also designed the LISP programming language that was heavily pitched for use with AI, and what inspired LISP machines (the specialized AI computer).

The Legacy Stays

The ideas, collaborations, and research directions that emerged from this gathering laid the groundwork for all of the progress that has been made in the field over the past six decades. Those in attendance, namely Marvin Minsky, Herb Simon, Allen Newell, and John McCarthy founded the three leading AI and computer science programs in the U.S.: MIT, Stanford, and Carnegie Mellon.

The workshop's legacy continues to this day. 

Subscribe now

The 1960s

The 1960s was an eventful decade for AI, with a shift towards knowledge-based and expert systems and early experiments in natural language processing. Leading this charge was Edward Feigenbaum, a student of Herbert Simon, a pioneer in decision theory and cognitive science.

Ed Feigenbaum and the Birth of Expert Systems

Feigenbaum's work with Herbert Simon sparked his interest in creating machines that could mimic human expertise. Inspired by Simon's theories on bounded rationality and human problem-solving, Feigenbaum and Alan Newell embarked on a quest to build "thinking machines” after Herbert Simon handed them the manual for the IBM 701 vacuum tube computer, this was their first commercial binary computer experience. 

“Taking the manual home, reading it, by the morning, I was born again.”  ~ Ed Feigenbaum's Search for A.I.

This quest led to the publication of "Computers and Thought" in 1963, a collection of articles on artificial intelligence that quickly became a bestseller. This was followed by the development of EPAM (Elementary Perceiver and Memorizer), a computer model that simulated human learning and memory.

Building on these foundations, Feigenbaum and his team created Dendral (1965), the first expert system. Dendral used knowledge of chemistry and heuristic search to identify organic molecules from mass spectrometry data, effectively automating scientific inference in a specialized domain (this reminded me of DeepMind’s AlphaFold 🙂).

ELIZA: The 1966 AI that beat GPT-3.5!

In 1964, Joseph Weizenbaum created ELIZA, an early natural language processing program that simulated a Rogerian psychotherapist.  Though simple in design, ELIZA's ability to hold seemingly meaningful conversations surprised many, even prompting some users to believe they were interacting with a real person. It’s surprising to me that in 1964 we had the same interfaces we have now (chatting with intelligence). I’d say Eliza is probably the closest to ChatGPT conceptually, but the tech, the intelligence itself was rule-based, it was creative but still followed a deterministic log. That said, it still beat GPT 3.5 in the Turing test! 

We evaluated GPT-4 in a public online Turing test. The best-performing GPT-4 prompt passed in 49.7% of games, outperforming ELIZA (22%) and GPT-3.5 (20%), but falling short of the baseline set by human participants (66%) from [2310.20216] Does GPT-4 pass the Turing test? 

Here is the link if you want to chat with Eliza: Eliza, Computer Therapist, also below is classic video showing the demo 🙂

The 1960s laid the groundwork for much of what we see in AI today. The focus on knowledge representation, heuristic search, and early natural language processing set the stage for the development of expert systems, machine learning, and, eventually, the large language models that power today's conversational AI.

The 1970s

In the 1970s, AI research centered around the idea that "computational intelligence," as Edward Feigenbaum put it, was the "Manifest Destiny of computer science." 

Refining Expert Systems: From Dendral to Meta-Dendral

Building on the success of Dendral, researchers developed Meta-Dendral (1975), a system that could automatically generate rules for mass spectrometry from vast datasets. The Dendral project also highlighted the importance of extracting, structuring, and representing expert knowledge in a way that computers could understand aka knowledge engineering. This involved collaborating with domain experts, codifying their knowledge, and creating inference engines to apply that knowledge to new problems.

“Knowledge engineers elicit the knowledge from the minds of human experts, shape the knowledge so that programmers can transform it into viable program codes (knowledge base), and create the inference system that uses a knowledge base to derive specific results.“ ~ (PDF) The Fifth Generation 

If you think about it, this is not very different from the goals we have today for AI agents, RAGs, and fine-tuned LLMs.

​​The LISP Machine: Pioneering AI Hardware

With the architecture and the design of expert systems ready, hardware capable of doing the job was needed to take it to the next step. John McCathy had developed LISP in the 50s, and the community at the time found it suitable for running expert systems because of its ability to do symbolic manipulation, a key aspect of AI research. The question was, what hardware would be optimized down to the instruction set to execute LISP programs? Please welcome the LISP machine. Those machines even came with IDEs optimized for running/debugging LISP.  

Many notable expert systems at the time were developed and deployed on LISP machines, including MYCIN (the medical expert), XCON (the digital equipment expert), and R1 (the sales assistant). The 70s were the rise of LISP machines. 

Early Network Effects for AI and Expert Systems

The 70s also saw the rise of ARPANET, the precursor to the internet. This enabled researchers to share knowledge and collaborate on AI projects across different institutions and disciplines. For instance, Feigenbaum's team at Stanford granted access to their AI tools via ARPANET, attracting hundreds of users from fields like biology and pharmaceuticals. Actually, they exposed the only non-DARPA machine at the time on the ARPANET. This added some network effects to AI research back then (but not enough to prevent winter from coming, keep on reading 🙂).

Scaling Even More

The Heuristic Programming Project at Stanford, led by Feigenbaum, aimed to scale up AI research by harnessing the talent of numerous graduate students (early open source 🙂?). This effort resulted in the creation of the "Handbook of Artificial Intelligence" a three-volume encyclopedia that became a standard reference for AI researchers worldwide. Mostly used for teaching and aligning researchers and post-grads on state-of-the-art.

The 1980s

In 1981, Japan launched the Fifth Generation Computer Systems project, a 1 billion dollar project (a lot at the time), aiming to develop powerful computers with AI capabilities like natural language understanding and machine learning. The four generations of computing that existed: 

• the first generation being diodes and vacuum tubes 

• the second generation is transistor-based

• third are integrated circuits

• fourth generation using microprocessors

“Parallel processing will replace the one-step-at-a time procedures of the standard yon Neumann architecture of today's computers; jobs will be divvied up so that many subjobs can be in the works simultaneously” from the fifth generation paper

Finally, the fifth generation, which is viewed as whatever is able to run artificial intelligence. Arguably, the Japanese stressed the fact that parallel processing is needed, which makes you think about GPUs but also quantum computers. 

"Whoever establishes superiority in knowledge and technology will control the balance of world power regulating both cost and the availability of knowledge" ~ McCorduck and Feigenbaum from the fifth generation paper

“The Japanese expect that use will be tremendously expanded if people can communicate with the system in natural language such as Japanese, English or Swahili” from fifth generation paper

The Japanese did have the foresight indeed, Natural language as a UX did wonders and “tremendously expanded” the use of AI (look at ChatGPT 🙂).

Finally, another quote to show you how close the thinking was to where we are today: 

“Are your plants unaccountably turning yellow? Have you found that even your broker can’t keep up with the latest money market options? Are you afraid of doctors and lawyers or too poor to call them in when you need them? Expert systems will be on call day or night, providing instant, specific solutions to your problems.”

Toward the end of the paper, the author asks Ed Feigenbaum the question about the ethical and Legal ramifications of expert systems. The answer was "No," he said, there will always be "a human in the loop" taking full responsibility. Expert systems are designed to aid decision-makers, not replace them. 

Despite the potential, AI researchers recognized the limitations of expert systems (we will talk about AI Winters in the coming sections). These systems struggled to learn independently, lacked common sense reasoning, and were unable to replicate the intuitive and creative aspects of human thought. Nevertheless, the work of the 1970s and the 1980s laid a crucial foundation for the AI breakthroughs that would follow.

Winter is Here!

The science kept coming, and the research did not stop, but the application was nowhere near. Working as a researcher myself for years, I know it can get very interesting to just keep digging more and more without a clear path of viability. If I can publish a paper at a well-known conference and contribute my learnings, I win, or that’s how I thought back then. 

After the Dartmouth summer proposal for AI research, many optimistic predictions were made, especially those ignited by the promise of groundbreaking applications. Throughout the 1960s, AI research received significant funding from DARPA, with few requirements for delivering tangible results.

First AI Winter

More AI research and writing, for example, the “Perceptrons” book by Marvin Minsky and Seymour Papert or the checker’s player by Arthur Samuel, but little commercial application or use for them. In 1973, Sir Jamel Lighthill, a professor, and an applied mathematician, published a report that critically assessed AI research, he was supportive of the idea and applications of AI for industrial automation, but had concerns about the research trying to meld it with the analysis of brain function. It’s said that this report strongly led to funding cuts in the UK and raised doubts about the field's potential globally, including the U.S. Which might have led to the first “AI Winter”,  a quiet period for AI research and development.

I did a bit of digging to see if research and development were really affected, and whether this period of time deserved to be referenced as a “Winter”. It didn’t seem that there was a stop to content being generated be it research or otherwise, on the contrary, there has been a steady increase in the amount of “events” (see the circles) related to AI AND the amount of content generated for AI, i.e., AI research was more active than ever during the first “AI Winter”.

I’d rather consider the first AI winter, a clash of expectations, but as an early field, people (especially the scientific community) knew there was still progress to be made. I.e., it didn’t create an economic bubble burst but rather instilled some doubts which did impact viability for commercial use or delayed it.

1980-1987: Renewed AI Excitement

The 1980s ignited interest in AI after the skepticism in the 70s, this was mainly driven by the emergence of expert systems, Ed Feigenbaum even published a book about the applications of expert systems, highlighting case studies from many companies, including IBM, FMC, Toyota, American Express, and more. This optimism and references to commercial applications attracted increased funding, leading to a period of renewed excitement in the field. Also, as you can see from the graph below, the AI curve and Ngram mentions reached their peak!

1987-1994: The Real “AI Winter”

The enthusiasm for expert systems in the 80s was short-lived as their limitations became apparent. These systems struggled to handle novel information and situations that fell outside their pre-programmed knowledge base (they were IF/ELSE rule-based). Additionally, the collapse of the market for Lisp machines, a key platform for the development of expert systems. Here some of the reasons: 

• Compared to LISP, there were more versatile and cheaper programming languages/technologies like C++ and RISC computers which made specialized LISP machines (requiring a custom TTL processor) less appealing.

• The existing Lisp Machine software was not flexible or user-friendly enough for the average developer (don’t we always have this problem even today 🙂).

In addition to the above, in 1987, DARPA's decision to cut AI funding again, coupled with the failure of the Japanese Fifth Generation Computer project in 1991, deepened the second AI winter, increased doubts, and might have led to a burst in the AI economic bubble.

> Note: I find the timing of starting Nvidia (1993) interesting, just a bit after the decommissioning of the fifth-generation project, which initially had aimed to optimize computing for AI applications using a form of parallel processing. Didn’t dig much here, but the correlation could not escape me. Not sure if Jensen Huang had that far ahead foresight. 

The combination of shifting market demands from LISP machines/expert systems, management and financial struggles, technological limitations, and competition from more versatile and cheaper technologies are considered reasons behind the “AI Winter.” Below, you can see the trend going down after 1987, and there were also fewer publications and writings (see the circles). 

More readings on AI winters here. 

1994-Present: Steady State & Boom!

Despite the downturn faced in the 1980s with the AI winter, AI research kept beating, and interest in the research community didn’t suffer as much. This is another way to say that AI stayed in the “lab” but didn’t abandon the world after the realization that more work was needed to produce better ROI for enterprises, at least enough to justify spending on skill/hardware to make it happen. Let’s see what happened by looking at the timeline:

So we can see that AI content peaked in the 80s, AI winter started, and less content was produced. But research and progress kept on. Of the noticeable periods in AI history is the moment when IBM’s Deep Blue defeats Garry Kasparov, the world’s chess champion at the time, marking an end to mono-task models of intelligence, which was AND still a goal for AI research. Noticeably, the late 1990s towards 2010, most of AI development happened in robotics, domestic robots, etc. 

https://twitter.com/olimpiuurcan/status/1524249481512103937

2009 onwards saw numerous manifestations of AI in areas such as autonomous vehicles, AI ethics and governance, quantum computing research, and the integration of AI into various industries, including healthcare, finance, and entertainment. Notably, Google introduced the self-driving car project in 2009. In the 2011, IBM introduced Watson, which defeated human champion Jeopardy (Historically, it seems like IBM knows how to win with AI, be it Deep Blue or Watson). 

In addition to self-driving cars, robotics, and game-playing AI, deep learning with neural networks started to flourish around this time span. AlexNet, a deep neural network won the imageNet large scale visual recognition challenge which sparked interest in deep learning and neural networks. 

In 2015, Google DeepMind developed AlphaGo, which defeated Lee Sedol, the world champion of Go at the time. In 2016, Generative Adversarial Networks (GANs) were introduced, which started to reshape the generation of very realistic images and videos.

In 2017 Researchers began to apply reinforcement learning to a range of tasks, from playing video games to robotic control, demonstrating machines' ability to learn complex strategies, and the publication of the papers that introduced attention with transformers. The following year, 2018, Google introduced BERT (Bidirectional Encoder Representations from Transformers), making a dent in optimizing context awareness in natural language processing and paving the way for future applications (Mr. ChatGPT). 

From 2019 to 2024, AI witnessed major milestones. OpenAI's GPT-2, GPT-3, and GPT-4  (also Claude, Gemini, etc.) showcased language models' power to generate human-like text (also raising questions/concerns about ethical AI use). Noteworthy is that in 2020, AI became a very useful tool in combating the COVID-19 pandemic, even aiding in prediction and vaccine development. DeepMind's Alpha Fold 2 solved the long-standing protein folding problem the following year, potentially revolutionizing biology and medicine. Very recently, Google announced Alpha Fold 3, which aids in accurately predicting the structure of proteins, DNA, RNA, ligands and more, and how they interact. We hope it will transform our understanding of the biological world and drug discovery. 

I’d say the 2020s is finally the decade of AI viability for commercial application at a large scale, the growth trend of AI is rising exponentially with no signs of stopping That said, we have seen a similar trend in the early 1980s, followed by rapid fall in interest by the mid 1980s (so we have a couple more years to break the record 🙂), see the figure below:

I personally see no sign of this stopping or decaying now, but I could see the overpromise of AGI, setting the wrong expectations might lead to dampening the growth trend, after all, humans continuously raise their expectations when exposed to better experiences, better things, and expectations are hard to reverse, so we should be careful what and how we communicate to avoid an economic bubble burst! 

Below is another view of the AI trend for the 2000s:

AI History Visualization

Before we part ways, I’ll point you to the visual I used throughout this post to point you to the AI timeline.

You can find the interactive visual here and the code here (feel free to contribute :))

Next post, we will explore the lessons we learned from the history of AI in this post, and muse on how we can apply them to inform how we build successful innovations, products, and businesses. Stay tuned!

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

Looking into NVIDIA’s market cap, I am reminded of the importance of considering external factors when devising strategy. I found PESTEL to be one good model for mapping the factors.

The illustration above was inspired by James Eagle's post here. The image and the PDF are available here.

Subscribe now

PESTEL

Below is a brief PESTEL analysis of NVIDIA, a framework that assesses the Political, Economic, Social, Technological, Environmental, and Legal factors influencing the company. It is important to consider how external factors impact NVIDIA's market capitalization and strategic direction.

More on PESTEL here.

Political

📉 2022: Geopolitical Tension between Russia and Ukraine: The conflict between Russia and Ukraine has led to widespread sanctions and export controls, particularly in technology sectors. For NVIDIA, this resulted in decreased sales in affected regions and disruptions in their supply chain, impacting their revenue and cost structures.

Economic

📉 2022: Inflation and Rise in Interest Rates: Inflation and higher interest rates generally increase the cost of borrowing and can slow down consumer spending. For a tech giant like NVIDIA, this translated to reduced demand for expensive consumer electronics and professional graphics solutions, impacting their overall sales and profitability.

Technological

📈 2015: Bitcoin Mining Craze: NVIDIA’s GPUs are heavily utilized in cryptocurrency mining. The initial surge in Bitcoin mining increased demand for NVIDIA’s GPUs, boosting their sales.

📉 2018 Bitcoin Crash: “In November 2018, the total market capitalization for Bitcoin fell below $100 billion for the first time since October 2017, and the price of Bitcoin fell below $4,000, representing an 80 percent decline from its peak the previous January. Bitcoin reached a low of around $3,100 in December 2018.” ~ Crypto Bubble

More on what happened here.

📈 2022/2023: Release of ChatGPT: The release of AI models like ChatGPT has spurred interest in AI and machine learning, areas where NVIDIA’s powerful GPUs are essential. This has enhanced NVIDIA's market position in the AI sector, driving demand for their AI-optimized hardware solutions.

Environmental & Social

📈 2019: Pandemic: The COVID-19 pandemic increased social isolation and boosted demand for home entertainment and remote work technologies. For NVIDIA, this translated into higher sales of gaming GPUs and professional GPUs for remote work setups and research in health technologies.

Legal

📉 2022 Supply Chain Issues, Tariffs, Embargo: Legal and trade issues such as tariffs and embargoes, especially given the geopolitical tensions and the global nature of NVIDIA’s supply chain, pose significant risks. These factors increased costs and supply shortages, impacting NVIDIA stocks.

Subscribe now

Conclusion

NVIDIA's market capitalization reflects both internal strategic decisions and the fluctuating landscape of external forces.

PESTEL analysis highlights the interplay of political, economic, social, technological, environmental, and legal factors that can either boost or hinder NVIDIA's performance.

Geopolitical tensions, economic trends like inflation, technological shifts with cryptocurrency volatility and AI advancements, the social impact of events like the pandemic, and the evolving legal landscape all shape NVIDIA's trajectory. This analysis emphasizes the following key insights:

• External Volatility: NVIDIA (and many others) is highly susceptible to factors beyond its direct control, making it crucial to monitor these trends closely.

• Diversification is Key: Mitigating risk requires strategic diversification across markets, technologies, and applications to avoid over-reliance on specific sectors.

• Adaptability: Maintaining agility to adapt products, strategies, and resource allocation in response to external changes is essential for continued success.

To ensure its long-term growth and robust market capitalization, companies like NVIDIA must remain vigilant and proactive in addressing the challenges and opportunities presented by these dynamic external forces.

WHO/WHAT are you PESTELing next?

The full mindmap PDF can be found here.

I recommend going through the mindmap first, internalizing it, and then coming back and looking for answers in the long form here. With that in mind, here goes!

The full mindmap PDF can be found here.

Subscribe now

Model Pre-training

Growing up, I competed as a professional swimmer. Before the competition, a day, a week, depending on availability, I’d go to that Olympic 50-meter swimming pool and take a feel for it, even do entire training sessions there. If I could, I’d familiarize myself with the surroundings minus the crowd, the shouting, the whistles, etc. But it would give me the much-needed confidence for the next task to come, the actual race, which is a whole different game, mentally and physically.  I was practically pretraining for the competition 🙂

Formally pretraining is defined as:

Pre-training refers to the process of initializing a model with pre-existing knowledge before fine-tuning it on specific tasks or datasets. In the context of AI, pre-training involves leveraging large-scale datasets to train a model on general tasks, enabling it to capture essential features and patterns across various domains. ~ Lark

Pretraining helps the model perform better when it later learns tasks specific to a smaller, targeted dataset. It improves the model’s accuracy and efficiency, as it starts with a solid foundation of knowledge.

Pretraining is a necessary evil, and someone has to do it. That someone or something needs to have POWER 💪 and MONEY 💰. But once done, many can benefit from using it (if there was an intention for that in the first place, e.g., open source models). Technomistically speaking, we would be talking about sunk costs and Amortization.

• Sunk Costs: Pretraining requires a LARGE initial investment in computational resources and data acquisition. These costs are considered sunk as they cannot be recovered. However, once a model (big or small, still needs alot of resources) is trained, and compressed to represent a virtual version (it doesn’t really “copy” the data or memorize it) of the data it has been trained on, one can start to amortize these costs, leading to another fancy term called economies of scale (the initial high cost of training the model is spread out over many uses for inference).

• Opportunity Cost: Mentioned in the mindmap above is Self-Supervised Learning (SSL), which is basically a model trying to figure out the meaning of life by looking at ALOT of data, learning patterns between them, and generating it’s own labels (magic ✨). In doing so, SSL minimizes the need for explicitly labeling data, thus reducing both the direct costs of data preparation and the opportunity costs associated with extensive data labeling processes.

By now, you might be thinking, well, it’s great that someone did some pre-training (“pre” because the assumption is implicit in that it’s not yet ready for whatever you want it to do, i.e., it’s an invitation to train the model more). "I am pre-trained. Please train me properly sensei”

Optimizations

After pretraining a language model, it can be optimized further depending on the outcome desired. Examples of outcome optimizations are fine-tuning, prompt engineering, instruction finetuning, Retrieval augment generation (RAG), and so on. Notice that here, I intentionally said outcome optimizations and not model optimizations. To optimize outcomes, there are invasive and noninvasive methods, similar to when you visit a doctor with a problem, the doctor presents you with options like surgery (invasive because they modify something in you) or physical therapy (sometimes just reminding your body how to do things), here we also have options:

• Noninvasive: prompt engineering, tuning, and RAG (aka engineering around the model, which I personally consider a form of prompt optimization). These methods are cost-effective because they utilize existing resources to enhance output without significant expenditure. However, there is a limit to how much the model can improve with them.

• Invasive: requires more training on unseen data to improve the model's ability to produce certain outcomes. Examples are fine-tuning and its variants and methods (e.g., instruction fine-tuning). Such methods involve a higher initial cost, which might be needed to surpass the “limits” of noninvasive methods in terms of how the model performs on various tasks/benchmarks.

There are other optimizations, not necessarily to produce a better outcome, but to optimize how to get to the same outcome. For example, quantization is a form of cost/energy optimization (by mapping floating point representations to lower bits), done for training and inference. Distillation (distilling the same amount of knowledge onto a smaller model), and more. All of these take the desired outcome as a goal and work around how to get to the same outcome with the least of effort/energy/gas. I like the concept of “gas” here, borrowed from blockchain because it describes a fee without the specifics of that fee. 

Gas is the fee required to successfully conduct a transaction or execute a contract on the Ethereum blockchain platform. ~ How Gas Fees Work on the Ethereum Blockchain. 

Of all the optimizations, fine-tuning is one of the hottest topics in the field these days. 

Full & Parameter-Efficient Fine Tuning (e.g., LoRA)

I am jumping ahead here because we don’t need to get into the nitty gritty. After pre-training, you could go and do Full Parameter Fine Tuning (FPFT), which would mean that you take that pre-trained model and re-tune it to a specific task (e.g., teach it kungfu).

Full-parameter here would mean that would have to:

• Update all the weights and parameters of the Neural Network (which can be very large) frequently.

• Go through many Hyperparameter optimization rounds to avoid side effects.

• Run it multiple times through your data (aka, multiple Epochs)

• …

While you are not running the training over the entire internet as a dataset, only a subset of the data (e.g., KungFu), it’s still considered “costly”.

There are ways to solve for this with Parameter Efficient Fine Tuning (PEFT), which has the following benefits:

• Leaves pre-trained model weights fixed and only adopts a small number of task-specific parameters during fine-tuning.

• Reduces storage memory because you are not updating the entire model parameters (there are multiple techniques, again mentioned in the mindmap).

This makes fine-tuning cheaper and accessible on modest hardware. Techniques include Low Rank Adaptation (LoRA), which decomposes larger weights matrix representations into smaller matrices with low ranks and other variants such as adaptive layers, prefix tuning, and more.

PEFT provides incremental cost reductions 📉 through the efficient use of hardware, which also leads to more optimized resource allocations, lower energy consumption 🍀, and enhanced overall operational efficiency.

Reinforcement Learning from Human Feedback (RLHF)

As mentioned above, there are multiple ways to improve the model’s performance. Another example is Reinforcement Learning from Human Feedback (RLHF) which is a  “Human in the loop” strategy to basically teach models “principles”, i.e., what it takes to be “helpful”, “harmless”, and so on. 

RLHF works briefly as follows:

• First, humans judge a model's output. These humans review these outputs and provide feedback. This feedback could be in the form of rankings, ratings, or direct corrections. The key is that these human evaluators assess the model's responses based on desired behavior, as described above. 

• The feedback from human evaluators is then used to fine-tune the model. This may involve training the model to predict human preferences or directly optimizing the model's parameters based on the feedback. The goal here is to align the model more closely with human values and expectations.

While RLHF has shown good results, it is still expensive (find the humans and make them do the work). Second, it is difficult to encode human values into the model (you can try). Third, the model may not generalize well to new situations. Despite these limitations, RLHF is a valuable tool for AI safety research and has the potential to contribute to the development of safer AI systems.

Incorporating RLHF entails additional costs due to human involvement. These costs must be balanced against the marginal utility (benefit-to-cost ratio for each human involved) derived from improved model accuracy and compliance with ethical standards, enhancing user trust and market acceptability, all of which are hard(er) to quantify but are important nevertheless.

Conclusion & Recommendations

Models are pretrained to get a basic understanding of the world (depending on the data it fed on), and are optimized to get “better” at achieving an outcome. The outcome may vary. It could be to get better at math, or writing poetry, or better at following instructions (in English or other languages), or better at adhering to what aligns with human values and principles (causing no harm, no discrimination, ethics,...).

Other outcomes could revolve around making the model more efficient cost and energy wise, faster at inference, etc. We can call those efficiency gains.

With this in mind, here are some recommendations:

• Adapt and Fine-tune: Use in-context learning and fine-tuning strategies to adapt to new requirements and teach your model more about the tasks/outcomes/use-cases you want it to be good at without the need for extensive retraining which helps conserve resources and allows you to respond quickly address your organization’s needs. 

• Invest in Parameter-Efficient Fine Tuning: pretrained models are expensive to train. Luckily, we are seeing investments in making models open and thus available to a wider audience. The basic models are rarely useful without additional tuning, make use of existing advancements in training (e.g., LoRA which we will talk about more in a later post), which can reduce the tuning costs and improve throughput, as well as alleviate logistical challenges associated with updating AI models, making it ideal for continuous improvement cycles.

A comparison of training throughput (tokens per second) for the 7B model with a context length of 512 on a p4de.24xlarge node. The lower memory footprint of LoRA allows for substantially larger batch sizes, resulting in an approximate 30% boost in throughput. ~ Fine-Tuning LLMs: LoRA or Full-Parameter? An in-depth Analysis with Llama 2 

• Balance Human Input with Automated Processes: While human feedback is crucial for ensuring model reliability and ethical alignment, it is a balancing act to weigh these benefits against the costs of human involvement and to optimize the use of automation were beneficial.

• Invest in Cost/Energy and Computational Optimizations: Implementing computational optimizations such as quantization and quantized-aware training (see the mindmap) should be prioritized to reduce operational expenses (OPEX), such as energy consumption and maintenance. It will also reduce capital expenditure costs (CAPEX)  by eliminating the need for expensive and high-performance computing hardware.

Subscribe now

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

⛔ For the busy folks, here is a TL;DR version:

• Platform Principles: Create a foundation for streamlined development of value. I.e., Platforms provide tools, services, and automation to simplify and accelerate value delivery.

• Platform Archetypes: Platforms come in various shapes and forms. Examples are Cloud platforms (Google Cloud, etc.), business platforms (Salesforce, etc.), and Internal Developer Platforms (IDPs). All offer unique value and are targeted to a set of personas.

• Key Considerations:

  • APIs: Inspired by Amazon’s API mandate, platforms should provide consumable and persona-focused APIs that cover the right set of needs.

  • Purposeful Design: Understand the value you want to deliver and tailor a platform to serve those needs. Failure to do so results in overserving (surplus) or underserving (shortage). You will want to strike the balance (equilibrium).

  • Product of Platform vs. Platform for Product: Determine if the platform itself or the value it brings is the desirable.

  • Skills and Tools: Platform creation requires a wide range of skills (frontend, backend, infrastructure, etc.). Assemble the right people and tools to get the job done. Assemble your platform engineering fellowship.

• To Remember: Keep your platform user-focused, simplify wherever possible, and continuously evolve it based on feedback.

Subscribe now

What REALLY is platform engineering? 

I’d define it simply as:

The foundation upon which value is being delivered

But it’s also a mashing of building blocks, tools, and services that enable the target persona (developers, ML engineers, data scientists, ...) to craft value. 

There are many contexts in which you’ll see platforms mentioned. I will use the following format to describe them “_platform such as platform_name that provides _tools_services  to enable _those_enabled to _value

• Cloud Platforms, such as Google Cloud Platform, AWS, Azure, provide services (databases, networking, IDEs) for developers to use to generate value. 

• Business and Marketplace Platforms, such as Salesforce or Shopify, provide pre-built modules for customer management, payment processing, and e-commerce functionality to enable businesses to operate and sell online.

• Social Media Platforms, such as Facebook or Twitter, provide social networking, content distribution, and API access to enable individuals and businesses to connect, engage with audiences, and build communities.

I left the best for last, Internal Developer Platforms (IDP), garnering attention, are no different. Let’s use the format above: 

• Internal Developer Platforms (IDPs) provide self-service portals, automation, and standardized workflows to enable developers to provision resources, deploy code for value-generating applications, and manage them easily.

Depending on the level of abstraction needed, a platform exposes a construct (or an Application Programming Interface aka API) to request for a service needed to provide value. Let’s talk more about this. 

The API Mandate, the Chef’s Analogy, and Purposing Platforms

When I think about platforms nowadays, I think about two things:

• The API Mandate - thanks to Jeff Bezos 

• Indian food (delicious) 🥘 - thanks to Kelsey Hightower

Food and APIs, what are you talking about, man? Let me explain 🙂

Amazon’s API mandate, as championed by Jeff Bezos at Amazon, essentially states that all functionalities within a platform should be accessible through well-defined APIs. I.e., all value generated by any team should be exposed via a well-defined interface, the side effect of that mandate was that Amazon was able to externalize those set of APIs to wider consumptions, and also what became AWS cloud platform (Augusto Marietti has a more detailed writeup on this here, good read!). That is probably one of the earliest appearances of a “platform”. The question is what kind of APIs to expose? 

As highlighted in Working Backwards by Bill Carr and Colin Bryar, Amazon’s teams are grouped by mission vs. function, so depending on each team’s mission, and desired outcome, an API will emerge to allow others to consume the artifact produced by those Two-Pizza Teams (or should we call them distributed platform pizza teams 🙂).

Indian food is a specialized cuisine that is known for its unique taste and flavors. In one of those Twitter spaces, Kelsey Hightower gave the example of an Indian restaurant where quality is tied to specialty. You wouldn’t go to an Indian restaurant to eat Italian food. If that restaurant served Italian, you’d typically question the quality of the food!  Sure, you can stock a kitchen with ingredients from all cuisines, but until you curate a menu, you might not have something of true value to offer. 

The combined lesson here is to understand who you are serving and for what purpose, to build the right abstractions. Generalize too much, and you risk complexity, you overserve your target persona, and you drive them away from using your abstractions altogether (negative value).  Specialize too much, and you risk undeserving the target persona, they don’t get what they need, and they will start building their own abstractions on top to fill the gaps (shadow IT). 

In economics, the right terms to use are supply surplus and shortage. Supply surplus denotes a situation in which the quantity supplied of a good exceeds the quantity demanded. In the case of platforms here, too many APIs, too many unused templates, and too much functionality that has the side effect of increasing support times, engineering development time, etc. Whereas shortage represents the opposite scenario where the quantity demanded is greater than the quantity supplied, for example, developers need APIs to produce services of value, lack of APIs can be an example of a shortage. Here is an excerpt from Stevey’s Google Platform Rant describing an example of this:

Google+ is a prime example of our complete failure to understand platforms from the very highest levels of executive leadership (hi Larry, Sergey, Eric, Vic, howdy howdy) down to the very lowest leaf workers (hey yo). We all don't get it. The Golden Rule of platforms is that you Eat Your Own Dogfood. The Google+ platform is a pathetic afterthought. We had no API at all at launch, and last I checked, we had one measly API call. One of the team members marched in and told me about it when they launched, and I asked: "So is it the Stalker API?" She got all glum and said "Yeah." I mean, I was joking, but no... the only API call we offer is to get someone's stream. So I guess the joke was on me.

It’s a balancing act and no easy feat, but when you get there, you are a platform karate black belt! 

Is the Platform the P-roduct OR the p-roduct

Platforms are value bridges, at least when considering why you would build one in the first place. If you treat the platform as the “main” thing, you’d be overlooking the purpose and the reason a platform exists in the first place, at  the same time, if you JUST focus on the value the platform delivers, you might not even make it to where value is finable/consumable, well, you still need something, a vehicle of sorts to get there! 

This has been a long-standing topic of discussion, and it is expressed in different terminologies/contexts such as: 

• The mean vs. the end 

• The platform product vs. the experience product

• The P-roduct vs. the p-roduct (lower case and upper case)

Let’s take the last one. How can the p-product (lowercase) serve the uppercase P-roduct. How can we build platforms with purpose? You might have different goals depending on who you are (a supplier or a consumer) and your strategy. 

Shreyas Doshi talked about the generic concept in one of his posts. Here are some of the examples/results based on surveys from his followers (who are mostly product builders) on what they consider the P-roduct: 

• Netflix: Most view licensed content as The Product, suggesting that content is more crucial than the technology (app or CDN) or algorithms.

• Uber: The ride is viewed as The Product, more than the app or the matching algorithms.

• Instacart: The groceries and the delivery system are almost equally viewed as The Product, indicating the importance of the end product and its delivery.

• Amazon: The delivered item is considered The Product, underscoring the end result over the platform or logistics (the listings, the marketplace, etc). 

• YouTube: The creators are seen as The Product, highlighting the value of content creators over the technological features (search, the app, the video viewer).

As a supplier (a vendor), you might consider the platform itself as a P-roduct to streamline how platforms are built, making it easier to produce content on top. As a consumer, you might think more of the content you offer on top of platform while placing constraints on platform usability (that acts as a filter to which vendors you “buy” the platform from). 

I don’t believe there is a clear cut answer, I’d be interested on your thoughts and how you approach this as a consumer or supplier of platforms. Let chat!

Picking the Right Skills & Tools (e.g., AI/ML Platform)

Once you understand the core value you are delivering via the platform and the type of platform you want to build, it’s time to make sure there is enough skill in the house to build it. Here, you might have to mix and mash skills. For example, you might need a mix of frontend, backend, and infrastructure, as well as one or more of DevOps, MLOps, SecOps, ... xOps 😁, whatever makes sense to get the job done (e.g., reduce cognitive load for developers).

To reduce the cognitive load of developers, the Platform team should cover the entire tech stack: Infrastructure/DevOps/SRE, but also frontend, backend, and security topics. ~ from Platform Engineering, Part 2: WHAT Are The Goals of a Platform Engineering Team? | by Benoit Hediard | Stories by Agorapulse | Medium 

To make things a bit more realistic, let’s go with an example of an AI/ML Platform and the typical components involved. The closer the component is to the top, the more visible it is to the users.

Let’s break it down relative to the end-user (inspired by Wardley maps but without the X-axis):

• Top Row (High Value to Users): This represents the User Interface Layer where direct user interactions happen. AI tooling, including chat interfaces (e.g., streamlit), Dev environments (e.g., Jupyter), and maybe an orchestration UI (e.g. Kubeflow)

• Second Row (Medium Value to User): Data Processing and Workflow Management tools (e.g., Kubeflow pipelines, Airflow, DAGs,…) that are crucial for the operational aspects but less visible to the end user.

• Third Row (Medium to Low Value to User): Model Monitoring and Operations tools (Promethus/Grafana, Evidently,…), which are essential for maintaining the platform's health and performance.

• Bottom Row (Lowest Visibility): Infrastructure (AWS, GCP, Azure,…) and Platform Management along with DevOps and Collaboration Tools, which are foundational yet mostly invisible to the end user, another name for this layer can be the “commodity” layer.

I will not go more into the technical details here, but there are multiple venues to explore if you are interested specifically in AI/ML platforms and personas. One venue to explore is the Cloud Native AI working group, especially the most recent AI whitepaper. Another venue for exploring platform engineering is the platform engineering white paper.

The components will change depending on the type of platform you are building. You don’t need everything in a single platform. You need to curate and specialize based on feedback from your target platform users (remember specialization and Indian food 🥘). The left side of the figure above shows what a typical “generic” platform would be comprised of.

As such, you will need to ensure that you have the right skill set to build your target platform, for the personas you are targeting.

Conclusions & Reflections

So here are some parting recommendations:

• Define Clear Objectives and APIs: Start by clearly defining your platform's purpose and the APIs it will offer (how will your platform be consumed). This clarity will enable your platform to provide specific, valuable functionalities that are easy for users to implement and integrate. This is another way to say, start with the end in mind (from the 7 habits). 

• Focus on User Needs: keep the end-user in mind during development. Understand their pain points, requirements, and usage patterns. This understanding will help in creating features that are not just innovative but also highly relevant and user-friendly.

• Keep it lean and Evolve with time: Avoid adding unnecessary features/functions that do not add clear value. Overcomplication can deter users and increase the cognitive load, making the platform less appealing and harder to use. Always aim for simplicity and efficiency in design which itself will drive more straightforward execution.

• Continuous Feedback and Improvement: Implement a robust feedback loop with your users to continuously improve the platform. Regular updates based on user feedback and emerging trends will keep the platform relevant and highly functional. I found this post by Abi Noda and Tim Cochran helpful as a guide for collecting feedback (via qualitative and quantitative metrics). I will also discuss feedback collection in upcoming posts. 

Different people have different definitions of value. What’s important is that within your realm, among your people, you align on the core value you want to deliver. Here are some questions to ask to sanity check if you are on the right path:

• Do I understand what problem I'm solving (the core job to be done)? Can I articulate it better than the customers themselves?

• Is this tech the right fit for the use case at hand? It’s not about the fanciest tool, but one that gets the job done.

• How well do the tools and systems I implement meet the evolving needs of your users? This usually translates to how modular is my architecture.

• Am I adding value or just complexity? Sometimes, the simplest option is the right one, beware of overengineering (been there, done that)

• Can I do better? In what ways can you further automate to reduce costs and cognitive load? Are you collecting enough feedback to know this?

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. Looking forward to hearing from you!