The Problem MCP Solves

At its foundation, MCP rests on a simple but powerful principle: AI models are only as effective as the context they are given. Imagine asking a colleague for help without providing any background information. Their assistance will likely be limited. MCP addresses this by providing a standardized way for AI applications and agents to access and interact with external data and tools, enriching their context and enabling more intelligent and useful behavior.

Before MCP, integrating AI with external systems (databases, APIs, local files, even web services) was often a bespoke and lengthy (developer time) process.

Each new integration required custom code, leading to a "many-to-many" problem (or some call it the “N x M” problem). This created a fragmented landscape, slowing down development and hindering interoperability.

MCP proposes a "one-to-many" solution, a universal interface that connects any MCP-compatible AI applications to any MCP-compatible server, where the logic is detail abstracted by the unified interface, but let’s not get ahead of ourselves and take a quick look at MCP’s building blocks.

Prompts, Tools, and Resources (the building blocks)

MCP isn't a monolithic entity. It's built upon three well-defined blocks, each granting a different level of control:

• Prompts (User-Controlled): These are predefined templates for common interactions initiated by the user. Think of slash commands in a text editor, IDE, or Slack (e.g., /summarize to summarize a pull request, /translate to translate selected text, /code-review to review code/PRs/…). Prompts provide a consistent and user-friendly way to interact with specific services, ensuring well-formed input for the language model. They are essentially user-invoked “tools”.

• Resources (Application-Controlled): Resources represent data exposed by the server to the client application. Unlike tools, which are actions, resources are information. Importantly, the application (not the model) determines how to utilize this data. This could involve displaying an image, attaching a file to a message, providing contextual information to the user, or using the data in subsequent computations. Resources can be static (like a file) or dynamic (like data fetched from an API that changes over time). This can also be used with prompts, for example, when invoking prompts or tools from the CLI, resource URIs could be passed as parameters or arguments

• Tools (Model-Controlled): This is where the AI model's agency comes into play. A server exposes a set of tools, and functions that perform specific actions (or call one or more APIs). These could range from reading and writing files to querying databases, accessing APIs, or even controlling external devices. The thing to note here is that the language model that is used by the client application is what decides when, what, and how to invoke these tools based on the task at hand.

I personally have not seen that much use of resources and prompts; most of the excitement has been around tools (exceptions do exist, though, just not as common).

For more details on the building blocks, see the protocol specification and check out this talk: Building Agents with Model Context Protocol - Full Workshop with Mahesh Murag of Anthropic

Communication Modes

In addition to the trifecta above, MCP can operate in different modes:

• STDIO: Good for quick starts and onboarding. Here, the client often spawns the server as a subprocess, piping JSON-RPC messages via STDIO.

• HTTP + Server-Sent Events (SSE) [Older comms]: Used two separate channels: regular HTTP POSTs (Client->Server) and a dedicated SSE stream (Server->Client). Required stateful servers for the SSE part. This had some drawbacks, including incompatibility with serverless, resource-intensive(ness), and didn’t really get adopted that much, also MCP allows for hierarchical servers relationships, which complicates things, especially with SSE

• Streamable HTTP [Newer comms]: No more "always-on" connections to remote HTTP servers. Uses a single endpoint. Leverages standard HTTP POST. The server chooses whether to reply with a single JSON response or upgrade the POST response itself into an SSE stream. Also allows a separate GET for a dedicated SSE stream if needed. More flexible, supports stateless servers, and adds resum-ability.

Now, debate exists regarding the trade-offs between MCP's complexity and its utility across the different communication modes. Remote MCP, particularly when implemented over HTTP, is frequently perceived as complex due to added layers like JSON-RPC wrappers, especially when compared to traditional REST APIs defined by OpenAPI specifications (Why not JUST OpenAPI, man?). However, some argue this complexity is justified by enabling a reasonably powerful paradigm: “users dynamically adding tools and context sources to AI applications at runtime”, much like browser extensions, going beyond static, developer-defined integrations. In contrast to remote and its perceived complexity, local MCP via stdio, commonly used for inner-loop developer tools (e.g., requiring filesystem access or command execution), is generally viewed as more straightforward and practical for its purpose, making it the more prevalent implementation observed so far.

The Value Proposition: A Win-Win-Win-Win Scenario

MCP offers some tangible benefits to a wide range of users:

• WIN 1 | Application Developers: The most immediate benefit is the elimination of custom integrations. Once an application is MCP-compatible (i.e., an application becomes a client), it can connect to any MCP server with zero additional coding. This reduces development time and effort, allowing developers to focus on the core functionality of their applications rather than the plumbing of integrations.

• WIN 2 | Tool and API Providers: By building an MCP server once, tool and API providers can expose their services to a broad ecosystem of MCP-compatible applications. This provides a standardized "on-ramp" for AI, increasing their reach and potential user base. It's a way to "AI-enable" their existing services with minimal overhead.

• WIN 3 | End Users: The ultimate beneficiaries are the end users, who gain access to more powerful, context-rich, and personalized AI applications. These applications can understand and act upon their data and the real world in ways that were previously impossible or required significant manual intervention. Take Zapier MCP actions, for example. For a non-technical consumer, you’d just point to Zapier, and magically be able to “chat” with all available integrations. I chose Zapier because while MCP standardized tool interactions through a protocol, Zapier had standardized access to “integrations/tools” with no/low-code, and now they MCPed all their integrations via Actions.

• WIN 4 | Enterprises: MCP offers a clear way to separate concerns and streamline internal development workflows. Teams responsible for managing infrastructure (e.g., vector databases, CRM systems, internal APIs) can expose these resources as MCP servers. Now, other teams can consume these services with natural language without needing to build custom integrations or understand the underlying implementation details.

Agents ♥️ MCP

Perhaps the most useful incarnation of MCP lies in its role as a foundational protocol for AI agents to use tools. An agent, in this context, can be thought of as an "augmented LLM", a language model that is enhanced with the ability to interact with retrieval systems (for accessing information), tools (for taking actions), and memory (for storing and retrieving information). MCP provides a standardized interface for these interactions. Here is the Agents equation:

Agents = Model + Augmentation(Tools + retrieval + memory) + Loop

More importantly, MCP enables agents to dynamically discover and utilize new tools and data sources, even after they have been initialized. This means an agent can "learn" new capabilities by connecting to new MCP servers, without requiring any changes to its core code. This alone is a big win, but it comes with trade-offs, for example, MCP servers overwhelming the Agent’s LLM context and rendering it “kaput”.

Most of the known “agent” frameworks had started to bring MCP client supportability in as a framework-native feature, most prominently, Open-AI bought in, and this more or less settles it!

And looks like Google will be biting soon as well:

Another seemingly powerful concept is composability. Any application or agent can be both an MCP client and an MCP server. This allows for the creation of layered, multi-agent systems where specialized agents can interact with each other and with external services in complex workflows. Composability makes hierarchies possible, but the downside is that it makes everything else a tad harder. For example, security (credentials propagating downstream), debuggability (the load shifts on building that on the server application, so it must be very well instrumented), latency/performance (the length of the hierarchy/chain of client/servers can be unpredictable, so unless there is a fixed tested n-depth, performance can’t be guaranteed).

Standard OR “Overhyped Marketing Fluff”

Short answer: OpenAI’s adoption of MCP in its agent SDK solidified its stance towards being a standard rather than just “Fluff”. Langchain’s twitter poll did as well 🙂

I recently came across this post from Harrison Chase (CEO of LangChain) and Nuno Campos (lead of LangGraph), where they debate the practicality and usefulness of MCP for developers AND the rest of the crowd. Let’s get into the details.

Optimism and Skepticism Striking the Right Balance

In the post, Nuno expresses some doubts about MCP's utility beyond very basic tool replacement. He argues that effective tool use in most production agents requires tailoring of the agent's system message and architecture to those specific tools (MCP “just” ships the tool, no questions asked, no customizations done, unless you’d go and modify the server’s code base and the prompts). He argues that even with tailored agents and tools, models often fail to call the correct tool (true story on my side, getting LLMs to work well with tools depends on LLM’s parameter size, quality, and many other factors).

I tried to depict this a bit below, it’s like going around with a Swiss army knife, not really knowing when you’d use any of the tools in there, but in times of need, they sorta/kinda do the job! Compare that with walking with a bag of tools (scissors, can opener, the handyman you are!) all around. A Swiss knife is lighter, and it DOES the job (though poorly, and takes a lot of time). On the other hand, having the perfect tool for the job saves time but adds the overhead of optimizing the time and place (you never know!)

Harrison, on the other hand, positions MCP as particularly valuable when you want to integrate tools with an AI agent whose underlying logic you cannot directly control. This opens the door for non-developers to extend the capabilities of their existing AI tooling/clients/agents (e.g., Claude or Cursor) with custom integrations and tools. He put the emphasis here on non-technical users to build agents without needing to code the agent's core logic (no-code/low-code), plug this MCP server and be on your to doing something productive.

Utility for Non-Developers AND Developers

While MCP might not be predictably reliable/predictable (you never how well those tools are tested). For non-developers, they could be just enough to get the job done. In a previous post, we discussed non-consumption and the different types of innovation. Nonconsumption occurs when potential customers are shut out from existing solutions due to barriers like cost, access, complexity, or, crucially, in this context, required technical skill.

Many tools, locked behind APIs requiring code, remain inaccessible to those who could benefit from them but lack programming abilities. LLMs began tackling this by democratizing access to information through a natural language interface. MCP takes it a step further by potentially democratizing action and integrations, allowing a non-developer to plug or even just use existing integrations via an MCP server (abstracting access to their AI client, it bridges the skill gap and turns previously inaccessible tool functionalities into consumable services via a natural language interface). Some examples:

• A marketing manager using Claude could connect an MCP server for their company's Google Analytics account. Without writing code, they could ask Claude, "Summarize the top 5 traffic sources for our latest campaign landing page based on GA data," a task previously requiring manual navigation or analyst support.

• A project manager using Cursor could integrate an MCP server for their team's Jira instance. They could then ask Cursor, "List all open 'bug' tickets assigned to me in the 'Backend' project and summarize their description" accessing project data directly within their coding-assistance tool.

• A small business owner could use an AI assistant connected to an MCP server wrapping their simple customer database (perhaps even a managed Airtable or Google Sheet). They could ask, "Find customers who purchased Product X in the last 6 months but haven't purchased Product Y."

That said, for developers getting started, not really sure what tools are the right ones for the job, experimenting with existing integrations built by someone else and exposed via the MCP server abstraction might not be a terrible idea. Optimization can come later, but as a starting point, for identifying the use-case, and understanding the landscape, it can be quite useful, and who knows, maybe it’s just good enough 🙂

MCP’s Creator Economy & Network Effects

If you have not noticed, there are thousands of MCP servers out there (and there is no stopping), MCP triggered a new creator economy around AI integrations. Developers are building, sharing (in the open), and sometimes even monetizing MCP servers via marketplaces, subscriptions, or sponsorships, creating valuable assets like premium servers for niche/complex tools to generate revenue (like app stores).

MCP’s ecosystem will have Clients (with MCP integrated within like Cursor), Servers (for almost anything has an API), and infrastructure services and platforms, i.e., where to host those servers for scale, security, and who hosts them for me.

The diversity will give birth to new network effects. As more developers and organizations adopt MCP, the number of MCP servers and connectors between AI models and external tools will keep growing. Each new server (e.g., for Slack, GitHub, or a niche blockchain tool) adds net-new value to the ecosystem by expanding the range of tools that MCP-compatible AI clients can access. For example, if a developer builds an MCP server for a CRM, every MCP user gains the ability to integrate their AI with Salesforce without additional effort. I.e., the more participants, the more useful it becomes (“direct network effect”).

Additionally, as major companies integrate MCP, as seen with OpenAI, Block, Apollo, and Microsoft’s Copilot Studio, the protocol will keep gaining credibility and reach. This draws in smaller firms and developers, creating an "indirect network effect." For instance, if Google or AWS builds MCP servers (servers as complements) for their platforms, the protocol’s utility skyrockets, pulling in more users and developers to leverage those integrations and funneling more use of the platform other services (“Indirect network effect”).

On the macro level, MCP provides AI with superpowers by streamlining access to tools and integrations. This drives more and more automation enhancing productivity across various industries and amplifying aggregate economic value.

HOWEVER, this all does not come for free. For MCP to be considered more than just “hype” by enterprises, many will need to invest in learning about how AI systems work and how to implement them reliably. Legacy systems, fragmentation risk, and security concerns (requiring robust permissions) are hurdles, that could potentially slow uptake for those enterprises.

The Future

MCP's long-term impact depends on reaching critical mass, which will depend on how easy/pragmatic it is to adopt for enterprise use-cases. If widely adopted, it could redefine the AI ecosystem, shifting focus from integration plumbing to faster and more accessible innovation. The future is full of surprises and questions:

• How to differentiate?

  • The focus may shift from “the best API design” to “the best collection of discoverable tools” for agents

  • More Specialization? The separation of concerns inherent in MCP (client, server, tools, resources) could lead to specialization among developers. Some might focus on building robust and reliable MCP servers (back to software engineering), while others concentrate on creating user-friendly AI applications (back to good design principles).

• How will pricing structures change? Dynamic, market-driven tool adoption based on agent assessment of speed, cost, and relevance could emerge, favoring modular, high-performing tools over merely popular ones. Also, servers-as-a-service, subscriptions (MCP would add more value, potentially justifying subscription amounts), Usage-Based Pricing (Pay-as-You-Go), outcome-based pricing (MCP makes that more tangible, e.g., pay for a code commit), and other modes of monetization will start to emerge.

• How to design APIs with Tools? Tools and APIs are not 1:1 mapping, a tool can combine multiple API calls (e.g., draft_and_send_email vs. just send_email).

• How and Where to host MCP servers? real-time load balancing across MCP servers, scale, etc (same old same, or is it 🙂)

• ..

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

Introduction

The AI landscape is constantly shifting, in ~3 years we are at “manus” level (standing on the shoulders giants of course). That said, each year, there is the unavoidable question, “Do we still need Retreival Augemented Generation (RAG)?” The answer (still in 2025) is “Absolutely, what are you talking about?!.

But why? The key here is that the definition of RAG evolves with time! In this post, we will explore RAG's modern origins, how RAG is being used and implemented, its most prominent optimizations, and finally answer the core question: "Do we still need RAG?"

Important Taxonomy

Before we proceed, let’s cover two important concepts:

• Parametric Memory: This refers to the knowledge stored within the parameters (weights and biases) of a machine learning model. Knowledge is Implicit, like memorizing facts for an exam. The knowledge is embedded in your brain structure (analogous to the model's parameters).

• Non-Parametric Memory: This refers to knowledge stored outside of the model's parameters, typically in an external database or index. Here is that the structure of the model doesn't change - we are NOT changing parameters but adding additional data the model can access. Knowledge is explicit, the facts are readily available and understandable. This is like having access to a textbook during the exam. The knowledge is stored externally and you can retrieve it as needed.

We will be referring to those in the rest of the post.

Standalone LLMs Handicaps and RAG origins

So why do we need RAG? What problems does RAG solve?

Large Language Models (LLMs) are, at their core, pattern-matchers trained on massive datasets. While impressive, they suffer from key limitations:

• Limited ability to access and precisely manipulate knowledge: While large language models store factual knowledge in their parameters, their ability to access and utilize this knowledge accurately for specific tasks is still constrained.

• Lagging performance on knowledge-intensive tasks: On tasks that heavily rely on external knowledge, the performance of purely parametric models often falls behind.

• Challenges in providing provenance: It is difficult to determine why a parametric model makes a particular prediction, and thus providing evidence or sources for its decisions is an open problem.

• Knowledge Cutt-off: Parametric models cannot easily incorporate new information or revise existing knowledge without undergoing further training. This is referred to as the "knowledge cutoff" and is a barrier to building reliable and up-to-date AI applications.

• Potential for "hallucinations": Parametric models can generate factually incorrect information, which is a significant drawback for knowledge-intensive applications.

To solve the above problems, we need to bridge the gap between the knowledge storage capacity of large language models and the need for precise, up-to-date, and verifiable knowledge in many AI applications.

RAG to the rescue! The original paper for RAG set to address the limitations above. RAG was defined in the paper as a:

“general-purpose fine-tuning recipe for which combine pre-trained parametric and non-parametric memory for language generation"

The main components of RAG as defined in the paper are (as shown below):

• Retriever: This component is responsible for taking an input sequence (query) and retrieving relevant text documents from a non-parametric memory. The retriever consists of two main parts:

  • Query Encoder (q(x)): This is a model that creates a dense vector representation of an input query.

  • Document Index (d(z)): This is a dense vector index of text passages (documents). Each document is represented by a vector. The document encoder and index usually remain unchanged during fine-tuning.

• Generator: This component takes the original input sequence x and the retrieved document(s) z as context and generates the target sequence y.

🧳 Use-case: Customer Support at a Tech Company

Imagine a tech company like Slack deploying an AI-powered support bot. The bot needs to answer queries about the latest features, pricing updates, and troubleshooting steps, i.e., Information that changes frequently and isn’t fully captured in an LLM’s training data. With RAG, the bot retrieves the most recent documentation and support tickets from an internal knowledge base, ensuring responses are accurate and up-to-date, even for features released last week.

RAG and Finetuning: A Good Match

Today, there's sometimes confusion around the terms fine-tuning and RAG. Where Fine-tuning usually refers to adapting a pre-trained LLM (the generator) to a specific task. This is done by adjusting the LLM's internal parameters (weights) – essentially, retraining parts of the LLM on a new dataset. RAG, on the other hand, usually refers to improving how information is retrieved and presented to the LLM, often without changing the LLM's core parameters.

However, the original definition of RAG in the 2020 paper was broader. It envisioned a system where both the “retriever” component and the “generator” component (the model) were fine-tuned together aka 'end-to-end' fine-tuning.

The core of RAG is a general-purpose fine-tuning approach where both the retriever and the generator are trained jointly and end-to-end on downstream NLP tasks. This means that the parameters of the retriever (specifically the query encoder) and the generator are adjusted based on the task-specific data

According to the original paper, RAG is a “general fine-tuning recipe”, “It combines a pre-trained LLM with a retriever that accesses an external knowledge source. The Fine-tuning in the original paper puts the emphasis on the system (retriever + generator), which fine-tuned together so the retriever learns to find documents helpful for the LLM.

Today, fine-tuning often refers to adaptations of a pre-trained LLM (not the system) to a specific task by adjusting its internal parameters (weights), while RAG emphasizes optimizations on the retrieval flow, the store, re-arranging the results, often independent from the generator.

Why did we regress? Not sure!

Tweaking just the retrieval parameters is oftentimes called “naive RAG”, and while this can be useful, the real power comes from further optimizing the retrieval process. This is where retrievers finetuning enters the picture, specifically the finetuning of encoders/embeddings models.

Finetuning an embedding model on in-domain data (e.g., a company’s internal documents) improves retrieval accuracy, and don’t just believe the 2020 paper (which puts the emphasis on tuning retrievers + generators). A recent Databricks blog post titled "Improving Retrieval and RAG with Embedding Model Finetuning" provides reasonable evidence of this, showing substantial gains in retrieval metrics (like Recall@10) on datasets like FinanceBench and ManufactQA after finetuning.

The takeaways?

• Better Embeddings = Better Retrieval: Finetuned embeddings capture the nuances of specific data, leading to more relevant search results.

• Better Retrieval = Better RAG: More relevant context enables the LLM to generate accurate, grounded responses, reducing hallucinations.

• Finetuning Embedding Models Can Outperform Reranking: In many cases, finetuned embeddings match or exceed reranking models, simplifying the RAG pipeline.

The gotchas?

Finetuning embedding models is dataset-dependent: it worked well for some (e.g., FinanceBench) but not all (e.g., Databricks DocsQA). I.e., don’t assume finetuning embeddings will always solve your problems. You need to identify the true bottleneck in your system (retrieval, generation, or something else) and target your efforts accordingly.

Identify your “good enough” and remain there if it gets the job done, or risk diminishing returns from unnecessary spending. Moreover, along with fine-tuning the embedding model, adjusting the generator model (the LLM) can yield further improvements. However, this shouldn’t be your first move, see:

🧳 Use Case: Legal Research for a Law Firm

A law firm needs an AI tool to assist with case law research. Legal documents are dense with jargon and context-specific terms like "tort" or "estoppel" that generic embeddings might misinterpret. By finetuning an embedding model on a corpus of legal briefs, statutes, and case law, the firm’s RAG system retrieves highly relevant precedents for queries like "What are the latest rulings on non-compete clauses in California?" The result? Faster, more accurate research that keeps lawyers ahead of the curve.

Long-Context Models: A Powerful Tool, But Not a Replacement Retrieval/RAG

The rise of LLMs with longer context windows has led some to speculate that RAG might become obsolete. If an LLM can process an entire document, or multiple documents, in its context window, we don’t need to bother with retrieval, right, right?? Wrong!

A recent (close to the date of this writing) Snowflake blog post, "Long-Context Isn’t All You Need: How Retrieval & Chunking Impact Finance RAG" provides a counterpoint. Their research on financial filings shows that retrieval and effective chunking remain essential, even with long-context models. Here’s why:

• Context Confusion: Dumping large amounts of text into an LLM’s context window can overwhelm it with irrelevant information, making it hard to pinpoint what’s crucial for the query. Retrieval acts as a "focusing mechanism."

• The Art of Chunking: How documents are divided into chunks impacts retrieval effectiveness. Moderate chunk sizes, combined with retrieving more chunks, often yield the best results (compared to retrieving large chunks), enhanced by adding LLM-generated global document context.

• Retrieval Quality Trumps Generative Power (Sometimes): A well-tuned retrieval pipeline with smart chunking can enable a smaller LLM to nearly match a larger model (take note here, if you ever want to consume smaller models for Q&A, knowledge retrieval, consider this).

• Efficiency and Scalability: Processing vast text in a single pass is computationally expensive. Optimized RAG offers a more efficient, scalable/cost-efficient solution.

🧳 Use Case: Financial Analysis at an Investment Bank

An investment bank uses an AI system to analyze quarterly earnings reports from multiple companies. A long-context LLM could ingest entire filings, but irrelevant sections (e.g., boilerplate disclaimers) might dilute its focus. With RAG, the system retrieves only the most relevant chunks, say, revenue breakdowns or risk factors, using a chunking strategy tailored to SEC filings. This allows a smaller, finetuned model to deliver precise insights, saving compute costs while matching the accuracy of larger models.

The Rise of Agentic RAG: Beyond Simple Retrieval

Standard RAG involves a single retrieval step before generation. But Agentic RAG, or Retrieval-Augmented Generation with Agents, takes it further. An agent—an LLM-powered component capable of reasoning, planning, and acting, can:

• Interact with Multiple Tools: It chooses the best tool for a query, like a vector database, web search, API, or internal knowledge base.

• Perform Multi-Step Retrieval: It manages complex, multi-turn retrieval processes, refining its strategy as it gathers information.

• Reformulate Queries: If initial results are lacking, it rephrases the query and tries again.

• Validate Retrieved Information: It assesses the quality and relevance of context before generation, reducing hallucinations.

• Integrate with External Systems: It can send emails, access calendars, or perform calculations via APIs.

I liked the analogy provided in this post by weaviate:

Think of it this way: Common (vanilla) RAG is like being at the library (before smartphones existed) to answer a specific question. Agentic RAG, on the other hand, is like having a smartphone in your hand with a web browser, a calculator, your emails, etc.

The Weaviate blog post likens standard RAG to a library (access to information) and Agentic RAG to a smartphone (access plus processing and action). Surveys on Agentic RAG highlight its dynamic retrieval management and adaptability for complex tasks.

🧳 Use Case: Healthcare Decision Support

A hospital deploys an AI assistant to help doctors diagnose rare diseases. A patient’s query: "What’s causing my persistent fever and joint pain?" would require more than a single lookup. An Agentic RAG system searches medical journals, cross-references the patient’s electronic health record via an API, and queries a drug database for side effects. It reformulates the query if initial results are inconclusive, validates findings against recent studies, and suggests a differential diagnosis—all in real time.

In the context of agents, you can think of RAG as a single-step look-up for the agent, where the single step is just doing retrieval/look-up on a static base once. On the other hand, agentic RAG involves multiple steps, the emphasis here is on retrieving the right information to solve for the problem. I.e., its not a one-off

🤔 When to Use Standard RAG vs. Agentic RAG

• Standard RAG is sufficient when:

  • You have a single, well-defined knowledge source (e.g., a company wiki).

  • Queries are simple (e.g., "What’s our return policy?").

  • Cost and latency are critical.

  • Use Case - Standard RAG in E-Commerce: An online retailer uses standard RAG for a chatbot answering "What’s the warranty on this laptop?" The bot retrieves the answer from a product catalog, a single, structured source, keeping responses fast and cheap.

• Agentic RAG is important when:

  • Multiple, diverse sources are needed (e.g., web, internal docs, APIs).

  • Queries are complex (e.g., "Plan a marketing campaign based on competitor analysis").

  • Iterative retrieval and validation are required.

  • Integration with external tools is essential.

  • Use Case - Agentic RAG in Supply Chain Management: A logistics firm needs an AI to optimize shipping routes. The agent queries weather APIs, traffic data, and warehouse inventories, iteratively refining its plan based on real-time constraints, then schedules deliveries via an external system. Standard RAG couldn’t handle this multi-step complexity.

Conclusion: RAG’s Enduring Legacy

RAG (more importantly, retrieval optimization) is not a passing trend. It’s a fundamental architecture for building AI systems that access and reason over information. While LLMs evolve, the need for external knowledge persists (be it static or dynamic).

To summarize, here is why RAG remains essential, even in 2025 and beyond:

• RAG is more than "Naive RAG": The original definition of RAG emphasized end-to-end fine-tuning of both the retriever and the generator. We somehow lost sight of that and kept focusing on retrieval, separately from generation (i.e., fixing the generator LLM mode, aka “naive” RAG). While "naive RAG" (just optimizing retrieval parameters) can be a good starting point, the true power of RAG lies in optimizing the entire system. This includes fine-tuning embedding models for specific domains, employing sophisticated chunking strategies, and even leveraging re-ranking techniques.

• Fine-tuning Matters: Fine-tuning embedding models on in-domain data significantly boosts retrieval accuracy, leading to more relevant context for the LLM. This, in turn, improves response quality and reduces hallucinations. The benefits of fine-tuning are not universal, however, and careful evaluation is necessary to determine its effectiveness for a given dataset and task.

• Long-Context Models are Not a Silver Bullet: While long-context LLMs can process larger amounts of text, they don't eliminate the need for RAG. Retrieval acts as a crucial "focusing mechanism," preventing the LLM from being overwhelmed by irrelevant information. Strategic chunking and optimized retrieval can even enable smaller, more efficient LLMs to match the performance of larger models on certain tasks.

• Agentic RAG Extends RAG's Capabilities: Agentic RAG elevates RAG beyond simple retrieval. By incorporating AI agents, RAG systems can dynamically manage multi-step retrieval processes, interact with various tools (databases, APIs, web search), reformulate queries, validate retrieved information, and even take actions. This makes RAG suitable for complex, real-world scenarios that require more than a single information lookup.

• Choosing the Right RAG Approach: Standard RAG is often sufficient for simpler tasks with well-defined knowledge sources. Agentic RAG shines in situations requiring complex reasoning, multi-source integration, and interaction with external systems. The choice depends on the specific application's needs and constraints.

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

The Accelerated Ripple Effect and Hidden Tech Debt of Generative AI

Traditional AI models followed a predictable pipeline and still had a hidden technical debt. Now, with the rise of generative AI models like large language models (LLMs), the debt has been exacerbated. Alongside these models, there are additional layers, such as data retrieval, embeddings, safety, governance, etc. All of which need to be integrated into the AI/ML life cycle. Below are a few examples:

• Data Management: predictive models are typically trained on structured, labeled datasets, focusing on historical data to predict future outcomes. Generative models use larger, more diverse datasets, often unstructured, to learn patterns and generate novel content. Managing this data for quality, bias, and provenance presents a greater challenge.

• Model Training (and fine-tuning): predictive training optimizes specific metrics like accuracy or precision.  The process is generally more straightforward and less computationally intensive than generative models. Training LLMs (emphasis on L) needs more computational resources and longer training pipelines. Continuous learning, fine-tuning, and techniques like Reinforcement Learning from Human Feedback (RLHF) are often necessary (see this post for more details).

• Deployment:  predictive model deployment typically involves integrating the model into an existing application or system for real-time or batch predictions. Deploying LLMs at scale adds more challenges, be it low-latency inference, high availability, or managing fluctuating workloads, which requires specialized infrastructure (compute, storage, and networking) and inference optimization techniques.

With more components in generative AI (especially with retrieval involved)  comes more changes. Changes in one area, such as the training data or model architecture, can trigger cascading effects across the entire system, impacting everything from retrieval effectiveness to monitoring metrics (Change one thing, changes everything or CACE++).  

A Longer AI Lifecycle

More components lead to a denser lifecycle. Each step in the lifecycle encompasses more steps. Let’s explore those a bit:

Data-centric tasks:

Generative AI introduces new stages and expands existing ones to handle the unique data requirements of complex AI systems.

• Synthetic Data Generation (SDG): Traditional ML primarily focuses on cleaning and transforming real-world data. Generative AI adds the step of generating synthetic data (SDG) to augment existing datasets, especially in cases with limited or sensitive data (usually necessarily for training foundational models).

• Data Collection & Management: Traditional ML pipelines typically deal with structured data and simpler data management processes. Generative AI requires managing diverse data sources like synthetic data, embeddings, and external knowledge bases, demanding more sophisticated data management strategies. 

• Embedding Management:  Traditional ML relies heavily on manual feature engineering. Generative AI introduces embedding management as a core component, using techniques like pre-trained language models (e.g., BERT) to generate vector representations of data. Storing these embeddings in specialized (vector) databases enables semantic similarity searches and allows models to understand relationships between data points, moving beyond manual feature creation.

Model-Centric tasks:

Generative AI introduces new approaches to model training and customization to cater to the advanced capabilities of modern AI systems.

• Model Training: While traditional ML trains task-specific models on static datasets, Generative AI leverages the power of transfer learning by pretraining foundation models on massive datasets and then fine-tuning them for specific tasks. For instance, instead of training a chatbot from scratch, developers can fine-tune a pre-trained GPT-4 language model on a dataset of customer service conversations to achieve better conversational abilities.

• Model Customization & Alignment: Traditional ML typically evaluates model performance on held-out data. Generative AI emphasizes continuous fine-tuning and alignment with human feedback, ensuring models remain relevant and adapt to evolving data and user needs. 

AI System Deployment & Orchestration tasks:

Generative AI tackles the challenges of deploying and managing AI systems with multiple interdependent components. Traditional ML primarily deploys single models with fixed logic. Generative AI focuses on deploying and managing compound AI systems with multiple interacting components, including LLMs, other AI models, agents, and APIs. This requires special orchestration and integration techniques. 

Safety & Monitoring tasks:

Generative AI models are unpredictable, which gives rise to the need for new strategies for ensuring responsible AI (guardrails for outputs) and maintaining high performance in compound systems:

• Guardrails Implementation: Traditional ML focuses on monitoring model performance for degradation. Generative AI extends this by implementing guardrails and safety mechanisms that prevent unsafe, biased, or inaccurate outputs (especially when there is no transparency on how models were trained and to account for model emergent behavior). This involves incorporating ethical considerations and regulatory compliance into the development process. 

• Real-Time Retrieval Integration: While traditional ML models rely solely on training data, Generative AI applications more often rely on retrieval integration, allowing AI systems to access external knowledge bases during inference. This ensures the system can provide up-to-date responses and adapt to new information without requiring full retraining. Examples include a customer support chatbot retrieving the latest product specifications from a knowledge base during a conversation or a legal research tool retrieving relevant case law from a legal database in real-time.

• Monitoring & Feedback: Traditional ML monitors basic performance metrics. Generative AI expands monitoring to include human interactions, model outputs, and retrieval performance for comprehensive system evaluation and improvement. Human feedback plays a crucial role in ensuring responsible AI. This could involve collecting user feedback on the helpfulness of a chatbot's responses, analyzing conversation logs to identify patterns of misunderstanding, and using this feedback to improve the chatbot's training data and fine-tune its language model.

• Prompt Tuning and Optimization: Model behavior is usually fixed after training in traditional ML. Generative AI introduces the concept of prompt engineering, tuning, and optimization, which is particularly relevant for LLMs. This involves refining model behavior by optimizing the initial prompts provided to the model based on real-world feedback, enabling continuous adaptation without requiring full retraining. That said, I doubt prompt engineering is a long-term solution (see this post for more details)

The denser AI lifecycle introduces more optimization knobs and more questions to be answered. How much data should be synthesized?  How elaborate should the guardrails be?  Organizations must weigh the marginal benefits of each additional unit of effort (e.g., another data point, a more refined prompt) against the marginal costs.

Emerging Roles & Disciplines Beyond MLOps

More components lead to new roles and disciplines.  Machine Learning Operations (MLOps) has provided a solid foundation for operationalizing traditional machine learning models, but it falls short of addressing the challenges posed by generative AI systems. This entices the creation of new operational disciplines like LLMOps and GenAIOps, and the rise of new personas such as the "AI engineer”,  responsible for tweaking the many knobs of AI applications to ensure that applications meet their specific use case requirements. 

But before discussing AI engineering, let’s try to understand the distinctions between various “ops” disciplines, starting with MLOps. 

• MLOps: Focuses on the end-to-end lifecycle of traditional machine learning models, including data preparation, model training, evaluation, deployment, and monitoring. MLOps emphasizes automation, reproducibility, and scalability for individual models. 

• LLMOps: can be thought of as a branch of MLOps that deals specifically with the operational challenges of LLMs. LLMOps address aspects like prompt engineering, fine-tuning, model alignment, and managing the unique infrastructure requirements of LLMs. 

• GenAIOps: Broader than LLMOps, GenAIOps encompasses the operationalization of all generative AI models, including LLMs, image generation models, and others. It focuses on managing the complexities of training, deploying, and monitoring generative AI systems.GenAIOps addresses the challenges of deploying and scaling generative models, which often require specialized hardware (e.g., GPUs) and efficient resource allocation.

• RagOps:  Focuses specifically on the operational aspects of Retrieval Augmented Generation (RAG) systems, which combine LLMs with external knowledge sources to improve accuracy and grounding. RagOps addresses challenges like knowledge base management, retrieval model optimization, and ensuring consistency between the LLM and the retrieved information. 

• AI Engineering: This represents the next stage in the evolution of AI operations. AI Engineering entails managing the entire lifecycle of compound AI systems that may include LLMs, other AI models (including generative models), retrieval mechanisms (like those used in RAG), external knowledge bases, and human-in-the-loop components. AI Engineering emphasizes system-level considerations, including orchestration, integration, safety, and continuous learning.

Navigating the New Found Challenges of Generative AI

Generative AI, particularly LLMs and multi-modal models, are reshaping the tech landscape and introducing new use cases.  We are moving beyond static predictive models to a dynamic world where AI can generate novel content, engage in conversations, and even drive decision-making processes.  This transition, however, is not “free”. 

The complexity of those generative AI systems increases the technical debt of traditional AI, introducing new layers of complexity related to data management, model training, deployment, safety, guardrails, and more.  Organizations looking to capitalize on the promise of generative AI should approach this paradigm with a broader perspective, acknowledging both the opportunities and the costs and complexities involved.

One of the not-so-obvious changes is the expansion of the AI lifecycle, be it around the data, the model, or the deployment machinery. Each of these expanded stages introduces new "knobs" to tune and optimize, creating more overhead for AI practitioners. This overhead also gives rise to specialized roles and personas like the AI engineer, who is now responsible for managing this complexity.

Before we close out, here are some thoughts to approach this new shift: 

Think Big Picture:  Before jumping on the generative AI bandwagon, consider a cost-benefit analysis to account for the hidden costs associated with new requirements such as data management (e.g., SDG),  specialized infrastructure, and the need for new talent (or skills, i.e., bigger training budges). Be willing to adapt or abandon existing infrastructure that is no longer suitable for generative AI to unlock new capabilities faster.

Build the right abstractions (and Platforms): Consider building customizable AI platforms that can reduce the generative AI tech debt and reduce time to value when building AI applications (usually through well-defined abstractions, interfaces, and APIs). Here are some quick ideas:

• Built-in prompt management: Prompts are artifacts. Make it easier to experiment with new prompts, share versions and distribute them. 

• Automating training: Implement automated workflows for data preparation, hyperparameter optimization, and model selection. Includes pre-trained models.

• Simplify Evals: provide a dashboard with performance metrics, reporting, and alerting for new models.

• Simplify Deployment: provide APIs to automate  AI system deployment to various environments (with pre-built templates).

• Surface alerts and monitoring dashboards: make it easier to instrument AI applications for monitoring and logging, and simplify the configuration alerts.

• Build responsibly: integrate security policies, add access control, and audit knobs.

• Tailor to the needs: Prioritize value and tailor the platform to the specific needs of different user personas. For example, you can start by offering tiered access and functionalities. 

Cultivate a Culture of Continuous Learning: Generative AI is constantly evolving.  Invest in training and development for your employees, encourage knowledge sharing, and stay informed about the latest advancements to ensure your organization remains at the forefront of this transformative technology.

By far, Generative AI is the fastest-evolving technology I have encountered in my professional life. It's not merely a new tool, it’s a paradigm shift that is forcing organizations to reassess their approach to existing use-cases. It also encourages them to consider how AI can accelerate their business by discovering new workflows, and use-cases.

For more on building AI products, read my previous post on how to model value through the adapted whole-product framework:

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

This convergence is happening amidst growth in the global AI market, projected to reach $826 billion by 2030 from $196.63 billion in 2023 [Markets & Markets]. Within this growing market, the AI cybersecurity portion of it is an expanding segment, anticipated to reach $134 billion by 2030 from $24.3 billion in 2023 [Statista].

Below is an executive summary.

Key Drivers:

• Generative AI Surge: Generative AI is rapidly gaining traction, impacting various functions within organizations and fueling demand for AI-powered applications. [Gartner Forecast Analysis]

• Widespread Adoption: A large majority (84%) of organizations leverage AI-based cybersecurity tools, with 75% focusing on network security [World Economic Forum, Statista].

• Generative AI Spending Surge: Expenditure on generative AI solutions is expected to skyrocket, accounting for 35% of total AI spending by 2027, up from a mere 8% in 2023 [Gartner].

• Booming Market: The AI cybersecurity market is poised for exponential growth, projected to reach a staggering $134 billion by 2030, reflecting a dramatic increase from its 2023 valuation of $24.3 billion [Statista].

• Rapid CAGR: This translates to a remarkable compound annual growth rate (CAGR) of 22.3% from 2023 to 2033 [Market.us], driven by escalating cyber threats, widespread cloud adoption, and the demand for proactive and scalable security solutions.

• AI Software Spending Boom: Global spending on AI software is projected to reach $297.9 billion by 2027, with generative AI software expenditure increasing significantly. [Gartner Forecast Analysis]

• Expansion of the Attack Surface: Cloud, growth in the number of IoT devices has expanded the attack surface, creating new vulnerabilities that require AI-driven security measures. [Market.us]

• Growing Awareness and Investment: Organizations are increasingly recognizing the importance of AI in cybersecurity, leading to increased investments in research, development, and deployment of AI-powered security tools. [World Economic Forum]

• Cybersecurity Talent Gap: The cybersecurity industry faces a significant talent shortage, highlighting the need for AI-powered solutions to automate tasks and augment human expertise. [World Economic Forum]

• Regulatory Landscape: Cybersecurity and privacy regulations are becoming increasingly stringent, influencing the development and adoption of AI-powered security solutions. [World Economic Forum]

• Cybersecurity Talent Shortage: The cybersecurity industry faces a significant talent gap, making it difficult for organizations to effectively manage security operations. AI can help bridge this gap by automating tasks, providing insights, and augmenting human expertise. [World Economic Forum]

• Collaborations on Trustworth AI: focus on advancing safe, secure, and trustworthy AI, aligning regulatory frameworks, promoting ethical AI research, and broadening cooperation in AI protection and cybersecurity. [UAE and US Collaboration on AI ]

  • The U.S. and the UAE reaffirm their commitment to advancing safe, secure, and trustworthy artificial intelligence (AI) technologies.

  • Key principles include fostering international AI frameworks, aligning regulatory frameworks, and promoting ethical AI research and development.

  • The collaboration will focus on advancing safe, secure, and trustworthy AI, aligning regulatory frameworks, promoting ethical AI research, and broadening cooperation in AI protection and cybersecurity.

  • President Biden and President H.H. Sheikh Mohamed bin Zayed Al Nahyan will oversee the development of the memorandum of understanding.

  • The U.S. and the UAE are committed to deepening collaboration in AI and related technologies for a more prosperous and secure future.

Key Trends

• Integration of AI into Enterprise Applications: Over 70% of independent software vendors are expected to integrate generative AI capabilities into their enterprise applications by 2026. [Gartner Forecast Analysis]

• AI Adoption Phases: Organizations are at various stages of AI adoption, with many still in the experimentation or planning phases. [Gartner Forecast Analysis]

• Generative AI's Impact on Security: While generative AI offers benefits for content creation and automation, it also presents new security risks, such as the generation of malicious code and sophisticated phishing attacks. Organizations must adapt their security strategies to address these emerging threats. [CBH]

• Explainable AI for Enhanced Trust: The lack of transparency in some AI algorithms can hinder trust and adoption. The development of explainable AI, which provides insights into how AI systems make decisions, is crucial for building confidence in AI-driven security solutions. [Gartner Forecast Analysis]

• Regulatory Landscape: Evolving cybersecurity and privacy regulations, such as GDPR and CCPA, are influencing the development and deployment of AI-powered security solutions, emphasizing data protection and responsible AI practices. [World Economic Forum]

Opportunities:

• Development of Advanced Threat Detection and Response Solutions: AI can significantly enhance threat detection accuracy, accelerate incident response times, and minimize the impact of cyberattacks.

• Proactive Security Measures: AI enables predictive analytics and threat intelligence, allowing organizations to identify and address potential vulnerabilities before they are exploited proactively. [Markets & Markets]

• Automated Security Operations: AI can automate routine security tasks, freeing up security professionals to focus on more complex and strategic initiatives. [Gartner Forecast Analysis]

• Enhanced Security for Cloud and IoT Environments: AI-powered solutions can address the unique security challenges posed by cloud computing and the growing number of connected devices. [Market.us

Challenges:

• Ethical Concerns and Bias: AI algorithms can be biased, raising ethical concerns regarding fairness and accountability. [CBH]

• Adversarial AI: Attackers can leverage AI to develop sophisticated attacks, posing new challenges for cybersecurity professionals. 

• Data Privacy and Security: AI systems rely on vast amounts of data, raising concerns about data privacy and security. [CBH]

• Explainability and Transparency: Lack of transparency in AI algorithms can hinder trust and adoption [Gartner Forecast Analysis]

Conclusion:

The AI security market is experiencing rapid evolution, driven by the need for solutions to counter rising cyber threats. With power comes great responsibility though, while AI offers potential for enhancing security, organizations must also address the risks associated with its malicious use. 

• Enhancing Security: AI offers significant potential for bolstering cybersecurity defenses, with organizations increasingly employing it for:

• Automating routine tasks: Freeing up human analysts for more complex threats.

• Real-time threat detection: Enabling rapid response and mitigation.

• Predictive analytics: Proactively identifying and addressing vulnerabilities.

• Emerging Risks: However, AI also presents new challenges as adversaries exploit it for malicious purposes:

  • Phishing attacks.

  • Automated malware development.

Related musing here:

Going in Details

AI Market Statista

Reference: Artificial Intelligence - Global | Statista Market Forecast 

Important Technologies Impacting Market Growth: 

• Machine Learning: Forms the foundation of the market and exhibits consistent growth throughout the period.

• Autonomous & Sensor Technology: Shows steady growth, gaining momentum towards the latter half of the projection period.

• Natural Language Processing: Experiences significant growth, becoming a major contributor to the overall market size by 2030.

• Computer Vision: Demonstrates strong and consistent growth, becoming the second largest segment by 2030.

• AI Robotics: While starting with a smaller market size, it exhibits rapid growth, becoming a prominent segment by 2030.

Key Observations:

• The overall AI market is projected to experience high growth, reaching nearly 800 billion USD by 2030.

• Machine Learning remains a dominant segment, while Computer Vision and AI Robotics emerge as important growth drivers.

• Natural Language Processing also shows notable growth, increasing its importance in various applications.

As for the top 10 countries, this bar chart depicts the estimated market size of the AI market in the top 10 countries for 2030.

• United States: Dominates the market with a projected size of 223.7 billion USD, significantly larger than any other country. The US remains the clear leader in the AI market, driven by factors like technological advancements, research and development investments, and a thriving tech ecosystem.

• China: Ranks second with a market size significantly smaller than the US but considerably larger than the remaining countries.it can be considered a strong contender, though with export controls

• Germany, France, Russia, Brazil, etc.: Exhibit relatively smaller market sizes than the US and China.

Gartner Forecast Analysis

Reference: Forecast Analysis: AI Software Market by Vertical Industry, 2023-2027 

AI software spending will grow to $297.9 billion by 2027. Over the next five years, market growth will accelerate from 17.8% to 20.4% in 2027, with a 19.1% CAGR. Government has the largest spend of over $70 million by 2027, but oil and gas is growing fastest with a 25.2% CAGR.

McKinsey: GenAI Adoption Spikes

Reference: The state of AI in early 2024: Gen AI adoption spikes and starts to generate value 

• Gen AI Adoption Surge: 65% of organizations regularly use gen AI, a significant increase from the previous year.

• Common Use Cases: Gen AI adoption is most prevalent in marketing, sales, product development, and IT, with an average of two functions utilizing it.

• Overall AI Adoption Growth: 72% of organizations now utilize AI, up from 50% previously, with increased adoption across various industries.

• Increased AI Investments: Organizations are allocating budgets for both gen AI and analytical AI, expecting further investment growth over the next three years.

• Gen AI Risks: 44% of respondents reported negative consequences from gen AI, including inaccuracy, cybersecurity concerns, and explainability issues, highlighting the need for risk mitigation.

• High-Performer Success: Organizations effectively leveraging gen AI attribute a significant portion of their EBIT to its deployment, often utilizing it across multiple functions and implementing risk best practices.

• High-Performer Challenges: These organizations also face challenges with data governance, integration, and operating models, emphasizing the importance of addressing these for successful gen AI adoption.

• Diverse Research Sample: The research involved a diverse group of participants across various regions, industries, company sizes, and functional specialties, providing valuable insights into the current AI landscape.

StackOverflow Developer Survey 

Reference: AI | 2024 Stack Overflow Developer Survey 

Similar to last year, developers remain split on whether they trust AI output: 43% feel good about AI accuracy and 31% are skeptical. Developers learning to code are trusting AI accuracy more than their professional counterparts (49% vs. 42%).

Professional developers agree the issue is not user error: twice as many professionals cite lack of trust or understanding the codebase as the top challenge of AI tools compared to proper training.

Market.us

Reference: AI In Cybersecurity Market Size, Share | CAGR of 22.3% 

• The Global AI In Cybersecurity Market size is expected to be worth around USD 163.0 Billion by 2033, from USD 22 Billion in 2023, growing at a CAGR of 22.3% during the forecast period from 2024 to 2033.

• AI in cybersecurity refers to the application of artificial intelligence technologies in detecting, preventing, and mitigating cyber threats and attacks. It involves using machine learning, natural language processing, and other AI techniques to analyze vast amounts of data, identify patterns, and detect anomalies that may indicate potential security breaches or malicious activities.

• The AI in cybersecurity market is rapidly growing as organizations recognize the need for advanced solutions to combat the increasing complexity and sophistication of cyber threats. AI technologies can enhance traditional cybersecurity measures by providing real-time threat detection, automated incident response, and predictive analytics to proactively identify and address potential vulnerabilities.

• AI in cybersecurity encompasses a range of applications, including malware detection, network intrusion detection, user behavior analytics, and threat intelligence. By leveraging AI algorithms, cybersecurity systems can continuously learn from new data and adapt to evolving threats, improving their accuracy and effectiveness in detecting and mitigating security incidents.

• The market for AI in cybersecurity is driven by factors such as the growing frequency and severity of cyberattacks, the prevalence of cloud,  IoT in the past decade, and the need for scalable and intelligent security solutions. Vendors and emerging startups are investing in AI capabilities to develop innovative products and services that address the evolving threat landscape.

• According to a report by Security Intelligence, the average total cost of a data breach in 2022 rose to ~$4.35 million, representing a modest increase of 2.6% compared to the previous year’s average of ~$4.24 million. This upward trend highlights the growing financial impact that cyberattacks can have on organizations worldwide.

• European Union Agency for Cybersecurity (ENISA) observed a noteworthy surge in the adoption of AI-based security solutions, with a remarkable 30% increase over the past year. This surge can be attributed to organizations’ proactive efforts to bolster their cyber resilience and protect sensitive data from sophisticated attacks.

World Economic Forum

• The World Economic Forum’s “Global Cybersecurity Outlook 2023” (see World Economic Forum's Global Cybersecurity Outlook 2023) report revealed that an impressive 84% of surveyed organizations are leveraging AI-based tools to bolster their cybersecurity capabilities. These organizations recognize the potential of artificial intelligence in fortifying their defenses and mitigating risks associated with cyber threats.

  • Cybersecurity Perception & Prioritization:

    • 95% of business executives and 93% of cyber executives agree that cyber resilience is integrated into their enterprise risk-management strategies.

    • Only 36% of organizational leaders who meet at least monthly on cybersecurity feel confident in their organization's cyber resilience.

    • 90% of respondents are concerned about the cyber resilience of third parties who have direct connections to or process their organization's data.

    • Only 25% of respondents stated that their most senior cybersecurity executive reports directly to the CEO.

    • 56% of cyber leaders meet with business leaders monthly or more often to discuss cybersecurity.

• Cybersecurity Talent Gap:

  • 59% of business leaders and 64% of cyber leaders ranked talent recruitment and retention as a key challenge.

  • Less than half of respondents reported having the people and skills needed today to respond to cyberattacks.

• Impact of Regulations

  • 73% of respondents agree that cyber and privacy regulations effectively reduce their organization’s cyber risks, a significant increase from the previous year.

  • 76% of business leaders and 70% of cyber leaders agree that stronger regulation enforcement would increase their organization's cyber resilience.

Markets & Markets

Reference: Artificial Intelligence in Cybersecurity Market Share, Forecast | Growth Analysis & Opportunities [2030]

The AI in Cybersecurity market report outlines a technology roadmap from 2023 to 2028, breaking anticipated advancements into short-term, mid-term, and long-term phases.

Short-term Roadmap (2023-2025)

• Automating Incident Response: AI will be increasingly utilized to automate routine incident response tasks, reducing response times and minimizing manual errors.

• Integration with Threat Intelligence: AI-driven threat intelligence platforms will be integrated with security tools, providing contextual awareness to enhance detection capabilities.

• Explainable AI Adoption: The incorporation of explainable AI will improve transparency and enable a better understanding of threat detection mechanisms, helping security teams trust and verify AI decisions.

Mid-term Roadmap (2025-2028)

• Intelligent Security Systems: Combining AI, cognitive computing, and automation will result in systems that can reason, learn, and autonomously make decisions, transforming cybersecurity operations.

• Federated Machine Learning Models: The use of federated learning models will rise, allowing organizations to collaboratively enhance threat intelligence without compromising data privacy.

Long-term Roadmap (2029-2030)

• Human-AI Collaboration in Security Operations: AI and human operators will collaborate seamlessly, leveraging each other’s strengths to optimize decision-making and threat response.

• Quantum-Resistant Encryption: AI will be integrated with quantum-resistant encryption protocols to secure sensitive data against emerging threats.

• AI and Blockchain Collaboration: AI systems will work with blockchain technology to safeguard data transactions, ensuring data integrity and security.

Market Overview

AI Cybersecurity Market

• Market Size: Valued at approximately $22.4 billion in 2023.

• Growth Projection: Expected to grow at a Compound Annual Growth Rate (CAGR) of 20.8%, reaching around $147.5 billion by 2033.

Overall AI Market

• Market Size: Estimated at $196.63 billion in 2023.

• Growth Projection: Projected to grow to approximately $826 billion by 2030.

Market Comparison

The AI cybersecurity market constitutes a substantial segment of the broader AI market, although smaller in size. In 2023, it represented about 11.4% of the overall AI market, with the potential to increase to 18% by 2030. Both markets are witnessing rapid expansion, driven by AI's growing role in safeguarding digital assets and the increasing importance of cybersecurity in an interconnected world.

CBH

Reference: How to Craft a Proactive Generative AI Strategy To Manage Cybersecurity Risks 

• 35% of global companies are currently using some form of artificial intelligence (AI) in their operations, with 42% exploring AI integration, as per an Exploding Topics analysis.

• Generative AI and LLMs are beneficial for content creation but poses risks such as publication of misleading content, data breaches, bias perpetuation, and legal/ethical violations. Risks involves establishing governance programs, implementing defense in depth, and ensuring model security through techniques like input data validation and anomaly detection.

• The National Institute of Standards and Technology (NIST) has developed the Artificial Intelligence Risk Management Framework (AI RMF) to help businesses mitigate AI risks. See: NIST AIRC - Playbook 

• Characteristics of a trustworthy (a better umbrella here is responsible AI) generative AI system include safety, reliability, secure data handling, explainability, and fairness in training data.

• Organizations should continuously evaluate and enhance their security measures to adapt to the evolving AI landscape and regulatory frameworks.

Use-cases and technological trends

Sovereign AI

• Sovereign AI: What it is, and 6 ways states are building it | World Economic Forum 

  • Security & Control: Maintaining complete control over data, algorithms, and infrastructure to mitigate risks of foreign interference, data breaches, and misuse of AI.

    • Secure Data Centers: Physically secure facilities with high-performance computing capabilities for data processing and storage.

    • Data Management Platform: Tools for data acquisition, cleaning, transformation, governance, and secure storage (cloud, data lakes, etc.).

    • AI and Data Usage Policies: Establishing processes for reviewing the use of data in AI training and fine-tuning. Establishing processes for reviewing proposed use cases for AI. Publishing and enforcing these decisions through technical and non-technical means.

    • Supply Chain Security for AI: Generating and using AI BOM and provenance data. Signing and verifying AI models.

  • AI Development Platform:

    • ML Frameworks & Tools: Providing access to leading frameworks (TensorFlow, PyTorch) and development tools for building, testing, deploying, and monitoring AI models.

    • AI Hardware: Investing in or securing access to specialized hardware like GPUs and TPUs to accelerate AI training and inference.

    • Internal Development Platforms: Enhancing existing IDPs to give developers access to the AI hardware they need, generate supply chain security artifacts for AI, and enforce AI security and compliance policies; all through automation rather than paperwork.

  • Software & Applications:

    • Vertical AI Solutions: Developing AI applications tailored to specific national priorities, such as healthcare, finance, defense, etc.

    • Core AI Technologies: Building expertise and tools in key areas like NLP, computer vision, and robotics.

  • Ethical Considerations: Ensuring the responsible and ethical development of AI, addressing bias, transparency, and societal impact.

• Real stories backing up the use-case

  • Indonesia: Study: 51 percent of Indonesian financial institutions are focusing on deploying GenAI for daily tasks - TNGlobal 

  • India: Finding The Sweet Spot For India’s Sovereign AI 

  • UK: Why The UK Should Have Its Own Exascale AI/HPC Machine, And How 

AI Technology Trends

In a previous article, we explored the idea of compound AI systems as key building blocks in the making of whole products. We discussed agents/agentic, RAG, and other enabling technologies. These technologies emerged from trends through which the next generation of AI products will be made of, including those for AI security. These trends/technologies will be used to amplify a system's security but are also systems that need to be secured. For example:

• Long-context reasoning: By leveraging long-context models, AI security systems can analyze vast amounts of data over extended time periods, identifying patterns and anomalies that may indicate emerging threats. This can be important for detecting sophisticated attacks that may involve slow, subtle manipulations over time. For instance, detecting insider threats that gradually escalate privileges or analyzing long-term trends in network traffic to identify subtle deviations indicative of a compromise.

• Text-to-action: Combining long-context reasoning with text-to-action capabilities allows AI security systems to not only identify threats but also automatically generate and execute appropriate responses. For example, an AI system could analyze a complex security incident report, identify the root cause, and automatically initiate actions like isolating infected devices, patching vulnerabilities, or alerting relevant personnel.

• Agents/Agentic will enhance AI security by enabling proactive and autonomous threat detection and response. These AI agents can continuously monitor systems, analyze data, and adapt to evolving threats, automatically triggering remediation actions to neutralize attacks. They can also conduct proactive threat hunting, automate investigations, personalize security policies based on user behavior and context, and improve collaboration between AI systems and human analysts. Ultimately, agents/agentic technology empowers organizations to build more robust and resilient security systems that can effectively combat the ever-growing complexity of cyber threats.

These new AI trends/technologies can be integrated into security products to enhance their effectiveness. For example, integrating agent-based threat detection into a SIEM (Security Information and Event Management) system can improve its ability to identify and respond to threats in real-time. Similarly, incorporating long-context reasoning into vulnerability management tools can enable predictive analysis and proactive patching.

Summary

The AI cybersecurity market is well positioned for growth, driven by the increasing sophistication of cyber threats and the widespread adoption of AI-powered security solutions. While AI offers the potential for enhancing cybersecurity, organizations must also address the ethical concerns and risks associated with its use.

Key Takeaways:

• Rapid Market Growth: The AI cybersecurity market is expected to reach $134 billion by 2030, with a CAGR of 22.3%.

• Generative AI is a Key Driver:  Spending on generative AI solutions is rising, and it's being rapidly integrated into enterprise applications.

• AI Enhances Security: AI enables proactive threat detection, automated security operations, and enhanced security for cloud and IoT environments.

• Emerging Risks:  The malicious use of AI, including phishing attacks and malware development, presents new challenges.

• Responsible AI is important: Addressing ethical concerns, bias, and data privacy is essential for the responsible development and deployment of AI in cybersecurity.

Looking Ahead:

• Organizations must proactively manage the risks associated with AI while harnessing its potential to strengthen cybersecurity defenses.

• To ensure the safe and ethical development of AI for cybersecurity, continued collaboration between governments, industry, and researchers is needed.

• Trends like long-context reasoning, text-to-action capabilities, and the development of autonomous AI agents will shape the future of AI in cybersecurity.

By navigating the opportunities and challenges responsibly, organizations can leverage AI to build a more secure and resilient digital future.

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

Introduction

The other day, I found myself reflecting on a classic concept that I was taught in business school, Maslow’s hierarchy of needs,  a simple but powerful framework for understanding human motivation, with basic physiological needs at the foundation (air, food, water, shelter, sleep, clothing,...) and the pursuit of self-actualization at the pinnacle.

This got me thinking, in the world of tech (especially AI) and products, what is an equivalent? I mean, users always have needs, and needs in the product vary significantly subject to the use case and the problem being solved, but a spectrum definitely exists. Is there a model or a framework we can use to identify what constitutes the “right” product for customers and what customers would expect of the product? Luckily, Geoffrey Moore's "Crossing the Chasm" provides some answers. In his book, Moore references Levitt’s Whole Product Model, and goes further to simplify by introducing the “Simplified Whole Product Model”. 

In this post, we will internalize Moore's model, expand it, and show how it can be applied specifically to AI products (applies to any product as well). We'll dive into the trade-offs inherent in building AI applications and illustrate these concepts with real-world examples.

My goal is that after you read this post, you should have a mental model and a framework for building great/usable AI products, which would help you not only think about the technology, but also how it fits in the big picture. 

The Whole Product Primer (Plus it’s Descendants)

The "Whole Product" model revolves around the idea that a core/generic product must be complemented by additional services and interfaces (aka enablers) making up the Whole Product which should provide a solution to the customer’s problem and to address their needs. 

In Geoffery Moore’s book, the core/generic product is defined as the fundamental offering or technology that a company produces, which may not be sufficient to fully solve the customer's problem or meet their needs. 

This is where the outer ring comes into play. It represents the whole (expected) product, which is divided into sectors. This outer ring encompasses all the additional elements that customers expect or require to make the core product fully functional and valuable to them, let’s call them the “enablers”.

The Adapted (Simplified) Whole Product Model

In the tech industry, companies often prefer to build upon existing open-source projects or technologies rather than developing everything from scratch. These companies focus on adding unique value through layers of customization, support, consulting services, integrations, and proprietary patterns, creating a whole product that is more than the sum of its parts.

Furthermore, any successful technology is bound to become commoditized over time, a strategy we often see in tech employed by competitors who gain from doing so, forcing value into higher layers in the value chain (which they usually have thus wanting to commoditize). Recognizing this, companies need to continually innovate and differentiate their offerings to maintain a competitive edge (related, see a previous post on AI market dynamics and what companies in the space focus their efforts on).

Therefore, let’s adapt the simplified whole product model with two key adjustments. First, we'll shift from fixed sectors to a more modular, petal-like structure. This reflects the interconnected yet distinct components that comprise the whole product layer. Second, we'll introduce a new layer above the whole product layer, called the "differentiated product layer". This layer will highlight the unique value propositions that set companies and their products apart, showcasing how they create the most value for their customers. 

To be more concrete, let’s show how this can be applied to Slack for example (this is just for illustration purposes, the real differentiators could very well be very different). 

In addition to representing the product's enablers differently using petal-like modular components, we added a new layer to highlight the differentiators. In the example above and in the case of Slack, enablers could be threads, Slack Connect, the workflow builder, and/or Slack AI. 

We are very close to being done here with the adaptations, so we will add one last thing to our new framework. In addition to the differentiated layer, we would like to model customizability for products. I.e., one customer's whole product may not be the same for another. I.e., not all customers desire exactly the same features, so it’s important to cater based on customers' constraints/needs. For example, generically, some customers value safety/security over cost, others might value speed, etc. 

Let's continue the slack example. Slack might have different customers to cater for. Enterprise customers, use it mainly as a means for company-wide communication, in that case, the focus will be security and compliance with the company’s communication policy, leading to: 

• Prioritized Enablers: Enterprise-grade security, granular permissions, compliance features (e.g., data retention policies)

• Emphasized Differentiators: Slack Connect for secure external collaboration, integration with enterprise security tools

Another use-case, focus area might be on developers, and Slack being part of their dev/test workflows. In that case, the focus will be on developer productivity and collaboration, leading to:

• Prioritized Enablers: Integrations with development tools (e.g., GitHub, Jira), code snippets, powerful search

• Emphasized Differentiators: Workflow Builder for automating tasks, Slack AI for code suggestions and knowledge retrieval

The takeaway here is that versatility can be a core differentiator on its own because it allows for tailored product experiences. Another way to look at it is that the constraint being imposed defines the core value proposition of the product and how it is shaped to best serve and differentiate in a particular space.  

In our example, Slack can tailor its offering to different customer segments, highlighting the features and capabilities that are most relevant to each group. This customization not only enhances the user experience but also strengthens Slack's value proposition in a competitive market.

Towards Whole AI Products (aka Systems) 

Hopefully, you have a handle on the adapted simplified whole product framework by now. Next, we will focus on using the framework and mapping it to the super exciting world of AI applications. 

Key Ingredients to Building AI Applications

Before the mapping, let's do a quick primer on the core ingredients of AI products and applications (a sample not an exhaustive list). We will cover the key ideas, but we won’t delve into the technical intricacies. For that, there are many resources available, some of which I will be referencing as we go for further reading.

LLMs AND/OR SLMs

In a previous post, I introduced the model product possibilities frontier, a framework for studying the tradeoffs and use cases of large language models (LLMs) and Small Language Models (SLMs), which I will not be repeating here for brevity. That said, the choice of which models and their size to use is a key ingredient for building generative AI applications and products. 

Here are a few considerations/questions to ask yourself when reasoning about the tradeoffs:

• What are your most favorable constraints? Is it speed, quality, cost, etc? 

• What about privacy? Do you value data staying in-house (Small models are easier/cheaper to deploy, train, and serve on-premise)

• How are you going to evaluate the performance of your AI applications that make use of these models?  

• Is a smaller model easier to test and evaluate (think about the specificity as truth vs the versatility of LLMs which introduces more variability/hallucination, and thus makes it harder to test)

While we did not call it out explicitly, large or small models can be fine-tuned and aligned. This is covered in greater detail in this post. 

Retrieval Augmented Generation (RAG)

I’d say 2023 was the year of RAG. We went from naive RAG to Advanced RAG. I liked naive tbh, it communicated simplicity, but well, these days advanced is perceived as better, something we are yet to fix, but that’s a different story 🙂. This paper provides more details. 

RAG workflows are comprised of many moving pieces and optimizations. The goal is to retrieve the best content to augment the context for LLMs (text generation) with necessary information. In that case, LLMs become curators rather than innovators/generators of sorts (they shape the retrieval results and make them relatable as an output to a user but are not the source of knowledge themselves). To give you an idea of the moving pieces involved with RAG, here is a rough brain dump (feel free to surf the mindmap as you please, I will not enumerate the details here for brevity). 

When considering RAG for building AI applications, some questions come to mind around tradeoffs and decisions, usually between RAG, long context, and Fine-tuning. Again, we won’t cover details, but here are a set of questions that you can ask to inform your decision. 

• Does the application require access to external data sources to provide accurate and up-to-date responses (RAG usually makes sense if data freshness is important, especially since language models are point-in-time trained)?

• Is it crucial for the model to adapt its behavior, writing style, or domain-specific knowledge to match specific requirements (RAG does not customize behavior, fine-tuning would make sense if behavior customization is a goal)? 

• How critical is it to minimize the risk of the model generating false or fabricated information (hallucinations)? 

• How much labeled training data is available for fine-tuning? Does it adequately represent the target domain and tasks? 

• How frequently does the underlying data change? How important is it for the model to have access to the latest information?

• Is it important to understand the reasoning behind the model's responses and trace them back to specific data sources?

• How important is minimizing computational costs for your project or organization?

• Do your typical queries require multi-step reasoning (complex queries or simple questions)?

• How important is the ability to scale your solution to handle a large number of queries?

Finally, here is a short guide I created to help you make informed decisions about RAG/Fine-tuning if you wish to use it: 

For more information, check the below papers which I found very useful in understanding the differences and the trade-offs:

• [2407.16833] Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

• [2401.08406] RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture 

• [2312.05934] Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs 

RAG today has become synonymous with building AI applications in some contexts. What’s clear is that RAG is not one component, it's a system comprised of many moving pieces with levers to turn on/off for what makes sense most subject to context and use-case. 

Agents ft. Agentic OR Agentless! 

In addition to the model (LLM/SLM), RAG, there is the notion of agents and agentic workflows (also agentless to counter 🙂). While this is again not going to be a deep-dive, let’s cover the basics. 

What are agents, what is agentic behavior, and why agentless sometimes?

The notion of agents is not new. Agents have existed for decades (see this for examples), they are officially called Intelligent agents.  Below is the definition of an Intelligent Agent.  

In intelligence and artificial intelligence, an intelligent agent (IA) is an agent acting in an intelligent manner. It perceives its environment, takes actions autonomously in order to achieve goals, and may improve its performance with learning or acquiring knowledge. An intelligent agent may be simple or complex: A thermostat or other control system is considered an example of an intelligent agent, as is a human being, as is any system that meets the definition, such as a firm, a state, or a biome.^[1]

What’s changed is that with the advent of LLMs is that agents got a capability boost, from symbolic, rule-based, predefined simple actions with low autonomy (see the history post for more details, you may be reminded of expert systems) to being able to understand and generate natural language, learn and adapt across diverse domains, and perform complex, autonomous actions. In today’s context,  An agent is a software entity possessing autonomy, goal-oriented behavior allowing it to operate and generalize cross-domains and take complex actions. 

Agentic behavior, in this context, refers to an agent's ability to operate independently, make decisions aligned with its objectives, and execute actions (potentially with tools/functions-calling,...) to achieve those goals. The level of agency can vary based on factors like the complexity of the environment, the agent's goals, and the degree of user supervision required. More agentic systems can operate autonomously in intricate environments, pursue complex objectives, and utilize advanced techniques such as planning and tool use

Finally, there is the notion of flow-engineered / AGENTLESS which relies on determinism and only interfaces with language models for specific clarifying actions, in a sense similar to intelligent agents of the past, with the exception of having access to external intelligence capable of better identifying areas where the predefined action could be taken. 

To simplify your life, I've included this visual below (higher resolution here) to help you build a clearer mental picture of agents/agentic. 

Other components

Besides agents, RAG, the models, there are multiple other ingredients that go into building an AI applications, going through each and every one is out of scope for this post, but here is a non-exhaustive list for reference: 

• Data Pipeline: System for collecting and processing data, think extractions, transformation. 

• Knowledge Base: where the processed knowledge/data is stored. 

• User Interface: Web or app interface for users.

• Query/prompt Cache: avoid unnecessary query round-trips which can greatly reduce costs. 

• APIs: To interface with other systems. 

• Infrastructure: an important component that is usually overlooked, where to host the model/app, how to scale it, etc. 

• Observability: be able to log, monitor, trace an AI application. 

• Model Gateways: to interface between the user-query and it’s destination. Along the way, it makes sure the query is authenticated/authorized, masked/audited for sensitive content (e.g., PII), and finally routed to the best model to serve the query (best here is dependent on the use-case, see this post)

• <Many more>

As I was writing this, I came across this blog post, which discusses the technical details of some of the most used components for AI applications. 

Compounds AI Systems

You have come a long way brave reader, the end is near, and you shall be rewarded. So far we have been separately covering important components and ingredients that are key to the making of AI applications, but what makes the interconnection of these components towards achieving a shared goal? A system!

A system is a group of interacting or interrelated elements that act according to a set of rules to form a unified whole

Zaharia et. al recently introduced the notion of Compound AI Systems. In their post, they define it as: 

A system that tackles AI tasks using multiple interacting components, including multiple calls to models, retrievers, or external tools. In contrast, an AI Model is simply a statistical model, e.g., a Transformer that predicts the next token in text.

The authors also emphasize the complexity of designing AI systems: 

While compound AI systems can offer clear benefits, the art of designing, optimizing, and operating them is still emerging. On the surface, an AI system is a combination of traditional software and AI models, but there are many interesting design questions. For example, should the overall “control logic” be written in traditional code (e.g., Python code that calls an LLM), or should it be driven by an AI model (e.g. LLM agents that call external tools)? Likewise, in a compound system, where should a developer invest resources—for example, in a RAG pipeline, is it better to spend more FLOPS on the retriever or the LLM, or even to call an LLM multiple times?

In their post, they showcase a table of AI systems, and the components they are composed of. Additionally, they highlight the need for optimization across the chosen components to build reliable AI systems. Below we extract the components mentioned in the post and categorize them into Ops (i.e., operations), Tools, Context/Knowledge, and models. 

If you remember in the previous section, we covered similar components and more as ingredients to build AI applications. The takeaway here is that building reliable AI applications takes a system not a singleton component. I.e., “the whole is more than the sum of the parts”

Another way to visualize it is to consider a dashboard looking like a cockpit with all knobs needed to build your AI application, here is an example of what that could look like:

Without abstraction, you’d have to configure all these knobs manually (i.e., you’d have to understand what each of these means). Nowadays, there exist many frameworks to do the orchestration which to a good extent abstracts away some if not all these details. Is that a good thing? I will let you decide. My take? It can be a good thing if you are experimenting, learning, but if reliability, performance, and security are concerns (and they should be), you’d still have to understand what all these knobs mean before you pick up automation/orchestration tooling. Think of it this way, do pilots just take on their license without understanding what each and every knob in their cockpit means? I would guess not! But when they do, they can auto-pilot if they choose to because at any point they CAN switch back to pilot-mode and turn on the right knobs to fly the plane safely. 

From Compound Systems to Whole AI Products

Now that we understand the key ingredients needed to build AI applications and compound AI systems, which is the technical pattern we will use to describe the components of an AI application and how they intermingle, let’s go ahead and map that back to our adapted simplified whole product framework. 

As you can see in the diagram above (higher resolution here), we took the components of compound AI as we categorized them in the previous section, and mapped them to the generic/core (right in the middle), and the whole product layer comprised of one or more enablers. You may notice that we left out the differentiated product layer, that’s intentional. We will cover that in a coming section. What about the constraints? Let’s model them as well well.

The constraints will heavily depend on the use-case, I used “Enterprise” here as an example. For enterprise AI use-cases, safety, and reliability are important concerns. Using the constraints, we put emphasis on specific parts of the whole product, highlighting key enablers. In that case we chose legal, ops, gateway, and UX.  

Different use-cases will place different emphasis on the whole product, resulting in some layers being more important than others. Some use-cases even simplify the whole product by losing unneeded enablers, making the whole product leaner and more directed towards solving the problem/use case at hand. 

Defensibility AND Compound MOATS

Previously we took a tour to compare and contrast the current AI Market Landscape. We showed how companies that have a mission to better something other than just the model might have better odds in surviving in a competitive market (I.e., AI as an enabler vs. AI as the core product). We have also shown how companies are releasing open-source language models, which increases competitiveness and commoditizes the model layer completely making it pertinent for startups and companies to see defensibility through differentiation, i.e., what is the company’s MOAT? 

For defensibility, let’s summarize the most prominent strategies: 

• Having strong using communities and strong user engagement.

• Transitioning from Foundational Models to Purpose-Based Approaches

• Building Layers of Value Beyond the Model

• Differentiating at Various Layers of the AI Stack

Let’s briefly get into each. 

• Fostering a strong community and high user engagement: This involves cultivating a rapidly growing user base, harnessing the power of network effects, and creating a vibrant community that engages users across different generations. I.e., Who will use my product, what value to I provide beyond just the model, and why do I have a community in the first place?

• Transitioning from general foundational models to purpose-built applications: By focusing on specific user needs and problems, companies can tailor their AI solutions to provide more value and differentiate themselves in the market using existing business models, E.g., I already a social network, I make good money from Ads, how can I add more value to the existing community by incorporating AI?

• Building layers of value beyond the model: Invest in research to continually improve models and applications, leverage proprietary data (data as moat) for enhanced performance (after all garbage in, garbage out, gold in, gold out), and continuously refine products based on user feedback. By building a loyal customer base and offering unique value propositions, companies can establish a strong competitive advantage. 

• Differentiate by focusing various layers of the AI stack: This can involve developing superior AI models or smaller niche models (focusing on a tiny use-case but beating anyone else at doing it), providing scalable and efficient AI infrastructure, or creating user-friendly interfaces and seamless integrations (a GPT store, for example?). Each layer presents an opportunity for differentiation and can contribute to a company's overall defensibility.

These are just but some strategies that can be used to build moats, it is rarely a single component, it’s the sum of multiple to make a better whole defensible product. Compound MOATs ™ are the way! The last strategy is the one with lowest chances of surviving alone, so I’d consider at least two of the above strategies to start differentiating. Some questions to ask: 

• What processes do you have in place to ensure that AI models are being leveraged as enablers rather than being treated as end products?

• What strategies are you employing to rapidly grow your user base, create network effects, and foster a sense of community?

• What investments are you making in research, data, product refinements, and customer acquisition to build layers of value?

• What resources are you allocating to differentiate your company at the model layer, infrastructure layer, or application layer

• How are you evaluating and prioritizing potential areas of differentiation to ensure a sustainable competitive advantage?

Adding The Differentiated Product Layer

Alright alright alright, now that we understand moats/defensibility strategies, how do we model them back into our framework?! Using any (or additional) defensibility strategies to differentiate, additional components are added to the differentiated product layer in the model. In that case, we added strong community, integration with partners, a store/marketplace, innovations at the application layer (value above the model), and unique data. This layers makes a company’s set of Compound MOATs, which are also what create brand differentiation, loyalty, retention, etc.

AI Whole Products in Practice

It’s 2024, almost two years after the release of ChatGPT, almost 70 years after the perceptron, the first manifestation of neural networks (see this post for more details), and ~40 years after the creation of expert systems which was the closest Applied AI could get. In the post, I go into the details of why expert systems did not pan out (and partially led to an AI winter), but for brevity, it was a consumption gap, what we had back then in terms of compute, community, and technology was a far cry from where we are today. 

With LLMs showing a glimpse of what can be achieved with natural language, and with the maturity of predictive AI and deep neural networks, applied AI is a reality now more than ever. In this section, we show hope AI applications are built using compound AI systems in the wild. There are many sources of knowledge about applications of AI that can be found on the internet. I chose to use the Federal AI use-case inventory to extract some examples use-cases, followed by a real case of how Uber and OpenAI make use of compound AI systems to build whole AI products and map them to our adapted simplified whole product framework. 

Federal AI Use-Cases Examples

Below is the breakdown for 6 example use-cases from the inventory after we have applied the framework (use the codes to find them in the inventory). 

Note: Higher resolution of the image below can be found here.

Example 1:  TowerScout (HHS-0022-2023)

• Problem: Identifying potential sources of Legionnaires' Disease outbreaks during investigations.

• Constraints:  Accuracy, speed of detection, ability to process aerial imagery.

• Core Product: Object detection and image classification models trained to recognize cooling towers.

• Enablers:

  • Data Pipeline: System to acquire, process, and store aerial imagery.

  • Knowledge Base: Geographic data on building locations, potential water sources.

  • Tools: Image annotation tools, model training infrastructure, visualization software (GIS).

• Differentiated Product Layer:

  • Integration:  Direct integration with CDC outbreak investigation workflows and databases.

  • Unique Data: Access to CDC's epidemiological data for model training and validation.

Example 2:  USDA Cropland Data Layer (USDA-0026-2023)

• Problem:  Classifying crop types and land use for agricultural monitoring and statistics.

• Constraints: Accuracy, national coverage, consistency over time, ability to handle satellite data.

• Core Product: Machine learning algorithms (likely Random Forest) trained to classify crops from satellite imagery.

• Enablers:

  • Data Pipeline: System to acquire, process, and store multi-temporal satellite imagery.

  • Knowledge Base: Ground truth data from farm surveys, historical crop patterns, weather data.

  • Tools:  Image processing software, model training infrastructure, geospatial analysis tools.

• Differentiated Product Layer:

  • Long-Term Data:  Historical CDL data provides valuable insights into agricultural trends.

  • Public Availability:  Open access to CDL data makes it widely used by researchers and policymakers.

Example 3:  Human Resource Apprentice (OPM-0000-2023)

• Problem: Time-consuming and potentially subjective evaluation of applicant qualifications in government hiring.

• Constraints: Accuracy, fairness, ability to process applicant resumes and job descriptions, explainability.

• Core Product: AI model (NLP and potentially ranking algorithms) trained on data from previous hiring decisions.

• Enablers:

  • Data Pipeline: System to acquire and process applicant data from applications and resumes.

  • Knowledge Base: Job descriptions, qualification requirements, competency frameworks.

  • Tools:  NLP libraries, model training infrastructure, user interface for HR specialists.

• Differentiated Product Layer:

  • Bias Mitigation:  Robust testing and evaluation for fairness and adverse impact mitigation.

  • Explainability: Ability for the system to provide clear rationale for applicant rankings.

Example 4:  HaMLET (Harnessing Machine Learning to Eliminate Tuberculosis) - HHS-0023-2023 (CDC)

• Problem: Improving the accuracy and efficiency of overseas health screenings for immigrants and refugees, specifically for tuberculosis.

• Constraints:  Accuracy, speed (high throughput), ability to process chest x-rays, potential resource limitations in overseas settings.

• Core Product: Computer vision models trained to detect TB from chest x-rays.

• Enablers:

  • Data Pipeline: System for acquiring, digitizing, and storing chest x-rays.

  • Knowledge Base: Large, labeled dataset of chest x-rays with confirmed TB diagnoses.

  • Tools: Image annotation tools, model training infrastructure, potentially lightweight deployment for use on less powerful devices.

• Differentiated Product Layer:

  • Public Health Impact: Potential to significantly reduce TB transmission and improve global health outcomes.

  • Resource Efficiency: Automating screening can reduce the need for specialized personnel, making it more feasible in resource-constrained settings.

Example 5:  RelativityOne (DHS-0026-2023 - Dept. of Homeland Security)

• Problem:  Inefficient and time-consuming document review in litigation, FOIA requests, and other legal processes involving large volumes of documents.

• Constraints: Accuracy, speed, ability to handle diverse document formats, legal and ethical considerations around data privacy and access.

• Core Product:  A document review platform using machine learning techniques (continuous active learning, clustering).

• Enablers:

  • Data Pipeline: System for ingesting, processing, and indexing large volumes of documents.

  • Knowledge Base: Legal frameworks, case law, and other relevant information for model training.

  • Tools:  Text extraction and analysis tools, user interface for legal professionals to review and manage documents and results.

• Differentiated Product Layer:

  • Enhanced Efficiency:  Significantly reduces the time and resources required for document review.

  • Improved Accuracy: ML models can identify relevant documents and patterns that humans might miss.

  • Compliance and Security: Strong focus on data security and compliance with legal and ethical requirements.

Example 6:  Cybersecurity Threat Detection (HHS-0015-2023 - ASPR)

• Problem:  Effectively analyzing the massive volume of cybersecurity threat data to identify and respond to real threats.

• Constraints:  Speed, accuracy, ability to handle diverse data sources, evolving nature of cyber threats.

• Core Product:  AI and ML models trained to detect anomalies and malicious activity in network traffic and other security data.

• Enablers:

  • Data Pipeline: Real-time data ingestion from various security tools (firewalls, intrusion detection systems, etc.)

  • Knowledge Base: Databases of known threats, attack patterns, and vulnerabilities.

  • Tools:  Data visualization and analysis tools, security orchestration and automation platforms for incident response.

• Differentiated Product Layer:

  • Proactive Threat Detection:  AI models can identify emerging threats and zero-day attacks that traditional rule-based systems might miss.

  • Automated Response: AI can automate incident response actions, such as quarantining infected devices, to contain threats faster.

Companies & Products

Beyond the federal AI use-cases, let us apply the framework to products released out in the open by well-known companies and startups. We will be covering Uber, and OpenAI.

Uber’s Michael Angelo

Recently, I came across this post and this post covering Uber's journey in developing and refining their AI platform, Michelangelo, over the past 8 years. According to the posts, Michelangelo plays a critical role in powering nearly every aspect of Uber's operations, from core functions like ETA prediction and ride matching to fraud detection and customer support. Additionally, since 2023, Uber has been building various internal generative AI applications and platforms to provide a good foundation for building those applications (see this post on how to build platforms for more details). Here is a distribution of their generative AI use-cases/goals: 

With that in mind, let’s apply our adapted whole product framework to Uber’s internal AI use-case with Michaelangelo and building an AI platform. 

• Problem: Lack of a standardized and scalable system for developing, deploying, and managing ML across Uber's diverse business needs with tiering/prioritization.

• Goal: Harness the power of both traditional ML and LLMs to improve core operations (ETA, pricing), enhance user experiences (customer support, app features), and boost internal productivity.

• Constraints:

  • Scale:  Managing massive data volume and real-time prediction demands of a global user base.

  • Latency: Delivering low-latency predictions for time-sensitive applications.

  • Security & Privacy: Protecting user data, particularly PII, especially when using external LLMs.

  • Collaboration:  Supporting efficient workflows for diverse teams of data scientists, ML engineers, and application developers.

  • Adaptability:  Rapidly evolving to integrate new AI/ML technologies and adapt to the changing landscape.

  • Cost-Effectiveness: Managing the computational expenses of large-scale AI, optimizing where possible.

• Core Product: Fine-tuned / Custom self-hosted LLMs tailored for Uber’s internal use-cases.

• Enablers:

  • Data Pipeline:

    • Palette:  Feature store for managing, sharing, and accessing features across Uber.

    • Data Processing & Prep: Tools for collecting, cleaning, and transforming data for both traditional ML and LLMs.

    • Knowledge Integration:  Connecting LLMs to knowledge bases, APIs, and Uber-specific data sources for grounding and context.

  • Tools:

    • Development: Michelangelo Studio (MA Studio) for UI-based workflows; Canvas for code-driven development, version control, and CI/CD.

    • Training:  Horovod, Ray, Spark, support for TensorFlow and PyTorch; specialized tools for LLM fine-tuning and optimization.

    • Serving:  Triton Inference Server, Michelangelo's real-time prediction service (OPS).

    • Monitoring: Model Excellence Score (MES) for quality assessment, feature monitoring, SLA integration, and LLM performance tracking.

  • Gateways: Uber’s Specialized Gateways such as (GenAI, CO Inference) abstracting complexities and providing easier access to AI capabilities.

  • User Interfaces: Michelangelo Studio: Unified UI for managing ML workflows.

  • Legal & Operations:

    • Security & Compliance:  PII redaction, access controls, bias detection, and mechanisms for ensuring responsible AI usage.

    • Cost Management:  Tracking LLM usage, setting budgets, and implementing cost optimization strategies.

    • Model Versioning & Artifact Management: Ensuring reproducibility, tracking experiments, and managing model deployments.

• Differentiated Product Layer:

• Scale and Operational Efficiency:  Michelangelo and its integrated gateways are built to handle the complexities of AI/ML at Uber's global scale.

• Internal Platform Expertise: Uber's AI platform team has deep knowledge of the company's unique data, business needs, and engineering environment.

• Focus on Developer Experience: Tools like MA Studio and Canvas, combined with the abstraction layers of gateways, prioritize developer productivity and ease of use.

• Hybrid Approach:  Combining traditional ML and LLMs through a unified architecture allows Uber to address a wider range of use cases.

If you have noticed, and in the mapping we have done so far for Michael Angelo, the whole product is the platform. It’s what enables developers to build products that customers love, take their mobile application for example. I have discussed platforms as products or products of the platforms in more length in this post. Feel free to take a refresher trip if you are looking for more details on the distinction. 

OpenAI’s ChatGPT

By now you most likely have used a variant of ChatGPT, what you have not seen is what’s running under the hood to allow you to use the interface exposed and get the chat experience you get. Below is a diagram from an OpenAI talk about what the platform looks like under the hood and what it takes to run ChatGPT and expose to the world.

To get more visibility, let’s apply the adapted whole product framework to ChatGPT :

• Problem: How to providing accessible, versatile, and powerful AI assistance for a wide range of tasks and queries.

• Constraints:

  • Safety and ethical considerations

  • Scalability to handle massive user demand

  • Accuracy and reliability of outputs

  • Cost-effectiveness of compute resources

• Core Product: Large Language Models (GPT series)

• Enablers:

  • Models: GPT-3.5, GPT-4, and other specialized models

  • Context/Knowledge: Fine-tuning datasets for specific tasks and safety alignment

  • Tool-use

    • ChatGPT for general conversation and task assistance

    • DALL-E for image generation

    • Codex for code generation and understanding

  • UX:  the ChatGPT web interface + the Mobile app 

  • Ops:

    • Scalable infrastructure for model training and inference

    • Monitoring and logging systems

    • User feedback collection and analysis

• Differentiated Product Layer:

  • GPT Store: Marketplace for custom GPTs created by users and organizations

  • Strong Community and User Engagement: Rapidly growing user base for ChatGPT as well as an active developer community using OpenAI API (in a sense it’s become the standard)

  • Continuous Model Improvements: Regular updates (e.g., GPT-3 to GPT-4) and Integration capabilities with other tools and platforms

  • State-of-the-Art Performance: Leading performance in various language tasks

  • Unique Data and Feedback Loop: 

    • Massive web-scraped dataset for pre-training

    • Vast amounts of user interaction data for model improvement

  • Innovation at Application Layer:

    • GPT-4 with visual input capabilities

    • ChatGPT plugins ecosystem

    • Realistic Voice with imitation

    • Assistant API for creating AI agents

  • Strategic Partnerships: Microsoft partnership for exclusive access to GPT models increasing distribution blast radius to all Azure users. 

  • Infrastructure: Access to large-scale infrastructure and compute (partially enabled by the Microsoft partnership as well)

The (Adapted) Market Development Life Cycle 

So far we have been traveling across the lands of the adapted simplified whole product framework. Along the way, we have also covered some real examples to demonstrate how the framework is (or can be) used.  It wouldn’t be a whole product framework adaptation if we didn’t adapt it to Moore’s Market Development Life Cycle model though.

Note: higher resolution of the image below can be found here.

It all starts with a Generic (core) Product, a barebones model appealing to innovators/techies who prioritize core functionality. If you would pick an open-source LLM (maybe fine-tuned to solve a specific problem?) and just put it to the test, that would be an example of a core/generic product (the enabling technology which is at the heart of making a future whole product possible). ​​Innovators here are tinkering with the tech that you seemingly are building your product around (or that you might have built yourself). Questions they might ask here: how does it fair, do we need it, do we have better alternatives, would we require additional support (skill/knowledge?), do they (your company) have it?... you’ll neeed to make sure you have answers to those questions. 

To cross to the Early Adopters and their desire for somewhat practical solutions, your product should find a way to meet the expectations (aka the Expected Product), for the problem your customer is trying to solve, what are some of the key enablers you made sure to add to create a Minimum Viable Product (MVP)? Here you must have started to target a specific niche, and started to provide enough enablers in the product that it solves 80% of their use-case (they might be willing to help because now they SEE the value of what you are offering).  At this stage, relationships and feedback matter. 

Now it’s the moment of truth, to cross the chasm to the early majority. This stage often makes or breaks your product/value prop. You will have to navigate a tradeoff: maintain the speed and innovation that attracted early adopters while at the same time also addressing the reliability/demands to make this product Whole. Make no mistake, the likelihood of others doing the same is high at this stage, but you will need to cross here anyways. Examples of enablers at this stage: 

• An efficient pipeline for data acquisition, processing, and storage. Think of Uber's Michelangelo platform, with its specialized data management tools like Palette.

• User-friendly interfaces, efficient model training infrastructure, observability (think compound systems here, and tailor to constraints). Using our Uber’s example, think Michelangelo Studio and their AI gateway (AuthN/Z, routing, etc). 

• Knowledge Integration, connecting the AI to relevant knowledge bases (RAG maybe), well-defined APIs, and domain-specific data sources to enhance its capabilities. 

Once you do cross, know you have augmented your product just enough to make it whole, welcome to the land of the pragmatists, and congratulations, you have an augmented whole product with well-defined key-enablers that solve the customer’s problem. 

You are not done though!  Now you get a chance to tell the world why you are different, ruffle your feathers, and be ready to differentiate, welcome to the Differentiated Product layer. At this stage, you’ll need to focus on highlighting your unique value proposition and solidify your maots. Examples here could:

• Foster an active community around the product (if you have that already, you might be a winner) and encourage user contributions/feedback. Both Slack and OpenAI have cultivated vibrant communities around their products (there are different ways to do that, but that’s not the topic of this post maybe more on this later). 

• Collaborate with key partners to expand reach, access valuable resources, and enhance the product’s capabilities. For example, OpenAI's partnership with Microsoft exemplifies this, granting them access to compute and distribution, 

• Leverage unique datasets, if you have a community, you likely also have data unique to your products/services (with consent of course I hope). Develop and customize your models, and refine your core optimizatoions to create a competitive edge. Uber's Michelangelo leverages their vast ride-sharing data and internal expertise to optimize AI for their specific business needs.

As you move through the stages, you’ll notice how the product's complexity increases, natural and reflects the evolving needs and expectations of each customer segment/use-case. The visual above hopefully acts as a guide/framework to highlight the importance adapting your AI product strategy accordingly to achieve success in each phase of the lifecycle.  Failing to adapt will leave you behind, while successfully listening and continuously building/iterating can give your company and your product a boost into a temporarily blue-ocean (we will talk about that later) where you excel for what you do.

Putting it All Together: Building Whole AI Products

You MADE IT! By now, you understand what it takes to build whole AI products! Let’s quickly recap.

In this post, we went together on a journey that started from classic business principles like Maslow's hierarchy of needs to the world of compound AI systems AND how they map and transform into whole AI products. We've explored the critical components of successful AI products and applications, adapting Moore’s "Simplified Whole Product Model" along the way, and finally fitted our new framework into Moore’s infamous Model Development Lifecycle framework (again with some adaptations/opinions). Here are some take-aways from our journey:

1. It's Not Just About the Model: While LLMs and SLMs are powerful (open-source or not), they are just one ingredient in the recipe for a successful AI product. And yes open source unlocks many potential benefits (out of scope), but it does NOT mean it rivals whole products! 

2. Compound AI Systems make a good pattern/foundation for whole AI products: The true power of AI is unleashed when you combine models, data pipelines, knowledge bases, retrieval mechanisms (like RAG), agents, user interfaces, and robust infrastructure (and more) into a cohesive system that works well with the defined constraints.

3. Differentiation is key: In a rapidly evolving AI landscape, establishing a moat (see above) is essential for long-term success. Focus on building strong communities, transitioning to purpose-built applications, creating value beyond the model, and differentiating at various layers of the AI stack. Compound MOATs (read above) are the way to go!

4. Constraints Shape Your Product: Clearly define the problem you're solving and the specific constraints of your target audience. These constraints will guide your choices regarding the core product, enablers, and even the differentiators.

5. The Adapted Whole Product Framework Provides a Roadmap:  By considering each layer of the framework, the generic/core product, enablers, constraints, and differentiated product layer, you can develop a complete understanding of what constitutes a valuable and defensible AI product.

Building AI products is not a one-size-fits-all endeavor. The examples from the Fed-AI use-case inventory, Uber’s Michaelangelo, or OpenAI’s ChatGPT (some of many examples in the wild) highlight the different approaches and strategies companies/institutions are employing today to build AI products and applications.  By focusing on user needs, and continuously innovating/iterating/discovering, you can navigate the uncertainties of the AI landscape and create AI products that truly deliver on their promise.

With all that said and done, now It's Your Turn, friend:

Think about an AI product you are working on or envisioning. Use the adapted simplified whole product framework and the guiding questions posed throughout this post to analyze its strengths, weaknesses, and opportunities for differentiation. Remember, building successful AI products requires building a perspective that goes beyond just the technology itself, remember the “whole is greater than the sum of it’s parts”, so make sure how you connect the parts resonates will with your brand, mission, and strategy. 

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

The Types of Innovations

Before delving into nonconsumption, it's important to get a hold on the fundamentals of innovation and its various forms. Innovation is not one-size-fits-all, different types of innovation trigger distinct responses in the market. This section will cover how these innovations influence both supply and demand, and how they shape market dynamics. Understanding this interplay matters to grasp how innovations, such as AI, can transform industries and create new opportunities.

Supply-Side Vs. Demand-Side

When we look at innovations, we have to consider two sides: supply and demand.  Supply-side innovations act as a catalyst for enhanced production efficiency, potentially fattening profit margins. This is not merely about tweaking a few nuts and bolts, it involves engineering overhauls across various segments of the supply chain or the value stream. Those advancements can lead to increases in the quantity of supply available on the market and sometimes, even shifts in the supply curve. Picture a machine running smoother and faster, churning out higher-quality goods at a quicker rate.

Demand-side Innovations, on the other hand, look into their impact on consumer demand. They affect how consumers perceive and desire products. A great demand-side innovation would result in outward shifts in the demand curve (e.g., by creating new markets) or increases in the quantity demanded. An example would be introducing a revolutionary gadget that everyone didn't know they needed, but now can't imagine living without once they used it (sounds familiar?).

Disruptive Vs. Sustaining

Every now and then, we experience technological breakthroughs, while often exciting advancements, they don't always translate into market success or new products/services. They are fundamental discoveries or inventions that push the boundaries of what's possible. However, it's the disruptive innovations built upon these breakthroughs that truly reshape markets.  Example of breakthroughs: 

• Quantum Computing: While still in its early stages, quantum computing represents a leap in processing power, promising to solve complex problems beyond the reach of classical computers. This breakthrough has the potential to disrupt industries like pharmaceuticals, materials science, and finance, but its full-fledged applications are still years away (for now).

• Brain-Computer Interfaces (BCIs): BCIs, like Neuralink's projects, aim to connect the human brain directly to computers, potentially enabling new forms of communication and treatment for neurological disorders. While groundbreaking, widespread adoption and market disruption are still a long way off, though there are signs of it starting:

• Genetic Engineering: CRISPR-Cas9 and other gene-editing tools offer immense possibilities for treating genetic diseases, modifying crops, and even creating designer babies. However, ethical concerns, regulatory hurdles, and technical challenges mean that the full disruptive potential of genetic engineering is yet to be realized.

Building upon breakthroughs, disruptive innovations introduce new products or services that significantly alter the market landscape. These innovations often create new market opportunities by offering cheaper, more convenient, or more accessible alternatives to existing products (we will discuss nonconsumption in the next section).  Disruptive innovations can shift the demand curve to the right as they attract new customers who were previously underserved or not served at all by existing products. 

It’s important to note that disruptive innovations are “disruptive” because of their impact on aggregate demand for an underserved need (either latent or blatant). Here are some examples of disruptive innovations and the breakthroughs they were built upon.

• Smartphones (Disrupting Mobile Phones and PCs): Smartphones, like the iPhone, are built upon breakthroughs in touchscreen technology, mobile processors, and wireless communication. They disrupted the traditional mobile phone market and even impacted the personal computer industry by providing a more convenient and versatile alternative.

• E-commerce (Disrupting Retail): Amazon and other e-commerce platforms leveraged breakthroughs in internet technology, logistics, and online payment systems. They disrupted brick-and-mortar retail by offering a wider selection, competitive prices, and the convenience of shopping from home.

• Ride-sharing (Disrupting Taxis): Companies like Uber and Lyft utilized breakthroughs in GPS technology, mobile apps, and real-time data processing. They disrupted the taxi industry by offering on-demand rides, transparent pricing, and a more user-friendly experience.

Most technological breakthroughs undergo several epochs/phases before they are finally usable enough (commercially viable) to cause disruption in markets (impacting demand curves). Not all breakthroughs lead to commercially viable products or services that can disrupt markets.

Sustaining innovations, on the other hand, are incremental improvements that maintain and enhance the competitiveness of existing products or services within established markets. Unlike disruptive innovations that create new markets or alter existing ones (by shifting demand curves), sustaining innovations focus on enhancing the features, performance, or efficiency of current offerings. These innovations are important for businesses to keep pace with market demands and ensure customer satisfaction and continued market relevance.

If we put it all in a chart, it would start with an incubation phase, where breakthroughs can emerge but remain “researchy”, lacking application and viability. As these breakthroughs evolve, a usable product could be developed, getting us to the Adoption phase. This stage typically shows rapid growth due to Disruptive Innovation, where the application of breakthrough technology reshapes markets and unlocks new possibilities by fulfilling previously unmet needs. With time, the innovation enters the Sustaining phase, where growth stabilizes and is driven by incremental enhancements to the existing product. However, failure to maintain or sustain can lead to a decline, underscoring the importance of incremental improvements to remain competitive (even starting a new “S-curve” with another disruptive application).

This “S-curve of Innovation” shows the evolution from breakthroughs to commercially viable products that can be disruptive but must be sustained. 

To go a step further, here is a diagram that shows how innovations interact with supply and demand dynamics, leading to different types of market impacts.

Now that we understand the types of innovation, let’s explore in more detail the main drivers of disruptive innovations, especially nonconsumption.

Nonconsumption

It does not matter if there is a breakthrough if no one can make use of it. As described in the previous section, breakthroughs can only become disruptive if they can be consumed/used, which is not always the case. Welcome nonconsumption!

Nonconsumption typically describes a scenario where potential customers are underserved by the current market offerings. These customers either cannot afford the existing solutions, do not have access to them, or find that the solutions do not perfectly meet their needs. Nonconsumption can be an opportunity for businesses to innovate and create products or services that address these unmet latent AND blatant needs, thus turning nonconsumers into consumers. It is also a risk to established companies (aka incumbents) that are too focused on serving their existing customer base but forgo opportunities outside of their comforting bubble, leaving more breathing room for startups (the new entrants). 

Nonconsumers usually fall into one of the two categories: 

• Underserved nonconsumers: those who do not have access to a product or service due to various barriers such as cost, lack of information, lack of infrastructure, etc.  For example, due to a lack of infrastructure, people in rural areas without access to high-speed internet are nonconsumers of streaming services/social networks and broader knowledge in general.

• Overserved nonconsumers: those who do not use a product or service because it exceeds their needs or is too complex. For example, someone who only needs a basic phone for calls and texts but is offered a high-end smartphone with features they won't use is an overserved customer:

Clayton Christensen introduced me to the concept through his book, “The Innovator’s Solution.” In it, he discusses how new-market disruptions can target non-consumption by creating products that enable a larger population of people, who previously lacked the money or skill, to begin using a product and doing the job for themselves.  Also see Targeting nonconsumption: The most viable path to growth - Christensen Institute 

Note: brand plays a role here as well. Sometimes, nonconsumption can be by design via positioning/targeting a specific slice of consumers. This usually happens via brand design, that said, even then companies with great brands are at risk of disruption. We probably will talk about this in more detail in upcoming posts. 

Subscribe now

AI Winter Nonconsumption

In part I, we journeyed through the peaks and valleys of AI history and covered AI winters. However, we stopped short of exploring the reasons behind them. Winter came, multiple times actually, but WHY? Let’s go behind that wall today!

Take a quick look at your AI scroll, and cast your mind back to the mid-80s, when AI had one of its highest peaks in history. Back then, expert systems were seeing some signs of commercial viability as companies such as IBM, FMC, Toyota, American Express, and others started to find use cases for it. This led to renewed excitement and hope up until 1987, when expert systems started to show limitations and struggled to handle novel information and situations that fell outside its pre-programmed knowledge base, i.e., expert systems underserved consumers, and the tech was way behind in serving the needs properly, as a result, was non-consumable.

ℹ️ The spike in the graph does not reflect the demand curve but rather content generated, which is a rather poor proxy to adoption, but it’s all that we have in that timeline beyond the narratives described in the previous post.

Parallel to expert systems, many theories were getting close to actionable forms, especially for neural networks with Hopfield nets,  Boldtzman machines, perceptrons, and backprop networks, see A Very Short History of Artificial Neural Networks | by James V Stone for more details. So at this point, the mid 80s, most of AI theories were formulated. Since the 50s, there have been multiple knowledge breakthroughs, yet no disruption on the horizon (remmember disruption hinges of demand). One could say that expert systems ALMOST did it, but it was a short dream back then. 

So really, why?

With the benefit of hindsight, the field of artificial intelligence stems from the research originally done on Hopfield nets, Boltzmann machines, the backprop algorithm, and reinforcement learning. However, the evolution of backprop networks into deep learning networks had to wait for three related developments: 1) much faster computers, 2) massively bigger training data sets, and, 3) incremental improvements in learning algorithms ~ from A Very Short History of Artificial Neural Networks | by James V Stone 

One way I would sum it up is AI winters were a result of a negative consumption gap where expectations of what’s possible from AI exceeded what was being delivered, due to many factors such as much faster computers, massively bigger training data sets, and incremental improvements in learning algorithms. The figure below illustrates the consumption gap. 

These are but three factors, let’s dig deeper.

Not Accessible by Design 

When Ed Feigenbaum's expert systems came to life when the IBM 701 was connected to the early ARPANET, the reach was very limited. Only a select few researchers had the privilege of contributing and accessing shared knowledge (which was a massive boost by the way compared to how computing was done). This rhymes well with Christensen's wording of nonconsumption: a potentially transformative technology was out of reach for the vast majority due to restrictions and a lack of infrastructure (in that case, access to the server and the knowledge to evolve and do more research). 

In addition to the lack of accessibility, there were multiple “Tech Have-Nots” that the entire field required to grow to something more usable (which is what’s needed for demand to increase). 

The "Tech Have-Nots"

The 80s saw a surge of interest in artificial intelligence, but progress was subpar for many reasons:

• Limited Internet Access: While ARPANET existed, it was primarily restricted to academic and research institutions. The concept of a globally interconnected "World Wide Web" only started to sprout in the late 80s. This meant that collaboration and knowledge sharing were pretty much confined to small, isolated groups.

• Data Scarcity: High-quality datasets were hard to come by, making it difficult to train effective AI models. The concept of "big data" was far off, and the ability to collect and store vast amounts of information was limited. Most 

• Hardware Constraints: computers of the 1980s had limited processing power and memory (speaking from the future, easy to judge). This made it difficult to run complex AI algorithms, hindering research and development.

• The "Haves" and "Have-Nots": Even with access to the limited resources available, significant disparities existed. Well-funded institutions and researchers, like Ed Feigenbaum with his DARPA-backed IBM 701, had an advantage over others who lacked access to such cutting-edge technology.

To put those limitations in perspective, let's have a look at some data and explore NetTalk, one of the first manifestations of Neural networks that was designed to learn how to pronounce written English text.

In the 80s, I’d say NETtalk, with its 18,629 adjustable weights and 1000 data points, was the marvel. It couldn’t do a whole lot though with that much data and those weights compared to a GPT-3, which is when we first realized how powerful neural networks can be when trained on the right amount of data using enough hardware for the outcomes to make sense.

Compared to NetTalk, GPT-3, is a general-purpose model with 175 billion parameters and a trillion data points, an exponential growth in model size over the decades. This is a result of leaps in computational power which was not an overnight quest. In addition to knowledge boosts (e.g., with Transformers), there were many incremental advancements in algorithms, the internet, hardware, and data availability. GPT-3, as the first truly usable form of a language model based on deep neural networks (we had GPT-2, but wouldn’t say that was viable), showcases the difference between a theoretical breakthrough in science and its readiness and commercial variability (i.e., the time, the research, the hardware, the effort it takes to beat nonconsumption).

The AI Spring

Fast forward to the present. Where are we now? Have we solved nonconsumption? Let’s see.

We now have the internet! While not everyone has access to it, compared to the ARPANET in the 80s, about 66% of the world population now (2024) does. It’s still a major issue of nonconsumption on its own, but we made progress! Using the internet, anyone can share, collaborate, and access data/knowledge.

This widespread of the mother internet has led to an explosion of user-generated data, via social media, Wikipedia, articles, and papers, creating large datasets known as "big data". This abundance of data has become the main resource for training today’s AI and large language models, enabling them to learn and improve from diverse and extensive information. Contrast that with the limited availability of data in the 1980s, which was a choke point for further development and advancement of AI and neural networks (NetTalk had a 1k dataset, GPTs datasets are in billions and going trillion). 

We have also made progress on the computing front. The development of parallel processing with GPUs and chips specifically designed for AI workloads (e.g., TPUs) has been a game-changer. This leap in computing made it possible to train large and complex deep-learning models on big datasets, which was simply not feasible in the 1980s due to the limitations of hardware at the time.

Research continued, never stopping, which also led to breakthroughs in AI techniques and algorithms. These algorithms, particularly for deep learning and neural networks, have enabled certain applications of AI to automate many of the tasks that only humans were capable of. Techniques such as convolutional neural networks (CNNs) and transformer models have been crucial in getting us to where we are today. In the 1980s, the algorithms were just not as advanced or effective.

In addition to computing, data, and widespread of knowledge, we saw the rise of open-source software which is helping democratize access to AI and machine learning software. Tools and libraries such as TensorFlow, PyTorch, and others are accessible to anyone with an internet connection, allowing for a global participation in AI development. This openness not only accelerates innovation but also reduces the entry barriers that once existed due to proprietary systems. Open source also gave birth to new business models focused on building value not gating IP. 

Is it Summer yet then? 

Not quite! Despite the improvements, the supply side of compute for AI is still highly inaccessible. Training state-of-the-art large language models requires massive compute resources costing millions of dollars, primarily for high-end GPUs and cloud resources. The costs have been increasing exponentially as models get larger. Only well-resourced tech giants and a few research institutions can currently afford to train the largest LLMs. It actually fits a power law quite nicely, the major players having enough capital and access to data through their current operating business,  so you will find that a minority of companies have access to the majority of compute/data (more about the AI market in a previous post).

Additionally, current economics and global tensions around trade policies have led to what is often referred to as a "chip war", impacting the production and distribution of advanced semiconductor chips. This situation introduces disparities in computational access. Countries and companies that can produce or secure these chips have a competitive edge in developing and deploying AI technologies. So yes internet solved some of it, politics always gets in the way 🙂

What about data? According to scaling and chinchilla laws, model performance in language models scales as a power law with both model size and training data, but this scaling has diminishing returns, there exists a minimum error that cannot be overcome by further scaling. That said, it’s not unlikely that we will figure out how to overcome this in the near future. 

For a fixed compute budget, an optimal balance exists between model size and data size, as shown by DeepMind's Chinchilla laws. Current models like GPT-4 are likely undertrained relative to their size and could benefit significantly from more training data (quality data in fact). Future progress in language models will depend on scaling data and model size together, constrained by the availability of high-quality data.

Beyond data, and despite the progress, AI still faces technical limitations such as the lack of common sense reasoning, vulnerability to adversarial attacks, and difficulties in generalizing from training data to new, unseen situations, not to mention hallucinations. Finally, current models can be more “creative” than we want them to sometimes, which makes them hard for tasks that require reproducibility or accuracy. So we are not quite there yet. 

So is it Disruptive Innovation or AI Winter 2.0?

Generative AI seed funding drops 76% as investors take wait-and-see approach. On one hand, $50 billion has been invested in AI, yet only $3 billion in revenue has been generated. One could argue that this is upfront capital deployed to establish infrastructure and capacity for future earnings (an investment), however, today’s GPUs lose value quickly, especially as newer, more efficient models are continually released, so it’s hard to tell!

This reality is causing investors to pause and reassess their strategies. Past AI hype cycles, like expert systems in the 80s and Japan's 5th generation program early 90s, remind us that overenthusiasm can sometimes lead to disillusionment (see part I of this series).

That said, dismissing AI's potential for disruption would be a mistake. Unlike previous iterations, today's AI, despite its flaws and hallucinations, offers some value (might not be crystal clear, but the productivity gains are not to be reckoned with). This could potentially slow down an "AI winter" as businesses find ways to integrate AI into their workflows, subject to the value perceived. 

History repeats itself in a way, we have seen a similar pattern before with the dot-com bubble, where there was a surge of investment in companies, often with inflated valuations and unrealistic expectations, which eventually led to a market correction, with many startup companies failing or being acquired. We're likely to see the same, where the weaker players are likely to struggle and sway away. As discussed in a previous post, existing incumbents and larger players have an existing business model that does NOT revolve around JUST AI, it’s ads, e-commerce, software, consulting, and so on. This gives those bigger players the runway they need to fight the long fight, make bets, and potentially acquire some winners from the new entrant's pool, power law again! 

So to answer the original question, I don’t think we will see a “bad” AI winter soon (we might see an autumn but not a winter), at the same time, there is still the debate on whether the current version of AI, though way better than the 80s, qualifies as disruptive innovation on it’s own. I clearly see the potential for AI, the technology, as a disruptive force, but I lean more toward calling the current versions of its application sustaining innovations, especially since we are seeing how it can accelerate and optimize revenue streams of existing incumbent companies as they incorporate it into their pre-existing fly-wheels. 

That said, applications that make use of AI to target areas of nonconsumption with more affordable and accessible means to solve problems can unlock new markets, and become disruptive. Here are a couple of examples: 

• AI as an enabler for personalized education for underserved students in areas where education is infeasible and not accessible.

• AI-powered diagnostics expanding healthcare access in developing countries where healthcare is infeasible and not accessible.

The question becomes whether the tech powering today’s AI is enough to give birth to those disruptive innovations (the data, the compute...). I’d say not yet, what say you?

That’s it for this series! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

That said, for someone who wants to understand the market or even start an AI business, it’s important to build awareness of the landscape, how those companies position themselves in the market, what the main products they sell, how they make money, and finally, how does AI contribute their business livelihood. That’s exactly what this post is about. 

In this post, we will explore the AI Market Dynamics Quadrant. We will focus on how open or closed the offerings from those companies are when it comes to generative AI models (open vs. closed) and how coupled the models/GenAI in general are to their business model (direct vs. indirect).

Subscribe now

The Glossary

We will be segmenting the AI market into four key categories based on two dimensions: the openness of the AI model (open vs. closed) and the integration of the AI model into the business model (direct vs. indirect sell).

• Open Models: AI models that are accessible to the public, often leveraging permissible licenses to share the weights, code, architecture, etc, or community licenses (with conditions), to encourage usage and contribution. They also usually rank higher in the foundational model transparency index. 

• Closed Models: AI models that are proprietary, with controlled access to ensure exclusivity and competitive advantage (the model here is usually the MOAT).

• Direct Business Models: Companies that monetize AI models directly through subscriptions, licenses, or selling the technology itself. The model is the product!

• Indirect Business Models: Companies that use AI models to enhance and complement their existing products and services, rather than selling the AI models directly. The model is the proxy!

These are all words you’ll need to follow along and understand the quadrant. That said, let’s try to get more insights into how the companies in the quadrant make money today before starting to analyze the dynamics and the landscape through the quadrant.

The Players: Incumbents & New Entrants

In the landscape quadrant, I didn’t include all the companies in the world. I sampled based on activity and heat, i.e., these are the top players potentially with the largest influence and diffusion power on the AI scene.  

Let’s start with the incumbents, the well-established, those who know what they are doing, the seniors. 

Sample Incumbents

Below is a sample of incumbents. 

IBM

Software is IBM's largest revenue driver. It includes Watsonx (AI and data platform), Red Hat (Hybrid Cloud, RHEL, OpenShift Platform, and Ansible), security and zero-trust, automation, and transaction processing (which powers IBM Z). 

In second place is IBM Consulting, which is a key driver for digital transformation for many organizations. It also partners with other large enterprises AWS, Microsoft, and SAP, and utilizes Watsonx for procurement solutions. 

In third place is IBM Infrastructure, which ensures reliable, secure solutions for critical workloads across public, private, hybrid clouds, and the edge. Its strengths include transaction processing on IBM-Z and AI-driven transformation. 

Finally, IBM financing which supports the acquisition of IBM's technology solutions and services by providing various financial options, including loans, leases, and customized payment plans. 

Generally, generative AI models are not the main revenue stream here. IBM makes money by providing comprehensive tech solutions and transformation services and leveraging open-source models to drive consulting and software sales.

Microsoft

Microsoft generates revenue through a diverse range of products and services, primarily categorized into three main segments: Intelligent Cloud, Productivity and Business Processes, Personal Computing, and as of very recently, AI-enabled services. Here is a detailed breakdown of these revenue streams, followed by a simplified revenue distribution based on the available data: 

• Intelligent Cloud segment is Microsoft's largest revenue driver, which includes Azure (their cloud), SQL Server, Windows Server, GitHub, and Enterprise Services. Azure, in particular, is a major component, offering a wide range of cloud services and holding a 24% market share, second only to AWS. This segment accounts for 38% of Microsoft's total revenue. 

• Personal Computing is the second largest revenue driver. It encompasses Windows (operating system), Surface (devices), Xbox (gaming consoles and services), and search advertising (Bing). This segment aims to enhance the user experience across various devices and platforms. In 2023, this segment represented 25% of the total revenue. 

• Productivity and Business Process is the third largest revenue driver. This includes Office Commercial products and cloud services (Office 365), Office Consumer products and cloud services (Microsoft 365), LinkedIn, and Dynamics 365. This segment focuses on enhancing corporate productivity, communication, and information services. In 2023, this segment represented 23% of the total revenue

• AI-enabled services, particularly through the Copilot platform, have become a major component of Microsoft's revenue streams. Copilot, integrated into Microsoft 365 and other products, leverages AI to enhance productivity and user experience. In 2023, AI-enabled services, including Copilot, comprised of 6% of Microsoft's revenue.

Generally speaking, Microsoft does not sell AI directly, it infuses it into their products/services like Office 365, search and Azure. It has also a pretty even and diverse set of revenue streams. Worth noting that the slightest capture in search from Google would yield considerable margins.

Google

Google's revenue model is more or less advertising driven, i.e., it remains by a large margin the primary source of Google’s income. However, it has diverse revenue streams. In addition to search and ads, it has Google Cloud, subscriptions, platforms, and devices, as well as its other bets segment (which gets rotated/killed often). 

The diversity of Google’s revenue stream ensures it’s presence in both consumer and enterprise markets. Let’s do a brief break-down to understand a bit more (data from Alphabet’s q4 earnings report): 

• Advertising is Google's largest revenue driver, accounting for the majority of its income. This includes ads on Google Search, YouTube, and Google Network properties. In 2023, Google Search, YouTube, and Google network Ads contributed ~77% of Google’s total revenue (lion’s share). 

• Google Cloud comes in second place, which includes Google Cloud Platform (GCP) and Google Workspace. Google Cloud provides infrastructure, data analytics, and collaboration tools for enterprise customers. In 2023, Google Cloud contributed about 11% to total revenue, driven by the increasing demand for cloud services and AI tools.

• Subscriptions, Platforms, and Devices are the third largest revenue driver. This includes revenue from YouTube Premium, YouTube TV, Google Play, and hardware sales such as Pixel smartphones and Google Nest devices. In 2023, this segment contributed 10.1% of the total revenue. 

• Other bets: Finally, Google's Other Bets segment includes various experimental and emerging businesses such as Waymo (self-driving cars), Verily (life sciences), and other innovative projects. Although this segment is smaller, it represented 0.7% of the total revenue.

Generally speaking, Google makes money through advertising, cloud services, and enhancing user retention across its suite of products. Google uses (or will use) AI models to improve its services, be it search, its Gsuite or its cloud offerings.

Compared to other incumbents, Google’s revenue stream might be diversified in numbers but not in weight. Google heavily relies on its Ads business, any threat to this stream canbe an existential threat to Alphabet/Google. 

Amazon

Amazon's revenue model is characterized by its strong focus on e-commerce, cloud computing, and advertising. The company leverages its extensive product selection, efficient delivery services, and robust technological infrastructure to drive growth across various segments. AWS, in particular, plays a crucial role in Amazon's profitability, while advertising and subscription services continue to expand, contributing to the company's overall financial success.

• Online stores are Amazon's largest revenue driver, which includes first-party sales of products directly to consumers. This segment accounts for approximately 40% of Amazon's total revenue. The online stores segment benefits from Amazon's extensive product selection, competitive pricing, and efficient delivery services

• In second place are Amazon's third-party seller services, which include commissions, shipping fees, and other services provided to third-party sellers on Amazon's marketplace. This segment accounts for about 24% of the total revenue. The third-party marketplace allows Amazon to expand its product offerings without directly managing inventory, enhancing its overall product selection and customer experience

• AWS is Amazon's cloud computing division and the third-largest revenue driver. It provides a variety of services, such as computing power, storage, and databases, to businesses and developers. In 2023, AWS accounts for approximately 16% of Amazon's total revenue. AWS is also the most profitable segment, contributing significantly to Amazon's operating income

• Amazon's advertising business has experienced rapid growth, offering targeted advertising solutions to brands and sellers. This segment makes up about 8.2% of the total revenue. Amazon's advertising services include sponsored ads, display ads, and video ads, which help brands reach a wide audience on Amazon's platform

• Amazon's subscription services, including Amazon Prime, Kindle Unlimited, Amazon Music, and Audible, are another revenue stream. In 2023, subscription services represented about 7.1% of the total revenue. Amazon Prime, in particular, fosters customer loyalty and drives additional sales through its ecosystem of benefits.

• Amazon's physical stores, including Whole Foods Market and Amazon Go, accounted for approximately 3.5% of the total revenue in 2023. These stores complement Amazon's online presence and provide additional channels for customer engagement

• Finally, Amazon's "Other" segment, which includes various smaller revenue streams such as co-branded credit card agreements and certain licensing and distribution of video content, makes up about 0.8% of the total revenue

Amazon's diverse revenue streams, including online stores, AWS, subscription services, and advertising, allow it to withstand losses in other areas due to its well-rounded portfolio, making it less dependent on AI. That said, in the AI world, AWS exposes a family of models through AWS bedrock led by Anthropic’s Claude, to which it has committed up to $4 billion in investments, and provides infrastructure through inferentia and trainium for inference and training, amongst the normal storage/compute/networking and the wide selection of instances. We will talk more about Anthropic and Claude in a bit. 

Apple

Apple's business model is centered around product design and vertical integration. Apple’s focus on user experience and design, helped it build a brand with high equity, resulting in a very high willingness to pay from its customer base (especially as you own more Apple devices the value offered becomes higher). The company's revenue streams are diversified, with the iPhone being a major driver, supported by services like Apple Music and the App Store, as well as other products like the iPad, Mac, and Apple Watch. Here is a quick breakdown (sources: How Apple Makes Money and Apple Statistics):

• Apple's largest revenue driver is the iPhone, accounting for around 52% of the company's total revenue.

• Apple's Services are second, representing approximately 22% of total sales. Services include the App Store, Apple Music, iCloud, Apple TV+, Apple Arcade, and various other subscription offerings.

• The third largest revenue contributor is Wearables, Home, and Accessories (10.4% of total revenue). This segment encompasses products like AirPods, Apple Watch, HomePod, Apple TV, Beats headphones, and other accessories.

• Next is the Mac lineup of computers, including MacBook Air, MacBook Pro, iMac, and others making up around 7.7% of Apple's total sales.

• Finally, the iPad tablet lineup, consisting of models like iPad Pro, iPad Air, and iPad Mini, contributing about 7.4% of Apple's revenue.

Apple has been lurking in the shadows when it comes to  AI compared to other incumbents, but the company has made significant investments in this area. Since 2017, Apple had 32 AI startup acquisitions under it’s belt, focusing on technologies like self-driving cars, voice design, music generation, and image recognition. These investments have led to the integration of AI features across Apple's product lineup, such as personal voice, live voicemail transcription, and advanced health monitoring on the latest iPhones and Apple Watches. Apple should also not be take lightly when it comes to the potential for personalization with AI, after all they ace the wearables! 

Apple's AI strategy involves a hybrid approach, leveraging both in-house and external resources. The company is investing in generative AI and plans to "break new ground" in this area, as stated by CEO Tim Cook. Apple is also exploring partnerships with AI companies like OpenAI and Google to integrate their large language models into future products and services.

In summary, Apple's revenue model is driven by its diverse product (mostly physical) lineup, with the iPhone and Services being the primary contributors. 

Meta

Meta's revenue model is dominated by advertising, with growing contributions from Reality Labs, Payments, and Other Fees. Let’s break it down: 

• Advertising is Meta's largest revenue driver, generating approximately 98% of the company's total revenue. This includes ad placements across its family of apps, such as Facebook, Instagram, Messenger, and WhatsApp. Meta's advertising revenue success is driven by its sophisticated ad targeting, large user base, and continuous enhancements in ad formats and measurement tools.

• Reality Labs, Meta's division focused on virtual and augmented reality, generated around 2% of total sales. This segment includes hardware like Oculus VR headsets and software related to the metaverse. Despite still being a smaller revenue contributor, Reality Labs is a significant part of Meta's long-term strategy to lead in the development of the metaverse and immersive experiences.

• Meta's Payments and Other Fees segment makes up less than 1% of the company's total revenue. This includes fees from Facebook Pay, peer-to-peer payments, and other financial services. Although currently a minor part of the revenue mix, Meta is investing in expanding its financial services offerings to create new revenue streams.

Meta's revenue model is dominated by advertising, with growing contributions from Reality Labs, Payments, and Other Fees. That said, Meta has been an active contributor (actually a solid contributor, e.g., PyTorch, Llama, etc.) to the AI scene. 

AI is not the direct sell, nor it is the main revenue stream, but making one of Meta’s dependencies a commodity (the foundation model), helps it scavenge talent from the entire tech landscape to make its models better. In doing so, it helps it make material improvements for its ad business via better content recommendation, enhanced ad targeting,  automatic content moderation, translation, and copy creation. As ad performance improves, Meta attracts more advertisers and commands higher ad rates, ultimately boosting their revenue. Not to mention that Meta’s “open” approach enhances their brand reputation and attracts top talent. 

Summary for Incumbents

Technology incumbents like IBM, Microsoft, Google, Amazon, Meta, and Apple have long-established revenue streams from their core businesses, such as cloud computing, operating systems, search engines, e-commerce, and social media. They have existing means to make money (not AI). AI can be important in these industries mainly as an enabler for their core business and revenue streams but is not the main source of income. 

There are exceptions, though. For example, Google is in the Ad business. Ads happen when you are able to connect the consumer with the advertiser. Google does this by offering “free” search (after selling the soul) and then surfacing Ads customized to the consumer. Facebook collects data based on social browsing and customizes ads (via cookies, again selling the soul) to improve click-through rates and bring them to the consumer, and so on. 

In that case, where there is no actual product/software/service being sold but the Ad (as the main source of revenue), AI can be detrimental. If there is a better way to connect the consumer with the advertiser, it becomes a threat, hence why Google/Meta are strong players in the AI and foundational models game besides the AI new entrants (we talk more about those next) whose main mission in the world is to thrive of generative AI. 

Below is an aggregation of incumbents' revenue stream distribution charts. If you look closely, you will see that there are a few with a balanced staircase distribution where all the streams are contributing their fair share to the aggregate revenue (e.g., IBM, Microsoft, Amazon, Apple). On the other hand, there are those that rely heavily on one stream (e.g., advertisement) to varying degrees (Meta wins in its sole dependence on Ads, followed by Google!). 

Also, some have their streams under one category (e.g., Apple is focused on devices, wearables, and physical products), while others horizontally surf different verticals (cloud, e-commerce, devices, etc). 

Sample New Entrants

Below we will cover new entrants in the tech and AI scene. They don’t necessarily make money (profit) at this point, but we will quickly cover their business model to understand better how they fit in the AI Market Dynamics quadrant.

Mistral

Mistral AI's revenue model is driven by its AI models and services, enterprise solutions, and open-source offerings. The company's investments in AI research and its partnerships with leading tech firms have positioned it as a key player in the AI industry, with a strong focus on performance, efficiency, and customization. Let’s do a quick breakdown: 

• AI models and services are Mistral AI's largest revenue driver. This includes their flagship models like Mistral Large and Mixtral 8x22B, available through APIs and platforms such as Amazon Bedrock and Google Cloud's Vertex AI (also on Azure as of recently). These models serve various applications, including text summarization, question answering, and code completion. Mistral’s key play here is high performance and cost-efficiency for their models. 

• In second place are Mistral AI's enterprise solutions, which provide tailored AI capabilities to businesses across various industries. These solutions include custom AI development and fine-tuning services, allowing enterprises to adapt Mistral's models to their specific needs. Mistral AI collaborates with European integrators and clients to co-build solutions, ensuring that the models meet industry-specific requirements and compliance standards

Generally, Mistral focuses on providing open AI models through APIs and partnerships. They generate revenue through collaborations and by offering services that integrate their open AI models into other businesses.

OpenAI

OpenAI, which began as a research lab in 2015 and transformed into a for-profit organization in 2019, generates revenue primarily through API consumption fees for its foundational models (the GPTs), used by startups, and even large enterprises to create AI-infused applications (mostly also for prototyping). But also, through premium/plus subscriptions which give access to more features like the create your own GPTs, the GPT store, better models (GPT4/4o/turbo/...), and more input/output tokens to use those models. 

One of OpenAI’s largest partners/distributors is Microsoft, which is also a minority owner. In that case, Microsoft earns revenue through hosting OpenAI's APIs on Azure and distributing them to users, as well as through the sale of applications built on Azure. The partnership between OpenAI and Microsoft has led to new products like GitHub Copilot, and OpenAI's generative models may be integrated into Microsoft's offerings like Bing, Office 365, and now CoPilot + PC (similar to rewind.ai but eats more data, stay put and take care of your data 🙂).  

On the other hand, Microsoft Azure serves as the foundation for OpenAI's AI infrastructure, enabling the pre-training of models, and OpenAI's APIs are distributed through Azure, allowing users to access them directly or through the Azure Enterprise platform (a wider variety of users). 

OpenAI also has a somewhat spaghetti corporate structure. We will not discuss it in this post, but I left the diagram below for your own leisure. 

Control Alt delete: Exploring OpenAI's corporate structure, after Sam Altman’s shock dismissal

In general, OpenAI uses a closed model, offering products like ChatGPT through subscriptions, APIs, and licenses. The main monetization is achieved through direct sales to personal and enterprise customers. I.e., the model is the product! 

Anthropic

Like OpenAI, Anthropic's revenue model is driven by its mostly AI services, enterprise solutions, and custom AI development. The company's investments in AI research and its partnerships with leading tech firms have positioned it as a key player in the AI scene. Their AI game is around safety and ethical AI deployment. 

Here is a quick breakdown: 

• AI services are Anthropic's largest revenue driver. This includes the Claude family of models, such as Claude 3 Opus, Sonnet, and Haiku, which offer AI capabilities for tasks like reasoning, content creation, and coding. These models are available through APIs and platforms like Amazon Bedrock and Google Cloud's Vertex AI. 

• In second place are Anthropic's enterprise solutions, which provide tailored AI capabilities to businesses across various industries. The Team plan, for example, offers access to all three of Anthropic's latest Claude models with expanded usage limits, admin tools, and billing management. This plan is designed to support multi-step conversations and process lengthy documents targeting sectors such as technology, financial services, legal services, and healthcare. The Team plan costs $30 per user per month. 

• The third-largest revenue contributor is custom AI development, where Anthropic collaborates with clients to develop bespoke AI solutions (individual clients design custom AI solutions). This includes dedicated capacity options for high-throughput needs, ensuring that clients can access the necessary resources to support their extensive AI applications (I.e., the infrastructure required to build customized AI solutions). 

To add, similar to OpenAI and Microsoft, Anthropic and AWS are strategically collaborating with AWS becoming Anthropic’s primary cloud provider. Earlier in 2024, Anthropic made a $4 billion investment in Anthropic, securing minority ownership in the company. Anthropic is set to use AWS’s infrastructure (including AWS’s Trinium and Inferntia chips) to build, train, and serve its foundation models. This gives AWS early access to Anthropic’s models (with options to customize/finetune) and exposes Anthropic to AWS's massive customer base. The models are available through Amazon Bedrock.  The play here is mainly centering around responsible AI as a differentiating tenet. 

Generally, Anthropic provides closed AI models, focusing on products like Claude and recycles that as subscriptions for directly interacting with the models, through API consumption, or licenses for partners to use their models, which benefits both the partner and Anthropic. The main monetization is achieved through direct sales to personal and enterprise customers. I.e., the model is the product! 

Summary for New Entrants

In a sense, the business models for Mistral, OpenAI, and Anthropic are quite similar. Subscriptions, APIs, and partnerships with one of the incumbents to give them access to people and infrastructure. Each tries to go for a unique play. Anthropic goes for ethical and responsible AI, Mistral goes for lean, efficient, and open models, and finally, OpenAI goes for AGI and best-in-breed quality. 

That said, they all have one thing in common. Unlike the Incumbents who have various revenue streams making them money and can potentially loss-lead for a long while, this group operates on the basis that the model IS the main product and the revenue stream. Take the model away, poof, they are gone, finito! 

Analysis & Categorization 

No, let’s summarize all the above and explain the quadrant. There will be four categories, namely: 

• Open & Direct: Companies that pushed their foundational models to the open, and still rely on the model being the product. 

• Open & Indirect: Companies that pushed their foundational models to the open, but don’t rely on the model being the product, rather the proxy.  

• Closed & Direct: Companies that DID NOT push their foundational models to the open, and still rely on the model being the product. 

• Closed & Direct: Companies that DID NOT push their foundational models to the open AND don’t rely on the model being the product, rather the proxy.  

Open and Direct

Mistral is located here, among other new players (which are out of scope for this post). Mistral focuses on providing open AI models directly through APIs and partnerships, as described in the new entrants overview. Whether that will sustain a living or not is unclear for now, but they are definitely moving more towards offering their premium models (time leaps ahead) with a higher price tag, and they won’t be planning to open-source those models. 

Open and Indirect

IBM and Meta fall here. 

IBM utilizes open AI models (while wearing its open-by-default Red Hat to provide accessible training/finetuning via InstructLab) within its broader suite of software and consulting services. It enhances its offerings by integrating open-source models and providing comprehensive tech solutions without directly selling the AI models or at least not as their main revenue stream. IBM pushed their Granite models weights/arch/.. under the Apache license (a very permissive license). IBM also ranks highly in  the Model Transparency Index (as of May 2024). 

Similarly, Meta, which relies mainly on Ads to make a living, is releasing its models under a community license. In doing so, it gets contributions from the entire world to improve its internal services (for ad targeting), which helps it attract more advertisers and command higher ad rates, ultimately boosting its revenue. Not to mention that Meta’s “open” approach enhances its brand reputation and attracts top talent.

You might have noticed that I didn’t mention the eye of Sauron (Jensen Huang / Nvidia). I didn’t feel I needed to, they are right in the middle of it (see the quadrant), and no matter what happens, their chip rules them all (for now)!

”𝘍𝘰𝘳 𝘦𝘷𝘦𝘳𝘺 $1 𝘴𝘱𝘦𝘯𝘵 𝘰𝘯 𝘕𝘝𝘐𝘋𝘐𝘈 𝘈𝘐 𝘪𝘯𝘧𝘳𝘢𝘴𝘵𝘳𝘶𝘤𝘵𝘶𝘳𝘦, 𝘤𝘭𝘰𝘶𝘥 𝘱𝘳𝘰𝘷𝘪𝘥𝘦𝘳𝘴 𝘩𝘢𝘷𝘦 𝘢𝘯 𝘰𝘱𝘱𝘰𝘳𝘵𝘶𝘯𝘪𝘵𝘺 𝘵𝘰 𝘦𝘢𝘳𝘯 $5 𝘪𝘯 𝘎𝘗𝘜 𝘪𝘯𝘴𝘵𝘢𝘯𝘵 𝘩𝘰𝘴𝘵𝘪𝘯𝘨 𝘳𝘦𝘷𝘦𝘯𝘶𝘦 𝘰𝘷𝘦𝘳 4 𝘺𝘦𝘢𝘳𝘴" ~ 𝘕𝘝𝘐𝘋𝘐𝘈 𝘊𝘍𝘖

From silicon to software, each layer of the stack adds value by leveraging the "𝘳𝘢𝘸 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭" of the layer below. For the cloud providers (IBM, AWS, GCP, Azure, etc.), the raw material is the NVIDIA chip (or the AMD, Intel, etc.). For the Model/Inference API providers like Claude/OpenAI/Mistral, it's the infra to train/serve. For startups, wrappers, and application providers, it's the inference APIs, and the chain continues.

The success of the provider hinges on the success of the consumer and vice versa with varying dependencies.

Subscribe now

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

Everyone is speed publishing, according to the AI Index report from stanford. Every year, the number of AI publications increases by the thousands as more researchers join the AI ranks.  With all this progress and FOMO, it's easy to lose sight of the long, winding road that led us here, I mean, it didn’t happen overnight, ChatGPT didn’t happen overnight, Artificial General Intelligence (AGI) will not happen overnight, its foundation upon the foundation, it could take years, decades, even centuries sometimes!

This post takes a step back into the 1930s (and even before), 50s, 60s, 70s, and 80s, when most of the concepts behind what we know as AI today were forged. Here, we take a trip together down memory lane. In this post, we are going to explore the key events in history, meet key people who made it all happen, look into how they influenced AI, and how AI was posed as the ethos of computer science, the ultimate goal upon which all grounding concepts were built.  We will also look at AI winters, why they happened, and where we are today. 

Note: As I was writing this, I wrote way more sections than I should have, so I decided to break it down into two parts (it will not likely fit an E-mail anyway). This here is part I, in part II, we will compare and contrast history and present, look at AI and innovations in general from the business and economic perspective, and brainstorm on how to build better products catering to a larger group of consumers from the lessons learned in part I. 

Before the 1930s

The journey towards artificial intelligence began long before the 1930s. Many early thinkers and inventors laid the groundwork for future AI advancements through a series of theoretical, philosophical, and mechanical innovations. Be it the Ancient Greek myths, such as those of Talos and Galatea, the mechanical men presented by Yan Shi to King Mu of Zhou in 10th century BC China, the logical groundwork for AI with concepts such as the syllogism and means-ends analysis (ALOT of details here). Even during the medieval period, the Banu Musa brothers created a programmable music automaton, and Al-Khawarizmi’s work in arithmetic and algebra introduced the term “algorithm” (9th century). Fast forward to the 1600s, with Schickard’s first calculating clock, Hobbes, and the mechanical theory of cognition by Hobbes (19th century). Onwards to the 1700s, Jonathan Swift described a knowledge-generating machine in "Gulliver's Travels, and the 1800s where Samuel Butler speculated on machine consciousness in "Darwin Among the Machines". 

Photo from: https://zkm.de/en/automatische-hydraulische-orgel-der-banu-musa-ibn-shakir 

These early milestones/inventions/philosophies showcase the longstanding fascination with creating intelligent machines, which set the stage for AI advancements in the 20th century, the century when modern AI was born. 

The 1930s to 1950s

Alan Turing,  considered as the father of theoretical computer science and AI, produced a foundational paper in 1936 on computable numbers which introduced the concept of the Turing machine, providing a theoretical framework for future computers. By 1941, Konrad Zuse had constructed the first working program-controlled general-purpose computer, and in 1948, Norbert Wiener coined the term "cybernetics," integrating the study of control systems and communication theory

In 1950, Alan Turing wrote his paper on COMPUTING MACHINERY AND INTELLIGENCE, asking the question, “Can machines think?” He explored and simplified the definition of “intelligence” with the goal of modeling it. Additionally, he proposed the imitation game, the Turing test, as a way to test and evaluate intelligence. 

Note on the Turing Test: Today, we discuss Artificial General Intelligence (AGI) and how we get there. The Turing test is one approach to identifying that state. Some argue that the Turing test is not a practical measure of AI. It's easily gamed with simple tricks that exploit human biases to appear “human” rather than demonstrate genuine intelligence, which is arguably not the goal we have in mind. We need a more robust and meaningful assessment of AI capabilities.

In 1955, Newell, Simon & Shaw Developed the First Artificial Intelligence Program, the Logic Theorist, the first program designed to imitate human problem-solving skills. The program was created to prove theorems in propositional calculus similar to those in Principia Mathematica by Whitehead and Russell. In 1956, The program was showcased at the Dartmouth Summer Session on Artificial Intelligence in 1956, marking a key moment in the history of AI research.  

The Dartmouth Summer Research Project of 1956 is often taken as the event that initiated AI as a research discipline ~ Artificial Intelligence (AI) Coined at Dartmouth 

The Dartmouth conference brought together many of the field's great minds in one place and also formed the basis for future research and innovations in the field. In the next section, we will discuss the first artificial intelligence white paper, one of the outcomes/artifacts of this gathering of researchers and scientists.

In 1957, John McCarthy (also one of the founding fathers of AI, mainly as he coined the phrase “artificial intelligence”) introduced LISP, short for, List Processing) which later became the favored programming language for AI. 

Before we finish the 1950s, I’d like to highlight one of the very first demos of AI. If you have been around, you have probably seen “The mother of all demos'' by Douglas Engelbart in the 60s, demonstrating the computer mouse as a UX interface, but have you seen Claude Shannon’s demo on Theseus, the AI mouse? If not, you should! Theus is probably one of the first manifestations of applied “intelligence”. Shannon’s goal was optimizing telephones and relays but this remains a great token of how scientists at the time approached the problem. 

The Dartmouth Workshop | 1956

In 1955, McCarthy, then a mathematics professor at Dartmouth College, formally proposed a summer workshop to the Rockefeller Foundation. The proposal, titled "A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence" outlined the workshop's goals and marked the first official use of the term "artificial intelligence"

With funding secured from the Rockefeller Foundation, the workshop was held in the summer of 1956 on the Dartmouth College campus. While the original plan envisioned a two-month collaborative effort, the workshop ultimately unfolded as a more loosely structured gathering, with participants coming and going throughout the summer.  

The Dartmouth workshop was attended by the founders of AI, other key figures were invited to the Dartmouth Workshop as well. Among them were future Nobel prize winners John F. Nash Jr. (1928–2015) and Herbert A. Simon (1916–2001). 

A Bold Conjecture: Machines That Learn and Think

During the Dartmouth workshop, a proposal for a summer research project was presented. Here is a quote from the proposal:

We propose that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it ~ from A proposal for the Dartmouth Summer Research Project on Artificial Intelligence 

This quote encapsulated the core ambition, the vision, of AI research at the time, which can be boiled down to creating machines capable of learning, reasoning, and problem-solving on par with humans. 

During the workshop, scientists explored a wide range of topics, most of which form the basis and the core of what we know as AI. It's all there since the 50s 🙂. Amongst those topics, I find two to be quite important: 

• Neural Nets: Early researchers in the field, like Marvin Minsky, delved into the idea that artificial neural nets (networks today) could simulate the brain's learning processes. 

“It may be speculated that a large part of human thought consists of manipulating words according to rules of reasoning and rules of conjecture.”

“How can a set of (hypothetical) neurons be arranged so as to form concepts? Considerable theoretical and experimental work has been done on this problem by Uttley, Rashevsky and his group, Farley and Clark, Pitts and McCulloch, Minsky, Rochester and Holland, and others. Partial results have been obtained, but the problem needs more theoretical work.”

• Self-Improvement: An ambitious goal at the time was the concept of machines capable of self-improvement. The idea from the paper hinted at the potential for AI systems to enhance their own capabilities continuously. Are we here yet in 2024? I don’t think so, we have self-supervised learning (see my earlier post on pretraining and fine-tuning for more details) but not self-improvement as a broader concept that encompasses more than just learning from data during training/initial phases, it includes any changes and AI might make to improve its performance, capabilities, or efficiency, potentially including but not limited to learning processes. Is the AI self-aware enough? 

Not Enough Algorithm-ability

“The major obstacle is not lack of machine capacity, but our inability to write programs taking full advantage of what we have.”

This quote reminds me alot of where we are today, we have tons of computers, and we did make breakthroughs with transformers, but arguably we might be hitting scaling laws here. After GPT-4, we are in catch-up mode, but who knows, time will tell. I still think we need to spend more time doing efficiency optimizations and not just throw computers at the problem. Anyway, let’s get back to the founders!

Founding Fathers & Key Figures

The Dartmouth workshop brought together some of the most influential figures in the early history of AI:

Claude Shannon

A pioneer in information theory, Shannon's work laid the foundation for understanding how information is transmitted and processed, both in machines and in biological systems. Remember Theseus, the AI mouse 🙂? 

During my college years, we studied Shanon’s Entropy and information theory. I have since been fascinated by the simplicity of the approach, which I personally think applies not only to senders/receivers between machines in an electrical communication system but also to humans. 

We humans communicate via language. Verbally, we have accents, which are noise to the receiver. Sometimes, we are even lazy to say the whole thing or lazy to provide context, which is, again, noise. We tend to rely on repeating what we say or changing how we formulate it to improve the quality of the knowledge we communicate, that’s the loop 🙂. Actually, Shannon’s theory can be applied in many areas, even in biology! 

Marvin Minsky

Minsky's research on neural networks and cognitive science shaped the AI field for decades to come. What’s fascinating is that Minsky had built his prototype of the neural network using mechanical components, he called it the SNARC machine,  a machine “made out of about 400 vacuum tubes and a couple of hundred relays and a bicycle chain” for simulating learning by nerve nets.  The machine he built simulated rats running in a maze and the circuit was reinforced each time the simulated rat reached the goal (kinda like Theseus, it could only do one thing, solve the maze). 

“A feedback loop reinforced correct choices by increasing the probability that the computer would make them again—a more complicated version of Shannon’s method, and a level closer to how our minds really work. Eventually, the rat learned the maze.”

Conceptually this is very close to how we do it today, with optimizers and error functions! In many ways, Minsky had a “technical” vision of how things would work. 

Nathaniel Rochester

An early advocate (also an IBMer) for using computers to solve problems traditionally done by humans, Rochester helped build the first AI programs. In 1956, he published a classic article in which he and colleagues simulated a network of neurons on IBM 701 and 704 calculators. See Nathaniel Rochester III (1919-2001) - Dharmendra S. Modha 

John McCarthy

The organizer of the Dartmouth workshop, McCarthy coined the term "artificial intelligence" (which earned him the top spot amongst the founding fathers of AI) and played a crucial role in defining the field's research agenda. “McCarthy has worked on a number of questions connected with the mathematical nature of the thought process, including the theory of Turing machines, the speed of computers, the relation of a brain model to its environment, and the use of languages by machines”

McCarthy also designed the LISP programming language that was heavily pitched for use with AI, and what inspired LISP machines (the specialized AI computer).

The Legacy Stays

The ideas, collaborations, and research directions that emerged from this gathering laid the groundwork for all of the progress that has been made in the field over the past six decades. Those in attendance, namely Marvin Minsky, Herb Simon, Allen Newell, and John McCarthy founded the three leading AI and computer science programs in the U.S.: MIT, Stanford, and Carnegie Mellon.

The workshop's legacy continues to this day. 

Subscribe now

The 1960s

The 1960s was an eventful decade for AI, with a shift towards knowledge-based and expert systems and early experiments in natural language processing. Leading this charge was Edward Feigenbaum, a student of Herbert Simon, a pioneer in decision theory and cognitive science.

Ed Feigenbaum and the Birth of Expert Systems

Feigenbaum's work with Herbert Simon sparked his interest in creating machines that could mimic human expertise. Inspired by Simon's theories on bounded rationality and human problem-solving, Feigenbaum and Alan Newell embarked on a quest to build "thinking machines” after Herbert Simon handed them the manual for the IBM 701 vacuum tube computer, this was their first commercial binary computer experience. 

“Taking the manual home, reading it, by the morning, I was born again.”  ~ Ed Feigenbaum's Search for A.I.

This quest led to the publication of "Computers and Thought" in 1963, a collection of articles on artificial intelligence that quickly became a bestseller. This was followed by the development of EPAM (Elementary Perceiver and Memorizer), a computer model that simulated human learning and memory.

Building on these foundations, Feigenbaum and his team created Dendral (1965), the first expert system. Dendral used knowledge of chemistry and heuristic search to identify organic molecules from mass spectrometry data, effectively automating scientific inference in a specialized domain (this reminded me of DeepMind’s AlphaFold 🙂).

ELIZA: The 1966 AI that beat GPT-3.5!

In 1964, Joseph Weizenbaum created ELIZA, an early natural language processing program that simulated a Rogerian psychotherapist.  Though simple in design, ELIZA's ability to hold seemingly meaningful conversations surprised many, even prompting some users to believe they were interacting with a real person. It’s surprising to me that in 1964 we had the same interfaces we have now (chatting with intelligence). I’d say Eliza is probably the closest to ChatGPT conceptually, but the tech, the intelligence itself was rule-based, it was creative but still followed a deterministic log. That said, it still beat GPT 3.5 in the Turing test! 

We evaluated GPT-4 in a public online Turing test. The best-performing GPT-4 prompt passed in 49.7% of games, outperforming ELIZA (22%) and GPT-3.5 (20%), but falling short of the baseline set by human participants (66%) from [2310.20216] Does GPT-4 pass the Turing test? 

Here is the link if you want to chat with Eliza: Eliza, Computer Therapist, also below is classic video showing the demo 🙂

The 1960s laid the groundwork for much of what we see in AI today. The focus on knowledge representation, heuristic search, and early natural language processing set the stage for the development of expert systems, machine learning, and, eventually, the large language models that power today's conversational AI.

The 1970s

In the 1970s, AI research centered around the idea that "computational intelligence," as Edward Feigenbaum put it, was the "Manifest Destiny of computer science." 

Refining Expert Systems: From Dendral to Meta-Dendral

Building on the success of Dendral, researchers developed Meta-Dendral (1975), a system that could automatically generate rules for mass spectrometry from vast datasets. The Dendral project also highlighted the importance of extracting, structuring, and representing expert knowledge in a way that computers could understand aka knowledge engineering. This involved collaborating with domain experts, codifying their knowledge, and creating inference engines to apply that knowledge to new problems.

“Knowledge engineers elicit the knowledge from the minds of human experts, shape the knowledge so that programmers can transform it into viable program codes (knowledge base), and create the inference system that uses a knowledge base to derive specific results.“ ~ (PDF) The Fifth Generation 

If you think about it, this is not very different from the goals we have today for AI agents, RAGs, and fine-tuned LLMs.

​​The LISP Machine: Pioneering AI Hardware

With the architecture and the design of expert systems ready, hardware capable of doing the job was needed to take it to the next step. John McCathy had developed LISP in the 50s, and the community at the time found it suitable for running expert systems because of its ability to do symbolic manipulation, a key aspect of AI research. The question was, what hardware would be optimized down to the instruction set to execute LISP programs? Please welcome the LISP machine. Those machines even came with IDEs optimized for running/debugging LISP.  

Many notable expert systems at the time were developed and deployed on LISP machines, including MYCIN (the medical expert), XCON (the digital equipment expert), and R1 (the sales assistant). The 70s were the rise of LISP machines. 

Early Network Effects for AI and Expert Systems

The 70s also saw the rise of ARPANET, the precursor to the internet. This enabled researchers to share knowledge and collaborate on AI projects across different institutions and disciplines. For instance, Feigenbaum's team at Stanford granted access to their AI tools via ARPANET, attracting hundreds of users from fields like biology and pharmaceuticals. Actually, they exposed the only non-DARPA machine at the time on the ARPANET. This added some network effects to AI research back then (but not enough to prevent winter from coming, keep on reading 🙂).

Scaling Even More

The Heuristic Programming Project at Stanford, led by Feigenbaum, aimed to scale up AI research by harnessing the talent of numerous graduate students (early open source 🙂?). This effort resulted in the creation of the "Handbook of Artificial Intelligence" a three-volume encyclopedia that became a standard reference for AI researchers worldwide. Mostly used for teaching and aligning researchers and post-grads on state-of-the-art.

The 1980s

In 1981, Japan launched the Fifth Generation Computer Systems project, a 1 billion dollar project (a lot at the time), aiming to develop powerful computers with AI capabilities like natural language understanding and machine learning. The four generations of computing that existed: 

• the first generation being diodes and vacuum tubes 

• the second generation is transistor-based

• third are integrated circuits

• fourth generation using microprocessors

“Parallel processing will replace the one-step-at-a time procedures of the standard yon Neumann architecture of today's computers; jobs will be divvied up so that many subjobs can be in the works simultaneously” from the fifth generation paper

Finally, the fifth generation, which is viewed as whatever is able to run artificial intelligence. Arguably, the Japanese stressed the fact that parallel processing is needed, which makes you think about GPUs but also quantum computers. 

"Whoever establishes superiority in knowledge and technology will control the balance of world power regulating both cost and the availability of knowledge" ~ McCorduck and Feigenbaum from the fifth generation paper

“The Japanese expect that use will be tremendously expanded if people can communicate with the system in natural language such as Japanese, English or Swahili” from fifth generation paper

The Japanese did have the foresight indeed, Natural language as a UX did wonders and “tremendously expanded” the use of AI (look at ChatGPT 🙂).

Finally, another quote to show you how close the thinking was to where we are today: 

“Are your plants unaccountably turning yellow? Have you found that even your broker can’t keep up with the latest money market options? Are you afraid of doctors and lawyers or too poor to call them in when you need them? Expert systems will be on call day or night, providing instant, specific solutions to your problems.”

Toward the end of the paper, the author asks Ed Feigenbaum the question about the ethical and Legal ramifications of expert systems. The answer was "No," he said, there will always be "a human in the loop" taking full responsibility. Expert systems are designed to aid decision-makers, not replace them. 

Despite the potential, AI researchers recognized the limitations of expert systems (we will talk about AI Winters in the coming sections). These systems struggled to learn independently, lacked common sense reasoning, and were unable to replicate the intuitive and creative aspects of human thought. Nevertheless, the work of the 1970s and the 1980s laid a crucial foundation for the AI breakthroughs that would follow.

Winter is Here!

The science kept coming, and the research did not stop, but the application was nowhere near. Working as a researcher myself for years, I know it can get very interesting to just keep digging more and more without a clear path of viability. If I can publish a paper at a well-known conference and contribute my learnings, I win, or that’s how I thought back then. 

After the Dartmouth summer proposal for AI research, many optimistic predictions were made, especially those ignited by the promise of groundbreaking applications. Throughout the 1960s, AI research received significant funding from DARPA, with few requirements for delivering tangible results.

First AI Winter

More AI research and writing, for example, the “Perceptrons” book by Marvin Minsky and Seymour Papert or the checker’s player by Arthur Samuel, but little commercial application or use for them. In 1973, Sir Jamel Lighthill, a professor, and an applied mathematician, published a report that critically assessed AI research, he was supportive of the idea and applications of AI for industrial automation, but had concerns about the research trying to meld it with the analysis of brain function. It’s said that this report strongly led to funding cuts in the UK and raised doubts about the field's potential globally, including the U.S. Which might have led to the first “AI Winter”,  a quiet period for AI research and development.

I did a bit of digging to see if research and development were really affected, and whether this period of time deserved to be referenced as a “Winter”. It didn’t seem that there was a stop to content being generated be it research or otherwise, on the contrary, there has been a steady increase in the amount of “events” (see the circles) related to AI AND the amount of content generated for AI, i.e., AI research was more active than ever during the first “AI Winter”.

I’d rather consider the first AI winter, a clash of expectations, but as an early field, people (especially the scientific community) knew there was still progress to be made. I.e., it didn’t create an economic bubble burst but rather instilled some doubts which did impact viability for commercial use or delayed it.

1980-1987: Renewed AI Excitement

The 1980s ignited interest in AI after the skepticism in the 70s, this was mainly driven by the emergence of expert systems, Ed Feigenbaum even published a book about the applications of expert systems, highlighting case studies from many companies, including IBM, FMC, Toyota, American Express, and more. This optimism and references to commercial applications attracted increased funding, leading to a period of renewed excitement in the field. Also, as you can see from the graph below, the AI curve and Ngram mentions reached their peak!

1987-1994: The Real “AI Winter”

The enthusiasm for expert systems in the 80s was short-lived as their limitations became apparent. These systems struggled to handle novel information and situations that fell outside their pre-programmed knowledge base (they were IF/ELSE rule-based). Additionally, the collapse of the market for Lisp machines, a key platform for the development of expert systems. Here some of the reasons: 

• Compared to LISP, there were more versatile and cheaper programming languages/technologies like C++ and RISC computers which made specialized LISP machines (requiring a custom TTL processor) less appealing.

• The existing Lisp Machine software was not flexible or user-friendly enough for the average developer (don’t we always have this problem even today 🙂).

In addition to the above, in 1987, DARPA's decision to cut AI funding again, coupled with the failure of the Japanese Fifth Generation Computer project in 1991, deepened the second AI winter, increased doubts, and might have led to a burst in the AI economic bubble.

> Note: I find the timing of starting Nvidia (1993) interesting, just a bit after the decommissioning of the fifth-generation project, which initially had aimed to optimize computing for AI applications using a form of parallel processing. Didn’t dig much here, but the correlation could not escape me. Not sure if Jensen Huang had that far ahead foresight. 

The combination of shifting market demands from LISP machines/expert systems, management and financial struggles, technological limitations, and competition from more versatile and cheaper technologies are considered reasons behind the “AI Winter.” Below, you can see the trend going down after 1987, and there were also fewer publications and writings (see the circles). 

More readings on AI winters here. 

1994-Present: Steady State & Boom!

Despite the downturn faced in the 1980s with the AI winter, AI research kept beating, and interest in the research community didn’t suffer as much. This is another way to say that AI stayed in the “lab” but didn’t abandon the world after the realization that more work was needed to produce better ROI for enterprises, at least enough to justify spending on skill/hardware to make it happen. Let’s see what happened by looking at the timeline:

So we can see that AI content peaked in the 80s, AI winter started, and less content was produced. But research and progress kept on. Of the noticeable periods in AI history is the moment when IBM’s Deep Blue defeats Garry Kasparov, the world’s chess champion at the time, marking an end to mono-task models of intelligence, which was AND still a goal for AI research. Noticeably, the late 1990s towards 2010, most of AI development happened in robotics, domestic robots, etc. 

https://twitter.com/olimpiuurcan/status/1524249481512103937

2009 onwards saw numerous manifestations of AI in areas such as autonomous vehicles, AI ethics and governance, quantum computing research, and the integration of AI into various industries, including healthcare, finance, and entertainment. Notably, Google introduced the self-driving car project in 2009. In the 2011, IBM introduced Watson, which defeated human champion Jeopardy (Historically, it seems like IBM knows how to win with AI, be it Deep Blue or Watson). 

In addition to self-driving cars, robotics, and game-playing AI, deep learning with neural networks started to flourish around this time span. AlexNet, a deep neural network won the imageNet large scale visual recognition challenge which sparked interest in deep learning and neural networks. 

In 2015, Google DeepMind developed AlphaGo, which defeated Lee Sedol, the world champion of Go at the time. In 2016, Generative Adversarial Networks (GANs) were introduced, which started to reshape the generation of very realistic images and videos.

In 2017 Researchers began to apply reinforcement learning to a range of tasks, from playing video games to robotic control, demonstrating machines' ability to learn complex strategies, and the publication of the papers that introduced attention with transformers. The following year, 2018, Google introduced BERT (Bidirectional Encoder Representations from Transformers), making a dent in optimizing context awareness in natural language processing and paving the way for future applications (Mr. ChatGPT). 

From 2019 to 2024, AI witnessed major milestones. OpenAI's GPT-2, GPT-3, and GPT-4  (also Claude, Gemini, etc.) showcased language models' power to generate human-like text (also raising questions/concerns about ethical AI use). Noteworthy is that in 2020, AI became a very useful tool in combating the COVID-19 pandemic, even aiding in prediction and vaccine development. DeepMind's Alpha Fold 2 solved the long-standing protein folding problem the following year, potentially revolutionizing biology and medicine. Very recently, Google announced Alpha Fold 3, which aids in accurately predicting the structure of proteins, DNA, RNA, ligands and more, and how they interact. We hope it will transform our understanding of the biological world and drug discovery. 

I’d say the 2020s is finally the decade of AI viability for commercial application at a large scale, the growth trend of AI is rising exponentially with no signs of stopping That said, we have seen a similar trend in the early 1980s, followed by rapid fall in interest by the mid 1980s (so we have a couple more years to break the record 🙂), see the figure below:

I personally see no sign of this stopping or decaying now, but I could see the overpromise of AGI, setting the wrong expectations might lead to dampening the growth trend, after all, humans continuously raise their expectations when exposed to better experiences, better things, and expectations are hard to reverse, so we should be careful what and how we communicate to avoid an economic bubble burst! 

Below is another view of the AI trend for the 2000s:

AI History Visualization

Before we part ways, I’ll point you to the visual I used throughout this post to point you to the AI timeline.

You can find the interactive visual here and the code here (feel free to contribute :))

Next post, we will explore the lessons we learned from the history of AI in this post, and muse on how we can apply them to inform how we build successful innovations, products, and businesses. Stay tuned!

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

I recommend going through the mindmap first, internalizing it, and then coming back and looking for answers in the long form here. With that in mind, here goes!

The full mindmap PDF can be found here.

Subscribe now

Model Pre-training

Growing up, I competed as a professional swimmer. Before the competition, a day, a week, depending on availability, I’d go to that Olympic 50-meter swimming pool and take a feel for it, even do entire training sessions there. If I could, I’d familiarize myself with the surroundings minus the crowd, the shouting, the whistles, etc. But it would give me the much-needed confidence for the next task to come, the actual race, which is a whole different game, mentally and physically.  I was practically pretraining for the competition 🙂

Formally pretraining is defined as:

Pre-training refers to the process of initializing a model with pre-existing knowledge before fine-tuning it on specific tasks or datasets. In the context of AI, pre-training involves leveraging large-scale datasets to train a model on general tasks, enabling it to capture essential features and patterns across various domains. ~ Lark

Pretraining helps the model perform better when it later learns tasks specific to a smaller, targeted dataset. It improves the model’s accuracy and efficiency, as it starts with a solid foundation of knowledge.

Pretraining is a necessary evil, and someone has to do it. That someone or something needs to have POWER 💪 and MONEY 💰. But once done, many can benefit from using it (if there was an intention for that in the first place, e.g., open source models). Technomistically speaking, we would be talking about sunk costs and Amortization.

• Sunk Costs: Pretraining requires a LARGE initial investment in computational resources and data acquisition. These costs are considered sunk as they cannot be recovered. However, once a model (big or small, still needs alot of resources) is trained, and compressed to represent a virtual version (it doesn’t really “copy” the data or memorize it) of the data it has been trained on, one can start to amortize these costs, leading to another fancy term called economies of scale (the initial high cost of training the model is spread out over many uses for inference).

• Opportunity Cost: Mentioned in the mindmap above is Self-Supervised Learning (SSL), which is basically a model trying to figure out the meaning of life by looking at ALOT of data, learning patterns between them, and generating it’s own labels (magic ✨). In doing so, SSL minimizes the need for explicitly labeling data, thus reducing both the direct costs of data preparation and the opportunity costs associated with extensive data labeling processes.

By now, you might be thinking, well, it’s great that someone did some pre-training (“pre” because the assumption is implicit in that it’s not yet ready for whatever you want it to do, i.e., it’s an invitation to train the model more). "I am pre-trained. Please train me properly sensei”

Optimizations

After pretraining a language model, it can be optimized further depending on the outcome desired. Examples of outcome optimizations are fine-tuning, prompt engineering, instruction finetuning, Retrieval augment generation (RAG), and so on. Notice that here, I intentionally said outcome optimizations and not model optimizations. To optimize outcomes, there are invasive and noninvasive methods, similar to when you visit a doctor with a problem, the doctor presents you with options like surgery (invasive because they modify something in you) or physical therapy (sometimes just reminding your body how to do things), here we also have options:

• Noninvasive: prompt engineering, tuning, and RAG (aka engineering around the model, which I personally consider a form of prompt optimization). These methods are cost-effective because they utilize existing resources to enhance output without significant expenditure. However, there is a limit to how much the model can improve with them.

• Invasive: requires more training on unseen data to improve the model's ability to produce certain outcomes. Examples are fine-tuning and its variants and methods (e.g., instruction fine-tuning). Such methods involve a higher initial cost, which might be needed to surpass the “limits” of noninvasive methods in terms of how the model performs on various tasks/benchmarks.

There are other optimizations, not necessarily to produce a better outcome, but to optimize how to get to the same outcome. For example, quantization is a form of cost/energy optimization (by mapping floating point representations to lower bits), done for training and inference. Distillation (distilling the same amount of knowledge onto a smaller model), and more. All of these take the desired outcome as a goal and work around how to get to the same outcome with the least of effort/energy/gas. I like the concept of “gas” here, borrowed from blockchain because it describes a fee without the specifics of that fee. 

Gas is the fee required to successfully conduct a transaction or execute a contract on the Ethereum blockchain platform. ~ How Gas Fees Work on the Ethereum Blockchain. 

Of all the optimizations, fine-tuning is one of the hottest topics in the field these days. 

Full & Parameter-Efficient Fine Tuning (e.g., LoRA)

I am jumping ahead here because we don’t need to get into the nitty gritty. After pre-training, you could go and do Full Parameter Fine Tuning (FPFT), which would mean that you take that pre-trained model and re-tune it to a specific task (e.g., teach it kungfu).

Full-parameter here would mean that would have to:

• Update all the weights and parameters of the Neural Network (which can be very large) frequently.

• Go through many Hyperparameter optimization rounds to avoid side effects.

• Run it multiple times through your data (aka, multiple Epochs)

• …

While you are not running the training over the entire internet as a dataset, only a subset of the data (e.g., KungFu), it’s still considered “costly”.

There are ways to solve for this with Parameter Efficient Fine Tuning (PEFT), which has the following benefits:

• Leaves pre-trained model weights fixed and only adopts a small number of task-specific parameters during fine-tuning.

• Reduces storage memory because you are not updating the entire model parameters (there are multiple techniques, again mentioned in the mindmap).

This makes fine-tuning cheaper and accessible on modest hardware. Techniques include Low Rank Adaptation (LoRA), which decomposes larger weights matrix representations into smaller matrices with low ranks and other variants such as adaptive layers, prefix tuning, and more.

PEFT provides incremental cost reductions 📉 through the efficient use of hardware, which also leads to more optimized resource allocations, lower energy consumption 🍀, and enhanced overall operational efficiency.

Reinforcement Learning from Human Feedback (RLHF)

As mentioned above, there are multiple ways to improve the model’s performance. Another example is Reinforcement Learning from Human Feedback (RLHF) which is a  “Human in the loop” strategy to basically teach models “principles”, i.e., what it takes to be “helpful”, “harmless”, and so on. 

RLHF works briefly as follows:

• First, humans judge a model's output. These humans review these outputs and provide feedback. This feedback could be in the form of rankings, ratings, or direct corrections. The key is that these human evaluators assess the model's responses based on desired behavior, as described above. 

• The feedback from human evaluators is then used to fine-tune the model. This may involve training the model to predict human preferences or directly optimizing the model's parameters based on the feedback. The goal here is to align the model more closely with human values and expectations.

While RLHF has shown good results, it is still expensive (find the humans and make them do the work). Second, it is difficult to encode human values into the model (you can try). Third, the model may not generalize well to new situations. Despite these limitations, RLHF is a valuable tool for AI safety research and has the potential to contribute to the development of safer AI systems.

Incorporating RLHF entails additional costs due to human involvement. These costs must be balanced against the marginal utility (benefit-to-cost ratio for each human involved) derived from improved model accuracy and compliance with ethical standards, enhancing user trust and market acceptability, all of which are hard(er) to quantify but are important nevertheless.

Conclusion & Recommendations

Models are pretrained to get a basic understanding of the world (depending on the data it fed on), and are optimized to get “better” at achieving an outcome. The outcome may vary. It could be to get better at math, or writing poetry, or better at following instructions (in English or other languages), or better at adhering to what aligns with human values and principles (causing no harm, no discrimination, ethics,...).

Other outcomes could revolve around making the model more efficient cost and energy wise, faster at inference, etc. We can call those efficiency gains.

With this in mind, here are some recommendations:

• Adapt and Fine-tune: Use in-context learning and fine-tuning strategies to adapt to new requirements and teach your model more about the tasks/outcomes/use-cases you want it to be good at without the need for extensive retraining which helps conserve resources and allows you to respond quickly address your organization’s needs. 

• Invest in Parameter-Efficient Fine Tuning: pretrained models are expensive to train. Luckily, we are seeing investments in making models open and thus available to a wider audience. The basic models are rarely useful without additional tuning, make use of existing advancements in training (e.g., LoRA which we will talk about more in a later post), which can reduce the tuning costs and improve throughput, as well as alleviate logistical challenges associated with updating AI models, making it ideal for continuous improvement cycles.

A comparison of training throughput (tokens per second) for the 7B model with a context length of 512 on a p4de.24xlarge node. The lower memory footprint of LoRA allows for substantially larger batch sizes, resulting in an approximate 30% boost in throughput. ~ Fine-Tuning LLMs: LoRA or Full-Parameter? An in-depth Analysis with Llama 2 

• Balance Human Input with Automated Processes: While human feedback is crucial for ensuring model reliability and ethical alignment, it is a balancing act to weigh these benefits against the costs of human involvement and to optimize the use of automation were beneficial.

• Invest in Cost/Energy and Computational Optimizations: Implementing computational optimizations such as quantization and quantized-aware training (see the mindmap) should be prioritized to reduce operational expenses (OPEX), such as energy consumption and maintenance. It will also reduce capital expenditure costs (CAPEX)  by eliminating the need for expensive and high-performance computing hardware.

Subscribe now

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. I look forward to hearing from you!

⛔ For the busy folks, here is a TL;DR version:

• Platform Principles: Create a foundation for streamlined development of value. I.e., Platforms provide tools, services, and automation to simplify and accelerate value delivery.

• Platform Archetypes: Platforms come in various shapes and forms. Examples are Cloud platforms (Google Cloud, etc.), business platforms (Salesforce, etc.), and Internal Developer Platforms (IDPs). All offer unique value and are targeted to a set of personas.

• Key Considerations:

  • APIs: Inspired by Amazon’s API mandate, platforms should provide consumable and persona-focused APIs that cover the right set of needs.

  • Purposeful Design: Understand the value you want to deliver and tailor a platform to serve those needs. Failure to do so results in overserving (surplus) or underserving (shortage). You will want to strike the balance (equilibrium).

  • Product of Platform vs. Platform for Product: Determine if the platform itself or the value it brings is the desirable.

  • Skills and Tools: Platform creation requires a wide range of skills (frontend, backend, infrastructure, etc.). Assemble the right people and tools to get the job done. Assemble your platform engineering fellowship.

• To Remember: Keep your platform user-focused, simplify wherever possible, and continuously evolve it based on feedback.

Subscribe now

What REALLY is platform engineering? 

I’d define it simply as:

The foundation upon which value is being delivered

But it’s also a mashing of building blocks, tools, and services that enable the target persona (developers, ML engineers, data scientists, ...) to craft value. 

There are many contexts in which you’ll see platforms mentioned. I will use the following format to describe them “_platform such as platform_name that provides _tools_services  to enable _those_enabled to _value

• Cloud Platforms, such as Google Cloud Platform, AWS, Azure, provide services (databases, networking, IDEs) for developers to use to generate value. 

• Business and Marketplace Platforms, such as Salesforce or Shopify, provide pre-built modules for customer management, payment processing, and e-commerce functionality to enable businesses to operate and sell online.

• Social Media Platforms, such as Facebook or Twitter, provide social networking, content distribution, and API access to enable individuals and businesses to connect, engage with audiences, and build communities.

I left the best for last, Internal Developer Platforms (IDP), garnering attention, are no different. Let’s use the format above: 

• Internal Developer Platforms (IDPs) provide self-service portals, automation, and standardized workflows to enable developers to provision resources, deploy code for value-generating applications, and manage them easily.

Depending on the level of abstraction needed, a platform exposes a construct (or an Application Programming Interface aka API) to request for a service needed to provide value. Let’s talk more about this. 

The API Mandate, the Chef’s Analogy, and Purposing Platforms

When I think about platforms nowadays, I think about two things:

• The API Mandate - thanks to Jeff Bezos 

• Indian food (delicious) 🥘 - thanks to Kelsey Hightower

Food and APIs, what are you talking about, man? Let me explain 🙂

Amazon’s API mandate, as championed by Jeff Bezos at Amazon, essentially states that all functionalities within a platform should be accessible through well-defined APIs. I.e., all value generated by any team should be exposed via a well-defined interface, the side effect of that mandate was that Amazon was able to externalize those set of APIs to wider consumptions, and also what became AWS cloud platform (Augusto Marietti has a more detailed writeup on this here, good read!). That is probably one of the earliest appearances of a “platform”. The question is what kind of APIs to expose? 

As highlighted in Working Backwards by Bill Carr and Colin Bryar, Amazon’s teams are grouped by mission vs. function, so depending on each team’s mission, and desired outcome, an API will emerge to allow others to consume the artifact produced by those Two-Pizza Teams (or should we call them distributed platform pizza teams 🙂).

Indian food is a specialized cuisine that is known for its unique taste and flavors. In one of those Twitter spaces, Kelsey Hightower gave the example of an Indian restaurant where quality is tied to specialty. You wouldn’t go to an Indian restaurant to eat Italian food. If that restaurant served Italian, you’d typically question the quality of the food!  Sure, you can stock a kitchen with ingredients from all cuisines, but until you curate a menu, you might not have something of true value to offer. 

The combined lesson here is to understand who you are serving and for what purpose, to build the right abstractions. Generalize too much, and you risk complexity, you overserve your target persona, and you drive them away from using your abstractions altogether (negative value).  Specialize too much, and you risk undeserving the target persona, they don’t get what they need, and they will start building their own abstractions on top to fill the gaps (shadow IT). 

In economics, the right terms to use are supply surplus and shortage. Supply surplus denotes a situation in which the quantity supplied of a good exceeds the quantity demanded. In the case of platforms here, too many APIs, too many unused templates, and too much functionality that has the side effect of increasing support times, engineering development time, etc. Whereas shortage represents the opposite scenario where the quantity demanded is greater than the quantity supplied, for example, developers need APIs to produce services of value, lack of APIs can be an example of a shortage. Here is an excerpt from Stevey’s Google Platform Rant describing an example of this:

Google+ is a prime example of our complete failure to understand platforms from the very highest levels of executive leadership (hi Larry, Sergey, Eric, Vic, howdy howdy) down to the very lowest leaf workers (hey yo). We all don't get it. The Golden Rule of platforms is that you Eat Your Own Dogfood. The Google+ platform is a pathetic afterthought. We had no API at all at launch, and last I checked, we had one measly API call. One of the team members marched in and told me about it when they launched, and I asked: "So is it the Stalker API?" She got all glum and said "Yeah." I mean, I was joking, but no... the only API call we offer is to get someone's stream. So I guess the joke was on me.

It’s a balancing act and no easy feat, but when you get there, you are a platform karate black belt! 

Is the Platform the P-roduct OR the p-roduct

Platforms are value bridges, at least when considering why you would build one in the first place. If you treat the platform as the “main” thing, you’d be overlooking the purpose and the reason a platform exists in the first place, at  the same time, if you JUST focus on the value the platform delivers, you might not even make it to where value is finable/consumable, well, you still need something, a vehicle of sorts to get there! 

This has been a long-standing topic of discussion, and it is expressed in different terminologies/contexts such as: 

• The mean vs. the end 

• The platform product vs. the experience product

• The P-roduct vs. the p-roduct (lower case and upper case)

Let’s take the last one. How can the p-product (lowercase) serve the uppercase P-roduct. How can we build platforms with purpose? You might have different goals depending on who you are (a supplier or a consumer) and your strategy. 

Shreyas Doshi talked about the generic concept in one of his posts. Here are some of the examples/results based on surveys from his followers (who are mostly product builders) on what they consider the P-roduct: 

• Netflix: Most view licensed content as The Product, suggesting that content is more crucial than the technology (app or CDN) or algorithms.

• Uber: The ride is viewed as The Product, more than the app or the matching algorithms.

• Instacart: The groceries and the delivery system are almost equally viewed as The Product, indicating the importance of the end product and its delivery.

• Amazon: The delivered item is considered The Product, underscoring the end result over the platform or logistics (the listings, the marketplace, etc). 

• YouTube: The creators are seen as The Product, highlighting the value of content creators over the technological features (search, the app, the video viewer).

As a supplier (a vendor), you might consider the platform itself as a P-roduct to streamline how platforms are built, making it easier to produce content on top. As a consumer, you might think more of the content you offer on top of platform while placing constraints on platform usability (that acts as a filter to which vendors you “buy” the platform from). 

I don’t believe there is a clear cut answer, I’d be interested on your thoughts and how you approach this as a consumer or supplier of platforms. Let chat!

Picking the Right Skills & Tools (e.g., AI/ML Platform)

Once you understand the core value you are delivering via the platform and the type of platform you want to build, it’s time to make sure there is enough skill in the house to build it. Here, you might have to mix and mash skills. For example, you might need a mix of frontend, backend, and infrastructure, as well as one or more of DevOps, MLOps, SecOps, ... xOps 😁, whatever makes sense to get the job done (e.g., reduce cognitive load for developers).

To reduce the cognitive load of developers, the Platform team should cover the entire tech stack: Infrastructure/DevOps/SRE, but also frontend, backend, and security topics. ~ from Platform Engineering, Part 2: WHAT Are The Goals of a Platform Engineering Team? | by Benoit Hediard | Stories by Agorapulse | Medium 

To make things a bit more realistic, let’s go with an example of an AI/ML Platform and the typical components involved. The closer the component is to the top, the more visible it is to the users.

Let’s break it down relative to the end-user (inspired by Wardley maps but without the X-axis):

• Top Row (High Value to Users): This represents the User Interface Layer where direct user interactions happen. AI tooling, including chat interfaces (e.g., streamlit), Dev environments (e.g., Jupyter), and maybe an orchestration UI (e.g. Kubeflow)

• Second Row (Medium Value to User): Data Processing and Workflow Management tools (e.g., Kubeflow pipelines, Airflow, DAGs,…) that are crucial for the operational aspects but less visible to the end user.

• Third Row (Medium to Low Value to User): Model Monitoring and Operations tools (Promethus/Grafana, Evidently,…), which are essential for maintaining the platform's health and performance.

• Bottom Row (Lowest Visibility): Infrastructure (AWS, GCP, Azure,…) and Platform Management along with DevOps and Collaboration Tools, which are foundational yet mostly invisible to the end user, another name for this layer can be the “commodity” layer.

I will not go more into the technical details here, but there are multiple venues to explore if you are interested specifically in AI/ML platforms and personas. One venue to explore is the Cloud Native AI working group, especially the most recent AI whitepaper. Another venue for exploring platform engineering is the platform engineering white paper.

The components will change depending on the type of platform you are building. You don’t need everything in a single platform. You need to curate and specialize based on feedback from your target platform users (remember specialization and Indian food 🥘). The left side of the figure above shows what a typical “generic” platform would be comprised of.

As such, you will need to ensure that you have the right skill set to build your target platform, for the personas you are targeting.

Conclusions & Reflections

So here are some parting recommendations:

• Define Clear Objectives and APIs: Start by clearly defining your platform's purpose and the APIs it will offer (how will your platform be consumed). This clarity will enable your platform to provide specific, valuable functionalities that are easy for users to implement and integrate. This is another way to say, start with the end in mind (from the 7 habits). 

• Focus on User Needs: keep the end-user in mind during development. Understand their pain points, requirements, and usage patterns. This understanding will help in creating features that are not just innovative but also highly relevant and user-friendly.

• Keep it lean and Evolve with time: Avoid adding unnecessary features/functions that do not add clear value. Overcomplication can deter users and increase the cognitive load, making the platform less appealing and harder to use. Always aim for simplicity and efficiency in design which itself will drive more straightforward execution.

• Continuous Feedback and Improvement: Implement a robust feedback loop with your users to continuously improve the platform. Regular updates based on user feedback and emerging trends will keep the platform relevant and highly functional. I found this post by Abi Noda and Tim Cochran helpful as a guide for collecting feedback (via qualitative and quantitative metrics). I will also discuss feedback collection in upcoming posts. 

Different people have different definitions of value. What’s important is that within your realm, among your people, you align on the core value you want to deliver. Here are some questions to ask to sanity check if you are on the right path:

• Do I understand what problem I'm solving (the core job to be done)? Can I articulate it better than the customers themselves?

• Is this tech the right fit for the use case at hand? It’s not about the fanciest tool, but one that gets the job done.

• How well do the tools and systems I implement meet the evolving needs of your users? This usually translates to how modular is my architecture.

• Am I adding value or just complexity? Sometimes, the simplest option is the right one, beware of overengineering (been there, done that)

• Can I do better? In what ways can you further automate to reduce costs and cognitive load? Are you collecting enough feedback to know this?

That’s it! If you want to collaborate, co-write, or chat, reach out via subscriber chat or simply on LinkedIn. Looking forward to hearing from you!