The Data Advantage in the Foundation Models Era

When building across the ML lifecycle, there is a consistent debate on whether the gains are due to data or model optimization. Venture capital investment has also cycled between the two several times over recent history. Recently, a focus on data-intensive machine learning drove improvements in data quality and mining through programmatic labeling, enrichment and data modeling. Now, we’re shifting into the foundation model era** with key innovations in transformer models driving a new era of research and investment on the modeling side. Of course, this is followed by a wave of investment in foundation model opportunities.

More foundation models are being released, leading to the modeling space becoming hyper-competitive. This raises the question of what differentiator will generate a true edge. Historically, data-intensive machine learning has generated a true moat and this new era will be no different. For companies with foundation models or startups leveraging these models, differentiation comes from quantity, quality and applicability of data for the use case.

We can examine the implications for development by breaking down the workflow. In the past, data scientists had end-to-end ownership of all development. This was a frustrating mismatch for a field of experts who tend to have deep expertise rather than a broad skill set.

Before, teams who built machine learning applications had to build entire platforms and collect data into data models. Then, teams shifted to business problems and leveraged their analytical and mathematical toolkit. This is still the development process, but the key difference is the outsourcing of each step to enable depth instead of breadth. In the most extreme case, this may look like a SaaS platform at every stage of development.

The key shift is the acceleration in development towards the application layer. A popular and long-standing metaphor in development is Maslow’s hierarchy of data science needs, as shown on the left in the figure below. This visual shows the importance of taking each step before moving up the pyramid to AI products, as is the inclination. In the post-foundation model state, we operate with a new set of assumptions. Improvements in the modern data stack has moved data scientists up the pyramid towards business questions rather than infrastructure.

In the post-foundation model era, companies can outsource steps of the pyramid to new startups. This will enable some startups to focus on the application layer. On the far right, we hone in on core innovations that will enable the new development process.

This shift has disrupted some of big tech’s advantages by democratizing access to quality resources. In the past, big tech and those with significant data stacks were best able to leverage the benefits of AI. Prior platform shifts to cloud and mobile created an opportunity for startups to oppose incumbents who would have to rebuild on a new platform. There was an obvious innovator’s dilemma as they’d have to rip out old profitable products and reinvent their business model to compete. Innovations in foundation models have accelerated and unlocked new opportunities for platform unbundling.

Foundation models will enable a similar dynamic for incumbents who have built products and services on end-to-end platforms built in house that may underperform foundation models. Yet, many of the best foundation models are being released by big tech incumbents such as Google’s flan, Microsoft’s copilot or Meta’s make-a-video. This means the opportunity for startups is creative application and the use of reinforcement learning as an agent. And another opportunity for startups that enable this application at scale, through a generalized intelligence or deployment layer. This is a dynamic to watch as more foundation models emerge… would open source demand forecasting models outperform models large F500’s have built in house (albeit with some tuning)?

This outsourcing of foundation models has an important consequence. Because everyone theoretically has access to the same foundation models, the competitive advantage becomes the data and feedback loop.

The real differentiator is the collection, curation and use of data to improve user experience of end users, whether for the development of foundation models or in applications. Foundation model companies will build curated quality datasets through proprietary processes. Examples being human sentiment and emotion or ecommerce multimodal datasets. Application businesses will ingest and aggregate data sources into deep personalization for content generation for people and brands. It is critical in the next wave of generative AI businesses that we stay rooted to the principle of data-intensive machine learning.

**We align with the foundation model terminology versus LLMs because we believe this trend is bigger than LLMs, but to each their own.

Reach out if you have a perspective to share or are building in this space: madison@costanoavc.com.

Resources: