AI chip market evolution: Cloud vs. edge in training and inference

Part 2: Edge training, inference, and market trends

The AI chip market is undergoing significant transformations, which can be understood through two key dimensions: deployment environments (cloud vs. edge) and market segments (training vs. inference).

Cloud-based training currently dominates the market and is expected to maintain strong growth in the future. Training is critical for AI model development, requiring immense computational power to process vast amounts of data, which is why it is primarily concentrated in cloud data centers. NVIDIA is the dominant player in this space, but competition from AWS, Google TPU, and Meta could reshape the market landscape in the coming years.

Alongside training, the cloud inference market is also experiencing rapid growth. Inference refers to applying trained AI models to real-world scenarios and making decisions based on real-time data. The increasing demand for inference is driven by advancements in AI models and emerging application scenarios. While inference requires lower computational power than training, it demands high accuracy and low latency, making cloud-based inference expansion essential for scaling AI applications.

On the edge, both edge training and edge inference cater to demands for low latency and data privacy. Compared to cloud-based training, edge training remains a smaller market but is expected to grow with the rise of smart devices and autonomous vehicles. Meanwhile, edge inference focuses on real-time, on-device processing, which is crucial for applications such as autonomous driving and other latency-sensitive use cases. While the edge market remains a niche segment, its importance will increase as these specialized applications continue to evolve.

In conclusion, training and inference in both cloud and edge environments present unique challenges and opportunities. Companies capable of delivering high-performance, cost-effective solutions in these areas will have the potential to challenge existing market leaders.

Following our previous discussion on the cloud AI training and inference market, this article will focus on the on-premises AI chip market for training and inference.

Our Analysis

Compared to the cloud market, on-premises AI solutions offer distinct advantages in low latency and data privacy. As emerging applications such as autonomous vehicles and smart devices grow, on-premises AI training and inference are expected to be key drivers of future market expansion. This article will analyze the trends shaping this segment and explore efficient, cost-effective solutions to address competitive challenges. Finally, we will summarize the key findings from both articles in a comparative table and discuss the future trajectory of the AI chip market, highlighting strategic areas that potential competitors and market players should closely monitor.

3. Edge Training

In the AI chip market, edge training represents the smallest segment, accounting for approximately 3–5% of the market. Edge training requires high computational efficiency and is highly sensitive to latency. These applications must perform AI training at the edge rather than relying on cloud-based computing resources. Examples include autonomous vehicles that require real-time learning and adaptation of their perception systems, industrial machines and robots that train and adapt based on local data, wearable devices such as smartwatches and health monitoring systems that continuously learn and adjust based on user data in real time, and smart city applications that require immediate data processing.

With growing concerns over data privacy and security, businesses are increasingly opting for edge training to ensure sensitive data remains protected. In this market, NVIDIA remains the dominant supplier, while AMD holds a smaller share. If emerging players like DeepSeek can provide cost-effective AI training solutions optimized for edge devices—especially amid growing demands for data privacy and real-time processing—they could become serious challengers to NVIDIA.

4. Edge Inference

Edge inference accounts for approximately 10–15% of the AI chip market, primarily serving enterprise inference needs by running trained models on endpoint devices or edge computing systems. These applications demand low latency and real-time responsiveness, with significantly lower computational requirements than AI training. Key use cases include:

1. Smart security: real-time image analysis for detecting suspicious behavior or individuals
2. Smart home: rapid response to user commands and real-time environmental adjustments
3. Smart transportation: traffic monitoring, autonomous driving, and intersection surveillance
4. Drones – real-time image analysis for navigation and filming
5. Healthcare monitoring – real-time data processing to assess user health conditions
6. Industrial IoT – data collection and analysis to ensure smooth production operations

As concerns over data privacy and security grow, more businesses and institutions are shifting inference processing to local devices to prevent sensitive data leakage. This trend is especially prominent in industries requiring strict privacy protection, such as finance, healthcare, and government.

NVIDIA remains the leading supplier, while AMD continues to gain traction with advancements in its products. Competitors offering high-efficiency, low-power edge inference solutions—tailored to smart home, security, and industrial IoT applications—could challenge NVIDIA and AMD in this evolving market.

Conclusion

The AI chip market is undergoing rapid transformation, with different market segments showing varying growth trends and challenges, as shown in Table 1.

Table 1   AI chip market structure analysis and comparison

Source: Researcher and Research

The growth of the edge market and its divergence from the cloud market compel us to consider the potential impact of emerging competitors. Among these, DeepSeek’s technological innovations are particularly noteworthy, as they have the potential to disrupt the current market landscape. Although large companies like NVIDIA and Google currently dominate the market, DeepSeek’s rise—whether in hardware acceleration or breakthroughs in AI training—could significantly alter this dynamic.

Despite the ongoing emergence of Cloud Service Providers (CSPs) developing their own ASICs, ICs, and other competitors, we believe that NVIDIA continues to maintain a strong market position and technological advantage. The primary reasons for this are:

1. GPU Advantage

NVIDIA has long dominated the GPU market, with accelerators like the A100 and H100 becoming industry standards for AI training and inference. NVIDIA’s GPUs not only support training but also handle large-scale inference, playing a critical role in many AI applications. As a result, even with the competition from CSPs developing their own ASICs and ICs, NVIDIA maintains a strong advantage in applications requiring general-purpose and high-performance computing.

2. Robust Software Ecosystem

NVIDIA boasts a comprehensive developer ecosystem, including tools like CUDA, cuDNN, and TensorRT, making it easy for developers to build AI applications on NVIDIA hardware. In contrast, competitors like CSPs with in-house ASICs and DeepSeek need to invest significant time and resources in developing their own software ecosystems, giving NVIDIA a clear edge.

3. Efficient AI Computing Platform

NVIDIA’s high-performance computing (HPC) and AI platforms offer highly optimized hardware and software integration, providing powerful acceleration for a variety of AI workloads, such as natural language processing and image recognition. The optimization of these platforms gives NVIDIA a performance advantage in processing large datasets and models, surpassing other competitors.

However, NVIDIA also faces significant challenges, primarily from two forces in the specialized competition space:

1. Development of CSP-Developed Chips

In an effort to reduce reliance on third-party chip suppliers and achieve cost control and hardware customization, many CSPs have opted to develop their own chips. For example, Google’s TPU focuses on neural network inference, while Amazon’s Inferentia is optimized for inference scenarios. These in-house chips offer more efficient, cost-competitive solutions for specific applications.

2. Breakthroughs in Specialized ASICs

Some competitors may pose a threat to NVIDIA by achieving breakthroughs in specific areas, such as low-latency inference or other specialized acceleration needs, and developing highly specialized ASICs. Especially in cost-sensitive markets or those with niche requirements, NVIDIA’s high-end GPUs (such as the A100 and H100) may not be as attractive as these specialized ASIC solutions due to their higher price points.

Therefore, NVIDIA currently maintains a significant competitive advantage in the general AI training and inference market. However, if CSPs aggressively develop in-house ASICs or competitors make breakthroughs in specialized areas, NVIDIA will face increased competitive pressure. Future competition will depend on whether these challengers can surpass NVIDIA products in terms of performance, efficiency, price, and ecosystem support.

In conclusion, when discussing the AI chip market, the training and inference needs in both cloud and edge markets each present different challenges and opportunities. The development of these four areas is interwoven, and the characteristics of each can influence the future direction of the market. As AI technology evolves, businesses that can provide higher-performance, cost-effective solutions in these areas will not only effectively address current market challenges but also capture growth opportunities in the future. Such solutions have the potential to challenge the current market leaders and carve out a strong position in these rapidly developing sectors.

This article is part of our Future Scenarios and Design series.
It explores how possible futures take shape through trend analysis, strategic foresight, and scenario thinking, including shifts in technology, consumption, infrastructure, and business models.

See more in this category, or explore more notes here.

AI chip market evolution: Cloud vs. edge in training and inference

Part 1: Cloud training and inference

The AI chip market is undergoing significant transformations, which can be understood through two key dimensions: deployment environments (cloud vs. edge) and market segments (training vs. inference).

Cloud-based training currently dominates the market and is expected to maintain strong growth in the future. Training is critical for AI model development, requiring immense computational power to process vast amounts of data, which is why it is primarily concentrated in cloud data centers. NVIDIA is the dominant player in this space, but competition from AWS, Google TPU, and Meta could reshape the market landscape in the coming years.

Alongside training, the cloud inference market is also experiencing rapid growth. Inference refers to applying trained AI models to real-world scenarios and making decisions based on real-time data. The increasing demand for inference is driven by advancements in AI models and emerging application scenarios. While inference requires lower computational power than training, it demands high accuracy and low latency, making cloud-based inference expansion essential for scaling AI applications.

On the edge, both edge training and edge inference cater to demands for low latency and data privacy. Compared to cloud-based training, edge training remains a smaller market but is expected to grow with the rise of smart devices and autonomous vehicles. Meanwhile, edge inference focuses on real-time, on-device processing, which is crucial for applications such as autonomous driving and other latency-sensitive use cases. While the edge market remains a niche segment, its importance will increase as these specialized applications continue to evolve.

In conclusion, training and inference in both cloud and edge environments present unique challenges and opportunities. Companies capable of delivering high-performance, cost-effective solutions in these areas will have the potential to challenge existing market leaders.

This article, “AI Chip Market Evolution: Cloud vs. Edge in Training and Inference,” is structured into two parts to provide a comprehensive analysis of the market’s diverse landscape and future trends.

Part 1: Cloud Market
This section explores Cloud Training and Cloud Inference, examining their growth potential, key industry players, and evolving competitive dynamics.

Part 2: Edge Market
This section delves into Edge Training and Edge Inference, focusing on low latency, data privacy, and emerging applications, while assessing their technological advancements and market opportunities.

By adopting this structured approach, the article offers a clearer understanding of the AI chip market’s distinct segments, delivering valuable insights to industry stakeholders for strategic planning and competition.

Our Analysis

When examining the future of the AI industry, understanding the market structure is essential. The AI landscape can be broadly categorized into four key markets: Cloud Training, Cloud Inference, Edge Training, and Edge Inference. While each of these markets operates with distinct mechanisms and applications, they are deeply interconnected, forming the foundation of AI technology.

• Cloud Training: The Core of AI Development
  Cloud training serves as the backbone of AI advancement, responsible for large-scale data processing and model training. As data volumes continue to grow and computational demands rise, cloud training remains the cornerstone of AI’s rapid technological progress.
• Cloud Inference: Enabling Real-Time AI Applications
  Once AI models are trained, cloud inference plays a crucial role in applying them to real-world scenarios. Unlike training, inference focuses on enabling AI to deliver rapid, real-time responses. The evolution of cloud inference is tightly linked to cloud training, and as demand for AI-powered solutions expands, the cloud inference market continues to grow.
• Edge Training & Edge Inference: Addressing Privacy and Latency Needs
  The edge AI market—comprising edge training and edge inference—is gaining momentum, particularly in applications requiring high data privacy and low latency. By shifting training and inference processes from the cloud to local devices, edge AI enhances security and reduces response times. In certain use cases, edge solutions offer distinct advantages over cloud-based alternatives.

These four market segments not only highlight how AI technologies are applied across industries today but also signal the future direction of AI development. Despite their unique operational models, their interdependencies will continue to drive AI innovation and industry-wide progress.

This article will provide an in-depth exploration of the cloud sector within the AI chip market, focusing specifically on cloud training and cloud inference markets. As artificial intelligence technologies advance, these two markets are rapidly becoming key growth areas. Given the critical role of training and inference in the AI model lifecycle, both will significantly influence the structure of the market and future competitive dynamics. In this article, we will analyze the current state and future growth potential of the cloud training market, along with the competitive landscape of key players. Additionally, we will explore the development trends of cloud inference and its impact on emerging application scenarios. This section will lay the foundation for the subsequent discussion on the edge market and provide a comprehensive understanding of the current landscape and challenges within the AI chip market.

Among these four markets, the cloud training market forms the foundation of AI development, as large-scale data processing and model training require substantial computational power, which cloud platforms provide. With the advancement of AI technologies and the explosion of data, the cloud training market has become the core driver of AI technology growth. In the cloud training market, the deep learning process of AI models demands vast computational resources, typically relying on the infrastructure provided by large cloud service providers. The successful completion of this process sets the stage for subsequent inference and applications, fueling the rapid growth of the cloud inference market.

1. Cloud Training

The cloud training market holds the largest share of the AI chip market, estimated at 50%-70%. The demand for training large AI models such as GPT-4, Gemini, and Claude allows NVIDIA to maintain its market leadership. The H100’s competitive edge lies in its CUDA ecosystem and TensorRT, while Google TPU leverages its massive internal training clusters and networking technologies. Additionally, companies like AWS with Trainium, Meta, and Anthropic are developing their own training ASICs. While these companies are unlikely to challenge NVIDIA’s market dominance in the short term, they could change the market landscape in the long term.

The market will continue to grow, but the growth rate may slow as AI training remains expensive, and companies are looking for ways to reduce costs. These include techniques such as sparsity, mixed-precision training, more efficient AI acceleration chips, and the reusability of pre-trained models (with improvements in fine-tuning technology).

However, even if technological optimizations reduce the cost of AI training, key considerations remain:

1. companies still need high-performance AI training chips,
2. generative AI use cases are rapidly expanding, and
3. advancements in training technologies and changes in market demand remain uncertain.

Therefore, the training market led by NVIDIA and Google TPU will continue to experience strong demand in the short term.

If DeepSeek can provide innovative AI acceleration technologies and efficient training solutions in the cloud training market, it could challenge the established leaders, such as NVIDIA and Google TPU. Cloud training relies heavily on high-performance chips and energy-efficient technologies to handle large-scale datasets and models. If DeepSeek can offer low-cost, high-performance AI training solutions, particularly by making breakthroughs in sparsity techniques and mixed-precision training, it has the potential to disrupt the existing market structure.

After discussing the development of the cloud training market, we now turn to explore the cloud inference market. Although these two markets may seem distinct, they are actually closely related. Cloud training involves large-scale computations based on vast datasets, while cloud inference applies the trained models to real-world business scenarios. Therefore, the development of both markets is interdependent.

2. Cloud Inference

The rapidly growing cloud inference market has become the second-largest segment of the AI chip market, accounting for approximately 15-25% of the market share. This includes applications such as chatbots, speech recognition, recommendation systems, computer vision (e.g., autonomous driving perception systems), financial risk assessment, medical diagnostics, and industrial inspection.

Currently, the market is dominated by NVIDIA, although AMD has started gaining some market share through its partnerships with Meta and Microsoft. Meanwhile, Intel is maintaining its competitive edge in the inference market with its Habana Gaudi AI chips and Xeon CPUs. Tenstorrent is actively developing its own AI accelerator, attempting to challenge NVIDIA’s leadership position. Qualcomm is focusing on low-power AI inference applications, launching the Cloud AI 100 accelerator, while Mythic explores analog computing technologies, which may further impact the AI inference market in the future. Cerebras and Groq are also competing by renting out inference compute power through self-built small cloud service providers (CSPs), adding more options to the market.

Looking ahead, the cloud inference market is expected to grow significantly, driven by the following key factors:

1. Advances in AI models: This is the most critical factor. As more advanced AI models (such as OpenAI’s GPT-5 and DeepSeek’s R1) are released, there will be a significant increase in the demand for inference computing, which will, in turn, drive the overall market development.
2. Emerging application scenarios: Applications like autonomous driving, financial risk assessment, and medical diagnostics will greatly drive the demand for inference computing. These applications not only expand the scope of AI use but also increase the demand for efficient inference, further stimulating market growth.
3. Reduction in inference costs: As inference costs decrease, more companies will enter the market and apply AI technologies in more fields, which will be a long-term driver of market growth.
4. Innovation and improvement in hardware infrastructure: As specialized hardware for AI inference (such as AI ASICs and FPGAs) is developed, inference efficiency will greatly improve, further reducing costs and enhancing performance.
5. Stricter data privacy and security requirements: As AI is applied in sensitive fields like finance and healthcare, increasing requirements for data privacy and security will drive demand for more efficient and regulation-compliant inference technologies. This could also promote the development of related technologies, such as encrypted inference and edge computing.
6. Enhancement of traditional software by AI: AI’s enhancement of traditional software can significantly improve the performance of existing systems, providing more efficient tools for traditional businesses and possibly giving rise to new business models. Although the impact of this factor is indirect, it plays a stabilizing role in driving long-term market growth.

As AI technology continues to evolve, the demand for more efficient and advanced inference computing will intensify, leading to a surge in demand for inference models. We expect that the computational demand for inference models could exceed the current demand for large language models (LLMs) by more than ten times. This change in demand will drive CSPs to accelerate the development of self-designed ASIC chips to improve inference performance and reduce reliance on third-party hardware (like NVIDIA). For example, AWS aims for 50% of its chips to be self-designed ASICs, Meta plans 70%, and Microsoft could reach up to 80%.

Moreover, as the demand for efficient inference computing increases in AI applications like autonomous driving and medical diagnostics, DeepSeek has the potential to become a strong competitor by introducing accelerators or software solutions that surpass existing technologies.

This would have a significant impact on NVIDIA, which currently dominates the market. As Google’s TPUs, AWS’s Inferentia, and other specialized ASIC technologies gradually increase their market share, more competitors will join the market, making the competition increasingly fierce.

After understanding the market dynamics of cloud inference, the next piece will focus on the development of the edge market, another field that complements cloud inference. While edge training and inference are independent of the cloud market, they offer more efficient and low-latency solutions in many situations, especially in industries with high data privacy requirements, where edge solutions hold unmatched advantages.

This article is part of our Future Scenarios and Design series.
It explores how possible futures take shape through trend analysis, strategic foresight, and scenario thinking, including shifts in technology, consumption, infrastructure, and business models.

See more in this category, or explore more notes here.