
A big reason why AI, particularly GenAI, became possible was the advances in semiconductor technology – chips that could process vast amounts of data very quickly. Now, as AI advances, as more solutions emerge, and more people start using them, the need for upgrading data centre and edge infrastructure to meet the needs of AI has become urgent.
Sanjay Gajendra, COO & co-founder of semiconductor company Astera Labs, says announcements by hyperscalers like Amazon, Microsoft, Google, and Meta indicate that there will be capital investment of $300 billion to upgrade their infrastructure next year. That’s about 25% higher than this year, and double of what it was in 2023. “People call it the arms race to deploy AI at scale,” he says.

As that upgrade happens, a vital element of that will also be the upgrade of the connectivity elements inside a server, and between servers. Gajendra says various reports show only 51-52% of a GPU (the main AI processor) is utilised currently – “meaning, about 48% of the time, the GPUs are simply sitting idle waiting for data or memory, because the connectivity infrastructure is not fast or efficient enough.”
People are trying to solve this problem in different ways. Shivananda Koteshwar, MD and head of Astera’s newly established India site, says it’s a hugely complex problem. Traditionally, servers had maybe two CPUs. Now, the AI server motherboard, he says, has multiple GPUs, CPUs, NIC (network interface controller) cards, accelerator cards. “For this, you need strong connectivity solutions that leverage all the different protocols required for the connectivity, whether it’s PCI, Ethernet, CXL (compute express link), and solutions that understand these protocols very well. And since all these components are talking to each other (at very high speeds and therefore more potential for things to go wrong), the interconnect solutions need to be very reliable, very efficient, very low power, and ensure there is high throughput and low latency,” he says.

But a single server or a small cluster of servers is pointless in AI. Gajendra notes that AI training and inference workloads require giant clusters with 100,000-500,000 GPUs all running together. Connecting multiple clusters to create such giant clusters uses protocols like Ethernet and InfiniBand.
Astera has created precisely such connectivity solutions, solutions that address the data, networking and memory bottlenecks that are limiting the efficiency of these AI infrastructure today. “Astera is fortunate to be in the heart of all the major AI infrastructure that’s being built out by hyperscalers all around the world,” says Gajendra, who co-founded the company with two other semiconductor professionals Jitendra Mohan and Casey Morrison in 2017. They did an IPO earlier this year; the share has surged in the past few months.
Silicon photonics
Other efforts too are on to make AI connectivity more efficient. Jitendra Chaddah, head of India for GlobalFoundries, one of the world’s leading semiconductor manufacturers, says his company is working with customers to use silicon photonics for faster connectivity. Silicon photonics combines electronics and photonics (light), and allows optical and electrical components to be integrated on a single chip. Optical signals can carry much more data compared to electrical signals, and these interconnects consume less power than electrical ones.

Chaddah says AI at the edge will make edge connectivity also critical in the coming times.