GPUs are the most important hardware in artificial intelligence. But when AI models grow huge with trillions of parameters, training these giant models requires thousands of GPUs working together, and their interconnection must allow constant sharing of huge amounts of data at very high speeds.
Today, Nvidia uses special high-speed copper cables – InfiniBand DAC cables – to connect GPU servers. These thick cables work well inside one rack or between nearby racks which are usually less than 3 meters, allowing fast bandwidth and low-latency data transfer among GPUs during model training. However, copper has significant problems when the distance increases or the speed goes higher.
Copper wires generate heat because of electrical resistance, and the longer the cable, the weaker the signal becomes. At a high speed of 500 Gigabits per second, copper cables can only work well under one meter. To go further, extra amplifiers are needed, but this complicates the setup and makes cooling harder.
In early March, Nvidia announced major investments: US$2 billion (HK$15.6 billion) each in two optical companies, Lumentum and Coherent. Nvidia’s chief executive Jensen Huang said the next generation of AI infrastructure will use light instead of electricity to connect GPUs.
The physics behind this is simple: optical fiber uses photons instead of electrons for data transfer. Since photons do not generate heat like electrons do, they travel at the speed of light and lose almost no energy even over long distances. One fiber can also carry much more data by using different wavelengths – or colors – of light at the same time, which solves the power and distance problems of copper.
As of today, optical technology is still expensive because of the special transceivers required at both ends. But since the raw glass fiber itself is cheaper than copper, the economics will make sense at larger production scales. Nvidia plans to use Co-Packaged Optics starting in 2026, which will place the light engines right next to the chips.
For short distances inside a rack, copper will still be used for a few more years before light finally takes over. In the future, optical connections will make AI training faster, cheaper to run, and much greener.
Allen Au is a tech startup founder, AI architect, and YouTuber
