Inside Intel’s delays in delivering a crucial new microprocessor

Last May, Sandra Rivera, a senior executive at chip giant Intel, received some alarming news.

Engineers had worked for more than five years to develop a powerful new microprocessor to perform data center computing tasks and were confident that they had finally found the right product. But signs of a potentially serious technical fault surfaced during a regular morning meeting to discuss the project.

The problem was so troublesome that Sapphire Rapids, the codename for the microprocessor, had to be delayed – the latest in a series of setbacks for one of Intel’s most important products in years.

“We were pretty discouraged,” said Ms. Rivera, executive vice president of Intel’s data center and artificial intelligence group. “It was a painful decision.”

The Sapphire Rapids launch ended up being pushed back from mid-2022 to Tuesday, nearly two years later than originally planned. The lengthy development of the product — which combines four chips in a single package — underscores some of the challenges facing a turnaround effort at Intel as the United States tries to assert dominance in core computing technology.

Since the 1970s, Intel has been a big player in the small silicon wafers that run most electronic devices, best known for a variety called microprocessors, which act as the electronic brains in most computers. But the Silicon Valley company has lost its long-standing lead in manufacturing technology in recent years, which helps determine how fast the chips can compute.

Patrick Gelsinger, who became Intel’s chief executive in 2021, has pledged to restore its manufacturing edge and build new factories in the United States. He was a leading figure when Congress debated and passed legislation last summer aimed at reducing US reliance on chipmaking in Taiwan, which China claims as its territory.

Sapphire Rapids’ bumpy development has implications for Intel’s ability to bounce back to deliver future chips on time. It’s an issue that could affect dozens of computer makers and cloud service providers, not to mention the millions of consumers who use online services likely to be powered by Intel technology.

“What we want is a stable and predictable cadence,” said Kirk Skaugen, executive vice president responsible for server sales at Lenovo, a Chinese company that plans 25 new systems based on the new processor. “Sapphire Rapids is the start of a journey.”

For Intel, the pressure is there. Along with declining demand for chips used in personal computers, the company faces fierce competition in server chips which are its most profitable business. That question has worried Wall Street, with Intel’s market value having fallen more than $120 billion since Mr. Gelsinger took office.

Intel plans to hold an online event on Tuesday to discuss Sapphire Rapids, which is named after part of the Colorado River. More formally, the product is called the 4th Generation Intel Xeon Scalable Processor.

In an interview, Mr. Gelsinger said Sapphire Rapids had the makings of a hit, despite the delays. He chose Ms. Rivera in 2021 to take charge of the development unit, where she uses lessons from experience to change the way Intel designs and tests its products. He said Intel had conducted several internal reviews of what happened with Sapphire Rapids, and “we’re not done.”

Sapphire Rapids began in 2015, with discussions between a small group of Intel engineers. The product was the company’s first attempt at a new approach in chip design. Companies now routinely pack tens of billions of tiny transistors onto each piece of silicon, but competitors like Advanced Micro Devices and others had begun making processors from multiple chips bundled together in plastic packages.

Intel engineers came up with a design with four chips, each with 15 processor “cores” that act as individual calculators for general-purpose computing work. The company also decided to include additional circuit blocks for special tasks – including artificial intelligence and encryption – and to communicate with other components, such as chips that store data.

The interaction between so many elements is “very complex,” said Shlomit Weiss, who co-leads Intel’s design engineering group. “Complexity usually brings problems.”

The Sapphire Rapids team has struggled with bugs, flaws caused by design errors, or manufacturing issues that can cause a chip to miscalculate, run slowly, or stop working. They were also affected by delays in the product manufacturing process.

But in December 2019, the engineers had taken a step called “tape-in”. This is when electronic files containing a completed design are transferred to a factory to manufacture sample chips.

Chip samples arrived in early 2020 as Covid-19 forced lockdowns. Engineers quickly got Sapphire Rapids’ compute cores to communicate with each other, said project lead engineer Nevine Nassif. But there was still more work than expected.

One of the main tasks was “validation”, a testing process in which Intel and its customers run software on sample chips to simulate computing tasks and detect bugs. Once defects are found and fixed, designs can return to the factory to make new test chips, which usually takes over a month.

Repeating this process led to missed deadlines. Ms Nassif said Sapphire Rapids was designed to counter AMD’s Milan processor, which was introduced in March 2021. But it still wasn’t ready in June, when Intel announced a delay until the year next to allow more validation.

That’s when Ms. Rivera stepped in. The longtime Intel executive had successfully built a business in networking products before being appointed in 2019 as chief human resources officer.

“We had to get our execution mojo back,” Gelsinger said. “I needed someone who was going to run into the fire and fix this thing for me.”

In October 2021, Ms. Rivera and a senior design official established weekly meetings on the state of Sapphire Rapids, held every Monday at 7 a.m. These rallies showed steady progress in finding and fixing bugs, she said, boosting confidence that production will start in the second quarter of 2022.

Then came the discovery of the fault last May. Ms. Rivera wouldn’t describe it in detail, but said it affected CPU performance. In June, it used an investor event to announce a delay of at least a quarter, which pushed Sapphire Rapids later than the launch of a competing AMD chip in November.

“We were ready to ship,” Ms. Nassif said. The final delay “was so sad considering all the effort that had gone into it”.

Ms Rivera has learned a series of lessons from setbacks. The first was simply that Intel packed too much innovation into Sapphire Rapids, rather than delivering a less ambitious product sooner.

She also concluded that the team should have spent more time refining and testing their design using computer simulations. Finding bugs before they are in chip samples is cheaper and would have removed features to simplify the product, Ms. Rivera said. She has since moved to bolster Intel’s simulation and validation capabilities.

“We used to have a lot of this kind of muscle that we let atrophy,” Ms. Rivera said. “Now we are rebuilding.”

She also determined that Intel planned more products than its engineers and customers could easily handle. So it streamlined that product roadmap, including pushing back a Sapphire Rapids successor to 2024 from 2023.

More broadly, Ms. Rivera and other Intel leaders have pushed the organization to develop better processes for documenting technical issues and sharing that information inside and outside the company.

Some Intel customers say communication has improved.

“Did everything go well?” No,” said Lenovo’s Skaugen, who once ran Intel’s server chip business. “But we were much less surprised than in the past.”