You can have it fast, cheap, or secure — pick any two.
It seems to be possible as long as “secure” isn’t one of your choices.
“Our IT industry is inexorably international, and anyone involved in the process can subvert the security of the end product,” Schneier wrote.
We don’t often hear about intentional efforts to subvert the security of the technology supply chain simply because these incidents tend to get quickly classified by the military when they are discovered.
Indeed, noted security expert Bruce Schneier calls supply-chain security “an insurmountably hard problem.”
Most of the U.S. government’s efforts to police the global technology supply chain seem to be focused on preventing counterfeits — not finding secretly added spying components.
Finally, it’s not clear that private industry is up to the job, either. At least not yet.
Supply chain challenges definitely fit into category “things I can’t change“.
109 Comments
Tomi Engdahl says:
Computer Networks Are Now Permanently Hackable. Have Fun With That.
https://www.bloomberg.com/view/articles/2018-10-04/the-big-hack-global-supply-village-is-stuck-with-trapdoors?utm_source=facebook&utm_campaign=socialflow-organic&utm_medium=social&cmpid=socialflow-facebook-business&utm_content=business
The web of parts makers, assemblers, testers and contractors is almost impossible to untangle.
Tomi Engdahl says:
China’s Big Hack Has Big Implications
https://www.bloomberg.com/view/articles/2018-10-04/china-s-big-hack-has-big-implications
It ratchets up international tensions and exposes the global supply chain’s vulnerability.
Tomi Engdahl says:
Bloomberg Reports China Infiltrated the Supermicro Supply Chain We Investigate
https://www.servethehome.com/bloomberg-reports-china-infiltrated-the-supermicro-supply-chain-we-investigate/
Bloomberg today came out with an industry shocker. The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies. In that article, Bloomberg reports that the PLA managed to infiltrate Supermicro’s supply chain and add small chips that allowed Chinese agencies to hack into 30 companies such as Apple and Amazon.
What is this hack?
According to Bloomberg, the hack involves a small IC inserted into the Supermicro motherboard PCB. In previous generations, this would have been a surface mount component. The story claims current generations have these devices embedded in PCB.
There, of course, has to be much more than a simple chip. That chip needs to tap into electrical signals both for power and for data transfer. That means that not only must a component be inserted, but also PCB wires. Bloomberg says it is in line with memory to CPUs to intercept some password validation code. By changing this code in Linux, it allows remote attackers to access the servers and potentially phone home.
That is a little strange frankly from a technical standpoint. Where could these chips be located?
DRAM memory traces are very complex.
The other candidates are more probable. The first is using the onboard SATADOM wires. SATADOMs are small flash memory devices used to load base operating systems. SATA cables are 7-pin designs with three ground wires and two A/B +/- pairs. Supermicro SATADOM connectors have an extra power capability.
The final, and perhaps most likely vector would be the BMC. We have a piece Explaining the Baseboard Management Controller or BMC in Servers. A hardware chip that could impact the BMC firmware is more probable.
Each BMC has local storage ever since the 1998 IPMI 1.0 spec was announced.
The BMC has root console access to the server. It is on before the server boots. It can mount media and has network access.
BMCs are amazingly hacked devices. The Bloomberg story’s comments from Amazon and Apple both point to the BMC and IPMI firmware/ management interfaces. We think this is the most likely vector.
The bad news is that BMC’s are extremely dangerous. They are also pervasive with a few points under 100% of servers having them these days.
The security community, as a whole, knows that BMCs are both useful if not mandatory in today’s infrastructure.
Bottom line, if this Supermicro attack vector is to the BMC, then the Bloomberg story is no bigger than the Dell EMC PowerEdge iDRACula story or any others. Saying there is a vulnerability in a BMC is like saying the sun is hot.
STH
Home Server Other Components
ServerOther Components
Bloomberg Reports China Infiltrated the Supermicro Supply Chain We Investigate
By Patrick Kennedy – October 4, 2018 31
Supermicro SBI-7128RG-X
Supermicro SBI-7128RG-X
Bloomberg today came out with an industry shocker. The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies. In that article, Bloomberg reports that the PLA managed to infiltrate Supermicro’s supply chain and add small chips that allowed Chinese agencies to hack into 30 companies such as Apple and Amazon. The company also published, in a different article, statements from Amazon, Apple, and Supermicro strongly rebutting the story. See The Big Hack: Statements From Amazon, Apple, Supermicro, and the Chinese Government. Something is certainly strange here, and at STH, we review more server platforms than anywhere else on the Internet, including those from Supermicro. We also, by chance, started diving into the BMC security space more recently so it is clearly time to investigate.
What is this hack?
According to Bloomberg, the hack involves a small IC inserted into the Supermicro motherboard PCB. In previous generations, this would have been a surface mount component. The story claims current generations have these devices embedded in PCB.
There, of course, has to be much more than a simple chip. That chip needs to tap into electrical signals both for power and for data transfer. That means that not only must a component be inserted, but also PCB wires. Bloomberg says it is in line with memory to CPUs to intercept some password validation code. By changing this code in Linux, it allows remote attackers to access the servers and potentially phone home.
That is a little strange frankly from a technical standpoint. Where could these chips be located?
DRAM memory traces are very complex. 288 pins per DDR4 DRAM module (not all are data of course), times 8 modules per CPU times two CPUs and that is a lot of pins to monitor from such a small IC. Even in the older 240-pin DDR3 generation, with 16 modules there is no way a small IC can monitor that many wires. Also, memory traces in motherboards are often an area where PCB designers spend a lot of time to get correct lengths and timings on the wires. Inserting a small IC would not be the easiest feat there.
The other candidates are more probable. The first is using the onboard SATADOM wires. SATADOMs are small flash memory devices used to load base operating systems. SATA cables are 7-pin designs with three ground wires and two A/B +/- pairs. Supermicro SATADOM connectors have an extra power capability.
Supermicro X11SDV 4C TLN2F SATA And Oculink
Supermicro X11SDV 4C TLN2F SATA And Oculink
This would be a lower pin count option to exploit. The problem, of course, is that most large shops encrypt data on the SATADOMs. Most SATADOMs do not have self-encrypting capabilities which means it is host encryption. The Bloomberg article said that the hardware would intercept storage to CPU transfers. If the data is encrypted when transferred, it would be nearly impossible in that IC footprint to crack reasonable encryption and change the OS in-line.
The final, and perhaps most likely vector would be the BMC. We have a piece Explaining the Baseboard Management Controller or BMC in Servers. A hardware chip that could impact the BMC firmware is more probable.
ASRock EPC612D8A-TB ASPEED 2400 BMC
ASRock EPC612D8A-TB ASPEED 2400 BMC
Each BMC has local storage ever since the 1998 IPMI 1.0 spec was announced.
Intel IPMI V1 September 1998 BMC
Intel IPMI V1 September 1998 BMC
This is generally a very small flash module for storage, often a few MB in size. The BMC usually runs a flavor of Linux. Getting root access to the BMC is bad, but it is not the same as getting full access to the main server OS.
ASPEED AST2500 Diagram
ASPEED AST2500 Diagram
The BMC has root console access to the server. It is on before the server boots. It can mount media and has network access. Think of it as an administrator sitting at the machine, but bringing that functionality anywhere in the world.
BMCs are amazingly hacked devices. The Bloomberg story’s comments from Amazon and Apple both point to the BMC and IPMI firmware/ management interfaces. We think this is the most likely vector.
The bad news is that BMC’s are extremely dangerous. They are also pervasive with a few points under 100% of servers having them these days. The Bloomberg article cites the well-known Supermicro BMC/ IPMI vulnerabilities. Supermicro is not alone. Every Dell EMC PowerEdge server (edit: 13th generation and older, the new 14th generation has a fix to prevent this) has a local and remote exploit available that the company can mitigate with patches, but cannot fix. We broke this story with iDRACula. If you think you are safe with HPE or Lenovo servers, here are BMC vulnerabilities for other vendors.
The security community, as a whole, knows that BMCs are both useful if not mandatory in today’s infrastructure. As a result, the security community, and major hyper-scale vendors are putting a lot of effort in researching security solutions.
One of the more interesting bits is that if it is a BMC vulnerability or anything that “phones home” over a network interface, one would expect that security researchers would have seen it. There are companies that put boxes on networks just to see what network traffic they create. Supermicro tends to build common designs that it ships to multiple customers. It would be slightly interesting if only some Supermicro servers, e.g. for certain customers were impacted. If China did not do this, it would have been caught earlier. If China did limit to a few customers, it would be difficult to target them at PCB. As we will show shortly, Supermicro PCBs are used across products.
Bottom line, if this Supermicro attack vector is to the BMC, then the Bloomberg story is no bigger than the Dell EMC PowerEdge iDRACula story or any others. Saying there is a vulnerability in a BMC is like saying the sun is hot.
Some higher-resolution areas of MicroBlade BMCs
We had some similar generation Supermicro MicroBlades where we could provide higher-resolution photos of their BMC areas. This is where the hacked chips are located on the board that Bloomberg depicts. This also shows that a Supermicro PCB is spun for multiple products. That makes it extremely difficult to target a specific customer at the time of PCB construction. Here we have two different products built on the same underlying PCB.
For our less technical readers, this is what the actual PCB looks like. For our more technical readers, you may want to see for yourself.
Here are two MicroBlades of that era the Supermicro B1SD1-TF and the B1SD2-TF. The “2” represents that the PCB houses two complete server nodes. We highlight this because if the attack is present on this platform, presumably it would require a second inserted chip which would not be required on the B1SD1-TF.
There are a ton of ICs there. I know we have STH readers who will want to look. Have at it.
The Counterpoint Published Outside of the Main Story
Bloomberg posts statements from companies, not in their main article, but linked in a separate article.
Amazon, Apple, and Supermicro all deny that this is happening.
Just for a taste, here is an excerpt from Apple’s statement:
“We are deeply disappointed that in their dealings with us, Bloomberg’s reporters have not been open to the possibility that they or their sources might be wrong or misinformed. Our best guess is that they are confusing their story with a previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs. That one-time event was determined to be accidental and not a targeted attack against Apple.” (Source: Bloomberg/ Apple)
This is a little strange. All three are public companies. A simple “no comment” would have sufficed. Or a “we would not be allowed to comment on your classified source story” perhaps. Supermicro one can dismiss their lack of knowledge to perhaps the intelligence community not wanting to alert anyone there. Apple and Amazon went beyond a simple “no recollection” or “no comment” type response. They should not be allowed to make these types of responses if they are untrue since they would be potentially misleading investors. Even if they could not speak about the issues, they did not have to go into the depth that they did.
Indeed, when we broke iDRACula the persistent (with mitigations) non-fixable Dell EMC PowerEdge issue impaction tens of millions of their servers, we held the entire story while Dell EMC’s confirmation went through legal and management approvals. Having just broken a similar story, the responses from parties are in an absolutely sharp contrast.
Where the Bloomberg Piece Makes No Sense
There is one area where the Bloomberg piece makes no sense. Supermicro servers are procured for US Military contracts and use to this day. Supermicro’s government business is nowhere near a large as some other vendors, but there are solutions providers who sell Supermicro platforms into highly sensitive government programs.
If the FBI, or other intelligence officials, had reason to believe Supermicro hardware was compromised, then we would expect it would have taken less than a few years for this procurement to stop.
Assuming the Bloomberg story is accurate, that means that the US intelligence community, during a period spanning two administrations, saw a foreign threat and allowed that threat to infiltrate the US military. If the story is untrue, or incorrect on its technical merits, then it would make sense that Supermicro gear is being used by the US military.
First and foremost, I think we need to call for an immediate SEC investigation around anyone who has recently taken short positions or sold shares in Supermicro.
There seems to have been over 20 people that knew about this.
Further, with public companies making statements on the impact, unless there is a valid national security/ classified reason that they gave the responses they did, there is a mismatch. Apple and Amazon did not say “no comment” they called Bloomberg’s account false.
Second, we need a formal investigation into why, if this is a true and serious threat, why it was not flagged in military procurement years ago.
one key takeaway: server security is a big deal. Perhaps the bigger takeaway is that this is a 21st-century battleground that is active every day. Government agencies from China, the US, Russia, Israel, and others all have ways to impact servers and more broadly computing devices.
Tomi Engdahl says:
https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies
Tomi Engdahl says:
U.S. government sides with Apple and Amazon, effectively denying Bloomberg ‘spy chip’ report
https://techcrunch.com/2018/10/07/homeland-security-denies-bloomberg-spy-chip-report/?sr_share=facebook&utm_source=tcfbpage
Homeland Security has said it has “no reason to doubt” statements by Apple, Amazon and Supermicro denying allegations made in a Bloomberg report published earlier this week.
It’s the first statement so far from the U.S. government on the report, casting doubt on the findings. Homeland Security’s statement echos near-identical comments from the U.K.’s National Cyber Security Center.
Bloomberg said, citing more than a dozen sources, that China installed tiny chips on motherboards built by Supermicro, which companies across the U.S. tech industry — including Amazon and Apple — have used to power servers in their datacenters. The chip can reportedly compromise data on the server, allowing China to spy on some of the world’s most wealthy and powerful companies.
Apple, Amazon and Supermicro later published statements on their websites. Bloomberg said it’s sticking by its story. And yet, this latest twist isn’t likely to leave anyone less confused
The reality is that days after this story broke, it seems many of the smartest, technically minded, rational cybersecurity experts still don’t know who to believe — Bloomberg, or everyone else.
And until someone gets their hands on these apparent chips, don’t expect that to change any time soon.
Tomi Engdahl says:
David Brunnstrom / Reuters:
DHS and UK’s NCSC say they have “no reason to doubt” statements from Apple, Amazon, and others rejecting Bloomberg’s story on China spying via microchips — WASHINGTON (Reuters) – The U.S. Department of Homeland Security said on Saturday it currently had no reason to doubt statements …
DHS says no reason to doubt firms’ China hack denials
https://www.reuters.com/article/us-china-cyber-dhs/dhs-says-no-reason-to-doubt-firms-china-hack-denials-idUSKCN1MH00Y
“The Department of Homeland Security is aware of the media reports of a technology supply chain compromise,” DHS said in a statement.
“Like our partners in the UK, the National Cyber Security Centre, at this time we have no reason to doubt the statements from the companies named in the story,” it said.
Tomi Engdahl says:
DHS and GCHQ join Amazon and Apple in denying Bloomberg chip hack story
https://www.zdnet.com/article/dhs-and-gchq-join-amazon-and-apple-in-denying-bloomberg-chip-hack-story/
US and UK officials stand by Amazon and Apple’s statements regarding Bloomberg chip hack story.
Tomi Engdahl says:
Apple tells Congress that it has found no sign of microchip tampering
https://www.theverge.com/2018/10/7/17948924/apple-cybersecurity-microchip-george-stathakopoulos-denial-congress
Apple, Amazon, and Supermicro each released a forceful denial that their systems were tampered with following the publication of a Bloomberg Businessweek report last week, which alleged that Chinese agents introduced microchips into servers manufactured in the country. In a letter to Congressional officials, Apple reiterated its denial, saying that it has found no sign of tampering.
Tomi Engdahl says:
This Tech Would Have Spotted the Secret Chinese Chip in Seconds
https://spectrum.ieee.org/riskfactor/computing/hardware/this-tech-would-have-spotted-the-secret-chinese-chip-in-seconds
University of Florida engineers use X-rays, optical imaging, and AI to spot spy chips in computer systems
According to Bloomberg Businessweek, spies in China managed to insert chips into computer systems that would allow external control of those systems. Specialized servers purchased by Amazon, Apple, and others around 2015 and manufactured in China by San Jose–based Super Micro were reportedly at issue, as may have been systems built for the U.S. military.
The institute’s semiautomated system “could have identified this part in a matter of seconds to minutes,” says Tehranipoor, an IEEE Fellow. The system uses optical scans, microscopy, X-ray tomography, and artificial intelligence to compare a printed circuit board and its chips and components with the intended design.
Tomi Engdahl says:
Apple, Amazon Strongly Refute Server Infiltration Report
https://threatpost.com/apple-amazon-strongly-refute-server-infiltration-report/137950/
An explosive Bloomberg report, if true, would highlight the largest supply chain attack to have been launched against American corporations.
Apple and Amazon are strongly refuting a report claiming that Chinese spies infiltrated third-party motherboards used on servers by U.S. companies.
The Big Hack: Statements From Amazon, Apple, Supermicro, and the Chinese Government
https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack-amazon-apple-supermicro-and-beijing-respond
Tomi Engdahl says:
Malicious Component Found on Server Motherboards Supplied to Numerous Companies
https://hackaday.com/2018/10/04/malicious-component-found-on-server-motherboards-supplied-to-numerous-companies/
How the Hack Works
The attacks work with the small chip being implanted onto the motherboard disguised as signal couplers. It is unclear how the chip gains access to the peripherals such as memory (as reported by Bloomberg) but it is possible it has something to do with accessing the bus. The chip controls some data lines on the motherboard that likely provide an attack vector for the baseboard management controller (BMC).
Hackaday spoke with Joe FitzPatrick (a well known hardware security guru who was quoted in the Bloomberg article). He finds this reported attack as a very believable approach to compromising servers. His take on the BMC is that it’s usually an ARM processor running an ancient version of Linux that has control over the major parts of the server. Any known vulnerability in the BMC would be an attack surface for the custom chip.
Data centers house thousands of individual servers that see no physical interaction from humans once installed. The BMC lets administrators control the servers remotely to reboot malfunctioning equipment among other administrative tasks. If this malicious chip can take control of the BMC, then it can provide remote access to whomever installed the chip. Reported investigations have revealed the hack in action with brief check-in communications from these chips though it’s difficult to say if they had already served their purpose or were being saved for a future date.
What Now?
Adding hardware to a design is fundamentally different than software-based hacking: it leaves physical evidence behind. Bloomberg reports on US government efforts to investigate the supply chain attached to these parts. It is worth noting though that the article doesn’t include any named sources while pointing the finger at China’s People’s Liberation Army.
The solution is not a simple one if servers with this malicious chip were already out in the field. Even if you know a motherboard has the additional component, finding it is not easy. Bloomberg also has unconfirmed reports that the next-generation of this attack places the malicious component between layers of the circuit board. If true, an x-ray would be required to spot the additional part.
Tomi Engdahl says:
Apple sends letter to Congress about alleged China spy chip
https://www.cultofmac.com/581542/apple-sends-letter-to-congress-about-alleged-china-spy-chip/
“Apple’s proprietary security tools are continuously scanning for precisely this kind of outbound traffic, as it indicates the existence of malware or other malicious activity. Nothing was ever found,” Apple wrote in its letter to Congress. The letter was signed by Apple’s Vice President for Information Security George Stathakopoulos.
Stathakopoulos said that it has never found any of the vulnerabilities mentioned in the article, or been contacted by the FBI as a result of such concerns.
“Over the course of the past year, Bloomberg has contacted us multiple times with claims, sometimes vague and sometimes elaborate, of an alleged security incident at Apple. Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them.
Tomi Engdahl says:
We need new TLA:
SPM – Spying Platform Module
Tomi Engdahl says:
UK, US Security Agencies Deny Investigating Chinese Spy Chips
https://www.securityweek.com/uk-us-security-agencies-deny-investigating-chinese-spy-chips
The U.S. Department of Homeland Security (DHS) and the U.K. National Cyber Security Centre (NCSC) have denied investigating the presence of Chinese spy chips in Supermicro servers, as claimed by a bombshell report published last week by Bloomberg.
According to Bloomberg, the Chinese government planted tiny chips in Supermicro motherboards in an effort to spy on more than 30 organizations in the United States, including government agencies and tech giants such as Apple and Amazon.
The report, on which Bloomberg reporters have been working for the past year using information from 17 sources, claims that Chinese agents masquerading as government or Super Micro employees pressured or bribed managers at the Chinese factories where the motherboards are built. Once the chips were planted, they would allow attackers to remotely access the compromised devices.DHS and NCSC respond to reports on Chinese spy chips
Apple and Amazon allegedly discovered the malicious hardware implants and contacted the FBI.
Tomi Engdahl says:
China Tech Stocks Lenovo, ZTE Tumble After Chip Hack Report
https://www.securityweek.com/china-tech-stocks-lenovo-zte-tumble-after-chip-hack-report
Chinese tech stocks Lenovo Group and ZTE Corp. tumbled in Hong Kong on Friday following a news report Chinese spies might have used chips supplied by another company to hack into U.S. computer systems.
Lenovo shares closed down 15.1 percent while ZTE lost 11 percent.
Bloomberg News cited unidentified U.S. officials as saying malicious chips were inserted into equipment supplied by Super Micro Computer Inc. to American companies and government agencies.
Lenovo, with headquarters in Beijing and Research Triangle Park, North Carolina, is the biggest global manufacturer of personal computers and has a growing smartphone brand.
“Super Micro is not a supplier to Lenovo in any capacity,” said Lenovo in a statement. “Furthermore, as a global company we take extensive steps to protect the ongoing integrity of our supply chain.”
Tomi Engdahl says:
Mikey Campbell / AppleInsider:
Hardware security expert Joe Fitzpatrick, a named source in Bloomberg’s China spy chip investigation, casts doubt on the story’s accuracy — Security researcher Joe Fitzpatrick, one of the few sources named in Bloomberg Businessweek’s bombshell Chinese hack investigation …
Security researcher cited in Bloomberg’s China spy chip investigation casts doubt on story’s veracity
By Mikey Campbell
Monday, October 08, 2018, 07:24 pm PT (10:24 pm ET)
http://appleinsider.com/articles/18/10/08/security-researcher-cited-in-bloombergs-china-spy-chip-investigation-casts-doubt-on-storys-veracity
Security researcher Joe Fitzpatrick, one of the few sources named in Bloomberg Businessweek’s bombshell China hack investigation, in a podcast this week said he felt uneasy after reading the article in part because its claims almost perfectly echoed theories on hardware implants he shared with journalist Jordan Robertson.
The security specialist first spoke with Robertson last year, just prior to giving a presentation on hardware implants at the DEF CON hacking convention. The impetus behind Robertson’s questioning was not made clear to Fitzpatrick until last month.
In his conversations with the journalist, Fitzpatrick detailed how hardware implants work, specifically noting successful proof-of-concept devices he demonstrated at Black Hat in 2016. While he is a security researcher, Fitzpatrick is not in the business of selling such devices to customers — let alone nation states — and is for the most part working off theories derived from years of teaching others how to secure their own hardware.
“It was surprising to me that in a scenario where I would describe these things and then he would go and confirm these and 100% of what I described was confirmed by sources.”
Further, the story as told “doesn’t really make sense.” As Fitzpatrick notes, there are easier, more cost-effective methods of attaining backdoor access into a target computer network.
Bloomberg in its article claimed Chinese operatives managed to sneak a microchip smaller than a grain of rice onto motherboards produced by hardware supplier Supermicro. Supposedly designed by the Chinese military, the chip acted as a “stealth doorway onto any network” and offered “long-term stealth access” to attached computer systems.
Nearly 30 companies were reportedly impacted by the breach, though only Amazon and Apple were mentioned by named in the story. Both companies have released strongly worded denials, with Apple characterizing the report as “wrong and misinformed.”
“Spreading hardware fear, uncertainty and doubt is entirely in my financial gain, but it doesn’t make sense because there are so many easier ways to do this,” Fitzpatrick said, referring to the purported hardware implant. “There are so many easier hardware ways, there are software, there are firmware approaches. There approach you are describing is not scalable. It’s not logical. It’s not how I would do it. Or how anyone I know would do it.”
“And you know I’m still skeptical. I followed up being like, ‘Yeah, okay if they wanted to backdoor every single Supermicro motherboard, I guess this is the approach that makes sense,” he told Gray. “But I still in my mind I couldn’t rationalize that this is the approach any one would choose to take.”
Robertson was unable to produce photographic evidence of the chips in question, saying they were described to him by protected sources. Indeed, Robertson in September asked Fitzpatrick what a “signal amplifier or coupler” looks like, suggesting the publication narrowed the attack package down to that particular component. Fitzpatrick sent Robertson a link to a very small signal coupler sold by Mouser Electronics.
“Turns out that’s the exact coupler in all the images in the story,” Fitzpatrick said.
While the illustration used in the Bloomberg story is just that, Fitzpatrick argues similar components would be an unlikely choice for the attack vector described.
pint-size signal couplers are not standard fare for server motherboards that do not include Wi-Fi or LTE.
Tomi Engdahl says:
Brian Krebs / Krebs on Security:
Why it is basically impossible to secure supply chains from attacks like the alleged Chinese embedded chip hack, and how we can mitigate the consequences
https://krebsonsecurity.com/2018/10/supply-chain-security-is-the-whole-enchilada-but-whos-willing-to-pay-for-it/
Tomi Engdahl says:
Analysts Foresee Supply Chain Impact from Chip Hack Report
https://www.eetimes.com/document.asp?doc_id=1333839
Analysts expect changes in the global electronics supply chain following a report that Chinese spies planted chips in the servers of nearly 30 U.S. companies, including Amazon and Apple.
A Bloomberg report, citing U.S. government and corporate sources speaking off the record, said a unit of China’s People’s Liberation Army (PLA) was behind the effort to hack into the operations of U.S. companies and compromise the global supply chain.
The Bloomberg report, which Apple and Amazon refuted, comes as the Trump administration escalates its trade war with China, targeting computer and networking hardware in its latest round of sanctions. White House officials expect companies to shift their supply chains to other countries as a result.
Analysts said the impact of the reported spying will be substantial.
“There’s going to be structural changes in how hardware gets validated, tested and approved across the supply chain following this,”
Perception is part of reality. The Bloomberg report comes at a time when trade tensions are high, and Ma said he won’t be surprised if it fuels more political agendas.
“In a nutshell, it will raise the risk concern on outsourcing electronic manufacturing to China,” said Bernstein analyst Mark Li.
According to the Bloomberg report, which cited numerous sources in the U.S. government and insiders at companies including Apple and Amazon, one of the first signs of the hack came in 2015, when Amazon Web Services (AWS) hired a third-party company to evaluate the security of software compression firm Elemental Technologies, a company Amazon was planning to acquire.
That security check raised a flag, prompting AWS to scrutinize Elemental’s main product: servers that customers installed to handle video compression.
The AWS checks revealed a rogue chip on the motherboards, the Bloomberg report said. Amazon reported the discovery to the U.S. government, sounding an alarm in the intelligence community. Elemental’s servers were in Department of Defense data centers, the CIA’s drone operations and networks of Navy warships. Elemental was just one of hundreds of Supermicro customers.
A top-secret probe, which Bloomberg said is ongoing, showed that the chips created a backdoor into networks running the altered servers. The chips were inserted on motherboards at factories run by subcontractors in China, according to the report.
Tomi Engdahl says:
Apple, Amazon Refute China Chip Hacking Story
https://www.eetimes.com/document.asp?doc_id=1333831
Tomi Engdahl says:
Industry Reactions to Chinese Spy Chips: Feedback Friday
https://www.securityweek.com/industry-reactions-chinese-spy-chips-feedback-friday
Ian Pratt, co-founder and president, Bromium:
“From the publicly available information it sounds like the implant was intended to compromise the Baseboard Management Controller (BMC) that is present on most server hardware to allow remote management over a network. The BMC has a lot of control over the system. It can provide remote keyboard/video/mouse access to the system over the network. It also typically has access to lots of information about the host, such as its name, domain, IP addresses etc, and can query other information from the host via SNMP. The BMC can also be used to upgrade or modify the firmware used by the main CPU and Management Engine (ME), providing a great scope for stealthy malfeasance.
Based on the photographs, the device appears to be an SPI bus interposer, which would be inserted into the SPI bus between the BMC and the flash memory chip it boots from.
Jack Jones, Co-Founder and Chief Risk Scientist, RiskLens:
“We all know that the Chinese have been persistent in their campaigns to steal intellectual property and government intelligence through digital infiltration. We’ve also always known that hardware backdoors are a potential vector for this activity. In fact, many information security professionals have been warning of this for years. Why then, have companies and government agencies continued to purchase vast amounts and varieties of technologies from China?
Brian Vecci, Technical Evangelist, Varonis:
“This attack is about as surprising as catching Cookie Monster with his hand in the cookie jar. Compromising digital assets has become industrialized with advanced threats’ careful planning and organization. These threat actors are playing a long game with pre-attacks like these that position themselves for devastating attacks down the road– they are testing their abilities and an organization’s vulnerabilities to see how far they can go. What is surprising is that it has only taken decade or two for the digital world to become so inter-dependent – not just with hardware but with software — today many systems have so much code in common that any upstream compromise is a widespread threat.
Sanjay Beri, CEO, Netskope:
“Chinese cyber infiltration is nothing new, as proven by ongoing recent attacks from elite Chinese institutions diligently working to gain access to assets from the west. Today’s news proves that it’s clear we have exited the honeymoon period created by the deal President Obama struck with President Xi Jinping in back in 2015, where the two pledged that each of their governments would refrain from targeted cyber attacks toward another for commercial gain.
Itzik Kotler, CTO and Co-Founder, SafeBreach:
“Like many recent attacks, this is low-level, stealthy, and widespread. The combination of these three makes it especially frightening at first, and it certainly is rare to see such an attack in the wild.
Dave Weinstein, VP Threat Research, Claroty:
“While the denials from Apple and Amazon have been relatively unprecedented in their strength and specificity, the reality is that the supply chain – for everything from consumer products, to technology, to heavy machinery – has been a perpetual source of concern for many years as a morass of potential exposure, and one that renders most security tools obsolete.
Malcolm Harkins, Chief Security and Trust Officer, Cylance:
“Unfortunately the only surprising element about this attack is that it’s taken so long to be uncovered in a report. Supply chain compromise has been a concern for a long time, and there are multiple nation states with endless motivations who make attacks of this scale a certainty rather than a probability.
Tim Bandos, Vice President, Cybersecurity, Digital Guardian:
“The fact that China manufactures many of the components that go into servers, it would be relatively simple to install and disguise a hidden chip enabling backdoor communications and control with those endpoints. Also, given where these chips reside – lower in the stack – most technologies such as EDR and AV have a visibility gap and wouldn’t be able to identify anything being tampered with at the hardware-level. This (once again) demonstrates that determined adversaries have capabilities exceeding that of defenders; hopefully, this will inspire the development of methods and techniques to detect when hardware tampering has taken place. Until then, diversifying supply chain vendors and staying vigilant on outbound and inbound network traffic is highly advised.”
Tomi Engdahl says:
China Tech Stocks Lenovo, ZTE Tumble After Chip Hack Report
https://www.securityweek.com/china-tech-stocks-lenovo-zte-tumble-after-chip-hack-report
Tomi Engdahl says:
UK, US Security Agencies Deny Investigating Chinese Spy Chips
https://www.securityweek.com/uk-us-security-agencies-deny-investigating-chinese-spy-chips
The U.S. Department of Homeland Security (DHS) and the U.K. National Cyber Security Centre (NCSC) have denied investigating the presence of Chinese spy chips in Supermicro servers, as claimed by a bombshell report published last week by Bloomberg.
Tomi Engdahl says:
A major U.S. telecommunications company discovered manipulated hardware from Super Micro Computer Inc. in its network and removed it in August, fresh evidence of tampering in China of critical technology components bound for the U.S., according to a security expert working for the telecom company.
The security expert, Yossi Appleboum, provided documents, analysis and other evidence of the discovery following the publication of an investigative report in Bloomberg Businessweek that detailed how China’s intelligence services had ordered subcontractors to plant malicious chips in Supermicro server motherboards over a two-year period ending in 2015.
Based on his inspection of the device, Appleboum determined that the telecom company’s server was modified at the factory where it was manufactured. He said that he was told by Western intelligence contacts that the device was made at a Supermicro subcontractor factory in Guangzhou, a port city in southeastern China. Guangzhou is 90 miles upstream from Shenzhen, dubbed the `Silicon Valley of Hardware,’ and home to giants such as Tencent Holdings Ltd. and Huawei Technologies Co. Ltd.
The tampered hardware was found in a facility that had large numbers of Supermicro servers, and the telecommunication company’s technicians couldn’t answer what kind of data was pulsing through the infected one, said Appleboum, who accompanied them for a visual inspection of the machine. It’s not clear if the telecommunications company contacted the FBI about the discovery. An FBI spokeswoman declined to comment on whether it was aware of the finding.
Representatives for AT&T Inc. and Verizon Communications Inc. had no immediate comment on whether the malicious component was found in one of their servers. T-Mobile U.S. Inc. and Sprint Corp. didn’t immediately respond to requests for comment.
New Evidence of Hacked Supermicro Hardware Found in U.S. Telecom
https://www.bloomberg.com/news/articles/2018-10-09/new-evidence-of-hacked-supermicro-hardware-found-in-u-s-telecom
Discovery shows China continues to sabotage critical technology components bound for America
Tomi Engdahl says:
A new twist in Bloomberg’s ‘spy chip’ report implicates U.S. telecom
https://techcrunch.com/2018/10/09/a-new-twist-in-bloombergs-spy-chip-report-implicates-u-s-telecom/?utm_source=tcfbpage&sr_share=facebook
There’s a new wrinkle in the Bloomberg’s ongoing but controversial series on alleged hardware hacks affecting U.S. tech giants — despite heavy skepticism after the named companies rebuffed the allegations and critics poked holes in the reporting.
Bloomberg’s new report out Tuesday said that a U.S. telecom discovered that hardware it used in its datacenters was “manipulated” by an implant designed to conduct covert surveillance and exfiltrate corporate or government secrets.
The implant was found on an Ethernet connector
on a motherboard developed by Supermicro, a major computer manufacturer that was named in the first Bloomberg story.
Although the report claims “fresh evidence of tampering” by China, it does not explicitly link the tampering to similar attacks on Apple and Amazon, or others.
security researcher said he inspected the implant first-hand, rather than the reporters having to rely on descriptions from several sources who allegedly had knowledge of the implants.
Yossi Appleboum, co-founder of Sepio Systems and former Israeli intelligence officer, provided Bloomberg with evidence and documentation — which wasn’t published alongside the story — that the alleged implant was introduced at the factory where the telecom’s equipment was built. He also said there are many ways that China’s supply chain is compromised and implants could be introduced.
Plot twist: Bloomberg didn’t name the telecom because of a non-disclosure agreement
Bloomberg said as of Monday that it stood by its reporting.
But it’s difficult not to be skeptical
Tomi Engdahl says:
New Evidence of Hacked Supermicro Hardware Found in U.S. Telecom
https://www.bloomberg.com/news/articles/2018-10-09/new-evidence-of-hacked-supermicro-hardware-found-in-u-s-telecom
The discovery shows that China continues to sabotage critical technology components bound for America.
Tomi Engdahl says:
U.S. Republican senator seeks briefings on reported China hacking attack
https://www.reuters.com/article/us-china-cyber-apple/u-s-republican-senator-seeks-briefings-on-reported-china-hacking-attack-idUSKCN1MJ2O8
The top Republican on the Senate Commerce Committee has asked Apple Inc, Amazon.com Inc and Super Micro Computer Inc for staff briefings about a Bloomberg report that the Chinese government implanted malicious hardware into server motherboards provided by Super Micro.
“Allegations that the U.S. hardware supply chain has been purposely tampered with by a foreign power must be taken seriously,” Thune wrote.
Tomi Engdahl says:
Motherboard:
Sprint, AT&T, Verizon, and T-Mobile deny being the “US telecom” mentioned in today’s Bloomberg story alleging compromised Supermicro hardware — No one is really sure who to believe after Businessweek’s bombshell story on an alleged Chinese supply chain attack against Apple, Amazon, and others.
The Cybersecurity World Is Debating WTF Is Going on With Bloomberg’s Chinese Microchip Stories
https://motherboard.vice.com/en_us/article/qv9npv/bloomberg-china-supermicro-apple-hack
No one is really sure who to believe after Businessweek’s bombshell story on an alleged Chinese supply chain attack against Apple, Amazon, and others.
It is not clear in the article that Bloomberg knows which telecom is apparently affected; it notes that Appleboum is covered by an non-disclosure agreement. Motherboard has reached out to 10 major US telecom providers, and the four biggest telecoms in the US have denied to Motherboard that they were attacked: In an email, T-Mobile denied being the one mentioned in the Bloomberg story. Sprint said in an email that the company does not use SuperMicro equipment, and an AT&T spokesperson said in an email that “these devices are not a part of our network, and we are not affected.” A Verizon spokesperson said: “Verizon’s network is not affected.”
“You should know that Bloomberg provided us with no evidence to substantiate their claims and our internal investigations concluded their claims were simply wrong,” the letter, signed by George Stathakopoulos, vice president of information security at Apple, reads.
“Hardware is a nightmare. We can barely validate software, and all our assumptions rely on the hardware working correctly. Pull away that assumption, it’s like removing the screws from a piece of IKEA furniture,” Matthew Green, associate professor at Johns Hopkins University, previously told Motherboard in an online chat. The article itself was based mostly on anonymous sources, both inside impacted companies and those who had been briefed on the incident.
The prospect of this kind of attack is very real, but the fact that both Bloomberg and the companies named in the story are doubling down is confusing everyone, and a sign that we are probably not done hearing about this story anytime soon.
A Bloomberg spokesperson said in a statement: “As is typical journalistic practice, we reached out to many people who are subject matter experts to help us understand and describe technical aspects of the attack. The specific ways the implant worked were described, confirmed, and elaborated on by our primary sources who have direct knowledge of the compromised Supermicro hardware. Joe FitzPatrick was not one of these 17 individual primary sources that included company insiders and government officials, and his direct quote in the story describes a hypothetical example of how a hardware attack might play out, as the story makes clear.”
Tomi Engdahl says:
Bloomberg:
Yossi Appleboum, a security expert working for a “major” US telco provides docs allegedly showing Supermicro sent it hacked hardware; Supermicro stock down ~16%
https://www.bloomberg.com/news/articles/2018-10-09/new-evidence-of-hacked-supermicro-hardware-found-in-u-s-telecom
Tomi Engdahl says:
Risky Business Feature: Named source in “The Big Hack” has doubts about the story
https://risky.biz/RB517_feature/
Tomi Engdahl says:
Tampered Chinese Ethernet port used to hack ‘major US telecom,’ says Bloomberg report
https://www.theverge.com/2018/10/9/17955848/supermicro-telecom-server-hack-apple-amazon
It’s the first time a Bloomberg source has gone on the record following the original report from last week
Joe Fitzpatrick, went on a podcast Monday explaining that he was uncomfortable with the story and was unsure whether it was entirely accurate.
https://risky.biz/RB517_feature/
Tomi Engdahl says:
NSA cybersecurity head can’t find corroboration for iCloud spy chip report
https://iphone.appleinsider.com/articles/18/10/10/nsa-cybersecurity-head-cant-find-corroboration-for-icloud-spy-chip-report
The senior advisor for Cybersecurity Strategy to the director of the National Security Agency has advised there is a lack of evidence relating to both of Bloomberg’s recent espionage-related stories, and has openly requested for people with knowledge of the situation to provide assistance.
Joyce’s comments suggest he disbelieves the entirety of the report, through checking via his own sources.
In response to Wall Street Journal reporter Dustin Volz’s query on the allegations, Joyce advised “What I can’t find are any ties to the claims in the article,” adding “We’re befuddled.”
Joyce to plea to others to bring clarity
Joyce then admits he has no confidence that there’s something to the story. “I worry about the distraction that it is causing.”
Fitzpatrick advised he had previously spoken to the reporter about proof-of-concept devices demonstrated at Black Hat 2016, but found it strange that the ideas he mentioned were confirmed by other sources of the publication.
Two U.S. senators have written to Supermicro demanding answers over the reports
Tomi Engdahl says:
Homeland Security has ‘no reason to doubt’ China spy chip refutals
It’s not content to sit by the wayside.
https://www.engadget.com/2018/10/07/homeland-security-backs-amazon-apple-refutals-of-china-spy-chip/
Tomi Engdahl says:
Chinese Super Micro ‘spy chip’ story gets even more strange as everyone doubles down
Bloomberg puts out related story while security experts cast doubt on research and quotes
https://www.theregister.co.uk/2018/10/09/bloomberg_super_micro_china_spy_chip_scandal/
Tomi Engdahl says:
So who lied? Was it Bloomberg Business Week? Or is the liar Apple, Amazon and Supermicro?
https://medium.com/silicon-valley-global-news/so-who-lied-50e3aad7a2da
Tomi Engdahl says:
NSA denies spy chip claims, FBI says it’s not allowed to
https://9to5mac.com/2018/10/11/spy-chip-claims/
The weight of evidence that Bloomberg’s spy chip claims were based on a misunderstanding continues to grow.
An NSA spokesperson has said that the story had people ‘chasing shadows’ and even with the ‘great access’ the organization has to secret information, it can still find no evidence to support Bloomberg’s claims …
Addressing the issue of whether Apple and Amazon might be lying about the spy chip claims to protect their reputations, Joyce didn’t buy it.
Those companies will “suffer a world of hurt” if regulators later determine that they lied.
Joyce also said that fallout from the story wasn’t limited to damage to the reputation of the companies concerned.
[I have] grave concerns about where this has taken us […] I worry that we’re chasing shadows right now. I worry about the distraction that it is causing.
While the NSA would certainly be willing to keep things quiet if it determined that were in the interests of national security, that idea ceased making sense the moment the story was published.
Tomi Engdahl says:
Yossi Appleboum on How Bloomberg is Positioning His Research Against Supermicro
https://www.servethehome.com/yossi-appleboum-disagrees-bloomberg-is-positioning-his-research-against-supermicro/
Tomi Engdahl says:
Supply Chain Security 101: An Expert’s View
https://krebsonsecurity.com/2018/10/supply-chain-security-101-an-experts-view/
Brian Krebs (BK): Do you think Uncle Sam spends enough time focusing on the supply chain security problem? It seems like a pretty big threat, but also one that is really hard to counter.
Tony Sager (TS): The federal government has been worrying about this kind of problem for decades. In the 70s and 80s, the government was more dominant in the technology industry and didn’t have this massive internationalization of the technology supply chain.
But even then there were people who saw where this was all going, and there were some pretty big government programs to look into it.
BK: Why do you think more companies aren’t insisting on producing stuff through code and hardware foundries here in the U.S.?
TS: Like a lot of things in security, the economics always win. And eventually the cost differential for offshoring parts and labor overwhelmed attempts at managing that challenge.
TS: Suppose a nation state dominates a piece of technology and in theory could plant something inside of it. The attacker in this case has a risk model, too. Yes, he could put something in the circuitry or design, but his risk of exposure also goes up.
Could I as an attacker control components that go into certain designs or products? Sure, but it’s often not very clear what the target is for that product, or how you will guarantee it gets used by your target. And there are still a limited set of bad guys who can pull that stuff off. In the past, it’s been much more lucrative for the attacker to attack the supply chain on the distribution side, to go after targeted machines in targeted markets to lessen the exposure of this activity.
BK: So targeting your attack becomes problematic if you’re not really limiting the scope of targets that get hit with compromised hardware.
TS: Yes, you can put something into everything, but all of a sudden you have this massive big data collection problem on the back end where you as the attacker have created a different kind of analysis problem. Of course, some nations have more capability than others to sift through huge amounts of data they’re collecting.
BK: Is there anything about the way these cloud-based companies operate….maybe just sheer scale…that makes them perhaps uniquely more resilient to supply chain attacks vis-a-vis companies in other industries?
TS: That’s a great question. The counter positive trend is that in order to get the kind of speed and scale that the Googles and Amazons and Microsofts of the world want and need, these companies are far less inclined now to just take off-the-shelf hardware and they’re actually now more inclined to build their own.
BK: Can you give some examples?
TS: There’s a fair amount of discussion among these cloud providers about commonalities — what parts of design could they cooperate on so there’s a marketplace for all of them to draw upon. And so we’re starting to see a real shift from off-the-shelf components to things that the service provider is either designing or pretty closely involved in the design, and so they can also build in security controls for that hardware. Now, if you’re counting on people to exactly implement designs, you have a different problem. But these are really complex technologies, so it’s non-trivial to insert backdoors. It gets harder and harder to hide those kinds of things.
Tomi Engdahl says:
https://www.tivi.fi/Kaikki_uutiset/lista-kiinasirun-kiistajista-pitenee-nsa-jahtaamme-varjoja-6744672
Tomi Engdahl says:
Software Update Supply Chain Attacks: What You Need to Know
https://medium.com/threat-intel/software-update-supply-chain-attacks-what-you-need-to-know-f5bd3ba9718e
Software update supply chain attacks have been one of the big trends in cyber crime in 2018. Find out more about this cyber attack technique.
Some of the most high-profile cyber attacks of recent times have been perpetrated after cyber criminals compromised a third-party supplier of the targeted company and used their access to get on to the victim’s network: attacks of this nature are known as software update supply chain attacks.
What is a software update supply chain attack?
We define a software update supply chain attack as follows: “Implanting a piece of malware into an otherwise legitimate software package at its usual distribution location; this can occur during production at the software vendor, at a third-party storage location, or through redirection.”
Tomi Engdahl says:
BuzzFeed News:
Tim Cook calls for a retraction of Bloomberg’s China chip hacking story: “There is no truth in their story about Apple. They need to do the right thing” — “There is no truth in their story about Apple. They need to do the right thing.” — Apple CEO Tim Cook …
Apple CEO Tim Cook Is Calling For Bloomberg To Retract Its Chinese Spy Chip Story
https://www.buzzfeednews.com/article/johnpaczkowski/apple-tim-cook-bloomberg-retraction
“I feel they should retract their story. There is no truth in their story about Apple. They need to do that right thing.”
Apple CEO Tim Cook, in an interview with BuzzFeed News, went on the record for the first time to deny allegations that his company was the victim of a hardware-based attack carried out by the Chinese government. And, in an unprecedented move for the company, he called for a retraction of the story that made this claim.
Earlier this month Bloomberg Businessweek published an investigation alleging Chinese spies had compromised some 30 US companies by implanting malicious chips into Silicon Valley–bound servers during their manufacture in China. The chips, Bloomberg reported, allowed the attackers to create “a stealth doorway” into any network running on a server in which they were embedded.
According to Bloomberg, the company discovered some sabotaged hardware in 2015, promptly cut ties with the vendor, Supermicro, that supplied it, and reported the incident to the FBI.
Apple, however, has maintained that none of this is true
Bloomberg has stood steadfastly by its story and even published a follow-up account that furthered the original’s claims.
Apple Insiders Say Nobody Internally Knows What’s Going On With Bloomberg’s China Hack Story
https://www.buzzfeednews.com/article/johnpaczkowski/apple-china-hacking-bloomberg-servers-spies-fbi
“I don’t know if something like this even exists.”
Tomi Engdahl says:
Bloomberg reiterated its previous defense of the story. “Bloomberg Businessweek’s investigation is the result of more than a year of reporting, during which we conducted more than 100 interviews,” a spokesperson told BuzzFeed News in response to a series of questions. “Seventeen individual sources, including government officials and insiders at the companies, confirmed the manipulation of hardware and other elements of the attacks.
https://www.buzzfeednews.com/article/johnpaczkowski/apple-tim-cook-bloomberg-retraction
Tomi Engdahl says:
Much Ado About Hardware Implants
https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2018/october/much-ado-about-hardware-implants/
By now most people have seen the three Bloomberg articles [1],[36],[49] detailing the alleged conspiracy to install back-doors on servers assembled by SuperMicro via a tiny microchip. There are plenty of great takes already
Supply chain attacks are not new
nor are those using hardware implants
Several questions need to be answered here:
1) Is it possible from a purely technical standpoint?
2) Did it actually happen?
3) What do we do about it?
Technical Feasibility
There are simply not enough technical details in the article to forensically disassemble how such an attack occurred. So, let’s turn this around: How would I implement such an attack from a hardware/firmware/software perspective?
Baseboard Management Controller
These are server-grade motherboards. Most such systems have a dedicated Baseboard Management Controller (BMC) on board, which is a secondary, dedicated computer for server management. This controller gives the administrators the capability to remotely write to the hard disks to install operating systems, upgrade the system BIOS, turn the power on and off, monitor health metrics such as temperatures, and view the video screen remotely just as though they were sitting at the console. It is well connected within the host system, sitting on the Peripheral Component Interconnect Express (PCIe) and Low Pin Count (LPC) buses and has direct connections to the network interface
BMCs come in all shapes, sizes, and brands. SuperMicro uses a very common BMC solution from ASPEED, HP has their iLO, Dell has their iDRAC, Facebook has created OpenBMC
Attacking the BMC
Without Secure Boot stopping us, the easiest attack would be to simply modify the BMC firmware in flash. No hardware implant is needed. But (a) the main article clearly states there was a hardware device, (b) this attack can be detected in an audit; pulling the flash chip and comparing its contents to a known-good copy would be easy to do, and (c) it does not work against the most modern devices that are implementing Secure Boot, so it lacks generality. Bloomberg’s second article does indicate that this sort of attack might have been attempted, and it is possible that it was just an earlier iteration before the attackers evolved into more advanced hardware implants.
Similarly, using outdated firmware with known (or even intentionally crafted) vulnerabilities is a much simpler attack than a hardware implant, and provides a comfortable amount of plausible deniability. With most BMC implementations there are many vulnerabilities to choose from.
Now a common implementation flaw in embedded systems occurs when data is read from flash memory multiple times: once to validate it and again to use it. This is often referred to as a Time-of-Check-Time-of-Use (TOCTOU) issue, a double-read, or more generically a race-condition.
Risks in the supply chain
Outsourcing anything, from an open source software library to purchasing a server from overseas, is a risk. You are offloading control of your system security to another entity, one that may not have the same motivations and appetite for risk as you. Your suppliers may in turn outsource things to yet others. It is turtles all the way down. And when you get nearer the bottom, it is astounding the types of creative shenanigans that can happen. Malware on factory test stations, refurbished components sold as new, cash bribes paid by organized crime to factory workers, etc. What is alleged in the story is completely within the realm of possibility, and in-line with my professional experience when investigating actual factory product security breaches.
Establishing network connections
The BMC is frequently configurable to support either a dedicated management network connection or to piggyback over the host systems network interface. This allows users flexibility in the deployment, but also makes the attackers job of reaching out to the Internet much less predictable. Malware is commonly designed to reach out to a command and control (C&C) server that the attacker controls. This type of rogue traffic is probably the easiest way to detect an attack such as the one described in the article. Any organization with a large robust security program, and certainly all of the victims named in the article, would catch this immediately. Now it is entirely possible that this backdoor operates silently until activated. It could monitor network traffic passively waiting for a magic string to activate its payload. Researchers demonstrated such an attack in a hard drive controller in 2012
Did it actually happen?
The article does not give any actual technical details, publishes no sources, and relies on anonymous information almost entirely.
There is speculation that the authors have conflated this with other stories involving bad firmware updates and common IPMI bugs
What next?
First of all, don’t panic. There are enough arguments against the allegations in the article to warrant a healthy skepticism.
Things like Secure Boot, authenticated debug, storage encryption, and transport encryption are table-stakes for embedded device security in 2018. Releasing modern, network-connected devices for use in critical business infrastructure without performing significant security due-diligence is both irresponsible and bad business. Regarding supply chain attacks specifically, here are some things that can help:
Random product audits: Automated optical and X-ray inspections, precision weight and RF measurements, chemical analysis, and other techniques can help determine if there are rogue implants. Make the attacker work hard to hide their work. This is a cat-and-mouse game of obfuscation [13], but it is as useful for product quality as it is for security.
Diversification: Multi-sourcing components with multiple vendors and multiple factories makes you harder for an attacker to predict. Diversification in your shipping logistics can help mitigate interdiction attacks.
Strong monitoring: Your factory networks, processes, and people are all subject to exploitation. Detection can go a long way, but if you don’t look, you won’t find.
Outsourcing, at the end of the day, is a business decision. You can rewrite all of your software libraries from scratch in-house, but your product will not ship on time. You can design and build all your servers in-house, but your profit margins would decrease. So, you outsource.
Tomi Engdahl says:
Makena Kelly / The Verge:
AWS CEO Andy Jassy and Supermicro CEO Charles Liang join Apple CEO Tim Cook in calling for a retraction of Bloomberg’s spy chip story — ‘[Tim Cook] is right. Bloomberg story is wrong about Amazon, too.’ — Today, executives from both Amazon and the server manufacturer, Super Micro …
Amazon exec and Super Micro CEO call for retraction of spy chip story
‘[Tim Cook] is right. Bloomberg story is wrong about Amazon, too.’
https://www.theverge.com/2018/10/22/18011138/china-spy-chip-amazon-apple-super-micro-ceo-retraction
Today, executives from both Amazon and the server manufacturer, Super Micro, are calling for the retraction of a Bloomberg report published earlier this month. The report alleged that these chips were able to compromise the computer networks of as many as 30 companies, including networks belonging to Amazon.
Last week, Apple CEO Tim Cook called for Bloomberg to retract a report claiming that Chinese spies smuggled malicious microchips into a company server.
The other two companies named in the story, Amazon and Super Micro, decided to follow in Apple’s footsteps today, offering their own statements condemning the allegations.
On Monday, Super Micro said that the company would continue to investigate the claims and review its motherboards in search of any hardware manipulations. Just hours after, Super Micro CEO Charles Liang said, “Bloomberg should act responsibly and retract its unsupported allegations.”
The report cites 17 unnamed sources and no compromised hardware has surfaced in the weeks since publication. The report garnered nearly instant criticism when it was published earlier this month from cybersecurity experts who were unconvinced by the available evidence.
Tomi Engdahl says:
Patrick Kennedy / ServeTheHome:
A critical analysis of Bloomberg’s claims about the alleged compromised Supermicro motherboards and why the reported technical details are implausible
Investigating Implausible Bloomberg Supermicro Stories
https://www.servethehome.com/investigating-implausible-bloomberg-supermicro-stories/
Technical Lightness or Inaccuracy
This is a long article. In the first section, we are going to discuss why there are some fairly astounding plausibility and feasibility gaps in Bloomberg’s description of how the hack worked. The weakness in this section of the Bloomberg article makes it extremely difficult to navigate and it is light on details. We are going to evaluate some of the parts in isolation, and also discuss some of the logical outcomes.
The Lynchpin of How Bloomberg’s Device Activates is Not Plausible
We are going to focus on a few key parts of one of the opening paragraphs from the story where functionality is described.
Since the implants were small, the amount of code they contained was small as well. But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off. (Source: Bloomberg with emphasis added to highlight key points for discussion)
That first part starting with “telling the device…” is nonsensical. If you are in the industry or read our Basic BMC and IPMI Management Security Practices piece, you would know that this is false.
Even smaller organizations with a handful of servers generally have segregated BMC networks. That basic starting point, from where large companies take further steps
One of the biggest reasons people in the server management industry doubt the Bloomberg report is that it is not plausible for these chips to function in the networks of their intended targets, allegedly “30 companies.”
Impossible Access to Code on Crashed or Turned Off Servers
The next inaccuracy to this paragraph is the line describing BMCs as “giving them access to the most sensitive code even on machines that have crashed or are turned off.” That is not how this technology works.
Baseboard management controllers or BMCs are active on crashed or turned off servers. They allow one to, for example, power cycle servers remotely.
When a server is powered off it is not possible to access a server’s “most sensitive code.” OS boot devices are powered off. Local storage is powered off for the main server. Further encrypted sensitive code pushed from network storage is not accessible, and a BMC would not authenticate.
This line from the Bloomberg is technically inaccurate because a powered off server’s storage with its sensitive code has no power and cannot be accessed.
Bloomberg Reports China Infiltrated the Supermicro Supply Chain We Investigate
https://www.servethehome.com/bloomberg-reports-china-infiltrated-the-supermicro-supply-chain-we-investigate/
Tomi Engdahl says:
Investigating Implausible Bloomberg Supermicro Stories
https://www.servethehome.com/investigating-implausible-bloomberg-supermicro-stories/
Tomi Engdahl says:
http://www.etn.fi/index.php/13-news/8600-applen-paajohtaja-tuotteissamme-ei-ole-kiinalaisia-vakoilupiireja
Tomi Engdahl says:
Forgotten that Chinese spy chip story? We haven’t – it’s still wrong, Super Micro tells SEC
Server maker drags Bloomberg in note to customers, watchdog, still checking its motherboards
https://www.theregister.co.uk/2018/10/22/super_micro_chinese_spy_chip_sec/
Big price to pay
The Bloomberg article – published on October 3 – wiped more than 40 per cent off Super Micro’s share price within a matter of hours. But, despite all the three main companies included in the report – Apple, Amazon and Super Micro – all strenuously denying the story was true, Super Micro’s share price has not recovered.
It hit a low of $12.46 following the story, a gut-wrenching plunge from $21.40, but as of the time of writing, the share price is $14.74. That represents a recovery of 9 per cent but it is still down 31 per cent from before the story was published.
Super Micro stresses that no one has come to the support of Bloomberg’s article, and that numerous officials, including FBI director Christopher Wray, NSA Senior Cybersecurity Advisor Rob Joyce, Director of National Intelligence Dan Coats, the US Department of Homeland Security, and the UK’s GCHQ have all questioned the story.
Due to the nature of the allegations however – a highly confidential state-sponsored hacking effort – everyone, and particularly the stock market, remains wary. A long history of the intelligence services issuing misleading and at times downright false statements does not make them the most reliable sources of information.
Apple and Amazon have also demonstrated a strong tendency to spin their way out of embarrassing situations with carefully constructed denials. Plus, of course, it is very strongly in Super Micro’s interests to deny the story.
That said, the denials have been unusually specific and categorical. As time has passed, the growing consensus appears to be that Bloomberg got the story wrong. Although a better explanation may be that it accurately reported a misinformation campaign put together by some part of the intelligence services.
Implausible
For its part, Super Micro tried to soothe customers by arguing that it has comprehensive checks on its products and would have noticed any effort to interfere with them.
Retract?
While that explanation would suffice for most situations, the fact that Super Micro’s motherboards are used by companies like Amazon and Apple as well as the US military, it remains plausible that the Chinese government would be willing to invest the enormous resources necessary to pull off such a hack.
The statement does reflect growing consensus that Bloomberg bombed the story, however. Last week, Apple CEO Tim Cook called for the newswire to retract the story, effectively admitting it got it wrong. And this morning, the head of Amazon Web Services, Andy Jassy, joined that call
Tomi Engdahl says:
Apple CEO Tim Cook Is Calling For Bloomberg To Retract Its Chinese Spy Chip Story
https://www.buzzfeednews.com/article/johnpaczkowski/apple-tim-cook-bloomberg-retraction
“I feel they should retract their story. There is no truth in their story about Apple. They need to do the right thing.”
Tomi Engdahl says:
Two new supply-chain attacks come to light in less than a week
https://arstechnica.com/information-technology/2018/10/two-new-supply-chain-attacks-come-to-light-in-less-than-a-week/
As drive-by attacks get harder, hackers exploit the trust we have in software providers.
Most of us don’t think twice about installing software or updates from a trusted developer.
As developers continue to make software and webpages harder to hack, blackhats over the past few years have increasingly exploited this trust to spread malicious wares. Over the past week, two such supply-chain attacks have come to light.
The first involves VestaCP, a control-panel interface that system administrators use to manage servers.
“The VestaCP installation script was altered to report back generated admin credentials to vestacp.com after a successful installation,”
The attackers, Léveillé said, then likely used the passwords to log in to servers over their secure shell interface.
Using SSH, the attackers infected the servers with ChachaDDoS, a relatively new strain of malware used to wage denial-of-service attacks on other sites.
Chacha runs on 32- and 64-bit ARM, x86, x86_64, MIPS, MIPSEL, and PowerPC. On Tuesday, researchers from security firm Sophos described a newly discovered DDoS botnet they call Chalubo
Clipboard hijacker sneaked into PyPI
The second supply-chain attack to come to light this week involves a malicious package that was slipped into the official repository for the widely used Python programming language. Called “Colourama,” the package looked similar to Colorama, which is one of the top-20 most-downloaded legitimate modules in the Python repository. The doppelgänger Colourama package contained most of the legitimate functions of the legitimate module, with one significant difference: Colourama added code that, when run on Windows servers, installed this Visual Basic script. It constantly monitors the server’s clipboard for signs a user is about to make a cryptocurrency payment.
Tomi Engdahl says:
Building a Proof of Concept Hardware Implant
https://hackaday.com/2018/10/24/building-a-proof-of-concept-hardware-implant/
You’ve no doubt heard about the “hardware implants” which were supposedly found on some server motherboards, which has led to all sorts of hand-wringing online. There’s no end of debate about the capabilities of such devices, how large they would need to be, and quite frankly, if they even exist to begin with. We’re through the looking-glass now, and there’s understandably a mad rush to learn as much as possible about the threat these types of devices represent.
[Nicolas Oberli] of Kudelski Security wanted to do more than idly speculate, so he decided to come up with a model of how an implanted hardware espionage device could interact with the host system. He was able to do this with off the shelf hardware, meaning anyone who’s so inclined can recreate this “Hardware Implant Playset” in their own home lab for experimentation. Obviously this is not meant to portray a practical attack in terms of the hardware itself, but gives some valuable insight into how such a device might function.
https://research.kudelskisecurity.com/2018/10/23/build-your-own-hardware-implant/