Updated Intel’s Atom C2000 processor family has a fault that effectively bricks devices, costing the company a significant amount of money to correct. But the semiconductor giant won’t disclose precisely how many chips are affected nor which products are at risk.
On its Q4 2016 earnings call earlier this month, chief financial officer Robert Swan said a product issue limited profitability during the quarter, forcing the biz to set aside a pot of cash to deal with the problem.
“We were observing a product quality issue in the fourth quarter with slightly higher expected failure rates under certain use and time constraints, and we established a reserve to deal with that,” he said. “We think we have it relatively well-bounded with a minor design fix that we’re working with our clients to resolve.”
Coincidentally, Cisco last week issued an advisory warning that several of its routing, optical networking, security and switch products sold prior to November 16, 2016 contain a faulty clock component that is likely to fail at an accelerated rate after 18 months of operation.
“If the LPC clock(s) stop functioning the system will no longer be able to boot,” Intel’s documentation explains.
Last week, Paul Alcorn over at Tom’s Hardware picked up on an interesting statement made by Intel in their Q4 2016 earnings call. The company, whose Data Center group’s profits had slipped a bit year-over-year, was “observing a product quality issue in the fourth quarter with slightly higher expected failure rates under certain use and time constraints.” As a result the company had setup a reserve fund as part of their larger effort to deal with the issue, which would include a “minor” design (i.e. silicon) fix to permanently resolve the problem.
A bit more digging by Paul further turned up that the problem was with Intel’s Atom C2000 family, better known by the codenames Avoton and Rangeley.
C2000 is an important part of Intel’s product lineup – especially as it directly competes with various ARM-based processors in many of its markets
The Problem: Early Circuit Degradation
AVR54. System May Experience Inability to Boot or May Cease Operation
Problem: The SoC LPC_CLKOUT0 and/or LPC_CLKOUT1 signals (Low Pin Count bus clock outputs) may stop functioning.
Implication: If the LPC clock(s) stop functioning the system will no longer be able to boot.
Workaround: A platform level change has been identified and may be implemented as a workaround for this erratum.
At a high-level, the problem is that the operating clock for the Low Pin Count bus can stop working. Essentially a type of legacy bus, the LPC bus is a simple bus for simple peripherals, best known for supporting legacy devices such as serial and parallel ports. It is not a bus that’s strictly necessary for the operation of a computer or embedded device, and instead its importance depends on what devices are being hung off of it. Along with legacy I/O devices, the second most common device type to hang off of the LPC is the boot ROM/BIOS– owing to the fact that it’s a simple device that needs little bandwidth – and this is where the C2000 flaw truly rears its head.
As Intel’s errata succinctly explains, if the LPC bus breaks, then any system using it to host the boot ROM will no longer be able boot, as the system would no longer be able to access said boot ROM. The good news is that Intel has a workaround (more on that in a second), so it’s an avoidable failure, but it’s a hardware workaround, meaning the affected boards have to be reworked to fix them. Complicating matters, since Atom C2000 is a BGA chip being used in an embedded fashion, an LPC failure means that the entire board (if not the entire device) has to be replaced.
Intel revealed during its Q4 2016 earnings call that unspecified CPU failures have affected its bottom line. The company enjoyed record Q4 revenue and a record $59.4 billion in overall 2016 revenue, but CPU failures reduced the Data Center Group’s financial performance.
Exclusive The flaw in Intel’s Atom C2000 family of chips has been vexing Intel’s hardware customers for at least a year and a half, according to a source at one affected supplier, but it wasn’t immediately obvious that Intel’s silicon was to blame.
The well-placed insider, who spoke to The Register on condition of anonymity, said the problem – which results in bricked systems – became apparent to engineers at product makers when the return rate on gear spiked about 18 months ago.
It took additional time to figure out that Intel’s hardware was responsible.
In January, with the publication of errata documenting the flawed LPC clock outputs in its Atom C2000 system-on-chips, Intel acknowledged the issue publicly. Around the same time, chief financial officer Robert Swan said profitability during Intel’s fourth quarter last year had been constrained by an unspecified product issue, prompting the company to establish a cash reserve to deal with the blunder.
Cisco’s subsequent publication of an advisory warning that a variety of its products could fail after 18 months of use because of a faulty clock component
Companies that appear to be affected by the Atom C2000 flaw include not only those mentioned in our previous report – Aaeon, Dell, HP, Infortrend, Lanner, NEC, Newisys, Netgate, Netgear, Quanta, Supermicro, Synology, and ZNYX Networks – but also ASRock Rack (C2550D4I and C2750D4I), iXsystems (FreeNAS Mini), Seagate (NAS Pro), and Sophos (UTM firewall SG 125).
Our industry source suggested that non-disclosure agreements with Intel prevent companies from releasing or discussing information about technical issues related to Intel products.
Netgear on Thursday said that four of its product lines may experience “higher-than-normal failure rates” and that it is contacting affected customers to assess whether replacement or repair is appropriate.
“While this situation is not an immediate cause for concern about the operation of any of these models, we understand that customers might have questions,” the company said in a statement. “We will be reaching out to all registered customers of the affected models over the next several months to determine the appropriate actions for their situations.”
The four product lines – ReadyNAS devices RN3130 and RN3138, and Wi-Fi managers WC7500 and WC7600v2 – happen to have a common component, Intel’s Atom C2000 line chips, which Intel has acknowledged had problems until recently. As many as 16 specific models in this group of four lines may be affected.
Other vendors and service providers that have used Intel’s Atom C2000 chips and appear to have been affected include: Aaeon, ASRock Rack, Checkpoint, Cisco, Dell, Fortinet, HP, Infortrend, iXsystems, Online/Scaleway, Lanner, NEC, Newisys, Netgate, Quanta, Seagate, Sophos, Supermicro, Synology, and ZNYX Networks.
We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.
We are a professional review site that has advertisement and can receive compensation from the companies whose products we review. We use affiliate links in the post so if you use them to buy products through those links we can get compensation at no additional cost to you.OkDecline
7 Comments
Tomi Engdahl says:
http://www.theregister.co.uk/2017/02/06/cisco_intel_decline_to_link_product_warning_to_faulty_chip/
Tomi Engdahl says:
Intel’s Atom C2000 chips are bricking products – and it’s not just Cisco hit
Chipzilla and Switchzilla won’t confirm connection but the writing is on the wall
https://www.theregister.co.uk/2017/02/06/cisco_intel_decline_to_link_product_warning_to_faulty_chip/
Updated Intel’s Atom C2000 processor family has a fault that effectively bricks devices, costing the company a significant amount of money to correct. But the semiconductor giant won’t disclose precisely how many chips are affected nor which products are at risk.
On its Q4 2016 earnings call earlier this month, chief financial officer Robert Swan said a product issue limited profitability during the quarter, forcing the biz to set aside a pot of cash to deal with the problem.
“We were observing a product quality issue in the fourth quarter with slightly higher expected failure rates under certain use and time constraints, and we established a reserve to deal with that,” he said. “We think we have it relatively well-bounded with a minor design fix that we’re working with our clients to resolve.”
Coincidentally, Cisco last week issued an advisory warning that several of its routing, optical networking, security and switch products sold prior to November 16, 2016 contain a faulty clock component that is likely to fail at an accelerated rate after 18 months of operation.
“If the LPC clock(s) stop functioning the system will no longer be able to boot,” Intel’s documentation explains.
Tomi Engdahl says:
Semi-Critical Intel Atom C2000 SoC Flaw Discovered, Hardware Fix Required
by Ryan Smith on February 8, 2017 10:30 AM EST
http://www.anandtech.com/show/11110/semi-critical-intel-atom-c2000-flaw-discovered
Last week, Paul Alcorn over at Tom’s Hardware picked up on an interesting statement made by Intel in their Q4 2016 earnings call. The company, whose Data Center group’s profits had slipped a bit year-over-year, was “observing a product quality issue in the fourth quarter with slightly higher expected failure rates under certain use and time constraints.” As a result the company had setup a reserve fund as part of their larger effort to deal with the issue, which would include a “minor” design (i.e. silicon) fix to permanently resolve the problem.
A bit more digging by Paul further turned up that the problem was with Intel’s Atom C2000 family, better known by the codenames Avoton and Rangeley.
C2000 is an important part of Intel’s product lineup – especially as it directly competes with various ARM-based processors in many of its markets
The Problem: Early Circuit Degradation
AVR54. System May Experience Inability to Boot or May Cease Operation
Problem: The SoC LPC_CLKOUT0 and/or LPC_CLKOUT1 signals (Low Pin Count bus clock outputs) may stop functioning.
Implication: If the LPC clock(s) stop functioning the system will no longer be able to boot.
Workaround: A platform level change has been identified and may be implemented as a workaround for this erratum.
At a high-level, the problem is that the operating clock for the Low Pin Count bus can stop working. Essentially a type of legacy bus, the LPC bus is a simple bus for simple peripherals, best known for supporting legacy devices such as serial and parallel ports. It is not a bus that’s strictly necessary for the operation of a computer or embedded device, and instead its importance depends on what devices are being hung off of it. Along with legacy I/O devices, the second most common device type to hang off of the LPC is the boot ROM/BIOS– owing to the fact that it’s a simple device that needs little bandwidth – and this is where the C2000 flaw truly rears its head.
As Intel’s errata succinctly explains, if the LPC bus breaks, then any system using it to host the boot ROM will no longer be able boot, as the system would no longer be able to access said boot ROM. The good news is that Intel has a workaround (more on that in a second), so it’s an avoidable failure, but it’s a hardware workaround, meaning the affected boards have to be reworked to fix them. Complicating matters, since Atom C2000 is a BGA chip being used in an embedded fashion, an LPC failure means that the entire board (if not the entire device) has to be replaced.
CPU Failures Hurt Intel’s Bottom Line
http://www.tomshardware.com/news/intel-cpu-failure-atom-processor,33538.html
Intel revealed during its Q4 2016 earnings call that unspecified CPU failures have affected its bottom line. The company enjoyed record Q4 revenue and a record $59.4 billion in overall 2016 revenue, but CPU failures reduced the Data Center Group’s financial performance.
Tomi Engdahl says:
Intel Atom chips have been dying for at least 18 months – only now is truth coming to light
Vendors clocked C2xxx flaw when returns spiked
https://www.theregister.co.uk/2017/02/07/intel_atom_failures_go_back_18_months/
Exclusive The flaw in Intel’s Atom C2000 family of chips has been vexing Intel’s hardware customers for at least a year and a half, according to a source at one affected supplier, but it wasn’t immediately obvious that Intel’s silicon was to blame.
The well-placed insider, who spoke to The Register on condition of anonymity, said the problem – which results in bricked systems – became apparent to engineers at product makers when the return rate on gear spiked about 18 months ago.
It took additional time to figure out that Intel’s hardware was responsible.
In January, with the publication of errata documenting the flawed LPC clock outputs in its Atom C2000 system-on-chips, Intel acknowledged the issue publicly. Around the same time, chief financial officer Robert Swan said profitability during Intel’s fourth quarter last year had been constrained by an unspecified product issue, prompting the company to establish a cash reserve to deal with the blunder.
Cisco’s subsequent publication of an advisory warning that a variety of its products could fail after 18 months of use because of a faulty clock component
Companies that appear to be affected by the Atom C2000 flaw include not only those mentioned in our previous report – Aaeon, Dell, HP, Infortrend, Lanner, NEC, Newisys, Netgate, Netgear, Quanta, Supermicro, Synology, and ZNYX Networks – but also ASRock Rack (C2550D4I and C2750D4I), iXsystems (FreeNAS Mini), Seagate (NAS Pro), and Sophos (UTM firewall SG 125).
Our industry source suggested that non-disclosure agreements with Intel prevent companies from releasing or discussing information about technical issues related to Intel products.
Tomi Engdahl says:
Intel® Atom™ Processor C2000
Product Family Specification Update
http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/atom-c2000-family-spec-update.pdf
Tomi Engdahl says:
Intel’s dying Atom chips strike again: Netgear recalls four ReadyNAS, Wi-Fi management lines
Hardware maker offers to replace or repair at-risk kit
https://www.theregister.co.uk/2017/03/03/netgear_recalling_hardware_with_bad_intel_atoms/
Netgear on Thursday said that four of its product lines may experience “higher-than-normal failure rates” and that it is contacting affected customers to assess whether replacement or repair is appropriate.
“While this situation is not an immediate cause for concern about the operation of any of these models, we understand that customers might have questions,” the company said in a statement. “We will be reaching out to all registered customers of the affected models over the next several months to determine the appropriate actions for their situations.”
The four product lines – ReadyNAS devices RN3130 and RN3138, and Wi-Fi managers WC7500 and WC7600v2 – happen to have a common component, Intel’s Atom C2000 line chips, which Intel has acknowledged had problems until recently. As many as 16 specific models in this group of four lines may be affected.
Service Note for RN3130, RN3138, WC7500, and WC7600v2
https://kb.netgear.com/000037344/Service-Note-for-RN3130-RN3138-WC7500-and-WC7600v2
Tomi Engdahl says:
Other vendors and service providers that have used Intel’s Atom C2000 chips and appear to have been affected include: Aaeon, ASRock Rack, Checkpoint, Cisco, Dell, Fortinet, HP, Infortrend, iXsystems, Online/Scaleway, Lanner, NEC, Newisys, Netgate, Quanta, Seagate, Sophos, Supermicro, Synology, and ZNYX Networks.
Source: https://www.theregister.co.uk/2017/03/03/netgear_recalling_hardware_with_bad_intel_atoms/