On February 2, 2017 Cisco announced a major issue affecting products from their routing, switching, security and server product lines. Reference Tables A through D below for a full list of affected products and part numbers.
The issue in question is caused by a degradation of a clock signal supplied by a third-party manufactured component within the affected equipment. This degradation of clock signal leads to a failure that Cisco describes as:
“Once the component has failed, the system will stop functioning, will not boot, and is not recoverable.”
This does not sound good in anyone’s book. The good news (if you can call it that) is that this issue does not rear its ugly head until at least 18 months of device runtime:
“Although the issue may occur beginning at 18 months in operation, based on information provided by the supplier, we don’t expect an unusual spike in failures until year three of runtime,” Cisco said in its advisory.
This fact gives us some time to proactively swap out these faulty devices in a controlled manor before this failure has a chance to impact our networks. All Cisco products that have this component have been identified and Cisco has worked with the supplier to put a fix ion place. Cisco is offering to provide replacement products for those products under warranty or covered by any valid services contract, which have this component, even if they have not failed.
LookingPoint has outlined the below two steps to help identify affected products and begin the remediation process.
STEP 1: Do I have equipment impacted by this defect.
Refer to tables A through D below for a list of affected product PID’s (Product IDs) and VID’s (Version ID).
The PID and VID of your devices can be obtained by running thee “show inventory” command from the cli.
Any devices with a VID number that is equal to or earlier than that shown in the Possibly Affected VID column are possibly affected by this issue. Record the serial number of these devices and move onto step 2
The output below shows an example of how we identified one of our LookingPoint lab ISR 4331 routers as being susceptible to this issue.
STEP 2: Complete Cisco Replacement Order Spreadsheet and Submit a Case.
Download the “Replacement Product Order Spreadsheet” and fill out the required 4 fields
- Product ID
- Serial Number
- Requester Name and Email
- Shipping Address
Once complete the spreadsheet will display an impact status confirming the issue and how many months have elapsed since manufacturing.
Send the completed replacement order spreadsheet to:
clock-signal-replacement-request@cisco.com.
Once Cisco receives the request, we will validate that the provided Serial Numbers contain this clock signal component and are entitled to be replaced. Note, you must have a current service contract or the device must have been under warranty dated as of November 16, 2016. A case number will be sent to you to track your request.
Reference Tables
Table A
Table B
Table C
Table D
ISSUE Fast Facts:
Q: Does this issue effect currently shipping products?
No. All products shipping currently do not have this issue.
Q: When did Cisco become aware of this issue?
Cisco became aware of the issue in late November 2016 and have since then been working with the 3rd party manufacturer of the faulty component to identify the scale and breadth of this issue.
Q: Who manufactures the faulty component?
Cisco’s official line is that they do not intend to publicly name the supplier of this component, although internet blogs and articles have been quick call out Intel’s Atom C2000 processor as the cause of this issue.
Q: Does this issue only affect Cisco products?
No, other companies also use this component from the supplier in their products. LookingPoint advises customers to check any non-Cisco products they use for possible issues.
Q: How long will replacements take?
Due to the age-based nature of the failure and the volume of replacements, Cisco will be prioritizing requests based on the products’ time in operation. Therefore, fulfillment requests may be delayed and your request may be fulfilled in multiple shipments.
Q: Where can I find more information regarding this?
Cisco Advisory: http://www.cisco.com/c/en/us/support/web/clock-signal.html#~overview
Cisco Field Notices: http://www.cisco.com/c/en/us/support/docs/field-notices/642/fn64253.html
Cisco FAQ: http://www.cisco.com/c/en/us/support/web/clock-signal.html#~faqs
Written By: Chris Marshall, LookingPoint Senior Solutions Architect - CCIE #29940