Home Blog Cisco Device Upgrades

Blog

Mar 7
Cisco Device Upgrades
Posted by Lee Jolly

Upgrading Cisco software (IOS, IOS-XE, ASA code, etc.) can be a little daunting sometimes.  For the most part, installations are straight forward, but there is usually a fair amount at stake.  Ideally, your environment should be designed with high-availability in mind, allowing you to perform maintenance (and endure unplanned outages) without business impact.  But that isn’t always financially possible, especially for remote sites and smaller branch offices.  This makes things especially difficult for the engineers tasked with performing this maintenance remotely.  Losing access to a device and watching your continuous ping not receive replies for 5, 10, 15 minutes or even longer sometimes is always taxing!  That said, they almost always come back, usually just shortly after real panic starts to set in.  While we can’t be 100% sure everything is going to work every time (we didn’t write the code or design the hardware), if we do our due diligence we can increase our chances of success. 

As part of LookingPoint’s Managed Services we support hundreds of Cisco devices across many organizations, which has given us the opportunity to hone our process for these upgrades.  Occasionally we track certain bug fixes or features in specific IOS versions, in order to solve certain issues or enable some specific feature, but short of that, the process I’ll describe below is what we typically do for all upgrades performed. 

On-site console access

We routinely perform upgrades remotely, to devices all over the world, with no on-site personnel available.   I am not trying to strike fear in you, as I’ve said, the vast majority of the time, the upgrades go flawlessly.  But we always recommend having someone available at the site with a console cable, a laptop, and some way to get internet access for a WebEx session.  If any downtime beyond the successful reboot and reload is unacceptable, it’s imperative to have someone on-site.  With a console session, as almost any software related issue can be overcome. 

Invariably there are challenges getting this console access if the on-site resource does not have a technical background.  A lot of new Cisco devices have a USB cable for console that requires a special driver install.  Typically people use the older console port which requires a serial to USB adapter, since very few laptops actually have a COM port.  Some of these Serial to USB adapters install without needing additional drivers and reboots and others don’t.  For these reasons, and others (typically availability of an on-site person), we rarely have this safety net.  Doesn’t stop us from recommending it though! 

On-site console access is a very nice safety net in case things go wrong, but we can work without it.  If someone is available, make sure they are prepared before- hand.

Version Selection – Cisco Recommended Release

There is no single right answer for the version you select to run.  It would be nice if you could just click update and be done, but unfortunately, it’s not that easy.  Within the last couple years, Cisco has added a recommended release marked with the ‘star’ which they describe as “Cisco Suggested release based on software quality, stability, and longevity”.  That sounds great, and is the best version to install 90% of the time (or more), but occasionally, as is the case in this screenshot below for the ISR 4331, there are multiple recommended versions.  Sometimes, as in the case of Firepower on the ASA (also below), there are no versions recommended at all.

iOS Image 1.jpg

iOS Image 2.jpg

If there is only one recommended version, we try to end up there regardless of what is already installed on the device.  Just as we have a fairly large pool of devices and experiences, so does Cisco but on an even larger scale, so it is worth taking their suggestions.  That said, we still read the release notes and do all the checks we’ll touch on later, but our goal is to install the recommended release.  If there are multiple recommended releases such is the case with the ISR4331 above, we try to minimize impact by trying to stay a similar train, and if that’s not possible, we read both release notes and make determinations based on that.  Same for devices that do not recommend a version.  The Firepower screenshot above, for example, requires a LOT of reading.  6.2.2.2 is the highest release number, but 6.2.0.5 is listed under ‘latest’.  Why?  We have to read it all to know and decide. 

Cisco’s recommended releases are a great starting point, but you still must read the release notes and decide for yourself.

Version Selection – Release notes

The release notes are great.  They are thorough, so can be slightly overwhelming, but are packed with great information.  These release notes for the ISR 4331 Denali-16.3.6(MD) for example, has critical information about migrating to Denali from IOS XE 3.x and when ROMMON upgrades are potentially needed to support it.    There is so much to read that I can’t focus you on just one piece.  Caveats, limitations and restrictions, and new features are all important.

iOS Image 3.jpg

Essentially, we are looking for stability above all else.  So pay particular attention to caveats and potential bugs that can be introduced.  The current open caveats for this Denali version are listed here:

iOS Image 4.jpgOf these, they are all pretty specific to technologies not a lot of our customers use.  One in particular I would read further on is the bug that caused the SA creation failure.   I wouldn’t rule out an upgrade unless it was very specifically affected the caveat. 

Read the release notes!

Image Copy

Depending on the device, the new image should be copied to flash:, bootflash:, disk0:, etc.  That location should be checked for available disk space before the image is copied over.  Really old Cisco devices had such limited space on flash that it was VERY common that you could not fit the new image without first removing others.  Some devices were so constrained and the newer images got so large, that you actually have to delete the currently running version!  Pretty scary really.  Fortunately devices come with a lot more space now, but this should still be checked.

Once the image is copied to the device, you should verify the hash against the MD5 or SHA512 hash against what is documented on the downloads page.  I know you never verify a windows executable you download, or drivers, or anything else really, so this seems like overkill.  But when you copy a file over TFTP and see how slow it is and you sometimes see the ASCII progress status go “!!!!!!..!!!.!!!!!!” instead of the very soothing “!!!!!!!!!!!!!!!”, you really want to make sure the file integrity is good.  It’s quick and easy.

iOS Image 5.jpg

Verify the image, don’t be lazy

Pre-upgrade Device State

Taking a config backup is absolutely necessary.  Our customers have device backups through our monitoring tool, but I still take a backup myself before I do any upgrades.  On devices such as the ASA that store password encrypted, I run: more system:running-config to get the passwords in plain text for easier recovery.  I also run: show ip interface brief.  I like to see a status of the ports, SVI’s, etc. in case something doesn’t come back up correctly that is attached to this device.  If the device is a voice gateway, I like to see the status of the PRI’s.  Sometimes customers migrate away from a certain PRI and you end up troubleshooting a T1 that is not supposed to be working in the first place. 

Take backups and log the session as you collect pre-upgrade device state info

Set the boot image (or install the software package) and reboot!

Each product’s installation procedure is different and this blog is more about selection and verification, rather than a step by step guide.  Actually, the release notes typically contain the real step by step instructions and should be followed.  There are very specific procedures for updating 3850 switch stacks, ASA’s in failover pairs, etc.  Each one is alike but different.  All the more reason to be thorough and read, read, read.  Or let us do it! 

Learn More About LookingPoint  Next > Care Managed Services

Written By: Lee Jolly, LookingPoint, Inc. Systems Engineer VCP CCNP

Written By:

subscribe to our blog

Get New Unique Posts