Cisco C9800-CL sits idle at GRUB Loading Stage2…

Cisco C9800-CL sits idle at GRUB Loading Stage2…

I have been using the Cisco Catalyst 9800-CL (Wireless Controller for cloud) for a while now. Recently, I accidentally powered off the wrong VMWare server, resulting in a wireless disruption. Priority 1 at home! And of course, just before I had a session with Shawn preparing for CiscoLive Barcelona…

After restarting the vSphere server, my C9800-CL wasn’t booting up, with a message: “GRUB Loading stage2… ” And it just sat there, for minutes..  Eventually, during the WebEx Call, I managed to fix it and got my controller back up and running. 

Steps to fix the issue

These are the steps that I used for fixing this issue.

First, power off the VM in vSphere. We need to change some settings in the BIOS.

Next, go and select “Edit Settings” of your VM and click “VM Options” at the top to view some advanced settings and click “Boot options” open. Change the Boot delay to “8000” milliseconds, so that you have enough time when you boot the VM.

Hit Save after you have changed the settings.

Just to make sure, open the settings of the VM again, and click open the first CD/DVD Drive. Check that there is an image named “_deviceImage-0.iso” and that it is connected at powerup.

When I used the vCenter convertor to move the VM off to a new server, I found that this iso wasn’t copied with the controller and it is needed.

Hit Save when you know the ISO image is there.

Follow the next steps to get the C9800-CL booting up again

  1. Open up the console of the VM in the browser (it saves you time)
  2. Power on the VM
  3. Once the Bios is shown, click in the console and hit “ESC
  4. The boot order menu is shown, like the image on the left
  5. Scroll down to highlight “CD-ROM Drive” 
  6. And hit “Enter
  7. Now the VM will boot normally and your controller will start as expected.

Summary

It seems that Grub (the bootloader on the first disk) is not configured correctly the C9800-CL, which leads to a VM / Appliance that is not booted because it cannot find any kernel to load. By selecting the CD image, the right bootloader is selected and the controller is started with the correct configuration. 

I do assume this is a caveat/bug in the Cloud version and will be fixed in a newer release. I do hope you can use this info to fix your C9800-CL deployment sooner. 

FDM Application fails after upgrade

FDM Application fails after upgrade

This is just a quick blog post for those that might have FDM issues after upgrading your FTD software.

I have recently updated my Firepower appliance from 6.5.0 to 6.5.0.2. One of the reasons to update is not only that 6.5.0 is a .0 release, but also that I noticed some failed rule-update deployments that set snort to block all traffic.

Unfortunately, after upgrading, FDM reported an error that it could not be launched with an application failure error. The suggested action was to remove the manager, add a new local manager and begin from scratch. This is the error: “The Firepower Device Manager application cannot be opened. Please try again”

While googling for a possible caveat of this behavior on 6.5.0.2, I came across a caveat in 6.2.3 that has the same behavior. 

That caveat has supported me in fixing my solution. What I did was executing the following commands:

 

> expert
**************************************************************
NOTICE - Shell access will be deprecated in future releases
         and will be replaced with a separate expert mode CLI.
**************************************************************
admin@na-grm-ftd01:~$ sudo su -
Password: 
root@my-ftd01:httpd# cd /ngfw/var/cisco/ngfwWebUi/
root@my-ftd01:ngfwWebUi# ls -a
.   .bootstrap-failed  clifile    deploy                      ha_pkg  lina_cli_sqlite_stores   pjb_output  sslCiphers  variables.ftd_onbox
..  bin                clisyncer  ftd_onbox_6.5.0.2_previous  libs    ngfw_onbox_bootstrap.sh  sru         tomcat      version

root@my-ftd01:ngfwWebUi# rm .bootstrap-failed 
root@my-ftd01:pmtool disablebyid tomcat
root@my-ftd01:pmtool enablebyid tomcat

Basically, you go into expert mode, find the tomcat directory used for FDM and then remove a status file and try to restart it.

With me, this worked and helped me get back access to FDM. Should you run into issues with FDM after an upgrade, this “hack” might help you.

Disclaimer: You are entering expert mode of FTD, it means you can DESTROY your FTD configuration and box. Be aware of what you are doing and make sure you have a backup. 

Hitting 100% CPU after restore on Firepower Management Center

I recently purchased a new microserver to reduce my power footprint at home. And I had to move the FMC (Firepower Management Center) from the OpenStack deployment that I previously ran at home. However, I ran into several bugs in OpenStack that were fixed in later versions, but I couldn’t upgrade because of another bug. Essentially I hit a catch-22 and had to deploy a new FMC on that new microserver and use a restore to move the data and policies. In that process I did hit a bug for which I’d like to share some info on. (more…)