Restoring firmware on an EdgeRouter Lite

Ubiquiti’s EdgeRouter Lite is decent piece of networking hardware, but the routers often encounter issues with storage after a few years. The internal USB key “forgets” some of the data, and then the router doesn’t boot anymore.

This post is about restoring the firmware the easy way – no need to muck around with TFTP or serial consoles. It’s really just documentation for future me.

I believe these instructions work for the EdgeRouter PoE 5 as well, but I have not personally tried to recover that device.

What should I have done before my router stopped booting?

Save a copy of your config! If you have that, it’s really easy to recover in the future. The web UI’s config download works just fine.

You might be able to retrieve your config from the old flash drive, but I wasn’t able to the two times that the key became corrupt. It’ll be in the /w/ directory of the filesystem.

Wait, a USB key?

Yes, the internal storage is just a USB key. The key itself is super small to fit inside the case. You have a little more room if your ERL is the modern rectangular box instead of the chamfered box.

The USB key supplied with my 2013 ERL has lost data twice now. The first time this happened, I reflashed it and moved on with my life. After a year (and one power outage), it lost data again and I switched to another USB key.

To access the USB key, you have to take the ERL apart. There are two Phillips screws on the bottom. After removing the screws, hold the bottom and slide the top case over the Ethernet jacks.

What USB keys work as a replacement?

You’ll want something physically small that can store at least 2GB. If the key is wider than a USB port, it’s not going to fit. The Kingston DataTraveler SE9 USB 2.0 (pictured in the middle) is physically about the right form factor.

I’ve also successfully used a SanDisk Ultra Flair USB 3.0 (pictured at the right) key since it was cheap on Amazon. It barely fits, thanks to the black loop at the end, and also required an additional bootloader tweak to make work.

I’d avoid buying a USB 3.0 key for two reasons. First, the internal interface only supports USB 2.0 so there’s no point getting something fancy. Second, my totally unsupported theory is that USB 3.0 keys take longer to enumerate, causing the bootloader to think there are no USB keys connected and requiring the tweak mentioned above.

If you really want to be sure the replacement key won’t die, you could buy an industrial-grade key from somewhere like Digi-Key or Mouser, but I’m not sure it’s worth it.

I have a USB key, now what?

There are two main ways to restore the firmware. The method I suggest below uses your computer to format and prepare the USB key.

The other, original, method is the EMRK (EdgeMax Rescue Kit) which is kinda fiddly and requires a serial console for your ERL plus running a TFTP server. This guide is pretty comprehensive. This Ubiquiti forum post was the original guide.

Putting a fresh EdgeOS system on a USB key

I’ve only done this from a Linux system (or VM). Plausibly you could make this work on any system that can partition the USB key and make an EXT3 partition.

I suggest using mkeosimg (forum thread). The repo has instructions on how to use the tool. It’ll let you do stuff like flash a USB key with the latest EdgeOS and your configuration backup. Hopefully you made a configuration backup.

The first time I restored my router, I used the original script (and submitted a couple patches). While that script still works, I didn’t remember all the correct arguments for the scripts.

I flashed my USB key and the router doesn’t always boot!

Some USB keys take a relatively long time to show up and the router’s bootloader doesn’t detect the key. I think USB 3.0 keys might be more likely to cause this issue but I have no real evidence to support my assertion. You’ll need a serial console (sorry) to tweak the bootloader settings.

Pretty much any “USB to rollover cable” or “USB to RJ45 console” will work. The cable will probably be light blue because that’s the color Cisco used forever ago. You’ll want to set the bit rate to 115200bps. There are good resources on the internet on how to make a serial console work. The pictured cable is serial-only; I have a USB to serial adapter that I plug the rollover cable into.

First, you’ll want to plug your serial cable into the management port on the ERL.

As the ERL boots, you’ll want to break into the bootloader console. Once you have, the following commands (from the EdgeRouter-Recovery repo) will delay the bootloader long enough to let your slow USB key show up. If you’re typing the commands in manually, the first setenv is all on one line even though it looks wrapped here.

setenv oldbootcmd 'fatload usb 0 $loadaddr vmlinux.64;bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@3072k(eeprom)'
setenv bootcmd 'sleep 1; usb reset; sleep 1; $(oldbootcmd)'
setenv bootdelay 0
saveenv