OTA Firmware Updates Bricking Devices? How I Design Fail-Safe Updates
One bad over-the-air update can brick a whole fleet. How I build update systems that always fall back to working firmware.
The problem clients bring: "An update went out last week and a batch of devices never came back. We are scared to ship the next fix." Without safe updates, every bug fix becomes a gamble.
What is really going wrong
- Firmware overwrites itself in place, so a power cut mid-update leaves nothing bootable.
- The new image is never verified before it runs.
- There is no way to tell that a new version is broken and roll back automatically.
- Updates are pushed to every device at once.
How I solve it
- Use two firmware slots: the update writes to the inactive slot while the device keeps running.
- Verify size, hash and signature before switching.
- Boot the new image on trial; if it does not confirm health within a set time, the bootloader rolls back.
- Roll updates out in stages, starting with a small group of devices.
- On Linux products, apply the same A/B approach to the whole root file system.
What the client gets
The confidence to ship fixes often, because the worst case of a bad update is an automatic rollback, not a support crisis.