Implement a system to update the various firmwares easily and safely
Open, NormalPublic

Description

People will update the firmwares multiple times, so this must be a safe and easy procedure.

  • os
  • fpga bitstream
  • remote control firmware
  • sensor interface board fpga code

All those would in an ideal world be updatable using an sd card with firmware binaries in root of the card, ready for update.

  • checksum ?
  • how to make it failsafe?
  • revert to previous version

Influence on remote control physical design for example : the remote control must have the appropriate connections to be updatable from the camera (jtag or whatever is needed).

Can all this be done?

philippej updated the task description. (Show Details)
philippej raised the priority of this task from to Normal.
philippej added a project: AXIOM Beta Hardware.
philippej added a subscriber: philippej.

Isnt it the simplest approach to provide a single ISO image for the operating system SD card (containing FPGA bitstream as well)?

I assume there are lots of well established tools (also GUIs) for all popular operating systems to write an ISO image to an SD card?

The only problem I see here is that any custom settings/user files need to be placed outside of that card...

Yes, sure, but if you want to update only the remote for example, you won't like to download a 300 mb file just to upgrade a 64kb firmware.

I created this task also for the implications on the hardware-side : let's not forget what connectivity is required to burn the firmware on various chips. We don't want to ask everyone to have a complete SDK (often including expensive cables) for every chip we use. Think like the USB bridge on the arduino.

The AXIOM remote would have its own micro SD card and software image I assume.

Bertl added a comment.Nov 27 2014, 3:23 PM

Presuming "ISO" refers to ISO 9660/ECMA-119 the compact disc filesystem.

First, this filesystem is not very well suited for SD cards for several reasons, so writing one to an SD card wouldn't be a good idea.

Secondly, we want to update certain portions individually, as for example custom settings and preferences should be retainable.

Finally, we need to allow "secure" (i.e. signed) updates for folks who do not want to test/develop their own stuff, to prevent malware from destroying the camera hardware.

Best,
Herbert

Personally I like the idea of a single button upgrade from an SD card. But like Philippe said I don't like the idea of writing 300MB to update 64 kb. Moreover, a "single button push" upgrade can be dangerous. I don't want to upgrade my camera by mistake.
But connecting it through USB to my PC and using a program to do the update is something I prefer. Like that I know that i cannot use the camera during the upgrade and I won't do any mistake (or at least less), I don't need that the camera is running to give be information about the upgrade being made.

I developed/used update systems a few times. My experience is:

  • Forget about JTAG connectors. It is way to clumsy for an average user
  • Be very careful when a micro-controller/processor has to update itself. You don't want to brick your system because you lost power or connection (or user mistake)
  • Update must be modular. You can use on board partitions or IDs in the header of binaries in order to identify for which part there have to be used.
  • a small terminal like interface is useful for maintenance. You can at least try to find out what went wrong.
  • Using a dedicated micro-controller for system management is a good idea. It can acts as a JTAG interface/USB bridge... interface. It does not have to be upgraded at all (unless...). I had good experience with ATMEL AVR32 chips and PLCXpresso cards. By the way EZ-USB FX3 usb controller from cypress has a pass through interface.
  • You need at least a CRC in your images and preferably a signed key also. Integrate in it all the information of the chips you will upgrade. You don't want to load a firmware for the wrong hardware revision.

Benoit

Hi Benoit Callebaut

Very good feedback, we will definitely consider your suggestions!
Would you be available to help implement some of them?

Yes of course.
Once you have more detailed information I would be glad to help implements/discuss concrete design.

sebastian assigned this task to Bertl.EditedMay 1 2016, 10:40 AM

As discussed in the last days lets set up an rsync server and test the firmware update procedure over the internet concept.

The idea is to have a script inside the camera that you can run to update your camera automatically (with rsync) to the latest version when its connected to the internet. Also you can specify particular versions to update to (developer, stable, nightly, etc.) and we can maintain a complete firmware image on a central server that developers can commit to.

Since it can be a problem if the update procedure is interrupted we will also provide a "golden" image that provides a minimal image that enables the AXIOM Beta to boot and run. Since all the firmware is stored on a microsd card any broken firmware can also easily be replaced when plugging in the microsd card in a pc and restoring the firmware there.

mithro added a subscriber: mithro.May 22 2016, 7:12 AM

On the topic of firmware management, as you have multiple different images parts and versions, it is really important that you embed as much version information into each part as possible.

Embedding things like the git-info really makes it easier to understand how this system was built. Ideally you want *all* your firmware to be reproducibly buildable. See https://reproducible-builds.org/

For the HDMI2USB project, this is the version information we currently output;

HDMI2USB>version 
hardware version info
===============================================
           DNA: 013b226b831ac479

gateware version info
===============================================
      platform: opsis
        target: HDMI2USB
      revision: b50566518c649eec8f79c1b9e4ceecaa5dbf08a9
misoc revision: 61a14086

firmware version info
===============================================
    git commit: b50566518c649eec8f79c1b9e4ceecaa5dbf08a9
    git branch: master
  git describe: v0.0.0-699-gb505665-dirty
    git status:
    --
     M ../../third_party/liteeth
    --

         built: Apr 17 2016 06:40:30
-----------------------------------------------

Things that I think are missing here include;

  • toolchain versions used to compile stuff
  • info about who did the building
  • repository links

We also have embedded EEPROMs on the Opsis boards containing things like the git revision the PCB was made from. With your "always upgradable" model, I think it is really important you do something similar with *every* part. This allows software/firmware to check that it is running on hardware it expects (and allows things like dynamic reconfiguration). Nothing kills time like your users trying to use some feature on a board which doesn't have it. I designed a little protocol for our TOFE boards here -> https://hdmi2usb.tv/tofe/EEPROM.html -- It's based on a similar design to what the RPi Hats use but designed to be a lot more compact (suitable for 128 byte EEPROMs). As USB has shown us, self describing hardware is awesome.


On the process of upgrading firmware; all the Xilinx parts support "multiboot". This is where if firmware fails to load correctly, it will boot a backup image. This can make flashing safer because if things go wrong, you just end up booting into the backup image. See info like http://www.wiki.xilinx.com/Zynq-7000+AP+SoC+Boot+-+Multiboot+Tech+Tip

All our plugin modules have an EEPROM on the I2C bus.
Recent Power Boards feature en EEPROM on the I2C bus as well.
The Main Board can be uniquely identified via PICs and MachXO2s.

On the AXIOM Beta, the MicroZed is configured to boot entirely from SD card, so going back to the old firmware is only a matter of replacing said card.

Thanks for the input though,
Herbert

And I think we all agree that its important to clearly version releases and label everything accordingly so any release can be tracked/reproduced.
With rsync we can just have a "latest" repo plus repos with increasing version numbers in case anyone wants to try a specific version or downgrade to test/compare something.

Maybe this is the right time and place to suggest NixOS.
It is not used primarily in embedded devices and is also not very enduser friendly right now, but has some advantages like atomic updates (what would help here to easily roll back updates) and declarative configuration.
It is build around the Nix package manager, that can be also used on other Linux distributions and even OS X (theoretically also on Windows, but no one is working on that). They are working on reprodicible builds.
The community is very active and helpful! I use NixOS on most servers at home.
You can find my OS configurations here: https://github.com/davidak/nixos-config

Here are some resources for NixOS on ARM: