create firmware image creation script
Closed, ResolvedPublic

Description

Draft bash creation script - far from complete. Very much a draft and work in progress currently - some parts just pseudo code yet.

#!/bin/bash

#definitions

NAME="AXIOM Beta"
VERSION="0.1"
BUILD_LOG=build.log


#Setup

# install depencies
# lets start with Ubuntu as main host build system and expand it for other distros later on



sudo apt-get install build-essential git
sudo apt-get install libc6-armel-cross libc6-dev-armel-cross
#sudo apt-get install binutils-arm-linux-gnueabi
#sudo apt-get install libncurses5-dev
#sudo apt-get install libc6-armel-cross libc6-dev-armel-cross
sudo apt-get install gcc-arm-linux-gnueabi


mkdir build
cd build/


# acquire arch linux
# see https://archlinuxarm.org/platforms/armv7/xilinx/zedboard

#TODO
#wget http://os.archlinuxarm.org/os/ArchLinuxARM-zedboard-latest.tar.gz
#bsdtar -xpf ArchLinuxARM-zedboard-latest.tar.gz -C mnt

#TODO uboot
#TODO acquire kernel from 13thfloor
#TODO disable pacman kernel updates in arch, see: https://www.digitalocean.com/community/tutorials/pacman-syu-kernel-update-solved-how-to-ignore-arch-kernel-upgrades

#TODO acquire PIC bitstream from 13thfloor
#TODO acquire FPGA bitstream from 13th floor




#collect sources

git clone https://github.com/apertus-open-source-cinema/beta-software.git

# TODO cross compile all beta-software for ARM CPU in microzed

cd beta-software/mimg/
CROSS_COMPILE=arm-linux-gnueabi- make



# TODO copy software binaries to system directories /usr/bin?
# TODO copy beta scripts 
# TODO copy source code to system directories (where?)


#TODO remove build dir after everything was built successfully
#rm -r build


# package

#TODO dd -> iso file for micro sd card
davidak created this task.Nov 10 2016, 11:29 PM
davidak updated the task description. (Show Details)
davidak raised the priority of this task from to Needs Triage.
davidak added a project: AXIOM Beta Software.
davidak added a subscriber: davidak.
sebastian updated the task description. (Show Details)Mar 2 2017, 2:25 PM

Good solution for this thing is to install Jenkins, e.g. through Docker, and let it build the packages for you. Either manually or fully automatic, e.g. nightly builds. Jenkins is not limited just to building, it's a batch task machine. So you could give it all the tasks which should produce repeatable results, like firmware images, OC nightlies, unit testing.

CI definitely sounds like a good plan for this task, +1 from me

Just a little note:
As it is currently built with archlinux, I would stay on this system until it's finished and have another iteration for switching the system (if needed). This way we could implement CI in "no time" without having to look at changing packet(dependencies). No time = 1 work day

Suggest main-tasks for this ... task:

Btw, I root for travis, as we get it for free on github: https://travis-ci.org/

Bertl added a subscriber: Bertl.Mar 3 2017, 2:00 AM

Here a few quick comments far from being complete:

  • We use an almost unmodified kernel branch maintained by Xilinx (git repository) with a custom devicetree binary.
  • We are currently using an unmodified Arch Linux on the Beta.
  • U-Boot is customized and needs to be combined with the Xilinx FSBL (see T37).
  • The PIC code is currently compiled on and programmed directly from the Beta.
  • The Bitstreams for the Lattice FPGAs still required Lattice Diamond to build. .- The Bitstreams for the Zynq (yes there are several) require Vivado to build.
  • There is no initramfs, the OS is booted directly from the SD card (second partition)
  • FSBL + U-Boot, uEnv.txt and kernel are on the first partition together with a golden version.
  • C tools are currently compiled on the Beta (currently /opt/BERTL) and would optimally become Arch packages.
  • Scripts are currently stored in /root and should also get packaged up (maybe subpackages).
  • The SD card has three partitions, first one needs to be VFAT for the boot process to work, second one is the rootfs and the third one has all the sources.
  • The Zynq has two ARMv7 cores with NEON-SIMD and up to 1GB shared DDR memory.

Feel free to ask for more details and I will try to add them and more whenever I find the time.
For quick questions, IRC is the way to go.

Thanks and good luck,
Herbert

Added by @sebastian:
Beauty of this task is it's modularity. E.g. we could add the fpga bitstreams as binaries in the beginning.

sebastian triaged this task as High priority.May 23 2017, 7:55 PM

raising to high priority because this is the main blocking task for the entire firmware not getting developed any further

Angel added a subscriber: Angel.Jul 23 2017, 1:00 AM
jatha added a subscriber: jatha.EditedJul 25 2017, 2:48 PM

I think travis wouldn't be the optimal solutions for firmware builds, because we need some special tools with proprietary licenses for building the fpga bitstreams and more build time than the 50 minutes travis gives you for free. Imo http://concourse.ci would be a good solution, because it would allow us to create quite modular pipelines for the firmware.

generally i like https://drone.io/ as a CI. it executes the build in docker containers, so you have a clean invironment every time but it is as easy as Travis. it is open source, you can install it on your own server that has docker (it runs in a docker container itself). they currently don't have a public instance, but will privide paid services in the future and probaply free for open source projects like in the past.

i havn't read this ticket yet but will do so and think about it. maybe i can help here :)

What about gitlab-ci? I have an ansible playbook for setup so it would be doable in minutes as long as someone provides a server. We could mirror the github repo and have gitlab as (more or less) transparent tool in the middle.

@maltefiala GitLab is a good software, so thair CI is probaply also good. i never used it...

Maybe we an use the hosted version at https://gitlab.com for CI?

Good idea. Let's wait for @sebastian to create an account I guess. Additionally it seems as if gitlab.com allows the EE feature "repository mirroring": https://docs.gitlab.com/ee/workflow/repository_mirroring.html

jatha added a comment.Jul 25 2017, 6:22 PM

gitlab ci is indeed a good idea :). the mirroring from github is also possible in the public instance but very bugy. to work around this we can create another repository with gitlab ci, which only pulls from github and pushes to gitlab, and is triggered by a github webhook. I did exactly this setup for another project and it works very well...
(https://github.com/freifunkks/site-ffks/tree/beta) the mirror repo can be found here: https://gitlab.com/freifunkks/mirror-scripts

i can help creating this setup, when somebody gives me credentials or i can help somebody else to set things up :)

RexOr added a comment.Jul 25 2017, 7:14 PM

Can you message me your email address please Jatha?

jatha added a comment.EditedJul 25 2017, 9:29 PM

The beta-software is now mirrored to https://gitlab.com/apertus/beta-hardware. Wen somebody who's more into building the software manually could write a .gitlab-ci.yml and add it to the GitHub repository the builds would start automatically on every commit.

PS: here is the build badge
[![build status](https://gitlab.com/apertus/beta-software/badges/master/build.svg)](https://gitlab.com/apertus/beta-software/pipelines)

@RexOr please don't use apertus-open-source-cinema as name, it's toooooooooo long, nobody wants to type that in! you could use it as display name or description...

can we create a group on gitlab to give multiple people permissions like with organizations on github?

at this point of time this would be some more work. because the gitlab is only a mirror of GitHub, the repo doesn't need any maintenance, so I would vote for keeping the gitlab account a "person".

it will be way more work later. also i would like to have apertus as name on github too.

GitHub answered that the apertus organisation is not dormant but has only no public activity. Maybe we should decide an alternative short handle.

RexOr added a comment.EditedJul 26 2017, 7:36 AM

Often there's someone out there who gets into platforms and acquires the name apertus before us.

Kept things consistent with GitHub but I agree the URL is too long. apertusOSC or apertusOSCinema would be good but will leave that to Sebastian as I don't have permissions. Jatha should probably manage that area in the meantime as he's more familiar with things.

anyone up to create the mentioned .gitlab-ci.yml file?

jatha added a comment.Aug 14 2017, 9:48 PM

I began to do this with your help...

jatha claimed this task.Nov 1 2017, 10:54 AM
RexOr added a comment.Nov 1 2017, 7:22 PM

Any progress on this front?

jatha added a comment.EditedNov 2 2017, 9:52 PM

Yes! Just now i created this (https://github.com/apertus-open-source-cinema/beta-software/pull/7) pull Request. If you want to bring the work on this forwards, you could test this (https://gitlab.com/nein/beta-software/-/jobs/38788789/artifacts/download) image and report your findings. edit: bertl is already doing this.

Wow, that's great progress you made here!

PR done.

@Bertl created the original partition structure so I am sure he can comment on the point:

  • Only 2 partitions: one boot partition and one mixed OS/tools partition. I think this is handy during development and avoids a lot of fussing. But we have do discuss this
sebastian added a comment.EditedNov 3 2017, 10:23 AM

This is the current push URL of the github webhook:

*removed*

We do not have a "secret" in place yet and the REF_NAME looks like it needs to be replaced.

jatha added a comment.Nov 3 2017, 2:47 PM

@sebastian could you please remove the URL, as it shouldnt be public ;)

Question: I am trying to download this:

https://gitlab.com/apertus/beta-software/-/jobs/43672726/artifacts/download

But keep on getting a 503 - forbidden after downloading several hundred MB...

503 is not forbidden but service temporarily not available. maybe gitlab has a temproary outage (they sometimes do ;))

sorry I meant 403.

HTTP-Anforderung gesendet, auf Antwort wird gewartet ... 403 Forbidden
2018-02-21 15:13:53 FEHLER 403: Forbidden.

Downloading fine here. Speed significantly slowed down in the middle, but finished fine, after accelerating again.

Maybe something related:

https://stackoverflow.com/questions/36298959/gitlab-issuing-temporary-ip-bans-403-forbidden

In the meantime is there any place I can get a copy of a working image? I want to build a VM...

Did you get the image on IRC?

BAndiT1983 added a comment.EditedFeb 28 2018, 10:57 PM

@jatha Have you set Travis CI up?

Edit: Seems so, from chat. Is it possible to add Build for PRs, for quick tests before merge?

jatha added a comment.Feb 28 2019, 9:35 PM

we are much further now and this can be closed imho.

sebastian closed this task as Resolved.Feb 28 2019, 9:46 PM