diff --git a/site/news/MANIFEST b/site/news/MANIFEST index 0e73538c283f2f82fd0958c44b911e8e2563c59c..26c8f85cf795a0f928b8e49257d24be2268ed240 100644 --- a/site/news/MANIFEST +++ b/site/news/MANIFEST @@ -1,3 +1,4 @@ +audit1.md gnu.md canoeboot20240510.md canoeboot20240504.md diff --git a/site/news/audit1.md b/site/news/audit1.md new file mode 100644 index 0000000000000000000000000000000000000000..03bb7ce617f1bb4fd0716cff52bcdeb3e79ac3ae --- /dev/null +++ b/site/news/audit1.md @@ -0,0 +1,849 @@ +% Canoeboot Build System Audit 1 +% Leah Rowe +% 9 June 2024 + +**A new release is now available, with these changes. Learn more by reading +about the [Canoeboot 20240612 release](canoeboot20240612.md).** + +NOTE: This audit pertains to Canoeboot as of 9 June 2024 but the article is +published on 12 June 2024; nonetheless, the article date is set to June 9th. +For changes up to June 12th, please read the Canoeboot 20240612 announcement. + +Introduction +============ + +Canoeboot is a free/libre boot firmware project. It replaces your +proprietary BIOS/UEFI firmware, on supported x86 and ARM computers. It does +this by providing an automated build system to download, patch and compile +the various upstream sources (e.g. coreboot, GRUB, SeaBIOS). Coreboot is used +for hardware initialisation, configuring everything from your CPU, memory +controller all way to peripherals, readying the hardware so that it can run +software, e.g. GNU+Linux operating systems. You can essentially think of *cbmk*, +which is Canoeboot's build system, as a *source-based package manager*. It is +what the Canoeboot releases are built with. The *cbmk* build system essentially +implements a *coreboot distro*, the same way you might think of a GNU+Linux +distribution. + +Extensive auditing has been performed on cbmk, since the Canoeboot 20240504 +release. These audits fix bugs, reduce code bloat and generally improve the +efficiency of cbmk, adding and removing features in a careful, conservative +way, with a focus on *clean code*. Remember the magic words: code equals bugs. + +This article covers changes from Canoeboot 20240504, up to +revision `4f6fbfde81f5176e5892d1c00627f8f680fd3780` from 9 June 2024. + +Some notes about this audit +--------------------------- + +This is the *first* official Canoeboot audit. The initial Canoeboot releases, +from October/November 2023, incorporated changes from Libreboot Build System +Audit 3, and Canoeboot 20240504/20240510 included changed from Libreboot Build +System Audit 4, plus changes adapted from releases up to Libreboot 20240504 and +a bit beyond; however, Canoeboot used to be synced with Libreboot per each +Libreboot release - nowadays, it is synced *per commit* on both the build system +and the documentation, plus other repositories. + +Therefore, whenever a Libreboot audit is announced, a corresponding Canoeboot +audit will also be announced. To keep it clean, we will simply refer to this +one as the *first* official Canoeboot Build System Audit, or *audit 1*. + +ALSO: Canoeboot 20240510 was released *during* the audit; this changelog is in +reference to Canoeboot 20240504, *not* 20240510. + +Modest code size reduction +-------------------------- + +There are 1054 lines of shell script in the build system, versus 1208 in the +Canoeboot 20240504 release. Canoeboot's build system is written purely in +POSIX sh; not BASH, not KSH, not ZSH, jush sh! + +This is a difference of 154 lines, or a 13% reduction. Despite the reduction, +numerous features have been added and a large number of bugs were fixed. + +Summarised list of changes +========================== + +Changes are in order per category, from newest to oldest: + +Feature changes +--------------- + +* **Download crossgcc tarballs as dependencies, when cloning coreboot.** We + previously relied on the coreboot build system, which automatically fetches + these when running `make crossgcc`, which we run automatically. However, + the coreboot logic has to be patched for reliability because the GNU HTTP 302 + redirect often fails so we use a static mirror, and the logic has no + redundancy. With this new change, we use the same tarballs but we specify + two URLs, a main and a backup. This also means that the tarballs will once + again be included in Canoeboot release archives, enabling offline builds. +* **Support downloading files as submodules, in Git repositories.** This + complements the pre-existing feature where sub-repositories (Git) can be + cloned into a subdirectory of a given main repo. We use this for crossgcc, + as referenced above. +* New files under `config/dependencies/` for Fedora 40 and Ubuntu 24.04. Now + you can run `./build dependencies fedora40` + and `./build dependencies ubuntu2404` on each respective distro, to get + the right build dependencies for building Canoeboot from cbmk. +* **NEVER** run `git submodule update`, *ever*. Instead, rely *solely* on + config/submodule/ to define which dependencies should be downloaded, to each + given subdirectory within a main project. This is using a feature described + later on (in this audit report), whereby projects can have redundant + submodule repositories defined; initially, this feature was an *override* + where otherwise the submodule update command would be executed if + the `.gitmodules` file existed for a given project; this override is now + the *only* way to do it, and is thus the default behaviour. This may be + considered a preventative bug fix, in case certain projects auto-download + submodules that might cause us trouble in the future. It's better that we + maintain tight control of submodules. +* **Summarising the next few changes mentioned below: out-of-source builds + are now fully supported, for both single- and multi-tree projects.** (it was + previously only supported on multi-tree projects) +* Moved builds of coreboot utilities (e.g. cbfstool) to `elf/utilname`, + e.g. `elf/cbfstool/default/cbfstool` would be the new cbfstool binary location + for the one build from coreboot in the `default` tree. +* script/trees: Now single-tree builds are skipped if a build exists + under `elf/projectname/`, based on the presence of a `build.list` file; this + is consistent with the same behaviour pre-existing for multi-tree projects. +* When building memtest86plus, the binary is now placed out-of-source, + into `elf/memtest86plus`. +* When building flashprog, the binary is now placed out-of-source, + into `elf/flashprog/`. +* Use new function `singletree` to decide whether to use submodules, rather + than hardcoding a check for *coreboot* - NOTE: use of submodules was later + disabled during this audit, replaced with custom handling in cbmk. +* For error exits caused by *improper commands* (as opposed to fault conditions + while processing valid commands), don't directly call `err`; instead, call a + newly written function `badcmd` which says that much, and links to the + website (if `docs/` is present as in releases, it also points there). +* Added a `projectsite` file pointing to canoeboot.org, complementing the + existing `projectname` file which contains the word `canoeboot`. This is + used in the `version` command. +* **GRUB is now a multi-tree project.** Each given coreboot target can + specify which GRUB tree it wants to use, each containing its own revision + and patches, with its own GRUB configuration file. This can be used later on + to provide specific optimisations on each given mainboard, but it is used + at present to exclude xHCI patches on boards that don't need it; please also + read the bugfix section (of this audit report) pertaining to this same topic, + for more context. Before this change was implemented, all mainboards used + the exact same GRUB revision, with the same patches and the same config. +* grub.cfg: scan `grub2/` last, on each given device/partition; this speeds + up the boot time in most tests, because most setups use `grub/`, + but `grub2/` is still used on legacy setups so we have to support it and, for + reasons mentioned in the bullet point below, GRUB is very inefficient at + generating the list of devices/partitions when using the `*` wildcard, so + we can't scan `grub*/`. +* grub.cfg: it now scans a reduced set of devices/partitions by default, while + still ensuring (in practise, on real systems) that all such devices and + partitions will be scanned. We hardcode this, because the `*` wildcard in + GRUB is *very slow* on some machines, due to the way the GRUB kernel + constantly re-initialises the list of devices and partitions during operation. + Scanning an *excessive* number of hardcoded device/partition numbers slows + down the boot too, so this has been optimised. It has been tested and it + shouldn't cause any issues on machines/setups that people actually use. +* **grub.cfg: scan distro-provided grub.cfg from ESP;** we previously only + scanned the ESP for isolinux/syslinux configurations (which GRUB can parse). +* grub.cfg: Don't search for `*_grub.cfg` as this slows down the bootup + sequence, and nobody really uses this anymore; Canoeboot's GRUB is much more + robust these days, pretty much booting anything automatically, but you used + to have to (regularly) use a `canoeboot_grub.cfg` file to override the default + one provided by your distro. This legacy cruft has been removed, entirely! +* script/roms: Allow to override `grub_scan_disk` via `-s`, for + example: `./build roms -s nvme t1650_12mb` +* **grub.cfg: Use `grub_scan_disk` to set boot order (rather, boot order by + device type).** It is possible now to configure each mainboard with this + variable, so that certain types of devices are scanned in a precise order; + for example, scan NVMe SSDs first. +* **include/git.sh: Allow manual override of `git submodule` handling**, instead + directly downloading Git repositories using `git clone`, into the subdirectory + of a given main Git repository (as per `src/projectname` scheme). With this + feature, it is possible now to specify a *backup* submodule repository, for + redundancy, all while still allowing to reset the revision (and *patch* the + given submodule). This has been used to provide greater redundancy when + downloading coreboot submodules. It also allows to *limit* the number of + submodules, so now we only download the ones we need, thus saving bandwidth + especially during very large and long build sessions. - *NOTE: this was + later changed so as to be the ONLY method for downloading submodules, skipping + the actual git-submodule-update command entirely, on all projects.* +* **Native NVMe driver added to the GRUB payload**, allowing users to boot from + NVMe SSDs where present on a given mainboard. The patch is courtesy of + Mate Kukri, who ported SeaBIOS's own NVMe driver, converting all of the + code to run properly within GRUB's own kernel. NVMe SSDs are now fully + bootable on all machines that can have them, offering vastly superior + read and write performance when compared to SATA SSDs. +* include/git.sh: Allow patching git submodules (NOTE: support for submodules + was removed entirely, later in the audit, in favour of custom logic in cbmk + for the downloading of such dependencies). +* Added Portuguese keyboard support in the GRUB payload (patch courtesy of + the contributor by alias `samuraikid`). +* Removed all help commands, because it's just a duplication of documentation + that is already included in releases anyway, and people using the Git + repository require internet access anyway, so they can just use the website. +* Main build script: removed the functionality for generating source tarballs + where the only source code included is U-Boot; we do not need this, because + the larger source tarball containing all of Canoeboot also contains U-Boot. +* include/option.sh: Don't bother checking for GNU Tar, because we were only + using it for reproducible tarball generation which didn't work yet anyway; + there are still ways of doing it with BSD tar and so on. +* Print a two-line break before confirming the location of the generated + release archive, when running release builds. This makes it more obvious + to the operator. +* Removed all status checks from script/roms (formerly script/build/roms), + because it's better to document this instead, and rely on testing regardless. + +Bug fixes +--------- + +Some of these changes fix actual issues that were found in testing, while +others were fixed *before* being triggered/reported and are thus *preventative +bug fixes*. The logic in cbmk has been very intensively audited as is customary! + +The changes are, from newest to earliest: + +* script/trees: Exit with error status if a given project is not defined. It + was previously decided that this script could be used to directly run Makefiles + from any given directory, but this is no longer done as it was error-prone; + this change prevents such usage, thereby preventing unstable conditions within + the build system. +* **Create a lock file when running cbmk.** Only do it from the main parent + instance, but not child instances of it; delete it at the end, after exiting + from the parent process. If starting a separate parent process, that one + will now immediately exit (with error status) if the lock file exists. This + prevents the fault condition where the user accidentally runs the same cbmk + instance twice, from the same work directory; it is only designed to be + executed once, per work directory. This is similar to the locking feature + you find in package managers such as apt-get. Also do this in release/ + directories, while building (but don't include a lock file inside the tarball). +* include/git.sh: When doing a global check for files in every project all at + once, as defined by each respective (if existent) `nuke.list` file, hide + the output. Only show the output when running it on a specific project, not + the one in the for loop. This prevents user confusion / false bug reports. +* include/git.sh: Download coreboot as defined by `xtree` *before* downloading + the main project that defined it, to prevent a situation where the main project + is downloaded successfully but not the dependency (defined by `xtree`); this + is to maintain the integrity of the build system under fault conditions. +* include/lib.sh: When a download fails (running the `download` function), + don't then say that the file is "missing". Instead, actually say that the + download failed, so that the operator has a better understanding. +* include/lib.sh: Hide stderr on the `download` function, for the initial + check when verifying an existing file; although no problem existed on + technical terms, the output was confusing because it made the user think + there was a problem. The logic then downloads and re-verifies, and the + output indicating *that* verification has not been hidden; if the file + already exists, this is simply indicated by `e()`. This is considered a bug + fix, because it fixes the bug where users made erroneous bug reports, by + re-engineering the situation so that they do not make such erroneous reports. + TL;DR hide a totally benign (non-)error message. +* include/git.sh: Provide better user feedback about what is being downloaded + and where - although nothing was broken before, this lack of feedback was a + bug because it made debugging harder. Provide more clarity for the user. +* include/git.sh: Download dependencies *before*, not *after*, downloading the + project sources that depend on it. For example, pico-serprog depends on + pico-sdk. If you were to download pico-sdk *after* pico-serprog, the latter + may be downloaded and placed in src/, but then the former (sdk) could fail + due to bad internet, and now the overall downloaded code is corrupt, and there + was nothing checking for this after the fact; checking for it would be bloat. + By downloading the dependency *before*, then if *that* download fails, so + does the main one, and integrity is maintained within the build system. +* Preventative bugfix: don't check empty paths in `copy_elf` (of script/trees), + even though this potential bug was not yet triggered. Play it safe. +* script/trees: Don't check pre-existing builds in elf/ if `build.list` is + missing, otherwise it's too soon and builds are prevented in the first place; + this was caused initially when supporting out-of-source builds for single-tree + projects, as was already done on multi-tree. Now this is fixed. +* Documentation: only define the Untitled Static Site Generator + in `config/git` - the dependencies (markdown files and images) are now + defined in config/submodules/ instead. This prevents the bug where you could + download one of the dependencies first which would make the main project, + Untitled, un-downloadable, since the dependency projects go in subdirectories + of the main project that depends on them. +* Handle serprog dependencies in config/submodule instead of relying on + the git submodule update command, and only provide necessary modules. This + prevents the bug where downloading a dependency first later prevented the + main project from being downloaded, if the dependency was in a subdirectory + of what depends on it. +* Build coreboot utilities on a number of threads as defined by `XBMK_THREADS`; + although they already compiled, they would always do so on a single thread, + which is considered a bug. Now they can be compiled on multiple threads. +* include/lib.sh: Don't use `./update trees -f` to build coreboot *utilities*, + because it's quite error prone and not what that script is designed to do; + it is only designed to operate based on strictly defined single- and + multi-tree projects. Instead, call `make` directly. +* Don't use the presence of a `build.list` file to detect a multi-tree project + when running `./update trees`; instead, check the presence of `target.cfg` + down one level from `config/project/`, so: `config/project/*/target.cfg` + instead of `config/project/target.cfg`. This way, if someone working on cbmk + accidentally adds that `build.list` file in the wrong place, cbmk won't + become unusable. This also means that single-tree projects can now provide + a `build.list` file! (and some of them now do - look at the features section + on this page) +* Move check for *root user* to include/lib.sh, *before* the version/versiondate + files are written; these files need to be writeable by the standard user, + otherwise cbmk will exit. If you run cbmk as root, except when running the + dependencies command, it exits with error status; ironically, that very same + check then prevented running as root-root, causing cbmk to become unusable + until those files were either deleted or had ownership changed. This fix + prevents the bug from occuring ever again, but people who were previously + affected still have to fix these files (if they were written as root). +* Move dependency handling to include/lib.sh, *before* the version/versiondate + files are written, and *exit* before they are written; this prevents writing + the version/versiondate files as root, which previously occured when running + a command such as `./build dependencies debian` (installs build dependencies + from apt-get on a Debian machine). This bug ironically prevented cbmk from + running at all, under such conditions, because the dependencies script + required root, but cbmk exits with error status if running anything else as + root, and if version/versiondate are owned by root, that prevents cbmk from + running because writing to these files is the first thing it does, so an exit + with error status would otherwise occur. +* config/git/: Bump to a newer revision of Untitled (static site generator), + which thereby also imports the same fix as described in the next bullet + point below, because Untitled had (and now no longer has) the exact same bug. +* include/lib.sh: check environmental variables properly, for example + check that `${XBMK_RELEASE+x}` isn't unset; it was previously grepping + the output of `set`, which led to a bug report by a user who had the + variable `TMUX_TMPDIR` set, whereas `TMPDIR` was unset and cbmk was checking + the latter; in this example, the bug caused cbmk to act as though `TMPDIR` + was set, when it in fact wasn't, and code that used it then crashed because + cbmk does `set -u -e` (and it does this precisely to catch such bugs like the + one you're reading about now so that they can be fixed, like this one was!) +* **Re-configured GRUB so that none of the currently supported machines contain + xHCI support**. This is a mitigation against the bug reported in [lbmk + issue 216](https://codeberg.org/libreboot/lbmk/issues/216). This is done, by + using the new *multi-tree* GRUB handling, which is mentioned above in + in the section (of this audit report) pertaining to *feature changes*, whereby + each mainboard can have its own GRUB revisions and patches, with its own + GRUB configuration file (that could be uniquely optimised for it). + We do not need xHCI patches on any Canoeboot machines, but the patches are + free software regardless, and it's important to keep Canoeboot in sync with + Libreboot. It may be that you want to enable it on a custom configuration, + for example if you use a USB3 card on a KGPE-D16, but it is not currently + enabled by default on any Canoeboot machines. +* **Fix vboot build issue when running cbmk in i686 (32-bit) host machines**. + The patch, courtesy of *Luke T. Schumaker*, adapts vboot's vmlinuz extract + function so that it uses pointer logic directly, instead of defining + integers (of type `ssize_t`) which, the way it was written, caused GCC to + believe that there would be a buffer overflow in code; the new code is more + robust and should prevent such an issue. This is both an *acute* bug fix, + fixing a bug that was actually triggered, and a preventative bug fix as the + original code wasn't correct either, even on AMD64 hosts (where it happened + to compile anyway). +* **GRUB: Never run it as a primary payload on any target but QEMU**. This is + a preventative bug fix, after lbmk bug report issue 216: + <https://codeberg.org/libreboot/lbmk/issues/216> - although it was caused by + the xHCI patches, and only happened on Sandybridge hardware, and although + this was later removed on those boards, GRUB is very complex and likely has + a lot of memory corruption issues. SeaBIOS is more reliable, so: Canoeboot + only provides *SeaBIOS* as primary payload, but allows you to execute GRUB + from the SeaBIOS menu (the very same GRUB). Additionally: cbmk already + supported a configuration whereby SeaBIOS reads a `bootorder` file in CBFS, + making it try to run the GRUB payload first, while still allowing you to + interrupt by pressing ESC to bring up an alternative boot select menu. This + is now the *default*, on all x86 mainboards. This is a mitigation against + future instability in GRUB because, if such issues happen again, it will not + cause a brick since you can just use SeaBIOS instead, and skip booting to + the GRUB payload (on the affected machines, BIOS GRUB still worked, which + your distro provides and SeaBIOS executes it). *NOTE: GRUB was later made + into a multi-tree project, with certain mainboards using a version that + has the xHCI patches, if required, because the machines that actually need + xHCI support were not affected by the bug referenced in issue 216.* +* Main build script: Check SUID before checking Git name/email, otherwise the + version/versiondate files could be written as root and thus prevent building + of cbmk, which (for most commands) is intentionally engineered to exit (with + error status) if you run it as root. +* script/trees: Reset variable `makeargs` per target, so as to prevent + pollution of this variable when switching from one build target to the next. +* script/trees: Added `UPDATED_SUBMODULES=1` to the make command when running + any coreboot `make` command, to prevent coreboot from automatically fetching + certain Git submodules; this is a preventative fix, fixed before it became + a bug, which it likely would have become at some point as this is exactly + what the coreboot build system does! +* Main build script: hide the output of `git init` when cbmk re-initialises the + Git history, to prevent its output from being wrongly inserted into the + output of commands such as `./build roms list` - such pollution would cause + build errors, so it's important that the Git initialisation function either + doesn't output anything, or that it should cause an *exit* if output is to be + required. +* Added the `CHANGELOG` file to `.gitignore`. This means `./update release` + will now work, on release archives, because cbmk re-initialises Git history + when doing so, but the CHANGELOG file (when present) causes cbmk to skip + all source downloads (which the release builder relies on). +* **Fix garbled output on 1440x900 monitors when using the Dell Latitude E6400.** + The E6400 uses a reference clock (`DPLL_REF_SSCLK`) set to 100MHz, whereas + libgfxinit assumed 96MHz. This timing descrepancy did not cause an issue on + lower resolution displays, so we never caught it in earlier testing. Patch + courtesy of Nicholas Chin, who debugged this issue alongside the user who + reported it. It was fixed by making such timing configurable, within the + coreboot build system, setting it to 100MHz on Dell Latitude E6400. +* script/roms: Skip a target when its config directory is missing, so that + running a coreboot target with no configs in it will not yield an error; + instead, it will now cause a non-error return. +* include/option.sh: If `.git` is missing, in a bare copy of cbmk (not a + release archive), recreate the version/versiondate files manually so as to + prevent a build error. Use of `cbmk.git` or the release archives is + recommended, but some users directly download snapshots of `cbmk.git` from + sites such as Codeberg, and there's no way for us to turn off this feature; + even if we did, it may be present on other Git hosting sites, where users + might host their own copy of cbmk. +* include/option.sh: Don't return non-zero status from the function + named `mkrom_tarball`, because certain other functions rely on its return + value to always be *zero*; instead, call `err` which will then yield + an *exit* (with non-zero status). This means that the function will now + always *return* zero, when it returns. +* include/git.sh: Remove `.git` directories *per-project*, as and when each + project is being downloaded, instead of having it done all in bulk by the + main build script. This kicks in when `XBMK_RELEASE` is set (release builds), + to correct the over-use of disk space during such very large builds processes. + This makes the build system less likely to OOM when running it inside tmpfs. +* Main build script: initialise Git history *before* running any command, + because this is required for reliable use of the coreboot build system, which + the *inject* command makes heavy use of. This reduces the number of errors, + when running these commands from a release archive, where cbmk re-initialises + a new Git history when you run it for the first time. +* Main build script: define `xp` as a global variable, to prevent it from + being lost between functions. +* script/roms: Create full release tarball name, when generating releases. +* Main build script: exit (with error status) if not running directly from + the root of the cbmk work directory. + +General code cleanup +-------------------- + +In addition to *general* very sweeping code cleanup, condensing code lines +where possible and so on: + +* include/lib.sh: Simplified the `download` function (used for crossgcc tarballs). +* include/lib.sh: Simplified the `singletree` function. +* include/git.sh: Simplified the `link_crossgcc` function. +* include/git.sh: Simplified the `nuke` function, because it was over-engineered + to the extreme. Now it's more reasonable. +* include/lib.sh: Move download logic here from lbmk as of audit 5, for the + feature where *files* can be downloaded as submodules, within Git repositories. + Please read the notes about this in the *features* section. +* include/lib.sh: Shortened a string in the `e` function, so that the line + does not exceed a length of 80 characters. +* include/git.sh: Unified the handling of git clone/reset/am commands into a + single function, rather than duplicating the same logic across multiple + functions. +* script/trees: simplify the `copy_elf` function; don't create the elf directory, + create one defined by `dest_dir` instead (which is the elf directory with + the subdirectory for that project concatenated). Only create it within + the `copy_elf` function, which is only called if actually compiling the + code. This avoids creating empty directories under elf/, for example under + fault conditions. +* include/git.sh: Additional code cleanup, removing certain code that was in + place because the code used to handle both `git submodule update` and the + custom *override* logic for submodules; now only the override is used, at all + times, so the code was cleaned up and optimised only for this. +* include/git.sh: Reduced code indentation in function `fetch_submodule`. +* include/git.sh: Renamed a few variables for increased code clarity. +* script/trees: Unified handling of coreboot `makeargs`. +* Moved function `handle_coreboot_utils` to script/trees (and renamed it + to `check_coreboot_utils`), as it's only ever used from there. +* Moved cfgsdir/datadir variables to include/lib.sh, because it's also used + from script/roms and script/trees; unify them under a common location. +* Handle `build.list` from config/data/, not config/ - this avoids needing to + check for `build.list` in the `items` function on include/lib.sh, and it is + now avoided. +* include/lib.sh: More user-friendly output from the `e` function, telling the + user whether or not a file/directory exists. This is regularly used, for + example when trying to download a project and the source code was already + prepared. +* U-Boot on QEMU: removed the (currently) unused x86 target. +* grub.cfg: Split function `try_user_config` into multiple smaller functions. +* grub.cfg: Don't scan ESP on btrfs subvols as the ESP is always on a FAT32 + partition. This saves time during the bootup sequence. +* Renamed include/option.sh to include/lib.sh +* Main build script: simplified the logic for Git repository initialisation + by *returning non-zero status*, instead of calling err, and handling this + return status in the calling function. +* Main build script: condensed the logic for Git name/email checking into a + simply for loop running `eval`, rather than having lots of separate but very + similar Git commands. +* script/trees: Removed a few unused variables. +* include/git.sh: Moved logic for copying a Git repository to a new function. +* include/git.sh: Moved function `link_crossgcc` to a different location + within the file, for proper top-down order of logic (required as per the + cbmk coding style). +* include/git.sh: Split logic for crossgcc symlinking into its own function. +* include/git.sh: Skip submodule checks if `.gitmodules` missing (NOTE: later + replaced with custom submodule handling in cbmk). +* include/git.sh: Merged `patch_submodules` in `prep_submodules` (NOTE: ditto + to the same note below). +* include/git.sh: Split up submodule handling into a new function (NOTE: support + for submodules was later replaced with custom logic in cbmk). +* include/git.sh: Shortened a few variable names. +* include/git.sh: Removed redundant check for the existence of the patches + directory, when patching a given project. This is unnecessary, where it was + removed, because the patching function itself also checks this. Reduction + in code size by *one line*. +* include/git.sh: Removed function `fetch_from_upstream` and merged its logic + into calling function `fetch_project_trees`, the only calling function, since + the logic in `fetch_from_upstream` was very small and splitting made no sense. +* include/option.sh: Renamed `mktar_release` to `mkrom_tarball`. +* script/roms: Renamed function `moverom` to `copyrom`, because it runs `cp`, + not `mv`, therefore is is *copying* a file, not moving it. +* script/roms: Simplified the logic for listing available serprog build targets. +* script/roms: General simplification of configuration handling for payloads. +* Main build script: removed the `excmd` function and merged its logic into + the `main` function, and then `main` was cleaned up significantly. +* Main build script: don't make `script_path` a global variable; this allowed + a reduction in code size by precisely *one line of code*. +* Main build script: merged the functionality of function `check_git` into + the `main` function, then deleted function `check_git` (which was in + the file include/option.sh). +* Main build script: general simplification of the logic handling source code + downloads in function `fetch_trees`. +* Main build script: Use `UTC+0000` when initialising git repository commit + dates (for initial commits). +* Removed the `check_project` function, and placed its logic directly + inside `include/option.sh` so that it automatically runs in every script + that sources it. +* Main build script: General cleanup on the code handling file deletions + under function `fetch_trees`. +* Main build script: delete function `mkversion` and, in its calling function, + simply print the string contained in variable `relname`. +* Main build script: general cleanup on the logic that handles tarballs. +* Main build script: Remove `mkrom_images`, and move its logic into the only + calling function within that same file. +* include/option.sh: Removed the function `insert_version_files` and merged + its logic into its only calling function. +* Unified all logic for handling SHA512 checksums, placing it inside + include/option.sh for use elsewhere. +* Move image tarball generation to script/roms (formerly script/build/roms). +* Removed redundant function `extract_ref` from include/mrc.sh +* Removed an errant comment from include/git.sh +* Switched to a one-level directory structure for main scripts, rather than + two-level; for example, script/build/roms is now script/roms +* Merged script/update/release into the main build script +* Merged script/build/serprog into script/build/roms +* script/build/roms: remove unnecessary command (errant return) +* Merged include/err.sh with include/option.sh, into include/option.sh +* script/build/roms: fixed improper use of variable outside a function +* build/build/roms: more reliable exit status in `skip_board()` +* script/build/roms: split up `main()` into multiple smaller functions + +Revision updates +================ + +Some revisions were updated as part of standard routine, but happened to be +done during this audit. Those updates are as follows: + +SeaBIOS +------- + +Bump SeaBIOS to revision `e5f2e4c69643bc3cd385306a9e5d29e11578148c`, which has +these changes relative to the old one: + +``` +* e5f2e4c6 pciinit: don't misalign large BARs +* 731c88d5 stdvgaio: Only read/write one color palette entry at a time +* c5a361c0 stdvga: Add stdvga_set_vertical_size() helper function +* 22c91412 stdvga: Rename stdvga_get_vde() to stdvga_get_vertical_size() +* 549463db stdvga: Rename stdvga_set_scan_lines() to stdvga_set_character_height() +* c67914ac stdvga: Rename stdvga_set_text_block_specifier() to stdvga_set_font_location() +* aa94925d stdvga: Rework stdvga palette index paging interface functions +* 8de51a5a stdvga: Rename stdvga_toggle_intensity() to stdvga_set_palette_blinking() +* 96c7781f stdvga: Add comments to interface functions in stdvga.c +* 2996819f stdvga: Rename CGA palette functions +* 91368088 stdvgamodes: Improve naming of dac palette tables +* 70f43981 stdvgamodes: No need to store pelmask in vga_modes[] +* 1588fd14 vgasrc: Rename vgahw_get_linesize() to vgahw_minimum_linelength() +* d73e18bb vgasrc: Use curmode_g instead of vmode_g when mode is the current video mode +* 192e23b7 vbe: implement function 09h (get/set palette data) +* 3722c21d vgasrc: round up save/restore size +* 5d87ff25 vbe: Add VBE 2.0+ OemData field to struct vbe_info +* 163fd9f0 fix smbios blob length overflow +* 82faf1d5 Add LBA 64bit support for reads beyond 2TB. +* 3f082f38 Add AHCI Power ON + ICC_ACTIVE into port setup code +* 3ae88886 esp-scsi: terminate DMA transfer when ESP data transfer completes +* a6ed6b70 limit address space used for pci devices. +``` + +Flashprog +--------- + +Updated to revision 5b4fdd1 from 2 May 2024, rebasing the MX workaround patch. + +This imports upstream changes, relative to the previous revision: + +``` +* 5b4fdd1 z60_flashprog.rules: Add udev rule for CH347 +* 72c9e40 meson: Check for CPU families with known raw mem access +* 3458220 platform/meson: Port pciutils/pci.h workaround to Meson +* f279762 platform/meson: Check for libi386 on NetBSD +* 14da5f7 README: Convert to Markdown +* 8ddea57 README: Document branching and release policy +* 2522456 util/list_yet_unsupported_chips.sh: Fix path +* cbf9c11 spi: Don't cross 16MiB boundaries with long writes +* 823a704 dediprog: Skip warning on first attempt to read device string +* e8463c8 dediprog: Revise prefix check for given programmer id +* 38af1a1 dediprog: Revise id matching +* 4661e7c amd_spi100: Use flashprog_read_chunked() for progress reporting +* cdcfda2 read_memmapped: Use flashprog_read_chunked() for progress reporting +* 7679b5c spi25: Replace spi_read_chunked() with more abstract version +* ca1c7fd spi25: Normalize parameters of spi_nbyte_read() +* e36e3dc dediprog: Use default_spi_write_256 +* 522a86d linux_spi: Use default_spi_read()/_write_256() +* 806509b cli_classic: Turn progress reporting into a progress bar +* 842d678 libflashrom: Return progress state to the library user +* aa714dd flashprog.c: Let select_erase_functions() return byte count +* 2eed4cf serprog: Add SPI Mode and CS Mode commands +* 821a085 dediprog: Implement id reading for SF600 and later +* 274e655 dediprog: Read device string early +* 0057822 dediprog: Add protocol detection for SF700 & SF600Plus-G2 +* fb176d2 dediprog: Use more general 4BA write mode for newer protocols +* 0ab5c3d dediprog: Split device type and version parsing +* bdef5c2 dediprog: Use unsigned conversions to parse device string +* 5262e29 dediprog: Try to request 32B device string (instead of 16B) +* e76e21f dediprog: Get rid of some unnecessary hex constants +* 5a09d1e udelay: Lower the sleep vs delay threshold +* 03ad4a4 linux_mtd: Provide no-op delay implementation +* 211c6ec serprog: Refine flushing before synchronization +* 383b7fe serprog: Test synchronicity before trying to synchronize +* d7318ea serprog: Move synchronicity test into separate function +* 9a11cbf Let the flash context directly point to the used master +* aabb3e0 writeprotect: Hook wp functions into the chip driver +* 89569d6 memory_mapped: Reduce `decode_sizes` to a single `max_rom_decode` +* 929d2e1 internal: Pass programmer context down into chipset enables +* 7c717c3 internal: Pass programmer context down into board enables +* e3a2688 Pass programmer context to programmer->init() +* 2b66ad9 Start implementing struct flashprog_programmer +* 4517e92 memory_bus: Drop stale `size == 0` workaround and FIXME +* b197402 memory_bus: Split register mapping into own function +* 0e76d99 memory_bus: Move (un)map_flash_region into par master +* 9eec407 Perform default mapping only for respective chips +* 56b53dd wbsio_spi: Request memory mapping locally +* 5596190 it87spi: Request memory mapping locally +* 46449b4 spi25: Drop stale `bus == SPI` guards +* ab6b18f spi25: Move 4BA preparations into spi_prepare_4ba() hook +* 901fb95 Add prepare/finish_access() hooks for chip drivers +* a96aaa3 dediprog: Support long writes of 16MiB and more +* 1338936 Consider 4BA support when filtering erase functions +* 8d36db6 flashprog.8: Fix up serprog example +* d2ac303 flashprog.8: document new serprog cs parameter +* d1b9153 chipset_enable.c: Add Genoa to mendocino entry +``` + +As a reminder: + +Canoeboot now uses Flashprog instead of Flashrom; Flashprog is a fork of +Flashrom, lead by Nico Huber after a dispute with the new leadership of +Flashrom, and it was felt that Flashprog is a better choice for Canoeboot. + +Git log +======= + +This entire set of changelogs is based on the precise Git history in cbmk, +relative to Canoeboot 20240504 which is from where the audit began. + +The latest changes are listed first, going all the way down to earlier changes: + +``` +* 4f6fbfde81 minor code cleanup in the build system +* 070aee6728 re-add ability to use cbfs grub.cfg as default +* b4acd0f73c trees: exit with error if project undefined +* fd9664c567 build: also make a lock file during release build +* 686bad6d4e lib.sh: more useful lock message +* f1caf89a28 create a lock file during builds +* b6dc23bc67 git.sh: hide e() output on for loop +* e51eae0d25 lib.sh: fix regression +* 8b1a54d19e git.sh: download xtree *before*, not after +* 14bba2d789 git.sh: fix deletion path in nuke() +* ab4c4d406f lib.sh: less confusing error in download() +* 2eaaa63f58 lib.sh: hide stderr on download() +* 9e2584fbd9 lib.sh: simplify download() +* 79fb79d239 lib.sh: fix redundancy in download() +* e8b1d45631 lib.sh: simplify singletree() +* 90a8ef90b0 git.sh: further simplify nuke() +* c6b692208b git.sh: simplify link_crossgcc() +* c043e5810d git.sh: simplify nuke() +* 323a17d0c8 Add dependency scripts for Fedora 40 and Ubuntu 24.04 +* 62b2310a28 add crossgcc tarballs to config/submodules/ +* 8a34a0d338 git.sh: support downloading *files* as submodules +* 0730513709 git.sh: remove unnecessary line break +* ad05266f8d import file download function from lbmk c202dc61 +* b8e9eab0ba lib.sh: shorten a string in e() +* a29cf274bc git.sh: fix submodule path +* 7ac2264f53 git.sh: simplify prep_submodules() +* 7c8173ebd4 git.sh: unified handling of git clone/reset/am +* 573199c07d trees: simplified copy_elf() handling +* d0d9b1204f git.sh: simplify submodule handling +* df5d7c18bf git.sh: provide feedback for repository downloads +* 591c7d28e0 git.sh: download "depend" projects *before* +* 548d1e20c1 git.sh: reduced indentation in fetch_submodule +* 12a04e8de2 git.sh: reduced indentation in prep_submodules +* 9825e97a83 git.sh: *never* run git submodule update +* 860deb3e7e lib.sh: rename variable for clarity +* 8d5edd4f06 trees: don't check empty path in copy_elf() +* c1176bbd28 trees: fix build issue caused by bad elf check +* c88fb8c129 trees: fix listfile check in copy_elf() +* 9168d33741 trees: don't say check elf/ if build.list missing +* db09530905 trees: don't do elfcheck if build.list missing +* 99418a7e82 define mdfiles/images in config/submodules/docs/ +* 83d84797d8 libopencm3 to config/submodules/ on stm32-vserprog +* c3cabcddf9 add tinyusb to config/submodule/ for pico-sdk +* e4eb82e089 trees: unified coreboot makeargs +* f7170092c8 trees: use multiple threads to build cbutils +* 1d7a6f04c9 move handle_coreboot_utils to script/trees +* ff16d27991 put coreboot utils in elf/, not cbutils/ +* 3748f710c9 fix build issue building coreboot utils +* a30bfd334f trees: skip single-tree build if a build exists +* b682b4ddca use correct memtest86plus path in script/roms +* 4749a5a29f put memtest86plus builds in elf/memtest86plus/ +* 0e9d9b33b2 put flashprog builds in elf/flashprog/ +* 7fe0106fa0 trees: also print "DONE! check elf/dir" on single +* 74759d876a trees: handle build-test on multi-tree projects +* 98e9cf6864 git.sh: use singletree() to decide submodules +* b3b887567a remove cbcfgsdir variable (unused) +* cb446e7d24 move cfgsdir/datadir variables to lib.sh +* 7d99786a1a handle build.list from config/data/, not config/ +* a61794dfca don't use build.list to detect multi-tree projects +* 878056f37b move id check to lib.sh too +* 3900642471 move root check to lib.sh (bugfix) +* 740b1803fa bugfix: move dependencies handling to lib.sh +* 4e25e335ed bump untitled revision again +* 44ef38b335 bump untitled revision in git config +* 7b9431e336 lib.sh bugfix: check environmental variables right +* 2478252f67 lib.sh: more friendly output from e() +* d21fd016ac badcmd: don't print "no context given" +* 663de3bab4 badcmd: link directly to the maintenance manual +* 1d866d17d8 better help text on invalid commands +* 1204bc3c96 build: print the project website address on help +* ca0e9354f6 add projectsite file: point to canoeboot.org +* eb4ac3c334 make GRUB multi-tree and re-add xhci patches +* 347a104ae6 u-boot on qemu: remove currently unused x86 target +* 23e66c113d grub.cfg: scan /boot/grub.cfg last +* 6151316b91 grub.cfg: scan grub2/ last +* 36b3be95cf grub.cfg: search a reduced list of devs/partitions +* 71a17efc06 grub.cfg: scan grub.cfg from ESP +* 8bc7e3a539 grub.cfg: split up try_user_config +* cb4bacc9d9 grub.cfg: don't search for *_grub.cfg +* ea7e6e1659 grub.cfg: remove unnecessary path for isolinux +* 1beca3b781 grub.cfg: don't scan EFI on btrfs subvols +* 0662519cca Fix building vboot on i686 +* 224dce632b git.sh: do not remove .submodules +* a36504aa31 delete u-boot test/lib/strlcat.c using nuke() +* cdce8ba70b make nuke function more generic +* 2c1f6f5e7a do not allow dashes in coreboot target names +* 7dc5d35929 roms: allow user override of grub_scan_disk +* bcb65846d3 grub.cfg: actually support setting boot order +* 2887b77ae4 trees: use CPUS=x on regular coreboot make +* a056583762 update gitignore +* 1ac4f7409e roms: fix bad eval when comparing options +* 724dbfe0ce grub.cfg: add spdx header +* 66f5faac73 re-configure grub_scan_disk on various targets +* bb92776943 remove grub_scan_disk in all target.cfg files +* 935447b035 grub.cfg: use grub_scan_disk to set boot order +* 75b6fbf302 GRUB: remove XHCI patches for now (will re-add) +* 07340d9711 minor correction +* 9f489b43d5 roms: make grubfirst if seabios_withgrub=y +* fca9b19e18 coreboot: only run GRUB as a secondary payload +* b75490f8fc flashprog: bump to 5b4fdd1 from 2 May 2024 +* d147c5d915 rename include/option.sh to include/lib.sh +* f534b0e973 merge nuke() back into git.sh +* a02b152f44 rename nukeblobs to a more generic name +* cb1918c5d7 roms: remove errant reference +* 4cff3c7d33 roms: rename bstr variable +* dc487df12f git.sh: remove errant whitespace +* cbb2f4f8a9 general code cleanup in the build system +* 583135e548 build: simplify git_init() +* aaff90f5a5 build: do root check before git check +* 687fdacc78 build: simplify git checks +* 84ee6a1ed8 option.sh: fix bad check for version/versiondate +* 3554593fd8 trees: reset makeargs per target/project +* b09261a901 trees: also use UPDATED_SUBMODULES=1 on crossgcc +* 698548ac59 trees: add UPDATED_SUBMODULES to coreboot make +* c8c516703f trees: write -C on the make command first not last +* aa15eef32f config: add backup coreboot submodule repositories +* 9e88ef2449 coreboot/default: remove chromeec from module.list +* 27f21c32d3 git.sh: break if a submodule clone succeeds +* 38fca598fb coreboot: only download the necessary submodules +* b5aa8b2d35 git.sh: allow finer control of git submodules +* 9339c6f3fd build: hide git-init output +* 31e089aff3 option.sh: generate version file if .git not found +* 7ec023907b update/trees: remove unused variable +* 2b0e71412e git.sh: move repo copying to a new function +* d71c4d326e git.sh: move link_crossgcc to end of file +* 0d7c249c9b move deblob function to new file "deblob.sh" +* 1300f09e67 git.sh: move xgcc linking to a new function +* 24934e6569 git.sh: don't include --checkout in submodules +* 5e0129eb0f git.sh: skip submodules if .gitmodules missing +* 7f82622caf git.sh: merge patch_submodules in prep_submodules +* 9c0a7f14fc git.sh: split submodule handling to new function +* b593127795 git.sh: remove errant line break +* 19f694bf2a git.sh: remove another meaningless check +* 71a9fcced8 git.sh: shorter variable names +* 6693588857 git.sh: remove meaningless check +* 5c459ad4ac git.sh: remove variable not meaningfully used +* 7be7bb8edb add CHANGELOG to .gitignore +* 3b2ebda890 Fix E6400 display reference clock patches +* 995f052bb0 fix building coreboot images on i686 hosts +* 31d2c818eb Also try unlocking encrypted volume on NVMe +* 58f6741fb4 git.sh: fix invalid command in git_prep() +* f58b01c300 Add NVMe support to GRUB2 payload +* b892036edf Fix E6400 display issue with 1440 x 900 panel +* f81c7ed8e9 Add pt qwerty keymap to lbmk +* 849466c0ac git.sh: allow patching submodules +* 8d4d063ace git.sh: don't delete .git if src/project/project +* 0ecb062df0 build/roms: skip target if config/ dir missing +* 4783c5b90e more minor cleanup in the build system +* 10ecf41ee0 git.sh: remove fetch_from_upstream() +* ddcb793bd2 option.sh: don't return 1 in mkrom_tarball +* ae8637b620 option.sh: mktar_release to mkrom_tarball +* 309c3b1f33 build/roms: rename moverom to copyrom +* a39c95cfac minor code cleanup in the build system +* f102e21ab6 build/roms: simplify serprog list command +* 7a565c9f43 build/roms: simplified config payload checks +* a243dc2308 option.sh: err if config directory is missing +* c28166ff9e option.sh: print the config filename being checked +* 9fd504e24a git.sh: Remove .git if XBMK_RELEASE=y +* e4956478db build: remove initcmd() and simplify main() +* f2b3bb142d build: initialise git first (before commands) +* 571932d33e build: remove excmd() and simplify main() +* 525f5525d3 build: don't make script_path a global variable +* fbac2d8fe6 Implemented failsafe options at boot and inside menus for enabling/disabling serial, spkmodem and gfxterm +* 3e5db248dd cbmk: allow easier sync with lbmk +* e71189420f remove help commands (user should read docs) +* 23854de888 option.sh: delete check_git() +* 2c5f52ce29 build: define "xp" in the global variables +* 48c5c57cff build: simplify for loop in fetch_trees() +* c2baebc79a build: simplified downloads in fetch_trees() +* 18d0e53480 ./build release: don't do u-boot-only archives +* d8a923f766 build: use utc+0 when initialising git repo dates +* 0794127986 remove check_project() (always set variables) +* c8bc797f31 build: simplify deletions in fetch_trees() +* 363ec7512c build: delete mkversion() (just print relname) +* ae44676727 build/roms: clean up tarball handling +* 3469836f18 rm src/u-boot/*/test/lib/strlcat.c in u-boot +* c57dfefe91 build: remove mkrom_images +* 6ab8c2c446 build: use same tarball name on uboot-only release +* 21436c6a8f build/roms: create full release tarball name +* 90c528032b option.sh: don't bother checking for GNU tar +* 422d36a07c option.sh: remove insert_version_files() +* ca1806f20e cleanup: remove mkvdir +* a0ea7f7a92 unified sha512sum creation for tarballs +* 09fcc343a3 move rom tarball creation to script/roms +* 5c888669c6 disable x301 for next release (for now) +* 91c90d763f print two line breaks before confirming release +* d423421995 remove all status checks. only handle release. +* 4826364afb git.sh: remove errant comment +* 541430016f move script/*/* to script/ +* 9084ab15ab build: print usage for special commands +* f12c2f284f merge script/update/release into build +* 41f4ee3c2d Canoeboot 20240510 release +* 0580373ff9 bump seabios to e5f2e4c69643bc3cd385306a9e5d29e11578148c +* 17b5cb2749 further modify the README (stragglers) +* 628e91a3b9 build: further prevent non-cbmk-work-directory +* e761a494c8 build: exit if not running from cbmk directory +* eb8a02e808 build/roms: print serprog help +* a398011180 merge script/build/serprog with script/build/roms +* cd5c2573ac build/roms: remove unnecessary command +* da748de455 merge include/err.sh with include/option.sh +* 3acac46536 err.sh: correct copyright info +* 6bdbb70dbc build/roms: don't rely on x in handle_target +* 1c84d0fc9d build/roms: don't use exit status from skip_board +* 0ada63b629 build/roms: split up main() +* 5cecd9e394 build/roms: allow searching status by mismatch +* 97d502ccc8 tone the README way, way down +``` + +This is 206 changes, since Canoeboot 20240504.