From 2abd8aba00c3eda1f95fc0f818477df8ec50dc16 Mon Sep 17 00:00:00 2001 From: Zhe Yuan Date: Sun, 6 Sep 2026 00:28:54 -0400 Subject: [PATCH] Rewrite the README for the current script set MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README had been patched incrementally across several changes and no longer matched the repository: sections about adding a device and the package list format had ended up nested under Quick start, and the layout it described predated the per-device packages/ directory. Restructure it around what someone actually does: layout, requirements, a five-step quick start, then per-script reference. Document the parts that are easy to get wrong and were learned the hard way — the buildbot flags and why stripping them is safe, the Go toolchain options, the WSL PATH problem that breaks every Go package, and the git clean -xdff hazard. Co-Authored-By: Claude Opus 5 --- README.md | 366 +++++++++++++++++++++++++++++------------------------- 1 file changed, 200 insertions(+), 166 deletions(-) diff --git a/README.md b/README.md index c6b6dbd..f3ec1f6 100644 --- a/README.md +++ b/README.md @@ -1,113 +1,144 @@ # OpenWrt Build Helper Scripts -## Overview -This repository contains shell scripts that streamline preparing, configuring, and building OpenWrt from source on Debian/Ubuntu systems (x86_64 and ARM64). It also includes convenience utilities to manage package selections, fetch a device's official configuration, and add external LuCI packages. +Shell scripts that automate building OpenWrt from source: installing the host +dependencies, fetching the source tree, importing a device's official +configuration, and keeping a per-device package list. -## What’s included -- prepare-openwrt-env.sh - - Installs the build dependencies listed by the official OpenWrt build system guide. - - Supports apt (Debian/Ubuntu/Mint), dnf/yum (Fedora/RHEL/Rocky/Alma), pacman (Arch/Manjaro), zypper (openSUSE) and apk (Alpine). - - Probes each package against the running release and skips the ones that no longer exist, so a renamed or dropped package cannot abort the whole install. - - Handles x86_64 and ARM64 differences (native vs. cross multilib). - - Options: `-y/--yes` (no prompt), `-n/--dry-run` (only print), `--with-go` (also install the Go toolchain). -- prepare-openwrt.sh - - Clones or updates the OpenWrt source repo in the top-level `openwrt/` root. - - Lets you select a stable tag, a release branch, or the snapshot (`main`) interactively, or non-interactively via flags. - - Uses the script location as the source of truth, so running it from another current working directory is supported. - - On first run from a helper checkout, relocates that checkout to `openwrt/helper/` and leaves a symlink behind at the old path. - - Prints a plan and asks for confirmation; nothing on disk is touched before you confirm. - - Uses a partial clone (`--filter=blob:none`) by default, so the first fetch is far smaller than full history. - - Resumes automatically if a previous run was interrupted. - - Updates feeds and runs make defconfig. - - Options: `--stable []`, `--branch `, `--snapshot`, `-y/--yes`, `-n/--dry-run`, - `-f/--force`, `--full-clone`, `--allow-rc`, `--root `, `--skip-feeds`, - `--skip-defconfig`, `--repo-url `, `--no-compat-link`, `-h/--help`. -- add-external-repos.sh - - Clones or updates additional package repositories into package/. - - Currently includes luci-app-netspeedtest and luci-app-easytier (adjust REPOS as needed). -- download-config.sh - - Downloads config.buildinfo for a device and installs it as .config. - - Looks the device up in the official index, so any supported device can be picked by name; nothing is hardcoded. Use `--device ` for non-interactive runs. - - By default builds only the selected device instead of every model in the target, and turns off the buildbot flags (ALL_KMODS, ALL_NONSHARED, SDK, IB, MAKE_TOOLCHAIN, COLLECT_KERNEL_DEBUG, AUTOREMOVE). Neither changes the firmware contents — see "Buildbot flags" below. - - Downloads to a temp file and validates it before touching .config, and keeps a .config.download.bak. - - Prints the official sysupgrade/factory URLs (taken from profiles.json, so they are always correct) and a Firmware Selector link. -- gen-package-list.sh - - Builds a device's complete package list: the built-in part from the official profiles.json (default_packages + device_packages + the packages the Firmware Selector adds), combined with your own packages from `packages/.txt`. - - The built-in part never has to be copied by hand for a new release, and your packages are version independent — a new release needs no file changes at all. - - Writes the combined list to `.generated/` (not kept in git); `--apply` applies it straight to .config, `--full-stdout` just prints it. -- add-openwrt-packages.sh - - Applies a package list to .config: `##built-in` → =y, `##module` → =m, `##remove` or a `-pkg` prefix → =n. - - Rewrites existing CONFIG_PACKAGE_ lines in place, so repeated runs do not grow .config. - - Verifies each package against tmp/.config-package.in and tmp/.packageinfo and reports missing ones. - - After make defconfig it reports any package whose final state differs from what you asked for (a dependency pulling a removed package back in, for example). -- update-go-path.sh - - Detects the latest Go installation under /usr/lib/go-* and sets CONFIG_GOLANG_EXTERNAL_BOOTSTRAP_ROOT in .config. +The design goal is that **a new OpenWrt release requires no file changes** — +everything version specific is fetched from upstream, and only your own package +choices are stored in this repository. -- OS: Debian/Ubuntu, Fedora/RHEL, Arch, openSUSE or Alpine (the other scripts are developed and tested on Debian/Ubuntu) -- Architectures: x86_64/amd64 and aarch64/arm64 -- Tools: git, wget, curl, bash, python3 (used to read the official JSON indexes; jq is not required) -- Internet access for feeds and package downloads -- Optional: Go (required by some OpenWrt packages; install it with `./prepare-openwrt-env.sh --with-go`; configurable via update-go-path.sh) -- Note: with the default partial clone, switching to another tag or branch later fetches missing blobs on demand and therefore needs network access +## Layout + +`prepare-openwrt.sh` normalizes the checkout on first run: + +``` +before /openwrt-build-helper/ this repository +first run /openwrt/ the OpenWrt source root + /openwrt/helper/ this repository, moved in + /openwrt-build-helper -> openwrt/helper (symlink) +later runs the same /openwrt/ is updated in place +``` + +The symlink means shells, editors and agent sessions still sitting in the old +path keep working. Disable it with `--no-compat-link`. + +Inside `helper/`: + +| Path | In git | Purpose | +|---|---|---| +| `packages/.txt` | yes | your packages for one device, version independent | +| `.generated/` | no | combined package lists and saved buildinfo (build artefacts) | + +## Requirements + +- Linux with one of: apt (Debian/Ubuntu/Mint), dnf/yum (Fedora/RHEL/Rocky/Alma), + pacman (Arch/Manjaro), zypper (openSUSE), apk (Alpine) +- x86_64/amd64 or aarch64/arm64 +- git, curl, wget, bash, python3 (used to read the upstream JSON indexes; jq is + not required) +- Internet access, and roughly 30 GB of disk for a full build ## Quick start -1) Prepare the host machine -- Run: ./prepare-openwrt-env.sh -- This installs compilers, headers, Python tooling, and other build prerequisites. -- Preview without touching the system: ./prepare-openwrt-env.sh -n -2) Get OpenWrt sources -- Run: ./prepare-openwrt.sh -- Choose a stable tag, a release branch, or the snapshot. -- Non-interactive: ./prepare-openwrt.sh --stable 25.12.5 -y -- Preview only, changes nothing: ./prepare-openwrt.sh --stable -n -- The script: - - Reuses or creates the top-level OpenWrt root at `./openwrt`. - - Moves this helper checkout to `./openwrt/helper` on first run. - - Updates/install feeds and runs make defconfig. +```bash +# 1. host dependencies +./prepare-openwrt-env.sh # -n to preview, --with-go for Go -3) Add optional external packages -- From the OpenWrt root: - - ./helper/add-external-repos.sh -- This pulls extra LuCI packages into package/. +# 2. OpenWrt source (relocates this checkout to openwrt/helper on first run) +./prepare-openwrt.sh # or --stable 25.12.5 -y -4) Import a device config (optional but convenient) -- From the OpenWrt root: - - ./helper/download-config.sh -- Search for your device by name, or run it non-interactively: - - ./helper/download-config.sh --device linksys_mx8500 -y -- Add `--with-packages` to also generate the device's package list in the same run. -- Preview without changing anything: `./helper/download-config.sh --device linksys_mx8500 -n` +# from here on, run everything from the OpenWrt root +cd ../openwrt # or: cd openwrt -5) Enable additional packages (optional) -- Your own packages live in `helper/packages/.txt`, one file per device, independent of the OpenWrt version. -- Apply the built-in list plus your packages in one step: - - ./helper/gen-package-list.sh --device linksys_mx8500 --apply -- Or write the combined list out first and review it before applying: - - ./helper/gen-package-list.sh --device linksys_mx8500 - - ./helper/add-openwrt-packages.sh helper/.generated/packages_linksys_mx8500_.txt +# 3. the device's official configuration +./helper/download-config.sh --device linksys_mx8500 -y + +# 4. built-in packages + your own, applied to .config +./helper/gen-package-list.sh --device linksys_mx8500 --apply -y + +# 5. build +make -j$(nproc) download world +``` + +Firmware lands in `bin/targets///`. + +Run `./helper/download-config.sh` without `--device` to search for a device by +name. Add `--with-packages` to fold step 4 into step 3. ### Adding a device -1. `./helper/gen-package-list.sh --device --init-extras` — creates `packages/.txt` -2. Edit `packages/.txt` and list the packages you want on top of the defaults -3. `./helper/gen-package-list.sh --device --apply` -Use `./helper/download-config.sh` without `--device` to search for the id by name. +```bash +./helper/gen-package-list.sh --device --init-extras # creates packages/.txt +$EDITOR helper/packages/.txt # list what you want +./helper/gen-package-list.sh --device --apply +``` ### Upgrading to a new OpenWrt release -Nothing to edit. Check out the new version, then: -``` + +Nothing to edit — the built-in list is re-fetched and `packages/.txt` is +reused as is: + +```bash +./helper/prepare-openwrt.sh --stable -y +cd ../openwrt ./helper/download-config.sh --device -y ./helper/gen-package-list.sh --device --apply -y +make -j$(nproc) download world ``` -The built-in list is re-fetched for the new release and your `packages/.txt` is reused as is. -### Files -- `packages/.txt` — your packages, one file per device, kept in git and edited by hand. -- `.generated/` — combined package lists and saved buildinfo. Build artefacts, not kept in git. +## Scripts + +- **prepare-openwrt-env.sh** — installs the build dependencies from the official + build system guide. Dispatches on the package manager rather than guessing from + the distro version, and probes every package against the running release so a + renamed or dropped package cannot abort the install. + `-y`, `-n/--dry-run`, `--with-go`. + +- **prepare-openwrt.sh** — clones or updates the OpenWrt source in `openwrt/`. + Resolves the version over `git ls-remote` and prints a plan before touching + anything, so aborting at the prompt leaves the checkout untouched. Uses a + partial clone (`--filter=blob:none`) by default and resumes an interrupted + first run automatically. + `--stable []`, `--branch `, `--snapshot`, `-y`, `-n`, `-f/--force`, + `--full-clone`, `--allow-rc`, `--root `, `--skip-feeds`, + `--skip-defconfig`, `--repo-url `, `--no-compat-link`. + +- **download-config.sh** — installs a device's official `config.buildinfo` as + `.config`. Looks the device up in the official index, so any supported device + can be selected by name. Downloads to a temp file and validates it before + touching `.config`, keeping a `.config.download.bak`. By default builds only + the selected device and strips the buildbot flags (see below). + `--device `, `--target `, `--version `, `--snapshot`, + `--with-packages`, `--all-profiles`, `--keep-buildbot-flags`, + `--save-buildinfo`, `--offline`, `--output `, `-y`, `-n`. + +- **gen-package-list.sh** — builds the complete package list: the built-in part + from the target's `profiles.json` (`default_packages` + `device_packages` plus + the two packages the Firmware Selector adds), combined with your + `packages/.txt`. + `--device `, `--apply`, `--init-extras`, `--stdout`, `--full-stdout`, + `--out `, `--no-extras`, `-y`, `-n`. + +- **add-openwrt-packages.sh** — applies a package list to `.config`. Rewrites + existing `CONFIG_PACKAGE_` lines in place, so repeated runs do not grow the + file. After `make defconfig` it reports any package whose final state differs + from what you asked for. + `-y`, `-n`, `--no-defconfig`, `--no-backup`, `--strict`, and `-` to read stdin. + +- **update-go-path.sh** — points the build at an installed Go toolchain. + `--external` clears `CONFIG_GOLANG_BUILD_BOOTSTRAP` so the installed toolchain + builds host Go directly instead of OpenWrt building the whole bootstrap chain + from source. + `--external`, `--go-root `, `-n`. + +- **add-external-repos.sh** — clones extra package repositories into `package/`. + Edit the `REPOS` array to change the list. + +## Package list format -### Package list format Used by both `packages/.txt` and the generated lists: + ``` curl yq luci-app-ttyd packages, any number per line, set to =y -luci-app-wol a leading '-' removes a package (=n) @@ -117,95 +148,98 @@ curl yq luci-app-ttyd packages, any number per line, set to =y pkg # note text after a single '#' is a comment ``` +## Notes and gotchas + ### Buildbot flags -The official config.buildinfo is the buildbot's own configuration: it enables every -model in the target and sets ALL_KMODS / ALL_NONSHARED / SDK / IB / MAKE_TOOLCHAIN / -COLLECT_KERNEL_DEBUG / AUTOREMOVE. For a Linksys MX8500 that means 1248 extra module -packages plus the SDK, ImageBuilder and a toolchain tarball on every build. -download-config.sh turns these off by default. This was verified not to change the -firmware: the set of packages built into the image (`=y`) is identical either way — -193 packages before and after. The trade-off is that bin/ no longer contains prebuilt -packages for modules you did not select, so you cannot later install an arbitrary kmod -from your own build output. Use `--keep-buildbot-flags` to restore upstream behaviour, -and `--all-profiles` to build every model in the target. +The official `config.buildinfo` is the buildbot's own configuration: it enables +every model in the target and sets `ALL_KMODS`, `ALL_NONSHARED`, `SDK`, `IB`, +`MAKE_TOOLCHAIN`, `COLLECT_KERNEL_DEBUG` and `AUTOREMOVE`. For a Linksys MX8500 +that is 1248 extra module packages plus the SDK, ImageBuilder and a toolchain +tarball on every build. -6) Configure Go bootstrap path (if needed) -- From the OpenWrt root: - - ./helper/update-go-path.sh -- This sets CONFIG_GOLANG_EXTERNAL_BOOTSTRAP_ROOT to the latest /usr/lib/go-X.XX/ found. -- You can override it manually in .config if necessary. +`download-config.sh` turns these off by default. This was verified not to change +the firmware: the set of packages built into the image (`=y`) is identical either +way — 193 packages before and after. The trade-off is that `bin/` no longer +contains prebuilt packages for modules you did not select, so you cannot later +install an arbitrary kmod from your own build output. +`--keep-buildbot-flags` and `--all-profiles` restore upstream behaviour. +### Go packages +Packages such as tailscale, adguardhome, easytier and cloudflared need a host Go. +By default OpenWrt builds the entire bootstrap chain from source, which works but +is slow. To skip it, install a Go matching the version the tree expects and run: -- Ensure telephony feed is installed: - - ./scripts/feeds update telephony - - ./scripts/feeds install -a -- Then run: +```bash +sudo apt install golang-1.24-go +./helper/update-go-path.sh --external +``` -- Build the package: +### WSL: Windows PATH breaks Go package builds +On WSL, `PATH` includes Windows directories containing spaces and parentheses +(`/mnt/c/Program Files (x86)/...`). OpenWrt's Go build recipe interpolates `PATH` +unquoted into a shell command, so every Go package fails with: -- Common commands: - - make menuconfig - - make -j$(nproc) download world -- Artifacts will be in bin/ after the build completes. +``` +bash: -c: line 1: syntax error near unexpected token `(' +``` -## Script usage notes and tips -- Run locations: - - `prepare-openwrt.sh`: resolves paths from the script location, not your current shell directory. - - `prepare-openwrt-env.sh`: can be run before the first normalization from the fresh helper checkout. - - All other helper scripts: run them from inside the OpenWrt source root as `./helper/...` unless otherwise indicated. -- Feeds: - - If a package can’t be found, ensure feeds are updated/installed: - - ./scripts/feeds update -a && ./scripts/feeds install -a -- Go toolchain: - - On x86_64, Go is optional unless your selection pulls packages needing Go (e.g., firewall4 in some versions). If needed: - - sudo apt install golang -y - - Then use update-go-path.sh to set .config automatically. -- ARM64 multilib: - - The script installs cross multilib packages for better compatibility, but not all targets need them. If apt errors on specific multilib packages, remove or adjust as appropriate for your environment. +Either build with a sanitized PATH: + +```bash +CLEAN_PATH=$(echo "$PATH" | tr ':' '\n' | grep -v '^/mnt/' | paste -sd:) +env PATH="$CLEAN_PATH" make -j$(nproc) download world +``` + +or disable the interop PATH permanently in `/etc/wsl.conf` and restart WSL +(`wsl --shutdown`): + +```ini +[interop] +appendWindowsPath = false +``` + +### Hazard: `git clean -xdff` +`helper/` lives inside the OpenWrt Git worktree but is not tracked by it. +`prepare-openwrt.sh` adds `/helper/` to `openwrt/.git/info/exclude`, which keeps +`git status` clean and stops `git add -A` from staging it — but that does **not** +protect it from `git clean -xdff`: `-x` deliberately includes ignored files and +`-ff` removes Git's refusal to recurse into a nested repository. Use instead: + +```bash +git clean -xdf -e /helper +``` ## Troubleshooting -- Version detection in download-config.sh: - - It uses, in order: an explicit `--version`/`--snapshot`; the exact tag at HEAD; the branch (`main`/`master` → SNAPSHOT, `openwrt-XX.YY` → the newest matching tag merged into HEAD); otherwise it asks. If HEAD is ahead of the tag it warns that the downloaded config describes the release rather than your tree. - - If the device symbol does not survive `make defconfig`, the script stops and tells you the tree does not match that release, and points at the .config.download.bak it kept. -- Missing packages in add-openwrt-packages.sh: - - It verifies names against tmp/.config-package.in and tmp/.packageinfo. If a package is reported missing, ensure the corresponding feed is enabled in feeds.conf.default and run feeds update/install. -- A removed package (`-pkg`) comes back after make defconfig: - - Something else depends on it. The script reports this explicitly as "wanted =n, got =y". -- Permission errors: - - Scripts use umask 022 and do not require root except when installing apt packages. -- Clean up and rebuild a package: - - make package//{clean,compile} V=s -- Full rebuild: - - make clean; make -j$(nproc) download world -## Directory behavior -- If you start from a freshly cloned helper checkout at `/`, `prepare-openwrt.sh` will: - - Reuse or create `/openwrt/` as the OpenWrt source root. - - Move the entire helper checkout, including its `.git`, to `/openwrt/helper/`. - - Initialize or update the OpenWrt source tree directly in `/openwrt/`. - - Leave a symlink at the original checkout path pointing to `/openwrt/helper/`, so shells, editors and agent sessions whose working directory is still the old path keep working. Disable with `--no-compat-link`. -- If you later rerun `prepare-openwrt.sh` — from `openwrt/helper/` or through the symlink at the old path — it reuses the parent `openwrt/` directory and does not move anything again. -- The already-relocated layout is detected by content (the OpenWrt tree or its `origin` remote), not by directory name, so renaming `openwrt/` to something else does not trigger a second relocation. -- If a run is interrupted after the repository was created but before the source tree was checked out, the next run detects that state, reports `resuming it`, and completes. No manual cleanup is needed. -- If the script is already located in a real OpenWrt source root, it treats that directory as the source root and does not relocate it just because the directory name is `openwrt`. -- Use `--root ` to point at a different OpenWrt root in any ambiguous situation. +- **Version detection in download-config.sh** — it uses, in order: an explicit + `--version`/`--snapshot`; the exact tag at HEAD; the branch (`main`/`master` → + SNAPSHOT, `openwrt-XX.YY` → the newest matching tag merged into HEAD); + otherwise it asks. If HEAD is ahead of the tag it warns that the downloaded + config describes the release rather than your tree. Version queries are + anchored to the OpenWrt tree, not to whichever repository the script sits in. +- **The device symbol did not survive make defconfig** — your tree does not match + the release you downloaded a config for. The script stops and points at the + `.config.download.bak` it kept. +- **A package is reported missing** — names are verified against + `tmp/.config-package.in` and `tmp/.packageinfo`. Enable the relevant feed in + `feeds.conf.default`, then `./scripts/feeds update -a && ./scripts/feeds install -a`. +- **A removed package (`-pkg`) comes back after make defconfig** — something else + depends on it. Reported explicitly as "wanted =n, got =y". +- **A package is in .config but not in the manifest** — it is probably there under + a provider name (`libgcc` → `libgcc1`, `nftables` → `nftables-json`). +- **Rebuild one package** — `make package//{clean,compile} V=s` +- **Full rebuild** — `make clean; make -j$(nproc) download world` -### Known hazard: `git clean -xdff` -`helper/` lives inside the OpenWrt Git worktree but is not tracked by it. The script -adds `/helper/` to `openwrt/.git/info/exclude`, which keeps `git status` clean and stops -`git add -A` from staging it — but that does **not** protect it from `git clean -xdff`: -`-x` deliberately includes ignored files and `-ff` removes Git's refusal to recurse into -a nested repository. Running `git clean -xdff` in the OpenWrt root will delete the helper -checkout including its `.git`. Use this instead: +## Safety - git clean -xdf -e /helper - -## Security and safety -- Scripts use set -euo pipefail to stop on errors. -- prepare-openwrt.sh backs up `.config` to `.config..bak` before switching versions, and refuses to switch a tree with modified tracked files unless you pass --force. -- They modify .config and write under package/ and feeds/ inside your OpenWrt tree. -- Backup your .config if you need a known baseline. +- Scripts use `set -euo pipefail` and `umask 022`, and need root only to install + host packages. +- `.config` is backed up before it is replaced or before a version switch. +- `prepare-openwrt.sh` refuses to switch versions on a tree with modified tracked + files unless you pass `--force`. +- Scripts write under `package/` and `feeds/` inside your OpenWrt tree. ## License and authorship -- Scripts authored by Zhe Yuan with help from ChatGPT. + +- Scripts authored by Zhe Yuan. - License: MIT (unless you choose a different license; update this line accordingly).