Rewrite the README for the current script set

The README had been patched incrementally across several changes and no longer
matched the repository: sections about adding a device and the package list
format had ended up nested under Quick start, and the layout it described
predated the per-device packages/ directory.

Restructure it around what someone actually does: layout, requirements, a
five-step quick start, then per-script reference. Document the parts that are
easy to get wrong and were learned the hard way — the buildbot flags and why
stripping them is safe, the Go toolchain options, the WSL PATH problem that
breaks every Go package, and the git clean -xdff hazard.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-06 00:28:54 -04:00
parent a98b1e1c76
commit 2abd8aba00

366
README.md
View File

@@ -1,113 +1,144 @@
# OpenWrt Build Helper Scripts
## Overview
This repository contains shell scripts that streamline preparing, configuring, and building OpenWrt from source on Debian/Ubuntu systems (x86_64 and ARM64). It also includes convenience utilities to manage package selections, fetch a device's official configuration, and add external LuCI packages.
Shell scripts that automate building OpenWrt from source: installing the host
dependencies, fetching the source tree, importing a device's official
configuration, and keeping a per-device package list.
## Whats included
- prepare-openwrt-env.sh
- Installs the build dependencies listed by the official OpenWrt build system guide.
- Supports apt (Debian/Ubuntu/Mint), dnf/yum (Fedora/RHEL/Rocky/Alma), pacman (Arch/Manjaro), zypper (openSUSE) and apk (Alpine).
- Probes each package against the running release and skips the ones that no longer exist, so a renamed or dropped package cannot abort the whole install.
- Handles x86_64 and ARM64 differences (native vs. cross multilib).
- Options: `-y/--yes` (no prompt), `-n/--dry-run` (only print), `--with-go` (also install the Go toolchain).
- prepare-openwrt.sh
- Clones or updates the OpenWrt source repo in the top-level `openwrt/` root.
- Lets you select a stable tag, a release branch, or the snapshot (`main`) interactively, or non-interactively via flags.
- Uses the script location as the source of truth, so running it from another current working directory is supported.
- On first run from a helper checkout, relocates that checkout to `openwrt/helper/` and leaves a symlink behind at the old path.
- Prints a plan and asks for confirmation; nothing on disk is touched before you confirm.
- Uses a partial clone (`--filter=blob:none`) by default, so the first fetch is far smaller than full history.
- Resumes automatically if a previous run was interrupted.
- Updates feeds and runs make defconfig.
- Options: `--stable [<ver>]`, `--branch <name>`, `--snapshot`, `-y/--yes`, `-n/--dry-run`,
`-f/--force`, `--full-clone`, `--allow-rc`, `--root <dir>`, `--skip-feeds`,
`--skip-defconfig`, `--repo-url <url>`, `--no-compat-link`, `-h/--help`.
- add-external-repos.sh
- Clones or updates additional package repositories into package/.
- Currently includes luci-app-netspeedtest and luci-app-easytier (adjust REPOS as needed).
- download-config.sh
- Downloads config.buildinfo for a device and installs it as .config.
- Looks the device up in the official index, so any supported device can be picked by name; nothing is hardcoded. Use `--device <id>` for non-interactive runs.
- By default builds only the selected device instead of every model in the target, and turns off the buildbot flags (ALL_KMODS, ALL_NONSHARED, SDK, IB, MAKE_TOOLCHAIN, COLLECT_KERNEL_DEBUG, AUTOREMOVE). Neither changes the firmware contents — see "Buildbot flags" below.
- Downloads to a temp file and validates it before touching .config, and keeps a .config.download.bak.
- Prints the official sysupgrade/factory URLs (taken from profiles.json, so they are always correct) and a Firmware Selector link.
- gen-package-list.sh
- Builds a device's complete package list: the built-in part from the official profiles.json (default_packages + device_packages + the packages the Firmware Selector adds), combined with your own packages from `packages/<device-id>.txt`.
- The built-in part never has to be copied by hand for a new release, and your packages are version independent — a new release needs no file changes at all.
- Writes the combined list to `.generated/` (not kept in git); `--apply` applies it straight to .config, `--full-stdout` just prints it.
- add-openwrt-packages.sh
- Applies a package list to .config: `##built-in` → =y, `##module` → =m, `##remove` or a `-pkg` prefix → =n.
- Rewrites existing CONFIG_PACKAGE_ lines in place, so repeated runs do not grow .config.
- Verifies each package against tmp/.config-package.in and tmp/.packageinfo and reports missing ones.
- After make defconfig it reports any package whose final state differs from what you asked for (a dependency pulling a removed package back in, for example).
- update-go-path.sh
- Detects the latest Go installation under /usr/lib/go-* and sets CONFIG_GOLANG_EXTERNAL_BOOTSTRAP_ROOT in .config.
The design goal is that **a new OpenWrt release requires no file changes**
everything version specific is fetched from upstream, and only your own package
choices are stored in this repository.
- OS: Debian/Ubuntu, Fedora/RHEL, Arch, openSUSE or Alpine (the other scripts are developed and tested on Debian/Ubuntu)
- Architectures: x86_64/amd64 and aarch64/arm64
- Tools: git, wget, curl, bash, python3 (used to read the official JSON indexes; jq is not required)
- Internet access for feeds and package downloads
- Optional: Go (required by some OpenWrt packages; install it with `./prepare-openwrt-env.sh --with-go`; configurable via update-go-path.sh)
- Note: with the default partial clone, switching to another tag or branch later fetches missing blobs on demand and therefore needs network access
## Layout
`prepare-openwrt.sh` normalizes the checkout on first run:
```
before <parent>/openwrt-build-helper/ this repository
first run <parent>/openwrt/ the OpenWrt source root
<parent>/openwrt/helper/ this repository, moved in
<parent>/openwrt-build-helper -> openwrt/helper (symlink)
later runs the same <parent>/openwrt/ is updated in place
```
The symlink means shells, editors and agent sessions still sitting in the old
path keep working. Disable it with `--no-compat-link`.
Inside `helper/`:
| Path | In git | Purpose |
|---|---|---|
| `packages/<device-id>.txt` | yes | your packages for one device, version independent |
| `.generated/` | no | combined package lists and saved buildinfo (build artefacts) |
## Requirements
- Linux with one of: apt (Debian/Ubuntu/Mint), dnf/yum (Fedora/RHEL/Rocky/Alma),
pacman (Arch/Manjaro), zypper (openSUSE), apk (Alpine)
- x86_64/amd64 or aarch64/arm64
- git, curl, wget, bash, python3 (used to read the upstream JSON indexes; jq is
not required)
- Internet access, and roughly 30 GB of disk for a full build
## Quick start
1) Prepare the host machine
- Run: ./prepare-openwrt-env.sh
- This installs compilers, headers, Python tooling, and other build prerequisites.
- Preview without touching the system: ./prepare-openwrt-env.sh -n
2) Get OpenWrt sources
- Run: ./prepare-openwrt.sh
- Choose a stable tag, a release branch, or the snapshot.
- Non-interactive: ./prepare-openwrt.sh --stable 25.12.5 -y
- Preview only, changes nothing: ./prepare-openwrt.sh --stable -n
- The script:
- Reuses or creates the top-level OpenWrt root at `./openwrt`.
- Moves this helper checkout to `./openwrt/helper` on first run.
- Updates/install feeds and runs make defconfig.
```bash
# 1. host dependencies
./prepare-openwrt-env.sh # -n to preview, --with-go for Go
3) Add optional external packages
- From the OpenWrt root:
- ./helper/add-external-repos.sh
- This pulls extra LuCI packages into package/.
# 2. OpenWrt source (relocates this checkout to openwrt/helper on first run)
./prepare-openwrt.sh # or --stable 25.12.5 -y
4) Import a device config (optional but convenient)
- From the OpenWrt root:
- ./helper/download-config.sh
- Search for your device by name, or run it non-interactively:
- ./helper/download-config.sh --device linksys_mx8500 -y
- Add `--with-packages` to also generate the device's package list in the same run.
- Preview without changing anything: `./helper/download-config.sh --device linksys_mx8500 -n`
# from here on, run everything from the OpenWrt root
cd ../openwrt # or: cd openwrt
5) Enable additional packages (optional)
- Your own packages live in `helper/packages/<device-id>.txt`, one file per device, independent of the OpenWrt version.
- Apply the built-in list plus your packages in one step:
- ./helper/gen-package-list.sh --device linksys_mx8500 --apply
- Or write the combined list out first and review it before applying:
- ./helper/gen-package-list.sh --device linksys_mx8500
- ./helper/add-openwrt-packages.sh helper/.generated/packages_linksys_mx8500_<version>.txt
# 3. the device's official configuration
./helper/download-config.sh --device linksys_mx8500 -y
# 4. built-in packages + your own, applied to .config
./helper/gen-package-list.sh --device linksys_mx8500 --apply -y
# 5. build
make -j$(nproc) download world
```
Firmware lands in `bin/targets/<target>/<subtarget>/`.
Run `./helper/download-config.sh` without `--device` to search for a device by
name. Add `--with-packages` to fold step 4 into step 3.
### Adding a device
1. `./helper/gen-package-list.sh --device <id> --init-extras` — creates `packages/<id>.txt`
2. Edit `packages/<id>.txt` and list the packages you want on top of the defaults
3. `./helper/gen-package-list.sh --device <id> --apply`
Use `./helper/download-config.sh` without `--device` to search for the id by name.
```bash
./helper/gen-package-list.sh --device <id> --init-extras # creates packages/<id>.txt
$EDITOR helper/packages/<id>.txt # list what you want
./helper/gen-package-list.sh --device <id> --apply
```
### Upgrading to a new OpenWrt release
Nothing to edit. Check out the new version, then:
```
Nothing to edit — the built-in list is re-fetched and `packages/<id>.txt` is
reused as is:
```bash
./helper/prepare-openwrt.sh --stable -y
cd ../openwrt
./helper/download-config.sh --device <id> -y
./helper/gen-package-list.sh --device <id> --apply -y
make -j$(nproc) download world
```
The built-in list is re-fetched for the new release and your `packages/<id>.txt` is reused as is.
### Files
- `packages/<device-id>.txt` — your packages, one file per device, kept in git and edited by hand.
- `.generated/` — combined package lists and saved buildinfo. Build artefacts, not kept in git.
## Scripts
- **prepare-openwrt-env.sh** — installs the build dependencies from the official
build system guide. Dispatches on the package manager rather than guessing from
the distro version, and probes every package against the running release so a
renamed or dropped package cannot abort the install.
`-y`, `-n/--dry-run`, `--with-go`.
- **prepare-openwrt.sh** — clones or updates the OpenWrt source in `openwrt/`.
Resolves the version over `git ls-remote` and prints a plan before touching
anything, so aborting at the prompt leaves the checkout untouched. Uses a
partial clone (`--filter=blob:none`) by default and resumes an interrupted
first run automatically.
`--stable [<ver>]`, `--branch <name>`, `--snapshot`, `-y`, `-n`, `-f/--force`,
`--full-clone`, `--allow-rc`, `--root <dir>`, `--skip-feeds`,
`--skip-defconfig`, `--repo-url <url>`, `--no-compat-link`.
- **download-config.sh** — installs a device's official `config.buildinfo` as
`.config`. Looks the device up in the official index, so any supported device
can be selected by name. Downloads to a temp file and validates it before
touching `.config`, keeping a `.config.download.bak`. By default builds only
the selected device and strips the buildbot flags (see below).
`--device <id>`, `--target <t>`, `--version <ver>`, `--snapshot`,
`--with-packages`, `--all-profiles`, `--keep-buildbot-flags`,
`--save-buildinfo`, `--offline`, `--output <file>`, `-y`, `-n`.
- **gen-package-list.sh** — builds the complete package list: the built-in part
from the target's `profiles.json` (`default_packages` + `device_packages` plus
the two packages the Firmware Selector adds), combined with your
`packages/<device-id>.txt`.
`--device <id>`, `--apply`, `--init-extras`, `--stdout`, `--full-stdout`,
`--out <path>`, `--no-extras`, `-y`, `-n`.
- **add-openwrt-packages.sh** — applies a package list to `.config`. Rewrites
existing `CONFIG_PACKAGE_` lines in place, so repeated runs do not grow the
file. After `make defconfig` it reports any package whose final state differs
from what you asked for.
`-y`, `-n`, `--no-defconfig`, `--no-backup`, `--strict`, and `-` to read stdin.
- **update-go-path.sh** — points the build at an installed Go toolchain.
`--external` clears `CONFIG_GOLANG_BUILD_BOOTSTRAP` so the installed toolchain
builds host Go directly instead of OpenWrt building the whole bootstrap chain
from source.
`--external`, `--go-root <dir>`, `-n`.
- **add-external-repos.sh** — clones extra package repositories into `package/`.
Edit the `REPOS` array to change the list.
## Package list format
### Package list format
Used by both `packages/<device-id>.txt` and the generated lists:
```
curl yq luci-app-ttyd packages, any number per line, set to =y
-luci-app-wol a leading '-' removes a package (=n)
@@ -117,95 +148,98 @@ curl yq luci-app-ttyd packages, any number per line, set to =y
pkg # note text after a single '#' is a comment
```
## Notes and gotchas
### Buildbot flags
The official config.buildinfo is the buildbot's own configuration: it enables every
model in the target and sets ALL_KMODS / ALL_NONSHARED / SDK / IB / MAKE_TOOLCHAIN /
COLLECT_KERNEL_DEBUG / AUTOREMOVE. For a Linksys MX8500 that means 1248 extra module
packages plus the SDK, ImageBuilder and a toolchain tarball on every build.
download-config.sh turns these off by default. This was verified not to change the
firmware: the set of packages built into the image (`=y`) is identical either way —
193 packages before and after. The trade-off is that bin/ no longer contains prebuilt
packages for modules you did not select, so you cannot later install an arbitrary kmod
from your own build output. Use `--keep-buildbot-flags` to restore upstream behaviour,
and `--all-profiles` to build every model in the target.
The official `config.buildinfo` is the buildbot's own configuration: it enables
every model in the target and sets `ALL_KMODS`, `ALL_NONSHARED`, `SDK`, `IB`,
`MAKE_TOOLCHAIN`, `COLLECT_KERNEL_DEBUG` and `AUTOREMOVE`. For a Linksys MX8500
that is 1248 extra module packages plus the SDK, ImageBuilder and a toolchain
tarball on every build.
6) Configure Go bootstrap path (if needed)
- From the OpenWrt root:
- ./helper/update-go-path.sh
- This sets CONFIG_GOLANG_EXTERNAL_BOOTSTRAP_ROOT to the latest /usr/lib/go-X.XX/ found.
- You can override it manually in .config if necessary.
`download-config.sh` turns these off by default. This was verified not to change
the firmware: the set of packages built into the image (`=y`) is identical either
way — 193 packages before and after. The trade-off is that `bin/` no longer
contains prebuilt packages for modules you did not select, so you cannot later
install an arbitrary kmod from your own build output.
`--keep-buildbot-flags` and `--all-profiles` restore upstream behaviour.
### Go packages
Packages such as tailscale, adguardhome, easytier and cloudflared need a host Go.
By default OpenWrt builds the entire bootstrap chain from source, which works but
is slow. To skip it, install a Go matching the version the tree expects and run:
- Ensure telephony feed is installed:
- ./scripts/feeds update telephony
- ./scripts/feeds install -a
- Then run:
```bash
sudo apt install golang-1.24-go
./helper/update-go-path.sh --external
```
- Build the package:
### WSL: Windows PATH breaks Go package builds
On WSL, `PATH` includes Windows directories containing spaces and parentheses
(`/mnt/c/Program Files (x86)/...`). OpenWrt's Go build recipe interpolates `PATH`
unquoted into a shell command, so every Go package fails with:
- Common commands:
- make menuconfig
- make -j$(nproc) download world
- Artifacts will be in bin/ after the build completes.
```
bash: -c: line 1: syntax error near unexpected token `('
```
## Script usage notes and tips
- Run locations:
- `prepare-openwrt.sh`: resolves paths from the script location, not your current shell directory.
- `prepare-openwrt-env.sh`: can be run before the first normalization from the fresh helper checkout.
- All other helper scripts: run them from inside the OpenWrt source root as `./helper/...` unless otherwise indicated.
- Feeds:
- If a package cant be found, ensure feeds are updated/installed:
- ./scripts/feeds update -a && ./scripts/feeds install -a
- Go toolchain:
- On x86_64, Go is optional unless your selection pulls packages needing Go (e.g., firewall4 in some versions). If needed:
- sudo apt install golang -y
- Then use update-go-path.sh to set .config automatically.
- ARM64 multilib:
- The script installs cross multilib packages for better compatibility, but not all targets need them. If apt errors on specific multilib packages, remove or adjust as appropriate for your environment.
Either build with a sanitized PATH:
```bash
CLEAN_PATH=$(echo "$PATH" | tr ':' '\n' | grep -v '^/mnt/' | paste -sd:)
env PATH="$CLEAN_PATH" make -j$(nproc) download world
```
or disable the interop PATH permanently in `/etc/wsl.conf` and restart WSL
(`wsl --shutdown`):
```ini
[interop]
appendWindowsPath = false
```
### Hazard: `git clean -xdff`
`helper/` lives inside the OpenWrt Git worktree but is not tracked by it.
`prepare-openwrt.sh` adds `/helper/` to `openwrt/.git/info/exclude`, which keeps
`git status` clean and stops `git add -A` from staging it — but that does **not**
protect it from `git clean -xdff`: `-x` deliberately includes ignored files and
`-ff` removes Git's refusal to recurse into a nested repository. Use instead:
```bash
git clean -xdf -e /helper
```
## Troubleshooting
- Version detection in download-config.sh:
- It uses, in order: an explicit `--version`/`--snapshot`; the exact tag at HEAD; the branch (`main`/`master` → SNAPSHOT, `openwrt-XX.YY` → the newest matching tag merged into HEAD); otherwise it asks. If HEAD is ahead of the tag it warns that the downloaded config describes the release rather than your tree.
- If the device symbol does not survive `make defconfig`, the script stops and tells you the tree does not match that release, and points at the .config.download.bak it kept.
- Missing packages in add-openwrt-packages.sh:
- It verifies names against tmp/.config-package.in and tmp/.packageinfo. If a package is reported missing, ensure the corresponding feed is enabled in feeds.conf.default and run feeds update/install.
- A removed package (`-pkg`) comes back after make defconfig:
- Something else depends on it. The script reports this explicitly as "wanted =n, got =y".
- Permission errors:
- Scripts use umask 022 and do not require root except when installing apt packages.
- Clean up and rebuild a package:
- make package/<pkg>/{clean,compile} V=s
- Full rebuild:
- make clean; make -j$(nproc) download world
## Directory behavior
- If you start from a freshly cloned helper checkout at `<parent>/<origin_folder>`, `prepare-openwrt.sh` will:
- Reuse or create `<parent>/openwrt/` as the OpenWrt source root.
- Move the entire helper checkout, including its `.git`, to `<parent>/openwrt/helper/`.
- Initialize or update the OpenWrt source tree directly in `<parent>/openwrt/`.
- Leave a symlink at the original checkout path pointing to `<parent>/openwrt/helper/`, so shells, editors and agent sessions whose working directory is still the old path keep working. Disable with `--no-compat-link`.
- If you later rerun `prepare-openwrt.sh` — from `openwrt/helper/` or through the symlink at the old path — it reuses the parent `openwrt/` directory and does not move anything again.
- The already-relocated layout is detected by content (the OpenWrt tree or its `origin` remote), not by directory name, so renaming `openwrt/` to something else does not trigger a second relocation.
- If a run is interrupted after the repository was created but before the source tree was checked out, the next run detects that state, reports `resuming it`, and completes. No manual cleanup is needed.
- If the script is already located in a real OpenWrt source root, it treats that directory as the source root and does not relocate it just because the directory name is `openwrt`.
- Use `--root <dir>` to point at a different OpenWrt root in any ambiguous situation.
- **Version detection in download-config.sh** — it uses, in order: an explicit
`--version`/`--snapshot`; the exact tag at HEAD; the branch (`main`/`master`
SNAPSHOT, `openwrt-XX.YY` the newest matching tag merged into HEAD);
otherwise it asks. If HEAD is ahead of the tag it warns that the downloaded
config describes the release rather than your tree. Version queries are
anchored to the OpenWrt tree, not to whichever repository the script sits in.
- **The device symbol did not survive make defconfig** — your tree does not match
the release you downloaded a config for. The script stops and points at the
`.config.download.bak` it kept.
- **A package is reported missing** — names are verified against
`tmp/.config-package.in` and `tmp/.packageinfo`. Enable the relevant feed in
`feeds.conf.default`, then `./scripts/feeds update -a && ./scripts/feeds install -a`.
- **A removed package (`-pkg`) comes back after make defconfig** — something else
depends on it. Reported explicitly as "wanted =n, got =y".
- **A package is in .config but not in the manifest** — it is probably there under
a provider name (`libgcc``libgcc1`, `nftables``nftables-json`).
- **Rebuild one package** — `make package/<pkg>/{clean,compile} V=s`
- **Full rebuild** — `make clean; make -j$(nproc) download world`
### Known hazard: `git clean -xdff`
`helper/` lives inside the OpenWrt Git worktree but is not tracked by it. The script
adds `/helper/` to `openwrt/.git/info/exclude`, which keeps `git status` clean and stops
`git add -A` from staging it — but that does **not** protect it from `git clean -xdff`:
`-x` deliberately includes ignored files and `-ff` removes Git's refusal to recurse into
a nested repository. Running `git clean -xdff` in the OpenWrt root will delete the helper
checkout including its `.git`. Use this instead:
## Safety
git clean -xdf -e /helper
## Security and safety
- Scripts use set -euo pipefail to stop on errors.
- prepare-openwrt.sh backs up `.config` to `.config.<previous-ref>.bak` before switching versions, and refuses to switch a tree with modified tracked files unless you pass --force.
- They modify .config and write under package/ and feeds/ inside your OpenWrt tree.
- Backup your .config if you need a known baseline.
- Scripts use `set -euo pipefail` and `umask 022`, and need root only to install
host packages.
- `.config` is backed up before it is replaced or before a version switch.
- `prepare-openwrt.sh` refuses to switch versions on a tree with modified tracked
files unless you pass `--force`.
- Scripts write under `package/` and `feeds/` inside your OpenWrt tree.
## License and authorship
- Scripts authored by Zhe Yuan with help from ChatGPT.
- Scripts authored by Zhe Yuan.
- License: MIT (unless you choose a different license; update this line accordingly).