openwrt-build-helper/README.md

# OpenWrt Build Helper Scripts

Shell scripts that automate building OpenWrt from source: installing the host
dependencies, fetching the source tree, importing a device's official
configuration, and keeping a per-device package list.

The design goal is that **a new OpenWrt release requires no file changes** —
everything version specific is fetched from upstream, and only your own package
choices are stored in this repository.

## Layout

`prepare-openwrt.sh` normalizes the checkout on first run:

```
before        <parent>/openwrt-build-helper/       this repository
first run     <parent>/openwrt/                    the OpenWrt source root
              <parent>/openwrt/helper/             this repository, moved in
              <parent>/openwrt-build-helper -> openwrt/helper   (symlink)
later runs    the same <parent>/openwrt/ is updated in place
```

The symlink means shells, editors and agent sessions still sitting in the old
path keep working. Disable it with `--no-compat-link`.

Inside `helper/`:

| Path | In git | Purpose |
|---|---|---|
| `packages/<device-id>.txt` | yes | your packages for one device, version independent |
| `.generated/` | no | combined package lists and saved buildinfo (build artefacts) |

## Requirements

- Linux with one of: apt (Debian/Ubuntu/Mint), dnf/yum (Fedora/RHEL/Rocky/Alma),
  pacman (Arch/Manjaro), zypper (openSUSE), apk (Alpine)
- x86_64/amd64 or aarch64/arm64
- git, curl, wget, bash, python3 (used to read the upstream JSON indexes; jq is
  not required)
- Internet access, and roughly 30 GB of disk for a full build

## Quick start

```bash
# 1. host dependencies
./prepare-openwrt-env.sh                    # -n to preview, --with-go for Go

# 2. OpenWrt source (relocates this checkout to openwrt/helper on first run)
./prepare-openwrt.sh                        # or --stable 25.12.5 -y

# from here on, run everything from the OpenWrt root
cd ../openwrt        # or: cd openwrt

# 3. the device's official configuration
./helper/download-config.sh --device linksys_mx8500 -y

# 4. built-in packages + your own, applied to .config
./helper/gen-package-list.sh --device linksys_mx8500 --apply -y

# 5. build
make -j$(nproc) download world
```

Firmware lands in `bin/targets/<target>/<subtarget>/`.

Run `./helper/download-config.sh` without `--device` to search for a device by
name. Add `--with-packages` to fold step 4 into step 3.

### Adding a device

```bash
./helper/gen-package-list.sh --device <id> --init-extras   # creates packages/<id>.txt
$EDITOR helper/packages/<id>.txt                           # list what you want
./helper/gen-package-list.sh --device <id> --apply
```

### Upgrading to a new OpenWrt release

Nothing to edit — the built-in list is re-fetched and `packages/<id>.txt` is
reused as is:

```bash
./helper/prepare-openwrt.sh --stable -y
cd ../openwrt
./helper/download-config.sh --device <id> -y
./helper/gen-package-list.sh --device <id> --apply -y
make -j$(nproc) download world
```

## Scripts

Every script takes `-h/--help` and prints its full option list; the summaries
below cover the options worth knowing about.

- **prepare-openwrt-env.sh** — installs the build dependencies from the official
  build system guide. Dispatches on the package manager rather than guessing from
  the distro version, and probes every package against the running release so a
  renamed or dropped package cannot abort the install.
  `-y`, `-n/--dry-run`, `--with-go`.

- **prepare-openwrt.sh** — clones or updates the OpenWrt source in `openwrt/`.
  Resolves the version over `git ls-remote` and prints a plan before touching
  anything, so aborting at the prompt leaves the checkout untouched. Uses a
  partial clone (`--filter=blob:none`) by default and resumes an interrupted
  first run automatically.
  `--stable [<ver>]`, `--branch <name>`, `--snapshot`, `-y`, `-n`, `-f/--force`,
  `--full-clone`, `--allow-rc`, `--root <dir>`, `--skip-feeds`,
  `--skip-defconfig`, `--repo-url <url>`, `--no-compat-link`.

- **download-config.sh** — installs a device's official `config.buildinfo` as
  `.config`. Looks the device up in the official index, so any supported device
  can be selected by name. Downloads to a temp file and validates it before
  touching `.config`, keeping a `.config.download.bak`. By default builds only
  the selected device and strips the buildbot flags (see below).
  `--device <id>`, `--target <t>`, `--version <ver>`, `--snapshot`,
  `--with-packages`, `--all-profiles`, `--keep-buildbot-flags`,
  `--save-buildinfo`, `--offline`, `--output <file>`, `--refresh`,
  `--skip-defconfig`, `-y`, `-n`.

- **gen-package-list.sh** — builds the complete package list: the built-in part
  from the target's `profiles.json` (`default_packages` + `device_packages` plus
  the two packages the Firmware Selector adds), combined with your
  `packages/<device-id>.txt`.
  `--device <id>`, `--target <t>`, `--version <ver>`, `--snapshot`,
  `--apply`, `--init-extras`, `--extras-file <f>`, `--no-extras-file`,
  `--stdout`, `--full-stdout`, `--out <path>`, `--extras "<pkgs>"`,
  `--no-extras`, `--wrap <cols>`, `--refresh`, `-y`, `-n`.

  `--extras`/`--no-extras` control the two packages the Firmware Selector adds
  on top of the target's own set; `--extras-file`/`--no-extras-file` control
  your `packages/<device-id>.txt`. They are different things.

- **add-openwrt-packages.sh** — applies a package list to `.config`. Rewrites
  existing `CONFIG_PACKAGE_` lines in place, so repeated runs do not grow the
  file. After `make defconfig` it reports any package whose final state differs
  from what you asked for.
  `-y`, `-n`, `--no-defconfig`, `--no-backup`, `--strict`, and `-` to read stdin.

- **update-go-path.sh** — points the build at an installed Go toolchain.
  `--external` clears `CONFIG_GOLANG_BUILD_BOOTSTRAP` so the installed toolchain
  builds host Go directly instead of OpenWrt building the whole bootstrap chain
  from source.
  `--external`, `--go-root <dir>`, `--skip-defconfig`, `-n`.

- **add-external-repos.sh** — clones extra package repositories into `package/`.
  Edit the `REPOS` array to change the list.

## Package list format

Used by both `packages/<device-id>.txt` and the generated lists:

```
curl yq luci-app-ttyd      packages, any number per line, set to =y
-luci-app-wol              a leading '-' removes a package (=n)
##module                   packages after this are set to =m
##remove                   packages after this are set to =n
##built-in                 back to =y (the default)
pkg  # note                text after a single '#' is a comment
```

## Notes and gotchas

### Buildbot flags
The official `config.buildinfo` is the buildbot's own configuration: it enables
every model in the target and sets `ALL_KMODS`, `ALL_NONSHARED`, `SDK`, `IB`,
`MAKE_TOOLCHAIN`, `COLLECT_KERNEL_DEBUG` and `AUTOREMOVE`. For a Linksys MX8500
that is 1248 extra module packages plus the SDK, ImageBuilder and a toolchain
tarball on every build.

`download-config.sh` turns these off by default. This was verified not to change
the firmware: the set of packages built into the image (`=y`) is identical either
way — 193 packages before and after. The trade-off is that `bin/` no longer
contains prebuilt packages for modules you did not select, so you cannot later
install an arbitrary kmod from your own build output.
`--keep-buildbot-flags` and `--all-profiles` restore upstream behaviour.

### Go packages
Packages such as tailscale, adguardhome, easytier and cloudflared need a host Go.
By default OpenWrt builds the entire bootstrap chain from source, which works but
is slow. To skip it, install a Go from your distribution (or from go.dev) at
least as new as the version the tree expects, then run:

```bash
./helper/update-go-path.sh --external
```

The script searches `/usr/lib/go-*`, `/usr/local/go` and `/usr/lib/golang`,
picks the newest, and warns if it is older than the bootstrap version the tree
asks for. Point it elsewhere with `--go-root <dir>`.

### WSL: Windows PATH breaks Go package builds
On WSL, `PATH` includes Windows directories containing spaces and parentheses
(`/mnt/c/Program Files (x86)/...`). OpenWrt's Go build recipe interpolates `PATH`
unquoted into a shell command, so every Go package fails with:

```
bash: -c: line 1: syntax error near unexpected token `('
```

Either build with a sanitized PATH:

```bash
CLEAN_PATH=$(echo "$PATH" | tr ':' '\n' | grep -v '^/mnt/' | paste -sd:)
env PATH="$CLEAN_PATH" make -j$(nproc) download world
```

or disable the interop PATH permanently in `/etc/wsl.conf` and restart WSL
(`wsl --shutdown`):

```ini
[interop]
appendWindowsPath = false
```

### Hazard: `git clean -xdff`
`helper/` lives inside the OpenWrt Git worktree but is not tracked by it.
`prepare-openwrt.sh` adds `/helper/` to `openwrt/.git/info/exclude`, which keeps
`git status` clean and stops `git add -A` from staging it — but that does **not**
protect it from `git clean -xdff`: `-x` deliberately includes ignored files and
`-ff` removes Git's refusal to recurse into a nested repository. Use instead:

```bash
git clean -xdf -e /helper
```

## Troubleshooting

- **Version detection in download-config.sh** — it uses, in order: an explicit
  `--version`/`--snapshot`; the exact tag at HEAD; the branch (`main`/`master` →
  SNAPSHOT, `openwrt-XX.YY` → the newest matching tag merged into HEAD);
  otherwise it asks. If HEAD is ahead of the tag it warns that the downloaded
  config describes the release rather than your tree. Version queries are
  anchored to the OpenWrt tree, not to whichever repository the script sits in.
- **The device symbol did not survive make defconfig** — your tree does not match
  the release you downloaded a config for. The script stops and points at the
  `.config.download.bak` it kept.
- **A package is reported missing** — names are verified against
  `tmp/.config-package.in` and `tmp/.packageinfo`. Enable the relevant feed in
  `feeds.conf.default`, then `./scripts/feeds update -a && ./scripts/feeds install -a`.
- **A removed package (`-pkg`) comes back after make defconfig** — something else
  depends on it. Reported explicitly as "wanted =n, got =y".
- **A package is in .config but not in the manifest** — it is probably there under
  a provider name (`libgcc` → `libgcc1`, `nftables` → `nftables-json`).
- **Rebuild one package** — `make package/<pkg>/{clean,compile} V=s`
- **Full rebuild** — `make clean; make -j$(nproc) download world`

## Safety

- Scripts use `set -euo pipefail` and `umask 022`, and need root only to install
  host packages.
- `.config` is backed up before it is replaced or before a version switch.
- `prepare-openwrt.sh` refuses to switch versions on a tree with modified tracked
  files unless you pass `--force`.
- Scripts write under `package/` and `feeds/` inside your OpenWrt tree.

## Contributing and authorship

- Scripts authored by Zhe Yuan.
- No `LICENSE` file is present yet; add one before redistributing.
- [AGENTS.md](AGENTS.md) documents the layout, the shell conventions and the
  invariants to preserve — read it before changing a script.