openwrt-build-helper/README.md

# OpenWrt Build Helper Scripts

## Overview
This repository contains shell scripts that streamline preparing, configuring, and building OpenWrt from source on Debian/Ubuntu systems (x86_64 and ARM64). It also includes convenience utilities to manage package selections, fetch a device's official configuration, and add external LuCI packages.

## What’s included
- prepare-openwrt-env.sh
  - Installs the build dependencies listed by the official OpenWrt build system guide.
  - Supports apt (Debian/Ubuntu/Mint), dnf/yum (Fedora/RHEL/Rocky/Alma), pacman (Arch/Manjaro), zypper (openSUSE) and apk (Alpine).
  - Probes each package against the running release and skips the ones that no longer exist, so a renamed or dropped package cannot abort the whole install.
  - Handles x86_64 and ARM64 differences (native vs. cross multilib).
  - Options: `-y/--yes` (no prompt), `-n/--dry-run` (only print), `--with-go` (also install the Go toolchain).
- prepare-openwrt.sh
  - Clones or updates the OpenWrt source repo in the top-level `openwrt/` root.
  - Lets you select a stable tag, a release branch, or the snapshot (`main`) interactively, or non-interactively via flags.
  - Uses the script location as the source of truth, so running it from another current working directory is supported.
  - On first run from a helper checkout, relocates that checkout to `openwrt/helper/` and leaves a symlink behind at the old path.
  - Prints a plan and asks for confirmation; nothing on disk is touched before you confirm.
  - Uses a partial clone (`--filter=blob:none`) by default, so the first fetch is far smaller than full history.
  - Resumes automatically if a previous run was interrupted.
  - Updates feeds and runs make defconfig.
  - Options: `--stable [<ver>]`, `--branch <name>`, `--snapshot`, `-y/--yes`, `-n/--dry-run`,
    `-f/--force`, `--full-clone`, `--allow-rc`, `--root <dir>`, `--skip-feeds`,
    `--skip-defconfig`, `--repo-url <url>`, `--no-compat-link`, `-h/--help`.
- add-external-repos.sh
  - Clones or updates additional package repositories into package/.
  - Currently includes luci-app-netspeedtest and luci-app-easytier (adjust REPOS as needed).
- download-config.sh
  - Downloads config.buildinfo for a device and installs it as .config.
  - Looks the device up in the official index, so any supported device can be picked by name; nothing is hardcoded. Use `--device <id>` for non-interactive runs.
  - By default builds only the selected device instead of every model in the target, and turns off the buildbot flags (ALL_KMODS, ALL_NONSHARED, SDK, IB, MAKE_TOOLCHAIN, COLLECT_KERNEL_DEBUG, AUTOREMOVE). Neither changes the firmware contents — see "Buildbot flags" below.
  - Downloads to a temp file and validates it before touching .config, and keeps a .config.download.bak.
  - Prints the official sysupgrade/factory URLs (taken from profiles.json, so they are always correct) and a Firmware Selector link.
- gen-package-list.sh
  - Builds a device's complete package list: the built-in part from the official profiles.json (default_packages + device_packages + the packages the Firmware Selector adds), combined with your own packages from `packages/<device-id>.txt`.
  - The built-in part never has to be copied by hand for a new release, and your packages are version independent — a new release needs no file changes at all.
  - Writes the combined list to `.generated/` (not kept in git); `--apply` applies it straight to .config, `--full-stdout` just prints it.
- add-openwrt-packages.sh
  - Applies a package list to .config: `##built-in` → =y, `##module` → =m, `##remove` or a `-pkg` prefix → =n.
  - Rewrites existing CONFIG_PACKAGE_ lines in place, so repeated runs do not grow .config.
  - Verifies each package against tmp/.config-package.in and tmp/.packageinfo and reports missing ones.
  - After make defconfig it reports any package whose final state differs from what you asked for (a dependency pulling a removed package back in, for example).
- update-go-path.sh
  - Detects the latest Go installation under /usr/lib/go-* and sets CONFIG_GOLANG_EXTERNAL_BOOTSTRAP_ROOT in .config.

- OS: Debian/Ubuntu, Fedora/RHEL, Arch, openSUSE or Alpine (the other scripts are developed and tested on Debian/Ubuntu)
- Architectures: x86_64/amd64 and aarch64/arm64
- Tools: git, wget, curl, bash, python3 (used to read the official JSON indexes; jq is not required)
- Internet access for feeds and package downloads
- Optional: Go (required by some OpenWrt packages; install it with `./prepare-openwrt-env.sh --with-go`; configurable via update-go-path.sh)
- Note: with the default partial clone, switching to another tag or branch later fetches missing blobs on demand and therefore needs network access

## Quick start
1) Prepare the host machine
- Run: ./prepare-openwrt-env.sh
- This installs compilers, headers, Python tooling, and other build prerequisites.
- Preview without touching the system: ./prepare-openwrt-env.sh -n

2) Get OpenWrt sources
- Run: ./prepare-openwrt.sh
- Choose a stable tag, a release branch, or the snapshot.
- Non-interactive: ./prepare-openwrt.sh --stable 25.12.5 -y
- Preview only, changes nothing: ./prepare-openwrt.sh --stable -n
- The script:
  - Reuses or creates the top-level OpenWrt root at `./openwrt`.
  - Moves this helper checkout to `./openwrt/helper` on first run.
  - Updates/install feeds and runs make defconfig.

3) Add optional external packages
- From the OpenWrt root:
  - ./helper/add-external-repos.sh
- This pulls extra LuCI packages into package/.

4) Import a device config (optional but convenient)
- From the OpenWrt root:
  - ./helper/download-config.sh
- Search for your device by name, or run it non-interactively:
  - ./helper/download-config.sh --device linksys_mx8500 -y
- Add `--with-packages` to also generate the device's package list in the same run.
- Preview without changing anything: `./helper/download-config.sh --device linksys_mx8500 -n`

5) Enable additional packages (optional)
- Your own packages live in `helper/packages/<device-id>.txt`, one file per device, independent of the OpenWrt version.
- Apply the built-in list plus your packages in one step:
  - ./helper/gen-package-list.sh --device linksys_mx8500 --apply
- Or write the combined list out first and review it before applying:
  - ./helper/gen-package-list.sh --device linksys_mx8500
  - ./helper/add-openwrt-packages.sh helper/.generated/packages_linksys_mx8500_<version>.txt

### Adding a device
1. `./helper/gen-package-list.sh --device <id> --init-extras` — creates `packages/<id>.txt`
2. Edit `packages/<id>.txt` and list the packages you want on top of the defaults
3. `./helper/gen-package-list.sh --device <id> --apply`

Use `./helper/download-config.sh` without `--device` to search for the id by name.

### Upgrading to a new OpenWrt release
Nothing to edit. Check out the new version, then:
```
./helper/download-config.sh --device <id> -y
./helper/gen-package-list.sh --device <id> --apply -y
```
The built-in list is re-fetched for the new release and your `packages/<id>.txt` is reused as is.

### Files
- `packages/<device-id>.txt` — your packages, one file per device, kept in git and edited by hand.
- `.generated/` — combined package lists and saved buildinfo. Build artefacts, not kept in git.

### Package list format
Used by both `packages/<device-id>.txt` and the generated lists:
```
curl yq luci-app-ttyd      packages, any number per line, set to =y
-luci-app-wol              a leading '-' removes a package (=n)
##module                   packages after this are set to =m
##remove                   packages after this are set to =n
##built-in                 back to =y (the default)
pkg  # note                text after a single '#' is a comment
```

### Buildbot flags
The official config.buildinfo is the buildbot's own configuration: it enables every
model in the target and sets ALL_KMODS / ALL_NONSHARED / SDK / IB / MAKE_TOOLCHAIN /
COLLECT_KERNEL_DEBUG / AUTOREMOVE. For a Linksys MX8500 that means 1248 extra module
packages plus the SDK, ImageBuilder and a toolchain tarball on every build.
download-config.sh turns these off by default. This was verified not to change the
firmware: the set of packages built into the image (`=y`) is identical either way —
193 packages before and after. The trade-off is that bin/ no longer contains prebuilt
packages for modules you did not select, so you cannot later install an arbitrary kmod
from your own build output. Use `--keep-buildbot-flags` to restore upstream behaviour,
and `--all-profiles` to build every model in the target.

6) Configure Go bootstrap path (if needed)
- From the OpenWrt root:
  - ./helper/update-go-path.sh
- This sets CONFIG_GOLANG_EXTERNAL_BOOTSTRAP_ROOT to the latest /usr/lib/go-X.XX/ found.
- You can override it manually in .config if necessary.


- Ensure telephony feed is installed:
  - ./scripts/feeds update telephony
  - ./scripts/feeds install -a
- Then run:

- Build the package:

- Common commands:
  - make menuconfig
  - make -j$(nproc) download world
- Artifacts will be in bin/ after the build completes.

## Script usage notes and tips
- Run locations:
  - `prepare-openwrt.sh`: resolves paths from the script location, not your current shell directory.
  - `prepare-openwrt-env.sh`: can be run before the first normalization from the fresh helper checkout.
  - All other helper scripts: run them from inside the OpenWrt source root as `./helper/...` unless otherwise indicated.
- Feeds:
  - If a package can’t be found, ensure feeds are updated/installed:
    - ./scripts/feeds update -a && ./scripts/feeds install -a
- Go toolchain:
  - On x86_64, Go is optional unless your selection pulls packages needing Go (e.g., firewall4 in some versions). If needed:
    - sudo apt install golang -y
    - Then use update-go-path.sh to set .config automatically.
- ARM64 multilib:
  - The script installs cross multilib packages for better compatibility, but not all targets need them. If apt errors on specific multilib packages, remove or adjust as appropriate for your environment.

## Troubleshooting
- Version detection in download-config.sh:
  - It uses, in order: an explicit `--version`/`--snapshot`; the exact tag at HEAD; the branch (`main`/`master` → SNAPSHOT, `openwrt-XX.YY` → the newest matching tag merged into HEAD); otherwise it asks. If HEAD is ahead of the tag it warns that the downloaded config describes the release rather than your tree.
  - If the device symbol does not survive `make defconfig`, the script stops and tells you the tree does not match that release, and points at the .config.download.bak it kept.
- Missing packages in add-openwrt-packages.sh:
  - It verifies names against tmp/.config-package.in and tmp/.packageinfo. If a package is reported missing, ensure the corresponding feed is enabled in feeds.conf.default and run feeds update/install.
- A removed package (`-pkg`) comes back after make defconfig:
  - Something else depends on it. The script reports this explicitly as "wanted =n, got =y".
- Permission errors:
  - Scripts use umask 022 and do not require root except when installing apt packages.
- Clean up and rebuild a package:
  - make package/<pkg>/{clean,compile} V=s
- Full rebuild:
  - make clean; make -j$(nproc) download world

## Directory behavior
- If you start from a freshly cloned helper checkout at `<parent>/<origin_folder>`, `prepare-openwrt.sh` will:
  - Reuse or create `<parent>/openwrt/` as the OpenWrt source root.
  - Move the entire helper checkout, including its `.git`, to `<parent>/openwrt/helper/`.
  - Initialize or update the OpenWrt source tree directly in `<parent>/openwrt/`.
  - Leave a symlink at the original checkout path pointing to `<parent>/openwrt/helper/`, so shells, editors and agent sessions whose working directory is still the old path keep working. Disable with `--no-compat-link`.
- If you later rerun `prepare-openwrt.sh` — from `openwrt/helper/` or through the symlink at the old path — it reuses the parent `openwrt/` directory and does not move anything again.
- The already-relocated layout is detected by content (the OpenWrt tree or its `origin` remote), not by directory name, so renaming `openwrt/` to something else does not trigger a second relocation.
- If a run is interrupted after the repository was created but before the source tree was checked out, the next run detects that state, reports `resuming it`, and completes. No manual cleanup is needed.
- If the script is already located in a real OpenWrt source root, it treats that directory as the source root and does not relocate it just because the directory name is `openwrt`.
- Use `--root <dir>` to point at a different OpenWrt root in any ambiguous situation.

### Known hazard: `git clean -xdff`
`helper/` lives inside the OpenWrt Git worktree but is not tracked by it. The script
adds `/helper/` to `openwrt/.git/info/exclude`, which keeps `git status` clean and stops
`git add -A` from staging it — but that does **not** protect it from `git clean -xdff`:
`-x` deliberately includes ignored files and `-ff` removes Git's refusal to recurse into
a nested repository. Running `git clean -xdff` in the OpenWrt root will delete the helper
checkout including its `.git`. Use this instead:

    git clean -xdf -e /helper

## Security and safety
- Scripts use set -euo pipefail to stop on errors.
- prepare-openwrt.sh backs up `.config` to `.config.<previous-ref>.bak` before switching versions, and refuses to switch a tree with modified tracked files unless you pass --force.
- They modify .config and write under package/ and feeds/ inside your OpenWrt tree.
- Backup your .config if you need a known baseline.

## License and authorship
- Scripts authored by Zhe Yuan with help from ChatGPT.
- License: MIT (unless you choose a different license; update this line accordingly).