Generate the built-in package list from the official index

The ##built-in section of each package list was copied by hand from the
Firmware Selector for every new release. It is now generated from the
target's profiles.json (default_packages + device_packages, plus the two
packages the Selector adds on top), which reproduces the hand-written list
byte for byte, order included.

New lib-openwrt-upstream.sh holds the shared upstream access: URL
construction, cached fetches, version detection, device search over
.overview.json and profiles.json parsing. It needs python3, not jq.

New gen-package-list.sh writes config_<ver>/packages_<dev>_<ver>.txt,
replacing only the generated block so hand-written extras and the ##module
section survive; --stdout and --apply skip the file entirely.

download-config.sh:
- Look the device up in the official index instead of a hardcoded table of
  three devices, so any supported device can be selected by name.
- Build the firmware URLs from profiles.json images[]. The old code pasted
  the target into the filename with its '/' intact, so the URL was always
  wrong and the check always warned.
- Stage the download in a temp file and validate it before touching
  .config, and keep a backup. A 404 used to truncate .config to nothing.
- Fix version detection: `git describe --tags --abbrev=0` returns a tag
  whenever any tag is reachable, so the entire fallback chain below it was
  unreachable and a branch tip silently produced the wrong release URL.
- Build only the selected device instead of all 35 in the target, and turn
  off the buildbot flags (ALL_KMODS, ALL_NONSHARED, SDK, IB,
  MAKE_TOOLCHAIN, COLLECT_KERNEL_DEBUG, AUTOREMOVE). Verified not to change
  the firmware: the set of packages built into the image is identical
  either way (193 packages), while the module packages to build drop from
  1248 to 0. --keep-buildbot-flags and --all-profiles restore the old
  behaviour.
- Assert the device symbol survived make defconfig, so a tree that does not
  match the release fails loudly instead of silently building the default.

add-openwrt-packages.sh:
- Rewrite existing CONFIG_PACKAGE_ lines in place. Repeated runs used to
  append duplicates and grow .config every time.
- Support removal via a -pkg prefix or a ##remove section.
- Fix the parser dropping a lone '#' or an unknown ##header into the
  package list, where it was reported as a missing package named '#'.
- Read .config once into a map instead of grepping it per package, and
  compare package names exactly rather than as regexes.
- After make defconfig, report any package whose final state differs from
  what was asked for, which is the only way to see a removal that a
  dependency pulled back in.

Delete add-openwrt-packages_old.sh; nothing referenced it and it was a
strict subset of the current script.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

README.md

@@ -26,12 +26,20 @@ This repository contains shell scripts that streamline preparing, configuring, a
   - Clones or updates additional package repositories into package/.
   - Currently includes luci-app-netspeedtest and luci-app-easytier (adjust REPOS as needed).
 - download-config.sh
-  - Downloads config.buildinfo for a selected device based on your checked-out OpenWrt version (stable or snapshot).
-  - Writes .config and runs make defconfig.
-  - Prints firmware URL and a Firmware Selector link for reference.
+  - Downloads config.buildinfo for a device and installs it as .config.
+  - Looks the device up in the official index, so any supported device can be picked by name; nothing is hardcoded. Use `--device <id>` for non-interactive runs.
+  - By default builds only the selected device instead of every model in the target, and turns off the buildbot flags (ALL_KMODS, ALL_NONSHARED, SDK, IB, MAKE_TOOLCHAIN, COLLECT_KERNEL_DEBUG, AUTOREMOVE). Neither changes the firmware contents — see "Buildbot flags" below.
+  - Downloads to a temp file and validates it before touching .config, and keeps a .config.download.bak.
+  - Prints the official sysupgrade/factory URLs (taken from profiles.json, so they are always correct) and a Firmware Selector link.
+- gen-package-list.sh
+  - Generates the `##built-in` package list for a device from the official profiles.json (default_packages + device_packages + the packages the Firmware Selector adds), so it no longer has to be copied by hand for each release.
+  - Writes config_<version>/packages_<device>_<version>.txt, replacing only the generated block and preserving your own packages and the `##module` section verbatim.
+  - `--stdout` prints the list, `--apply` applies it straight to .config.
 - add-openwrt-packages.sh
-  - Enables packages listed in a text file by appending CONFIG_PACKAGE_<name>=y to .config (without removing existing settings).
-  - Verifies presence in package/ or feeds/ and reports missing ones, then runs make defconfig.
+  - Applies a package list to .config: `##built-in` → =y, `##module` → =m, `##remove` or a `-pkg` prefix → =n.
+  - Rewrites existing CONFIG_PACKAGE_ lines in place, so repeated runs do not grow .config.
+  - Verifies each package against tmp/.config-package.in and tmp/.packageinfo and reports missing ones.
+  - After make defconfig it reports any package whose final state differs from what you asked for (a dependency pulling a removed package back in, for example).
 - update-go-path.sh
   - Detects the latest Go installation under /usr/lib/go-* and sets CONFIG_GOLANG_EXTERNAL_BOOTSTRAP_ROOT in .config.
 - apply-dahdi-patches.sh
@@ -41,7 +49,7 @@ This repository contains shell scripts that streamline preparing, configuring, a
 ## System requirements
 - OS: Debian/Ubuntu, Fedora/RHEL, Arch, openSUSE or Alpine (the other scripts are developed and tested on Debian/Ubuntu)
 - Architectures: x86_64/amd64 and aarch64/arm64
-- Tools: git, wget, curl, bash
+- Tools: git, wget, curl, bash, python3 (used to read the official JSON indexes; jq is not required)
 - Internet access for feeds and package downloads
 - Optional: Go (required by some OpenWrt packages; install it with `./prepare-openwrt-env.sh --with-go`; configurable via update-go-path.sh)
 - Note: with the default partial clone, switching to another tag or branch later fetches missing blobs on demand and therefore needs network access
@@ -67,15 +75,45 @@ This repository contains shell scripts that streamline preparing, configuring, a
   - ./helper/add-external-repos.sh
 - This pulls extra LuCI packages into package/.
 
-4) Import a device config.buildinfo (optional but convenient)
+4) Import a device config (optional but convenient)
 - From the OpenWrt root:
   - ./helper/download-config.sh
-- Pick a device, and the script writes .config and runs make defconfig.
+- Search for your device by name, or run it non-interactively:
+  - ./helper/download-config.sh --device linksys_mx8500 -y
+- Add `--with-packages` to also generate the device's package list in the same run.
+- Preview without changing anything: `./helper/download-config.sh --device linksys_mx8500 -n`
 
 5) Enable additional packages from a list (optional)
-- Create a file packages.txt containing package names (whitespace or newlines).
-- From the OpenWrt root:
-  - ./helper/add-openwrt-packages.sh packages.txt
+- Generate the built-in list for your device:
+  - ./helper/gen-package-list.sh --device linksys_mx8500
+- Edit config_<version>/packages_<device>_<version>.txt and add your own packages below the generated block.
+- Apply it from the OpenWrt root:
+  - ./helper/add-openwrt-packages.sh helper/config_<version>/packages_<device>_<version>.txt
+
+### Package list format
+```
+##built-in                 packages after this are set to =y (the default)
+# >>> generated built-in list - do not edit by hand
+...                        regenerated by gen-package-list.sh; do not edit
+# <<< generated built-in list
+curl yq luci-app-ttyd      your own packages, any number per line
+-luci-app-wol              a leading '-' removes a package (=n)
+##module                   packages after this are set to =m
+##remove                   packages after this are set to =n
+```
+Text after a single `#` is a comment. Everything outside the generated block is preserved when the list is regenerated.
+
+### Buildbot flags
+The official config.buildinfo is the buildbot's own configuration: it enables every
+model in the target and sets ALL_KMODS / ALL_NONSHARED / SDK / IB / MAKE_TOOLCHAIN /
+COLLECT_KERNEL_DEBUG / AUTOREMOVE. For a Linksys MX8500 that means 1248 extra module
+packages plus the SDK, ImageBuilder and a toolchain tarball on every build.
+download-config.sh turns these off by default. This was verified not to change the
+firmware: the set of packages built into the image (`=y`) is identical either way —
+193 packages before and after. The trade-off is that bin/ no longer contains prebuilt
+packages for modules you did not select, so you cannot later install an arbitrary kmod
+from your own build output. Use `--keep-buildbot-flags` to restore upstream behaviour,
+and `--all-profiles` to build every model in the target.
 
 6) Configure Go bootstrap path (if needed)
 - From the OpenWrt root:
@@ -118,9 +156,12 @@ This repository contains shell scripts that streamline preparing, configuring, a
 - Telephony feed not found when applying DAHDI patch:
   - Run feeds update/install for telephony as shown above.
 - Version detection in download-config.sh:
-  - If you’re on a branch like openwrt-24.10, the script will try to find the latest v24.10.x tag automatically. Otherwise it prompts for a version (e.g., 24.10.4 or SNAPSHOT).
+  - It uses, in order: an explicit `--version`/`--snapshot`; the exact tag at HEAD; the branch (`main`/`master` → SNAPSHOT, `openwrt-XX.YY` → the newest matching tag merged into HEAD); otherwise it asks. If HEAD is ahead of the tag it warns that the downloaded config describes the release rather than your tree.
+  - If the device symbol does not survive `make defconfig`, the script stops and tells you the tree does not match that release, and points at the .config.download.bak it kept.
 - Missing packages in add-openwrt-packages.sh:
-  - Ensure the corresponding feed is enabled in feeds.conf.default and run feeds update/install.
+  - It verifies names against tmp/.config-package.in and tmp/.packageinfo. If a package is reported missing, ensure the corresponding feed is enabled in feeds.conf.default and run feeds update/install.
+- A removed package (`-pkg`) comes back after make defconfig:
+  - Something else depends on it. The script reports this explicitly as "wanted =n, got =y".
 - Permission errors:
   - Scripts use umask 022 and do not require root except when installing apt packages.
 - Clean up and rebuild a package: