diff --git a/README.md b/README.md index 9aa3840..55409bb 100644 --- a/README.md +++ b/README.md @@ -26,12 +26,20 @@ This repository contains shell scripts that streamline preparing, configuring, a - Clones or updates additional package repositories into package/. - Currently includes luci-app-netspeedtest and luci-app-easytier (adjust REPOS as needed). - download-config.sh - - Downloads config.buildinfo for a selected device based on your checked-out OpenWrt version (stable or snapshot). - - Writes .config and runs make defconfig. - - Prints firmware URL and a Firmware Selector link for reference. + - Downloads config.buildinfo for a device and installs it as .config. + - Looks the device up in the official index, so any supported device can be picked by name; nothing is hardcoded. Use `--device ` for non-interactive runs. + - By default builds only the selected device instead of every model in the target, and turns off the buildbot flags (ALL_KMODS, ALL_NONSHARED, SDK, IB, MAKE_TOOLCHAIN, COLLECT_KERNEL_DEBUG, AUTOREMOVE). Neither changes the firmware contents — see "Buildbot flags" below. + - Downloads to a temp file and validates it before touching .config, and keeps a .config.download.bak. + - Prints the official sysupgrade/factory URLs (taken from profiles.json, so they are always correct) and a Firmware Selector link. +- gen-package-list.sh + - Generates the `##built-in` package list for a device from the official profiles.json (default_packages + device_packages + the packages the Firmware Selector adds), so it no longer has to be copied by hand for each release. + - Writes config_/packages__.txt, replacing only the generated block and preserving your own packages and the `##module` section verbatim. + - `--stdout` prints the list, `--apply` applies it straight to .config. - add-openwrt-packages.sh - - Enables packages listed in a text file by appending CONFIG_PACKAGE_=y to .config (without removing existing settings). - - Verifies presence in package/ or feeds/ and reports missing ones, then runs make defconfig. + - Applies a package list to .config: `##built-in` → =y, `##module` → =m, `##remove` or a `-pkg` prefix → =n. + - Rewrites existing CONFIG_PACKAGE_ lines in place, so repeated runs do not grow .config. + - Verifies each package against tmp/.config-package.in and tmp/.packageinfo and reports missing ones. + - After make defconfig it reports any package whose final state differs from what you asked for (a dependency pulling a removed package back in, for example). - update-go-path.sh - Detects the latest Go installation under /usr/lib/go-* and sets CONFIG_GOLANG_EXTERNAL_BOOTSTRAP_ROOT in .config. - apply-dahdi-patches.sh @@ -41,7 +49,7 @@ This repository contains shell scripts that streamline preparing, configuring, a ## System requirements - OS: Debian/Ubuntu, Fedora/RHEL, Arch, openSUSE or Alpine (the other scripts are developed and tested on Debian/Ubuntu) - Architectures: x86_64/amd64 and aarch64/arm64 -- Tools: git, wget, curl, bash +- Tools: git, wget, curl, bash, python3 (used to read the official JSON indexes; jq is not required) - Internet access for feeds and package downloads - Optional: Go (required by some OpenWrt packages; install it with `./prepare-openwrt-env.sh --with-go`; configurable via update-go-path.sh) - Note: with the default partial clone, switching to another tag or branch later fetches missing blobs on demand and therefore needs network access @@ -67,15 +75,45 @@ This repository contains shell scripts that streamline preparing, configuring, a - ./helper/add-external-repos.sh - This pulls extra LuCI packages into package/. -4) Import a device config.buildinfo (optional but convenient) +4) Import a device config (optional but convenient) - From the OpenWrt root: - ./helper/download-config.sh -- Pick a device, and the script writes .config and runs make defconfig. +- Search for your device by name, or run it non-interactively: + - ./helper/download-config.sh --device linksys_mx8500 -y +- Add `--with-packages` to also generate the device's package list in the same run. +- Preview without changing anything: `./helper/download-config.sh --device linksys_mx8500 -n` 5) Enable additional packages from a list (optional) -- Create a file packages.txt containing package names (whitespace or newlines). -- From the OpenWrt root: - - ./helper/add-openwrt-packages.sh packages.txt +- Generate the built-in list for your device: + - ./helper/gen-package-list.sh --device linksys_mx8500 +- Edit config_/packages__.txt and add your own packages below the generated block. +- Apply it from the OpenWrt root: + - ./helper/add-openwrt-packages.sh helper/config_/packages__.txt + +### Package list format +``` +##built-in packages after this are set to =y (the default) +# >>> generated built-in list - do not edit by hand +... regenerated by gen-package-list.sh; do not edit +# <<< generated built-in list +curl yq luci-app-ttyd your own packages, any number per line +-luci-app-wol a leading '-' removes a package (=n) +##module packages after this are set to =m +##remove packages after this are set to =n +``` +Text after a single `#` is a comment. Everything outside the generated block is preserved when the list is regenerated. + +### Buildbot flags +The official config.buildinfo is the buildbot's own configuration: it enables every +model in the target and sets ALL_KMODS / ALL_NONSHARED / SDK / IB / MAKE_TOOLCHAIN / +COLLECT_KERNEL_DEBUG / AUTOREMOVE. For a Linksys MX8500 that means 1248 extra module +packages plus the SDK, ImageBuilder and a toolchain tarball on every build. +download-config.sh turns these off by default. This was verified not to change the +firmware: the set of packages built into the image (`=y`) is identical either way — +193 packages before and after. The trade-off is that bin/ no longer contains prebuilt +packages for modules you did not select, so you cannot later install an arbitrary kmod +from your own build output. Use `--keep-buildbot-flags` to restore upstream behaviour, +and `--all-profiles` to build every model in the target. 6) Configure Go bootstrap path (if needed) - From the OpenWrt root: @@ -118,9 +156,12 @@ This repository contains shell scripts that streamline preparing, configuring, a - Telephony feed not found when applying DAHDI patch: - Run feeds update/install for telephony as shown above. - Version detection in download-config.sh: - - If you’re on a branch like openwrt-24.10, the script will try to find the latest v24.10.x tag automatically. Otherwise it prompts for a version (e.g., 24.10.4 or SNAPSHOT). + - It uses, in order: an explicit `--version`/`--snapshot`; the exact tag at HEAD; the branch (`main`/`master` → SNAPSHOT, `openwrt-XX.YY` → the newest matching tag merged into HEAD); otherwise it asks. If HEAD is ahead of the tag it warns that the downloaded config describes the release rather than your tree. + - If the device symbol does not survive `make defconfig`, the script stops and tells you the tree does not match that release, and points at the .config.download.bak it kept. - Missing packages in add-openwrt-packages.sh: - - Ensure the corresponding feed is enabled in feeds.conf.default and run feeds update/install. + - It verifies names against tmp/.config-package.in and tmp/.packageinfo. If a package is reported missing, ensure the corresponding feed is enabled in feeds.conf.default and run feeds update/install. +- A removed package (`-pkg`) comes back after make defconfig: + - Something else depends on it. The script reports this explicitly as "wanted =n, got =y". - Permission errors: - Scripts use umask 022 and do not require root except when installing apt packages. - Clean up and rebuild a package: diff --git a/add-openwrt-packages.sh b/add-openwrt-packages.sh index f63ee4c..be0d34e 100755 --- a/add-openwrt-packages.sh +++ b/add-openwrt-packages.sh @@ -1,162 +1,340 @@ #!/bin/bash -# Add specified OpenWRT packages (from a text file) to .config without removing existing ones +# Apply a package list to the OpenWrt .config. +# +# The list is a plain text file of package names, whitespace separated, any +# number per line. Section headers switch how the following packages are +# applied, and a leading '-' on a name removes that package: +# +# ##built-in the packages after this are set to =y (default) +# ##module ... are set to =m +# ##remove ... are set to =n +# -luci-app-ttyd set to =n regardless of the current section +# pkg # comment text after a single # is ignored +# +# Existing CONFIG_PACKAGE_ lines are rewritten in place rather than appended, +# so running this repeatedly does not grow .config. +# +# Usage: ./add-openwrt-packages.sh [options] +# +# -y, --yes do not ask for confirmation +# -n, --dry-run show the resulting diff, change nothing +# --no-defconfig do not run make defconfig afterwards +# --no-backup do not keep a .config.bak +# --strict exit non-zero if any listed package was not found +# -h, --help show this help +# +# Run it from the OpenWrt root: ./helper/add-openwrt-packages.sh +# # Author: Zhe Yuan -# Usage: -# ./add-openwrt-packages.sh packages.txt -# Description: -# - Reads package names from a text file (supports both newline or space separation) -# - Supports ##built-in and ##module sections to control build mode (=y or =m) -# - Packages default to built-in (=y) if no section header is specified -# - Adds missing CONFIG_PACKAGE_xxx entries to .config -# - Keeps existing settings intact -# - Reports missing packages -# - Runs make defconfig at the end to sync dependencies -set -e +set -euo pipefail umask 022 -# Check argument -if [ "$#" -ne 1 ]; then - echo "❌ Usage: $0 " +PKG_FILE="" +ASSUME_YES=false +DRY_RUN=false +RUN_DEFCONFIG=true +KEEP_BACKUP=true +STRICT=false + +info() { echo "[INFO] $*"; } +warn() { echo "[WARN] $*"; } +err() { echo "[ERROR] $*" >&2; } + +usage() { sed -n '2,29p' "$0" | sed 's/^# \{0,1\}//;s/^#$//'; } + +confirm() { + local question="$1" default="$2" reply="" + [ "$ASSUME_YES" = true ] && return 0 + if [ "$default" = "yes" ]; then + read -r -p "$question [Y/n]: " reply || reply="" + case "${reply:-}" in [nN] | [nN][oO]) return 1 ;; *) return 0 ;; esac + else + read -r -p "$question [y/N]: " reply || reply="" + case "${reply:-}" in [yY] | [yY][eE][sS]) return 0 ;; *) return 1 ;; esac + fi +} + +while [ $# -gt 0 ]; do + case "$1" in + -y|--yes) ASSUME_YES=true ;; + -n|--dry-run) DRY_RUN=true ;; + --no-defconfig) RUN_DEFCONFIG=false ;; + --no-backup) KEEP_BACKUP=false ;; + --strict) STRICT=true ;; + -h|--help) usage; exit 0 ;; + -) PKG_FILE="-" ;; + -*) err "Unknown option: $1"; echo "Run '$0 --help' for usage." >&2; exit 1 ;; + *) + if [ -n "$PKG_FILE" ]; then + err "Only one package list can be given." + exit 1 + fi + PKG_FILE="$1" + ;; + esac + shift +done + +[ -t 0 ] || ASSUME_YES=true + +if [ -z "$PKG_FILE" ]; then + err "No package list given." + usage >&2 exit 1 fi -PKG_FILE="$1" - -# Verify file exists -if [ ! -f "$PKG_FILE" ]; then - echo "❌ Error: File not found: $PKG_FILE" +if [ "$PKG_FILE" != "-" ] && [ ! -f "$PKG_FILE" ]; then + err "Package list not found: $PKG_FILE" exit 1 fi -# Verify we are in OpenWRT source root -if [ ! -d "package" ] || [ ! -f "Makefile" ]; then - echo "❌ Error: Please run this script in the OpenWRT source root directory." +if [ ! -d package ] || [ ! -f Makefile ] || [ ! -f feeds.conf.default ]; then + err "Must run inside the OpenWrt root directory." + err "From the OpenWrt root, run it as: ./helper/$(basename "$0")" exit 1 fi -# Ensure .config exists -if [ ! -f ".config" ]; then - echo "⚙️ Generating initial .config..." - make defconfig +if [ ! -f .config ]; then + info "No .config yet; running make defconfig first..." + make defconfig >/dev/null fi PACKAGE_INDEX="tmp/.config-package.in" PACKAGE_INFO="tmp/.packageinfo" - -# Ensure package metadata exists for reliable package name lookups. -if [ ! -f "$PACKAGE_INDEX" ] && [ ! -f "$PACKAGE_INFO" ]; then - echo "⚙️ Generating package metadata..." +if [ ! -f "$PACKAGE_INDEX" ] || [ ! -f "$PACKAGE_INFO" ]; then + info "Generating package metadata..." make defconfig >/dev/null fi -is_already_builtin() { - local pkg="$1" - - awk -v package_y="CONFIG_PACKAGE_${pkg}=y" \ - -v default_y="CONFIG_DEFAULT_${pkg}=y" \ - '$0 == package_y || $0 == default_y { found=1; exit } - END { exit !found }' .config +# --- Parse the list ------------------------------------------------------ +# Section headers are handled BEFORE stripping comments: the old +# `sub(/#[^#].*/,"")` left a bare '#' behind, which was then reported as a +# missing package named '#'. +parse_list() { + awk ' + BEGIN { mode = "y" } + { + line = $0 + if (line ~ /^[ \t]*##/) { + if (line ~ /^[ \t]*##built-in/) { mode = "y"; next } + if (line ~ /^[ \t]*##module/) { mode = "m"; next } + if (line ~ /^[ \t]*##remove/) { mode = "n"; next } + print "[WARN] unknown section header, ignored: " line > "/dev/stderr" + next + } + sub(/#.*$/, "", line) + n = split(line, t, /[ \t]+/) + for (i = 1; i <= n; i++) { + tok = t[i]; m = mode + if (tok == "") continue + if (substr(tok, 1, 1) == "-") { tok = substr(tok, 2); m = "n" } + if (tok == "") continue + print tok "\t" m + } + } + ' } -is_module() { - local pkg="$1" +if [ "$PKG_FILE" = "-" ]; then + ENTRIES="$(parse_list)" + PKG_LABEL="(stdin)" +else + ENTRIES="$(parse_list < "$PKG_FILE")" + PKG_LABEL="$PKG_FILE" +fi - grep -qx "CONFIG_PACKAGE_${pkg}=m" .config -} +if [ -z "$ENTRIES" ]; then + err "No packages found in $PKG_LABEL." + exit 1 +fi +# --- Read the current state once ----------------------------------------- +declare -A CUR=() +declare -A DEF=() +while IFS= read -r line; do + case "$line" in + CONFIG_PACKAGE_*=*) CUR["${line%%=*}"]="${line#*=}" ;; + "# CONFIG_PACKAGE_"*" is not set") + s="${line#\# }" + CUR["${s% is not set}"]="n" + ;; + CONFIG_DEFAULT_*=y) DEF["${line%%=*}"]="y" ;; + esac +done < .config + +# --- Verify and classify -------------------------------------------------- package_exists() { local pkg="$1" - if [ -f "$PACKAGE_INDEX" ] && - awk -v key="PACKAGE_${pkg}" '$1 == "config" && $2 == key { found=1; exit } + awk -v key="PACKAGE_$pkg" '$1 == "config" && $2 == key { found = 1; exit } END { exit !found }' "$PACKAGE_INDEX"; then return 0 fi - if [ -f "$PACKAGE_INFO" ] && grep -Fqx "Package: $pkg" "$PACKAGE_INFO"; then return 0 fi - return 1 } -echo "📦 Reading packages from $PKG_FILE ..." +DESIRED="$(mktemp "${TMPDIR:-/tmp}/desired.XXXXXX")" +NEWCFG="$(mktemp ./.config.new.XXXXXX)" +trap 'rm -f "$DESIRED" "$NEWCFG"' EXIT -# Parse the file into "package:mode" pairs, respecting ##built-in / ##module sections. -# Default mode is "y" (built-in). Inline comments (#xxx but not ##) are stripped. -ENTRIES=$(awk ' - BEGIN { mode = "y" } - /^##built-in/ { mode = "y"; next } - /^##module/ { mode = "m"; next } - { - sub(/#[^#].*/, "") # strip inline comments (single #), preserve ## headers - for (i = 1; i <= NF; i++) print $i ":" mode - } -' "$PKG_FILE") +declare -a MISSING=() SKIPPED=() PLANNED=() INVALID=() -ADDED=() -UPGRADED=() -MISSING=() +while IFS=$'\t' read -r pkg mode; do + [ -z "$pkg" ] && continue -for entry in $ENTRIES; do - pkg="${entry%:*}" - mode="${entry##*:}" - CONFIG_NAME="CONFIG_PACKAGE_${pkg}" - - if [ "$mode" = "y" ]; then - # --- built-in mode --- - if is_already_builtin "$pkg"; then - echo "✅ Already built-in: $pkg" - continue - fi - - if is_module "$pkg"; then - sed -i "s/^CONFIG_PACKAGE_${pkg}=m$/CONFIG_PACKAGE_${pkg}=y/" .config - UPGRADED+=("$pkg") - echo "⬆️ Module → built-in: $pkg" - continue - fi - else - # --- module mode --- - if is_module "$pkg"; then - echo "✅ Already module: $pkg" - continue - fi - - if is_already_builtin "$pkg"; then - echo "✅ Already built-in (keeping): $pkg" - continue - fi + if ! [[ "$pkg" =~ ^[A-Za-z0-9._+-]+$ ]]; then + INVALID+=("$pkg") + continue fi - # Check if package exists in OpenWrt package metadata. - if ! package_exists "$pkg"; then - echo "⚠️ Package not found: $pkg" + sym="CONFIG_PACKAGE_$pkg" + cur="${CUR[$sym]:-}" + + # Already provided by the target defaults: nothing to write, no churn. + if [ "$mode" = "y" ] && [ -n "${DEF[CONFIG_DEFAULT_$pkg]:-}" ] && [ "$cur" != "m" ]; then + SKIPPED+=("$pkg (target default)") + continue + fi + + if [ "$cur" = "$mode" ]; then + SKIPPED+=("$pkg (already =$mode)") + continue + fi + + # A removal of something that was never enabled needs no check. + if [ "$mode" != "n" ] && ! package_exists "$pkg"; then MISSING+=("$pkg") continue fi - # Append package config - echo "${CONFIG_NAME}=${mode}" >> .config - ADDED+=("$pkg (=${mode})") - echo "➕ Added: $pkg (=${mode})" -done + printf '%s\t%s\n' "$sym" "$mode" >> "$DESIRED" + PLANNED+=("$pkg: ${cur:-unset} -> $mode") +done <<< "$ENTRIES" -# Run defconfig to sync dependencies -echo -echo "⚙️ Running make defconfig..." -make defconfig >/dev/null - -echo -echo "✅ Done!" -echo "📋 Packages newly added: ${#ADDED[@]}" -[ ${#ADDED[@]} -gt 0 ] && echo " → ${ADDED[*]}" -echo "📋 Packages upgraded to built-in: ${#UPGRADED[@]}" -[ ${#UPGRADED[@]} -gt 0 ] && echo " → ${UPGRADED[*]}" -if [ ${#MISSING[@]} -gt 0 ]; then - echo "⚠️ Packages not found (check feeds or spelling):" - for m in "${MISSING[@]}"; do - echo " - $m" - done +if [ ${#INVALID[@]} -gt 0 ]; then + warn "Ignored ${#INVALID[@]} entries with invalid package names:" + printf ' %s\n' "${INVALID[@]}" +fi + +if [ ! -s "$DESIRED" ]; then + info "Nothing to change; .config already matches $PKG_LABEL." + if [ ${#MISSING[@]} -gt 0 ]; then + warn "Packages not found (check feeds or spelling):" + printf ' %s\n' "${MISSING[@]}" + warn "Try: ./scripts/feeds update -a && ./scripts/feeds install -a" + [ "$STRICT" = true ] && exit 1 + fi + exit 0 +fi + +# --- Merge into .config --------------------------------------------------- +# Existing symbols are rewritten where they already are, so the file neither +# grows nor reorders; duplicates left behind by older versions collapse. +awk -F'\t' ' + FNR == NR { want[$1] = $2; order[++nk] = $1; next } + { + key = "" + if ($0 ~ /^CONFIG_PACKAGE_[^=]+=/) { key = $0; sub(/=.*$/, "", key) } + else if ($0 ~ /^# CONFIG_PACKAGE_[^ ]+ is not set$/) { + # Extract by string, not by field: FS is a tab here, so $2 is empty + # on .config lines and the symbol would never be matched. + key = $0; sub(/^# /, "", key); sub(/ is not set$/, "", key) + } + if (key != "" && (key in want)) { + if (!(key in done)) { + done[key] = 1 + if (want[key] == "n") print "# " key " is not set" + else print key "=" want[key] + } + next + } + print + } + END { + for (i = 1; i <= nk; i++) { + k = order[i] + if (k in done) continue + if (want[k] == "n") print "# " k " is not set" + else print k "=" want[k] + } + } +' "$DESIRED" .config > "$NEWCFG" + +echo +echo "========================================" +echo " Planned .config changes" +echo "========================================" +printf ' %s\n' "${PLANNED[@]}" +echo " ---" +echo " change: ${#PLANNED[@]} unchanged: ${#SKIPPED[@]} not found: ${#MISSING[@]}" +echo "========================================" + +if [ ${#MISSING[@]} -gt 0 ]; then + warn "Packages not found (check feeds or spelling):" + printf ' %s\n' "${MISSING[@]}" + warn "Try: ./scripts/feeds update -a && ./scripts/feeds install -a" +fi + +if [ "$DRY_RUN" = true ]; then + echo + diff -u .config "$NEWCFG" || true + echo + info "Dry run complete; .config was not modified." + [ "$STRICT" = true ] && [ ${#MISSING[@]} -gt 0 ] && exit 1 + exit 0 +fi + +echo +if ! confirm "Apply these changes to .config?" yes; then + info "Aborted by user." + exit 0 +fi + +[ "$KEEP_BACKUP" = true ] && cp -a .config .config.bak +mv -f "$NEWCFG" .config +chmod 644 .config +trap 'rm -f "$DESIRED"' EXIT + +if [ "$RUN_DEFCONFIG" = true ]; then + info "Running make defconfig..." + make defconfig >/dev/null + + # defconfig silently reverts a removal whose package something else selects, + # and can demote =y to =m. Without this check that stays invisible. + declare -a REVERTED=() + declare -A FINAL=() + while IFS= read -r line; do + case "$line" in + CONFIG_PACKAGE_*=*) FINAL["${line%%=*}"]="${line#*=}" ;; + "# CONFIG_PACKAGE_"*" is not set") + s="${line#\# }" + FINAL["${s% is not set}"]="n" + ;; + esac + done < .config + + while IFS=$'\t' read -r sym mode; do + got="${FINAL[$sym]:-unset}" + [ "$got" = "$mode" ] || REVERTED+=("${sym#CONFIG_PACKAGE_}: wanted =$mode, got =$got") + done < "$DESIRED" + + if [ ${#REVERTED[@]} -gt 0 ]; then + warn "make defconfig changed ${#REVERTED[@]} of the requested states:" + printf ' %s\n' "${REVERTED[@]}" + warn "Usually a dependency pulled the package back in." + fi +fi + +echo +echo "========================================" +echo "OK Applied ${#PLANNED[@]} change(s) from $PKG_LABEL" +[ "$KEEP_BACKUP" = true ] && echo " Backup: .config.bak" +echo "========================================" + +if [ "$STRICT" = true ] && [ ${#MISSING[@]} -gt 0 ]; then + exit 1 fi diff --git a/add-openwrt-packages_old.sh b/add-openwrt-packages_old.sh deleted file mode 100755 index 8e59167..0000000 --- a/add-openwrt-packages_old.sh +++ /dev/null @@ -1,134 +0,0 @@ -#!/bin/bash -# Add specified OpenWRT packages (from a text file) to .config without removing existing ones -# Author: Zhe Yuan -# Usage: -# ./add-openwrt-packages.sh packages.txt -# Description: -# - Reads package names from a text file (supports both newline or space separation) -# - Adds missing CONFIG_PACKAGE_xxx=y entries to .config -# - Keeps existing settings intact -# - Reports missing packages -# - Runs make defconfig at the end to sync dependencies - -set -e -umask 022 - -# Check argument -if [ "$#" -ne 1 ]; then - echo "❌ Usage: $0 " - exit 1 -fi - -PKG_FILE="$1" - -# Verify file exists -if [ ! -f "$PKG_FILE" ]; then - echo "❌ Error: File not found: $PKG_FILE" - exit 1 -fi - -# Verify we are in OpenWRT source root -if [ ! -d "package" ] || [ ! -f "Makefile" ]; then - echo "❌ Error: Please run this script in the OpenWRT source root directory." - exit 1 -fi - -# Ensure .config exists -if [ ! -f ".config" ]; then - echo "⚙️ Generating initial .config..." - make defconfig -fi - -PACKAGE_INDEX="tmp/.config-package.in" -PACKAGE_INFO="tmp/.packageinfo" - -# Ensure package metadata exists for reliable package name lookups. -if [ ! -f "$PACKAGE_INDEX" ] && [ ! -f "$PACKAGE_INFO" ]; then - echo "⚙️ Generating package metadata..." - make defconfig >/dev/null -fi - -is_already_builtin() { - local pkg="$1" - - awk -v package_y="CONFIG_PACKAGE_${pkg}=y" \ - -v default_y="CONFIG_DEFAULT_${pkg}=y" \ - '$0 == package_y || $0 == default_y { found=1; exit } - END { exit !found }' .config -} - -is_module() { - local pkg="$1" - - grep -qx "CONFIG_PACKAGE_${pkg}=m" .config -} - -package_exists() { - local pkg="$1" - - if [ -f "$PACKAGE_INDEX" ] && - awk -v key="PACKAGE_${pkg}" '$1 == "config" && $2 == key { found=1; exit } - END { exit !found }' "$PACKAGE_INDEX"; then - return 0 - fi - - if [ -f "$PACKAGE_INFO" ] && grep -Fqx "Package: $pkg" "$PACKAGE_INFO"; then - return 0 - fi - - return 1 -} - -echo "📦 Reading packages from $PKG_FILE ..." - -# Read package tokens from file, while allowing inline comments. -PACKAGES=$(awk '{ sub(/#.*/, ""); for (i = 1; i <= NF; i++) print $i }' "$PKG_FILE") - -ADDED=() -MISSING=() - -for pkg in $PACKAGES; do - CONFIG_NAME="CONFIG_PACKAGE_${pkg}" - - # Skip if already built-in - if is_already_builtin "$pkg"; then - echo "✅ Already built-in: $pkg" - continue - fi - - # Upgrade module to built-in - if is_module "$pkg"; then - sed -i "s/^CONFIG_PACKAGE_${pkg}=m$/CONFIG_PACKAGE_${pkg}=y/" .config - ADDED+=("$pkg") - echo "⬆️ Module → built-in: $pkg" - continue - fi - - # Check if package exists in OpenWrt package metadata. - if ! package_exists "$pkg"; then - echo "⚠️ Package not found: $pkg" - MISSING+=("$pkg") - continue - fi - - # Append package config - echo "${CONFIG_NAME}=y" >> .config - ADDED+=("$pkg") - echo "➕ Added: $pkg" -done - -# Run defconfig to sync dependencies -echo -echo "⚙️ Running make defconfig..." -make defconfig >/dev/null - -echo -echo "✅ Done!" -echo "📋 Packages newly added: ${#ADDED[@]}" -[ ${#ADDED[@]} -gt 0 ] && echo " → ${ADDED[*]}" -if [ ${#MISSING[@]} -gt 0 ]; then - echo "⚠️ Packages not found (check feeds or spelling):" - for m in "${MISSING[@]}"; do - echo " - $m" - done -fi diff --git a/download-config.sh b/download-config.sh index c68378b..793b28c 100755 --- a/download-config.sh +++ b/download-config.sh @@ -1,96 +1,276 @@ #!/bin/bash -# Download OpenWRT config.buildinfo, generate .config and run defconfig +# Fetch the official config.buildinfo for a device and install it as .config. +# +# The device is looked up in the official index, so any of the ~1900 supported +# devices can be selected by name; nothing is hardcoded. +# +# Two things are changed in the downloaded config by default, because it is +# the buildbot's configuration rather than a personal one: +# - only the selected device is built, instead of every model in the target +# - the buildbot flags (ALL_KMODS, ALL_NONSHARED, SDK, IB, MAKE_TOOLCHAIN, +# COLLECT_KERNEL_DEBUG, AUTOREMOVE, ...) are turned off +# Neither changes the contents of the firmware image; they only stop the build +# from also producing every optional module, the SDK and the ImageBuilder. +# +# Usage: ./download-config.sh [options] +# +# --device profile id, e.g. linksys_mx8500 +# --target disambiguate an id present in several targets +# --version override version detection (e.g. 25.12.5) +# --snapshot shorthand for --version SNAPSHOT +# --with-packages also refresh the device's package list file +# --all-profiles keep every device in the target (upstream default) +# --keep-buildbot-flags keep ALL_KMODS / SDK / IB / ... as upstream ships +# --save-buildinfo keep a copy in config_/_.buildinfo +# --offline use a previously saved buildinfo, no network +# --output write here instead of ./.config (for testing) +# --refresh ignore the cached JSON indexes +# --skip-defconfig do not run make defconfig afterwards +# -y, --yes do not ask for confirmation +# -n, --dry-run show the plan, change nothing +# -h, --help show this help +# +# Run it from the OpenWrt root: ./helper/download-config.sh +# # Author: Zhe Yuan -set -e +set -euo pipefail umask 022 -# --- Check if inside OpenWRT source directory --- -if [ ! -f "feeds.conf.default" ] || [ ! -d "package" ]; then - echo "[ERROR] Must run inside OpenWRT root directory." +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" +# shellcheck source=lib-openwrt-upstream.sh +source "$SCRIPT_DIR/lib-openwrt-upstream.sh" + +DEVICE_ID="" +DEVICE_TARGET="" +VERSION_OVERRIDE="" +WITH_PACKAGES=false +SINGLE_PROFILE=true +STRIP_BUILDBOT=true +SAVE_BUILDINFO=false +OFFLINE=false +OUTPUT="" +RUN_DEFCONFIG=true + +usage() { sed -n '2,37p' "$0" | sed 's/^# \{0,1\}//;s/^#$//'; } + +need_value() { + if [ "$2" -lt 2 ] || [ -z "${3:-}" ]; then + err "$1 requires a value." exit 1 -fi + fi +} -# --- Detect Git version --- -VERSION=$(git describe --tags --abbrev=0 2>/dev/null || echo "") -VERSION="${VERSION#v}" -IS_LATEST=false - -if [ -z "$VERSION" ]; then - BRANCH=$(git rev-parse --abbrev-ref HEAD) - if [[ "$BRANCH" =~ openwrt-([0-9]+\.[0-9]+) ]]; then - PARTIAL_VERSION="${BASH_REMATCH[1]}" - # Find latest patch tag - LATEST_TAG=$(git tag -l "v${PARTIAL_VERSION}.*" | sort -V | tail -n1) - if [ -z "$LATEST_TAG" ]; then - echo "[ERROR] Cannot find latest tag for $PARTIAL_VERSION.*" - exit 1 - fi - VERSION="${LATEST_TAG#v}" - IS_LATEST=true - elif [[ "$BRANCH" == *SNAPSHOT* ]]; then - VERSION="SNAPSHOT" - else - read -rp "Cannot detect version, input manually (e.g., 24.10.4 or SNAPSHOT): " VERSION - fi -fi - -echo "[INFO] OpenWRT version: $VERSION" - -# --- Devices list --- -# Format: Name Target FirmwareDevice -DEVICES=( -"Linksys_MX8500 qualcommax/ipq807x linksys_mx8500" -"Cudy_TR3000_128MB mediatek/filogic cudy_tr3000-v1" -"Cudy_TR3000_256MB mediatek/filogic cudy_tr3000-256mb-v1" -) - -echo "Available devices:" -for i in "${!DEVICES[@]}"; do - NAME=$(echo "${DEVICES[$i]}" | awk '{print $1}') - echo " $i) $NAME" +while [ $# -gt 0 ]; do + case "$1" in + --device) need_value "$1" $# "${2:-}"; DEVICE_ID="$2"; shift ;; + --target) need_value "$1" $# "${2:-}"; DEVICE_TARGET="$2"; shift ;; + --version) need_value "$1" $# "${2:-}"; VERSION_OVERRIDE="$2"; shift ;; + --snapshot) VERSION_OVERRIDE="SNAPSHOT" ;; + --with-packages) WITH_PACKAGES=true ;; + --all-profiles) SINGLE_PROFILE=false ;; + --keep-buildbot-flags) STRIP_BUILDBOT=false ;; + --save-buildinfo) SAVE_BUILDINFO=true ;; + --offline) OFFLINE=true ;; + --output) need_value "$1" $# "${2:-}"; OUTPUT="$2"; shift ;; + --refresh) REFRESH=true ;; + --skip-defconfig) RUN_DEFCONFIG=false ;; + -y|--yes) ASSUME_YES=true ;; + -n|--dry-run) DRY_RUN=true ;; + -h|--help) usage; exit 0 ;; + *) err "Unknown option: $1"; echo "Run '$0 --help' for usage." >&2; exit 1 ;; + esac + shift done -read -rp "Select device number: " DEV_INDEX -if [[ -z "${DEVICES[$DEV_INDEX]}" ]]; then - echo "[ERROR] Invalid selection." +[ -t 0 ] || ASSUME_YES=true + +require_python3 + +if [ -n "$OUTPUT" ]; then + DEST="$OUTPUT" + RUN_DEFCONFIG=false +else + require_openwrt_root + DEST=".config" +fi +DEST_DIR="$(dirname -- "$DEST")" + +VER="$(detect_version)" + +if [ "$OFFLINE" != true ] && ! version_exists "$VER"; then + err "Version '$VER' is not published at $BASE_URL." + err " $(url_overview "$VER")" + exit 1 +fi + +select_device "$VER" + +SAVED_BUILDINFO="$SCRIPT_DIR/config_$VER/${DEV_ID}_${VER}.buildinfo" + +SYSUPGRADE_URL="" +FACTORY_URL="" +PKG_COUNT="?" +if [ "$OFFLINE" != true ]; then + PROFILES_JSON="$(fetch_profiles "$VER" "$DEV_TARGET")" + PKG_COUNT="$(profile_packages "$PROFILES_JSON" "$DEV_ID" | grep -c . || true)" + while IFS= read -r img; do + case "$img" in + *squashfs-sysupgrade*) SYSUPGRADE_URL="$(url_image "$VER" "$DEV_TARGET" "$img")" ;; + *squashfs-factory*) FACTORY_URL="$(url_image "$VER" "$DEV_TARGET" "$img")" ;; + esac + done < <(profile_images "$PROFILES_JSON" "$DEV_ID") +fi + +CONFIG_URL="$(url_buildinfo "$VER" "$DEV_TARGET")" + +echo +echo "========================================" +echo " Planned actions" +echo "========================================" +echo " Device : $DEV_ID ($DEV_LABEL)" +echo " Target : $DEV_TARGET" +echo " Version : $VER" +if [ "$OFFLINE" = true ]; then + echo " Source : $SAVED_BUILDINFO (offline)" +else + echo " Source : $CONFIG_URL" +fi +echo " Destination : $DEST$([ -f "$DEST" ] && echo ' (backed up first)')" +echo " Device scope : $([ "$SINGLE_PROFILE" = true ] && echo 'this device only' || echo 'all devices in target')" +echo " Buildbot flags: $([ "$STRIP_BUILDBOT" = true ] && echo 'stripped' || echo 'kept')" +echo " Built-in pkgs : $PKG_COUNT (per profiles.json)" +[ -n "$SYSUPGRADE_URL" ] && echo " Sysupgrade : $SYSUPGRADE_URL" +echo " Post : $([ "$RUN_DEFCONFIG" = true ] && echo 'make defconfig' || echo 'skipped')" +echo "========================================" + +if [ "$DRY_RUN" != true ]; then + echo + if ! confirm "Proceed?" yes; then + info "Aborted by user." + exit 0 + fi +fi + +# --- Download into a temp file next to the destination ------------------- +# Same directory keeps the final mv atomic; `curl -f` plus this staging is +# what stops a 404 from truncating an existing .config. +TMP="$(mktemp "$DEST_DIR/.config.dl.XXXXXX")" +trap 'rm -f "$TMP" "$TMP.pp"' EXIT + +if [ "$OFFLINE" = true ]; then + if [ ! -f "$SAVED_BUILDINFO" ]; then + err "No saved buildinfo at $SAVED_BUILDINFO" + err "Run once without --offline (add --save-buildinfo) to create it." exit 1 -fi - -# --- Extract device info --- -read -r DEV_NAME DEV_TARGET DEV_FW <<<"${DEVICES[$DEV_INDEX]}" -echo "[INFO] Selected device: $DEV_NAME" - -# --- Construct URLs --- -if [[ "$VERSION" == "SNAPSHOT" ]]; then - CONFIG_URL="https://downloads.openwrt.org/snapshots/targets/$DEV_TARGET/config.buildinfo" - FIRMWARE_URL="https://downloads.openwrt.org/snapshots/targets/$DEV_TARGET/openwrt-SNAPSHOT-$DEV_TARGET-$DEV_FW-squashfs-sysupgrade.bin" + fi + info "Using the saved buildinfo (offline)." + cp "$SAVED_BUILDINFO" "$TMP" else - CONFIG_URL="https://downloads.openwrt.org/releases/$VERSION/targets/$DEV_TARGET/config.buildinfo" - FIRMWARE_URL="https://downloads.openwrt.org/releases/$VERSION/targets/$DEV_TARGET/openwrt-$VERSION-$DEV_TARGET-$DEV_FW-squashfs-sysupgrade.bin" + info "Downloading $CONFIG_URL" + if ! fetch_url "$CONFIG_URL" "$TMP"; then + err "Download failed; $DEST was not touched." + exit 1 + fi fi -echo "[INFO] Config URL: $CONFIG_URL" -echo "[INFO] Firmware URL: $FIRMWARE_URL" +# --- Validate before letting it near .config ------------------------------ +if [ ! -s "$TMP" ] || [ "$(wc -l < "$TMP")" -lt 5 ]; then + err "Downloaded config looks empty or truncated; $DEST was not touched." + exit 1 +fi +if ! grep -qx "CONFIG_TARGET_${DEV_TSYM}=y" "$TMP"; then + err "Downloaded config is not for target $DEV_TARGET; $DEST was not touched." + exit 1 +fi +if ! grep -q "_DEVICE_${DEV_ID}=y" "$TMP"; then + err "Downloaded config does not contain device $DEV_ID; $DEST was not touched." + exit 1 +fi -# --- Download config.buildinfo --- -echo "[INFO] Downloading config.buildinfo..." -wget -q --show-progress -O ".config" "$CONFIG_URL" -echo "[INFO] Saved as .config" +# --- Post-process --------------------------------------------------------- +# One awk pass: a chain of `sed -i` would rewrite the file repeatedly and the +# symbol names contain '-' and '+', which are awkward in sed replacements. +awk -v tsym="$DEV_TSYM" -v prof="$DEV_ID" \ + -v single="$SINGLE_PROFILE" -v strip="$STRIP_BUILDBOT" ' + # Matches both CONFIG_TARGET_DEVICE__DEVICE_=y and + # CONFIG_TARGET_DEVICE_PACKAGES__DEVICE_="". It cannot match the + # single-profile symbol appended below, which has no DEVICE_ prefix. + single == "true" && /^CONFIG_TARGET_DEVICE_/ { next } + single == "true" && /^CONFIG_TARGET_MULTI_PROFILE=y$/ { print "# CONFIG_TARGET_MULTI_PROFILE is not set"; next } + single == "true" && /^CONFIG_TARGET_ALL_PROFILES=y$/ { print "# CONFIG_TARGET_ALL_PROFILES is not set"; next } + single == "true" && /^CONFIG_TARGET_PER_DEVICE_ROOTFS=y$/ { print "# CONFIG_TARGET_PER_DEVICE_ROOTFS is not set"; next } + strip == "true" && $0 ~ /^CONFIG_(ALL_KMODS|ALL_NONSHARED|BUILDBOT|SDK|SDK_LLVM_BPF|IB|MAKE_TOOLCHAIN|COLLECT_KERNEL_DEBUG|JSON_CYCLONEDX_SBOM|AUTOREMOVE)=y$/ { + sym = $0; sub(/=y$/, "", sym); print "# " sym " is not set"; next + } + { print } + END { + # CONFIG_TARGET_PROFILE is deliberately not written: Kconfig derives it. + if (single == "true") print "CONFIG_TARGET_" tsym "_DEVICE_" prof "=y" + } +' "$TMP" > "$TMP.pp" -# --- Run make defconfig --- -echo "[INFO] Running make defconfig..." -make defconfig >/dev/null -echo "[INFO] defconfig done." +if [ "$DRY_RUN" = true ]; then + echo + info "Resulting config (first differences against the download):" + diff -u "$TMP" "$TMP.pp" | head -40 || true + echo + info "Dry run complete; $DEST was not touched." + exit 0 +fi -# --- Check firmware existence --- -if curl --head --silent --fail "$FIRMWARE_URL" >/dev/null; then - echo "[INFO] Firmware exists: $FIRMWARE_URL" +if [ "$SAVE_BUILDINFO" = true ]; then + mkdir -p "$(dirname "$SAVED_BUILDINFO")" + cp "$TMP" "$SAVED_BUILDINFO" + info "Saved the raw buildinfo to $SAVED_BUILDINFO" +fi + +if [ -f "$DEST" ]; then + cp -a "$DEST" "$DEST.download.bak" + info "Previous config kept at $DEST.download.bak" +fi +mv -f "$TMP.pp" "$DEST" +chmod 644 "$DEST" +info "Wrote $DEST" + +if [ "$RUN_DEFCONFIG" = true ]; then + info "Running make defconfig..." + make defconfig >/dev/null + + # A config whose symbols do not exist in this tree is silently dropped by + # Kconfig, leaving a config that looks fine and builds the wrong thing. + if [ "$SINGLE_PROFILE" = true ]; then + if ! grep -qx "CONFIG_TARGET_${DEV_TSYM}_DEVICE_${DEV_ID}=y" .config || + ! grep -qx "CONFIG_TARGET_PROFILE=\"DEVICE_${DEV_ID}\"" .config; then + err "The device symbol did not survive make defconfig." + err "Your tree ($(git describe --tags 2>/dev/null || echo unknown)) probably" + err "does not match release $VER. Restore with: cp $DEST.download.bak $DEST" + exit 1 + fi + fi +fi + +echo +echo "========================================" +echo "OK Configuration installed." +echo " Device: $DEV_ID ($DEV_LABEL)" +echo " Target: $DEV_TARGET" +echo " Version: $VER" +echo "========================================" +[ -n "$SYSUPGRADE_URL" ] && echo " Official sysupgrade: $SYSUPGRADE_URL" +[ -n "$FACTORY_URL" ] && echo " Official factory: $FACTORY_URL" +echo " Firmware Selector: $(selector_url "$VER" "$DEV_TARGET" "$DEV_ID")" +echo + +if [ "$WITH_PACKAGES" = true ]; then + GEN_ARGS=(--device "$DEV_ID" --target "$DEV_TARGET" --version "$VER") + [ "$ASSUME_YES" = true ] && GEN_ARGS+=(-y) + "$SCRIPT_DIR/gen-package-list.sh" "${GEN_ARGS[@]}" else - echo "[WARN] Firmware not found or URL invalid." + echo "Next steps:" + echo " ./helper/gen-package-list.sh --device $DEV_ID --version $VER" + echo " ./helper/add-openwrt-packages.sh helper/config_$VER/packages_${DEV_ID//-/_}_$VER.txt" + echo " make menuconfig" + echo " make -j\$(nproc) download world" + echo fi - -# --- Print Firmware Selector URL --- -TARGET_ESC=$(echo $DEV_TARGET | sed 's|/|%2F|g') -FS_URL="https://firmware-selector.openwrt.org/?version=$VERSION&target=$TARGET_ESC&id=$DEV_FW" -echo "[INFO] Check Installed Packages at: $FS_URL" diff --git a/gen-package-list.sh b/gen-package-list.sh new file mode 100755 index 0000000..a5d7d37 --- /dev/null +++ b/gen-package-list.sh @@ -0,0 +1,256 @@ +#!/bin/bash +# Generate the built-in package list for a device from the official index. +# +# The list is default_packages + device_packages from the target's +# profiles.json, plus the packages the Firmware Selector adds on top +# (luci, luci-app-attendedsysupgrade). This is exactly what the Selector +# shows, so it replaces copying that list by hand for every new release. +# +# By default the list is written into config_/packages__.txt, +# replacing only the block between the generated-list sentinels inside the +# ##built-in section. Everything else in the file -- your own extra packages, +# the ##module section, blank lines -- is preserved verbatim. +# +# Usage: ./gen-package-list.sh [options] +# +# --device profile id, e.g. linksys_mx8500 +# --target disambiguate an id present in several targets +# --version override version detection (e.g. 25.12.5) +# --snapshot shorthand for --version SNAPSHOT +# --extras "" override the Firmware Selector extras +# --no-extras emit only default_packages + device_packages +# --wrap wrap the generated list (default 0 = one line) +# --file write to this file instead of the default path +# --stdout print the list, one package per line, and exit +# --apply apply the list to .config instead of writing a file +# --no-migrate do not replace an existing unmarked built-in line +# --refresh ignore the cached JSON indexes +# -y, --yes do not ask for confirmation +# -n, --dry-run show what would change, write nothing +# -h, --help show this help +# +# Author: Zhe Yuan + +set -euo pipefail +umask 022 + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" +# shellcheck source=lib-openwrt-upstream.sh +source "$SCRIPT_DIR/lib-openwrt-upstream.sh" + +DEVICE_ID="" +DEVICE_TARGET="" +VERSION_OVERRIDE="" +EXTRAS="$SELECTOR_EXTRAS" +WRAP=0 +OUT_FILE="" +MODE="file" # file | stdout | apply +MIGRATE=true + +BEGIN_MARK="# >>> generated built-in list - do not edit by hand" +END_MARK="# <<< generated built-in list" + +usage() { sed -n '2,35p' "$0" | sed 's/^# \{0,1\}//;s/^#$//'; } + +need_value() { + if [ "$2" -lt 2 ] || [ -z "${3:-}" ]; then + err "$1 requires a value." + exit 1 + fi +} + +while [ $# -gt 0 ]; do + case "$1" in + --device) need_value "$1" $# "${2:-}"; DEVICE_ID="$2"; shift ;; + --target) need_value "$1" $# "${2:-}"; DEVICE_TARGET="$2"; shift ;; + --version) need_value "$1" $# "${2:-}"; VERSION_OVERRIDE="$2"; shift ;; + --snapshot) VERSION_OVERRIDE="SNAPSHOT" ;; + --extras) need_value "$1" $# "${2:-}"; EXTRAS="$2"; shift ;; + --no-extras) EXTRAS="" ;; + --wrap) need_value "$1" $# "${2:-}"; WRAP="$2"; shift ;; + --file) need_value "$1" $# "${2:-}"; OUT_FILE="$2"; shift ;; + --stdout) MODE="stdout" ;; + --apply) MODE="apply" ;; + --no-migrate) MIGRATE=false ;; + --refresh) REFRESH=true ;; + -y|--yes) ASSUME_YES=true ;; + -n|--dry-run) DRY_RUN=true ;; + -h|--help) usage; exit 0 ;; + *) err "Unknown option: $1"; echo "Run '$0 --help' for usage." >&2; exit 1 ;; + esac + shift +done + +[ -t 0 ] || ASSUME_YES=true + +if ! [[ "$WRAP" =~ ^[0-9]+$ ]]; then + err "--wrap needs a non-negative number." + exit 1 +fi + +require_python3 + +VER="$(detect_version)" +select_device "$VER" + +PROFILES_JSON="$(fetch_profiles "$VER" "$DEV_TARGET")" + +PKGS="$(profile_packages "$PROFILES_JSON" "$DEV_ID")" + +# Extras go last and never duplicate something the target already provides, +# which is what keeps the output identical to the hand-written lists. +for extra in $EXTRAS; do + if ! printf '%s\n' "$PKGS" | grep -qxF "$extra"; then + PKGS="$PKGS"$'\n'"$extra" + fi +done + +COUNT="$(printf '%s\n' "$PKGS" | grep -c . || true)" + +if [ "$MODE" = "stdout" ]; then + printf '%s\n' "$PKGS" + exit 0 +fi + +# --- Render the package block ------------------------------------------- +render_block() { + echo "$BEGIN_MARK" + echo "# device: $DEV_ID target: $DEV_TARGET version: $VER" + echo "# source: profiles.json default_packages + device_packages + selector extras" + if [ "$WRAP" -gt 0 ]; then + printf '%s\n' "$PKGS" | tr '\n' ' ' | fold -s -w "$WRAP" | sed 's/[[:space:]]*$//' + else + printf '%s\n' "$PKGS" | tr '\n' ' ' | sed 's/[[:space:]]*$//' + echo + fi + echo "$END_MARK" +} + +if [ "$MODE" = "apply" ]; then + info "Applying $COUNT built-in packages to .config via add-openwrt-packages.sh" + APPLY_ARGS=() + [ "$ASSUME_YES" = true ] && APPLY_ARGS+=(-y) + [ "$DRY_RUN" = true ] && APPLY_ARGS+=(-n) + { echo "##built-in"; printf '%s\n' "$PKGS"; } | + "$SCRIPT_DIR/add-openwrt-packages.sh" "${APPLY_ARGS[@]}" - + exit $? +fi + +# --- File mode ----------------------------------------------------------- +if [ -z "$OUT_FILE" ]; then + OUT_FILE="$SCRIPT_DIR/config_$VER/packages_${DEV_ID//-/_}_$VER.txt" +fi +OUT_DIR="$(dirname -- "$OUT_FILE")" + +echo +echo "========================================" +echo " Planned actions" +echo "========================================" +echo " Device : $DEV_ID ($DEV_LABEL)" +echo " Target : $DEV_TARGET" +echo " Version : $VER" +echo " Packages : $COUNT built-in ($(printf '%s\n' "$PKGS" | grep -c . || true) total)" +echo " Extras : ${EXTRAS:-(none)}" +echo " File : $OUT_FILE $([ -f "$OUT_FILE" ] && echo '(update)' || echo '(create)')" +echo "========================================" + +NEW_FILE="$(mktemp "${TMPDIR:-/tmp}/pkglist.XXXXXX")" +trap 'rm -f "$NEW_FILE"' EXIT + +if [ -f "$OUT_FILE" ]; then + # Replace only the generated block; preserve everything the user wrote. + render_block > "$NEW_FILE.block" + awk -v blockfile="$NEW_FILE.block" -v migrate="$MIGRATE" ' + function emit_block( line) { + while ((getline line < blockfile) > 0) print line + close(blockfile) + emitted = 1 + } + BEGIN { inb = 0; skip = 0; emitted = 0; replaced = 0 } + /^[ \t]*##/ { + if (inb && !emitted) { + # Section ended without a marked block: insert before leaving. + emit_block() + } + inb = ($0 ~ /^[ \t]*##built-in/) + skip = 0 + print + next + } + inb && index($0, "# >>> generated built-in list") == 1 { skip = 1; emit_block(); next } + inb && index($0, "# <<< generated built-in list") == 1 { skip = 0; next } + skip { next } + { + # Migration: the pre-sentinel auto-generated line is the one carrying + # both base-files and libc, which are in default_packages for every + # target and never appear in a hand-written extras line. + if (inb && !emitted && migrate == "true" && !replaced) { + has_bf = 0; has_libc = 0 + for (i = 1; i <= NF; i++) { + if ($i == "base-files") has_bf = 1 + if ($i == "libc") has_libc = 1 + } + if (has_bf && has_libc) { emit_block(); replaced = 1; next } + } + print + } + END { + if (inb && !emitted) emit_block() + if (!emitted) { + print "##built-in" + print "" + emit_block() + } + } + ' "$OUT_FILE" > "$NEW_FILE" + rm -f "$NEW_FILE.block" +else + { + echo "##built-in" + echo + render_block + echo + echo "##module" + } > "$NEW_FILE" +fi + +if [ -f "$OUT_FILE" ] && cmp -s "$OUT_FILE" "$NEW_FILE"; then + info "No change: $OUT_FILE is already up to date." + exit 0 +fi + +if [ "$DRY_RUN" = true ]; then + echo + info "Changes that would be written:" + diff -u "$OUT_FILE" "$NEW_FILE" 2>/dev/null || true + echo + info "Dry run complete; nothing was written." + exit 0 +fi + +echo +if ! confirm "Write $OUT_FILE?" yes; then + info "Aborted by user." + exit 0 +fi + +mkdir -p "$OUT_DIR" +if [ -f "$OUT_FILE" ]; then + cp -a "$OUT_FILE" "$OUT_FILE.bak" + info "Previous version kept at $(basename "$OUT_FILE").bak" +fi +cp "$NEW_FILE" "$OUT_FILE" +chmod 644 "$OUT_FILE" + +echo +echo "========================================" +echo "OK Package list written." +echo " File: $OUT_FILE" +echo " Packages: $COUNT built-in" +echo "========================================" +echo +echo "Next steps:" +echo " Edit $(basename "$OUT_FILE") to add your own packages below the generated block" +echo " cd $(dirname "$SCRIPT_DIR")" +echo " ./helper/add-openwrt-packages.sh helper/${OUT_FILE#"$SCRIPT_DIR"/}" +echo diff --git a/lib-openwrt-upstream.sh b/lib-openwrt-upstream.sh new file mode 100644 index 0000000..17a7080 --- /dev/null +++ b/lib-openwrt-upstream.sh @@ -0,0 +1,431 @@ +#!/bin/bash +# Shared helpers for talking to downloads.openwrt.org. +# +# Sourced by download-config.sh and gen-package-list.sh. Not executable on its +# own. Callers are expected to set `set -euo pipefail`; this file does not. +# +# Author: Zhe Yuan + +[ -n "${_OWRT_LIB_LOADED:-}" ] && return 0 +_OWRT_LIB_LOADED=1 + +BASE_URL="${BASE_URL:-https://downloads.openwrt.org}" +CACHE_DIR="${CACHE_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/openwrt-build-helper}" +CACHE_TTL="${CACHE_TTL:-86400}" + +# What the Firmware Selector adds on top of the target's own package set. +# This is a UI convention, not an API, so keep it overridable. +SELECTOR_EXTRAS="${SELECTOR_EXTRAS:-luci luci-app-attendedsysupgrade}" + +# Devices offered when an interactive search is left empty. Only a convenience +# shortcut -- the target still comes from .overview.json, never hardcoded. +FAVOURITE_DEVICES="${FAVOURITE_DEVICES:-linksys_mx8500 cudy_tr3000-v1 cudy_tr3000-256mb-v1}" + +REFRESH="${REFRESH:-false}" +DRY_RUN="${DRY_RUN:-false}" +ASSUME_YES="${ASSUME_YES:-false}" + +# --- Output helpers ------------------------------------------------------- +# All logging goes to stderr so that functions returning a value on stdout +# (detect_version, profile_packages, ...) can be captured safely. +info() { echo "[INFO] $*" >&2; } +warn() { echo "[WARN] $*" >&2; } +err() { echo "[ERROR] $*" >&2; } + +# Wrap every mutating command so --dry-run prints instead of running. +run() { + if [ "$DRY_RUN" = true ]; then + printf '[DRY-RUN]' + printf ' %q' "$@" + echo + return 0 + fi + "$@" +} + +confirm() { + # confirm "" + local question="$1" default="$2" reply="" + + [ "$ASSUME_YES" = true ] && return 0 + + if [ "$default" = "yes" ]; then + read -r -p "$question [Y/n]: " reply || reply="" + case "${reply:-}" in + [nN] | [nN][oO]) return 1 ;; + *) return 0 ;; + esac + else + read -r -p "$question [y/N]: " reply || reply="" + case "${reply:-}" in + [yY] | [yY][eE][sS]) return 0 ;; + *) return 1 ;; + esac + fi +} + +require_python3() { + if ! command -v python3 >/dev/null 2>&1; then + err "python3 is required to parse the upstream JSON indexes." + exit 1 + fi +} + +require_openwrt_root() { + if [ ! -f feeds.conf.default ] || [ ! -f Makefile ] || [ ! -d scripts ]; then + err "Must run inside the OpenWrt root directory." + err "From the OpenWrt root, run it as: ./helper/$(basename "$0")" + exit 1 + fi +} + +# --- URL construction ----------------------------------------------------- +# Every upstream URL is built from this one function. +release_path() { + local ver="$1" + if [ "$ver" = "SNAPSHOT" ]; then + echo "snapshots" + else + echo "releases/$ver" + fi +} + +url_overview() { echo "$BASE_URL/$(release_path "$1")/.overview.json"; } +url_profiles() { echo "$BASE_URL/$(release_path "$1")/targets/$2/profiles.json"; } +url_buildinfo() { echo "$BASE_URL/$(release_path "$1")/targets/$2/config.buildinfo"; } +url_image() { echo "$BASE_URL/$(release_path "$1")/targets/$2/$3"; } + +selector_url() { + local ver="$1" target="$2" id="$3" + echo "https://firmware-selector.openwrt.org/?version=$ver&target=${target//\//%2F}&id=$id" +} + +# --- Fetching ------------------------------------------------------------- +# `curl -f` is the key flag: a 404 exits non-zero and never writes a body, so +# a failed download can never truncate the destination. +fetch_url() { + local url="$1" dest="$2" + + if ! curl -fsSL --retry 3 --retry-delay 2 --max-time 60 -o "$dest.part" "$url"; then + rm -f "$dest.part" + return 1 + fi + mv -f "$dest.part" "$dest" +} + +fetch_cached() { + local url="$1" cache="$2" age=0 + + mkdir -p "$(dirname "$cache")" + + if [ "$REFRESH" != true ] && [ -s "$cache" ]; then + age=$(( $(date +%s) - $(stat -c %Y "$cache" 2>/dev/null || echo 0) )) + if [ "$age" -lt "$CACHE_TTL" ]; then + return 0 + fi + fi + + if fetch_url "$url" "$cache"; then + return 0 + fi + + # A stale cache beats no data when the network is down. + if [ -s "$cache" ]; then + warn "Could not refresh $url; using the cached copy." + return 0 + fi + return 1 +} + +version_exists() { + curl -fsI --max-time 30 "$(url_overview "$1")" >/dev/null 2>&1 +} + +# --- Version detection ---------------------------------------------------- +# Replaces `git describe --tags --abbrev=0`, which returns a tag whenever ANY +# tag is reachable -- so the old script's whole fallback chain was dead code +# and a branch tip past a tag silently produced the wrong release URL. +detect_version() { + local ver="" branch="" tag="" drift=0 + + if [ -n "${VERSION_OVERRIDE:-}" ]; then + ver="${VERSION_OVERRIDE#v}" + info "Version from the command line: $ver" + echo "$ver" + return 0 + fi + + if ! git rev-parse --is-inside-work-tree >/dev/null 2>&1; then + err "Not inside a Git repository and no --version given." + exit 1 + fi + + if tag="$(git describe --tags --exact-match HEAD 2>/dev/null)"; then + ver="${tag#v}" + info "HEAD is exactly at tag $tag; using version $ver." + echo "$ver" + return 0 + fi + + if branch="$(git symbolic-ref --quiet --short HEAD 2>/dev/null)"; then + case "$branch" in + main | master) + info "On branch '$branch'; using SNAPSHOT." + echo "SNAPSHOT" + return 0 + ;; + openwrt-* | [0-9]*) + local series="${branch#openwrt-}" + # --merged HEAD keeps tags from unrelated branches out of the picture. + tag="$(git tag --merged HEAD -l "v${series}.*" 2>/dev/null | + grep -v -- '-rc' | sort -V | tail -n 1)" + if [ -n "$tag" ]; then + drift="$(git rev-list --count "$tag..HEAD" 2>/dev/null || echo 0)" + if [ "$drift" -gt 0 ]; then + warn "HEAD is $drift commit(s) past $tag." + warn "The downloaded config describes the release, not your tree." + fi + ver="${tag#v}" + info "Branch '$branch' -> newest merged tag $tag; using version $ver." + echo "$ver" + return 0 + fi + ;; + esac + fi + + if tag="$(git describe --tags --long 2>/dev/null)"; then + tag="${tag%-*-g*}" + drift="$(git rev-list --count "$tag..HEAD" 2>/dev/null || echo 0)" + warn "HEAD is not at a tag; nearest is $tag ($drift commit(s) behind HEAD)." + if [ "$ASSUME_YES" != true ] && + ! confirm "Use version ${tag#v}?" no; then + err "Aborted; pass --version explicitly." + exit 1 + fi + ver="${tag#v}" + info "Using version $ver." + echo "$ver" + return 0 + fi + + if [ "$ASSUME_YES" = true ] || [ ! -t 0 ]; then + err "Cannot determine the OpenWrt version; pass --version or --snapshot." + exit 1 + fi + + read -r -p "Enter version (e.g. 25.12.5 or SNAPSHOT): " ver || ver="" + if [ -z "$ver" ]; then + err "No version given." + exit 1 + fi + echo "${ver#v}" +} + +# --- Device index --------------------------------------------------------- +fetch_overview() { + local ver="$1" + # Separate statement on purpose: `local a=$1 b=$a` expands every word before + # assigning, so $a would be unbound under `set -u`. + local cache="$CACHE_DIR/overview-$ver.json" + + # Snapshots move daily, so never serve them from a day-old cache. + [ "$ver" = "SNAPSHOT" ] && REFRESH=true + + if ! fetch_cached "$(url_overview "$ver")" "$cache"; then + err "Cannot fetch the device index for version '$ver'." + err " $(url_overview "$ver")" + exit 1 + fi + echo "$cache" +} + +# Emits TSV: id target label +_overview_py() { + python3 - "$@" <<'PY' +import json, sys, difflib + +path, mode, arg = sys.argv[1], sys.argv[2], sys.argv[3] +data = json.load(open(path, encoding="utf-8")) +profiles = data.get("profiles") or [] + + +def label(p): + parts = [] + for t in p.get("titles") or []: + if not isinstance(t, dict): + continue + # Upstream entries come in three shapes: {title}, {vendor,model}, + # and {vendor,model,variant}. + if t.get("title"): + parts.append(t["title"]) + else: + parts.append(" ".join( + str(x) for x in (t.get("vendor"), t.get("model"), t.get("variant")) if x + )) + parts = [p for p in parts if p] + return " / ".join(parts) or p.get("id", "") + + +def emit(rows): + for p in rows: + print("%s\t%s\t%s" % (p.get("id", ""), p.get("target", ""), label(p))) + + +if mode == "id": + rows = [p for p in profiles if p.get("id") == arg] + if not rows: + names = [p.get("id", "") for p in profiles] + close = difflib.get_close_matches(arg, names, n=5, cutoff=0.5) + sys.stderr.write("no device with id '%s'\n" % arg) + if close: + sys.stderr.write("closest matches: %s\n" % ", ".join(close)) + sys.exit(2) + emit(rows) +elif mode == "search": + terms = [t.lower() for t in arg.split() if t] + rows = [] + for p in profiles: + hay = (p.get("id", "") + " " + label(p)).lower() + if all(t in hay for t in terms): + rows.append(p) + rows.sort(key=lambda p: p.get("id", "")) + emit(rows) +elif mode == "list": + wanted = arg.split() + by_id = {p.get("id"): p for p in profiles} + emit([by_id[i] for i in wanted if i in by_id]) +PY +} + +# Sets DEV_ID, DEV_TARGET, DEV_TSYM, DEV_LABEL +select_device() { + local ver="$1" json="" rows="" n=0 choice="" tries=0 query="" + + json="$(fetch_overview "$ver")" + + if [ -n "${DEVICE_ID:-}" ]; then + if ! rows="$(_overview_py "$json" id "$DEVICE_ID" 2>&1)"; then + err "$rows" + exit 1 + fi + if [ -n "${DEVICE_TARGET:-}" ]; then + rows="$(printf '%s\n' "$rows" | awk -F'\t' -v t="$DEVICE_TARGET" '$2 == t')" + fi + n="$(printf '%s\n' "$rows" | grep -c . || true)" + if [ "$n" -eq 0 ]; then + err "Device '$DEVICE_ID' exists but not for target '${DEVICE_TARGET:-}'." + exit 1 + fi + if [ "$n" -gt 1 ]; then + err "Device id '$DEVICE_ID' is present in more than one target:" + printf '%s\n' "$rows" | awk -F'\t' '{print " " $2 " (" $3 ")"}' >&2 + err "Disambiguate with --target ." + exit 1 + fi + else + if [ "$ASSUME_YES" = true ] || [ ! -t 0 ]; then + err "No device selected. Pass --device for non-interactive use." + exit 1 + fi + + while [ "$tries" -lt 3 ]; do + tries=$((tries + 1)) + read -r -p "Search device (vendor, model or id; empty for favourites): " query || query="" + + if [ -z "${query:-}" ]; then + rows="$(_overview_py "$json" list "$FAVOURITE_DEVICES")" + else + rows="$(_overview_py "$json" search "$query")" + fi + + n="$(printf '%s\n' "$rows" | grep -c . || true)" + if [ "$n" -eq 0 ]; then + warn "No device matched '$query'." + continue + fi + if [ "$n" -gt 25 ]; then + warn "$n devices matched; please narrow the search." + continue + fi + + echo + printf '%s\n' "$rows" | awk -F'\t' '{printf " %2d) %-32s %-22s %s\n", NR, $1, $2, $3}' + echo + read -r -p "Select device number [1-$n]: " choice || choice="" + + # 1-based plus an explicit numeric test: an unset/non-numeric index in + # an array subscript is an arithmetic context that silently yields 0. + if [[ "${choice:-}" =~ ^[0-9]+$ ]] && [ "$choice" -ge 1 ] && [ "$choice" -le "$n" ]; then + rows="$(printf '%s\n' "$rows" | sed -n "${choice}p")" + break + fi + warn "Invalid selection." + rows="" + done + + if [ -z "$rows" ]; then + err "No device selected." + exit 1 + fi + fi + + IFS=$'\t' read -r DEV_ID DEV_TARGET DEV_LABEL <<<"$rows" + DEV_TSYM="${DEV_TARGET//\//_}" + + if [ -z "$DEV_ID" ] || [ -z "$DEV_TARGET" ]; then + err "Could not resolve the device selection." + exit 1 + fi +} + +# --- profiles.json -------------------------------------------------------- +fetch_profiles() { + local ver="$1" target="$2" + local cache="$CACHE_DIR/profiles-$ver-${target//\//_}.json" + + [ "$ver" = "SNAPSHOT" ] && REFRESH=true + + if ! fetch_cached "$(url_profiles "$ver" "$target")" "$cache"; then + err "Cannot fetch profiles.json for $target at version $ver." + err " $(url_profiles "$ver" "$target")" + exit 1 + fi + echo "$cache" +} + +# default_packages + device_packages, order preserved, deduplicated. +profile_packages() { + python3 - "$1" "$2" <<'PY' +import json, sys + +data = json.load(open(sys.argv[1], encoding="utf-8")) +pid = sys.argv[2] +prof = (data.get("profiles") or {}).get(pid) +if prof is None: + sys.stderr.write("profile '%s' is not in this target's profiles.json\n" % pid) + sys.exit(2) + +seen, out = set(), [] +for p in list(data.get("default_packages") or []) + list(prof.get("device_packages") or []): + if p not in seen: + seen.add(p) + out.append(p) +print("\n".join(out)) +PY +} + +# Image filenames straight from profiles.json -- no string building, which is +# what made the old firmware URL wrong (the target still contained a '/'). +profile_images() { + python3 - "$1" "$2" <<'PY' +import json, sys + +data = json.load(open(sys.argv[1], encoding="utf-8")) +prof = (data.get("profiles") or {}).get(sys.argv[2]) or {} +for img in prof.get("images") or []: + name = img.get("name") + if name: + print(name) +PY +}