yuanzhe1999/openwrt-build-helper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Generate the built-in package list from the official index

Browse Source 
The ##built-in section of each package list was copied by hand from the
Firmware Selector for every new release. It is now generated from the
target's profiles.json (default_packages + device_packages, plus the two
packages the Selector adds on top), which reproduces the hand-written list
byte for byte, order included.

New lib-openwrt-upstream.sh holds the shared upstream access: URL
construction, cached fetches, version detection, device search over
.overview.json and profiles.json parsing. It needs python3, not jq.

New gen-package-list.sh writes config_<ver>/packages_<dev>_<ver>.txt,
replacing only the generated block so hand-written extras and the ##module
section survive; --stdout and --apply skip the file entirely.

download-config.sh:
- Look the device up in the official index instead of a hardcoded table of
  three devices, so any supported device can be selected by name.
- Build the firmware URLs from profiles.json images[]. The old code pasted
  the target into the filename with its '/' intact, so the URL was always
  wrong and the check always warned.
- Stage the download in a temp file and validate it before touching
  .config, and keep a backup. A 404 used to truncate .config to nothing.
- Fix version detection: `git describe --tags --abbrev=0` returns a tag
  whenever any tag is reachable, so the entire fallback chain below it was
  unreachable and a branch tip silently produced the wrong release URL.
- Build only the selected device instead of all 35 in the target, and turn
  off the buildbot flags (ALL_KMODS, ALL_NONSHARED, SDK, IB,
  MAKE_TOOLCHAIN, COLLECT_KERNEL_DEBUG, AUTOREMOVE). Verified not to change
  the firmware: the set of packages built into the image is identical
  either way (193 packages), while the module packages to build drop from
  1248 to 0. --keep-buildbot-flags and --all-profiles restore the old
  behaviour.
- Assert the device symbol survived make defconfig, so a tree that does not
  match the release fails loudly instead of silently building the default.

add-openwrt-packages.sh:
- Rewrite existing CONFIG_PACKAGE_ lines in place. Repeated runs used to
  append duplicates and grow .config every time.
- Support removal via a -pkg prefix or a ##remove section.
- Fix the parser dropping a lone '#' or an unknown ##header into the
  package list, where it was reported as a missing package named '#'.
- Read .config once into a map instead of grepping it per package, and
  compare package names exactly rather than as regexes.
- After make defconfig, report any package whose final state differs from
  what was asked for, which is the only way to see a removal that a
  dependency pulled back in.

Delete add-openwrt-packages_old.sh; nothing referenced it and it was a
strict subset of the current script.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Zhe Yuan
2026-09-05 21:14:36 -04:00
parent 351e7c41f3
commit 0255d2f890
6 changed files with 1291 additions and 339 deletions
Download Patch File Download Diff File Expand all files Collapse all files

406
add-openwrt-packages.sh
View File

@@ -1,162 +1,340 @@
#!/bin/bash
# Add specified OpenWRT packages (from a text file) to .config without removing existing ones
# Apply a package list to the OpenWrt .config.
#
# The list is a plain text file of package names, whitespace separated, any
# number per line. Section headers switch how the following packages are
# applied, and a leading '-' on a name removes that package:
#
# ##built-in the packages after this are set to =y (default)
# ##module ... are set to =m
# ##remove ... are set to =n
# -luci-app-ttyd set to =n regardless of the current section
# pkg # comment text after a single # is ignored
#
# Existing CONFIG_PACKAGE_ lines are rewritten in place rather than appended,
# so running this repeatedly does not grow .config.
#
# Usage: ./add-openwrt-packages.sh [options] <package_list.txt | ->
#
# -y, --yes do not ask for confirmation
# -n, --dry-run show the resulting diff, change nothing
# --no-defconfig do not run make defconfig afterwards
# --no-backup do not keep a .config.bak
# --strict exit non-zero if any listed package was not found
# -h, --help show this help
#
# Run it from the OpenWrt root: ./helper/add-openwrt-packages.sh <list>
#
# Author: Zhe Yuan
# Usage:
# ./add-openwrt-packages.sh packages.txt
# Description:
# - Reads package names from a text file (supports both newline or space separation)
# - Supports ##built-in and ##module sections to control build mode (=y or =m)
# - Packages default to built-in (=y) if no section header is specified
# - Adds missing CONFIG_PACKAGE_xxx entries to .config
# - Keeps existing settings intact
# - Reports missing packages
# - Runs make defconfig at the end to sync dependencies
set -e
set -euo pipefail
umask 022
# Check argument
if [ "$#" -ne 1 ]; then
echo "❌ Usage: $0 <package_list.txt>"
PKG_FILE=""
ASSUME_YES=false
DRY_RUN=false
RUN_DEFCONFIG=true
KEEP_BACKUP=true
STRICT=false
info() { echo "[INFO] $*"; }
warn() { echo "[WARN] $*"; }
err() { echo "[ERROR] $*" >&2; }
usage() { sed -n '2,29p' "$0" | sed 's/^# \{0,1\}//;s/^#$//'; }
confirm() {
local question="$1" default="$2" reply=""
[ "$ASSUME_YES" = true ] && return 0
if [ "$default" = "yes" ]; then
read -r -p "$question [Y/n]: " reply || reply=""
case "${reply:-}" in [nN] | [nN][oO]) return 1 ;; *) return 0 ;; esac
else
read -r -p "$question [y/N]: " reply || reply=""
case "${reply:-}" in [yY] | [yY][eE][sS]) return 0 ;; *) return 1 ;; esac
fi
}
while [ $# -gt 0 ]; do
case "$1" in
-y|--yes) ASSUME_YES=true ;;
-n|--dry-run) DRY_RUN=true ;;
--no-defconfig) RUN_DEFCONFIG=false ;;
--no-backup) KEEP_BACKUP=false ;;
--strict) STRICT=true ;;
-h|--help) usage; exit 0 ;;
-) PKG_FILE="-" ;;
-*) err "Unknown option: $1"; echo "Run '$0 --help' for usage." >&2; exit 1 ;;
*)
if [ -n "$PKG_FILE" ]; then
err "Only one package list can be given."
exit 1
fi
PKG_FILE="$1"
;;
esac
shift
done
[ -t 0 ] || ASSUME_YES=true
if [ -z "$PKG_FILE" ]; then
err "No package list given."
usage >&2
exit 1
fi
PKG_FILE="$1"
# Verify file exists
if [ ! -f "$PKG_FILE" ]; then
echo "❌ Error: File not found: $PKG_FILE"
if [ "$PKG_FILE" != "-" ] && [ ! -f "$PKG_FILE" ]; then
err "Package list not found: $PKG_FILE"
exit 1
fi
# Verify we are in OpenWRT source root
if [ ! -d "package" ] || [ ! -f "Makefile" ]; then
echo "❌ Error: Please run this script in the OpenWRT source root directory."
if [ ! -d package ] || [ ! -f Makefile ] || [ ! -f feeds.conf.default ]; then
err "Must run inside the OpenWrt root directory."
err "From the OpenWrt root, run it as: ./helper/$(basename "$0")"
exit 1
fi
# Ensure .config exists
if [ ! -f ".config" ]; then
echo "⚙️ Generating initial .config..."
make defconfig
if [ ! -f .config ]; then
info "No .config yet; running make defconfig first..."
make defconfig >/dev/null
fi
PACKAGE_INDEX="tmp/.config-package.in"
PACKAGE_INFO="tmp/.packageinfo"
# Ensure package metadata exists for reliable package name lookups.
if [ ! -f "$PACKAGE_INDEX" ] && [ ! -f "$PACKAGE_INFO" ]; then
echo "⚙️ Generating package metadata..."
if [ ! -f "$PACKAGE_INDEX" ] || [ ! -f "$PACKAGE_INFO" ]; then
info "Generating package metadata..."
make defconfig >/dev/null
fi
is_already_builtin() {
local pkg="$1"
awk -v package_y="CONFIG_PACKAGE_${pkg}=y" \
-v default_y="CONFIG_DEFAULT_${pkg}=y" \
'$0 == package_y || $0 == default_y { found=1; exit }
END { exit !found }' .config
# --- Parse the list ------------------------------------------------------
# Section headers are handled BEFORE stripping comments: the old
# `sub(/#[^#].*/,"")` left a bare '#' behind, which was then reported as a
# missing package named '#'.
parse_list() {
awk '
BEGIN { mode = "y" }
{
line = $0
if (line ~ /^[ \t]*##/) {
if (line ~ /^[ \t]*##built-in/) { mode = "y"; next }
if (line ~ /^[ \t]*##module/) { mode = "m"; next }
if (line ~ /^[ \t]*##remove/) { mode = "n"; next }
print "[WARN] unknown section header, ignored: " line > "/dev/stderr"
next
}
sub(/#.*$/, "", line)
n = split(line, t, /[ \t]+/)
for (i = 1; i <= n; i++) {
tok = t[i]; m = mode
if (tok == "") continue
if (substr(tok, 1, 1) == "-") { tok = substr(tok, 2); m = "n" }
if (tok == "") continue
print tok "\t" m
}
}
'
}
is_module() {
local pkg="$1"
if [ "$PKG_FILE" = "-" ]; then
ENTRIES="$(parse_list)"
PKG_LABEL="(stdin)"
else
ENTRIES="$(parse_list < "$PKG_FILE")"
PKG_LABEL="$PKG_FILE"
fi
grep -qx "CONFIG_PACKAGE_${pkg}=m" .config
}
if [ -z "$ENTRIES" ]; then
err "No packages found in $PKG_LABEL."
exit 1
fi
# --- Read the current state once -----------------------------------------
declare -A CUR=()
declare -A DEF=()
while IFS= read -r line; do
case "$line" in
CONFIG_PACKAGE_*=*) CUR["${line%%=*}"]="${line#*=}" ;;
"# CONFIG_PACKAGE_"*" is not set")
s="${line#\# }"
CUR["${s% is not set}"]="n"
;;
CONFIG_DEFAULT_*=y) DEF["${line%%=*}"]="y" ;;
esac
done < .config
# --- Verify and classify --------------------------------------------------
package_exists() {
local pkg="$1"
if [ -f "$PACKAGE_INDEX" ] &&
awk -v key="PACKAGE_${pkg}" '$1 == "config" && $2 == key { found=1; exit }
awk -v key="PACKAGE_$pkg" '$1 == "config" && $2 == key { found = 1; exit }
END { exit !found }' "$PACKAGE_INDEX"; then
return 0
fi
if [ -f "$PACKAGE_INFO" ] && grep -Fqx "Package: $pkg" "$PACKAGE_INFO"; then
return 0
fi
return 1
}
echo "📦 Reading packages from $PKG_FILE ..."
DESIRED="$(mktemp "${TMPDIR:-/tmp}/desired.XXXXXX")"
NEWCFG="$(mktemp ./.config.new.XXXXXX)"
trap 'rm -f "$DESIRED" "$NEWCFG"' EXIT
# Parse the file into "package:mode" pairs, respecting ##built-in / ##module sections.
# Default mode is "y" (built-in). Inline comments (#xxx but not ##) are stripped.
ENTRIES=$(awk '
BEGIN { mode = "y" }
/^##built-in/ { mode = "y"; next }
/^##module/ { mode = "m"; next }
{
sub(/#[^#].*/, "") # strip inline comments (single #), preserve ## headers
for (i = 1; i <= NF; i++) print $i ":" mode
}
' "$PKG_FILE")
declare -a MISSING=() SKIPPED=() PLANNED=() INVALID=()
ADDED=()
UPGRADED=()
MISSING=()
while IFS=$'\t' read -r pkg mode; do
[ -z "$pkg" ] && continue
for entry in $ENTRIES; do
pkg="${entry%:*}"
mode="${entry##*:}"
CONFIG_NAME="CONFIG_PACKAGE_${pkg}"
if [ "$mode" = "y" ]; then
# --- built-in mode ---
if is_already_builtin "$pkg"; then
echo "✅ Already built-in: $pkg"
continue
fi
if is_module "$pkg"; then
sed -i "s/^CONFIG_PACKAGE_${pkg}=m$/CONFIG_PACKAGE_${pkg}=y/" .config
UPGRADED+=("$pkg")
echo "⬆️ Module → built-in: $pkg"
continue
fi
else
# --- module mode ---
if is_module "$pkg"; then
echo "✅ Already module: $pkg"
continue
fi
if is_already_builtin "$pkg"; then
echo "✅ Already built-in (keeping): $pkg"
continue
fi
if ! [[ "$pkg" =~ ^[A-Za-z0-9._+-]+$ ]]; then
INVALID+=("$pkg")
continue
fi
# Check if package exists in OpenWrt package metadata.
if ! package_exists "$pkg"; then
echo "⚠️ Package not found: $pkg"
sym="CONFIG_PACKAGE_$pkg"
cur="${CUR[$sym]:-}"
# Already provided by the target defaults: nothing to write, no churn.
if [ "$mode" = "y" ] && [ -n "${DEF[CONFIG_DEFAULT_$pkg]:-}" ] && [ "$cur" != "m" ]; then
SKIPPED+=("$pkg (target default)")
continue
fi
if [ "$cur" = "$mode" ]; then
SKIPPED+=("$pkg (already =$mode)")
continue
fi
# A removal of something that was never enabled needs no check.
if [ "$mode" != "n" ] && ! package_exists "$pkg"; then
MISSING+=("$pkg")
continue
fi
# Append package config
echo "${CONFIG_NAME}=${mode}" >> .config
ADDED+=("$pkg (=${mode})")
echo "➕ Added: $pkg (=${mode})"
done
printf '%s\t%s\n' "$sym" "$mode" >> "$DESIRED"
PLANNED+=("$pkg: ${cur:-unset} -> $mode")
done <<< "$ENTRIES"
# Run defconfig to sync dependencies
echo
echo "⚙️ Running make defconfig..."
make defconfig >/dev/null
echo
echo "✅ Done!"
echo "📋 Packages newly added: ${#ADDED[@]}"
[ ${#ADDED[@]} -gt 0 ] && echo " → ${ADDED[*]}"
echo "📋 Packages upgraded to built-in: ${#UPGRADED[@]}"
[ ${#UPGRADED[@]} -gt 0 ] && echo " → ${UPGRADED[*]}"
if [ ${#MISSING[@]} -gt 0 ]; then
echo "⚠️ Packages not found (check feeds or spelling):"
for m in "${MISSING[@]}"; do
echo " - $m"
done
if [ ${#INVALID[@]} -gt 0 ]; then
warn "Ignored ${#INVALID[@]} entries with invalid package names:"
printf ' %s\n' "${INVALID[@]}"
fi
if [ ! -s "$DESIRED" ]; then
info "Nothing to change; .config already matches $PKG_LABEL."
if [ ${#MISSING[@]} -gt 0 ]; then
warn "Packages not found (check feeds or spelling):"
printf ' %s\n' "${MISSING[@]}"
warn "Try: ./scripts/feeds update -a && ./scripts/feeds install -a"
[ "$STRICT" = true ] && exit 1
fi
exit 0
fi
# --- Merge into .config ---------------------------------------------------
# Existing symbols are rewritten where they already are, so the file neither
# grows nor reorders; duplicates left behind by older versions collapse.
awk -F'\t' '
FNR == NR { want[$1] = $2; order[++nk] = $1; next }
{
key = ""
if ($0 ~ /^CONFIG_PACKAGE_[^=]+=/) { key = $0; sub(/=.*$/, "", key) }
else if ($0 ~ /^# CONFIG_PACKAGE_[^ ]+ is not set$/) {
# Extract by string, not by field: FS is a tab here, so $2 is empty
# on .config lines and the symbol would never be matched.
key = $0; sub(/^# /, "", key); sub(/ is not set$/, "", key)
}
if (key != "" && (key in want)) {
if (!(key in done)) {
done[key] = 1
if (want[key] == "n") print "# " key " is not set"
else print key "=" want[key]
}
next
}
print
}
END {
for (i = 1; i <= nk; i++) {
k = order[i]
if (k in done) continue
if (want[k] == "n") print "# " k " is not set"
else print k "=" want[k]
}
}
' "$DESIRED" .config > "$NEWCFG"
echo
echo "========================================"
echo " Planned .config changes"
echo "========================================"
printf ' %s\n' "${PLANNED[@]}"
echo " ---"
echo " change: ${#PLANNED[@]} unchanged: ${#SKIPPED[@]} not found: ${#MISSING[@]}"
echo "========================================"
if [ ${#MISSING[@]} -gt 0 ]; then
warn "Packages not found (check feeds or spelling):"
printf ' %s\n' "${MISSING[@]}"
warn "Try: ./scripts/feeds update -a && ./scripts/feeds install -a"
fi
if [ "$DRY_RUN" = true ]; then
echo
diff -u .config "$NEWCFG" || true
echo
info "Dry run complete; .config was not modified."
[ "$STRICT" = true ] && [ ${#MISSING[@]} -gt 0 ] && exit 1
exit 0
fi
echo
if ! confirm "Apply these changes to .config?" yes; then
info "Aborted by user."
exit 0
fi
[ "$KEEP_BACKUP" = true ] && cp -a .config .config.bak
mv -f "$NEWCFG" .config
chmod 644 .config
trap 'rm -f "$DESIRED"' EXIT
if [ "$RUN_DEFCONFIG" = true ]; then
info "Running make defconfig..."
make defconfig >/dev/null
# defconfig silently reverts a removal whose package something else selects,
# and can demote =y to =m. Without this check that stays invisible.
declare -a REVERTED=()
declare -A FINAL=()
while IFS= read -r line; do
case "$line" in
CONFIG_PACKAGE_*=*) FINAL["${line%%=*}"]="${line#*=}" ;;
"# CONFIG_PACKAGE_"*" is not set")
s="${line#\# }"
FINAL["${s% is not set}"]="n"
;;
esac
done < .config
while IFS=$'\t' read -r sym mode; do
got="${FINAL[$sym]:-unset}"
[ "$got" = "$mode" ] || REVERTED+=("${sym#CONFIG_PACKAGE_}: wanted =$mode, got =$got")
done < "$DESIRED"
if [ ${#REVERTED[@]} -gt 0 ]; then
warn "make defconfig changed ${#REVERTED[@]} of the requested states:"
printf ' %s\n' "${REVERTED[@]}"
warn "Usually a dependency pulled the package back in."
fi
fi
echo
echo "========================================"
echo "OK Applied ${#PLANNED[@]} change(s) from $PKG_LABEL"
[ "$KEEP_BACKUP" = true ] && echo " Backup: .config.bak"
echo "========================================"
if [ "$STRICT" = true ] && [ ${#MISSING[@]} -gt 0 ]; then
exit 1
fi