Files
nolimit-discounts/ci/site-test.sh
kristianwindandClaude Opus 5 2b289c6763 1.6.2: WP 7.1.2 and WC 11.1.2 declared, and the caller counts the assertions
Kristian asked the family to check WordPress 7.1.2. Run pinned rather than
against "latest", because latest drifts and then nobody knows which version a
green belonged to:

  PHP 7.4   69 assertions, 0 failures   "none from this plugin"
  PHP 8.3   69 assertions, 0 failures   "none from this plugin"
  PHP 8.4   69 assertions, 0 failures   "none from this plugin"

The container reported WordPress 7.1.2 and WooCommerce 11.1.2 on all three,
read out of the run rather than from the variable it was given. So there is no
problem with 7.1.2 here.

The findings were in the reporting, not the code.

The headers were stale: "Tested up to: 7.1" and "WC tested up to: 11.1.0"
against current 7.1.2 and 11.1.2. The WooCommerce one is the visible half — a
shop on 11.1.2 sees an untested warning on the WooCommerce status page, and on
this plugin that banner sits in front of somebody deciding whether to update a
discount engine.

And the suite said "All assertions passed" without a number, which is the exact
failure every other check here was fixed for. It now prints the count.

🔴 Then the floor for that count found a bigger hole than it was written for.
Putting it at the end of the smoke file does nothing: wp eval-file runs the
file in function scope, so a `return` partway through skips everything after
it, including the floor. Measured — truncating tests/smoke.php before the
shipping section ran two thirds of the assertions and ci/site-test.sh reported
GREEN.

A file cannot assert about its own early exit. ci/site-test.sh now captures the
output and checks the total against MIN_SITE_ASSERTIONS from outside, where an
early return cannot reach, and fails when the summary line is missing
altogether. The in-file floor stays for a short run that does reach the end.

Proven with the same fixture that exposed it: the truncated suite now exits 1
saying it printed no assertion total.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-25 08:51:46 +02:00

237 lines
11 KiB
Bash
Executable File

#!/usr/bin/env bash
# Stand up a throwaway WordPress + WooCommerce, install this plugin into it,
# and run the site tests from ci/plugin.conf inside it.
#
# WP_VERSION=latest|nightly|6.8 WC_VERSION=latest|beta|11.1.0 PHP_VERSION=8.3
#
# The point of the version knobs is the question this whole directory exists to
# answer: *will the shop still work after I press update?* Running the same
# tests against `latest` and against `nightly`/`beta` answers it before the
# update happens rather than after.
#
# Written for a Gitea Actions job, which runs INSIDE a container that talks to
# the HOST's docker daemon. A bind mount of $PWD there mounts an empty
# directory nobody created, and the symptom is "Could not open input file",
# which reads like a typo in the path. So the plugin goes in over `docker cp`
# and all state lives in named volumes. That also makes the script behave
# identically on GitHub's runners and on a laptop.
set -euo pipefail
cd "$(dirname "$0")/.."
. ci/lib.sh
ci_load_conf
WP_VERSION="${WP_VERSION:-latest}"
WC_VERSION="${WC_VERSION:-latest}"
PHP_VERSION="${PHP_VERSION:-8.3}"
# 🔴 NOT just $$. A Gitea job runs in a fresh container where the shell gets a
# low PID, and those REPEAT across runs — so "wpci-$$" collides with the
# leftovers of any earlier run that did not clean up, and the job dies on
# `network with name wpci-147-net already exists`. Measured on Gitea run 999:
# every run landing on PID 147 was wedged permanently, while the same script on
# a laptop never collides because PIDs there are large and varied.
#
# The epoch and $RANDOM make the name unique per run even inside a container
# that always starts at the same PID.
id="wpci-$$-$(date +%s)-${RANDOM:-0}"
net="$id-net"
# The name is unique to this run, so anything already using it is debris from a
# crashed run and is ours to clear. Without this a single wedged name stays
# wedged forever, which is exactly what happened on the Gitea runner.
docker rm -f -v "$id-db" "$id-cli" >/dev/null 2>&1 || true
docker network rm "$net" >/dev/null 2>&1 || true
# 🔴 -v, and it is the whole bug. `mariadb:lts` declares VOLUME
# /var/lib/mysql, so every run creates an ANONYMOUS volume, and `docker rm -f`
# does not remove anonymous volumes — only `docker rm -f -v` does. The named
# $id-wp below is removed correctly, which is exactly what made this function
# look complete: removing three of the four things you created is
# indistinguishable from removing all four unless you count.
#
# Measured 2026-09-17 after a sibling plugin traced it: the Gitea runner's disk
# was 100% full, 26 GB in 95 volumes with 1 in use, and a push there failed
# with `unable to create temporary object directory` — three layers from the
# cause. This machine had 127 dangling volumes and 15.2 GB reclaimable.
cleanup() {
# Ask the containers which volumes they own BEFORE removing them. A global
# before/after count would be wrong here: several sessions share this box and
# run these container pairs concurrently, so a neighbour's volume would read
# as this run's leak.
local owned=""
owned=$(docker inspect -f \
'{{range .Mounts}}{{if eq .Type "volume"}}{{.Name}} {{end}}{{end}}' \
"$id-db" "$id-cli" 2>/dev/null || true)
docker rm -f -v "$id-db" "$id-cli" >/dev/null 2>&1 || true
docker volume rm -f "$id-wp" >/dev/null 2>&1 || true
docker network rm "$net" >/dev/null 2>&1 || true
# Count rather than trust the flag. This is advisory on a failed run — a test
# that died halfway has more urgent news — but on a successful run a leftover
# volume is a real regression and says so.
local leaked=""
for v in $owned; do
docker volume inspect "$v" >/dev/null 2>&1 && leaked="$leaked $v"
done
if [ -n "$leaked" ]; then
echo "::warning::This run left docker volume(s) behind:$leaked — 95 of these filled the Gitea runner's disk."
# `return` from an EXIT trap does NOT change the script's exit status;
# only `exit` does. Without this the warning would print and the build
# would still go green, which is the failure mode of every other check
# this kit has had to fix tonight.
if [ "${site_test_ok:-0}" = 1 ]; then
ci_fail "The site test leaked the volume(s) above. Left alone this fills the runner's disk, and the symptom is a push failing with 'unable to create temporary object directory'."
exit 1
fi
fi
}
trap cleanup EXIT
echo "WordPress=$WP_VERSION WooCommerce=$WC_VERSION PHP=$PHP_VERSION plugin=$PLUGIN_SLUG"
ci_step "database"
docker network create "$net" >/dev/null
docker volume create "$id-wp" >/dev/null
docker run -d --name "$id-db" --network "$net" \
-e MARIADB_ROOT_PASSWORD=root -e MARIADB_DATABASE=wordpress \
-e MARIADB_USER=wordpress -e MARIADB_PASSWORD=wordpress mariadb:lts >/dev/null
# wordpress:cli runs as www-data (33); a fresh named volume is root-owned, so
# ownership is fixed from a root container before anything tries to write.
docker run --rm -v "$id-wp:/var/www/html" alpine chown 33:33 /var/www/html
docker run -d --name "$id-cli" --network "$net" -v "$id-wp:/var/www/html" \
-e WORDPRESS_DB_HOST="$id-db" "wordpress:cli-php$PHP_VERSION" sleep infinity >/dev/null
# The image ships no php.ini, so PHP's 128M default applies and `wp core
# download` dies unpacking the zip. WP_CLI_PHP_ARGS does not help here: the
# phar is executed directly, not through the wp launcher.
docker exec -u root "$id-cli" sh -c 'echo memory_limit=512M > /usr/local/etc/php/conf.d/zz-memory.ini'
wp() { docker exec "$id-cli" wp "$@"; }
for i in $(seq 1 60); do
docker exec "$id-db" mariadb -uwordpress -pwordpress -e 'select 1' wordpress >/dev/null 2>&1 && break
if [ "$i" = 60 ]; then echo "MariaDB never came up." >&2; exit 1; fi
sleep 2
done
ci_step "WordPress $WP_VERSION"
if [ "$WP_VERSION" = latest ]; then
wp core download --quiet
else
wp core download --version="$WP_VERSION" --quiet
fi
wp config create --dbname=wordpress --dbuser=wordpress --dbpass=wordpress \
--dbhost="$id-db" --quiet
# Deprecations are the way a WordPress or PHP upgrade actually breaks a plugin:
# nothing fails, a notice appears, and six months later it is a fatal. They are
# logged to a file (not the page) and checked at the end of this script.
wp config set WP_DEBUG true --raw --quiet
wp config set WP_DEBUG_LOG true --raw --quiet
wp config set WP_DEBUG_DISPLAY false --raw --quiet
wp core install --url=http://plugin.test --title="$PLUGIN_SLUG CI" \
--admin_user=admin --admin_password=admin --admin_email=ci@example.com \
--skip-email --quiet
echo " installed $(wp core version)"
if [ "$NEEDS_WOOCOMMERCE" = yes ]; then
ci_step "WooCommerce $WC_VERSION"
case "$WC_VERSION" in
latest)
wp plugin install woocommerce --activate --quiet
;;
beta)
# Resolved inside the container, which has PHP and network; the host
# runner is not guaranteed either.
target=$(docker exec -i -e SLUG=woocommerce "$id-cli" php < ci/latest-prerelease.php)
if [ -n "$target" ]; then
echo " prerelease available: $target"
wp plugin install woocommerce --version="$target" --activate --quiet
else
echo " no prerelease newer than stable right now — using the stable release"
wp plugin install woocommerce --activate --quiet
fi
;;
*)
wp plugin install woocommerce --version="$WC_VERSION" --activate --quiet
;;
esac
# HPOS, because that is what a shop set up in the last few years runs, and a
# plugin that touches orders behaves differently under it.
wp option update woocommerce_custom_orders_table_enabled yes >/dev/null
echo " installed $(wp plugin get woocommerce --field=version)"
fi
ci_step "$PLUGIN_SLUG"
# Exactly the files the release zip will contain, plus the site tests. Testing
# the working tree instead would test a plugin no shop ever receives: one
# unanchored .gitignore line is enough for a file to be present locally,
# absent from the release, and never missed until a customer installs it.
{ ./ci/dist-files.sh; printf '%s\n' ${SITE_TESTS:-}; } | tr '\n' '\0' | tar --null -T - -cf - \
| docker exec -i "$id-cli" sh -c \
"mkdir -p /var/www/html/wp-content/plugins/$PLUGIN_SLUG \
&& tar -xf - -C /var/www/html/wp-content/plugins/$PLUGIN_SLUG"
wp plugin activate "$PLUGIN_SLUG" --quiet
wp plugin list --name="$PLUGIN_SLUG" --field=status | grep -qx active \
|| { echo "The plugin did not stay active after activation." >&2; exit 1; }
echo " activated $(wp plugin get "$PLUGIN_SLUG" --field=version)"
# 🔴 The output is captured and the summary checked HERE, by the caller.
#
# A smoke file cannot assert about its own early exit: `wp eval-file` runs it
# in function scope, so a `return` partway through skips everything after it —
# including any floor the file carries at the bottom. Measured: truncating
# tests/smoke.php before the shipping section ran two thirds of the assertions
# and this script reported green, because the file's own floor was one of the
# things never reached.
#
# So the number the file prints is verified against MIN_SITE_ASSERTIONS out
# here, where an early return cannot reach. The floor inside the file stays as
# well; it catches a short run that still gets to the end.
for f in ${SITE_TESTS:-}; do
ci_step "$f"
out=$(wp eval-file "wp-content/plugins/$PLUGIN_SLUG/$f" 2>&1) || { printf '%s\n' "$out"; exit 1; }
printf '%s\n' "$out"
case "$f" in
*smoke*)
ran=$(printf '%s' "$out" | sed -n -E 's/^All ([0-9]+) assertions passed.*/\1/p' | tail -1)
if [ -z "$ran" ]; then
echo
ci_fail "$f printed no assertion total. It exited before its summary — which is what a file that returned halfway through looks like, and it cannot report that about itself."
exit 1
fi
if [ "$ran" -lt "$MIN_SITE_ASSERTIONS" ]; then
echo
ci_fail "$f ran only $ran assertions, expected at least $MIN_SITE_ASSERTIONS. Part of the suite did not run."
exit 1
fi
echo " $ran assertions, at or above the floor of $MIN_SITE_ASSERTIONS"
;;
esac
done
ci_step "PHP notices and deprecations"
# Only lines naming this plugin. WordPress core and WooCommerce deprecate their
# own things on every release, and failing on those would make the build red
# for something nobody here can fix — which is how a red build gets ignored.
log=$(wp eval 'echo WP_CONTENT_DIR;')/debug.log
if docker exec "$id-cli" test -f "$log"; then
if docker exec "$id-cli" grep -F "/plugins/$PLUGIN_SLUG/" "$log"; then
echo
ci_fail "The plugin logged the notices above on WordPress $(wp core version). They are the shape an update breaks in: harmless now, fatal two releases later."
exit 1
fi
echo " none from this plugin"
else
echo " nothing logged at all"
fi
# Set only here, so the leak check above is advisory on a run that failed for
# some other reason and fatal on one that otherwise passed.
site_test_ok=1
ci_step "done"
echo "WordPress $(wp core version)$( [ "$NEEDS_WOOCOMMERCE" = yes ] && echo " + WooCommerce $(wp plugin get woocommerce --field=version)" ) on PHP $PHP_VERSION: green"