hardening 2.2: 2meet-data-optimizer 1.2.0 (drift detection, scope, CLI and diagnostics) #6

Merged

wpdev merged 19 commits from hardening/2.2 into main 2026-09-27 10:56:49 +08:00

Conversation 0 Commits 19 Files Changed 52

+2862 -330

wpdev commented 2026-09-27 10:47:15 +08:00

Owner

Copy Link

Copy Source

Hardening 2.2 for 2meet-data-optimizer (1.2.0). Continues the search-index mode of ADR-004 (Release 2.2 design section added). Pairs with HivePress AddOn 0.3.0 and WooCommerce AddOn 0.2.0.

Summary

• Drift: wp tmdo drift-check <entity> [--group] [--sample] [--fix] (miss / diff / extra per key, rebuild from native, never in aeav_only); the post router keeps a group native while it is recorded as drifting; daily sample on the health-check event; full --fix 60 s after an entity leaves disabled or the Hook Bus is switched back on (tmdo_drift_resync). (PR-070〜074, 073b)
• Auto-promoter requires a clean drift check within 2 days; shadow_read diffs capped at 20 per key per hour. (PR-075)
• Scope: post/wc_product no longer registered (WooCommerce meta stays native). (PR-077)
• CLI: bridge-mode, member-backfill --all, backfill groups from the registry (json groups included), cleanup-hp-transients --options. (PR-067〜069, 080)
• Fixed: entity backfill invented default values; doctor / term-comment-diagnose / bridge-status false alarms; member-cutover fatal; doctor counting drift records of unregistered groups. (PR-066, 068, 076, 089a)
• Perf: sort-only searches drive from the flat table when every post has a row (rating sort ~2.1 s native → ~0.44 s). (PR-079)
• 2.1 audit notes (escaping, logger), live router tests compare sort keys on site-wide ties. (PR-078, 089b)

Acceptance on dev36 (100k listings): dev30/notes/tmdo-acceptance-2.2-2026-09-27.md. Backlog: dev30/notes/pr-backlog.md (PR-066〜089).

Test plan

• composer test 698, integration 572, phpcs 0, stan OK
• live suites 56/56 (drift, rest, resync, router, serialized, shadow_write, zero_side_effect)
• R-2.2 on dev36: registry without WC groups, drift 0, Hook Bus off → write → on → drift 0, doctor clean, V1–V7 (write overhead +6.3%)
• CI green on this PR
• After deploy: cleanup-hp-transients --options --confirm, bridge-restore-native post --group=wc_product --retire, first drift-check per entity (CHANGELOG 升級後)

Known, not addressed: keyword + filter searches are slower routed (COUNT cache key busted by HivePress's random post__not_in; since 2.1) — candidate for 2.3.

Hardening 2.2 for `2meet-data-optimizer` (1.2.0). Continues the search-index mode of ADR-004 (Release 2.2 design section added). Pairs with HivePress AddOn 0.3.0 and WooCommerce AddOn 0.2.0. ## Summary - **Drift**: `wp tmdo drift-check <entity> [--group] [--sample] [--fix]` (miss / diff / extra per key, rebuild from native, never in aeav_only); the post router keeps a group native while it is recorded as drifting; daily sample on the health-check event; full `--fix` 60 s after an entity leaves `disabled` or the Hook Bus is switched back on (`tmdo_drift_resync`). (PR-070〜074, 073b) - **Auto-promoter** requires a clean drift check within 2 days; shadow_read diffs capped at 20 per key per hour. (PR-075) - **Scope**: `post/wc_product` no longer registered (WooCommerce meta stays native). (PR-077) - **CLI**: `bridge-mode`, `member-backfill --all`, backfill groups from the registry (json groups included), `cleanup-hp-transients --options`. (PR-067〜069, 080) - **Fixed**: entity backfill invented default values; doctor / term-comment-diagnose / bridge-status false alarms; member-cutover fatal; doctor counting drift records of unregistered groups. (PR-066, 068, 076, 089a) - **Perf**: sort-only searches drive from the flat table when every post has a row (rating sort ~2.1 s native → ~0.44 s). (PR-079) - 2.1 audit notes (escaping, logger), live router tests compare sort keys on site-wide ties. (PR-078, 089b) Acceptance on dev36 (100k listings): `dev30/notes/tmdo-acceptance-2.2-2026-09-27.md`. Backlog: `dev30/notes/pr-backlog.md` (PR-066〜089). ## Test plan - [x] `composer test` 698, integration 572, phpcs 0, stan OK - [x] live suites 56/56 (drift, rest, resync, router, serialized, shadow_write, zero_side_effect) - [x] R-2.2 on dev36: registry without WC groups, drift 0, Hook Bus off → write → on → drift 0, doctor clean, V1–V7 (write overhead +6.3%) - [ ] CI green on this PR - [ ] After deploy: `cleanup-hp-transients --options --confirm`, `bridge-restore-native post --group=wc_product --retire`, first `drift-check` per entity (CHANGELOG 升級後) Known, not addressed: keyword + filter searches are slower routed (COUNT cache key busted by HivePress's random `post__not_in`; since 2.1) — candidate for 2.3.

wpdev added 19 commits 2026-09-27 10:47:16 +08:00

fix(2meet-data-optimizer): do not invent default rows when migrating an entity group ... 13ea341ef8

- migrate_group() REPLACEs the whole flat row; columns without a native row fell back to their DDL default (session_count / account_flags DEFAULT 0), so every backfilled user got invented zeros. Keys with no native row are now written as NULL; required NOT NULL columns keep their default.
- Covers the cron batch path (migrate_group_batch) and clears stale flat values on re-run.
- Verify: EntityMigrationEngineDefaultsTest (3 RED on main); dev36 user/activity backfill: checker extra 123 → 0 for session_count and account_flags, miss/diff unchanged.
- PR: PR-066

refactor(2meet-data-optimizer): list post groups for backfill from the registry ... 7807eae957

- Admin "backfill all" and `wp tmdo post-migrate-group` take any registered post group via TMDO_Post_Migration::backfillable_groups() instead of a hard-coded list of 7 (dev36 registers 11; the 4 HP add-on groups were never backfilled). Restored orphan groups are excluded and rejected — their values live only in the flat table.
- Groups with json fields (attachment, nav_menu_item) go row by row through the migration engine instead of leaving the json columns NULL in the bulk pivot.
- Verify: PostBackfillJsonTest (3 RED on hardening/2.2); dev36 post-migrate-group attachment / nav_menu_item / hp_draft_flags: json columns semantically equal native (24/24, 21/21), checker unchanged.
- PR: PR-067

feat(2meet-data-optimizer): backfill every registered user group with member-backfill --all ... 2022cf1d08

- member-backfill / member-audit take their groups from the registry (TMDO_Entity_Migration_Engine::backfillable_groups(), restored orphans excluded) instead of a hard-coded list that still named the never-registered `sso`; `--all` backfills every user group (R-2.1 had to use wp eval). TMDO_Post_Migration::backfillable_groups() now delegates to it.
- member-cutover checks the Mode_Manager result with is_wp_error(): a guarded or invalid transition used to crash with a fatal (WP_Error used as an array).
- Verify: CliMemberBackfillTest (--all and the orphan filter RED before; the --group case cannot fail under the non-throwing WP_CLI stub); dev36 `member-backfill --all --reset` 9 groups, 0 errors, checker user miss/diff/extra 0; `member-cutover --confirm` exits 1 with the Guard message, mode unchanged.
- PR: PR-068

feat(2meet-data-optimizer): set an entity bridge mode from the CLI under the Promotion Guard ... 382786cb9d

- New `wp tmdo bridge-mode <entity> <mode>` goes through TMDO_Mode_Manager::set(): unsafe jumps and aeav_only promotions refused by the Promotion Guard exit with the reason and leave the mode unchanged. Named apart from the Feature Flags `mode-set`.
- member-sso-sync and member-force-logout warn that they are deprecated: the user sso group they work on is no longer registered.
- Verify: CliBridgeModeTest (RED: method missing); dev36 bridge-mode user aeav_only refused (unsafe jump), term dual_write → shadow_read → aeav_only refused by the HivePress guard → back to disabled, wpdo_bridge_modes identical to before.
- PR: PR-069

feat(2meet-data-optimizer): report flat-vs-native drift with wp tmdo drift-check ... 02045c6f07

- New TMDO_Drift_Check compares every backfillable group of post / user / term / comment with native meta by value (first native row → safe unserialize → Type_Caster::to_db): miss / diff / extra per key, set-based SQL with BINARY exact matches, PHP comparison only for the remaining candidates. Serialized-vs-JSON and 50 vs 50.000000 are no longer drift; over-long text stored as NULL is not a miss; extra covers rows of other post types. Outcome per group goes to the wpdo_drift_status option; nothing else is written.
- `wp tmdo drift-check <entity> [--group] [--sample=<n>] [--format]`. A new class instead of extending the post-only shadow verifier, which queries row by row and writes shadow_diffs.
- Verify: DriftCheckTest; live suite drift; dev36 post miss=533 diff=2 extra=1 in 23 s (checker's 45 json false diffs gone), user/term/comment 0.
- PR: PR-070

feat(2meet-data-optimizer): repair drifted flat rows from native with drift-check --fix ... 3a347269c8

- TMDO_Drift_Check::fix() deletes flat rows of deleted entities and, for post groups, of posts outside the group's post types; rebuilds every other drifted entity's row from native through the new TMDO_Entity_Migration_Engine::migrate_entity() (keys without native rows become NULL, PR-066); keeps a row whose entity has no native value for the group but clears its columns (no NULL-only inserts, no deleted rows the router may join on); invalidates the row cache of each touched entity; then checks again. Native meta is never written.
- `wp tmdo drift-check <entity> --fix` (not combinable with --sample).
- Verify: DriftCheckTest fix cases (RED: method missing); dev36 --fix: post 42 rows rebuilt, 1 stray row (post 312, hp_booking in hp_draft_flags) deleted, all four entities 0 drift after, checker miss/extra 0.
- PR: PR-071

fix(2meet-data-optimizer): keep searches native while a group is known to drift ... ddefdc16d4

- The post query router routes no clause of a group whose last drift-check recorded miss / diff / extra; all or nothing, so such a query stays fully native. A group never checked keeps routing (decision 2026-09-27: no slowdown after upgrading; PR-073 samples daily).
- The check sits in route_clause(), which every routed clause, HivePress sort shape and meta_key sort goes through.
- Verify: PostQueryRouterDriftTest (drifting group and drifting sort key RED before); dev36 hp_listing price search in shadow_read: clean → flat, drift → native, unchecked → flat, found 90347 each.
- PR: PR-072

feat(2meet-data-optimizer): sample drift daily and record the result ... 6c782282c7

- TMDO_Drift_Check::cron_tick() runs on the existing wpdo_daily_health_check event (no new schedule to add or clear): every entity type whose flat tables are written (dual_write or higher) is checked on 500 random entities per group, and the outcome replaces the recorded status, so a group found drifting leaves the router that day.
- Verify: DriftCheckTest cron cases (RED: method / hook missing); dev36 with all four entities in dual_write: one tick over 22 groups in 10 s, 0 drift, status option restored.
- PR: PR-073

fix(2meet-data-optimizer): drift checks only where native meta is authoritative ... 1b21ee9add

- In aeav_only flat is the source of truth and native meta is stale by design, so drift there means nothing: drift-check --fix refuses (it would overwrite current flat values with stale native ones), the daily sample skips such entities, and the router keeps routing a drifting group instead of falling back to stale native meta.
- Found before PR-074 on hardening/2.2 (PR-071–073 not released; dev21 still runs aeav_only).
- Verify: DriftCheckTest / PostQueryRouterDriftTest aeav_only cases (RED before); dev36 fix() in a simulated aeav_only refused.
- PR: PR-073b

feat(2meet-data-optimizer): backfill writes made while the Hook Bus was off ... a26500206a

- When an entity goes from disabled to dual_write / shadow_read, or the Hook Bus flag is switched on, a single WP-Cron event (tmdo_drift_resync, one per entity, 60 s later) runs drift-check --fix for the entities that keep native meta, so values written while flat was bypassed land in flat. aeav_only is never touched.
- A full fix instead of the planned post_modified_gmt increment (decision 2026-09-27): meta-only writes such as WooCommerce's direct stock SQL do not bump post_modified. No wpdo_bridge_off_since.
- The resync hook is cleared on deactivate / uninstall (literal name: uninstall.php does not load TMDO_Drift_Check).
- Verify: DriftCheckTest resync cases (RED before); dev36 end to end with real WP-Cron: write while user disabled → miss 1 → bridge-mode user dual_write schedules tmdo_drift_resync [user] → run → drift 0.
- PR: PR-074

feat(2meet-data-optimizer): throttle shadow_read diffs per key and promote on clean drift checks ... 9d1a7c5591

- The shadow diff logger records at most 20 rows per entity type + key per hour across all objects, on top of the existing per-object 5-minute window: a key stale on every object (WooCommerce total_sales, written by direct SQL) no longer logs a row per product read.
- The auto-promoter's third condition is now TMDO_Drift_Check::clean_since(): every backfillable group of the entity checked within 2 days (sampled or full) with no drift; no record, no promotion. The shadow diff count no longer gates it (the filter's 4th argument is passed as 0). The Promotion Guard still refuses HivePress / WooCommerce sites.
- Reason classification was dropped (decision 2026-09-27): it needs a schema change and drift-check already reports miss / diff / extra per key.
- Verify: ShadowDiffThrottleTest, AutoPromoterDriftTest (RED before); dev36 simulated user shadow_read: clean_since yes, promotion still refused by the guard, mode unchanged.
- PR: PR-075

fix(2meet-data-optimizer): stop false alarms in doctor, term-comment-diagnose and bridge-status ... f15473527c

- doctor: entity flat tables come from the registry (TMDO_CLI::check_entity_tables()) instead of a hard-coded user list naming the never-registered sso group, which failed doctor on every site; it also reports Promotion Guard violations and the recorded drift status per entity (report_guard_and_drift()).
- term-comment-diagnose: filter status comes from each filter's is_enabled() (the garbage filter and misc bucket default to off but were shown enabled); a filter whose add-on is not active shows "not installed".
- bridge-status: flat_rows counts rows holding a value (wp_core keeps an all-NULL row per post: 15306% on dev36) and eav_ids of a post-type group counts only its post types.
- Tests: the WP_CLI stubs record their output in $GLOBALS['_wp_cli_output'].
- Verify: DoctorTest, CliFilterStatusTest, CliBridgeStatusTest (RED before); dev36 doctor exit 1 → 0, wp_core 15306% → 100%.
- PR: PR-076

refactor(2meet-data-optimizer): stop registering the wc_product post group ... f48df1a33f

- WooCommerce reads product meta directly and writes stock by SQL, so a flat copy drifted (R-2.1 V1 T4); product meta now stays in native only (plan decision 2). TMDO_Post_Fields registers wp_core, attachment and nav_menu_item. Existing wp_wpdo_post_wc_product tables are left in place and still dropped on uninstall; the orphan restore registers nothing for them while native meta holds the keys; `wp tmdo bridge-restore-native post --group=wc_product --retire` archives them.
- Tests that route products register the former group themselves (tmdo_test_register_wc_product_group(), same name and table); live router Q06 runs on wp_core's _wp_page_template instead of _sku. The post stress tester already skips flat tables that do not exist.
- Verify: PostFieldsRegistrationTest (RED before); dev36: wc_product absent from the registry and not restored as an orphan, a CLI order took stock 452 → 450 in native with no DB error (reverted), drift post 10 groups 0.
- PR: PR-077

chore(2meet-data-optimizer): address the 2.1 audit notes ... ad7acc40b7

- The four public REST routes say why they are public (viewable posts only, show_in_rest fields only, per_page cap, one view per IP per post per hour).
- Admin ternary echoes of literal strings are escaped for their context: esc_attr() in attributes, esc_html() for text; the object-cache badge is an if/else and aria-current is always printed ("false" off the current tab).
- Exceptions in the migration engine and the health cron go to TMDO_Logger instead of error_log().
- Verify: audit greps (ternary echoes 23 → 0, error_log outside the logger 0); dev36 renders 8 admin tabs with the change, no PHP notice, one aria-current="page" each.
- PR: PR-078

perf(2meet-data-optimizer): route sort-only searches from the flat table when every post has a row ... 84f6e32913

- A sort-only search (HivePress _sort, no routed filter) LEFT JOINs the flat table to keep posts without a row; it now drives from the flat table (STRAIGHT_JOIN, INNER JOIN) when every post of the type has a flat row. The router gives a new post of a type with its own groups an empty row (wp_insert_post); coverage is one anti-join per group, cached 15 min and per post-mode change (forget_coverage()).
- A flat-first search drops the GROUP BY posts.ID WordPress adds for any tax query when the flat row is the only join (WooCommerce's product_visibility NOT IN on searches is a subquery); a joining tax query keeps it.
- Verify: live router suite (5 new cases, RED before); dev36 rating__desc page, ids identical: native ~3.3 s, 2.1 routing ~1.5–1.9 s, now ~0.51–0.85 s (main query 440 → 46 ms, COUNT 400 → 160 ms). p50 ≤ 300 ms is still not met: page rendering and HivePress's own get_posts() in pre_get_posts (suppress_filters, ~100 ms) remain.
- PR: PR-079

chore(2meet-data-optimizer): clean HivePress transients left in wp_options ... 07955192d4

- `wp tmdo cleanup-hp-transients --options [--dry-run|--confirm] [--batch=<n>]` removes the rows the HivePress AddOn wrote to wp_options (`_transient_wpdo_hp_pm_*` and their timeouts) before AddOn 0.3.0 kept them in the object cache, deleting in batches (default 5000). It works without the AddOn active; the namespace is repeated in core for that.
- Verify: CleanupHpTransientsTest (RED before; only the AddOn's rows go, look-alike names stay); dev36 --dry-run counts 2,314,750 rows in 3.2 s. The --confirm run belongs after the AddOn change is deployed.
- PR: PR-080

docs(2meet-data-optimizer): 1.2.0 release notes ... fab07655a2

- Bump to 1.2.0; CHANGELOG / readme entries for PR-066–080, with the post-upgrade ops commands
- ADR-004 Release 2.2 design (scope, drift check / fix / router / resync) and CONTEXT.md drift paragraph
- Verify: composer test, integration, phpcs 0
- PR: PR-086

fix(2meet-data-optimizer): doctor ignores drift records of groups no longer registered ... edf9e6bc93

- A group's last drift record outlived its registration (wc_product and user hot on dev36, any add-on deactivated): doctor counted it and could warn forever with a --fix hint that cannot reach it. Only records of backfillable groups are read; the router and clean_since already go by the registry.
- Verify: DoctorTest (RED before); dev36 doctor user 9 → 8 groups
- PR: PR-089a

test(2meet-data-optimizer): compare site-wide rating sorts by sort key, not id ... 582ff13a11

- The PR-079 live router cases ran the HivePress rating sort on the whole site with an unqualified DECIMAL: native rounds 4.5 to 5 and orders its many ties by chance, so the id order passed or failed with the query plan. They now send DECIMAL(10,1), as the HivePress Rating field does, and compare each position's native sort key and found_posts; the flat-first / GROUP BY assertions stay.
- Verify: live router 18/18 twice; all suites 56/56
- PR: PR-089b

wpdev merged commit 28ac6e3cad into main 2026-09-27 10:56:49 +08:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

wpdev

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: wpdev/2meet-data-optimizer#6