hardening 2.1: 2meet-data-optimizer 1.1.0 (search-index mode, staged rollout) #4

Merged

wpdev merged 16 commits from hardening/2.1 into main 2026-09-26 21:22:49 +08:00

Conversation 0 Commits 16 Files Changed 52

+4395 -196

wpdev commented 2026-09-26 21:12:06 +08:00

Owner

Copy Link

Copy Source

Hardening 2.1 for 2meet-data-optimizer (version 1.1.0) — HivePress / WooCommerce search-index mode: native meta is the source of truth, flat tables are its search index. Decision and design: docs/adr-004-search-index-mode-for-hp-wc.md. Backlog: dev30/notes/pr-backlog.md (PR-052〜065 + 054b / 057b / 062b / 062c / 064b).

Pairs with 2meet-data-optimizer-hivepress-addon 0.2.0 (hardening/2.1).

Summary

• Promotion Guard: aeav_only and cleanup refused on HivePress / WooCommerce sites (tmdo_promotion_guard_allow).
• Hook Bus: shadow_read reads native (sampled flat compare) — fixes lost writes; dual_write / shadow_read flat values re-sync from native after every write; a NULL sync never inserts a row.
• Data: arrays in text fields serialized (wp tmdo repair-array-literals); bridge-restore-native overwrite / only-missing by mode, no invented defaults; zone tables never created are never queried.
• Post query router: routes only when the result equals native; LIKE wildcards; one join per group, no GROUP BY; meta orderby incl. the HivePress _sort shape; WordPress split query kept; flat-first for flat-sorted searches (tmdo_router_flat_first).
• Tests: new live layer tests/live (run inside a real site); composer phpcs uses phpcs.xml like CI.

Commits

• docs(2meet-data-optimizer): add PR-057b and PR-062c to the 1.1.0 notes
• perf(2meet-data-optimizer): drive flat-sorted searches from the flat table
• fix(2meet-data-optimizer): do not recreate a deleted entity's flat row
• docs(2meet-data-optimizer): ADR-004 search-index mode and 1.1.0 release notes
• perf(2meet-data-optimizer): keep WordPress's split query on routed searches
• feat(2meet-data-optimizer): route meta orderby including the HivePress sort clause
• fix(2meet-data-optimizer): join each routed group and drop the redundant GROUP BY
• fix(2meet-data-optimizer): route a meta query only when every registered clause is routable
• fix(2meet-data-optimizer): restore native meta without overwriting newer values or inventing defaults
• fix(2meet-data-optimizer): resync flat rows from native meta after each write
• fix(2meet-data-optimizer): skip zone deletes on tables that were never created
• fix(2meet-data-optimizer): serve native meta in shadow_read and compare flat values
• fix(2meet-data-optimizer): serialize arrays in text fields instead of storing "Array"
• fix(2meet-data-optimizer): skip cold-zone cache warming unless the cold module is active
• fix(2meet-data-optimizer): refuse aeav_only and cleanup on HivePress / WooCommerce sites
• test(2meet-data-optimizer): add live test runner and failing shadow_read write suite

Verification

• unit 687, integration 522, phpcs 0, phpstan OK; live suites 47/47 on dev36.
• R-2.1 acceptance on dev36 (100k listings): dev30/notes/tmdo-acceptance-2.1-2026-09-26.md — V1–V3, V5–V7 pass (V1 T4: WC direct-SQL stock, known); V4 results identical to native, p50 321–866 ms vs 300 target (accepted, carried into 2.2).

Rollout

Merging does not deploy: plugins/ (15 sites) stays on the current main until the staged rollout (dev22 first, 48 h). Existing aeav_only sites (dev20/21/22) need bridge-restore-native --demote with this version — see dev30/notes/ops-aeav-sites-audit-2026-09-26.md.

Hardening 2.1 for `2meet-data-optimizer` (version 1.1.0) — HivePress / WooCommerce **search-index mode**: native meta is the source of truth, flat tables are its search index. Decision and design: `docs/adr-004-search-index-mode-for-hp-wc.md`. Backlog: `dev30/notes/pr-backlog.md` (PR-052〜065 + 054b / 057b / 062b / 062c / 064b). Pairs with `2meet-data-optimizer-hivepress-addon` 0.2.0 (hardening/2.1). ## Summary - **Promotion Guard**: aeav_only and cleanup refused on HivePress / WooCommerce sites (`tmdo_promotion_guard_allow`). - **Hook Bus**: shadow_read reads native (sampled flat compare) — fixes lost writes; dual_write / shadow_read flat values re-sync from native after every write; a NULL sync never inserts a row. - **Data**: arrays in text fields serialized (`wp tmdo repair-array-literals`); `bridge-restore-native` overwrite / only-missing by mode, no invented defaults; zone tables never created are never queried. - **Post query router**: routes only when the result equals native; LIKE wildcards; one join per group, no GROUP BY; meta orderby incl. the HivePress `_sort` shape; WordPress split query kept; flat-first for flat-sorted searches (`tmdo_router_flat_first`). - **Tests**: new live layer `tests/live` (run inside a real site); `composer phpcs` uses `phpcs.xml` like CI. ## Commits - docs(2meet-data-optimizer): add PR-057b and PR-062c to the 1.1.0 notes - perf(2meet-data-optimizer): drive flat-sorted searches from the flat table - fix(2meet-data-optimizer): do not recreate a deleted entity's flat row - docs(2meet-data-optimizer): ADR-004 search-index mode and 1.1.0 release notes - perf(2meet-data-optimizer): keep WordPress's split query on routed searches - feat(2meet-data-optimizer): route meta orderby including the HivePress sort clause - fix(2meet-data-optimizer): join each routed group and drop the redundant GROUP BY - fix(2meet-data-optimizer): route a meta query only when every registered clause is routable - fix(2meet-data-optimizer): restore native meta without overwriting newer values or inventing defaults - fix(2meet-data-optimizer): resync flat rows from native meta after each write - fix(2meet-data-optimizer): skip zone deletes on tables that were never created - fix(2meet-data-optimizer): serve native meta in shadow_read and compare flat values - fix(2meet-data-optimizer): serialize arrays in text fields instead of storing "Array" - fix(2meet-data-optimizer): skip cold-zone cache warming unless the cold module is active - fix(2meet-data-optimizer): refuse aeav_only and cleanup on HivePress / WooCommerce sites - test(2meet-data-optimizer): add live test runner and failing shadow_read write suite ## Verification - unit 687, integration 522, phpcs 0, phpstan OK; live suites 47/47 on dev36. - R-2.1 acceptance on dev36 (100k listings): `dev30/notes/tmdo-acceptance-2.1-2026-09-26.md` — V1–V3, V5–V7 pass (V1 T4: WC direct-SQL stock, known); V4 results identical to native, p50 321–866 ms vs 300 target (accepted, carried into 2.2). ## Rollout Merging does **not** deploy: `plugins/` (15 sites) stays on the current main until the staged rollout (dev22 first, 48 h). Existing aeav_only sites (dev20/21/22) need `bridge-restore-native --demote` with this version — see `dev30/notes/ops-aeav-sites-audit-2026-09-26.md`.

wpdev added 16 commits 2026-09-26 21:12:07 +08:00

test(2meet-data-optimizer): add live test runner and failing shadow_read write suite ... f72a315986

- Add tests/live: a `wp eval-file` runner (PASS/FAIL/XFAIL/XPASS/SKIP) with
  fixtures that override entity modes in process (pre_option) and read
  native / flat values with plain SQL, so WP core meta flows run for real
- shadow_write suite reproduces the dev36 finding: in shadow_read,
  update_*_meta returns false and native meta is never written
  (3 XFAIL marked for PR-056); the dual_write control passes
- Move the dev36 integration harness into tests/live/tools; outputs go to
  cwd / $TMDO_LIVE_OUT, site URL and path are env-configurable
- Verify: wp --path=<dev36> eval-file tests/live/run.php shadow_write
  → 1 passed, 3 xfail; unit 658 OK, integration 480 OK,
  phpcs (phpcs.xml) 0, stan OK
- PR: PR-052

fix(2meet-data-optimizer): refuse aeav_only and cleanup on HivePress / WooCommerce sites ... 79a6c4d724

- Add TMDO_Promotion_Guard: HivePress models read every meta key with an
  empty-key get_*_meta() and WooCommerce does the same for products (and
  writes stock/sales with direct SQL), so on those sites native meta must
  stay authoritative. Promotion to aeav_only is refused in
  TMDO_Mode_Manager::set() (covers REST, CLI, phases, auto-promoter) for
  post/user/term/comment with HivePress and post/user with WooCommerce;
  TMDO_Post_Migration::cleanup() and the cleanup phase refuse too.
  Demotion stays open; filter tmdo_promotion_guard_allow overrides.
- Admin notice lists guarded entities already in aeav_only with the
  restore-native command (no forced demotion on read).
- Verify: PromotionGuardTest 8 OK; unit 668 OK, integration 480 OK,
  phpcs (phpcs.xml) 0, stan OK; dev36 live: 4 promotions + post-cleanup
  refused, wpdo_bridge_modes unchanged
- PR: PR-053

fix(2meet-data-optimizer): skip cold-zone cache warming unless the cold module is active ... 20bd59ddae

- TMDO_Cache_Layer::prefetch() / warm_post() (and so on_save_post /
  on_loop_start) queried wp_wpdo_cold_{post_type} whenever cold fields
  were registered, even with the cold module idle and the table never
  created: every HivePress / WC save logged "table doesn't exist" errors.
  They now return unless the module is write-active and the table exists.
- CacheLayerTest activates the module in setUp; new cases assert zero cold
  queries while idle or when the table is missing.
- Verify: CacheLayerTest 12 OK; unit 671 OK, integration 480 OK,
  phpcs (phpcs.xml) 0, stan OK; dev36 live: seeding listings/products/
  vendors with cold_hp_listing idle → 0 wpdo_cold errors (was hundreds)
- PR: PR-054

fix(2meet-data-optimizer): serialize arrays in text fields instead of storing "Array" ... 08f9e313e0

- TMDO_Type_Caster: text / textarea values that are arrays or objects are
  stored with maybe_serialize() (was (string) → the literal "Array", which
  bridge-restore-native then wrote back over native meta); from_db()
  unserializes them with TMDO_Safe_Unserialize. text columns are
  VARCHAR(255): longer values are stored NULL (native keeps them) and
  logged as text_overflow.
- Add `wp tmdo repair-array-literals [--entity] [--confirm]`
  (TMDO_Array_Literal_Repair): rewrites flat "Array" values from native
  meta and lists native rows that are "Array" themselves (backup needed).
- Verify: TypeCasterSerializeTest 13 OK, ArrayLiteralRepairTest 3 OK;
  unit 685 OK, integration 483 OK, phpcs (phpcs.xml) 0, stan OK;
  live `serialized` 3 PASS; dev36 repair dry-run lists the value corrupted
  by the 2026-09-26 round trip (user #1 community-events-location)
- PR: PR-055

fix(2meet-data-optimizer): serve native meta in shadow_read and compare flat values ... b84252d787

- In shadow_read, TMDO_Hook_Bus::intercept_get() returned the flat value.
  update_metadata() reads the old value through that filter after the Hook
  Bus has upserted the new one, so WP saw old === new and never wrote native
  meta: every update in shadow_read was lost on demotion (dev36 2026-09-26).
  shadow_read now returns $check (native) and compares flat on a sample of
  reads (filter tmdo_shadow_compare_sample_rate, default 0.1). aeav_only is
  unchanged; the post query router still routes in shadow_read.
- Live suite shadow_write: xfail markers removed, HivePress listing case
  added (new int key hp_rating_count); fixtures gain post() and a
  post_type-scoped field()/flat().
- Verify: live shadow_write + serialized 8/8 PASS (old code: 4 FAIL);
  unit 685 OK, integration 484 OK, phpcs (phpcs.xml) 0, stan OK
- PR: PR-056

fix(2meet-data-optimizer): skip zone deletes on tables that were never created ... 781f171770

- TMDO_Zone_Router::delete_post() deleted from wp_wpdo_hot_/cold_{post_type}
  whenever fields were registered for that zone, so every post delete on a
  HivePress site (cold fields registered, module idle, table never created)
  logged a "table doesn't exist" error. Hot and cold deletes now also need
  the table to exist; module state is not checked, so a table left by a
  rolled-back module still loses the rows of deleted posts.
- Live suite zero_side_effect: saving / updating / deleting an hp_listing
  issues no query on missing zone tables (covers PR-054 and this PR).
- Verify: ZoneRouterTest 22 OK; live zero_side_effect + shadow_write +
  serialized 9/9 PASS (old code: zero_side_effect FAIL); unit 687 OK,
  integration 484 OK, phpcs (phpcs.xml) 0, stan OK
- PR: PR-054b

fix(2meet-data-optimizer): resync flat rows from native meta after each write ... 77e338f9c3

- dual_write / shadow_read no longer write flat in the update_/add_/
  delete_{type}_metadata pre-filters, before WP decides whether native
  changes at all. added_/updated_/deleted_{type}_meta (priority 1) call
  TMDO_Hook_Bus::sync_from_native(), which copies the entity's first native
  row (plain SQL, NULL when gone) into flat. Flat now follows native when WP
  skips (same value), refuses (non-matching prev_value, add $unique on an
  existing key) or deletes (also delete_all: touched IDs are collected from
  the meta IDs in delete_{type}_meta), and a flat value left stale by a
  direct SQL write heals on the next write.
- update_metadata_by_mid key changes: the old key is remembered in
  update_{type}_metadata_by_mid and re-synced too.
- wpdo_after_write / wpdo_after_delete fire from the sync with the same
  arguments; aeav_only keeps the pre-filter short-circuit unchanged.
- $internal_ops lock key includes the entity type (user 5 ≠ post 5).
- Live suite resync: 4 entities × 2 modes × 3 scenarios (old code: 16 of
  24 FAIL); integration: pre-filter leaves flat to the sync, delete_all
  re-syncs every touched user.
- Verify: live resync + shadow_write + serialized + zero_side_effect 33/33
  PASS; unit 687 OK, integration 485 OK, phpcs (phpcs.xml) 0, stan OK
- PR: PR-057

fix(2meet-data-optimizer): restore native meta without overwriting newer values or inventing defaults ... 8c049f0cf2

- TMDO_Native_Restorer::run() gains a policy: overwrite (flat wins; the
  default in aeav_only, where writes only reached flat and native is
  stale) or only_missing (native wins, flat only fills gaps; the default in
  every other mode, where native is the source of truth and may be newer —
  on dev36 an aeav_only round trip overwrote WooCommerce stock 42 with 0).
  CLI: --overwrite / --only-missing.
- Native values are read first and each owned key is classified as
  missing / differs / same / default_skipped; unchanged values are no
  longer rewritten. A flat value equal to its field default is not
  written for an entity without a native row (dev36: 1,368 invented
  session_count / account_flags rows). --dry-run reports the counts.
- Verify: NativeRestorer* 17 OK (5 new: policy by mode, forced
  overwrite, only-missing in aeav_only, dry-run counts, no invented
  defaults); unit 687 OK, integration 490 OK, phpcs (phpcs.xml) 0,
  stan OK; dev36 `bridge-restore-native user --dry-run`: policy
  only_missing, activity 120 defaults skipped
- PR: PR-058

fix(2meet-data-optimizer): route a meta query only when every registered clause is routable ... b338925605

- Leave the query to WordPress on OR, nested groups with registered keys, several post types, orderby on a named routed clause, and compare/type pairs whose flat-column semantics differ from native (untyped numeric ranges, CHAR on decimals, NUMERIC on text)
- Wrap LIKE values as %esc_like%; hook pre_get_posts at priority 1000 so plugin clauses (HivePress) are seen
- Verify: wp eval-file tests/live/run.php router (7/7); qbench Q04/Q06/Q08/Q14/Q16 == native on dev36
- PR: PR-060

fix(2meet-data-optimizer): join each routed group and drop the redundant GROUP BY ... f2e261c522

- posts_join joins the flat table of every routed group (a wp_core key next to a post type group no longer fails with "Unknown column")
- Flat tables are one row per post (UNIQUE post_id): remove the posts_groupby hook; skip the INNER JOIN status/date copy route for NOT EXISTS, which must match posts without a flat row
- Verify: tests/live/run.php router (9/9); qbench Q05 475 == native, no SQL error; price filter EXPLAIN has no Using temporary
- PR: PR-061

feat(2meet-data-optimizer): route meta orderby including the HivePress sort clause ... ec75721976

- Sort shape (relation OR of NOT EXISTS + EXISTS on one key, named in orderby): drop it from meta_query, LEFT JOIN the flat table and order by its column, ties by ID
- Named routed clause as the sole orderby, and meta_key + meta_value_num (meta_value on string columns) with IS NOT NULL; stay native when the column would not order like CAST(meta_value AS type), orderby has several keys, or meta_value filters
- Verify: tests/live/run.php router (11/11); fe.sh price__asc / hourly_rate__desc / rating__desc ids == native
- PR: PR-062

perf(2meet-data-optimizer): keep WordPress's split query on routed searches ... aa2da274dd

- posts_request only drops SQL_CALC_FOUND_ROWS, but WordPress then treated the request as filtered and selected full post rows through a temporary table; split_the_query now keeps the ID-first split for routed searches when nothing else changed the request, and posts_request_ids drops SQL_CALC_FOUND_ROWS from the split ID query
- Verify: tests/live/run.php router (12/12, IDs-first request, found_posts == native); dev36 fe.sh price 589→340 ms, price__asc 961→427 ms, rating__desc 1724→872 ms, hourly_rate__desc 855→884 ms (was 1206)
- PR: PR-062b

docs(2meet-data-optimizer): ADR-004 search-index mode and 1.1.0 release notes ... 4fc59b757e

- ADR-004: HivePress / WooCommerce sites keep native meta as the source of truth and use flat tables as its search index; records the 2.1 write order, shadow_read, Promotion Guard, restore policy and router rules, with known limits
- CONTEXT.md Entity Bridge, BENCHMARK-SCALE.md front-end and write-chain measurements (dev36), CHANGELOG and readme for 1.1.0; version 1.1.0
- composer phpcs uses phpcs.xml like CI (was --standard=WordPress with 2312 false errors)
- Verify: composer phpcs (0); composer test; composer stan
- PR: PR-064

fix(2meet-data-optimizer): do not recreate a deleted entity's flat row ... a7134bfb80

- Deleting a post / user / term / comment drops its flat row in the adapter delete hook, then WordPress deletes each meta row; sync_from_native upserted NULL for each and brought the row back (512 orphan rows on dev36). With no native value it now only clears the column on an existing row
- Verify: tests/live/run.php resync (25/25, entity deletes in dual_write and shadow_read leave no flat row); mut.php T8 flat_row NO-ROW; integration 516
- PR: PR-057b

perf(2meet-data-optimizer): drive flat-sorted searches from the flat table ... dc9a8e9886

- A filtered search ordered by a flat column is rewritten to SELECT STRAIGHT_JOIN … FROM flat INNER JOIN wp_posts, with posts.ID tie-breaks (ours and plugin-appended) read as the equal flat post_id, so MariaDB sorts one table instead of looking every post up and sorting in a temporary table
- Only when the routed conditions already require a flat row on the sort group, without the copy route or a selective post filter; filter tmdo_router_flat_first turns it off; split_the_query compares with the same rewrite
- Verify: tests/live/run.php router (13/13); integration 522; dev36 page SQL Q07 263→147 ms, Q15 250→142 ms; fe.sh hourly_rate__desc 850→367 ms, ids == native
- PR: PR-062c

docs(2meet-data-optimizer): add PR-057b and PR-062c to the 1.1.0 notes ... e8731c5119

- CHANGELOG / readme 1.1.0: flat-first routing for flat-sorted searches (tmdo_router_flat_first) and the all-NULL flat row left after deleting an entity
- ADR-004: NULL sync never inserts a row; flat-first rule; front-end p50 from the R-2.1 acceptance run
- BENCHMARK-SCALE: front-end table from the acceptance run (flat tables backfilled) and the join-order finding behind PR-062c
- Verify: composer phpcs (0)
- PR: PR-064b

wpdev merged commit abbdb39acc into main 2026-09-26 21:22:49 +08:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

wpdev

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: wpdev/2meet-data-optimizer#4