Commit Graph

64 Commits

Author SHA1 Message Date
wpdev 2aa9058ee9 fix(2meet-data-optimizer): keep a post row's status and date copies when the migration engine rebuilds it
- migrate_entity() (drift-check --fix, per-row backfill) REPLACEs the whole
  flat row and left out the post_status / post_date copies, so a rebuilt row
  had NULL status. One such row turns the status/date copy route off for the
  whole group: on dev36 a single row rebuilt by the stage-3 drift fix made the
  date-ordered price search 1.47 s instead of driving from the flat index
- post rows now carry TMDO_Post_Status_Sync::insert_columns(); the bulk pivot
  backfill (ON DUPLICATE KEY UPDATE) was not affected
- Verify: PostStatusSyncTest (existing and new rows keep their copies),
  integration 593, unit 704, phpcs, stan; dev36 post-status-sync + front-end
- PR: PR-127 (found in 2.4 profiling)
2026-09-27 22:58:09 +08:00
wpdev 66db9e4c56 perf(2meet-data-optimizer): route HivePress's featured-listing lookup
- HivePress runs it on every listing search page from its own
  pre_get_posts: get_posts() with the search's meta query (still unrouted,
  including its sort group), meta_key hp_featured and orderby rand. The
  router declined both shapes, so the lookup stayed native: 2.26–2.38 s of a
  2.5 s page at 1.3M listings on dev36
- a sort group (key NOT EXISTS OR key EXISTS) that no orderby uses matches
  every post and is dropped; meta_key without meta_value and without a
  meta_value sort routes as EXISTS (WP_Query turns it into that clause)
- Verify: PostQueryRouterTest (featured lookup routed, meta_key with a value
  untouched), integration 592, unit 704, phpcs, stan; dev36 front-end
- PR: PR-122 (follow-up)
2026-09-27 22:54:17 +08:00
wpdev 2ed48c3ba3 perf(2meet-data-optimizer): route meta clauses without a value as EXISTS
- WP_Meta_Query adds a value condition only when 'value' is set, so a clause
  like array( 'key' => 'hp_featured' ) only tests that the key exists. The
  router read it as "= ''", declined it, and kept the whole query native
- HivePress runs such a query on every listing search page (featured-listing
  lookup in Attribute::set_search_query(): hp_featured plus the search's own
  filters, ORDER BY RAND()); at 1.3M listings on dev36 it took 2.26 s of a
  2.5 s page, routed or not
- Verify: PostQueryRouterTest (key-only clause routes as IS NOT NULL),
  integration 590, unit 704, phpcs, stan; dev36 front-end
- PR: PR-122
2026-09-27 22:46:52 +08:00
wpdev eac0955b76 perf(2meet-data-optimizer): let flat-first sorts use a (column, id) index
- a group index may end with the entity ID column (post_id, ...)
- flat-first rewrites the posts.ID tie-breaks once, in the sort's direction:
  with a (column, post_id) index MySQL returns rows in order instead of
  sorting the whole match. dev36 (1.3M listings), price range sorted by price:
  850 ms filesort -> 2 ms. Ties are ordered by ID the same way as the column
  (natively ID DESC); pagination stays stable, result sets are unchanged
- Verify: EntityRegistryIndexesTest, PostQueryRouterFoundRowsTest (ASC
  tie-break, single tie-break), unit 704, integration 589, phpcs, stan;
  dev36 with the HivePress indexes (PR-119)
- PR: PR-118
2026-09-27 22:40:59 +08:00
wpdev ca2cbdab65 perf(2meet-data-optimizer): keep cached routed counts across unrelated post meta writes
- cached counts followed posts' last_changed, which WordPress moves on any
  post meta write; themes write meta on page views (on dev36 Blocksy saved a
  dynamic-CSS descriptor for each listing a crawler visited), so the cache
  was invalidated every few seconds and keyword/filter searches recounted
- counts now follow their own last_changed group (tmdo_posts), bumped on
  clean_post_cache (post saved or deleted; hooked in every mode) and on flat
  writes through the Hook Bus, plus terms' last_changed; a routed search reads
  no unmanaged meta
- Verify: PostQueryRouterFoundRowsTest (unrelated meta write keeps the count,
  save or flat write recounts), integration 588, unit 703, phpcs, stan;
  dev36 front-end
- PR: PR-117 (2.3 acceptance)
2026-09-27 17:46:54 +08:00
wpdev 7851f150fa perf(2meet-data-optimizer): key the routed count cache on the SQL without wpdb's placeholder escape
- keyword searches carry wpdb's placeholder escape for "%" ({<hash>}), which
  is new in every request, so md5(count SQL) never matched across requests
  and every keyword search recounted (the 2.2 "keyword + filter routed slower
  than native" finding). The key now uses remove_placeholder_escape()
- found in the 2.3 acceptance: dev36 keyword + price routed 6.2 s vs native
  4.9 s; PR-091's in-process check reused one process, one escape
- Verify: PostQueryRouterFoundRowsTest (two escapes, one COUNT), integration
  588, unit 703, phpcs, stan; dev36 front-end
- PR: PR-116 (2.3 acceptance)
2026-09-27 17:39:36 +08:00
wpdev 1d70d13f45 fix(2meet-data-optimizer): close the gaps in deferred table upgrades found in the 2.3 review
- a deferred group whose stored definition matches again (reverted, or
  upgraded by another process) goes through end_deferral() instead of staying
  native for good (review F2)
- the tmdo_ensure_group_indexes handler upgrades deferred tables itself: with
  ALTERNATE_WP_CRON the event runs after init in a visitor's request (F3)
- a new group table created while the entity writes to flat stays native
  until the drift fix has filled it from native meta; an opted-in HivePress
  attribute no longer routes searches to an empty table (F6)
- direct_read() returns native values for a deferred group (F10); the Hook
  Bus ignores deferral in aeav_only, as the router does
- dbDelta failures: the definition is recorded only when every column exists
  (schema.upgrade_failed otherwise), so a failed ALTER is retried and the
  group is not released onto missing columns
- test bootstrap: get_metadata() reads the real meta table
- Verify: SchemaUpgradeDeferralTest (6), Hook Bus direct read, integration
  587, unit 703, phpcs, stan
- PR: PR-114 (2.3 acceptance correctness review)
2026-09-27 17:30:36 +08:00
wpdev 29adb05567 fix(2meet-data-optimizer): alter an existing flat table in one process at a time; keep a deferred group native until its drift fix
- security review H1: anonymous /wp-cron.php requests pass ddl_allowed(),
  and dbDelta on an existing table had no lock, so concurrent requests each
  ran the ALTER on a large table. The upgrade now runs under the same MySQL
  GET_LOCK as the index DDL (skipped when another process holds it) and
  re-checks the stored hash once locked; this covers aeav_only too. The table
  build moves unchanged to write_table(); lock()/unlock() are shared
- security review M1: after a deferred upgrade the group stays native until
  tmdo_drift_resync has rebuilt the rows written meanwhile
  (TMDO_Schema_Manager::resynced() from run_scheduled_fix()); with no flat
  writes (disabled) it is released at once
- Verify: SchemaUpgradeDeferralTest (lock held elsewhere, resync release),
  integration 585, unit 702, phpcs, stan
- PR: PR-111 (2.3 acceptance security review)
2026-09-27 17:22:57 +08:00
wpdev 06a3594417 fix(2meet-data-optimizer): take the index lock on the primary under read/write splitting
- when $wpdb has send_reads_to_masters() (HyperDB, LudicrousDB), it is called
  before SELECT GET_LOCK, which a replica would take without locking out the
  ALTER on the primary
- Verify: GroupCompositeIndexTest (call order through a wpdb proxy),
  integration 583, unit 701, phpcs, stan
- PR: PR-108
2026-09-27 17:09:10 +08:00
wpdev e18030b206 fix(2meet-data-optimizer): upgrade existing flat tables from WP-CLI or WP-Cron only
- a changed group definition no longer runs dbDelta on an existing table
  inside a web request (seconds to minutes on a large table, and every
  concurrent request tried it): the group is recorded in tmdo_schema_deferred
  and tmdo_ensure_group_indexes is scheduled, whose request upgrades it
- meanwhile the group stays native: Hook Bus field lookups return no field and
  the router treats it as drifting; afterwards a drift fix
  (tmdo_drift_resync) rebuilds the writes that skipped the flat table
- new tables are still created at once; aeav_only upgrades at once (the flat
  table is the only copy); doctor lists deferred groups
- Verify: SchemaUpgradeDeferralTest, Hook Bus and router tests, integration
  582, unit 701, phpcs, stan; dev36 live cycle in the 2.3 acceptance
- PR: PR-107
2026-09-27 17:07:48 +08:00
wpdev 5968bcd95d fix(2meet-data-optimizer): forget post status columns of groups whose flat table is gone
- on_transition() skips a recorded group whose table no longer exists and
  removes it from tmdo_post_status_columns; every post status change used to
  UPDATE the missing table and print a database error (seen on dev36 after a
  throwaway group's table was dropped)
- Verify: PostStatusSyncTest (dropped table), integration 578, unit 701, phpcs, stan
- PR: PR-106
2026-09-27 16:59:08 +08:00
wpdev b187c1c4d3 perf(2meet-data-optimizer): count routed searches from a cached base count minus the excluded posts
- a routed search with post__not_in (HivePress sends a random set per page)
  caches the COUNT without the exclusion and subtracts a live primary-key
  count of the excluded posts that match; the total stays exact
- the excluded count drops STRAIGHT_JOIN so MySQL starts from posts.ID
- Verify: PostQueryRouterFoundRowsTest, unit 698, integration 577, phpcs, stan;
  dev36 live router suite and keyword + price search
- PR: PR-091
2026-09-27 16:19:11 +08:00
wpdev ec8d69a6f4 fix(2meet-data-optimizer): add declared flat-table indexes from WP-CLI or WP-Cron only, one process at a time
- ensure_group_indexes() runs the ALTER only under WP-CLI or WP-Cron (filter
  tmdo_schema_ddl_allowed); web requests, including /wp-admin/ before
  auth_redirect() and admin-post.php, schedule tmdo_ensure_group_indexes instead
- MySQL GET_LOCK keeps it to one process (add_option() is INSERT ... ON
  DUPLICATE KEY UPDATE and cannot serve as a lock); released by MySQL if the
  process dies
- the applied signature is recorded only when every ALTER succeeds; failures
  log schema.index_failed and stay in pending_indexes(), which doctor reports
- new tables get their declared cidx_ indexes in CREATE TABLE
- test stub wpdb::query()/insert() return false on SQL errors like wpdb
- Verify: GroupCompositeIndexTest 7/7, unit 698, integration 576, phpcs, stan
- PR: PR-090a
2026-09-27 16:07:23 +08:00
wpdev edf9e6bc93 fix(2meet-data-optimizer): doctor ignores drift records of groups no longer registered
- A group's last drift record outlived its registration (wc_product and user hot on dev36, any add-on deactivated): doctor counted it and could warn forever with a --fix hint that cannot reach it. Only records of backfillable groups are read; the router and clean_since already go by the registry.
- Verify: DoctorTest (RED before); dev36 doctor user 9 → 8 groups
- PR: PR-089a
2026-09-27 10:07:02 +08:00
wpdev 07955192d4 chore(2meet-data-optimizer): clean HivePress transients left in wp_options
- `wp tmdo cleanup-hp-transients --options [--dry-run|--confirm] [--batch=<n>]` removes the rows the HivePress AddOn wrote to wp_options (`_transient_wpdo_hp_pm_*` and their timeouts) before AddOn 0.3.0 kept them in the object cache, deleting in batches (default 5000). It works without the AddOn active; the namespace is repeated in core for that.
- Verify: CleanupHpTransientsTest (RED before; only the AddOn's rows go, look-alike names stay); dev36 --dry-run counts 2,314,750 rows in 3.2 s. The --confirm run belongs after the AddOn change is deployed.
- PR: PR-080
2026-09-27 09:06:19 +08:00
wpdev f48df1a33f refactor(2meet-data-optimizer): stop registering the wc_product post group
- WooCommerce reads product meta directly and writes stock by SQL, so a flat copy drifted (R-2.1 V1 T4); product meta now stays in native only (plan decision 2). TMDO_Post_Fields registers wp_core, attachment and nav_menu_item. Existing wp_wpdo_post_wc_product tables are left in place and still dropped on uninstall; the orphan restore registers nothing for them while native meta holds the keys; `wp tmdo bridge-restore-native post --group=wc_product --retire` archives them.
- Tests that route products register the former group themselves (tmdo_test_register_wc_product_group(), same name and table); live router Q06 runs on wp_core's _wp_page_template instead of _sku. The post stress tester already skips flat tables that do not exist.
- Verify: PostFieldsRegistrationTest (RED before); dev36: wc_product absent from the registry and not restored as an orphan, a CLI order took stock 452 → 450 in native with no DB error (reverted), drift post 10 groups 0.
- PR: PR-077
2026-09-27 01:11:52 +08:00
wpdev f15473527c fix(2meet-data-optimizer): stop false alarms in doctor, term-comment-diagnose and bridge-status
- doctor: entity flat tables come from the registry (TMDO_CLI::check_entity_tables()) instead of a hard-coded user list naming the never-registered sso group, which failed doctor on every site; it also reports Promotion Guard violations and the recorded drift status per entity (report_guard_and_drift()).
- term-comment-diagnose: filter status comes from each filter's is_enabled() (the garbage filter and misc bucket default to off but were shown enabled); a filter whose add-on is not active shows "not installed".
- bridge-status: flat_rows counts rows holding a value (wp_core keeps an all-NULL row per post: 15306% on dev36) and eav_ids of a post-type group counts only its post types.
- Tests: the WP_CLI stubs record their output in $GLOBALS['_wp_cli_output'].
- Verify: DoctorTest, CliFilterStatusTest, CliBridgeStatusTest (RED before); dev36 doctor exit 1 → 0, wp_core 15306% → 100%.
- PR: PR-076
2026-09-27 01:00:13 +08:00
wpdev 9d1a7c5591 feat(2meet-data-optimizer): throttle shadow_read diffs per key and promote on clean drift checks
- The shadow diff logger records at most 20 rows per entity type + key per hour across all objects, on top of the existing per-object 5-minute window: a key stale on every object (WooCommerce total_sales, written by direct SQL) no longer logs a row per product read.
- The auto-promoter's third condition is now TMDO_Drift_Check::clean_since(): every backfillable group of the entity checked within 2 days (sampled or full) with no drift; no record, no promotion. The shadow diff count no longer gates it (the filter's 4th argument is passed as 0). The Promotion Guard still refuses HivePress / WooCommerce sites.
- Reason classification was dropped (decision 2026-09-27): it needs a schema change and drift-check already reports miss / diff / extra per key.
- Verify: ShadowDiffThrottleTest, AutoPromoterDriftTest (RED before); dev36 simulated user shadow_read: clean_since yes, promotion still refused by the guard, mode unchanged.
- PR: PR-075
2026-09-27 00:48:18 +08:00
wpdev a26500206a feat(2meet-data-optimizer): backfill writes made while the Hook Bus was off
- When an entity goes from disabled to dual_write / shadow_read, or the Hook Bus flag is switched on, a single WP-Cron event (tmdo_drift_resync, one per entity, 60 s later) runs drift-check --fix for the entities that keep native meta, so values written while flat was bypassed land in flat. aeav_only is never touched.
- A full fix instead of the planned post_modified_gmt increment (decision 2026-09-27): meta-only writes such as WooCommerce's direct stock SQL do not bump post_modified. No wpdo_bridge_off_since.
- The resync hook is cleared on deactivate / uninstall (literal name: uninstall.php does not load TMDO_Drift_Check).
- Verify: DriftCheckTest resync cases (RED before); dev36 end to end with real WP-Cron: write while user disabled → miss 1 → bridge-mode user dual_write schedules tmdo_drift_resync [user] → run → drift 0.
- PR: PR-074
2026-09-27 00:41:43 +08:00
wpdev 1b21ee9add fix(2meet-data-optimizer): drift checks only where native meta is authoritative
- In aeav_only flat is the source of truth and native meta is stale by design, so drift there means nothing: drift-check --fix refuses (it would overwrite current flat values with stale native ones), the daily sample skips such entities, and the router keeps routing a drifting group instead of falling back to stale native meta.
- Found before PR-074 on hardening/2.2 (PR-071–073 not released; dev21 still runs aeav_only).
- Verify: DriftCheckTest / PostQueryRouterDriftTest aeav_only cases (RED before); dev36 fix() in a simulated aeav_only refused.
- PR: PR-073b
2026-09-27 00:34:03 +08:00
wpdev 6c782282c7 feat(2meet-data-optimizer): sample drift daily and record the result
- TMDO_Drift_Check::cron_tick() runs on the existing wpdo_daily_health_check event (no new schedule to add or clear): every entity type whose flat tables are written (dual_write or higher) is checked on 500 random entities per group, and the outcome replaces the recorded status, so a group found drifting leaves the router that day.
- Verify: DriftCheckTest cron cases (RED: method / hook missing); dev36 with all four entities in dual_write: one tick over 22 groups in 10 s, 0 drift, status option restored.
- PR: PR-073
2026-09-27 00:28:00 +08:00
wpdev ddefdc16d4 fix(2meet-data-optimizer): keep searches native while a group is known to drift
- The post query router routes no clause of a group whose last drift-check recorded miss / diff / extra; all or nothing, so such a query stays fully native. A group never checked keeps routing (decision 2026-09-27: no slowdown after upgrading; PR-073 samples daily).
- The check sits in route_clause(), which every routed clause, HivePress sort shape and meta_key sort goes through.
- Verify: PostQueryRouterDriftTest (drifting group and drifting sort key RED before); dev36 hp_listing price search in shadow_read: clean → flat, drift → native, unchecked → flat, found 90347 each.
- PR: PR-072
2026-09-27 00:22:47 +08:00
wpdev 3a347269c8 feat(2meet-data-optimizer): repair drifted flat rows from native with drift-check --fix
- TMDO_Drift_Check::fix() deletes flat rows of deleted entities and, for post groups, of posts outside the group's post types; rebuilds every other drifted entity's row from native through the new TMDO_Entity_Migration_Engine::migrate_entity() (keys without native rows become NULL, PR-066); keeps a row whose entity has no native value for the group but clears its columns (no NULL-only inserts, no deleted rows the router may join on); invalidates the row cache of each touched entity; then checks again. Native meta is never written.
- `wp tmdo drift-check <entity> --fix` (not combinable with --sample).
- Verify: DriftCheckTest fix cases (RED: method missing); dev36 --fix: post 42 rows rebuilt, 1 stray row (post 312, hp_booking in hp_draft_flags) deleted, all four entities 0 drift after, checker miss/extra 0.
- PR: PR-071
2026-09-27 00:07:33 +08:00
wpdev 02045c6f07 feat(2meet-data-optimizer): report flat-vs-native drift with wp tmdo drift-check
- New TMDO_Drift_Check compares every backfillable group of post / user / term / comment with native meta by value (first native row → safe unserialize → Type_Caster::to_db): miss / diff / extra per key, set-based SQL with BINARY exact matches, PHP comparison only for the remaining candidates. Serialized-vs-JSON and 50 vs 50.000000 are no longer drift; over-long text stored as NULL is not a miss; extra covers rows of other post types. Outcome per group goes to the wpdo_drift_status option; nothing else is written.
- `wp tmdo drift-check <entity> [--group] [--sample=<n>] [--format]`. A new class instead of extending the post-only shadow verifier, which queries row by row and writes shadow_diffs.
- Verify: DriftCheckTest; live suite drift; dev36 post miss=533 diff=2 extra=1 in 23 s (checker's 45 json false diffs gone), user/term/comment 0.
- PR: PR-070
2026-09-26 23:56:23 +08:00
wpdev 2022cf1d08 feat(2meet-data-optimizer): backfill every registered user group with member-backfill --all
- member-backfill / member-audit take their groups from the registry (TMDO_Entity_Migration_Engine::backfillable_groups(), restored orphans excluded) instead of a hard-coded list that still named the never-registered `sso`; `--all` backfills every user group (R-2.1 had to use wp eval). TMDO_Post_Migration::backfillable_groups() now delegates to it.
- member-cutover checks the Mode_Manager result with is_wp_error(): a guarded or invalid transition used to crash with a fatal (WP_Error used as an array).
- Verify: CliMemberBackfillTest (--all and the orphan filter RED before; the --group case cannot fail under the non-throwing WP_CLI stub); dev36 `member-backfill --all --reset` 9 groups, 0 errors, checker user miss/diff/extra 0; `member-cutover --confirm` exits 1 with the Guard message, mode unchanged.
- PR: PR-068
2026-09-26 23:30:28 +08:00
wpdev 7807eae957 refactor(2meet-data-optimizer): list post groups for backfill from the registry
- Admin "backfill all" and `wp tmdo post-migrate-group` take any registered post group via TMDO_Post_Migration::backfillable_groups() instead of a hard-coded list of 7 (dev36 registers 11; the 4 HP add-on groups were never backfilled). Restored orphan groups are excluded and rejected — their values live only in the flat table.
- Groups with json fields (attachment, nav_menu_item) go row by row through the migration engine instead of leaving the json columns NULL in the bulk pivot.
- Verify: PostBackfillJsonTest (3 RED on hardening/2.2); dev36 post-migrate-group attachment / nav_menu_item / hp_draft_flags: json columns semantically equal native (24/24, 21/21), checker unchanged.
- PR: PR-067
2026-09-26 22:54:27 +08:00
wpdev 13ea341ef8 fix(2meet-data-optimizer): do not invent default rows when migrating an entity group
- migrate_group() REPLACEs the whole flat row; columns without a native row fell back to their DDL default (session_count / account_flags DEFAULT 0), so every backfilled user got invented zeros. Keys with no native row are now written as NULL; required NOT NULL columns keep their default.
- Covers the cron batch path (migrate_group_batch) and clears stale flat values on re-run.
- Verify: EntityMigrationEngineDefaultsTest (3 RED on main); dev36 user/activity backfill: checker extra 123 → 0 for session_count and account_flags, miss/diff unchanged.
- PR: PR-066
2026-09-26 22:46:01 +08:00
wpdev 336664473b fix(2meet-data-optimizer): keep registered fields out of the public listings REST endpoints
Anti-EAV Lint + Quality Gate / anti-eav-lint (pull_request) Successful in 15s
Tests / PHP Lint (pull_request) Successful in 17s
Tests / Unit Tests (pull_request) Successful in 51s
Tests / Integration Tests (pull_request) Successful in 1m22s
Tests / PHPCS (pull_request) Successful in 1m12s
Tests / PHPStan (pull_request) Successful in 1m10s
- show_in_rest defaulted to true, so GET /wpdo/v1/listings and /listings/{id} (public by design) returned every registered hot / cold field to anonymous visitors — e.g. infocards tmeetic_ig_access_token_enc / _token_expiry / _user_id and WooCommerce _purchase_note (confirmed on dev36)
- Fields are private unless registered with show_in_rest => true; protected (underscore) and credential-like keys (token, secret, password, api key, private key, credential, salt, nonce) are never exposed
- Zone A list path no longer returns whole rows when nothing is public, and filters / orderby accept public columns only
- Version 1.1.1; CHANGELOG, readme
- Verify: unit 690 (+3), integration 525 (+3 private-column tests, RED on 1.1.0), live suite rest 2/2 (RED on 1.1.0), all live 49/49, phpcs 0, stan OK
2026-09-26 22:22:40 +08:00
wpdev dc9a8e9886 perf(2meet-data-optimizer): drive flat-sorted searches from the flat table
- A filtered search ordered by a flat column is rewritten to SELECT STRAIGHT_JOIN … FROM flat INNER JOIN wp_posts, with posts.ID tie-breaks (ours and plugin-appended) read as the equal flat post_id, so MariaDB sorts one table instead of looking every post up and sorting in a temporary table
- Only when the routed conditions already require a flat row on the sort group, without the copy route or a selective post filter; filter tmdo_router_flat_first turns it off; split_the_query compares with the same rewrite
- Verify: tests/live/run.php router (13/13); integration 522; dev36 page SQL Q07 263→147 ms, Q15 250→142 ms; fe.sh hourly_rate__desc 850→367 ms, ids == native
- PR: PR-062c
2026-09-26 21:05:11 +08:00
wpdev a7134bfb80 fix(2meet-data-optimizer): do not recreate a deleted entity's flat row
- Deleting a post / user / term / comment drops its flat row in the adapter delete hook, then WordPress deletes each meta row; sync_from_native upserted NULL for each and brought the row back (512 orphan rows on dev36). With no native value it now only clears the column on an existing row
- Verify: tests/live/run.php resync (25/25, entity deletes in dual_write and shadow_read leave no flat row); mut.php T8 flat_row NO-ROW; integration 516
- PR: PR-057b
2026-09-26 20:34:28 +08:00
wpdev aa2da274dd perf(2meet-data-optimizer): keep WordPress's split query on routed searches
- posts_request only drops SQL_CALC_FOUND_ROWS, but WordPress then treated the request as filtered and selected full post rows through a temporary table; split_the_query now keeps the ID-first split for routed searches when nothing else changed the request, and posts_request_ids drops SQL_CALC_FOUND_ROWS from the split ID query
- Verify: tests/live/run.php router (12/12, IDs-first request, found_posts == native); dev36 fe.sh price 589→340 ms, price__asc 961→427 ms, rating__desc 1724→872 ms, hourly_rate__desc 855→884 ms (was 1206)
- PR: PR-062b
2026-09-26 20:14:29 +08:00
wpdev ec75721976 feat(2meet-data-optimizer): route meta orderby including the HivePress sort clause
- Sort shape (relation OR of NOT EXISTS + EXISTS on one key, named in orderby): drop it from meta_query, LEFT JOIN the flat table and order by its column, ties by ID
- Named routed clause as the sole orderby, and meta_key + meta_value_num (meta_value on string columns) with IS NOT NULL; stay native when the column would not order like CAST(meta_value AS type), orderby has several keys, or meta_value filters
- Verify: tests/live/run.php router (11/11); fe.sh price__asc / hourly_rate__desc / rating__desc ids == native
- PR: PR-062
2026-09-26 20:04:23 +08:00
wpdev f2e261c522 fix(2meet-data-optimizer): join each routed group and drop the redundant GROUP BY
- posts_join joins the flat table of every routed group (a wp_core key next to a post type group no longer fails with "Unknown column")
- Flat tables are one row per post (UNIQUE post_id): remove the posts_groupby hook; skip the INNER JOIN status/date copy route for NOT EXISTS, which must match posts without a flat row
- Verify: tests/live/run.php router (9/9); qbench Q05 475 == native, no SQL error; price filter EXPLAIN has no Using temporary
- PR: PR-061
2026-09-26 19:47:53 +08:00
wpdev b338925605 fix(2meet-data-optimizer): route a meta query only when every registered clause is routable
- Leave the query to WordPress on OR, nested groups with registered keys, several post types, orderby on a named routed clause, and compare/type pairs whose flat-column semantics differ from native (untyped numeric ranges, CHAR on decimals, NUMERIC on text)
- Wrap LIKE values as %esc_like%; hook pre_get_posts at priority 1000 so plugin clauses (HivePress) are seen
- Verify: wp eval-file tests/live/run.php router (7/7); qbench Q04/Q06/Q08/Q14/Q16 == native on dev36
- PR: PR-060
2026-09-26 19:41:49 +08:00
wpdev 8c049f0cf2 fix(2meet-data-optimizer): restore native meta without overwriting newer values or inventing defaults
- TMDO_Native_Restorer::run() gains a policy: overwrite (flat wins; the
  default in aeav_only, where writes only reached flat and native is
  stale) or only_missing (native wins, flat only fills gaps; the default in
  every other mode, where native is the source of truth and may be newer —
  on dev36 an aeav_only round trip overwrote WooCommerce stock 42 with 0).
  CLI: --overwrite / --only-missing.
- Native values are read first and each owned key is classified as
  missing / differs / same / default_skipped; unchanged values are no
  longer rewritten. A flat value equal to its field default is not
  written for an entity without a native row (dev36: 1,368 invented
  session_count / account_flags rows). --dry-run reports the counts.
- Verify: NativeRestorer* 17 OK (5 new: policy by mode, forced
  overwrite, only-missing in aeav_only, dry-run counts, no invented
  defaults); unit 687 OK, integration 490 OK, phpcs (phpcs.xml) 0,
  stan OK; dev36 `bridge-restore-native user --dry-run`: policy
  only_missing, activity 120 defaults skipped
- PR: PR-058
2026-09-26 19:22:34 +08:00
wpdev 77e338f9c3 fix(2meet-data-optimizer): resync flat rows from native meta after each write
- dual_write / shadow_read no longer write flat in the update_/add_/
  delete_{type}_metadata pre-filters, before WP decides whether native
  changes at all. added_/updated_/deleted_{type}_meta (priority 1) call
  TMDO_Hook_Bus::sync_from_native(), which copies the entity's first native
  row (plain SQL, NULL when gone) into flat. Flat now follows native when WP
  skips (same value), refuses (non-matching prev_value, add $unique on an
  existing key) or deletes (also delete_all: touched IDs are collected from
  the meta IDs in delete_{type}_meta), and a flat value left stale by a
  direct SQL write heals on the next write.
- update_metadata_by_mid key changes: the old key is remembered in
  update_{type}_metadata_by_mid and re-synced too.
- wpdo_after_write / wpdo_after_delete fire from the sync with the same
  arguments; aeav_only keeps the pre-filter short-circuit unchanged.
- $internal_ops lock key includes the entity type (user 5 ≠ post 5).
- Live suite resync: 4 entities × 2 modes × 3 scenarios (old code: 16 of
  24 FAIL); integration: pre-filter leaves flat to the sync, delete_all
  re-syncs every touched user.
- Verify: live resync + shadow_write + serialized + zero_side_effect 33/33
  PASS; unit 687 OK, integration 485 OK, phpcs (phpcs.xml) 0, stan OK
- PR: PR-057
2026-09-26 19:13:02 +08:00
wpdev b84252d787 fix(2meet-data-optimizer): serve native meta in shadow_read and compare flat values
- In shadow_read, TMDO_Hook_Bus::intercept_get() returned the flat value.
  update_metadata() reads the old value through that filter after the Hook
  Bus has upserted the new one, so WP saw old === new and never wrote native
  meta: every update in shadow_read was lost on demotion (dev36 2026-09-26).
  shadow_read now returns $check (native) and compares flat on a sample of
  reads (filter tmdo_shadow_compare_sample_rate, default 0.1). aeav_only is
  unchanged; the post query router still routes in shadow_read.
- Live suite shadow_write: xfail markers removed, HivePress listing case
  added (new int key hp_rating_count); fixtures gain post() and a
  post_type-scoped field()/flat().
- Verify: live shadow_write + serialized 8/8 PASS (old code: 4 FAIL);
  unit 685 OK, integration 484 OK, phpcs (phpcs.xml) 0, stan OK
- PR: PR-056
2026-09-26 18:56:45 +08:00
wpdev 08f9e313e0 fix(2meet-data-optimizer): serialize arrays in text fields instead of storing "Array"
- TMDO_Type_Caster: text / textarea values that are arrays or objects are
  stored with maybe_serialize() (was (string) → the literal "Array", which
  bridge-restore-native then wrote back over native meta); from_db()
  unserializes them with TMDO_Safe_Unserialize. text columns are
  VARCHAR(255): longer values are stored NULL (native keeps them) and
  logged as text_overflow.
- Add `wp tmdo repair-array-literals [--entity] [--confirm]`
  (TMDO_Array_Literal_Repair): rewrites flat "Array" values from native
  meta and lists native rows that are "Array" themselves (backup needed).
- Verify: TypeCasterSerializeTest 13 OK, ArrayLiteralRepairTest 3 OK;
  unit 685 OK, integration 483 OK, phpcs (phpcs.xml) 0, stan OK;
  live `serialized` 3 PASS; dev36 repair dry-run lists the value corrupted
  by the 2026-09-26 round trip (user #1 community-events-location)
- PR: PR-055
2026-09-26 18:45:59 +08:00
wpdev 79a6c4d724 fix(2meet-data-optimizer): refuse aeav_only and cleanup on HivePress / WooCommerce sites
- Add TMDO_Promotion_Guard: HivePress models read every meta key with an
  empty-key get_*_meta() and WooCommerce does the same for products (and
  writes stock/sales with direct SQL), so on those sites native meta must
  stay authoritative. Promotion to aeav_only is refused in
  TMDO_Mode_Manager::set() (covers REST, CLI, phases, auto-promoter) for
  post/user/term/comment with HivePress and post/user with WooCommerce;
  TMDO_Post_Migration::cleanup() and the cleanup phase refuse too.
  Demotion stays open; filter tmdo_promotion_guard_allow overrides.
- Admin notice lists guarded entities already in aeav_only with the
  restore-native command (no forced demotion on read).
- Verify: PromotionGuardTest 8 OK; unit 668 OK, integration 480 OK,
  phpcs (phpcs.xml) 0, stan OK; dev36 live: 4 promotions + post-cleanup
  refused, wpdo_bridge_modes unchanged
- PR: PR-053
2026-09-26 18:29:04 +08:00
wpdev 08492dd795 fix(2meet-data-optimizer): compare empty strings correctly when retiring a group
Anti-EAV Lint + Quality Gate / anti-eav-lint (pull_request) Successful in 14s
Tests / Unit Tests (pull_request) Successful in 33s
Tests / PHP Lint (pull_request) Successful in 36s
Tests / Integration Tests (pull_request) Successful in 57s
Tests / PHPCS (pull_request) Successful in 35s
Tests / PHPStan (pull_request) Failing after 50s
- retire() read native meta with wpdb::get_col(), which (via get_var())
  returns null for an empty string, so a flat '' written back as '' was
  counted as a mismatch and blocked the retirement. It now reads raw rows
  with get_results().
- The integration test wpdb stub's get_var() / get_col() now read '' as
  null like the real wpdb, so tests see WordPress behaviour.
- Found writing back the sso group on dev21 / dev22 (picture_url = '').
- Verify: core unit 658 / integration 480 green, phpcs clean
- PR: PR-042 follow-up
2026-09-26 04:24:14 +08:00
wpdev 44a188b7ee feat(2meet-data-optimizer): group-scoped native write-back and group retirement
- TMDO_Native_Restorer::run() takes an optional group; the CLI gains
  `bridge-restore-native <entity> --group=<g>`.
- New retire(): once native meta holds every non-NULL value of a group's
  flat table (checked against the persisted field definitions), forget
  its registry_meta row and rename the table to <table>_retired_<Ymd>, so
  orphan restore no longer routes its keys to the flat table. Refused
  while a plugin still registers the group or any value differs.
  CLI: `--group=<g> --retire`.
- For retiring the user `sso` group (SSO AddOn deprecation).
- Verify: core unit 659 / integration 479 green, phpcs clean
- PR: PR-042
2026-09-26 04:06:31 +08:00
wpdev 10671e7b78 perf(2meet-data-optimizer): cache found-rows counts for routed queries
- TMDO_Post_Query_Router drops SQL_CALC_FOUND_ROWS from routed searches
  (posts_request) and answers found_posts_query from a COUNT(*) of the
  same query, cached in the object cache under posts' last_changed.
- Hook Bus flat post writes and deletes move posts' last_changed: in
  aeav_only they skip update_metadata()'s own hooks, so query caches
  would otherwise outlive the write.
- Bench (1M): cached page 8 ms (was 694 / 363 ms for scenarios 1 / 2);
  after invalidation 8 ms + COUNT 608 / 344 ms, on par with before.
- REST X-WP-Total COUNT is on the frozen Hot zone path (ADR-003): not
  changed.
- Verify: core unit 659 / integration 475, HP 142/49, WC 10/22 green,
  phpcs clean
- PR: PR-030
2026-09-26 01:44:45 +08:00
wpdev 754f904e3b perf(2meet-data-optimizer): route status and date ordering to entity tables
- TMDO_Post_Query_Router: when a group's post_status / post_date copies
  are ready and none is NULL (one index probe per group per request), a
  single-post-type search with explicit statuses and date ordering INNER
  JOINs the flat table, filters the copied status, orders by the copied
  post_date and skips GROUP BY. Otherwise the query keeps its old shape.
- New posts_orderby filter; bench gains an `entity+copy` variant.
- Bench (1M, with SQL_CALC_FOUND_ROWS): scenario 1 p95 3,739 -> 773 ms
  (p50 670 -> 770); scenario 2 p50 356 -> 448 ms. Without found-rows the
  same shape runs in 8 ms (PR-030).
- Verify: core unit 659 / integration 471 green, phpcs clean
- PR: PR-029
2026-09-26 01:41:07 +08:00
wpdev aeca12b41d feat(2meet-data-optimizer): store post_status and post_date copies in post entity tables
- TMDO_Post_Status_Sync: post group tables get post_status / post_date
  columns (posts' collation) and an (post_status, post_date) index.
  Brand-new empty tables get them on creation; existing tables only via
  `wp tmdo post-status-sync [--dry-run] [--batch]`, never as DDL on a
  populated table in a web request. Per-group state in
  tmdo_post_status_columns ('columns' | 'ready').
- transition_post_status refreshes the copies; Hook Bus upserts that
  create a post row fill them in (existing rows keep theirs).
- Bench (1M rows): add columns 0.05 s, index 5.3 s, batched backfill 32 s,
  0 mismatches afterwards.
- Verify: core unit 659 / integration 466 green, phpcs clean
- PR: PR-028
2026-09-26 01:25:20 +08:00
wpdev 65c75dfc8d feat(2meet-data-optimizer): support composite indexes on entity groups
- register_group() accepts $args['indexes'] (name => ordered keys); indexes on
  unknown keys are dropped. Kept outside the field definition, so schema
  hashes are unchanged.
- Schema_Manager::ensure_group_indexes() adds missing cidx_* indexes to the
  existing flat table in place; the applied definition is remembered in
  tmdo_entity_indexes so later requests skip SHOW INDEX.
- Bench (1M listings, scenario 2): Entity p50 337 ms -> 278 ms with
  (hp_verified, hp_price), on par with Hot (269 ms).
- Verify: core unit 656 / integration 459 green, phpcs clean
- PR: PR-027
2026-09-26 01:05:23 +08:00
wpdev d86c0a7bac feat(2meet-data-optimizer): write flat-table data back to native meta
- TMDO_Native_Restorer copies an entity's flat-table values into its native meta table so the value each key reads today is what native returns afterwards: the owning group (by post type for posts) wins, values are stored as WP stores them, NULL removes the native row
- Resumable per-group id cursor, catch-up pass for rows updated during the run, optional demote to dual_write; CLI wp tmdo bridge-restore-native <entity> [--dry-run] [--batch=<n>] [--demote]
- Verify: NativeRestorerTest (5, user) + NativeRestorerPostTest (2, scoped vs legacy routing); core unit 652 / integration 456, HP 148, WC 10 OK; scale estimate dev21 user ~5.1M / post ~4.0M flat rows
- PR: PR-031
2026-09-26 00:44:53 +08:00
wpdev 1eedaa9b3c refactor(2meet-data-optimizer): stop registering HivePress entity groups in core
- Drop hp_listing_core / hp_request_core / hp_vendor_core from TMDO_Post_Fields and hp_user from TMDO_Member_Fields; the HivePress AddOn registers identical definitions (PR-019/020a/033b)
- Tests: core asserts it registers none of them; HP-specific core tests removed (covered by the AddOn); SyncBridgeEntityGuardTest registers its own hp_price group
- Verify: core unit 633 / integration 449, HP 145/49, WC 10/22 OK
- Deploy: activate the HivePress AddOn on dev20/21/22 first
- PR: PR-020b
2026-09-26 00:07:31 +08:00
wpdev e0683a19b1 feat(2meet-data-optimizer): reconcile post rows written to the wrong group table
- TMDO_Post_Group_Reconciler copies non-NULL values that legacy routing stored in another group's table into the post type's own group (NULL/missing legacy values keep the scoped value), resumable via a cursor, with a catch-up pass for rows updated during the run, then switches on tmdo_post_scoped_routing
- CLI: wp tmdo post-reconcile-groups [--dry-run] [--batch=<n>]; registry exposes get_group_post_types()
- Verify: PostGroupReconcilerPlanTest (4), PostGroupReconcilerTest (6); core unit 635 / integration 449, HP 145/49, WC 10/22 OK; read-only dry-run: dev20 320 values / 0 overwrites, dev21/22 none
- PR: PR-034
2026-09-26 00:04:38 +08:00
wpdev 3f4fe52e1b fix(2meet-data-optimizer): route post entity keys by post_type
- Entity groups can declare post_types; get_field() resolves a post's key against its own type's groups first, then unscoped ones (legacy last-wins lookup kept for callers without a post type)
- Hook Bus and the post query router pass the post type, so a listing's hp_rating / hp_status / hp_verified map to hp_listing_core instead of the vendor/request tables; core HP groups declare their post types (schema hashes unchanged)
- Behind tmdo_post_scoped_routing: on for sites without post flat rows, off where rows may sit in the wrong table until PR-034 reconciles them (dry-run: dev20–22 off, dev23/24/30/31/40 on)
- Verify: EntityRegistryScopeTest (8), PostScopedRoutingTest (4); core unit 630 / integration 443, HP 144/49, WC 10/22 OK
- PR: PR-033
2026-09-25 23:55:49 +08:00
wpdev 1653011f4f feat(2meet-data-optimizer): guard orphaned flat-table groups
- On init:1, re-register aeav_only groups whose flat table exists and whose definition is in wpdo_registry_meta but no active plugin registers them, so their data stays readable (routing only, never DDL)
- Keys owned by a live group, or with rows in native meta (newer data written while orphaned), are left alone; the native-meta check is cached for a day
- Surface restored groups via Site Health (wpdo_orphaned_entity_groups), the admin Doctor tab, and wp wpdo doctor
- Verify: EntityRegistryRestoreTest (9), OrphanGroupRestoreTest (7); core unit 622 / integration 439, HP 141/49, WC 10/22 OK; dry-run on dev20–22 restores 16/17/4 groups, keeps hp_taxonomy hp_sort_order/hp_icon on native
- PR: PR-018
2026-09-25 23:38:12 +08:00