= 0.8 overlap rule). * * @param string $text Text. * @return string[] */ public static function wordTokens( string $text ): array { $normalized = self::normalizeForCompare( $text ); if ( '' === $normalized ) { return array(); } $tokens = preg_split( '/\s+/u', $normalized ); return is_array( $tokens ) ? array_values( array_unique( array_filter( $tokens ) ) ) : array(); } /** * Extract number-like tokens ("249.00", "60", "4k", "10m") from $text, as * a normalized (lowercased), deduplicated set. Used by the "unsupported * number" heuristic: a numeric token the model wrote that never appeared * anywhere in the context it was given. * * A unit suffix only ever attaches when it is glued directly to the * digits with no separator ("10cm", "2x", "1080p"), kept in full however * long it runs ("100milligrams" stays "100milligrams", never cut down to * a few letters that a different, same-prefixed unit could also produce * - "100milliliters" must never compare equal to it). Nothing after the * digits across a space, tab or newline ever attaches, on either side of * the comparison this heuristic runs: a number at the end of one field * or sentence can then never swallow the next field's label or the * following word of the same sentence into one fabricated token ("001" * followed by "Categories:" across a newline, or by "best" across a * plain space, both stay "001"). * * This is a labelled heuristic, not a verified-fact checker: it only * compares surface tokens, never units or magnitude. * * @param string $text Text. * @return string[] */ public static function numericTokens( string $text ): array { $plain = self::toPlainText( $text ); if ( 0 === preg_match_all( '/\d+(?:[.,]\d+)*[\p{L}%]*/u', $plain, $matches ) ) { return array(); } $tokens = array(); foreach ( $matches[0] as $raw ) { $normalized = mb_strtolower( $raw, 'UTF-8' ); if ( '' !== $normalized ) { $tokens[] = $normalized; } } return array_values( array_unique( $tokens ) ); } /** * Strip tags and decode HTML entities, leaving plain text. * * @param string $text Text. * @return string */ private static function toPlainText( string $text ): string { $stripped = wp_strip_all_tags( $text ); return html_entity_decode( $stripped, ENT_QUOTES, 'UTF-8' ); } }