13 Commits
Author SHA1 Message Date
bachirandClaude Opus 5.5 54e7e28046 materiotheme: tab title from the search, safe terms of old URLs (F3)
- Base.vue: the browser tab shows the phrase searched (vue-meta), « Base »
  without a search.
- terms of an old or edited URL parsed without throwing: malformed JSON
  is ignored.
- Production build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShDspHaZGwHoRZCsPBJzsM
2026-11-17 21:14:09 +01:00
bachirandClaude Opus 5.5 fca4be3155 Remove the former search form, filters and SlimSelect (F2)
- materio_sapi: MaterioSapiSearchForm, AjaxSearchForm and the
  /materio_sapi/search_form route removed, with the fieldset legend hook
  they needed. The search block is only the place where the Vue field
  mounts (placement and access unchanged).
- materiotheme: filters removed from the search store and Base.vue (the
  parameter of former URLs is ignored), formhtml removed from main.js and
  SearchBlock.vue, filter styles and SlimSelect import removed.
- slim-select uninstalled, with its postinstall patch step.
- Production build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShDspHaZGwHoRZCsPBJzsM
2026-11-17 21:11:18 +01:00
bachirandClaude Opus 5.5 b73057f453 materiotheme: search field as a Vue form, one phrase (F1)
- SearchForm.vue: one search field (150 characters, like the backend) and
  a button; Enter and button submit the same way; route base with keys
  only (no more terms nor filters). Same ids and classes as the former
  Drupal form, so main.scss applies.
- SearchBlock.vue: no more fetching nor Vue.compile of the Drupal form.
- Base.vue: aria-live on the result message, announced by screen readers.
- Production build (npm run prod).

Suggestions in the field come back in F7 (they came from the jQuery UI
autocomplete of the Drupal form).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShDspHaZGwHoRZCsPBJzsM
2026-11-17 20:50:21 +01:00
bachirandClaude Opus 5.5 7786c2cf71 materio_sapi: neighbours cut relative to the search, OR query as fallback (H6)
- ScoreCut: the kNN search scores every visible node; keep the ones above
  the mean of this search by more than 2 standard deviations. A fixed
  threshold meant nothing (at 0.7, « bois » 664 nodes, « wood » 4 671);
  with k = 2, 373 and 382. Exact matches stay first.
- OR query off (OR_WITH_KNN), kept as the fallback when there are no
  neighbours: model stopped, slow or switched off, term chosen in the
  autocomplete.
- S4: 355 results (1 104), same exact matches; S5: 248 (3 338), better
  top results, 0.27 s instead of 1.3 s.
- Message « X exact result(s) out of N relevant result(s) for "…" », French
  text added by materio_sapi_update_11001 (translations live in the DB).
- ScoreCutTest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShDspHaZGwHoRZCsPBJzsM
2026-11-17 19:56:53 +01:00
bachirandClaude Opus 5.5 88ecfb7199 materio_sapi: ranking cached for the next pages (H9)
Each page of 15 results ran every query again (0.7 s for words, 1 s for a
phrase, up to 4 s). Base::rankedSearch() caches the ranking 10 min in the
materio_sapi bin, per user (results depend on node grants), invalidated by
node_list. Pages 2+ now take 1 ms, page 1 unchanged.

The OR query is not limited yet: tied to the tail cut of H6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShDspHaZGwHoRZCsPBJzsM
2026-11-17 19:10:10 +01:00
bachirandClaude Opus 5.5 6d8c64c94f materio_ai: cached search vectors, 1 s delay, pause and cron warm-up (H8)
- QueryEmbedder: vector of a search phrase cached 30 days (cache bin
  materio_ai), model given search_timeout seconds (1 by default, new
  setting, update 11001), left alone 60 s after a failure
  (ModelPausedException, not logged).
- NeighbourSearch honours the enabled setting: the search called the model
  even when materio_ai was switched off.
- materio_ai_cron() loads the model with the normal timeout: after a
  restart, loading (2.9 s) exceeds the search delay and Ollama drops it,
  so searches alone would never load it. Ultimate cron job exported.
- On any model failure the search answers with the keyword results only.
- QueryEmbedderTest: cache, delay, pause, cache during the pause, warm-up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShDspHaZGwHoRZCsPBJzsM
2026-11-17 18:26:19 +01:00
bachirandClaude Opus 5.5 a656427aec materio_sapi: RankFusionTest as a Drupal UnitTestCase with PHPUnit attributes (T1)
Runs with vendor/bin/phpunit -c web/core: 6 tests, 8 assertions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShDspHaZGwHoRZCsPBJzsM
2026-11-17 12:57:42 +01:00
bachirandClaude Opus 5.5 26a623b4f6 drupal/core-dev ^11.4 as dev dependency: PHPUnit, phpstan, coder (T1)
Installed with -W -m: 76 dev packages and 4 production packages changed,
symfony/dom-crawler 8.0.12 -> 7.4.22 (core-dev 11.4 needs 7.4; used by
wa72/htmlpagedom for printable), sebastian/diff 7.0.1 -> 6.0.2 (core config
diff), symfony/filesystem and symfony/yaml 7.4.18 -> 7.4.23. Printable HTML
and PDF, config diff and search identical before and after. deploy.sh runs
composer install --no-dev: no dev package on the server. allow-plugins:
phpstan/extension-installer and phpcodesniffer-composer-installer allowed,
php-http/discovery and tbachert/spi denied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShDspHaZGwHoRZCsPBJzsM
2026-11-17 12:57:42 +01:00
bachirandClaude Opus 5.5 797a5714f1 materio_sapi: hybrid search, keyword and kNN results merged by RRF (H5)
RankFusion::fuse(): exact matches (references, phrase, all words) first, in
their order; then the OR query and the kNN neighbours merged by reciprocal
rank fusion (k = 60, ranks only). Base::neighbours() asks materio_ai for 200
neighbours of the phrase without references; no model call for a chosen
term, references alone, a missing service, or an error (logged, keyword
results only). Unit test RankFusionTest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShDspHaZGwHoRZCsPBJzsM
2026-11-17 12:43:04 +01:00
bachirandClaude Opus 5.5 8229a80c57 materio_sapi: references as exact matches, any case (H4)
referenceQuery() runs first: nodes whose field_reference is one of the
references of the phrase, uppercased (the field is case-sensitive: w0117 and
t0122 found nothing, B2), first and counted as exact. The reference condition
is removed from the OR query, where it restricted the results to the
referenced nodes (« bois R0208 » found nothing). References are removed from
the keywords; references alone run a single query (isReferenceSearch()).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShDspHaZGwHoRZCsPBJzsM
2026-11-17 12:17:09 +01:00
bachirandClaude Opus 5.5 58ce1b0513 materio_sapi: OR query in terms mode, Solr syntax neutralized (H3)
The OR query no longer sends the input raw (direct mode): « fibre de verre \ »
broke Solr, « -bois » returned every other node as exact, « verre~ » ran a
fuzzy search. KeywordText turns syntax between words into a separator, drops
leading + - ! and symbol-only words (« || » broke the payload boost). Hyphens
are split for the OR query only, as the direct mode did. Same counts and exact
matches on S4, about twice as fast.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShDspHaZGwHoRZCsPBJzsM
2026-11-17 12:10:00 +01:00
bachirandClaude Opus 5.5 ae69b5c03b materio_sapi: split sapiQuery, stop words removed, short-input exact queries (H2)
phraseQuery, andQuery and orQuery return [nid, uuid] lists built by
keywordQuery (duplicates removed by key, no more in_array over 10 000 rows);
filters code removed. KeywordText strips the stop words of the Solr field
type of the language, French elisions and request words: the custom types of
materio_sapi keep stop words (« de » alone matched 2 236 nodes). The phrase
and AND queries run for 6 meaningful words at most: beyond they find nothing
and the AND query could exceed maxClauseCount.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShDspHaZGwHoRZCsPBJzsM
2026-11-17 11:53:13 +01:00
bachirandClaude Opus 5.5 7db02c8b30 materio_sapi: search keys kept as one phrase, filters ignored (H1)
parseRequest() no longer splits the input on commas (Tags::explode):
Base::cleanPhrase() collapses spaces and cuts at 150 characters. The phrase
is kept with its case for the model, lowercased in keys for the keyword
queries. The filters parameter of the assisted search is ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShDspHaZGwHoRZCsPBJzsM
2026-11-17 11:30:32 +01:00
53 changed files with 6250 additions and 1042 deletions
+8 -1
View File
@@ -182,7 +182,11 @@
"wikimedia/composer-merge-plugin": true,
"zaporylie/composer-drupal-optimizations": true,
"drupal/core-project-message": true,
"symfony/runtime": true
"symfony/runtime": true,
"phpstan/extension-installer": true,
"dealerdirect/phpcodesniffer-composer-installer": true,
"php-http/discovery": false,
"tbachert/spi": false
}
},
"autoload": {
@@ -319,5 +323,8 @@
".gitattributes": "../.gitattributes"
}
}
},
"require-dev": {
"drupal/core-dev": "^11.4"
}
}
Generated
+5057 -46
View File
File diff suppressed because it is too large Load Diff
+1
View File
@@ -5,4 +5,5 @@ url: 'http://ollama:11434'
model: 'bge-m3:567m'
dimension: 1024
timeout: 10
search_timeout: 1.0
num_thread: 4
@@ -0,0 +1,17 @@
uuid: 77f99dd9-eeab-40af-9925-95a9f7cd6cee
langcode: en
status: true
dependencies:
module:
- materio_ai
title: 'Default cron handler'
id: materio_ai_cron
weight: 0
module: materio_ai
callback: 'materio_ai#cron'
scheduler:
id: simple
launcher:
id: serial
logger:
id: database
-9
View File
@@ -7,7 +7,6 @@
"": {
"name": "materio.com",
"version": "1.1.0",
"hasInstallScript": true,
"license": "MIT",
"dependencies": {
"axios": "^0.21.0",
@@ -59,7 +58,6 @@
"mini-css-extract-plugin": "^1.3.1",
"querystring-es3": "^0.2.1",
"sass-loader": "^10.1.0",
"slim-select": "^2.11.0",
"style-loader": "^2.0.0",
"uglify-es": "^3.3.9",
"url-loader": "^4.1.1",
@@ -11061,13 +11059,6 @@
"url": "https://github.com/chalk/slice-ansi?sponsor=1"
}
},
"node_modules/slim-select": {
"version": "2.11.0",
"resolved": "https://registry.npmjs.org/slim-select/-/slim-select-2.11.0.tgz",
"integrity": "sha512-Y0FEJKIPBy8ax7jArUxICxtKAnhX8GH7ydaFhcrYH7OznfpvxCe8uU2etyuRPKhE23c2LhT2GROFvZzGVplA0g==",
"dev": true,
"license": "MIT"
},
"node_modules/snapdragon": {
"version": "0.8.2",
"resolved": "https://registry.npmjs.org/snapdragon/-/snapdragon-0.8.2.tgz",
+1 -3
View File
@@ -9,8 +9,7 @@
"lint": "eslint --ext .js,.vue src",
"lint:fix": "eslint --ext .js,.vue src --fix",
"test": "jest --config test/jest.config.js",
"test:debug": "node --inspect node_modules/.bin/jest --runInBand --config test/jest.config.js",
"postinstall": "patch --forward node_modules/slim-select/package.json < patches/slimselect-export-css.patch"
"test:debug": "node --inspect node_modules/.bin/jest --runInBand --config test/jest.config.js"
},
"keywords": [
"Vue",
@@ -68,7 +67,6 @@
"mini-css-extract-plugin": "^1.3.1",
"querystring-es3": "^0.2.1",
"sass-loader": "^10.1.0",
"slim-select": "^2.11.0",
"style-loader": "^2.0.0",
"uglify-es": "^3.3.9",
"url-loader": "^4.1.1",
@@ -7,6 +7,9 @@ model: 'bge-m3:567m'
dimension: 1024
# Délai maximal d'un appel au modèle, en secondes.
timeout: 10
# Délai maximal du calcul du vecteur d'une recherche, en secondes : au-delà, la
# recherche répond avec les seuls mots-clés (H8).
search_timeout: 1.0
# Threads de calcul envoyés à chaque appel (options.num_thread) : Ollama ne voit pas
# la limite de CPU du conteneur, sans ce réglage il lance trop de threads (mesure M2).
num_thread: 4
@@ -17,6 +17,9 @@ materio_ai.settings:
timeout:
type: integer
label: 'Timeout (seconds)'
search_timeout:
type: float
label: 'Search timeout (seconds)'
num_thread:
type: integer
label: 'Threads per call'
@@ -58,3 +58,13 @@ function materio_ai_schema(): array {
];
return $schema;
}
/**
* Adds the search timeout setting (H8).
*/
function materio_ai_update_11001(): void {
$config = \Drupal::configFactory()->getEditable('materio_ai.settings');
if ($config->get('search_timeout') === NULL) {
$config->set('search_timeout', 1.0)->save(TRUE);
}
}
@@ -7,6 +7,7 @@
declare(strict_types=1);
use Drupal\materio_ai\Exception\EmbeddingException;
use Drupal\node\NodeInterface;
use Drupal\taxonomy\TermInterface;
@@ -66,3 +67,22 @@ function materio_ai_taxonomy_term_translation_delete(TermInterface $translation)
function materio_ai_taxonomy_term_predelete(TermInterface $term): void {
\Drupal::service('materio_ai.vectorizer')->enqueueForTerm($term, TRUE);
}
/**
* Implements hook_cron().
*
* Loads the model: a search gives it 1 s, too short to load it after a
* restart of Ollama (H8).
*/
function materio_ai_cron(): void {
$embedder = \Drupal::service('materio_ai.query_embedder');
if (!$embedder->isEnabled()) {
return;
}
try {
$embedder->warmUp();
}
catch (EmbeddingException $e) {
\Drupal::logger('materio_ai')->warning('Model warm-up failed: @message', ['@message' => $e->getMessage()]);
}
}
@@ -35,9 +35,21 @@ services:
tags:
- { name: event_subscriber }
cache.materio_ai:
class: Drupal\Core\Cache\CacheBackendInterface
tags:
- { name: cache.bin }
factory: ['@cache_factory', 'get']
arguments: [materio_ai]
materio_ai.query_embedder:
class: Drupal\materio_ai\Search\QueryEmbedder
arguments: ['@materio_ai.embedding_client', '@cache.materio_ai', '@config.factory', '@datetime.time']
Drupal\materio_ai\Search\QueryEmbedder: '@materio_ai.query_embedder'
materio_ai.neighbour_search:
class: Drupal\materio_ai\Search\NeighbourSearch
arguments: ['@entity_type.manager', '@language_manager', '@config.factory', '@materio_ai.embedding_client']
arguments: ['@entity_type.manager', '@language_manager', '@config.factory', '@materio_ai.query_embedder']
Drupal\materio_ai\Search\NeighbourSearch: '@materio_ai.neighbour_search'
materio_ai.knn_query_subscriber:
@@ -12,12 +12,18 @@ interface EmbeddingClientInterface {
/**
* Returns the vector of one text.
*
* @param string $text
* The text to vectorize.
* @param float|null $timeout
* Longest wait in seconds; the configured timeout by default. A search
* passes a short one (search_timeout, H8).
*
* @return float[]
* A vector of getDimension() values.
*
* @throws \Drupal\materio_ai\Exception\EmbeddingException
*/
public function embed(string $text): array;
public function embed(string $text, ?float $timeout = NULL): array;
/**
* Returns the vectors of several texts, in the same order and with the same keys.
@@ -23,13 +23,13 @@ final class OllamaEmbeddingClient implements EmbeddingClientInterface {
/**
* {@inheritdoc}
*/
public function embed(string $text): array {
public function embed(string $text, ?float $timeout = NULL): array {
$text = trim($text);
if ($text === '') {
throw new EmbeddingException('Cannot vectorize an empty text.');
}
$config = $this->config();
$timeout = max(1, (int) $config->get('timeout'));
$timeout ??= (float) max(1, (int) $config->get('timeout'));
$url = rtrim((string) $config->get('url'), '/') . '/api/embed';
try {
@@ -42,7 +42,7 @@ final class OllamaEmbeddingClient implements EmbeddingClientInterface {
'options' => ['num_thread' => max(1, (int) $config->get('num_thread'))],
],
'timeout' => $timeout,
'connect_timeout' => min($timeout, 3),
'connect_timeout' => min($timeout, 3.0),
]);
}
catch (GuzzleException $e) {
@@ -0,0 +1,12 @@
<?php
declare(strict_types=1);
namespace Drupal\materio_ai\Exception;
/**
* The model is not called: it failed less than QueryEmbedder::PAUSE seconds ago.
*
* The failure itself was already logged; this one is not worth a log line.
*/
final class ModelPausedException extends EmbeddingException {}
@@ -63,6 +63,15 @@ final class SettingsForm extends ConfigFormBase {
'#required' => TRUE,
'#config_target' => 'materio_ai.settings:timeout',
];
$form['search_timeout'] = [
'#type' => 'number',
'#title' => $this->t('Search timeout (seconds)'),
'#description' => $this->t('Longest wait for the vector of a search. Past it, the search answers with keywords only.'),
'#min' => 0.1,
'#step' => 0.1,
'#required' => TRUE,
'#config_target' => 'materio_ai.settings:search_timeout',
];
$form['num_thread'] = [
'#type' => 'number',
'#title' => $this->t('Threads per call'),
@@ -7,7 +7,6 @@ namespace Drupal\materio_ai\Search;
use Drupal\Core\Config\ConfigFactoryInterface;
use Drupal\Core\Entity\EntityTypeManagerInterface;
use Drupal\Core\Language\LanguageManagerInterface;
use Drupal\materio_ai\Embedding\EmbeddingClientInterface;
use Drupal\search_api\Utility\Utility;
/**
@@ -41,7 +40,7 @@ final class NeighbourSearch {
private readonly EntityTypeManagerInterface $entityTypeManager,
private readonly LanguageManagerInterface $languageManager,
private readonly ConfigFactoryInterface $configFactory,
private readonly EmbeddingClientInterface $client,
private readonly QueryEmbedder $embedder,
) {}
/**
@@ -55,16 +54,21 @@ final class NeighbourSearch {
* Number of neighbours (topK).
*
* @return array<int, float>
* Scores keyed by nid, best first. Solr score of a cosine kNN query:
* Scores keyed by nid, best first, none when materio_ai is switched off.
* Solr score of a cosine kNN query:
* (1 + cosine) / 2.
*
* @throws \Drupal\materio_ai\Exception\EmbeddingException
* When the model fails.
* When the model fails or is too slow (QueryEmbedder).
* @throws \Drupal\search_api\SearchApiException
* When Solr fails.
*/
public function search(string $phrase, ?string $langcode = NULL, int $limit = 100): array {
return $this->searchVector($this->client->embed($phrase), $langcode, $limit);
// Switched off: no text is sent to the model, keyword results only.
if (!$this->embedder->isEnabled()) {
return [];
}
return $this->searchVector($this->embedder->embed($phrase), $langcode, $limit);
}
/**
@@ -0,0 +1,105 @@
<?php
declare(strict_types=1);
namespace Drupal\materio_ai\Search;
use Drupal\Component\Datetime\TimeInterface;
use Drupal\Core\Cache\CacheBackendInterface;
use Drupal\Core\Config\ConfigFactoryInterface;
use Drupal\materio_ai\Embedding\EmbeddingClientInterface;
use Drupal\materio_ai\Exception\EmbeddingException;
use Drupal\materio_ai\Exception\ModelPausedException;
/**
* Vector of a search phrase: cached, with a short delay and a pause (H8).
*
* A search must answer even when the model is slow or stopped:
* - vectors are cached by model and phrase (cache bin materio_ai, Redis on
* the servers); they hold nothing about the user, so one entry serves
* everybody;
* - the model gets search_timeout seconds (1 by default);
* - after a failure the model is left alone for PAUSE seconds: the searches
* of that time do not each wait for the delay again.
*
* On failure an EmbeddingException is thrown, and the caller answers with
* the keyword results.
*
* A model that took longer than the delay to load is unloaded again by
* Ollama when the call is dropped: searches alone would never load it after
* a restart (2.9 s measured locally). warmUp(), run by cron with the normal
* timeout, loads it.
*/
final class QueryEmbedder {
/**
* Lifetime of a cached vector, in seconds (30 days).
*/
public const LIFETIME = 2592000;
/**
* Seconds without calling the model after a failure.
*/
public const PAUSE = 60;
public function __construct(
private readonly EmbeddingClientInterface $client,
private readonly CacheBackendInterface $cache,
private readonly ConfigFactoryInterface $configFactory,
private readonly TimeInterface $time,
) {}
/**
* Whether the model may be called (setting « enabled »).
*/
public function isEnabled(): bool {
return (bool) $this->configFactory->get('materio_ai.settings')->get('enabled');
}
/**
* Loads the model with the normal timeout, and ends the pause.
*
* Uses the client options of the searches (num_thread): with other options
* Ollama would load the model again.
*
* @throws \Drupal\materio_ai\Exception\EmbeddingException
* When the model fails.
*/
public function warmUp(): void {
$this->client->embed('materiO');
$this->cache->delete('pause:' . $this->client->getModel());
}
/**
* Returns the vector of a phrase.
*
* @return float[]
* The vector.
*
* @throws \Drupal\materio_ai\Exception\EmbeddingException
* When the model fails or is too slow; ModelPausedException when it
* failed less than PAUSE seconds ago.
*/
public function embed(string $phrase): array {
$model = $this->client->getModel();
$cid = 'query:' . $model . ':' . hash('sha256', $phrase);
if ($cached = $this->cache->get($cid)) {
return $cached->data;
}
$pause_cid = 'pause:' . $model;
if ($this->cache->get($pause_cid)) {
throw new ModelPausedException(sprintf('Model %s paused after a failure.', $model));
}
$timeout = (float) ($this->configFactory->get('materio_ai.settings')->get('search_timeout') ?? 1.0);
try {
$vector = $this->client->embed($phrase, max(0.1, $timeout));
}
catch (EmbeddingException $e) {
$this->cache->set($pause_cid, TRUE, $this->time->getRequestTime() + self::PAUSE);
throw $e;
}
$this->cache->set($cid, $vector, $this->time->getRequestTime() + self::LIFETIME);
return $vector;
}
}
@@ -0,0 +1,159 @@
<?php
declare(strict_types=1);
namespace Drupal\Tests\materio_ai\Unit;
use Drupal\Component\Datetime\TimeInterface;
use Drupal\Core\Cache\MemoryBackend;
use Drupal\materio_ai\Embedding\EmbeddingClientInterface;
use Drupal\materio_ai\Exception\EmbeddingException;
use Drupal\materio_ai\Exception\ModelPausedException;
use Drupal\materio_ai\Search\QueryEmbedder;
use Drupal\Tests\UnitTestCase;
use PHPUnit\Framework\Attributes\CoversClass;
use PHPUnit\Framework\Attributes\Group;
/**
* Tests the cache, delay and pause of the search vectors (H8).
*/
#[CoversClass(QueryEmbedder::class)]
#[Group('materio_ai')]
final class QueryEmbedderTest extends UnitTestCase {
/**
* Fake model: counts the calls, fails on demand, records the timeouts.
*/
private object $client;
/**
* Request time, moved forward by the tests.
*/
private int $now = 1_000_000;
private MemoryBackend $cache;
private QueryEmbedder $embedder;
/**
* {@inheritdoc}
*/
protected function setUp(): void {
parent::setUp();
$this->client = new class implements EmbeddingClientInterface {
public int $calls = 0;
public bool $fail = FALSE;
/**
* @var array<float|null>
*/
public array $timeouts = [];
public function embed(string $text, ?float $timeout = NULL): array {
$this->calls++;
$this->timeouts[] = $timeout;
if ($this->fail) {
throw new EmbeddingException('cURL error 28: Operation timed out');
}
return [(float) strlen($text), 1.0];
}
public function embedMultiple(array $texts): array {
return array_map($this->embed(...), $texts);
}
public function getModel(): string {
return 'bge-m3:567m';
}
public function getDimension(): int {
return 2;
}
};
$time = $this->createStub(TimeInterface::class);
$time->method('getRequestTime')->willReturnCallback(fn () => $this->now);
$time->method('getCurrentTime')->willReturnCallback(fn () => $this->now);
$this->cache = new MemoryBackend($time);
$config = $this->getConfigFactoryStub(['materio_ai.settings' => ['enabled' => TRUE, 'search_timeout' => 1.0]]);
$this->embedder = new QueryEmbedder($this->client, $this->cache, $config, $time);
}
/**
* The second search of a phrase does not call the model.
*/
public function testCachedVector(): void {
$first = $this->embedder->embed('bois brûlé');
$this->assertSame($first, $this->embedder->embed('bois brûlé'));
$this->assertSame(1, $this->client->calls);
$this->embedder->embed('bois flotté');
$this->assertSame(2, $this->client->calls);
}
/**
* A search gives the model search_timeout seconds, not the normal timeout.
*/
public function testSearchTimeout(): void {
$this->embedder->embed('bois');
$this->assertSame([1.0], $this->client->timeouts);
}
/**
* After a failure the model is not called for PAUSE seconds.
*/
public function testPauseAfterFailure(): void {
$this->client->fail = TRUE;
try {
$this->embedder->embed('bois');
$this->fail('The failure is passed on.');
}
catch (EmbeddingException $e) {
$this->assertNotInstanceOf(ModelPausedException::class, $e);
}
$this->client->fail = FALSE;
$this->now += QueryEmbedder::PAUSE - 1;
try {
$this->embedder->embed('verre');
$this->fail('The model is paused.');
}
catch (ModelPausedException) {
}
$this->assertSame(1, $this->client->calls);
$this->now += 2;
$this->assertNotEmpty($this->embedder->embed('verre'));
$this->assertSame(2, $this->client->calls);
}
/**
* A cached vector is served even during the pause.
*/
public function testCacheDuringPause(): void {
$vector = $this->embedder->embed('bois');
$this->client->fail = TRUE;
try {
$this->embedder->embed('verre');
}
catch (EmbeddingException) {
}
$this->assertSame($vector, $this->embedder->embed('bois'));
}
/**
* The cron warm-up uses the normal timeout and ends the pause.
*/
public function testWarmUpEndsPause(): void {
$this->client->fail = TRUE;
try {
$this->embedder->embed('bois');
}
catch (EmbeddingException) {
}
$this->client->fail = FALSE;
$this->embedder->warmUp();
$this->assertNull(end($this->client->timeouts));
$this->assertNotEmpty($this->embedder->embed('bois'));
}
}
@@ -0,0 +1,44 @@
<?php
/**
* @file
* Install, update and uninstall functions for materio_sapi.
*/
declare(strict_types=1);
/**
* Source of the search message (H6) and its French translation.
*
* Interface translations live in the database, not in the repository: the
* update brings the French text to every site.
*/
function materio_sapi_search_infos_strings(): array {
return [
'source' => '@exactmatchcount exact result(s) out of @count relevant result(s) for “@keys”',
'fr' => '@exactmatchcount résultat(s) exact(s) sur @count résultat(s) pertinent(s) pour « @keys »',
];
}
/**
* Adds the French translation of the new search message (H6).
*/
function materio_sapi_update_11001(): string {
if (!\Drupal::moduleHandler()->moduleExists('locale')) {
return 'locale is off: nothing to translate.';
}
$strings = materio_sapi_search_infos_strings();
$storage = \Drupal::service('locale.storage');
$source = $storage->findString(['source' => $strings['source'], 'context' => '']);
if (!$source) {
$source = $storage->createString(['source' => $strings['source'], 'context' => ''])->save();
}
$storage->createTranslation([
'lid' => $source->lid,
'language' => 'fr',
'translation' => $strings['fr'],
'customized' => LOCALE_CUSTOMIZED,
])->save();
_locale_refresh_translations(['fr'], [$source->lid]);
return 'French translation of the search message added.';
}
@@ -8,7 +8,6 @@
use Drupal\Core\Routing\RouteMatchInterface;
use Solarium\QueryType\Select\Query\Query;
use Drupal\search_api\Query\QueryInterface;
use Drupal\Core\Template\Attribute;
/**
* Implements hook_help().
@@ -51,24 +50,6 @@ function materio_sapi_search_api_solr_query_alter(Query $solarium_query, QueryI
}
/**
* Prepares variables for fieldset element templates.
*
* Default template: fieldset.html.twig.
*
* @param array $variables
* An associative array containing:
* - element: An associative array containing the properties of the element.
* Properties used: #attributes, #children, #description, #id, #title,
* #value.
*/
function materio_sapi_preprocess_fieldset(&$variables) {
if (isset($variables['element']['#legend_attributes'])) {
$variables['legend']['attributes'] = new Attribute($variables['element']['#legend_attributes']);
}
}
// ? https://happyculture.coop/blog/drupal-8-declarer-un-champ-extrafield-calcule-computed-field
// ? https://fivejars.com/blog/entity-basefielddefinitions-fields-examples-drupal-8
/**
@@ -6,22 +6,6 @@ materio_sapi.search_autocomplete:
requirements:
_permission: 'access materio search'
materio_sapi.search_form:
path: '/materio_sapi/search_form'
defaults:
_controller: '\Drupal\materio_sapi\Controller\AjaxSearchForm::getForm'
_format: json
requirements:
_permission: 'access materio search'
# materio_sapi.materio_sapi_search_form:
# path: '/materio_sapi/form/materio_sapi_search'
# defaults:
# _form: '\Drupal\materio_sapi\Form\MaterioSapiSearchForm'
# _title: 'MaterioSapiSearchForm'
# requirements:
# _access: 'TRUE'
materio_sapi.getresults:
path: '/materio_sapi/getresults'
defaults:
@@ -0,0 +1,8 @@
services:
# Rankings of the searches, reused by the next pages (H9).
cache.materio_sapi:
class: Drupal\Core\Cache\CacheBackendInterface
tags:
- { name: cache.bin }
factory: ['@cache_factory', 'get']
arguments: [materio_sapi]
@@ -1,92 +0,0 @@
<?php
// https://www.qed42.com/blog/autocomplete-drupal-8
// https://www.drupal.org/docs/8/modules/search-api/developer-documentation/executing-a-search-in-code
namespace Drupal\materio_sapi\Controller;
use Drupal\Core\Controller\ControllerBase;
use Symfony\Component\DependencyInjection\ContainerInterface;
use Drupal\Core\Form\FormBuilder;
use Drupal\Core\Form\FormState;
use Symfony\Component\HttpFoundation\JsonResponse;
use Symfony\Component\HttpFoundation\Request;
use Drupal\Core\Cache\CacheableJsonResponse;
use Drupal\Core\Cache\CacheableMetadata;
use Drupal\core\render\RenderContext;
/**
* Defines a route controller for entity autocomplete form elements.
*/
class AjaxSearchForm extends ControllerBase {
/**
* The built form array.
*
* @var array
*/
protected $form_builded;
/**
* The form builder.
*
* @var \Drupal\Core\Form\FormBuilder
*/
protected $formBuilder;
/**
* Construct.
*/
public function __construct(FormBuilder $formBuilder) {
$this->formBuilder = $formBuilder;
}
/**
* {@inheritdoc}
*/
public static function create(ContainerInterface $container) {
return new static(
$container->get('form_builder')
);
}
private function getFormDefinition(){
$language = \Drupal::languageManager()->getCurrentLanguage()->getId();
\Drupal::logger('materio_sapi')->notice($language);
// $form_state = new FormState();
$this->form_builded = $this->formBuilder->getForm('Drupal\materio_sapi\Form\MaterioSapiSearchForm', $language);
}
/**
* Handler for getform request.
*/
public function getForm(Request $request) {
$this->getFormDefinition();
// $rendered = render($this->form_builded);
$rendered = \Drupal::service('renderer')->render($this->form_builded);
// $form_builded = $this->form_builded;
// $rendered = \Drupal::service('renderer')->executeInRenderContext(new RenderContext(), function () use ($form_builded) {
// return render($form_builded);
// });
$data = [
'rendered' => $rendered,
// '#cache' => [
// 'max-age' => \Drupal\Core\Cache\Cache::PERMANENT,
// 'tags' => [
// 'materio_sapi-search_form-cache',
// ]
// ]
];
$response = new JsonResponse();
$response->setData($data);
// $response = new CacheableJsonResponse($data);
// $response->addCacheableDependency(CacheableMetadata::createFromRenderArray($data));
return $response;
}
}
@@ -6,8 +6,11 @@ namespace Drupal\materio_sapi\Controller;
use Drupal\Core\Controller\ControllerBase;
use Symfony\Component\HttpFoundation\JsonResponse;
use Symfony\Component\HttpFoundation\Request;
use Drupal\Component\Utility\Tags;
use Drupal\Component\Utility\Unicode;
use Drupal\materio_ai\Exception\ModelPausedException;
use Drupal\materio_sapi\KeywordText;
use Drupal\materio_sapi\RankFusion;
use Drupal\materio_sapi\ScoreCut;
use Drupal\search_api\Entity\Index;
// https://drupal.stackexchange.com/questions/225008/programatically-use-search-api
@@ -53,153 +56,279 @@ class Base extends ControllerBase {
protected $phrase_query, $and_query, $or_query, $query, $count_query, $count, $exactematch_count;
/**
* Filter and term values.
* Term values (autocomplete choice) and items.
*
* @var mixed
*/
protected $filters, $terms, $items;
protected $terms, $items;
private $limit = 15;
private $offset = 0;
/**
* Longest phrase kept, in characters (H1, 150 decided on 17/11). The longest
* test phrase (S5) has 89 characters; beyond the limit the phrase is cut,
* not refused.
*/
public const MAX_PHRASE_LENGTH = 150;
/**
* The phrase typed by the user, whole and case kept, for the model (H5).
*
* @var string
*/
protected $phrase = '';
/**
* Material references found in the phrase (W0117…), uppercase (H4).
*
* @var string[]
*/
protected $references = [];
/**
* Pattern of a material reference: family letter and four digits.
*/
public const REFERENCE_PATTERN = '/\\b[WTRPCMFGSO]\\d{4}\\b/iu';
/**
* Neighbours asked to Solr (kNN topK): every visible node (8 073 for an
* anonymous visitor in 2026), so that ScoreCut sees the whole distribution
* of the scores (H6). 0.3 to 0.6 s measured locally.
*/
public const KNN_LIMIT = 20000;
/**
* Standard deviations above the mean kept by ScoreCut (H6, option f, k = 2
* on trial since 17/11).
*/
public const KNN_SIGMA = 2.0;
/**
* Whether the OR query is merged with the neighbours (H3, H5).
*
* Off on trial since 17/11 (H6): on a phrase it matched about 40 % of the
* catalogue. Kept: it still runs when there are no neighbours (model
* stopped, slow or switched off, term chosen in the autocomplete), so that
* the search answers as before.
*/
public const OR_WITH_KNN = FALSE;
/**
* Number of neighbours found, for the response tuning of H6.
*
* @var int
*/
protected $knn_count = 0;
/**
* The reference query, run before the others.
*
* @var \Drupal\search_api\Query\QueryInterface|null
*/
protected $reference_query;
/**
* Most meaningful words for which the phrase and AND queries run (H2, a).
*
* Beyond, they find nothing (no node holds every word of a long phrase) and
* the AND query may exceed maxClauseCount: only the OR query runs.
*/
public const SHORT_QUERY_WORDS = 6;
/**
* Lifetime of a cached ranking, in seconds (H9).
*/
public const RANKING_LIFETIME = 600;
/**
* Ranking of the search, computed once for all its pages (H9).
*
* Each page used to run every query again (0.7 s for words, 1 s for a
* phrase, up to 4 s, page 2 as slow as page 1). The ranking is cached per
* user: results depend on the node grants, so two accounts never share an
* entry; anonymous visitors share theirs. Dropped after RANKING_LIFETIME
* or when a node changes (node_list).
*/
private function rankedSearch(string $lang): void {
$cid = implode(':', [
'ranking',
$lang,
$this->currentUser()->id(),
hash('sha256', json_encode([$this->phrase, $this->terms])),
]);
$cache = \Drupal::cache('materio_sapi');
if ($cached = $cache->get($cid)) {
[$this->results, $this->exactematch_count, $this->knn_count] = $cached->data;
return;
}
$this->sapiQuery();
$cache->set($cid, [$this->results, $this->exactematch_count, $this->knn_count], \Drupal::time()->getRequestTime() + self::RANKING_LIFETIME, ['node_list']);
}
/**
* Runs the keyword queries and merges their results, exact matches first.
*
* The keys are the phrase without stop words (H2, b): the custom types of
* materio_sapi keep them, « de » alone matched 2 236 nodes.
*/
private function sapiQuery(){
// https://www.drupal.org/docs/8/modules/search-api/developer-documentation/executing-a-search-in-code
// https://www.hashbangcode.com/article/drupal-8-date-search-boosting-search-api-and-solr-search
// https://kgaut.net/blog/2018/drupal-8-search-api-effectuer-une-requete-dans-le-code.html
// Otherwise, if you use Solr, you can also set the parse mode to “Direct query”
// and use something like 'tm_name:alex AND tm_surname:john' as the keywords (field names may vary).
// That will also ensure scoring works for the keywords (the other way just adds filters, which don’t affect scoring).
// https://www.drupal.org/project/search_api/issues/3049097
$lang = \Drupal::languageManager()->getCurrentLanguage()->getId();
$this->index = Index::load('database');
// References first, as exact matches, whatever their case (H4, B2).
preg_match_all(self::REFERENCE_PATTERN, $this->phrase, $matches);
$this->references = array_values(array_unique(array_map('mb_strtoupper', $matches[0])));
$exact = $this->references ? $this->referenceQuery($this->references, $lang) : [];
if ($this->isReferenceSearch()) {
$this->exactematch_count = count($exact);
$this->results = ['uuids' => array_column($exact, 1), 'nids' => array_column($exact, 0)];
return;
}
// Keys without the references (« bois R0208 » searches « bois »).
$words = KeywordText::words(preg_replace(self::REFERENCE_PATTERN, ' ', $this->phrase) ?? $this->phrase, $lang);
// Only stop words: search the phrase as typed, as before.
$keys = $words ? implode(' ', $words) : implode(' ', $this->keys);
if (count($words) <= self::SHORT_QUERY_WORDS) {
// Exact material names (like "wood-skin"), then all words.
$exact += $this->phraseQuery($keys, $lang);
$exact += $this->andQuery($keys, $lang);
}
$this->exactematch_count = count($exact);
// Exact matches first, then the neighbours that stand out (H6), merged
// with the OR query when it is on (H5). No neighbours: the OR query alone.
$neighbours = $this->neighbours($lang);
$or = ($neighbours === NULL || self::OR_WITH_KNN) ? $this->orQuery($keys, $lang) : [];
$neighbours = $neighbours === NULL ? [] : ScoreCut::aboveMean($neighbours, self::KNN_SIGMA);
$this->knn_count = count($neighbours);
$nids = RankFusion::fuse(array_keys($exact), array_filter([array_keys($or), array_keys($neighbours)]));
$uuids = array_column($exact + $or, 1, 0) + $this->uuids(array_diff(array_keys($neighbours), array_keys($exact + $or)));
$this->results = [
'uuids' => [],
'nids' => []
'uuids' => array_map(static fn ($nid) => $uuids[$nid], $nids),
'nids' => $nids,
];
// ,---.| ,---.
// |---'|---.,---.,---.,---.,---. | |. .,---.,---., .
// | | || ,---|`---.|---' | || ||---'| | |
// ` ` '` `---^`---'`---' `---\`---'`---'` `---|
// `---'
// (to match exact materials names (like "wood-skin"))
$this->phrase_query = $this->index->query(['offset'=>0,'limit'=>10000]);
// set parse mode and conjunction
$parse_mode = \Drupal::service('plugin.manager.search_api.parse_mode')
->createInstance('phrase');
$parse_mode->setConjunction('AND');
$this->phrase_query->setParseMode($parse_mode);
// Set fulltext search keywords and fields.
if ($this->keys) {
$this->phrase_query->keys(implode(' ', $this->keys));
}
/**
* Nodes closest in meaning to the phrase (materio_ai, Solr kNN).
*
* Not for a term chosen in the autocomplete (direct search) nor without
* the materio_ai service. On a model or Solr error, past the delay of
* QueryEmbedder (1 s), or with materio_ai switched off, the keyword
* results stand alone, with no message: the search answers as before H5.
*
* @return array<int, float>|null
* Scores of every visible node keyed by nid, best first; NULL when there
* are no neighbours to use.
*/
private function neighbours(string $lang): ?array {
if ($this->terms || !\Drupal::hasService('materio_ai.neighbour_search')) {
return NULL;
}
// References are keywords: they mean nothing to the model.
$phrase = trim(preg_replace(self::REFERENCE_PATTERN, ' ', $this->phrase) ?? '');
if ($phrase === '') {
return NULL;
}
try {
// Empty when materio_ai is switched off.
return \Drupal::service('materio_ai.neighbour_search')->search($phrase, $lang, self::KNN_LIMIT) ?: NULL;
}
catch (\Throwable $e) {
// Paused model: its failure was logged when it happened.
if (!$e instanceof ModelPausedException) {
\Drupal::logger('materio_sapi')->warning('Neighbour search failed, keyword results only: @message', ['@message' => $e->getMessage()]);
}
return NULL;
}
}
/**
* UUIDs of nodes found only by the neighbour search.
*
* @param int[] $nids
* Node IDs.
*
* @return array<int, string>
* UUIDs keyed by nid.
*/
private function uuids(array $nids): array {
if (!$nids) {
return [];
}
return \Drupal::database()->select('node', 'n')
->fields('n', ['nid', 'uuid'])
->condition('nid', array_values($nids), 'IN')
->execute()
->fetchAllKeyed();
}
/**
* Nodes whose reference is one of the given ones.
*
* field_reference is case-sensitive: references are uppercased (B2).
*
* @param string[] $references
* Uppercase references.
*
* @return array<int, array{0: mixed, 1: string}>
* [nid, uuid] keyed by nid, in result order.
*/
private function referenceQuery(array $references, string $lang): array {
$this->reference_query = $this->keywordQuery('terms', 'OR', '', $lang, 'materio_sapi_search_reference_query');
$conditions = $this->reference_query->createConditionGroup('OR');
foreach ($references as $reference) {
$conditions->addCondition('field_reference', $reference);
}
$this->reference_query->addConditionGroup($conditions);
return $this->collect($this->reference_query);
}
/**
* Whether the phrase is only references: no need for the model (H4, H5).
*/
public function isReferenceSearch(): bool {
return $this->references && trim(preg_replace(self::REFERENCE_PATTERN, '', $this->phrase) ?? '', " \t,;") === '';
}
/**
* The whole keys in the title.
*
* @return array<int, array{0: mixed, 1: string}>
* [nid, uuid] keyed by nid, in result order.
*/
private function phraseQuery(string $keys, string $lang): array {
$this->phrase_query = $this->keywordQuery('phrase', 'AND', $keys, $lang, 'materio_sapi_search_phrase_query');
$this->phrase_query->setFulltextFields(['title']);
// Restrict the search to specific languages.
$this->phrase_query->setLanguages([$lang]);
return $this->collect($this->phrase_query);
}
// Do paging.
// $this->and_query->range($this->offset, $this->limit);
// retrieve all results
// $this->and_query->range(0, -1);
/**
* Every word in the thesaurus and tag fields, or the chosen terms.
*
* @return array<int, array{0: mixed, 1: string}>
* [nid, uuid] keyed by nid, in result order.
*/
private function andQuery(string $keys, string $lang): array {
$this->and_query = $this->keywordQuery('terms', 'AND', $keys, $lang, 'materio_sapi_search_and_query');
// Add sorting.
$this->phrase_query->sort('search_api_relevance', 'DESC');
// Tiebreaker: among equally relevant matches, show the most recently
// modified first (Solr otherwise falls back to internal doc order, i.e.
// indexation order, which puts the oldest materiaux first).
$this->phrase_query->sort('changed', 'DESC');
// Set one or more tags for the query.
// @see hook_search_api_query_TAG_alter()
// @see hook_search_api_results_TAG_alter()
$this->phrase_query->addTag('materio_sapi_search_phrase_query');
$phrase_results = $this->phrase_query->execute();
foreach ($phrase_results as $result) {
$this->results['uuids'][] = $result->getField('uuid')->getValues()[0];
$this->results['nids'][] = $result->getField('nid')->getValues()[0];
}
// ,---. | ,---.
// |---|,---.,---| | |. .,---.,---., .
// | || || | | || ||---'| | |
// ` '` '`---' `---\`---'`---'` `---|
// `---'
$this->and_query = $this->index->query(['offset'=>0,'limit'=>10000]);
// set parse mode and conjunction
$parse_mode = \Drupal::service('plugin.manager.search_api.parse_mode')
->createInstance('terms');
$parse_mode->setConjunction('AND');
$this->and_query->setParseMode($parse_mode);
// Set fulltext search keywords and fields.
if ($this->keys) {
$this->and_query->keys(implode(' ', $this->keys));
}
// in case of term id provided restrict the keys to taxo fields
// A term chosen in the autocomplete restricts to the taxonomy fields.
if ($this->terms && is_array($this->terms) && count($this->terms)) {
$term_conditions = $this->and_query->createConditionGroup('OR');
// $term = (int) $this->term;
foreach ($this->terms as $term) {
$tid = $term->value;
foreach (['tag_tid', 'thesaurus_tid'] as $field) {
$term_conditions->addCondition($field, (int) $tid);
$term_conditions->addCondition($field, (int) $term->value);
}
}
$this->and_query->addConditionGroup($term_conditions);
// INSTEAD TRY TO BOOST THE TAG AND THESAURUS FIELDS
// foreach ($taxoSolrFieldsName as $fname) {
// // $solarium_query->addParam('bf', "recip(abs(ms(NOW,{$solrField})),3.16e-11,10,0.1)");
// $bfparam = "if(gt(termfreq({$fname},{$this->term}),0),^21,0)";
// $this->or_query->addParam('bf', $bfparam);
// }
// look @ materio_sapi_search_api_solr_query_alter in materio_sapi.module
// $this->or_query->setOption('termid', $this->term);
}
if ($this->filters) {
// FILTERS
$filters_conditions = $this->and_query->createConditionGroup('AND');
foreach ($this->filters as $filter) {
$filter = (int) $filter;
foreach (['thesaurus_tid'] as $field) { // 'tag_tid',
$filters_conditions->addCondition($field, $filter);
}
}
$this->and_query->addConditionGroup($filters_conditions);
if(!$this->keys) {
// if no keys but filters switch query to direct and add wildcard solr keys *:*
$direct_and_parse_mode = \Drupal::service('plugin.manager.search_api.parse_mode')
->createInstance('direct');
$direct_and_parse_mode->setConjunction('AND');
$this->and_query->setParseMode($direct_and_parse_mode);
// $this->and_query->keys('*:*');
}
}
// else{
$fulltextFields = [];
// Recherche uniquement sur les champ thésaurus et tag
$fulltextFields += [
// Thesaurus and tag fields only.
$this->and_query->setFulltextFields([
'thesaurus_name_0',
'thesaurus_synonyms_0',
'thesaurus_name_1',
@@ -213,144 +342,67 @@ class Base extends ControllerBase {
'thesaurus_name',
'thesaurus_synonyms',
'tag_name',
'tag_synonyms'
];
if(count($fulltextFields)){
$this->and_query->setFulltextFields($fulltextFields);
'tag_synonyms',
]);
return $this->collect($this->and_query);
}
/**
* Any word, in every fulltext field.
*
* References are no longer a condition here (H4): it restricted the OR
* results to the referenced nodes (« bois R0208 » found nothing).
*
* @return array<int, array{0: mixed, 1: string}>
* [nid, uuid] keyed by nid, in result order.
*/
private function orQuery(string $keys, string $lang): array {
// Terms mode, not direct: Search API escapes the words, so punctuation
// and Solr syntax typed in a phrase (: ( " / \\ ~) are plain text (H3).
// Hyphenated words as separate words, as the direct mode did: in terms mode
// « bio-based » would be a quoted phrase (the phrase query keeps it whole).
$this->or_query = $this->keywordQuery('terms', 'OR', preg_replace('/(?<=\p{L})-(?=\p{L})/u', ' ', $keys), $lang, 'materio_sapi_search_or_query');
return $this->collect($this->or_query);
}
/**
* A keyword query on the database index, not executed yet.
*
* @return \Drupal\search_api\Query\QueryInterface
* The query.
*/
private function keywordQuery(string $parse_mode_id, string $conjunction, string $keys, string $lang, string $tag) {
$query = $this->index->query(['offset' => 0, 'limit' => 10000]);
$parse_mode = \Drupal::service('plugin.manager.search_api.parse_mode')->createInstance($parse_mode_id);
$parse_mode->setConjunction($conjunction);
$query->setParseMode($parse_mode);
if ($keys !== '') {
$query->keys($keys);
}
// }
// Restrict the search to specific languages.
$this->and_query->setLanguages([$lang]);
// Do paging.
// $this->and_query->range($this->offset, $this->limit);
// retrieve all results
// $this->and_query->range(0, -1);
// Add sorting.
$this->and_query->sort('search_api_relevance', 'DESC');
$query->setLanguages([$lang]);
$query->sort('search_api_relevance', 'DESC');
// Tiebreaker: among equally relevant matches, show the most recently
// modified first (Solr otherwise falls back to internal doc order, i.e.
// indexation order, which puts the oldest materiaux first).
$this->and_query->sort('changed', 'DESC');
// Set one or more tags for the query.
$query->sort('changed', 'DESC');
// @see hook_search_api_query_TAG_alter()
// @see hook_search_api_results_TAG_alter()
$this->and_query->addTag('materio_sapi_search_and_query');
$query->addTag($tag);
return $query;
}
$and_results = $this->and_query->execute();
foreach ($and_results as $result) {
// !! have to remove duplicates from phrase query
/**
* Executes a query and lists its nodes once each, in result order.
*
* @return array<int, array{0: mixed, 1: string}>
* [nid, uuid] keyed by nid.
*/
private function collect($query): array {
$list = [];
foreach ($query->execute() as $result) {
$nid = $result->getField('nid')->getValues()[0];
if ( !in_array($nid, $this->results['nids']) ) {
$this->results['uuids'][] = $result->getField('uuid')->getValues()[0];
$this->results['nids'][] = $result->getField('nid')->getValues()[0];
}
$list[$nid] ??= [$nid, $result->getField('uuid')->getValues()[0]];
}
$this->exactematch_count = count($this->results['nids']);
// ,---. ,---.
// | |,---. | |. .,---.,---., .
// | || | || ||---'| | |
// `---'` `---\`---'`---'` `---|
// `---'
$this->or_query = $this->index->query(['offset'=>0,'limit'=>10000]);
// Change the parse mode for the search.
// Les différentes possibilités sont
// - « direct » => Requête directe
// - « terms » => Multiple words
// - « phrase » => Single phrase
// - " edismax " => ???
$or_parse_mode = \Drupal::service('plugin.manager.search_api.parse_mode')
->createInstance('direct');
$or_parse_mode->setConjunction('OR');
$this->or_query->setParseMode($or_parse_mode);
// Set fulltext search keywords and fields.
if ($this->keys) {
$this->or_query->keys(implode(' ', $this->keys));
}
// // exclude results from previous and_query
// !! trigering solr "too many boolean clauses" error
// $exclude_and_results_conditions = $this->or_query->createConditionGroup('AND');
// foreach ($this->results['nids'] as $nid) {
// $exclude_and_results_conditions->addCondition('nid', $nid, '<>');
// }
// $this->or_query->addConditionGroup($exclude_and_results_conditions);
if (preg_match_all('/[WTRPCMFGSO]\d{4}/i', implode(' ', $this->keys), $matches)) {
// in case we search for material references like W0117
$ref_conditions = $this->or_query->createConditionGroup('OR');
foreach ($matches[0] as $key => $value) {
$ref_conditions->addCondition('field_reference', $value);
}
$this->or_query->addConditionGroup($ref_conditions);
}
if ($this->filters) {
// FILTERS
$or_filters_conditions = $this->or_query->createConditionGroup('OR');
foreach ($this->filters as $filter) {
$filter = (int) $filter;
foreach (['thesaurus_tid'] as $field) { // 'tag_tid',
$or_filters_conditions->addCondition($field, $filter);
}
}
$this->or_query->addConditionGroup($or_filters_conditions);
if(!$this->keys) {
// if no keys but filters switch query to direct and add wildcard solr keys *:*
$direct_or_parse_mode = \Drupal::service('plugin.manager.search_api.parse_mode')
->createInstance('direct');
$direct_or_parse_mode->setConjunction('OR');
$this->or_query->setParseMode($direct_or_parse_mode);
// $this->or_query->keys('*:*');
}
}
// Restrict the search to specific languages.
$this->or_query->setLanguages([$lang]);
// Do paging.
// $this->or_query->range($this->offset, $this->limit);
// retrieve all results
// $this->or_query->range(0, -1);
// Add sorting.
$this->or_query->sort('search_api_relevance', 'DESC');
// Tiebreaker: among equally relevant matches, show the most recently
// modified first (Solr otherwise falls back to internal doc order, i.e.
// indexation order, which puts the oldest materiaux first).
$this->or_query->sort('changed', 'DESC');
// Set one or more tags for the query.
// @see hook_search_api_query_TAG_alter()
// @see hook_search_api_results_TAG_alter()
$this->or_query->addTag('materio_sapi_search_or_query');
$or_results = $this->or_query->execute();
foreach ($or_results as $result) {
$nid = $result->getField('nid')->getValues()[0];
// !! have to remove duplicates instead of $exclude_and_results_conditions (solr too many boolean clauses)
if ( !in_array($nid, $this->results['nids']) ) {
$this->results['uuids'][] = $result->getField('uuid')->getValues()[0];
$this->results['nids'][] = $nid;
}
}
// todo you may like / more like this
return $list;
}
private function defaultQuery(){
@@ -379,22 +431,17 @@ class Base extends ControllerBase {
* get params from request
*/
private function parseRequest(Request $request){
// Get the typed string from the URL, if it exists.
$this->keys = $request->query->get('keys');
if($this->keys){
$this->keys = mb_strtolower($this->keys);
$this->keys = Tags::explode($this->keys);
// \Drupal::logger('materio_sapi')->notice($this->keys);
}
// One phrase, commas included: no more splitting into tags (H1).
// keys stays an array of one lowercased element for the keyword queries.
$this->phrase = self::cleanPhrase($request->query->get('keys'));
$this->keys = $this->phrase !== '' ? [mb_strtolower($this->phrase)] : NULL;
// get the exacte term id in case of autocomplete
// $this->terms = $request->query->get('terms');
$t = $request->query->get('terms');
// $this->terms = strlen($t) ? explode(',', $t) : null;
$this->terms = $t && strlen($t) ? json_decode($t) : null;
// get the filters of advanced search
$f = $request->query->get('filters');
$this->filters = $f && strlen($f) ? explode(',', $f) : null;
// The filters parameter of the assisted search (old URLs) is ignored.
// $this->allparams = $request->query->all();
// $request->attributes->get('_raw_variables')->get('filters')
//
@@ -402,6 +449,17 @@ class Base extends ControllerBase {
$this->limit = $request->query->get('limit') ?? $this->limit;
}
/**
* Normalizes the typed phrase: spaces collapsed, length limited.
*/
public static function cleanPhrase($keys): string {
if (!is_string($keys)) {
return '';
}
$phrase = trim(preg_replace('/[\s\p{Cc}]+/u', ' ', $keys) ?? '');
return mb_substr($phrase, 0, self::MAX_PHRASE_LENGTH);
}
/**
* Handler for ajax search.
*/
@@ -415,43 +473,24 @@ class Base extends ControllerBase {
),
];
if ($this->keys || $this->filters) {
if ($this->keys) {
$lang = \Drupal::languageManager()->getCurrentLanguage()->getId();
$this->sapiQuery();
$this->rankedSearch($lang);
$resp['keys'] = json_encode($this->keys);
$resp['terms'] = json_encode($this->terms);
$resp['filters'] = $this->filters;
// $resp['count'] = $this->results->getResultCount();
$resp['count'] = count($this->results['nids']);
$resp['exactematch_count'] = $this->exactematch_count;
$resp['infos'] = t('The search found @exactmatchcount exact match result(s) for @count total result(s) with', array(
"@exactmatchcount" => $resp['exactematch_count'],
"@count" => $resp['count']
));
if ($this->keys) {
$resp['infos'] .= t(' keywords @keys', array(
"@keys" => implode(', ', $this->keys)
));
}
if ($this->keys && $this->filters) {
$resp['infos'] .= " and";
}
if ($this->filters) {
// get the terms names from tid
$storage = \Drupal::entityTypeManager()->getStorage('taxonomy_term');
$filters_names = [];
foreach ($this->filters as $tid) {
/** @var \Drupal\Core\Entity\EntityInterface $entity */
$entity = $storage->load($tid);
$entity_trans = \Drupal::service('entity.repository')->getTranslationFromContext($entity, $lang);
$filters_names[] = $entity_trans->getName();
}
$resp['infos'] .= t(' filters @filters', array(
"@filters" => implode(', ', $filters_names)
));
}
// Relevant: exact matches and neighbours that stand out (H6); the
// count no longer depends on an arbitrary limit. French text added by
// materio_sapi_update_11001().
$resp['infos'] = t('@exactmatchcount exact result(s) out of @count relevant result(s) for “@keys”', [
'@exactmatchcount' => $resp['exactematch_count'],
'@count' => $resp['count'],
'@keys' => $this->phrase,
]);
// $resp['options'] = $this->query->getOptions();
// $uuids = [];
@@ -508,7 +547,7 @@ class Base extends ControllerBase {
];
if ($this->keys) {
$resp['#title'] = implode(', ', $this->keys);
$resp['#title'] = $this->phrase;
// $this->sapiQuery();
@@ -1,151 +0,0 @@
<?php
namespace Drupal\materio_sapi\Form;
use Drupal\Core\Form\FormBase;
use Drupal\Core\Form\FormStateInterface;
/**
* Class MaterioSapiSearchForm.
*/
class MaterioSapiSearchForm extends FormBase {
/**
* {@inheritdoc}
*/
public function getFormId() {
return 'materio_sapi_search_form';
}
/**
* {@inheritdoc}
*/
public function buildForm(array $form, FormStateInterface $form_state) {
$lang = \Drupal::languageManager()->getCurrentLanguage()->getId();
$keys = \Drupal::request()->get('keys');
$term = \Drupal::request()->get('term');
$filters = \Drupal::request()->get('filters');
$form['#cache'] = array(
'contexts' => array(
'url.query_args:keys',
'url.query_args:term',
'url.query_args:filters',
),
);
$form['search'] = [
'#type' => 'textfield',
// '#title' => $this->t('Search'),
'#maxlength' => 150,
'#size' => 25,
'#weight' => '0',
'#attributes' => [
"placeholder" => $this->t('Search'),
// "@keyup" => "keyup",
// "@keyup.enter" => "submit",
"v-model" => "typed",
"v-focus" => "",
// "v-on:select" => "typed",
],
'#autocomplete_route_name' => 'materio_sapi.search_autocomplete',
];
$form['searchautocomplete'] = [
'#type' => 'hidden',
'#attributes' => [
"v-model" => "autocomplete"
],
];
// $filters_array = [];
$filters_classes = [];
if($filters){
$filters_array = explode(',',$filters);
$filters_classes[] = 'open';
}
$form['filters'] = [
'#type' => 'fieldset',
'#title' => t('filters'),
'#legend_attributes' => [
"class" => ["test-attribute"],
'@click.prevent' => "onClickFilters",
],
'#attributes' => [
"class" => $filters_classes
]
];
// $form['filters']['switch'] = [
// '#type' => "label",
//
// ]
$query = \Drupal::entityQuery('taxonomy_term')
->condition('vid', "assisted_research")
->accessCheck(TRUE)
->sort('field_weight', 'ASC');
$tids = $query->execute();
$terms = \Drupal\taxonomy\Entity\Term::loadMultiple($tids);
foreach ($terms as $tid => $term) {
if($term->hasTranslation($lang)){
$term = \Drupal::service('entity.repository')->getTranslationFromContext($term, $lang);
}
$options = array($term->getName());
$childs = $term->get('field_terms')->getValue();
$selected = null;
foreach ($childs as $child) {
$child_term = \Drupal\taxonomy\Entity\Term::load($child['target_id']);
if($child_term->hasTranslation($lang)){
$child_term = \Drupal::service('entity.repository')->getTranslationFromContext($child_term, $lang);
}
$options[$child_term->id()] = $child_term->getName();
if (isset($filters_array) && in_array($child_term->id(), $filters_array)) {
$selected = $child_term->id();
}
}
$form['filters']['filter-'.$tid] = array(
'#type' => 'select',
// '#title' => $term->getName(),
// '#multiple' => true,
'#options' => $options,
'#value' => $selected
);
}
$form['submit'] = array(
'#type' => 'button',
// '#src' => '',//drupal_get_path('module', 'materio_search_api') . '/images/search.png',
'#value' => t('Search'),
// '#prefix' => '<div class="search-btn-wrapper">',
// '#suffix'=> '</div>',
'#attributes' => [
'title' => t('Search'),
'@click.prevent' => "submit",
]
);
return $form;
}
/**
* {@inheritdoc}
*/
public function validateForm(array &$form, FormStateInterface $form_state) {
parent::validateForm($form, $form_state);
}
/**
* {@inheritdoc}
*/
public function submitForm(array &$form, FormStateInterface $form_state) {
// Display result.
foreach ($form_state->getValues() as $key => $value) {
\Drupal::messenger()->addStatus($key . ': ' . $value);
}
}
}
@@ -0,0 +1,94 @@
<?php
declare(strict_types=1);
namespace Drupal\materio_sapi;
/**
* Keyword text of a phrase: stop words and request formulas removed (H2).
*
* The stop words are those of the Solr field type of the language
* (search_api_solr text_LANG_*, the source of stopwords_LANG.txt in the
* configset): the keyword queries also search the custom types of
* materio_sapi (tags, thesaurus, countries, n-grams), which keep them. The
* whole phrase still goes to the model.
*/
final class KeywordText {
/**
* Words that only ask, and say nothing of the material.
*/
private const REQUEST_WORDS = [
'fr' => ['cherche', 'recherche', 'cherchons', 'recherchons', 'besoin', 'possible', 'svp', 'merci'],
// The English list of search_api_solr is short (no « i », « am », « my »…).
'en' => ['i', 'am', "i'm", 'we', 'my', 'me', 'looking', 'look', 'searching', 'search', 'need', 'want', 'possible', 'something', 'please', 'thanks'],
];
/**
* Leading elisions of French, removed before the stop word check.
*/
private const ELISION = '/^(?:[cdjlmnst]|qu)[\'’]/u';
/**
* Stop words by language, loaded once.
*
* @var array<string, array<string, true>>
*/
private static array $stopWords = [];
/**
* Returns the words of a phrase that carry meaning, in order.
*
* @return string[]
* Lowercased words; empty when the phrase has only stop words.
*/
public static function words(string $phrase, string $langcode): array {
$skip = self::stopWords($langcode);
$words = [];
// Solr and Search API syntax between words is a separator (« bois:verre »,
// « bois/métal »); hyphens, apostrophes, and . , in numbers stay.
$phrase = preg_replace('#[:;/\\\\^&|()\[\]{}<>"«»~*?!=+\#@$]+#u', ' ', $phrase) ?? $phrase;
foreach (preg_split('/\s+/u', mb_strtolower($phrase), -1, PREG_SPLIT_NO_EMPTY) as $token) {
// Edge punctuation, and leading + - ! that Search API reads as operators
// (« -bois » excluded wood and returned every other node, H3).
$token = trim($token, ",;:!?.()[]{}\"«»'’<>");
$token = ltrim($token, '+-!');
$word = preg_replace(self::ELISION, '', $token) ?? $token;
// No letter or digit (« || », « & »): breaks the payload boost of Solr.
if (!preg_match('/[\p{L}\p{N}]/u', $word) || isset($skip[$word]) || isset($skip[$token])) {
continue;
}
$words[] = $word;
}
return $words;
}
/**
* Stop words and request words of a language.
*
* @return array<string, true>
* Words as keys.
*/
private static function stopWords(string $langcode): array {
if (!isset(self::$stopWords[$langcode])) {
$list = self::REQUEST_WORDS[$langcode] ?? [];
$storage = \Drupal::entityTypeManager()->getStorage('solr_field_type');
foreach (["text_{$langcode}_7_0_0", "text_{$langcode}_6_0_0"] as $id) {
/** @var \Drupal\search_api_solr\SolrFieldTypeInterface|null $type */
if ($type = $storage->load($id)) {
$text = $type->getTextFiles()['stopwords'] ?? '';
foreach (preg_split('/\R/u', $text) as $line) {
$line = trim(preg_replace('/#.*/u', '', $line) ?? '');
if ($line !== '') {
$list[] = mb_strtolower($line);
}
}
break;
}
}
self::$stopWords[$langcode] = array_fill_keys($list, TRUE);
}
return self::$stopWords[$langcode];
}
}
@@ -20,11 +20,16 @@ class MaterioSapiSearchBlock extends BlockBase {
* {@inheritdoc}
*/
public function build() {
$build = [];
// $build['materio_sapi_search_block']['#markup'] = 'Implement MaterioSapiSearchBlock.';
$form = \Drupal::formBuilder()->getForm('Drupal\materio_sapi\Form\MaterioSapiSearchForm');
$build['materio_sapi_search_block'] = $form;
return $build;
// Only a place in the page: the search field is a Vue component
// (materiotheme SearchForm.vue, F1) mounted on this block. Not empty,
// otherwise Drupal would not render the block.
return [
'placeholder' => [
'#type' => 'html_tag',
'#tag' => 'div',
'#attributes' => ['class' => ['materio-sapi-search-placeholder']],
],
];
}
/**
@@ -0,0 +1,66 @@
<?php
declare(strict_types=1);
namespace Drupal\materio_sapi;
/**
* Merges ranked result lists: exact matches first, the rest by RRF (H5).
*
* Reciprocal rank fusion: a node scores the sum of 1 / (k + rank) over the
* lists that hold it (rank from 1). Only ranks count, not the scores of each
* engine, which are not comparable (BM25 against cosine). Equal scores keep
* the order of first appearance, list by list: the keyword lists are already
* sorted by relevance then by date (fallback sort kept).
*/
final class RankFusion {
/**
* Usual constant of RRF: dampens the weight of the first ranks.
*/
public const K = 60;
/**
* Returns the merged order.
*
* @param array<int|string> $exact
* Node IDs of the exact matches (references, phrase, all words), in order.
* Always first, whatever the other lists say.
* @param array<array<int|string>> $rankings
* Ranked lists of node IDs (keyword OR query, kNN neighbours…).
* @param int $k
* RRF constant.
*
* @return array<int|string>
* Node IDs, each once.
*/
public static function fuse(array $exact, array $rankings, int $k = self::K): array {
$merged = [];
foreach ($exact as $nid) {
$merged[$nid] ??= $nid;
}
$scores = [];
$first = [];
$position = 0;
foreach ($rankings as $ranking) {
$rank = 0;
foreach ($ranking as $nid) {
$rank++;
if (isset($merged[$nid])) {
continue;
}
if (!isset($scores[$nid])) {
$scores[$nid] = 0.0;
$first[$nid] = [$position++, $nid];
}
$scores[$nid] += 1 / ($k + $rank);
}
}
uksort($scores, static fn ($a, $b): int => [$scores[$b], $first[$a][0]] <=> [$scores[$a], $first[$b][0]]);
foreach (array_keys($scores) as $nid) {
$merged[$nid] = $first[$nid][1];
}
return array_values($merged);
}
}
@@ -0,0 +1,44 @@
<?php
declare(strict_types=1);
namespace Drupal\materio_sapi;
/**
* Keeps the neighbours that stand out for this search (H6, option f).
*
* The kNN search scores every visible node, so each search has its own score
* distribution. A fixed threshold means nothing: the scale moves with the
* search and the language (at 0.7, « bois » kept 664 nodes and « wood »
* 4 671). A node is kept when its score is above the mean of this search by
* more than K standard deviations: with K = 2, « bois » 373 and « wood » 382,
* 150 to 500 nodes for one word, about 250 for a phrase (measured 17/11).
*/
final class ScoreCut {
/**
* Scores above mean + $k standard deviations, in their order.
*
* @param array<int, float> $scores
* Scores keyed by nid, best first, ideally of every visible node.
* @param float $k
* Number of standard deviations.
*
* @return array<int, float>
* The kept scores, keyed by nid.
*/
public static function aboveMean(array $scores, float $k): array {
$n = count($scores);
if ($n < 2) {
return $scores;
}
$mean = array_sum($scores) / $n;
$variance = 0.0;
foreach ($scores as $score) {
$variance += ($score - $mean) ** 2;
}
$limit = $mean + $k * sqrt($variance / $n);
return array_filter($scores, static fn ($score) => $score > $limit);
}
}
@@ -0,0 +1,66 @@
<?php
declare(strict_types=1);
namespace Drupal\Tests\materio_sapi\Unit;
use Drupal\materio_sapi\RankFusion;
use Drupal\Tests\UnitTestCase;
use PHPUnit\Framework\Attributes\CoversClass;
use PHPUnit\Framework\Attributes\Group;
/**
* Tests the merge of the keyword and kNN rankings (H5).
*/
#[CoversClass(RankFusion::class)]
#[Group('materio_sapi')]
final class RankFusionTest extends UnitTestCase {
/**
* Exact matches stay first, in their order, even when ranked low elsewhere.
*/
public function testExactFirst(): void {
$order = RankFusion::fuse([30, 10], [[1, 2, 10], [2, 1, 30]]);
$this->assertSame([30, 10], array_slice($order, 0, 2));
}
/**
* Each node appears once, exact matches included.
*/
public function testNoDuplicate(): void {
$order = RankFusion::fuse([5], [[5, 1, 2], [2, 5, 3]]);
$this->assertSame([5, 2, 1, 3], $order);
}
/**
* A node found by both lists beats a node first in one list only.
*/
public function testBothListsWin(): void {
// 2 : 1/62 + 1/61 ; 1 : 1/61 ; 9 : 1/61.
$this->assertSame([2, 1, 9], RankFusion::fuse([], [[1, 2], [9, 2]]));
}
/**
* Equal scores keep the order of first appearance (keyword list first).
*/
public function testTiesKeepFirstAppearance(): void {
$this->assertSame([7, 8, 3, 4], RankFusion::fuse([], [[7, 3], [8, 4]]));
}
/**
* Empty lists: the exact matches alone, or nothing.
*/
public function testEmpty(): void {
$this->assertSame([4], RankFusion::fuse([4], [[], []]));
$this->assertSame([], RankFusion::fuse([], []));
}
/**
* Numeric strings and integers are the same node, the first type is kept.
*/
public function testIdTypes(): void {
$this->assertSame([12, 13], RankFusion::fuse([], [[12, 13], ['12']]));
$this->assertSame(['8'], RankFusion::fuse(['8'], [[8]]));
}
}
@@ -0,0 +1,47 @@
<?php
declare(strict_types=1);
namespace Drupal\Tests\materio_sapi\Unit;
use Drupal\materio_sapi\ScoreCut;
use Drupal\Tests\UnitTestCase;
use PHPUnit\Framework\Attributes\CoversClass;
use PHPUnit\Framework\Attributes\Group;
/**
* Tests the cut of the neighbours relative to the search (H6).
*/
#[CoversClass(ScoreCut::class)]
#[Group('materio_sapi')]
final class ScoreCutTest extends UnitTestCase {
/**
* Only the scores above mean + k standard deviations stay, in order.
*/
public function testAboveMean(): void {
// 98 scores at 0.6, then 0.9 and 0.8: mean 0.6045, deviation ≈ 0.0337.
$scores = [7 => 0.9, 3 => 0.8] + array_fill_keys(range(100, 197), 0.6);
$this->assertSame([7 => 0.9, 3 => 0.8], ScoreCut::aboveMean($scores, 2.0));
$this->assertSame([7 => 0.9], ScoreCut::aboveMean($scores, 7.0));
}
/**
* The same distribution shifted keeps the same nodes: no fixed threshold.
*/
public function testShiftedScale(): void {
$scores = [7 => 0.9, 3 => 0.8] + array_fill_keys(range(100, 197), 0.6);
$shifted = array_map(static fn ($s) => $s + 0.15, $scores);
$this->assertSame(array_keys(ScoreCut::aboveMean($scores, 2.0)), array_keys(ScoreCut::aboveMean($shifted, 2.0)));
}
/**
* Equal scores stand out from nothing; fewer than two scores are kept.
*/
public function testEdgeCases(): void {
$this->assertSame([], ScoreCut::aboveMean([1 => 0.7, 2 => 0.7, 3 => 0.7], 2.0));
$this->assertSame([5 => 0.8], ScoreCut::aboveMean([5 => 0.8], 2.0));
$this->assertSame([], ScoreCut::aboveMean([], 2.0));
}
}
File diff suppressed because one or more lines are too long
Binary file not shown.
File diff suppressed because one or more lines are too long
Binary file not shown.
File diff suppressed because one or more lines are too long
Binary file not shown.
File diff suppressed because one or more lines are too long
Binary file not shown.
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -381,12 +381,9 @@ export let _v_sitebranding_block, _v_user_block, _v_header_menu, _v_language_swi
// console.log('router beforeEach to ', to)
const id = 'block-materiosapisearchblock'
let $search_block = document.getElementById(id)
let formhtml = null
if ($search_block) {
// get the search form html to pass it as template to the vue
// we gain display speed vs async downloaded data
formhtml = $search_block.innerHTML
} else {
// The block is only a place in the page: the search form is the Vue
// component SearchForm (F1).
if (!$search_block) {
// else create the empty block to fill it later with async data
$search_block = document.createElement('div')
$search_block.setAttribute('id', id)
@@ -399,7 +396,7 @@ export let _v_sitebranding_block, _v_user_block, _v_header_menu, _v_language_swi
_v_search_block = new Vue({
store,
i18n,
render: h => h(VSearchBlock, { props: { blockid: id, formhtml: formhtml } })
render: h => h(VSearchBlock, { props: { blockid: id } })
}).$mount('#' + id)
}
function initVLeftContent () {
@@ -1,10 +1,6 @@
@use "sass:math";
@use "sass:color";
// @import '~slim-select/scss';
@import 'slim-select/styles';
$mdi-font-path: './mdi/fonts/';
@import './mdi/css/materialdesignicons.css';
@import './mdi/scss/variables';
@@ -641,72 +637,6 @@ header[role="banner"]{
}
}
fieldset#edit-filters{
grid-row: 2 / 2;
grid-column: 1 / span 2;
border: none;
padding: 0;
>legend{
line-height: 0.6;
padding-bottom: 0.2em;
width: 100%;
text-align: right;
span{
cursor: pointer;
font-size: 0.756em;
color: #8f8f8f;
}
}
> div.fieldset-wrapper{
height: 0;
overflow: hidden;
}
&.open >div.fieldset-wrapper{
height: auto;
overflow: visible;
}
.ss-main{
font-size: 0.756em;
.ss-single-selected{
border-radius: 0.7em;
&.ss-open-below{
border-radius: 0.7em 0.7em 0 0.7em;
}
height:1.5em;
border: none;
padding: 0 0.5em;
background-color: #eeeaea;
span.placeholder{
color: $color-main-text;
line-height: 1;
span.ss-disabled{
color: #8f8f8f;
}
}
span.ss-arrow span{
border-color: #b1adad;
}
}
.ss-content{
width:auto;
border: none;
border-radius: 0.7em 0 0.7em 0.7em;
&.ss-open{
box-shadow: 0 0 10px #ccc;
}
right: 0;
.ss-list{
max-height: none;
.ss-option{
white-space: nowrap;
text-align: right;
}
}
}
}
}
.button.form-submit{
grid-row: 1 / 1;
grid-column: 2 / 2;
@@ -1,79 +1,19 @@
<template v-slot="default">
<div v-bind:id="blockid">
<SearchForm
v-if="displayform"
v-bind:form="form"></SearchForm>
<SearchForm v-if="canSearch"></SearchForm>
</div>
</template>
<script>
import qs from 'querystring-es3'
import SearchForm from 'vuejs/components/Form/SearchForm'
import { mapState, mapActions } from 'vuex'
import MA from 'vuejs/api/ma-axios'
import { mapState } from 'vuex'
export default {
props: ['blockid', 'formhtml'],
data(){
return {
form: null
}
},
props: ['blockid'],
computed: {
...mapState({
canSearch: state => state.User.canSearch,
keys: state => state.Search.keys,
terms: state => state.Search.terms,
filters: state => state.Search.filters
}),
displayform(){
// console.log('computed displayform')
return this.canSearch && this.form
}
},
beforeMount() {
// console.log('SearchBlock beforeMount')
this.form = this.formhtml
},
watch: {
canSearch(new_value, old_value) {
// console.log('canSearch changed, old: '+old_value+", new: "+new_value)
if (new_value && !this.form) {
this.getSearchForm()
}
if (!new_value && this.form) {
this.form = null
}
}
},
methods: {
getSearchForm(){
console.log('getSearchForm')
// var urlParams = new URLSearchParams(window.location.search);
// var urlParamsKeys = urlParams.keys()
const params = {
keys: this.keys,
terms: this.terms, //JSON.stringify(this.terms),
filters: this.filters
}
console.log('Search getSearchForm params', params)
const q = qs.stringify(params)
MA.get(`/materio_sapi/search_form?`+q)
.then(({data}) => {
// console.log('getSearchForm')
// guard against unexpected response body (truncated / non json)
// which would crash Vue.compile(undefined) in SearchForm
if (typeof data.rendered !== 'string') {
console.warn('SearchBlock getSearchForm unexpected data', data)
return
}
this.form = data.rendered
})
.catch((error) => {
console.warn('Issue with get searchform', error)
})
}
canSearch: state => state.User.canSearch
})
},
components: {
SearchForm
@@ -1,241 +1,82 @@
<template>
<!-- One field for a phrase in natural language, one button (F1).
Same ids and classes as the former Drupal form, so main.scss applies. -->
<form
id="materio-sapi-search-form"
class="materio-sapi-search-form"
role="search"
@submit.prevent="submit"
>
<div class="form-item form-item-search">
<input
id="edit-search"
ref="input"
v-model="typed"
class="form-text"
type="search"
name="search"
:maxlength="maxLength"
:placeholder="$t('default.Search')"
:aria-label="$t('default.Search')"
enterkeyhint="search"
autocomplete="off"
autocapitalize="off"
spellcheck="false"
>
</div>
<input
type="submit"
class="button form-submit"
:value="$t('default.Search')"
:title="$t('default.Search')"
>
</form>
</template>
<script>
import Vue from 'vue'
import router from 'vuejs/route'
import { mapState } from 'vuex'
import SlimSelect from 'slim-select'
export default {
router,
props: ['form'],
data() {
data () {
return {
template: null,
typed: null,
autocomplete: [],
slimFilters: [],
$input: null
// basePath: drupalSettings.path.baseUrl + drupalSettings.path.pathPrefix
typed: '',
// Same limit as the backend (Base::MAX_PHRASE_LENGTH, H1).
maxLength: 150
}
},
computed: {
...mapState({
keys: state => state.Search.keys,
terms: state => state.Search.terms,
filters: state => state.Search.filters
keys: state => state.Search.keys
})
},
methods: {
submit() {
console.log("slim search submited", this.typed, this.autocomplete, this.filters, this.slimFilters)
// unfocus the text input to hide the keyboard on mobile device
this.$input.blur()
// New search is triggered by Base.vue with router (which will also fill the store)
// cleaning slimfilters
let filters = []
this.slimFilters.forEach((filter, index) => {
console.log('onsubmit foreach slimFilters', filter)
if(filter){
filters.push(filter)
}
})
// // this is just in case we landed in the page with filters in url params,
// // this.slimFilters is not filled but store/search filters is
// if(!filters.length){
// filters = this.filters
// }
console.log('Cleaning filters',this.slimFilters, this.filters, filters)
// push router
this.$router.push({name:'base', query:{
keys:this.typed,
// terms:this.autocomplete.join(','),
terms: JSON.stringify(this.autocomplete),
filters:filters.join(',')
}})
phraseFromKeys (keys) {
return keys && keys.length ? keys.join(', ') : ''
},
onAutoCompleteSelect(event, ui){
event.preventDefault();
console.log('autoCompleteSelect begining', this.typed, event, ui)
// split typed into tag list
var tag_list = this.typed.split(', ')
console.log('tag_list', tag_list)
// remove the last item as it is the first letters of autocomplete (like ma for marbre)
tag_list.pop()
// add in typed the label
tag_list.push(ui.item.label)
this.typed = tag_list.join(', ') + ', '
// check if item is not already in autocmplete
let add = true
this.autocomplete.forEach( (term) => {
if (term.value == ui.item.value) {
add = false
return
}
})
if (add) {
this.autocomplete.push(ui.item)
}
// we have to wait for typed watch to reset autocomplete before setting it
// setTimeout(function(){
// console.log('update autocomplete value after settimeout')
// this.autocomplete.push(ui.item)
// if (this.typed !== this.keys || this.autocomplete !== this.term) {
// this.submit()
// }
// }.bind(this), 1)
console.log('autoCompleteSelect end : this.autocomplete', this.autocomplete)
},
onSelectFiltersChange(index, info){
console.log('slim onSelectFiltersChange', index, info, this.slimFilters, this.filters)
this.slimFilters[index] = info.value
console.log(this.slimFilters);
},
onClickFilters(e){
console.log('onClickFilters legend', e)
e.target.closest('fieldset').classList.toggle('open')
// TODO: reset the filters oon close
}
},
directives: {
focus: {
// directive definition
inserted: function (el) {
// do not focus the input as it popup the keyboard on mobile device
// el.focus()
}
}
},
beforeMount() {
// console.log('SearchForm beforeMount')
if (this._props.form) {
// console.log('SearchForm beforeMount if this._props.form ok')
this.template = Vue.compile(this._props.form)
// https://github.com/vuejs/vue/issues/9911
this.$options.staticRenderFns = [];
this._staticTrees = [];
this.template.staticRenderFns.map(fn => (this.$options.staticRenderFns.push(fn)));
submit () {
const phrase = this.typed.replace(/\s+/g, ' ').trim()
// Hide the keyboard on mobile devices.
this.$refs.input.blur()
// Base.vue reacts to the route change and runs the search.
// Only the phrase: no more terms (F7) nor filters.
this.$router.push({ name: 'base', query: phrase ? { keys: phrase } : {} })
.catch(() => {})
}
},
watch: {
typed(n, o){
console.log(`watch typed changed o:(${typeof o})${o} | n:(${typeof n})${n}`, o, n)
// if (typeof n === 'string') {
// // remove terms from autocomplete if removed from typed
// const r = /,\s?$/ // remove last comma space
// let tag_list = n.replace(r,'').split(', ')
// }else{
// let tag_list = n
// }
// remove terms from autocomplete if removed from typed
if (typeof n !== 'string') {
return
}
const r = /,\s?$/ // remove last comma space
let tag_list = n.replace(r,'').split(', ')
console.log('watch typed tag_list', tag_list)
console.log('watch typed autocomplete before', this.autocomplete)
// Keep only the terms whose label is still present in the typed text.
// Filtering into a new array avoids skipping items when splicing during
// iteration (which left stale terms in autocomplete, corrupting `terms`).
this.autocomplete = this.autocomplete.filter(term => tag_list.indexOf(term.label) >= 0)
console.log('watch typed autocomplete after', this.autocomplete)
},
keys(n, o){
console.log('watch keys changed', o, n)
this.typed = n.length ? n.join(', ') + ', ' : ''
},
// terms(n, o){
// // if term change from store
// console.log('watch terms changed', o, n)
// this.autocomplete = n
// }
},
created() {
// fill component values with store values in case of direct page loading
console.log('SearchForm created, this.keys', this.keys)
this.typed = this.keys.length ? this.keys.join(', ') + ', ' : ''
this.autocomplete = this.terms
},
mounted(){
// console.log('SearchForm mounted')
Drupal.attachBehaviors(this.$el);
// Route the native form submission (Enter key) through the same submit()
// method as the button (@click.prevent="submit"). Without this, pressing
// Enter either triggers a native POST reload or only fills the jQuery UI
// autocomplete selection, so keys/terms sent to the backend differ from a
// button click and the search results are inconsistent.
// Note: when the autocomplete menu is open, jQuery UI handles Enter itself
// (selects the highlighted suggestion) and prevents this submit event, which
// is the intended behaviour.
this.$el.addEventListener('submit', (e) => {
e.preventDefault()
this.submit()
})
// get the search input
this.$input = this.$el.querySelector('#edit-search')
// // focus on input
// $input.focus()
// Catch the jquery ui events for autocmplete widget
jQuery(this.$input).on('autocompleteselect', this.onAutoCompleteSelect);
// customize the select filters
// http://slimselectjs.com/options
const selects = this.$el.querySelectorAll('select')
console.log('slim selects', selects);
let slim;
selects.forEach((selectElement, index) => {
console.log('slim selects forEach', selectElement, index);
// get the first option to make the placeholder then empty it
const placeholder_option = selectElement.querySelector('option:first-child')
// console.log('placeholder_option', placeholder_option);
const placeholder = placeholder_option.innerText
placeholder_option.removeAttribute("value")
placeholder_option.setAttribute("data-placeholder", true)
placeholder_option.innerHTML = ''
slim = new SlimSelect({
events:{
afterChange: (newval) => {
console.log('slim slimselect onchange', index, newval, this)
this.onSelectFiltersChange(index, newval[0])
}
},
select: selectElement,
placeholder: placeholder,
allowDeselect: true,
showSearch: false,
closeOnSelect: true
})
console.log('slim', slim);
// console.log('slimselect selected', slim, slim.selected(), index)
this.slimFilters[index] = slim.getSelected()[0];
})
console.log('slimSelect slimFilters after init', this.slimFilters)
},
render(h) {
// console.log('searchForm render')
if (!this.template) {
return h('span', $t('default.Loading…'))
} else {
return this.template.render.call(this)
// The store changes when a search starts from the URL or another link.
keys (n) {
this.typed = this.phraseFromKeys(n)
}
},
created () {
// Direct page load: the store already holds the keys of the URL.
this.typed = this.phraseFromKeys(this.keys)
}
}
</script>
<style lang="scss" scoped>
</style>
@@ -4,7 +4,8 @@
<span>{{ $t('default.Loading…') }}</span>
</div>
<div class="cards-list" v-else>
<aside class="search-info">
<!-- aria-live: screen readers announce the result message of each search (F1). -->
<aside class="search-info" aria-live="polite">
{{ searchinfos }}
</aside>
<div class="loading" v-if="!items.length & !noresults">
@@ -50,7 +51,8 @@ export default {
count: state => state.Search.count,
noresults: state => state.Search.noresults,
limit: state => state.Search.limit,
searchId: state => state.Search.searchId
searchId: state => state.Search.searchId,
keys: state => state.Search.keys
}),
// items is a sparse array (pages placed by offset); drop the holes left by
// pages that returned fewer results than the limit before rendering.
@@ -58,7 +60,23 @@ export default {
return this.items.filter(Boolean)
}
},
// Title of the browser tab: the phrase searched (F3).
metaInfo () {
return {
title: this.keys && this.keys.length ? this.keys.join(', ') : 'Base'
}
},
methods: {
// terms of an URL: a JSON list, possibly malformed in an old or edited
// link; never break the page for it (F3).
parseTerms (value) {
try {
const terms = JSON.parse(value)
return Array.isArray(terms) ? terms : []
} catch (e) {
return []
}
},
...mapActions({
newSearch: 'Search/newSearch',
nextPage: 'Search/nextPage'
@@ -86,16 +104,12 @@ export default {
if (params.has('terms')) {
console.log('Base created, has terms', params.get('terms'))
// this.$store.commit('Search/setTerms', params.get('terms').split(','))
this.$store.commit('Search/setTerms', JSON.parse(params.get('terms')))
this.$store.commit('Search/setTerms', this.parseTerms(params.get('terms')))
} else {
this.$store.commit('Search/reSetTerms')
}
if (params.has('filters')) {
this.$store.commit('Search/setFilters', params.get('filters').split(','))
} else {
this.$store.commit('Search/reSetFilters')
}
// The filters parameter of former URLs is ignored (F2).
this.newSearch()
},
@@ -108,15 +122,10 @@ export default {
// to.query.terms is a raw JSON string (see SearchForm.submit) and must be
// parsed like in created(), otherwise getResults() double-encodes it and
// the backend silently drops the terms (less relevant results).
this.$store.commit('Search/setTerms', JSON.parse(to.query.terms))
this.$store.commit('Search/setTerms', this.parseTerms(to.query.terms))
}else{
this.$store.commit('Search/reSetTerms')
}
if (to.query.hasOwnProperty('filters')) {
this.$store.commit('Search/setFilters', to.query.filters)
}else{
this.$store.commit('Search/reSetFilters')
}
if (to.query.hasOwnProperty('keys')) {
const r = /,\s?$/
let keys = to.query.keys.replace(r,'').split(', ')
@@ -17,7 +17,6 @@ export default {
state: {
keys: [],
terms: [],
filters: [],
uuids: [],
items: [],
offset: 0,
@@ -92,13 +91,6 @@ export default {
reSetTerms (state) {
state.terms = []
},
setFilters (state, filters) {
console.log('store search setFilters', filters)
state.filters = typeof filters === 'string' ? filters.split(',') : filters
},
reSetFilters (state) {
state.filters = []
},
setInfos (state, infos) {
state.infos = infos
},
@@ -169,10 +161,6 @@ export default {
limit: state.limit
}
console.log('search store getResults, params', params)
if (state.filters) {
console.log('getResults filters', state.filters)
params.filters = state.filters.join(',')
}
// console.log('Search getResults params', params)
const q = qs.stringify(params)
// * Qs.Stringify () Arrayformat: 'Repeat' : https://programmerall.com/article/31901061156/