Add real test methodology document
Browse files- README.md +1 -0
- reports/TEST_METHOD.md +38 -0
README.md
CHANGED
|
@@ -62,6 +62,7 @@ Report reali eseguiti su corpus AIO locale (101k+ records, fonti pubbliche reali
|
|
| 62 |
Artifact pubblicati:
|
| 63 |
- `reports/REAL_CASE_TEST_RESULTS_LATEST.md`
|
| 64 |
- `reports/REAL_CASE_TEST_RESULTS_LATEST.json`
|
|
|
|
| 65 |
|
| 66 |
Scope case use testati:
|
| 67 |
- clima mediterraneo ed energia
|
|
|
|
| 62 |
Artifact pubblicati:
|
| 63 |
- `reports/REAL_CASE_TEST_RESULTS_LATEST.md`
|
| 64 |
- `reports/REAL_CASE_TEST_RESULTS_LATEST.json`
|
| 65 |
+
- `reports/TEST_METHOD.md`
|
| 66 |
|
| 67 |
Scope case use testati:
|
| 68 |
- clima mediterraneo ed energia
|
reports/TEST_METHOD.md
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# AIO Real Test Method
|
| 2 |
+
|
| 3 |
+
Generated on: 2026-02-21
|
| 4 |
+
|
| 5 |
+
## Goal
|
| 6 |
+
Publish real multi-case retrieval tests on the full local AIO corpus, then expose only results in the public showcase.
|
| 7 |
+
|
| 8 |
+
## Data Scope
|
| 9 |
+
- Corpus file: `aio_archive/records.json`
|
| 10 |
+
- Corpus size at test time: `101257` records
|
| 11 |
+
- Unique sources at test time: `1867`
|
| 12 |
+
- Source family: public knowledge files already ingested locally (wiki-based IT/EN and related public text sources).
|
| 13 |
+
|
| 14 |
+
## Engine Configuration
|
| 15 |
+
- Runtime: `python app_oracolo.py` (CLI mode)
|
| 16 |
+
- Embedding backend used in test: `sentence-transformers(local-path): aio_models/paraphrase-multilingual-MiniLM-L12-v2`
|
| 17 |
+
- Retrieval path: AIO native hybrid retrieval (vector + lexical overlap + domain coherence bonus).
|
| 18 |
+
|
| 19 |
+
## Test Procedure
|
| 20 |
+
1. Internal benchmark run:
|
| 21 |
+
- command: `AIO_CLI_BENCH=1 python app_oracolo.py`
|
| 22 |
+
2. Real use-case runs (8 cases):
|
| 23 |
+
- command per case: `AIO_CLI_TEST_QUERY=1 AIO_CLI_QUERY="<query>" python app_oracolo.py`
|
| 24 |
+
3. Parse output:
|
| 25 |
+
- `hits`, `score`, `domain`, `source`
|
| 26 |
+
4. Case pass criteria:
|
| 27 |
+
- minimum hits threshold (case-specific),
|
| 28 |
+
- top score threshold,
|
| 29 |
+
- at least one expected term matched in retrieved context.
|
| 30 |
+
|
| 31 |
+
## Published Artifacts
|
| 32 |
+
- `REAL_CASE_TEST_RESULTS_LATEST.json`
|
| 33 |
+
- `REAL_CASE_TEST_RESULTS_LATEST.md`
|
| 34 |
+
- timestamped copies (`REAL_CASE_TEST_RESULTS_20260221_222608.*`)
|
| 35 |
+
|
| 36 |
+
## Licensing Note
|
| 37 |
+
- Free for use and study.
|
| 38 |
+
- Commercial use requires written authorization: `info@rthitalia.com`
|