Datasets, Models & Tools — Practical Guide
Everything you need to actually run the experiments, with links and honest caveats.
1. Datasets
1.1 Amazon ESCI — Shopping Queries Dataset ⭐ methodological template
- What: 130K hard queries, 2.6M human (query, product) judgments, ESCI labels (Exact/Substitute/Complement/Irrelevant). EN/JA/ES only — no Arabic.
- Get it: https://github.com/amazon-science/esci-data · Paper PDF in repo (
esci-shopping-queries-amazon-2022.pdf) - Use for: (a) annotation methodology & label definitions; (b) optional machine-translated Arabic version (spot-check 20% by hand; disclose MT in thesis).
1.2 Wayfair WANDS ⭐ best annotation-guideline template
- What: 480 furniture queries, 43K products, 233K judgments, labels Exact/Partial/Irrelevant, MIT license, includes a public annotation-guidelines PDF.
- Get it: https://github.com/wayfair/WANDS · Paper: ECIR 2022 (Springer page, paywalled — dataset itself is open)
1.3 prestoai/arabic-ecom-search-bench — audited 2026-08-22 ⭐ Track A
Verdict: genuinely usable, with caveats — do not bet the thesis on it alone.
| Aspect | Finding |
|---|---|
| Format | Proper TREC-style IR benchmark: queries.jsonl + corpus.jsonl + qrels.tsv → plugs into ranx/pytrec_eval directly |
| Scale | 29,014 queries · 107,041 products · 262,599 judgments |
| Labels | Graded 0–3 plus −1 hard negatives (similar-but-wrong) — rare; enables dense-retrieval failure-mode analysis |
| Baselines included | Published Meilisearch run: nDCG@10 = 0.624, Recall@10 = 0.483, MRR@10 = 0.756 (report.json) — immediate number to beat |
| Dialect | Explicit MSA + Libyan dialect tags → feeds RQ5 |
| Tooling | Ships evaluate.py + Meilisearch sync example |
Red flags
- No methodology disclosure: judgments' provenance unknown (human? click logs? LLM-assisted?). Sample qrels look machine-clustered. No peer-reviewed paper — only a
@miscHF citation (authors: Okasha, Naji, Badi). - No license file in repo — email authors before redistributing derivatives.
- Corpus is titles-only (
item_id,name) — no descriptions/categories/brands; thinner text than ESCI/WANDS for hybrid experiments. - One-dialect skew (Libyan) — case-study value, not pan-Arabic coverage.
- Very new (Apr 2026), ~274 downloads — zero community validation.
Action: email authors early asking (a) judgment provenance, (b) redistribution permission. Their answer feeds the dataset chapter either way.
1.4 General Arabic retrieval benchmarks (sanity-check layer)
Not e-commerce, but standard enough that reviewers will expect you to know/use them:
| Benchmark | Arabic contents | Use in thesis | Paper |
|---|---|---|---|
| Mr. TyDi-ar | 7,327 train/dev/test queries over ~2M Wikipedia passages; dense-retrieval benchmark | Sanity-check embedding models & BM25 settings before product experiments | 📄 research/papers/benchmarks/mr-tydi-multilingual-dense-retrieval-2021.pdf (data) |
| MIRACL-ar | Denser annotations than Mr. TyDi (train 3,495 q / 25K judgments; test-A 936 q) | Same role, higher-quality qrels; Apache-2.0 | 📄 research/papers/benchmarks/miracl-multilingual-retrieval-tacl2023.pdf (data) |
| mMARCO-ar | MT-translated MS MARCO passage ranking | Optional third sanity set; disclose MT caveat | 📄 research/papers/benchmarks/mmarco-multilingual-msmarco-2022.pdf |
| TREC AFP 2001–2002 | Classic Arabic news ad-hoc collection | Historical baseline citations only | via Darwish TREC paper (already in library) |
Positioning sentence for thesis: "Arabic retrieval benchmarks exist for open-domain QA-style search (Mr. TyDi, MIRACL), but none targets e-commerce product search with catalog semantics — we fill that gap." This keeps the novelty claim honest while showing command of the literature.
1.5 Other corpus sources (no relevance labels — index material only)
- milistu/AMAZON-Products-2023-Arabic (HF, ~117K rows): real Amazon Arabic product catalog — good bulk corpus to scale ES indexing/load tests beyond the titles-only bench.
- Masader catalogue (https://arbml.github.io/masader/) — searchable index of 1,100+ Arabic NLP datasets; sweep it once when finalizing data choices.
- Commercial scrapers exist for Noon.com / Amazon.ae (e.g., Apify actors) — usable for a corpus if ToS-respected and disclosed, but do not build the thesis on scraped judgment-free data alone.
1.6 Building your own slice (Track B — primary artifact)
- Write 300–500 realistic queries across types: MSA head terms, MSA long-tail, dialect variants (Egyptian/Gulf/Levantine — deliberately beyond the Libyan-only bench), brand transliterations, mixed AR/EN.
- Recruit 2–3 annotators; adopt WANDS guidelines verbatim (translated); compute Cohen's κ ≥ 0.7.
- Judge against candidates pooled from multiple system runs (BM25 + dense + LLM-expanded) so no single retriever biases the pool.
- This becomes the Dataset chapter + citable artifact, independent of any external group's responsiveness.
2. Embedding Models (dense retrieval side)
| Model | Params | Why | Link |
|---|---|---|---|
| GATE (Arabic-Triplet-Matryoshka-V2) | 135M | SOTA Arabic STS on MTEB; Matryoshka dims 64–768 | https://huggingface.co/omernacar (see paper arXiv:2505.24581) |
| AraBERT sentence-transformers finetunes | 135M | widely used baseline | https://huggingface.co/hindalmayyali/sentence-transform-arabert |
| MARBERT (+NLI triplet variants) | 163M | dialect coverage | https://huggingface.co/UBC-NLP/MARBERT |
| multilingual-e5-base/large | 278M/560M | strong multilingual fallback | https://huggingface.co/intfloat/multilingual-e5-large |
| OpenAI/Cohere/Voyage embeddings | API | commercial baselines | — |
Selection protocol: rank candidates on an Arabic STS/retrieval dev set (MTEB-Arabic subset) before wiring into ES.
3. LLMs (expansion engine candidates)
| Model | Access | Size | Notes |
|---|---|---|---|
| ALLaM-7B-Instruct | Apache-2.0, GGUF/Ollama | 7B | Saudi national model; strongest Arabic cultural/linguistic scores among open 7Bs |
| Jais / Jais-chat | Apache-2.0 | 13B/70B | UAE; good dialect handling |
| AceGPT-chat | Apache-2.0 | 7B/13B | KAUST; culturally aligned; unmaintained since 2023 |
| Qwen2.5-Instruct | Apache-2.0 | 0.5–72B | excellent multilingual; weaker Arabic-specific nuance |
| Command-R / Aya-23 | research licenses | 8–35B | multilingual incl. Arabic |
| GPT-4o-mini / Claude Haiku / Gemini Flash | API | — | commercial upper bounds + latency/cost references |
Local serving: Ollama (fastest start) or vLLM (throughput). Quantize to Q4/Q5 for laptop experiments.
4. Arabic NLP Tooling
| Tool | Use | Link |
|---|---|---|
Elasticsearch arabic_normalization + stemmer(language:arabic) |
built-in lexical normalization/stemming | ES analyzer docs |
| CAMeL Tools | MSA/dialect processing, diacritization, utilities | https://github.com/CAMeL-Lab/camel_tools |
| Farasa | segmentation, diacritization (QCRI) | http://farasa.qcri.org |
| Elsner/Awesome Arabic datasets lists | finding more corpora | https://github.com/omarkh/all-arabic-datasets |
5. Evaluation Tooling
| Task | Tool |
|---|---|
| Ranking metrics (NDCG/MRR/Recall) | ranx (pip install ranx) or pytrec_eval |
| Significance tests | paired bootstrap/randomization (implement ~50 lines) or scipy.stats.wilcoxon on per-query deltas |
| Annotation UI | Label Studio self-hosted |
| Load/latency testing | Locust / k6 |
| Experiment tracking | plain CSV + git, or MLflow if you want polish |
6. Infrastructure Notes
- Elasticsearch free basic license includes dense vectors and the
rrfretriever (RRF technically flagged enterprise — client-side RRF viaranxis the documented workaround; OpenSearch has native RRF free). - Keep MySQL and ES on Docker Compose locally; one
make indexcommand rebuilds everything for reproducibility. - Version-pin: ES version, model weights (by SHA), prompts (in git), dataset snapshot date — examiners love reproducibility sections.