A Reproducible Benchmark for Search Intent Classification

1. Introduction and research question

Search intent classification assigns a query to the user’s underlying information need, such as navigation, product evaluation, comparison, transactional action, or exploratory search. In practice, benchmark quality often determines whether reported model differences reflect genuine generalization or only dataset-specific annotation habits. The present manuscript addresses a narrow methodological question: how should a reproducible benchmark for search intent classification be specified so that model comparisons remain meaningful across languages, domains, and annotation budgets? The scope is intentionally limited to benchmark design rather than empirical performance claims. The article offers a targeted narrative synthesis of prior work that is relevant for benchmark construction, with two evidence groups. The first group concerns text classification studies that demonstrate how transformer-based models, domain adaptation, and interpretability can support cross-domain evaluation in sentiment tasks. The second group concerns methodological work on search-guided model selection, standardized data pipelines, and virtual-to-real transfer design, all of which can be adapted to benchmark construction without treating them as direct intent-classification studies. [1] [2] [3] [4] [5] [6]

The benchmark problem is motivated by two common weaknesses in existing intent datasets.The target use case is a benchmark whose portability beyond its original language and platform has not been established. Second, labels are often treated as self-evident categories rather than operational definitions derived from query purpose, so that the same surface form can be assigned different labels across annotation teams. A reproducible benchmark must therefore define the unit of analysis, the admissible labels, the annotation decision rule, and the evaluation regime before any model is trained. This protocol treats those elements as the core object of the benchmark rather than a secondary description of data preparation.

2. Evidence scope and transferable lessons from prior work

The direct application evidence comes from multilingual and cross-domain sentiment classification. EMSA shows that transformer-based language models can be structured with explicit reasoning steps to improve interpretability while retaining classification performance across languages and domains; its relevance here lies in the use of a two-stage reasoning-to-label pipeline and the emphasis on transparent decision paths. SentXFormer demonstrates that transformer embeddings combined with domain adaptation can support cross-domain transfer when domain shift is a primary threat to validity. A RoBERTa-based consumer-insight pipeline shows how attention-based representations can be paired with SHAP and LIME to produce local and global explanations, which is relevant for checking whether intent labels are based on evidence in the query text rather than on dataset artifacts. [1] [2] [3]

The methodological foundations are more directly transferable. The search-guided regression ensemble proposes a principled way to choose among candidate learners by using a forward or backward search procedure under a fixed optimization budget and with uncertainty-aware evaluation. For benchmark design, the transferable idea is not the regression setting itself, but the rule that model selection should be explicit, budgeted, and reproducible rather than improvised after inspection of test results. The reproducible dataset-generation pipeline in building control research provides the stronger benchmark analogue: it combines standardized generation, a common interface across heterogeneous simulators, and full code release for comparisons under fixed seeds. The virtual-to-real framework adds a second transferable principle: train on a controlled synthetic source and test on a distinct real target when annotation scarcity makes standard supervised evaluation too narrow. In intent classification, these principles support a benchmark that includes both naturally collected and synthetic or weakly labeled queries, while keeping their provenance separated. [4] [5] [6]

3. Benchmark objective and label space

The proposed benchmark is intended to answer three questions. First, can an intent classifier identify a query’s primary goal from query text alone? Second, does the classifier remain stable when the same intent is expressed across languages, domains, or orthographic variants? Third, how much uncertainty does the classifier exhibit on ambiguous queries? To support these questions, the benchmark uses a hierarchical single-label scheme with an explicit ambiguity state. Level 1 contains four mutually exclusive classes: navigational, informational, transactional, and comparative. Exploratory search is not a separate Level 1 class. Instead, it is recorded as an optional Level 2 subtype within informational or comparative intent when the query explicitly seeks options, ideas, or open-ended discovery. A query receives one Level 1 label in the main benchmark or is assigned to the ambiguity set; the protocol does not use partially multi-label annotation.

Operational definition of intent: a query q belongs to class c when annotators can justify c from the query text alone by reference to the user action most directly expressed in q. Queries whose interpretation requires session history, click data, or hidden task context are not assigned by inference; they are placed in the ambiguity set unless the text itself supports a single class. The default annotation design uses two primary annotators and one adjudicator. Each primary annotator assigns exactly one Level 1 label and a confidence score on a fixed ordinal scale {1,2,3,4}, where 1 denotes low confidence and 4 denotes high confidence. All disagreements are adjudicated under the text-only codebook without restricting the adjudicator to the initially proposed classes. Ordinal confidence is recorded for uncertainty description and review priority, not summed into a cardinal vote. The adjudicator may choose any admissible class or ambiguity. Agreement is provisional until the independent agreed-case audit is complete. This defines one consistent workflow across the task definition and construction stages. Ambiguity coverage is reported alongside accuracy so that selective exclusion cannot masquerade as general classification success.

4. Proposed data construction protocol

The benchmark should distinguish core evaluation data from auxiliary stress-test data. The core benchmark consists of naturally observed queries collected from organic search traffic or domain-specific query logs where access and disclosure are permitted. Synthetic or templated queries are not part of the primary score-reporting test set. They form a separate stress-test source used only for coverage analysis, robustness checks, or synthetic-to-real transfer experiments. Following the logic of virtual-to-real evaluation, synthetic queries may be used as an auxiliary source split, while naturally observed queries remain the only default target for the main benchmark claims. Each record in any split should contain at minimum the raw query text, language tag, domain tag, source provenance, timestamp bucket if available, annotation status, and split identifier. No click, dwell-time, or session feature is required for the core benchmark, because portability depends on a schema that is available across collection settings. [6]

Two annotators independently assign a label or ambiguity using the frozen intent codebook, with an ordinal confidence category and a text-grounded rationale. Agreement fixes the provisional reference label. Every disagreement goes to an adjudicator, who may choose any admissible label or ambiguity after inspecting the query and both rationales. Confidence categories order review priorities and describe annotator uncertainty; they are not added as interval-scale votes and cannot constrain the adjudicator to a majority label. A random agreed-case sample is also audited to detect shared misunderstanding. Ambiguous queries are retained in a separate abstention evaluation with coverage reported; they are not silently erased from the benchmark population. No final test label is changed in response to model errors. Codebook changes produce a new benchmark version and require a fresh held-out assessment.

5. Experimental design for benchmark validation

The benchmark should include a fixed comparator set rather than an unrestricted model list. The minimum comparator family consists of: a sparse lexical baseline such as linear classification over n-grams; one transformer fine-tuned on the benchmark training split; one domain-adapted transformer when cross-domain data are available; and one search-guided selection baseline used as a reproducibility control. The search-guided comparator is included because prior work shows that learner selection can be framed as an explicit forward or backward search under a fixed optimization budget. In this benchmark, that procedure must be completely prespecified before any test evaluation: candidate model pool, tuning space for each candidate, common random seeds, number of training restarts, maximum number of validation evaluations, search order, stopping rule, and tie-break rule. A default budget rule is recommended: each candidate receives the same number of hyperparameter trials and the same number of seed repeats, and the search procedure may compare only validation summaries computed from those equal budgets. [4]

A recommended decision sequence is as follows. Step 1: train each candidate model on the training split with fixed preprocessing, fixed seeds, and the same tuning budget. Step 2: evaluate each tuned candidate on the validation split. Step 3: select the single final model, or the final search-guided ensemble if that comparator is used, by a lexicographic rule fixed in advance: maximize macro-F1; if models differ by no more than a preregistered tolerance on macro-F1, choose the model with lower expected calibration error; if still tied, choose the model with lower selective risk at a fixed abstention rate; if still tied, choose the computationally simpler model as defined by fewer trainable parameters or, for linear models, fewer active features. Step 4: freeze all parameters, thresholds, and abstention settings. Step 5: evaluate once on the test split. The benchmark prohibits any test-set-driven model selection or threshold adjustment. This separation follows reproducible pipeline principles based on a common interface, controlled seeds, and identical tuning budgets across candidate methods. [5] [4]

The primary outcome measures are macro-F1, per-class F1, expected calibration error, abstention-aware risk, and label stability under perturbation. Perturbation testing must use a fixed, released rule set rather than ad hoc edits. The default perturbation suite contains only transformations intended to preserve query meaning: punctuation insertion or removal, case variation, whitespace normalization, reversible transliteration where a verified mapping table exists, and constrained spelling variants from a prespecified dictionary. Word-order changes are excluded from the default suite because meaning preservation cannot be guaranteed uniformly across languages. Label stability is the proportion of perturbed variants whose predicted Level 1 label matches the original prediction for the same query. A secondary diagnostic is explanation alignment. For each query, annotators mark the minimal intent-bearing tokens, if any, during annotation. An explanation method is counted as aligned when at least one marked token appears in the top-k attributed tokens under a fixed k chosen before evaluation. This diagnostic is motivated by prior use of SHAP and LIME, but it is not treated as proof that the prediction is correct. [3] [1]

6. Validation regimes and prospective analyses

The validation design should contain four regimes. In-domain evaluation tests whether a model can classify queries drawn from the same distribution as the training set. Cross-domain evaluation tests transfer across query sources such as e-commerce, local services, academic search, and help-center search. Cross-lingual evaluation tests whether the label definitions remain stable across languages, with separate analysis for languages sharing script and those requiring transliteration or translation. Scarcity evaluation reduces the number of labeled training examples and assesses whether the benchmark still discriminates between models when annotation is limited. The last regime is important because intent datasets in practice are often much smaller than sentiment corpora used for large-scale transfer experiments. [1] [2] [6]

A reproducible benchmark should also include a synthetic-to-real regime as a separate auxiliary experiment. In that setting, models may be trained on controlled synthetic queries that cover rare phrasings or underrepresented languages and then evaluated on a real query set with human labels. Scores from this regime must be reported separately from the core benchmark results. Its role is to test transfer under controlled source construction, not to enlarge the main test set or to justify claims about natural-query performance. This design is most useful when rare intents are sparsely observed, because it offers a bounded test of extrapolation without assuming that synthetic queries are distributionally equivalent to real search logs. If a model improves only on synthetic validation but not on real evaluation, the benchmark should record a transfer failure in that auxiliary regime. [6]

Prospective analysis should include subgroup reporting for short queries, code-mixed queries, ambiguous queries, and queries with named entities. These strata are not intended as separate benchmarks; they are diagnostic slices. The reason for keeping them separate is that failure modes differ. Short queries may lack enough lexical evidence for reliable class assignment. Code-mixed queries may be misclassified because language identification is unstable. Ambiguous queries can appear inaccurate even when the model identifies a plausible secondary intent. Named-entity-heavy queries may overfit to surface brand or product names rather than to intent structure. A useful benchmark should make these errors visible instead of averaging them away.

7. Comparison with prior methods and design boundaries

Compared with the multilingual sentiment frameworks, this protocol does not assume that one transformer architecture should be preferred in all settings. Instead, those studies justify two benchmark components: explicit evaluation under language or domain shift, and a defined place for explanation methods in diagnostics. EMSA is relevant because it structures classification through transparent intermediate reasoning stages; however, search intent benchmarking should not require chain-of-thought outputs, because the protocol must remain model-agnostic. SentXFormer is relevant because it combines contextual embeddings with domain adaptation, which supports inclusion of a transfer regime. The RoBERTa explainability study is relevant because it operationalizes token-level explanations through SHAP and LIME. These precedents motivate benchmark design choices, but they do not establish the validity of the proposed benchmark on their own. [1] [2] [3]

Compared with the reproducible data-generation pipeline, this protocol extends the idea of a standardized interface from simulator outputs to text annotations. The main difference is that the benchmark here cannot rely on deterministic simulation alone, because language data contain annotation noise and pragmatic ambiguity. For that reason, reproducibility must be sought through explicit provenance, fixed split construction, and documented adjudication rather than through complete determinism. Compared with the virtual-to-real framework, the benchmark also differs in that the synthetic source is auxiliary rather than dominant. Synthetic queries are included to probe rare coverage and transfer, but the core scientific question remains whether models generalize to real search intent distributions. This boundary prevents synthetic coverage from being mistaken for external validity. [5] [6]

A further boundary condition concerns label taxonomy. Some search tasks can use very fine-grained intent labels, but increasing label depth can reduce reproducibility if the boundaries are not stable across annotators or languages. The protocol therefore recommends starting with a coarse label set and adding subclasses only where the subclass decision rule can be stated without reference to corpus-specific examples. This rule is conservative, but it reduces the risk that the benchmark becomes a one-off dataset rather than a reusable standard.

8. Failure cases and limits of generalization

The benchmark will fail if the label schema encodes platform-specific assumptions that do not generalize across search environments. It will also fail if annotators are asked to infer hidden user context that is unavailable in all deployment settings, because then the labels cannot be reproduced from query text alone. Another failure case is class leakage through source metadata. If a query source is strongly correlated with a class, a model may appear accurate while learning provenance rather than intent. The protocol therefore requires source balancing or source masking where feasible. A final failure case arises when the ambiguity set becomes too large, which indicates that the taxonomy is not adequately grounded in observable text. In that case, the benchmark should be revised rather than force-fit labels onto weak evidence.

Generalization is limited in at least three ways. First, the benchmark is designed for query-text classification and does not by itself resolve multimodal intent signals from voice, session context, or click behavior. Second, the protocol is suitable for languages and domains where annotated text is available, but it cannot remove the need for local validation in low-resource settings. Third, the proposed uncertainty and explanation measures are diagnostic; they do not guarantee causal interpretability or user trust. These limits are deliberate, because a reproducible benchmark should specify what is being measured rather than claim to measure all aspects of search behavior.

Attribution alignment and bounded interpretation

For attribution comparisons, normalize text to Unicode NFC and retain a reversible mapping to original character offsets. Freeze the multilingual tokenizer and attribution method before evaluation. Aggregate subword contributions to contiguous word spans by signed sum, then rank spans by absolute contribution with start-offset tie breaking. Cap top-k at the number of valid spans. An all-zero attribution vector or an unmappable span yields an unavailable alignment metric rather than perfect agreement. Compare human rationale spans with these offset-aligned spans using the registered overlap rule, and report alignment coverage by language. Rationale overlap is an interpretability diagnostic; it is not evidence of causal faithfulness or proof that the model used the same reasoning as an annotator. Changes in segmentation can change the metric, so tokenizer and offset-mapping versions accompany every score.

9. Conclusion

This manuscript specifies a conceptual benchmark protocol for search intent classification. Its methodological contribution is bounded: it defines a hierarchical single-label taxonomy with an explicit ambiguity state, a text-only annotation rule with fixed adjudication, a split strategy that separates core natural-query evaluation from auxiliary synthetic stress tests, a fixed comparator family, and prespecified validation regimes for in-domain, cross-domain, cross-lingual, and low-annotation conditions. The literature synthesis supports three benchmark choices: include transformer-based models among the comparators, evaluate transfer under domain shift, and treat uncertainty and explanation as diagnostic outputs rather than as evidence of correctness. The protocol also borrows two methodological principles from prior work: standardized pipelines with fixed seeds and equal budgets, and synthetic-to-real transfer testing reported as a separate auxiliary regime. [1] [2] [3] [4] [5] [6]

What remains untested is whether this specification will be practical and stable when implemented by independent annotation teams and benchmark users. The taxonomy has not yet been applied in a completed benchmark release, the adjudication rule and ambiguity handling have not yet been assessed on real annotation disagreements, and the proposed comparator set has not been run under the full protocol described here. The next step is a controlled implementation study that keeps the specification fixed in advance, records adjudication outcomes and ambiguity rates, and reports predictive, calibration, abstention, and perturbation-stability outcomes without revising the benchmark after inspection of test results. Such a study could determine whether the protocol is suitable for later use as a reusable benchmark specification.

References

1. Zhao, Li; Zhou, Jinwei; Cao, Jinde; Zhu, Weina (2025). EMSA: Explainable multilingual sentiment analysis models providing sentiment analysis across multiple languages. PLOS One. 20(11), e0333508. doi:10.1371/journal.pone.0333508.
2. Kumar, Ajeet; Abhishek, Kumar; Shafeeq B M, Ahamed (2025). SentXFormer: a transformer-enhanced hybrid deep learning framework for cross-domain sentiment analysis of customer reviews. Scientific Reports. 16(1), 3528. doi:10.1038/s41598-025-33526-1.
3. Shasha, Qi (2026). Analyzing digital consumer insights through RoBERTa LLM based sentiment analysis and topic modeling. Scientific Reports. 16(1), 18921. doi:10.1038/s41598-026-48162-6.
4. Chen, Lifei; Lim, Zhi Min; Lim, Wei Hong; Tiang, Sew Sun; Sharma, Abhishek; Syamsunur, Deprizon; Alharbi, Amal H.; Eid, Marwa E.; et al. (2026). Search-guided regression ensembles for accurate, interpretable, and uncertainty-aware construction cost estimation. Scientific Reports. 16(1), 21456. doi:10.1038/s41598-026-51706-5.
5. Domingo, Ferran Aran; Alonso, Pablo Fraile; Torrentó, Josep Rius; Batalla, Oriol Agost; Vilardell, Ignasi Barri; Mayoral, Jordi Vilaplana; Fornés, Jordi Mateo (2026). A reproducible method to generate multi-building, multi-climate HVAC operation datasets with a stochastic exploratory controller. MethodsX. 16, 103866. doi:10.1016/j.mex.2026.103866.
6. Zou, Yiquan; Liang, Tianxiang; Nabi, Jafri Syed Riaz un; Xu, Zhendong; Zhou, Liang; Xiong, Biao (2026). A BIM-Guided Virtual-to-Real Framework for Component-Level Semantic Segmentation of Construction Site Point Clouds. Sensors. 26(1), 308. doi:10.3390/s26010308.