Evaluating Hallucinations in Product Recommendation Explanations: A Conceptual Protocol for Explanation Fidelity, Uncertainty, and Traceability

1. Introduction

Product recommender systems now routinely produce explanatory text to justify item rankings or single-item recommendations. In this setting, the central methodological problem is not whether the recommendation is accurate, but whether the accompanying explanation truthfully reflects the factors that produced it. A hallucinated explanation may mention an item feature that was never used, overstate the role of a user preference not present in the input, or infer a causal relation that the model does not support. The relevant research question is therefore narrower than generic natural-language faithfulness: how can one evaluate hallucinations in product recommendation explanations with a protocol that distinguishes grounded explanations from plausible but unsupported narratives? [1] [2]

This manuscript is a research protocol rather than an empirical report. It uses three transferable methodological ingredients from prior work: local attribution methods such as SHAP and LIME, uncertainty-aware prediction with calibrated intervals, and feature-effect analysis for models that combine structured and unstructured inputs. Recommendation-domain studies are used only to specify the kinds of inputs and user-facing explanation displays that a future evaluation may encounter. The proposed protocol therefore treats prior sources as methodological precedents for interpretability, uncertainty, and traceable recommendation inputs, not as prior tests of explanation hallucination. [3] [4] [1] [5]

2. Evidence scope and rationale for a targeted narrative synthesis

The evidence base is intentionally selective. The aim is not to survey all recommendation or explanation research, but to assemble a minimal basis for a reproducible hallucination-evaluation protocol for recommendation explanations. The most relevant sources are those that operationalize local attribution with SHAP or LIME, those that connect predictions to calibrated uncertainty, and studies that expose traceable recommender inputs or user-facing explanation displays. One consumer-review study applies SHAP and LIME to textual product-related inputs, showing that local token attributions can be inspected beside predictions. An ensemble-estimation framework combines SHAP with calibrated prediction intervals, which is useful for keeping predictive uncertainty distinct from explanation content. A real-estate valuation study shows how SHAP can inspect feature effects in a task with mixed structured and image-derived inputs. Two recommender studies provide application context: one uses review, interaction, and hybrid representations for product recommendation, and another combines profile, academic, and personality information and presents an explanation on a user dashboard. [3] [2] [4] [1] [5]

This narrow scope reflects the main requirement of the protocol: a precise scoring rule that another researcher could implement without adding unstated assumptions. The protocol adopts, as prospective assumptions rather than established findings, three propositions. First, local attribution methods can serve as proxy signals about influential inputs and can therefore support a separate alignment check. Second, uncertainty estimates can indicate when explanatory language should be hedged rather than categorical. Third, recommendation systems built from explicit profiles, metadata, review text, or comparable traceable fields permit grounding checks against the evidence visible to the system. What the cited studies do not provide is a claim-level rule for deciding whether explanation text is supported by that evidence. That rule is proposed here. [3] [2] [1] [5]

3. Conceptual definition of hallucination in recommendation explanations

For this protocol, the unit of analysis is the atomic claim. A recommendation explanation is any textual or structured statement that purports to justify why a product was recommended, and an explanation may contain zero or more atomic claims. Hallucination is defined as the presence of one or more factual atomic claims that are unsupported or contradicted by the recommendation evidence record. The evidence record is limited to information that the recommender could actually access for that instance: user attributes explicitly used by the model, item attributes explicitly used by the model, interaction or review evidence explicitly used by the model, and any retrieved records made available to the model at generation time. SHAP, LIME, or comparable local attribution outputs are not part of this evidentiary record. They are auxiliary proxy signals used only in a second evaluation channel that tests alignment between the explanation narrative and influential inputs. Unsupported claims include invented attributes, fabricated comparative statements, wrong feature polarity, incorrect statements about available evidence, and causal assertions that cannot be grounded in the visible record. [4] [3] [5]

The operational rule separates grounding from alignment. Let H be the set of atomic claims extracted from an explanation, and let E be the set of evidence-supported propositions derived only from the instance-level evidence record. For each claim h in H, assign one grounding label: supported if h is directly entailed by E; contradicted if h conflicts with E; unsupported if h is neither entailed by nor contradicted by E; or hypothetical if the text explicitly marks h as conjectural, for example with forms such as "may," "might," or "could," and the sentence does not present the content as fact. Grounding labels are assigned without using SHAP or LIME. A separate alignment label is then assigned only to grounded reason-claims, using the mapping rule defined later in the protocol. The explanation is classified as hallucinated when it contains at least one contradicted or unsupported claim presented as factual. If H is empty, the explanation is classified as zero-claim and non-informative. Zero-claim explanations are excluded from claim-level hallucination rates but included in system-level reporting through a separate zero-claim proportion. [3] [2]

4. Proposed variables and operationalization

The protocol evaluates four variable groups. The first is instance-level evidence variables: every user field, item field, review-derived field, interaction-derived field, retrieved field, and engineered feature that is available to the recommender for the focal recommendation. The second is model-output variables: recommendation score, rank, or class label; a scalar uncertainty estimate; and the generated explanation text. The third is attribution variables: SHAP values, LIME weights, or comparable local contributions mapped to the same interpretable input fields used in the evidence record. The fourth is verification variables: for each atomic claim, a grounding label and, where applicable, an attribution-alignment label. A claim is eligible for alignment checking only if it refers to one or more interpretable input fields that can be matched deterministically to attribution variables. [3] [2] [1] [5]

The primary outcome is a claim-weighted hallucination score for each explanation. Let S, U, and C denote the counts of supported, unsupported, and contradicted factual claims, after excluding claims explicitly marked as hypothetical. Let T = S + U + C. If T = 0, the hallucination score is undefined and the explanation is recorded as zero-claim. Otherwise define HS = (U + 2C) / (S + U + 2C), which penalizes contradicted claims more heavily than unsupported claims. The default preregistered decision rule is to classify an explanation as unacceptable when HS > 0.20. Because no benchmark is established here, later empirical work should run a prespecified sensitivity analysis over thresholds from 0.10 to 0.30 in increments of 0.05. Secondary outcomes are contradiction rate C/T, unsupported-claim rate U/T, and zero-claim rate at system level. The protocol also defines an attribution-alignment rate for grounded reason-claims. To compute it, first map each reason-claim to one or more canonical feature labels by exact matching to a preregistered dictionary of interpretable fields and synonyms. Then rank features for the instance by absolute attribution magnitude after aggregating one-hot encodings or token-level attributions to the same canonical label. A grounded reason-claim is aligned under the default rule if at least one mapped canonical feature appears in the top k attributed features for that instance, with k = 3 preregistered as the default and k in {1, 5} reported as sensitivity alternatives. Claims referring only to generic recommendation quality, stylistic praise, or user benefit without a mappable feature label are coded not-applicable for alignment rather than misaligned. When several correlated fields are grouped into one canonical label, the group is treated as aligned if the summed absolute attribution for the group falls within the top k groups. [3] [4] [2]

5. Proposed evaluation workflow

The evaluation workflow has five stages. Stage 1 segments the explanation into atomic claims under a deterministic rule. Each finite clause expressing a distinct factual proposition is one claim; conjunctions are split when the conjuncts can differ in truth value; comparative statements are split into the compared attribute and the direction of comparison; and causal statements are split into an observed premise and a causal conclusion when each part can be judged separately. Stage 2 assigns grounding labels by comparing each claim only with the instance-level evidence record. Stage 3 performs attribution alignment for the subset of grounded reason-claims that can be mapped to canonical features. Under the default rule, alignment is binary and depends on whether at least one mapped feature is among the top three canonical attribution groups by absolute magnitude for that instance. Stage 4 evaluates uncertainty wording. If the instance is designated low confidence under the rule in Section 6, categorical expressions such as "because" or "this proves" are coded as overconfident unless the explanation includes an explicit hedge. Stage 5 aggregates claim-level labels into explanation-level and system-level summaries, with zero-claim explanations retained in system-level denominators for reporting frequency. [3] [2] [5]

The comparator set should include three explanation modes. The first is an evidence-constrained template baseline that may restate only fields present in the instance-level evidence record. The second is a free-form generation baseline without explicit evidence constraints. The third is the proposed evidence-grounded method, which first generates candidate claims, removes any claim that is unsupported by the evidence record, and then applies the attribution-alignment and uncertainty checks before release. An optional fourth comparator is an attribution-only display that lists the top attributed canonical features without a full sentence explanation. This comparator set isolates two different errors: factual hallucination and low explanatory coverage. The protocol does not score incompleteness as hallucination, but keeping the evidence-constrained baseline makes that distinction observable in later testing. [3] [2] [4]

6. Validation design and prospective testing plan

Because this article is a protocol, the validation design is prospective. Stage 1 should use hand-constructed cases in which the accessible evidence record and the correct claim labels are fixed in advance. These cases can test whether the scoring procedure detects invented features, wrong polarity, unsupported comparisons, unjustified causal wording, and overconfident language under low confidence. Stage 2 should use held-out recommendation instances from a product domain with explicit feature records and, if applicable, stored retrieval traces, so that each explanation can be checked against what the recommender actually saw. Stage 3 should compare human annotation with the operational score at the claim level, emphasizing agreement on supported, unsupported, contradicted, and hypothetical labels. No validation stage is reported here as completed. [4] [1] [5]

One prospective validation question is whether prediction uncertainty marks conditions under which explanation hallucination becomes more likely. Each recommendation instance should therefore receive one preregistered scalar uncertainty estimate before explanation generation. When calibrated prediction intervals are available, define normalized interval width as (upper bound - lower bound) divided by the width between the 95th and 5th percentiles of the model score on a preregistered development split; truncate values above 1 only for descriptive plots, not for classification. The default rule labels an instance low confidence when this normalized width exceeds the 80th percentile of the same development split. If prediction intervals are unavailable, the study must preregister one monotone scalar uncertainty measure produced by the recommender, such as ensemble variance or predictive entropy, and classify low confidence by the same percentile rule on the development split. This common percentile rule keeps the decision threshold comparable across uncertainty measures even though their raw scales differ. Later empirical work can then test whether low-confidence instances show higher hallucination scores or higher overconfidence-in-language rates than other instances. The same stage should test whether the attribution-alignment channel detects explanation drift, defined here as divergence between grounded reason-claims and the instance's highest-magnitude attributed feature groups. [3] [2]

7. Comparison with prior methodological foundations

The protocol differs from the consumer-review interpretability study in the role assigned to SHAP and LIME. In that study, these methods explain sentiment-model predictions by highlighting influential tokens or features. Here they are not treated as proof that a factual claim is grounded. They are used only after grounding has already been judged from the instance-level evidence record, and only to test whether the explanation emphasizes inputs that attribution methods also mark as influential. The protocol therefore reuses local interpretability tools for a narrower verification task without claiming that they recover the model's true causal reasoning. [3]

The protocol also adapts the uncertainty-aware ensemble literature by separating predictive reliability from explanation reliability. In that literature, SHAP and prediction intervals are combined to support interpretable and uncertainty-aware estimation. Here, calibrated intervals or another preregistered scalar uncertainty measure serve a different function: they determine when the explanation should be hedged and provide a variable for later testing of whether low-confidence cases have higher hallucination risk. Uncertainty is therefore not a substitute for evidence grounding or attribution alignment. It is a boundary condition on how strongly a generated reason may be stated. [2]

The real-estate valuation study shows that SHAP can be used to inspect local feature effects in a prediction task with heterogeneous inputs, including image-derived features. That point transfers at the level of method rather than domain: when inputs come from multiple sources, instance-level feature contributions can still be summarized in an interpretable space. Recommendation explanations add a further requirement because they are user-facing justifications rather than analyst-facing diagnostics. The present protocol therefore adds claim extraction, a strict separation between evidence grounding and attribution alignment, a deterministic mapping from claims to canonical features, and explicit aggregation rules. The degree-program recommender provides an example of a system that displays an explanation to the user and that relies on traceable profile, academic, and personality inputs. The sentiment-aware product recommender provides product-domain context in which review text, interactions, and hybrid representations contribute to recommendations. The bounded contribution claimed here is a proposed method for testing whether explanation text about a product recommendation is grounded in accessible evidence and aligned, as a secondary check, with local attribution proxies. [4] [5] [1]

8. Failure cases and limits of generalization

Several failure cases must be expected. First, explanations that are intentionally abstract may not map cleanly onto atomic claims, which can produce false positives in the hallucination score. Second, if the recommender uses latent factors that are not human-readable, the evidence set may be incomplete even though the recommendation is internally valid; in that case, the protocol can test only visible grounding, not full mechanistic truth. Third, explanations can be semantically correct but pragmatically misleading, for example when they state a true feature but overemphasize its importance. The proposed alignment check helps detect this, but it cannot eliminate the ambiguity of natural language. [2] [3]

Generalization is also limited by domain and data type. The protocol is most applicable where recommendations are derived from explicit user profiles, item metadata, review text, or hybrid feature sets, because those are the inputs that can be traced. It is less directly applicable to purely collaborative systems with minimal explicit features, or to recommenders whose internal representations cannot be meaningfully surfaced. The method should therefore be presented as a grounded evaluation framework for product recommendation explanations, not as a universal test for every recommender architecture. [1] [5] [4]

9. Conclusion

This manuscript proposes a narrow definition of hallucination in product recommendation explanations and a corresponding evaluation protocol. The proposed contribution is a two-channel framework that first checks claim-level grounding against the evidence record available to the recommender and then checks whether grounded reason-claims align with local attribution proxies under a deterministic feature-mapping rule. Within this design, SHAP and LIME are auxiliary signals for alignment rather than evidence for factual support or proof of causal reasoning. The protocol also specifies how uncertainty should affect explanation wording and how zero-claim outputs should be reported in system-level summaries. [3] [2] [4] [1] [5]

What remains untested is whether the proposed scoring rule, comparator set, and alignment checks produce stable judgments across product domains, explanation styles, and recommender architectures. The protocol therefore ends with a testable claim rather than a performance claim: if a recommender explanation cannot be traced to the same evidence used by the model, it should be treated as hallucinated unless it is explicitly framed as hypothetical. Whether this rule improves user trust, decision quality, or system adoption requires separate empirical evaluation. [3] [2]

References

1. Kollu, Siva Rama Prasad; Garapati, Yugandhar (2026). DeepSentRec: a deep learning-based sentiment-aware product recommendation system. Scientific Reports. 16(1), 10580. doi:10.1038/s41598-026-45953-9.
2. Chen, Lifei; Lim, Zhi Min; Lim, Wei Hong; Tiang, Sew Sun; Sharma, Abhishek; Syamsunur, Deprizon; Alharbi, Amal H.; Eid, Marwa E.; et al. (2026). Search-guided regression ensembles for accurate, interpretable, and uncertainty-aware construction cost estimation. Scientific Reports. 16(1), 21456. doi:10.1038/s41598-026-51706-5.
3. Shasha, Qi (2026). Analyzing digital consumer insights through RoBERTa LLM based sentiment analysis and topic modeling. Scientific Reports. 16(1), 18921. doi:10.1038/s41598-026-48162-6.
4. Deng, Lin (2025). Real estate valuation with multi-source image fusion and enhanced machine learning pipeline. PLOS One. 20(5), e0321951. doi:10.1371/journal.pone.0321951.
5. Rances, Christian Alexandra; Sipe, Rassel Aviel; Baris, Sherwin; Lopez, Adrian; Rosaldo, Asia Dominic; Bringula, Rex; Ulfa, Saida (2026). CoDeRS: a computing degree program recommender system using machine learning algorithms. Frontiers in Artificial Intelligence. 9, 1874202. doi:10.3389/frai.2026.1874202.