Evaluating Retrieval-Augmented Customer Support in PropTech: A Research Protocol for Knowledge-Grounded Conversational Assistance

1. Introduction and research question

Property technology systems increasingly mediate interactions that were previously handled by front-desk staff, leasing agents, call centers, or valuation specialists. In this setting, a conversational assistant is not only a language interface; it is a decision support layer that must locate policy documents, property records, service histories, and local rules before answering. This creates a narrower and more demanding problem than general chatbot quality. The central question of this protocol is: under what evaluation conditions does retrieval-augmented customer support improve factuality, task completion, and escalation behavior in PropTech compared with a non-retrieval conversational baseline? The question is intentionally bounded to customer support tasks rather than to all property analytics. It focuses on answer-grounding, not on autonomous decision making.

The evidence scope is a targeted narrative synthesis rather than a systematic review. The immediate customer-service analogue is a banking chatbot study that structures user queries into graphs, completes knowledge graphs, and classifies service requests using a multi-layer neural framework. Its relevance lies in the combination of query normalization, structured knowledge access, and service-channel evaluation, all of which transfer to property support workflows. Real-estate valuation research on human–machine collaboration is also relevant because it separates expert-only, machine-only, and hybrid conditions and compares both accuracy and time. Although that work concerns valuation rather than customer support, it provides a direct template for task division and for comparing human and machine contributions in a property domain. Methodological transfer is further supported by cross-domain customer-review modeling and by evidence-constrained AI-agent workflows that use retrieval and follow-up questioning to maintain traceability. [1] [2] [3] [4]

2. Prior work and transferable foundations

The closest direct customer-service evidence comes from the banking chatbot framework that converts natural-language queries into a structured graph representation and then applies knowledge graph completion before neural refinement. The key methodological lesson is not the banking context itself but the way the system separates language understanding from evidence organization. That separation is useful in PropTech because property queries often contain multiple referents, such as unit number, building, lease term, neighborhood, and policy clause. A retrieval layer can disambiguate these referents before generation. The study also evaluated channel-level differences between chatbot and human support, which is relevant for deciding which property questions should remain automated and which should be escalated. [1]

Real-estate collaboration research supplies a second foundation. In that study, experts, a machine-learning model, and a hybrid setup were compared in valuation tasks under controlled conditions, with time and accuracy measured jointly. The important transferable point is structural: model assistance was treated as a partner in a defined workflow, not as a free-form replacement for human judgment. This supports an evaluation design in which retrieval-augmented support is tested against expert-only and model-only conditions, and in which the assistant is allowed to recommend escalation when uncertainty remains high. The study also shows that property tasks can be decomposed into comparable conditions, making it reasonable to evaluate whether a system helps with routine cases while preserving expert control over atypical cases. [2]

Two further sources inform the protocol design. A cross-domain sentiment framework for customer reviews shows that models evaluated only in one domain can fail under domain shift, vocabulary change, and different contextual uses of similar terms. This is relevant for PropTech because tenant complaints, listing inquiries, warranty questions, and valuation questions differ in lexical pattern and factual structure. The study motivates explicit robustness testing across query types, not only aggregate accuracy. Finally, the evidence-constrained AI-agent framework for indoor inspection demonstrates how retrieval, follow-up questions, tool use, and a structured state representation can be composed into a transparent workflow. Although it is not a customer-support system, it supports a design in which the agent is prevented from answering before the necessary evidence is collected and in which unresolved gaps are visible to the evaluator. [3] [4]

3. Proposed task definition and system scope

The proposed system is a retrieval-augmented conversational assistant for property support. The task is to answer user questions about a bounded property knowledge base containing, at minimum, listing metadata, building facts, lease or ownership policy excerpts, maintenance procedures, office contact routes, and service-case status fields where permitted. The system does not make legally binding decisions, set prices autonomously, or infer unavailable facts from external memory. Its output space consists of direct answers, grounded summaries, clarifying questions, and escalation recommendations. This bounded scope is important because property support queries often require a distinction between what is documented, what is probable, and what must be referred to a human agent.

A query is defined as support-relevant if it requests one of four response types: factual property information, procedural guidance, status lookup, or policy interpretation limited to published rules. The protocol excludes open-ended advisory questions whose answer depends on legal judgment, negotiation, or confidential records not in the index. The retrieval corpus is assumed to be versioned and auditable. Each document is assigned a source class: authoritative policy, transactional record, maintenance record, public listing record, or human-authored note. Source classes are ordered for conflict resolution as follows: authoritative policy and current transactional record take precedence over maintenance records and public listing records, which in turn take precedence over human-authored notes unless the note is the only current record of a service event. When two retrieved passages conflict, the assistant must prefer the higher-ranked current source; if rank and recency do not resolve the conflict, the assistant must not answer the disputed point and must escalate or request clarification.

4. Operational variables and exact decision rules

The evaluation uses four primary variables. First, grounding precision: the proportion of answer claims that are directly supported by retrieved evidence. Second, task resolution rate: the proportion of queries answered or appropriately escalated within the protocol. Third, clarification rate: the proportion of ambiguous queries for which the system asks a follow-up question before final answering. Fourth, interaction cost: the number of retrieval steps, follow-up turns, and human escalations required per resolved query. These variables are chosen because a property support assistant is useful only if it is both correct and operationally efficient.

An exact operational rule governs whether the system may answer directly. Let E be the set of retrieved passages, and let c be a candidate answer claim. The claim c is answerable only if the cited passage or passages explicitly entail c, or if c is a faithful paraphrase that does not add unstated content, and no higher-ranked retrieved passage contradicts it. This same entailment standard is used later for evaluation; the protocol does not use a weaker notion of semantic support at answer time. For a single-attribute claim, one authoritative current passage is sufficient. For a multi-attribute claim, each attribute must be entailed by evidence, and the claim may be presented as settled only if either: (a) one authoritative passage contains all required attributes, or (b) the attributes are supported by at least two independent passages. Independence means that the passages come from different documents or from separately versioned record fields that can be audited independently; two adjacent sentences from one unversioned document do not count as independent support. These rules are the default analysis plan. Sensitivity analyses may vary the independence requirement and the source-precedence order under a preregistered protocol. [4]

A second rule addresses entity ambiguity. If a query contains more than one possible property referent, such as a unit number that matches multiple buildings or a policy term that appears in multiple versions, the assistant must classify the query as ambiguous and issue a clarifying question before retrieval expansion. This rule is motivated by the structured handling of complex service language in the banking chatbot framework and by the need for evidence traceability in agentic workflows. In both cases, the system should reduce ambiguity before generating a final response rather than relying on post hoc correction. [1] [4]

5. Proposed architecture and algorithmic sequence

The protocol assumes a five-stage pipeline. Stage 1 parses the user query into intent, entity slots, and ambiguity flags. Stage 2 retrieves candidate evidence from the property corpus using hybrid lexical and embedding search. Stage 3 reranks evidence by source-class precedence, version recency within source class, and semantic overlap with the parsed query; if these factors still leave a material conflict unresolved, the disputed claim is marked non-answerable. Stage 4 generates an answer, a clarification request, or an escalation note under citation constraints and the entailment rule defined above. Stage 5 logs the evidence path, source versions, final decision, and unresolved gaps for audit.

The decision sequence is as follows. Input query q is first normalized into a structured representation s(q). If ambiguity(s(q)) = true, the model asks a clarifying question. If ambiguity is false, the retriever returns top-k passages. The generator is allowed to compose an answer only from the retrieved passages and must attach source-grounded spans to each answer sentence. If the answer requires information absent from the retrieved set, the model must refuse that portion and either request more details or escalate. The architecture is deliberately closer to evidence-constrained workflow design than to free-form generative chat. That choice follows the AI-agent framework, which formalizes state and tool use, and the banking chatbot study, which transforms user language into a structured graph before completion and classification. [4] [1]

Compared with the direct customer-service study, this protocol does not rely on a predictive regression target such as channel revenue or complaint volume. Its main output is a grounded response quality assessment. Compared with the valuation collaboration study, it evaluates support interactions rather than numeric appraisal. The protocol therefore adapts the collaborative logic of human–machine comparison without copying the valuation task itself. This distinction matters because a support assistant should be penalized for unsupported certainty even when it sounds fluent. [2]

6. Comparator design and prospective validation

The primary comparator is a non-retrieval baseline conversational model that receives the same user query but only the prompt and conversation history. A secondary comparator is a retrieval system without explicit clarification and refusal rules. If human agents are included, the human comparator should operate under the same time cap and access constraints as the automated system. The point of this comparison is to isolate whether retrieval itself improves groundedness and whether clarification/refusal logic prevents unsupported answers that a more permissive generator might produce.

The validation design is prospective and should be executed on a stratified query set assembled from real or simulated support interactions with preserved query distribution. Query strata should include routine factual questions, ambiguous entity questions, policy questions, multi-hop questions, and cases that require escalation. Each query receives dual annotation under a preregistered rubric: one label for task type and one label for evidence sufficiency. Evidence sufficiency is coded at both query and claim level as sufficient to answer directly, sufficient only after clarification, insufficient and requiring escalation, or conflicting. Mixed answers are scored claim by claim, so a response may contain supported and unsupported components rather than receiving a single all-or-none label. Answers are evaluated by human assessors using the same entailment rule applied in generation: a claim is supported only when the cited passage explicitly entails it or when the answer is a faithful paraphrase of that passage. For prospective execution, data collection should continue until each stratum reaches its preregistered minimum count and the planned comparison between the primary arms achieves the preregistered precision target for the main outcome estimates; the protocol therefore uses an explicit stopping rule rather than an open-ended sample. This design follows evidence-constrained evaluation logic and the cross-domain principle that performance should be examined under heterogeneous inputs rather than one narrow query style. [4] [3]

The protocol also recommends a human–machine collaboration arm for difficult cases. In that arm, a property agent receives the model’s retrieved evidence and draft response, then edits, confirms, or rejects it. The comparison should record whether the machine reduces lookup time, whether the human accepts the retrieved evidence, and whether combined handling improves resolution of difficult queries. This follows the logic of valuation collaboration research, which compared human-only, machine-only, and hybrid conditions. However, here the hybrid condition is evaluated on support quality and response safety rather than on price accuracy. [2]

7. Failure cases, boundary conditions, and limits of generalization

Several failure modes are expected. The first is retrieval drift, in which semantically similar but outdated or wrong property records are returned. The second is referential ambiguity, where the assistant confuses a building, a unit, or a policy version. The third is evidence conflict, where a FAQ, a lease clause, and a staff note disagree. The fourth is over-refusal, where the assistant asks for clarification even when the evidence is sufficient. These failure cases are central to evaluation because a grounded assistant can still be operationally weak if it is too cautious or too eager.

Generalization is limited by corpus quality and by domain conventions. A system developed on one property management platform may not transfer cleanly to another if record schemas, terminology, or escalation pathways differ. This is the same type of transfer problem observed in cross-domain customer-review classification, where vocabulary and context shift degrade model behavior. The protocol therefore requires out-of-domain testing in at least one prespecified contrast, such as rentals versus sales or two administrative units with different policy language, using the same query strata and annotation rubric as the main evaluation. Each out-of-domain condition should satisfy the same preregistered minimum-count and precision-based stopping rule used for the primary validation. The protocol also requires manual review of disagreement cases because high aggregate grounding scores may hide systematic failures on rare but consequential query types. [3]

The protocol does not claim that retrieval alone will resolve property support problems. Some queries will remain unsuitable for automation because they require negotiated judgment, legal interpretation, or access to private account data. For these cases, the correct outcome is escalation rather than a fabricated answer. The evidence-constrained workflow source supports this boundary condition by formalizing unresolved gaps and iterative queries rather than forcing a single-pass output. The customer-service chatbot source similarly suggests that structured handling is most useful when the query can be mapped to a service class. Queries outside that class remain a limitation of the system, not a failure of the evaluation design. [4] [1]

8. Conclusion

This manuscript establishes a conceptual framework for evaluating retrieval-augmented customer support in PropTech. The proposed contribution is a bounded research protocol that defines the task, variables, decision rules, comparator conditions, and validation procedures for grounded conversational support in property settings. The synthesis shows that direct service-chatbot evidence supports structured query handling and knowledge-based response pipelines, real-estate collaboration research supports human–machine comparison, and methodological work on transfer and evidence-constrained agents supports robustness testing, clarification behavior, and refusal rules. [1] [2] [3] [4]

What remains untested is whether the proposed protocol improves factuality, task completion, and escalation quality in actual PropTech support traffic. It is also untested whether the proposed clarification and support thresholds are well calibrated across different property organizations, document structures, and regulatory settings. Those questions require a prospective evaluation with real query logs and expert annotation. Until then, the contribution of this manuscript is methodological: it specifies how retrieval-augmented customer support in PropTech should be tested so that answer quality, traceability, and operational safety are assessed together rather than in isolation.

References

1. He, Ji; Luo, Yi; Wang, Ting (2025). iDigiChat: intelligent digital marketing service chatbot for providing efficient customer services using artificial intelligence. Scientific Reports. 15(1), 33074. doi:10.1038/s41598-025-14722-5.
2. Kmen, Christopher; Navratil, Gerhard; Kattenbeck, Markus; Giannopoulos, Ioannis (2026). Evaluating human–machine collaboration through a comparative analysis of experts, machine learning, and hybrid approaches in real estate valuation. Scientific Reports. 16(1), 4044. doi:10.1038/s41598-025-34099-9.
3. Kumar, Ajeet; Abhishek, Kumar; Shafeeq B M, Ahamed (2025). SentXFormer: a transformer-enhanced hybrid deep learning framework for cross-domain sentiment analysis of customer reviews. Scientific Reports. 16(1), 3528. doi:10.1038/s41598-025-33526-1.
4. Jing, Zijian; Zhu, Liyi; Chen, Tianyi; Hovestadt, Ludger; Li, Li (2026). Vision-Based Digital Twin and AI Agent Framework for Low-Cost, Explainable Indoor Building Inspection and Safety Assessment. Sensors. 26(15), 4992. doi:10.3390/s26154992.