A Reproducibility Framework for PropTech Benchmarks

1. Introduction

This article develops a protocol for reproducible benchmarking in property technology. The problem is defined narrowly. A benchmark is treated here as a specified comparison task that includes a data object, inclusion rules, variable definitions, comparator conditions, outcome metrics, and reporting conventions. Reproducibility, in this protocol, means that an external team can reconstruct those benchmark elements from the released specification and can rerun the comparison under the same declared conditions. The article does not claim that the cited literature has already established a field-wide reproducibility crisis in PropTech. Instead, it argues that the combination of heterogeneous data sources, policy-sensitive outcomes, and scenario-based analytics makes benchmark specification itself a methodological object that warrants explicit design.

The protocol is motivated by six studies whose methods bear directly on benchmark design. A housing-price dataset article documents anchor-year selection, variable categories, units, data sources, and missingness at a level useful for replication-oriented benchmark construction. A vision-based digital-twin inspection framework links semantic entities, measurements, image observations, and regulatory knowledge within a structured scene-state and an evidence-constrained reasoning process. Four additional studies show benchmark-relevant task forms in real-estate and urban analytics: explainable forecasting of price-volume dynamics, mapping of climate-risk valuation methods, causal analysis of minimum energy performance standards in commercial real estate, and a parking and mobility digital twin that integrates historical and real-time data with predictive and generative components. [1] [2] [3] [4] [5] [6]

The article's contribution is therefore procedural rather than empirical. It proposes a benchmark protocol called the Reproducible PropTech Benchmark Protocol, or RPBP. The protocol is intended for future benchmark instances in valuation, climate-risk assessment, regulatory impact analysis, and digital-twin operations. Its status in this article is conceptual. Thresholds, weights, and validation tolerances are proposed protocol choices that require later testing; they are not presented as established facts or as findings from the cited studies.

2. Research question and evidence scope

The research question is the manuscript's own methodological question: what minimum specification should a PropTech benchmark contain so that an independent team can reconstruct the benchmarked comparison, including data construction, comparator logic, interpretation outputs, and any declared scenario claims? The question concerns benchmark design rather than open-science practice in general. It also concerns multiple PropTech task families rather than one predictive architecture or one dataset.

The evidence scope is a targeted narrative synthesis. The six selected studies are used because they expose benchmark-relevant design problems with unusual clarity. The housing dataset article is useful for variable-level documentation and anchor-year justification. The indoor inspection digital twin is useful for audit trails, hierarchical state representations, and evidence-constrained inference. The explainable forecasting article is useful because it treats explanation outputs as part of the model product rather than as an afterthought. The climate-risk map is useful because it documents heterogeneity in exposures, outcomes, asset classes, and methods. The minimum energy performance standards study is useful because it specifies a policy timeline, treated assets, comparators, and multiple outcomes. The parking digital twin is useful because it integrates descriptive, predictive, and scenario-generation modules within one urban system. [1] [2] [3] [4] [5] [6]

3. Benchmark-relevant lessons from the selected studies

The Nanning housing dataset article treats dataset construction as a reproducible object. It identifies the residential community as the basic unit of analysis and summarizes price as the average price within each residential community. It explains the choice of 2018 as an anchor year by reference to completeness and internal consistency after deduplication and quality control. It distinguishes raw, clean, and processed formats; reports variables, units, sources, and summary statistics; and discloses missingness by variable. It also states that population density is used as a structural contextual covariate rather than a contemporaneous market-shock variable. For benchmark design, these practices show that replication depends on much more than a final cleaned table. [1]

The indoor building inspection digital twin contributes a second lesson: benchmark outputs can be tied to auditable evidence structures rather than free-form narrative explanation. Its framework reconstructs a metric-scale 3D point cloud from smartphone video, parses semantic entities and spatial relationships, formalizes risk guidelines into computable prototypes, and binds geometry, semantic labels, image observations, and regulatory knowledge in a hierarchical SceneState. The AI agent follows a dual pathway: an initial scan and then follow-up queries through tool calls for measurement, knowledge retrieval, or visual cross-checking. The article also argues that conventional manual inspections often lack stable, traceable organization of evidence and that general multimodal models lack metric precision, 3D consistency, domain grounding, and evidential reliability for safety-critical assessment. For benchmark design, the transferable point is the explicit linkage between claims and their supporting evidence states. [2]

The explainable price-forecasting study contributes a lesson about benchmark outputs. Its EX-KAN framework uses lagged raw values of both price and volume, jointly forecasts future price and volume, and computes Jacobian-based elasticities that quantify how changes in past price and volume propagate into future predictions across horizons and market regimes. The article also states that post hoc explanation methods such as LIME and SHAP can be unstable approximations and that many existing approaches do not model changing lagged cross-variable effects. A benchmark in this area therefore needs to define the explanation object explicitly. It is not enough to request 'interpretability' in the abstract. [3]

The climate-risk systematic map contributes a lesson about heterogeneity and conflicting evidence. It distinguishes physical and transition risks; maps variation in hazards, asset classes, valuation outcomes, data sources, and analytical approaches; and reports that the evidence base is fragmented. It also notes mixed findings, with some studies finding negative capitalization of climate risks in housing prices and others finding limited or no effects. The map further reports concentration in the United States and a strong focus on one- and two-family housing. These observations imply that a benchmark should declare its exposure class, asset class, outcome family, and geographic scope rather than present a climate-risk task as a universal template. [4]

The Dutch office-market study on minimum energy performance standards contributes a lesson about causal comparators and outcome separation. The article describes a policy process beginning with public discussion in 2016, formal announcement in October 2018, and prohibition on use of substandard office buildings from January 2023. It employs a modified difference-in-differences design using offices as treated assets and other commercial property types, such as retail or leisure, as comparators. It also reports different responses by outcome: offices became more likely to improve energy performance, transaction prices reacted after formal adoption, and rental responses remained limited in the studied period. For benchmark design, this means that intervention windows, treated units, comparator units, and each outcome definition must be encoded separately. [5]

The urban parking and mobility digital twin contributes a lesson about multi-source integration and scenario status. The framework combines parking meter transactions, occupancy, violations, weather, temporal patterns, and shift schedules. It is organized into three components: descriptive statistics, predictive analytics, and a simulation engine that uses generative models for what-if scenarios. The article also identifies broader limitations in the literature, including isolated subsystem designs, scaling barriers, and sparse generative capability. For benchmark design, the key transfer is that predictive tasks and synthetic scenario tasks should be specified separately, because they operate on different evidential grounds. [6]

4. Design requirements derived from the synthesis

On the basis of that synthesis, this article proposes five design requirements for reproducible PropTech benchmarks. These are protocol decisions made here, not empirical findings from the six studies. First, a benchmark needs a task charter that defines unit of analysis, target construct, spatial scope, temporal scope, inclusion and exclusion rules, and intended benchmark family. Second, every released variable needs a registry entry containing role, unit, provenance, alignment rules, and transformation history. Third, every output needs an explicit evidence status so that measured values are not conflated with estimates or synthetic scenarios. Fourth, if explanation is part of the task, the benchmark must declare the formal explanation object to be evaluated. Fifth, the benchmark must state its domain boundary in terms of exposure type, asset class, and outcome family.

These requirements follow directly from the kinds of methodological variation observed in the selected studies. The Nanning dataset shows the importance of source-level and variable-level documentation. The SceneState framework shows the value of evidence-linked outputs. The climate-risk map shows that exposure, outcome, and asset class heterogeneity are not peripheral details. The Dutch energy-standard study shows that intervention timing and comparator construction alter the substance of a benchmarked claim. The parking digital twin shows that observational prediction and synthetic scenario generation must not be reported as one undifferentiated result class. [1] [2] [4] [5] [6]

5. The Reproducible PropTech Benchmark Protocol (RPBP)

RPBP is proposed as a six-layer benchmark specification. Layer 1 is the task charter. Layer 2 is the variable registry. Layer 3 is the evidence graph. Layer 4 is comparator design. Layer 5 is the metric specification. Layer 6 is the validation log. Each layer has mandatory fields, but the mandatory set differs by benchmark family. The protocol defines four families for present purposes: F1 forecasting and valuation, F2 climate-risk valuation, F3 policy-impact evaluation, and F4 digital-twin operational or scenario benchmarking. These four families are proposed by this article as an organizing device; they are not claimed as a complete taxonomy of PropTech.

The mandatory layers by family are defined as follows. For F1, mandatory layers are task charter, variable registry, comparator design, metric specification, and validation log; the evidence graph is mandatory if the benchmark includes explanation outputs or inferred variables that must be traced to source states. For F2, all six layers are mandatory. This default avoids case-by-case disputes over whether climate exposure was directly observed or constructed from linked hazard, location, or policy data. For F3, all six layers are mandatory because policy timing, treatment coding, and outcome derivation require explicit lineage. For F4, all six layers are mandatory because observational streams, inferred states, and synthetic scenarios must be distinguished object by object. Under RPBP, the phrase 'mandatory component' refers to the family-specific set just defined.

The reproducibility score is a descriptive summary of specification completeness, not an independent acceptance gate. For a benchmark instance B in family f, define R_f(B) = (w1*C + w2*V + w3*E + w4*P + w5*M + w6*L) / (w1 + w2 + w3 + w4 + w5 + w6), where C is task-charter completeness, V is variable-registry completeness, E is evidence-graph completeness, P is comparator-design completeness, M is metric-specification completeness, and L is disclosure-and-validation-log completeness. Each component takes a value in [0,1]. To make scoring reproducible, RPBP fixes one default rubric: for each layer j, let n_j be the number of mandatory fields for that family and let m_j be the number of those fields that are present, non-empty, and machine-checkable under the layer definition. Then the default completeness score is S_j = m_j / n_j. A field counts as machine-checkable only if an external team can identify it unambiguously in the released package and determine whether it is satisfied without discretionary interpretation. Family-specific optional fields may be reported, but they do not change S_j unless the charter preregisters an alternative weighted field rubric before release. The default layer-to-symbol mapping is C = S_charter, V = S_registry, E = S_evidence, P = S_comparator, M = S_metric, and L = S_validation. Weights must be non-negative real numbers, and the denominator must be strictly positive. The default rule is that each mandatory component for family f receives weight 1 and each non-mandatory component receives weight 0, unless the benchmark charter preregisters different weights with reasons. RPBP therefore uses one ordered decision rule. Gate 1 is registration completeness: every mandatory layer for family f must have S_j = 1. Gate 2 is Level A reconstructive validation under the family-specific class rules in Section 8. Gate 3 is Level B analytic validation when the benchmark claims analytic reproducibility of results rather than only reconstructability of the benchmark object. A benchmark is classed as RPBP-compliant only if it passes all gates required by its chartered claim type. Under this rule, R_f(B) cannot rescue failure at any gate; it is reported to summarize documentation completeness among benchmarks that have already been evaluated by the gate sequence. Proposed family score thresholds remain descriptive defaults for later calibration rather than acceptance criteria: tau_F1 = 0.80, tau_F2 = 0.85, tau_F3 = 0.90, and tau_F4 = 0.90. These values are provisional placeholders for prospective testing, not validated adequacy standards. Compute m_j/n_j only for required layers with a positive registered field count. A nonrequired layer is reported as not applicable and carries zero weight, rather than evaluating a zero-field ratio.

The variable registry contains the minimum fields: variable name, object class, role, unit, source type, source date, spatial resolution, temporal resolution, transformation chain, missingness rule, imputation rule if any, legal access status, and current evidence status. Role is drawn from the set {target, predictor, treatment, exposure, comparator, grouping, identifier, explanation output}. Evidence status is defined exactly as follows. Observed: directly recorded from a source system or released source file. Derived: deterministically computed from observed or previously derived variables by a declared function. Inferred: estimated by a trained model or statistical estimator from observed or derived inputs. Synthetic: generated to represent a counterfactual, simulation, or other non-observed state. A variable may change status across processing stages, but each status transition must be logged in the evidence graph with timestamp, operator, input nodes, transformation rule, and output identifier. Mixed-status composite variables require an additional rule. If a released variable is assembled from components with different statuses, the released variable receives the highest-status label on the ordered scale observed < derived < inferred < synthetic, and the registry must also list the status of each named component. Thus a deterministic index built from observed and inferred components is classified as inferred, while a scenario indicator that incorporates any synthetic component is classified as synthetic. This conservative rule is proposed to prevent composite indicators from masking their least directly evidenced component. [1] [6]

The evidence graph generalizes the logic of structured SceneState representation to benchmark governance. Each benchmark object, such as a residential community, office building, transaction, parking zone, or zone-hour, is represented as a state node with identifier, geometry if relevant, temporal index, linked variable values, and applicable rules. Edges encode provenance, transformation, measurement, inference, or simulation operations. If a benchmark includes explanation outputs, those outputs are also graph nodes linked to the data and model states from which they were produced. This design is proposed so that benchmark claims can be traced at object level rather than only through aggregate tables. [2] [3]

6. Operational definitions and benchmark variables

Several operational definitions are required for RPBP. Benchmark instance means one released benchmark package evaluated under one declared family and one fixed set of task, comparator, metric, and validation rules. Unit of analysis means the smallest benchmark object that receives a target value, treatment assignment, or prediction. Comparator integrity means that an independent team can reconstruct the comparator condition from the released inclusion and exclusion rules without discretionary reinterpretation. Explanation object means the formal evaluand of interpretability, such as a lag elasticity matrix, a treatment-effect estimate with declared identification assumptions, or an evidence chain from output claim to measured state. Acceptance criterion means the family-specific condition under which a reconstructed benchmark instance is judged to match the registered benchmark within declared tolerances.

RPBP proposes a benchmark-family variable schema. In F1 forecasting and valuation benchmarks, typical core variables include structural property characteristics, accessibility measures, environmental conditions, land-use context, and lagged market variables such as past prices or transaction volumes. The EX-KAN study is relevant here because it models price and volume jointly and reports regime-aware elasticities. In F2 climate-risk valuation, variables must identify risk type as physical or transition and must subclassify the exposure channel. The climate-risk map reports that physical risks are often operationalized through hazard maps or proxies such as elevation and shoreline distance, while transition-risk evidence is concentrated on channels such as energy-performance requirements, energy labels, and local externalities linked to renewable-energy deployment. F2 registration must also fix the valuation estimand before metric evaluation. The charter must label each reported climate-effect parameter as a marginal valuation effect from a stated econometric specification, a semi-elasticity, an elasticity, a mean contrast between declared exposure groups, or another explicitly defined estimand. In F3 policy benchmarks, the minimum required variables are treatment indicator, intervention timing, comparator definition, and at least one outcome defined over a declared window. In F4 digital-twin benchmarks, the minimum required variables are stream identifier, observation timestamp, object identifier, state variable, and scenario-control variable for any synthetic run. [3] [4] [5] [6]

RPBP also proposes an exact climate-exposure typing rule. If a benchmark variable encodes hazard likelihood, hazard intensity, expected damage from a natural hazard, or asset location relative to a natural hazard, then classify the variable as physical climate risk. If it encodes policy, legal, technology, market, or reputational consequences associated with decarbonization, then classify the variable as transition climate risk. This rule is a protocol adaptation of the categories reported in the climate-risk map. [4]

A second exact rule concerns evidence-status transitions and derived-variable verification. Let s_t(v) denote the status of variable v after processing stage t. A transition s_t(v) -> s_(t+1)(v) is valid only if the evidence graph records: (i) input node identifiers, (ii) transformation or estimation function identifier, (iii) operator identifier, and (iv) output node identifier. If any of these fields is missing, the transition is invalid for RPBP compliance. For validation, a derived variable is not accepted by rate threshold alone. The benchmark charter must preregister one of two verification modes for each derived variable class: exact equality, used by default for deterministic recoding, aggregation, joins, and arithmetic transformations represented in exact machine-readable form; or tolerance-bounded equality, used only when the derivation contains declared numerical approximation such as coordinate reprojection, interpolation, or floating-point optimization routines. If tolerance-bounded equality is used, the charter must declare the tolerance rule before release. This rule is proposed to preserve traceability when variables move from observed records to derived features, inferred outputs, or synthetic scenario states.

7. Comparator design and metric specification

Comparator rules in RPBP are task-specific. This article does not impose a universal baseline across PropTech because the selected studies show that benchmark validity depends on the substantive question. In F1 forecasting, an acceptable comparator may be a persistence model, an ARIMA-family model, or a simpler explainable architecture, provided the benchmark charter fixes the comparator class and input information set. The EX-KAN article is instructive here because it emphasizes fair comparison under a joint-input setting across KAN, LSTM, TSMixer, and Transformer models. RPBP therefore requires that any comparator use the same declared lag window and access the same admissible external variables unless the benchmark explicitly studies information advantage. [3]

In F3 policy-impact benchmarks, comparator design must encode untreated units and timing assumptions. The Dutch energy-standard study provides a model of this specificity by distinguishing public discussion from formal announcement, defining an intervention window, and comparing offices with other commercial property types. RPBP therefore requires a policy benchmark to register: treatment group, comparator group, intervention start, intervention formalization date if distinct, post-period definition, and outcome-specific observation windows. If multiple outcomes are studied, each outcome must have its own comparator statement if sample construction differs. [5]

In F4 digital-twin benchmarks, predictive modules and scenario modules cannot be treated as one comparator class. A predictive module estimates observed states. A scenario module generates plausible but non-observed states. RPBP therefore requires separate comparator logic for each. For predictive evaluation, the comparator must predict observationally grounded targets. For scenario evaluation, RPBP sets one default comparator rule: the comparator is a constrained ablation of the benchmarked system that uses the same scenario inputs, the same released hard constraints, and the same output schema, but removes at least one declared module, information source, or feedback path named in advance by the charter. This default preserves benchmark-specific fairness better than allowing unrestricted comparison across unrelated scenario engines. A benchmark may preregister a different scenario comparator, such as an external rule-based simulator or another scenario engine, but only if the charter also demonstrates information-set parity, identical scenario-control variables, and common hard plausibility constraints. One benchmark instance may include both module types, but then both must be represented within a shared evidence graph and a shared validation log, with module-specific nodes, edges, metrics, and acceptance decisions. [6] [2]

Metric specification is also family-specific. In F1, primary metrics may include absolute or percentage forecast error on the target series; secondary metrics may include explanation stability or regime-consistency checks if those outputs are benchmarked. In F2, primary metrics depend on the declared outcome family, which may be transaction prices, rents, expected damages, appraisals, or cost measures, as reported in the climate-risk map. In F3, primary metrics are treatment-effect estimates for declared outcomes and secondary metrics may include treatment-coding consistency or cohort reconstruction accuracy. In F4, primary metrics are split into observed-state prediction metrics and synthetic-scenario plausibility diagnostics. RPBP does not collapse these metrics into a single universal score, because the selected studies indicate that outcomes differ in meaning and evidential basis across tasks. [4] [5] [6] [3]

8. Validation design

RPBP proposes two validation levels. Level A is reconstructive validation. It asks whether an external team can rebuild the benchmark object, including cohort, variables, comparator conditions, and metric code paths, from the released specification. Level B is analytic validation. It asks whether the reconstructed benchmark yields the same benchmark statistics within predeclared tolerances. The distinction matters because good analytic agreement can arise despite poor documentation, and complete documentation can still encounter source volatility or access restrictions.

The reconstructive criterion is defined by evidence status, and RPBP evaluates each status class separately before any family-level pass-fail decision is made. For observed variables, record matching means equality of the preregistered key tuple and equality of the released value after applying any preregistered canonicalization rule. The default key tuple is object identifier plus time index when time is present; if the benchmark uses another key, the charter must declare it before release. Canonicalization is limited to deterministic formatting operations such as date normalization, case normalization, declared unit conversion, and ordering of categorical labels. For derived variables, validation has two parts. First, lineage verification checks whether the external team can reproduce the declared dependency graph from validated inputs to the released output node. Second, declare one derivation-conformance mode. Symbolic mode requires a released expression or transformation grammar, its input domain, and either canonical rule identity or a checkable equivalence argument under those domain restrictions. Finite numerical agreement is not a proof of symbolic equivalence. In executable/approximate mode, release a content-addressed implementation and fixed test fixtures covering valid inputs, boundaries and missing-value cases. A fixture passes when statuses agree and, for finite numeric outputs, |a-b| <= atol + rtol*max(|a|,|b|), with finite nonnegative tolerances fixed before reconstruction. Unexpected NaN or infinity fails rather than satisfying the tolerance check. This mode establishes fixture-level numerical agreement only. Record which mode was used and do not generalize its conclusion to untested inputs. Value verification then applies the preregistered equality rule for that derived-variable class: exact equality by default, or tolerance-bounded equality only where the charter has explicitly declared numerical approximation. For inferred variables, validation is output matching within a declared numerical tolerance epsilon after rerunning the specified model with the registered training and inference settings. To make this reproducible, the registration must also fix dependency versions, random-seed rules, hardware-sensitive settings when relevant, and a training-data snapshot or immutable query rule. If the estimator is nondeterministic, the charter must state whether reproducibility is evaluated by a fixed-seed run or by a preregistered seed set with an acceptance band over the resulting outputs. For synthetic variables, exact value matching is not required because scenario generation may be stochastic. RPBP therefore adopts one default rule for conceptual registration: a synthetic output is reconstructively matched only if the external team reproduces the registered scenario specification and the regenerated output satisfies every preregistered hard plausibility constraint. Hard constraints must be released as machine-checkable predicates on the scenario output, such as permitted value ranges, conservation or accounting identities, monotonic policy restrictions, temporal ordering rules, or network-capacity bounds. The basic audit unit is the seed-generated scenario record: the smallest generated object identified by a unique combination of scenario identifier, seed identifier, object identifier, and time index when time is present. Scenario plausibility is evaluated at that unit by default. If the charter instead defines a scenario bundle, the bundle rule must be preregistered and must state the bundle membership map and the bundle-level constraint operators. If two required hard constraints conflict on the same scenario domain, registration fails until the constraints are made jointly satisfiable. A priority ordering cannot silently discard a condition still labeled hard. If the benchmark charter allows stochastic generators, the team must run the registered number of seeds and report the Synthetic Plausibility Pass Rate = plausible seed-generated scenario records / eligible seed-generated scenario records. Satisfaction of hard constraints is necessary but may be insufficient when the charter also claims comparative scenario conclusions. For any registered scenario-ordering claim, such as policy A weakly dominating policy B on a declared outcome, the charter must release the outcome functional, the comparison operator, the evaluation sample, and one explicit seed-aggregation rule before release. The default seed-aggregation rule is majority support: a claim is reproduced only if it holds in strictly more than half of the registered seeds after applying the declared outcome functional to each seed separately. If the charter instead uses a pooled-seed functional, such as a seed-averaged outcome, that alternative must be preregistered together with its tie rule. Competing ordering claims are evaluated claim by claim; if their truth sets overlap under the registered rule, all may pass, but if they are logically incompatible under the declared operator, the benchmark fails registration unless the charter specifies a precedence rule or mutually exclusive claim domains. A synthetic module is reconstructively matched only if all hard constraints pass and every preregistered scenario-ordering claim is reproduced under the registered rule. In plain text, Reconstructive Match Rate = matched released records / eligible released records. The denominator is computed separately for each variable class and evidence status, not once for the benchmark as a whole. A released record is eligible for a given variable only if the external team can legally access the source fields required to reconstruct that variable under the benchmark's declared access conditions. These access-conditioned denominators are intended for legal-compliance accounting at variable level and for transparent reporting of what could be audited. Final pass-fail decisions still apply the family-specific class rules to the eligible records of each mandatory class; a benchmark does not pass merely because inaccessible records were excluded from unrelated denominators. For multiple scenario claims, the registry assigns each a unique claim ID, domain, outcome functional, comparison operator, required scenario list, seed set and aggregation/tie rule. Each claim uses the same registered seeds across the scenarios it compares; missing paired runs make that claim incomplete. Different claims may use different functionals and seed sets, but each is evaluated on its own declared inputs. A module passes only when every required claim passes; no average of claim pass rates replaces this conjunction. Conflicting required claims on an overlapping domain fail registration rather than being selected retrospectively. Any mandatory evidence class with zero accessible eligible records is incomplete, not a successful zero-denominator match.

A benchmark family is then assigned a proposed Level A acceptance criterion by ordered components rather than by one blanket threshold. Step 1 is registry completeness: all mandatory registry, comparator, metric, and validation-log fields for the family must be present. Step 2 is class-wise reconstruction: each evidence-status class is evaluated under its own rule and denominator. Step 3 is family aggregation: the benchmark passes Level A only if every mandatory class-specific condition is satisfied; no strong result in one class can compensate for failure in another. For F1 and F2, Level A passes only if observed variables achieve an Observed Match Rate of at least 0.99, each derived-variable class satisfies lineage verification and registered derivation-conformance and then achieves at least 0.99 under its preregistered equality rule, and inferred variables achieve an Inferred Match Rate of at least 0.95. For F3, exact cohort reconstruction is a hard prerequisite assessed first: the treated cohort, comparator cohort, and intervention-period coding must be reproduced exactly from the declared rules before any tolerance-based matching is evaluated. Conditional on that prerequisite, observed variables must achieve 0.99, each derived-variable class must satisfy lineage verification and registered derivation-conformance and then achieve 0.99 under its preregistered equality rule, and inferred variables must achieve 0.95. For F4, Level A passes only if four conditions hold jointly: observed-state variables meet the 0.99 threshold; each derived-state class satisfies lineage verification and registered derivation-conformance and then meets the 0.99 threshold under its preregistered equality rule; inferred state variables meet the 0.95 threshold; and synthetic scenario outputs achieve a Synthetic Plausibility Pass Rate of 1.00 on hard constraints and reproduce every preregistered scenario-ordering claim under the registered seed-aggregation rule and a complete scenario specification in the evidence graph. The default value 1.00 is strict because hard constraints are admissibility rules, not soft quality scores. Benchmark designers may preregister additional soft plausibility diagnostics, but those do not replace the hard-constraint requirement. All numerical cutoffs in this paragraph are provisional protocol defaults chosen for prospective calibration. They should be treated as placeholders rather than empirically justified standards until inter-team studies test whether they are attainable and discriminating in each family.

Level B criteria remain family-specific. F1 requires each primary prediction metric to agree within its registered tolerance and applies the registered tie band when comparing model order. The default F2 magnitude rule is restricted to beta defined as a mean valuation contrast between two registered exposure groups, expressed in the original outcome units. Only then set rho=|beta|/max(IQR(Y),eta), with eta>0 in those same units; the proposed descriptive bands are <0.05, [0.05,0.20), [0.20,0.50) and >=0.50. The analytic sample and quantile convention for IQR must be frozen. If IQR(Y)<eta, do not use these bands. A marginal slope, semi-elasticity, elasticity or other parameter requires a separate estimand-specific scale, tolerance and magnitude rule registered before release; a raw regression coefficient is never divided by outcome IQR merely because it is called beta. Missing type-specific rules make the effect-reproduction claim incomplete. Sign agreement uses the registered zero band: classify an effect as positive above delta_effect, negative below -delta_effect, and numerically zero otherwise. The reconstruction must reproduce both this sign class and the appropriate magnitude band. F3 requires outcome-effect agreement within delta_effect and exact intervention-period coding. F4 evaluates observed-state metrics and synthetic claims separately; no predictive match compensates for a failed required scenario claim. Every tolerance is a declared input in the units of its metric, rather than a universal performance guarantee.

A prospective evaluation design can test RPBP without claiming results in advance. The default design is a preregistered two-team reconstruction study for one benchmark instance at a time. Before implementation, the benchmark sponsor releases the RPBP package, the acceptance thresholds, the hard plausibility constraints for any scenario module, the number of stochastic seeds if relevant, and the tolerance parameters delta_metric, delta_effect, and epsilon. The registration must also state why the default values are retained or revised. One explicit default rule governs parameter choice: retain the protocol defaults unless the sponsor preregisters a benchmark-specific alternative and justifies it by measurement resolution, legal or platform limits on attainable matching, or a domain-safety requirement for stricter bounds. The registration must further include a calibration plan for each revised parameter. The default calibration plan is sensitivity-based rather than empirical-performance-based: each revised threshold is tested against the default value, the proposed value, and one stricter value of the same direction; each revised tolerance is tested at the proposed value and at symmetric tighter and looser values around it; and each stochastic generator is rerun on one alternative preregistered seed set. Two independent teams then reconstruct the same benchmark without undocumented assistance. Primary endpoints are agreement on cohort reconstruction, evidence-status transitions, comparator conditions, metric code paths, Reconstructive Match Rate, Synthetic Plausibility Pass Rate where applicable, and the resulting Level A and Level B decisions. The protocol is falsified for that benchmark instance if either team cannot execute the released specification or if the teams reach different pass-fail decisions under the same registered rules.

9. Comparison with prior methods

RPBP differs in purpose from the six selected studies. The Nanning article is a dataset paper rather than a benchmark protocol, but it offers a model of variable-level documentation that RPBP converts into a mandatory registry with source, unit, missingness, and transformation fields. The indoor inspection digital twin is an assessment framework rather than a benchmark specification, but its SceneState and evidence-constrained reasoning motivate RPBP's evidence graph and claim-to-evidence linkage. The EX-KAN article proposes a forecasting model, whereas RPBP remains model-agnostic and requires benchmark designers to predefine the explanation object, comparator information set, and acceptance criteria for explanation outputs. The climate-risk systematic map differs again: it does not prescribe one benchmark design, but it shows why climate tasks must declare exposure class, asset class, valuation outcome, and geographic scope instead of assuming one transferable template. The Dutch energy-standard study contributes a different comparison lesson by separating policy discussion, formal adoption, treated assets, comparator assets, and outcome-specific responses. The parking and mobility digital twin clarifies a final distinction central to RPBP: predictive modules and synthetic scenario modules may coexist in one system, yet they should be benchmarked under separate comparator logic while sharing a common provenance structure. [1] [2] [3] [4] [5] [6]

10. Boundary conditions, failure cases, and limits

RPBP is proposed for benchmarks built from structured property, urban, environmental, regulatory, or sensor data where benchmark claims depend on explicit variable semantics and evidence status. The protocol is less directly suited to settings where the benchmark target is purely unstructured marketing text or where source disclosure is legally impossible. Even in those cases, parts of the protocol, especially the task charter and variable registry, may still be useful as internal governance tools. This is a scope decision of the present article, not a literature-derived theorem.

Several failure cases are foreseeable. Source volatility can defeat reconstruction when listings disappear, APIs change, or real-time streams are not archived. Semantic drift can arise when variables such as energy labels, land-use classes, or zoning codes retain their names but change institutional meaning across jurisdictions. Inferred explanations can be unstable when the explanation object is sensitive to model perturbation. Synthetic scenarios can appear coherent while violating unstated physical or regulatory constraints. Policy benchmarks can fail because comparator groups are weak or intervention timing is ambiguously coded. RPBP does not remove these risks; it requires them to be declared, logged, and assigned to a specific benchmark layer. [1] [3] [5] [6]

11. Conclusion

This article proposes a research protocol for reproducible PropTech benchmarking. Using a targeted synthesis of six studies, it identifies transferable requirements for benchmark governance: replication-oriented data documentation, evidence-linked state representation, explicit explanation objects, task-specific comparator rules, and separation of observed, derived, inferred, and synthetic outputs. Those requirements are formalized as the Reproducible PropTech Benchmark Protocol, with benchmark families, mandatory layers, a reproducibility score, status-transition logging rules, family-specific acceptance criteria, and a prospective inter-team evaluation design. [1] [2] [3] [4] [5] [6]

What has been established here is conceptual and procedural. The article specifies a benchmark protocol and defines conditions under which future implementations could test it. What remains untested is whether the proposed thresholds, weights, floors, and tolerance rules are well calibrated; whether independent teams can apply RPBP with acceptable burden; and whether RPBP improves the stability of benchmark decisions. Under this protocol, RPBP would be considered empirically supported only after prospective inter-team studies show that registered benchmark instances can be reconstructed and that pass-fail decisions remain stable within the declared family-specific acceptance criteria.

References

1. Lin, Wenbo; Wu, Kaichuang; Zhang, Xiaolu; Gu, Xiaoling; Jiang, Ming; Wang, Jun; Han, Feng (2026). A comprehensive dataset for uncovering housing price drivers in emerging cities: Nanning as a case study. Data in Brief. 66, 112884. doi:10.1016/j.dib.2026.112884.
2. Jing, Zijian; Zhu, Liyi; Chen, Tianyi; Hovestadt, Ludger; Li, Li (2026). Vision-Based Digital Twin and AI Agent Framework for Low-Cost, Explainable Indoor Building Inspection and Safety Assessment. Sensors. 26(15), 4992. doi:10.3390/s26154992.
3. Le, Dat; Rajasegarar, Sutharshan; Luo, Wei; Nguyen, Thanh Thi; Angelova, Maia (2026). From prediction to explanation for price–volume dynamics in real estate market. Interface Focus. 16(3), 20250069. doi:10.1098/rsfs.2025.0069.
4. Piseddu, Tommaso; Vanhuyse, Fedra (2026). Existing evidence on the impact of climate risk on real estate valuations: a systematic map. Environmental Evidence. 15(1), 8. doi:10.1186/s13750-026-00389-6.
5. Eichholtz, Piet; Kok, Nils; Sun, Xudong (2026). The impact of minimum energy performance standards on the commercial real estate market. Nature Communications. 17(1), 4038. doi:10.1038/s41467-026-70684-w.
6. Piccialli, Francesco; Amitrano, Sara; Cerciello, Donato; Borrelli, Anna; Prezioso, Edoardo; Canzaniello, Marzia (2025). A digital twin framework for urban parking management and mobility forecasting. Nature Communications. 16(1), 9400. doi:10.1038/s41467-025-65306-w.