Auditing Predictive Coverage in Automated Property Valuation: A Reproducible Study of 414 Transactions

1. Introduction

Automated property valuation is commonly formulated as a regression task in which observable property and location characteristics are mapped to a transaction price or a standardized unit price. For lending, taxation, appraisal review, and investment oversight, that framing is incomplete when the system returns only a point estimate. A usable valuation also needs an uncertainty statement. If an interval accompanies the estimate, its operational value depends on whether realized transaction outcomes fall inside such intervals at approximately the advertised rate. Intervals that are too narrow invite unwarranted confidence, whereas intervals that are excessively wide may attain a nominal level while contributing little to decision support. [1] [2]

This study addresses that issue in a bounded empirical setting using a public real-estate valuation dataset from Sindian District, New Taipei City, Taiwan. The repository documents 414 complete observations, six predictors, and an outcome measuring house price per unit area. Because the data are public and the schema is compact, the dataset is suitable for a reproducible audit in which uncertainty calculations can be reconstructed exactly rather than summarized only at a high level. [3]

The interval method used here is split conformal prediction. In regression, conformal methods can wrap a base predictor and produce prediction intervals with finite-sample marginal coverage guarantees under exchangeability. That property is attractive when the analyst wants an auditing layer that does not rely on a correctly specified residual distribution or a correct functional form. In this study, conformal prediction is used in that restricted role: not as a valuation-specific innovation, but as a transparent benchmark for checking interval calibration on held-out data. [1] [2]

The empirical contribution is deliberately narrow. This article reports a completed secondary computational analysis performed through an automated workflow on one public housing dataset. The main question is whether ordinary least squares combined with split conformal prediction at nominal 90% coverage attains held-out empirical coverage near that target under repeated random partitions and under one coarse date-ordered stress check. The study does not propose a new conformal theorem, does not compare against uncomputed machine-learning alternatives, and does not claim automatic generalization to other markets or property types. [1] [3]

2. Prior work and evidence scope

The cited literature serves two distinct roles in this computational study. First, it provides the methodological basis for predictive-coverage auditing through conformal inference. Second, it supplies direct valuation context on automated valuation models, appraisal error, model evaluation, and human–machine comparison. The article does not present a separate review study; instead, the cited work is used to define the problem, justify the analytical design, and bound the interpretation of the completed computations. [1] [4] [5] [6]

The methodological foundation comes from work on distribution-free predictive inference for regression. That work formulates the prediction problem for independent and identically distributed observations, defines a target coverage level 1 − alpha, and shows how conformal procedures can construct prediction bands with finite-sample marginal validity. Split conformal prediction is especially relevant because it offers a practical compromise between computational cost and validity. [1]

A broader tutorial treatment helps interpret the method. It explains how a point predictor can be coupled with a nonconformity measure to produce prediction regions that are valid under exchangeability and nested across confidence levels. In regression settings these regions are typically intervals. This perspective supports treating interval calibration as an evaluable property distinct from point-prediction accuracy. [2]

The valuation studies serve a different function. One recent automated valuation article defines an automated valuation model as a mathematically based program for estimating property prices and argues that predictive performance depends on both the chosen learning algorithm and the selected feature configuration. In its application, performance varied materially across feature configurations and model pipelines, with tree-based methods performing strongly. These findings do not provide predictive-coverage guarantees, but they do support careful disclosure of variables, preprocessing choices, and validation design when a valuation model is audited. [4]

A commercial real-estate appraisal study is relevant because it examines deviations between appraisals and transaction prices and evaluates whether interpretable machine-learning methods can capture structured variation in those deviations. The verified passages support the narrower point that appraisal outputs can differ substantially from realized sale prices and that machine-learning applications in valuation must contend with heterogeneity, data limitations, and overfitting risk when data are scarce. These concerns reinforce the importance of reporting uncertainty performance rather than only central predictions. [5]

Research on human–machine collaboration in valuation informs evaluation design rather than interval theory. The cited study stresses validation on unseen data, direct comparison of predictive systems under controlled informational conditions, and the possibility that automated guidance may assist some experts yet hinder others when outputs conflict with existing workflows. These observations support the practical importance of reporting uncertainty measures that can later be inspected in appraisal review, even though they do not establish anything about conformal coverage itself. [6]

Taken together, the cited work reveals a specific gap. The valuation studies address prediction accuracy, feature engineering, appraisal deviations, and human–model interaction, but the verified passages do not establish finite-sample predictive coverage guarantees for automated valuation models. Interval-validity claims in this article therefore rest on conformal-prediction methodology, while the valuation papers are used for problem framing, variable context, and evaluation design. This separation matters because good point predictions do not imply calibrated intervals. [1] [4] [5] [6]

3. Research question, dataset, and operational definitions

The primary empirical question is: under a fixed six-covariate specification on the public housing dataset, how closely does split conformal prediction built on ordinary least squares achieve nominal 90% held-out coverage? A second question is procedural: what exact audit sequence, if documented fully, allows another analyst to reproduce the same calculations from the public files and the recorded software environment? [3] [1]

The empirical base contains 414 real-estate transactions collected from Sindian District, New Taipei City, Taiwan. The documented predictors are transaction date in decimal years, house age in years, distance to the nearest MRT station in meters, number of convenience stores in the walking living circle, latitude, and longitude. The outcome is house price per unit area, measured in units of 10,000 New Taiwan dollars per ping, where one ping equals 3.3 square meters. There are no missing values in the repository description. [3]

The analysis therefore concerns a small structured tabular specification defined by the public schema. Richer inputs often used in recent automated valuation studies, such as image-derived indicators, extended neighborhood features, or larger sets of engineered variables, are outside the scope of both the dataset and the present computation. That boundary is analytically useful because it isolates the coverage behavior of a simple, inspectable baseline. [3] [4]

The nominal miscoverage level is alpha = 0.1, so the target coverage level is 0.9. Empirical test coverage is defined exactly as coverage = (1/n_test) × sum I{L_i <= y_i <= U_i}, where [L_i, U_i] is the prediction interval for test case i. Mean interval width is defined as (1/n_test) × sum (U_i − L_i). Point accuracy is summarized by mean absolute error, MAE = (1/n_test) × sum |y_i − yhat_i|, and root mean squared error, RMSE = sqrt[(1/n_test) × sum (y_i − yhat_i)^2]. [1]

A central distinction is that between marginal and conditional coverage. Marginal coverage concerns average performance over future draws from the joint data-generating process. Conditional coverage would require intervals to attain the nominal rate within subpopulations or at specific covariate values. The conformal guarantee used here is marginal under exchangeability. It does not imply that every property subtype, location pattern, or accessibility band is equally well calibrated. Any subgroup findings reported below are therefore descriptive diagnostics rather than formal guarantees of local validity. [1] [2]

This distinction has practical consequences. A nominal 90% interval attached to a single property should not be read as a property-specific 90% conditional probability given its exact attributes. The formal claim is weaker and population based: under the required assumptions, about 90% of future outcomes will fall inside such intervals on average. A coverage audit should therefore verify aggregate calibration first and then inspect whether substantively important subsets suggest departures that may matter operationally. [1]

4. Computational design and reproducibility framework

The completed analysis used the public dataset created by I-Cheng Yeh. To support exact reconstruction, the workflow records the SHA-256 hash of the downloaded archive as aa437bdac3ca23200258a0a58d251c0af90651aeaf7c7a07aaa714f0f9f25e2c and the SHA-256 hash of the extracted CSV file as 9d84b73c188d2f1e463b5b7ec342906ed544c702e3b10648389f2f30c07f5be8. The software environment used Python 3.13.5, NumPy 2.3.0, and pandas 2.3.0. These details matter because implementation differences can alter fitted coefficients, residual rankings, and therefore conformal interval endpoints. [3]

The base predictor is ordinary least squares with an intercept and six standardized covariates. Standardization is estimated on the training partition only. For each predictor, the training mean and training standard deviation are computed, training observations are converted to z-scores, and the same fixed transformation is then applied to calibration and test observations. This train-only scaling rule prevents leakage from later partitions into model fitting or preprocessing. No interaction terms, nonlinear basis expansions, variable selection steps, or hyperparameter searches were added. [3] [1]

Partition sizes were fixed at 248 observations for training, 83 for calibration, and 83 for testing. This follows the logic of split conformal prediction: the regression model is fitted on one subset, while a disjoint calibration subset is used to estimate the residual threshold that determines interval width. The separation is not cosmetic. The validity argument depends on comparing a future residual with calibration residuals generated under the same fitted predictor. [1]

Two validation regimes were executed. First, 100 random partitions were generated using integer seeds from 0 through 99. Because the same 414 observations appear in different roles across these runs, the resulting summaries describe sensitivity to partition choice within this fixed dataset; they are not independent replications from newly sampled populations. Second, one coarse date-ordered split was created by sorting records by decimal-year transaction date and breaking ties by original record identifier. Because the date variable is recorded at a coarse decimal-year resolution, this design should be interpreted as a chronology stress check rather than as strict out-of-time forecasting. [3]

This restrained design differs from richer automated valuation applications in the reviewed literature. One recent study evaluates multiple regressors, feature-selection procedures, and image-fusion pipelines. Another analyzes structured appraisal deviations using boosting trees. The present study does none of those things. Its design objective is narrower: isolate a simple and inspectable baseline on which interval calibration can be audited exactly and reproduced from public materials. [4] [5]

5. Implemented split-conformal audit procedure

The audit sequence has eight steps. First, verify dataset identity by file hashing and confirm the six-predictor, one-outcome schema. Second, create train, calibration, and test partitions and record the exact split rule. Third, standardize predictors using training data only. Fourth, fit the ordinary least squares model with an intercept on the transformed training data. Fifth, compute calibration nonconformity scores as absolute prediction errors. Sixth, convert those scores into a residual threshold through an explicit order-statistic rule. Seventh, form prediction intervals for the test set from the point predictions and that threshold. Eighth, report point accuracy, interval width, empirical coverage, and limited subgroup diagnostics together. [1] [3]

The conformal rule can be stated exactly. Let the calibration scores be s_i = |y_i − yhat_i| for i = 1, ..., m, with m = 83. Sort the scores so that s_(1) <= s_(2) <= ... <= s_(m). Define k = ceil((m + 1) × (1 − alpha)). With m = 83 and alpha = 0.1, k = 76 and q = s_(76), the 76th ordered calibration score. For a general calibration size with k > m, q must be +infinity; clipping k to m would not preserve the stated finite-sample guarantee. For a test observation with point prediction yhat, the interval is [yhat − q, yhat + q]. Under exchangeability, this construction targets marginal coverage at least 1 − alpha. The procedure therefore attaches an uncertainty band to the base regression fit without assuming Gaussian residuals or correct linear specification. [1]

In this implementation, every test observation within a given split receives the same interval radius q. Interval centers differ because the point predictions differ, but interval widths are constant within a split. This makes the procedure easy to inspect and reproduce. It also highlights a limitation: if predictive uncertainty changes over the feature space, a single residual threshold may be wider than necessary in stable regions and too narrow in more volatile regions. The methodological literature discusses conformal extensions with locally varying length for heteroscedastic data, but such extensions were not implemented here. [1]

The present audit also differs from the methods emphasized in the application literature. The Hong Kong valuation study searches across multiple feature configurations and model classes to improve predictive performance. The commercial appraisal study focuses on boosting trees to capture structured appraisal deviations. By contrast, the current study fixes one low-complexity linear baseline so that interval behavior can be interpreted without the additional confounding introduced by extensive model tuning. This choice trades possible predictive gains for transparency in the coverage audit. [4] [5]

One exploratory subgroup check was added for the coarse date-ordered split: properties located at 1000 meters or less from the nearest MRT station versus those located farther away. This cutoff functions only as a simple transit-access proxy for descriptive inspection. It was not used to tune the model, choose the conformal threshold, or support formal inferential claims. Its purpose is to illustrate how acceptable global marginal coverage can coexist, in this dataset under this diagnostic, with differences across a substantively meaningful subset. [3]

6. Results

Across the 100 repeated random splits, the ordinary least squares plus split-conformal baseline achieved mean empirical coverage of 0.9051, close to the nominal target of 0.9000. The standard deviation of coverage across splits was 0.0464, with a minimum of 0.7229 and a maximum of 0.9880. Mean interval width was 25.0405 with standard deviation 3.1707, ranging from 20.2531 to 35.8763. Mean absolute error averaged 6.2481 with standard deviation 0.5633, while RMSE averaged 8.8590 with standard deviation 1.6418 and ranged from 6.6638 to 12.5598. On average, the coverage result is consistent with the intended marginal-calibration role of split conformal prediction, but the spread across partitions shows that realized performance on a single split can differ materially in a dataset of this size. [1] [3]

These repeated-split summaries require careful interpretation. The average coverage was near nominal, yet some individual splits fell well below 0.9. Each test partition contains only 83 observations, and the conformal radius is itself estimated from 83 calibration residuals. Realized coverage on a single partition can therefore depart noticeably from the long-run target even when the procedure is functioning as intended. Because the 100 runs reuse the same observations in different roles, their variation should be read as partition sensitivity within this dataset rather than as independent population replication. [1]

In the coarse date-ordered split, the training, calibration, and test partitions again contained 248, 83, and 83 observations. Mean absolute error was 7.2496 and RMSE was 9.1918. Empirical coverage at nominal 90% was 0.9157, corresponding to 76 covered outcomes among 83 test cases. Mean interval width was 25.5468. Relative to the repeated random-split summary, point-prediction error was somewhat higher while coverage remained close to nominal. Because the temporal ordering is only partial, this result should be interpreted as a chronology stress check rather than as evidence from a strict future holdout. [3] [1]

The recorded date ranges clarify the limit of that chronology. In the partially ordered design, calibration observations spanned decimal-year values from 2013.25 to 2013.4166667, whereas test observations spanned 2013.4166667 to 2013.5833333. The shared boundary value means that some calibration and test cases carried the same recorded transaction date and were separated only by original record order. The design therefore preserves coarse sequencing but does not isolate a distinct later period in the stronger sense often desired for market forecasting. [3]

The exploratory subgroup diagnostic suggested modest heterogeneity by MRT access. Among test observations located within 1000 meters of an MRT station, n = 45, empirical coverage was 0.9333 and MAE was 7.0234. Among observations farther than 1000 meters, n = 38, coverage was 0.8947 and MAE was 7.5174. These subgroup counts are small, so the differences should not be overstated. Their value is diagnostic: satisfactory overall marginal coverage can coexist in this dataset under this diagnostic with some local variation across a feature-defined subset. [3] [1]

Taken together, the results support a restrained empirical claim. On this public dataset, a simple linear predictor wrapped in split conformal inference produced held-out marginal coverage close to nominal on average while also yielding reportable point-prediction error and interval width. The analysis therefore shows that automated valuation assessment can include a joint uncertainty dashboard—MAE, RMSE, empirical coverage, and mean interval width—instead of relying exclusively on point metrics. [1] [3]

7. Interpretation in the context of valuation research

The principal finding is not that ordinary least squares is an especially strong valuation model. The finding is that a transparent coverage audit can be conducted on a disclosed automated-valuation baseline and can reveal whether nominal interval levels are approximately realized on held-out data. This distinction matters because point accuracy and interval calibration answer different questions. A model may perform adequately on MAE or RMSE while still providing misleading uncertainty intervals, or it may attain nominal coverage only by producing intervals so wide that they lose operational value. [1] [2]

The result also has to be understood against the application literature. The Hong Kong automated valuation study reports that predictive performance is sensitive to feature configuration and model choice, and its framework uses richer information than the six-variable tabular baseline analyzed here. The commercial appraisal study likewise shows that valuation deviations can contain structured patterns that machine-learning methods may capture. These papers suggest that stronger point predictors than the present baseline may be plausible, but they do not imply that interval calibration can be presumed. Any richer automated valuation model would still require its own coverage audit. [4] [5]

The human–machine collaboration study adds another practical angle. Its design compares machine predictions, expert judgments, and hybrid use under controlled information conditions, and its discussion notes that automated guidance can sometimes conflict with expert workflows. From that perspective, interval reporting may serve appraisal review by making model uncertainty explicit rather than leaving it implicit in a single predicted number. A reviewer who sees both a point valuation and an empirically audited interval has more information than one who sees only a point estimate. [6]

At the same time, the present results should not be extended beyond their evidence. The study concerns one district, one outcome definition, one sparse predictor set, one base regressor, and one conformal variant. It establishes that transparent interval auditing is feasible in this setting. It does not establish that the same interval widths, calibration properties, or subgroup patterns would hold in larger, more heterogeneous, or more rapidly changing property markets. [3] [1]

8. Reproducibility and availability

A substantive feature of this study is that the computations can be reconstructed from public data and recorded implementation details. The workflow specifies dataset identity, file hashes, split sizes, seed range, preprocessing rule, conformal order-statistic formula, and core software versions. Code and data accompany this article. Such reporting is especially important in conformal auditing because apparently minor choices—such as whether scaling used all observations or training observations only—can alter fitted values, calibration residuals, and the resulting interval threshold. [3] [1]

Reproducibility in this setting has two levels. The first is file-level reproducibility: an analyst can verify that the same raw and extracted files were used by checking the recorded hashes. The second is computational reproducibility: an analyst can rerun the workflow under the stated software environment and reconstruct the reported summaries for the repeated random splits and the coarse date-ordered split. Because the procedure contains no hyperparameter search, no stochastic model fitting beyond split generation, and no hidden feature-engineering stage, the computational pathway remains short enough for direct external inspection. [3]

This emphasis on exact reconstruction is consistent with the rationale of predictive-coverage auditing. Coverage claims are easy to state but can be sensitive to data partitioning, preprocessing, and residual-threshold implementation. By disclosing the complete audit sequence, the study allows readers to inspect whether the reported uncertainty behavior follows from the stated procedure rather than from undocumented analytical discretion. [1]

9. Limitations and failure cases

Several limitations bound interpretation. First, the dataset is small and geographically concentrated, containing 414 transactions from one district in Taiwan. The results therefore do not justify claims about transportability to other cities, countries, property types, or market regimes. Second, the public schema contains only six predictors. Broader automated valuation pipelines may include additional locational measures, image-derived variables, and more extensive feature engineering. If omitted structure is important, residual variation may increase and intervals required for marginal coverage may widen or behave differently. [3] [4]

Third, the chronology check is not a strict forecast experiment. The date field uses coarse decimal-year values, and the calibration-test boundary includes tied dates that were broken by original record identifier. Strong claims about temporal transport beyond this dataset are therefore not supported. Fourth, the 100 repeated random splits are descriptive rather than inferential replications because they recycle the same observations. Their means and standard deviations summarize sensitivity to partition choice within this fixed dataset, not sampling variation across newly collected studies. [3] [1]

Fifth, subgroup diagnostics remain limited. The MRT-distance split is exploratory, uses small cell counts, and does not establish conditional coverage, fairness, or the absence of localized appraisal bias. The public data do not contain protected demographic labels, and the design does not include expert valuations. The analysis therefore cannot assess coverage across protected groups, compare machine intervals with human uncertainty judgments, or test whether interval display improves human–machine collaboration during appraisal review. [6] [3]

Potential failure cases should be stated directly. If future observations are not exchangeable with the calibration sample, marginal coverage may move away from nominal. If market volatility changes over time, a threshold calibrated on earlier cases may become too small or too large. If uncertainty is strongly heteroscedastic across neighborhoods or property subtypes, the global symmetric interval used here may be inefficient or locally miscalibrated. If many small subgroups are inspected after the fact, apparent undercoverage may arise from sampling noise alone. These are not objections to conformal prediction as such; they are conditions under which a simple split-conformal baseline may need redesign or local adaptation. [1] [2]

10. Coverage-audit sequence as a practical protocol

Although this article reports completed calculations, its broader contribution is a coverage-audit sequence that can be reused for similarly structured valuation datasets. The sequence begins with public specification of the prediction target and covariates, followed by leakage-controlled preprocessing, clear partitioning into training, calibration, and test subsets, and paired reporting of point and interval metrics. The key analytic principle is that uncertainty assessment should be treated as a first-class evaluation target rather than an afterthought to point prediction. [1] [3]

In practical deployment, the same logic could be extended to richer models or additional valuation settings. A more complex automated valuation model may reduce MAE or RMSE, but the coverage audit would still ask whether intervals attain their nominal level and at what width. Application studies that emphasize model comparison, feature configuration, or human–machine workflows provide useful design ideas for such extensions, but they do not remove the need for direct interval evaluation on unseen cases. [4] [6] [5]

The present study also suggests a reporting norm for valuation research. When an automated valuation model is proposed, the minimum evaluation set should include one point metric, one interval-calibration metric, and one interval-size metric, all computed on held-out data. In small datasets, repeated partition summaries can supplement a single split by showing how sensitive performance is to sample allocation, provided that such summaries are described as descriptive rather than as independent replications. [1]

11. Conclusion

This computational study examined predictive-coverage auditing for automated property valuation on a public dataset of 414 residential transactions. Using ordinary least squares with six covariates, train-only standardization, and split conformal prediction at nominal 90% coverage, the implemented workflow produced mean empirical coverage of 0.905 across 100 repeated random splits and 0.916 in one coarse date-ordered stress check. Mean interval width was about 25 outcome units, with MAE and RMSE reported alongside coverage in both validation regimes. For this dataset and procedure, interval calibration could therefore be audited transparently rather than inferred from point-prediction error alone. [3] [1]

What has been established is bounded and practical: a reproducible coverage-audit sequence was implemented on a public valuation dataset, and its held-out results were reported exactly. What remains untested is equally important. The study does not establish conditional coverage, does not provide strict out-of-time validation, does not compare against alternative automated valuation models, and does not address demographic fairness or expert deployment. The appropriate conclusion is therefore modest: predictive coverage auditing is feasible and informative in this small automated valuation setting, but wider claims require richer data, broader model comparisons, and more demanding validation designs. [1] [4] [6] [5] [3]

References

1. Lei, Jing; G’Sell, Max; Rinaldo, Alessandro; Tibshirani, Ryan J.; Wasserman, Larry (2018). Distribution-Free Predictive Inference for Regression. Journal of the American Statistical Association. 113(523), 1094-1111. doi:10.1080/01621459.2017.1307116.
2. Shafer, Glenn; Vovk, Vladimir (2008). A Tutorial on Conformal Prediction. Journal of Machine Learning Research. 9(12), 371–421. Publisher record.
3. Yeh, I-Cheng (2018). Real Estate Valuation. UCI Machine Learning Repository. doi:10.24432/C5J30W.
4. Deng, Lin (2025). Real estate valuation with multi-source image fusion and enhanced machine learning pipeline. PLOS One. 20(5), e0321951. doi:10.1371/journal.pone.0321951.
5. Deppner, Juergen; von Ahlefeldt-Dehn, Benedict; Beracha, Eli; Schaefers, Wolfgang (2025). Boosting the Accuracy of Commercial Real Estate Appraisals: An Interpretable Machine Learning Approach. The Journal of Real Estate Finance and Economics. 71(2), 314-351. First published online 2023. doi:10.1007/s11146-023-09944-1.
6. Kmen, Christopher; Navratil, Gerhard; Kattenbeck, Markus; Giannopoulos, Ioannis (2026). Evaluating human–machine collaboration through a comparative analysis of experts, machine learning, and hybrid approaches in real estate valuation. Scientific Reports. 16(1), 4044. doi:10.1038/s41598-025-34099-9.