Received 10.02.2026, Revised 17.05.2026, Accepted 25.06.2026 Published 03.08.2026

A sequential decision framework for Bayesian A/B Tests: Reconciling interval precision, practical significance, and resource constraints

Artur Markov*

ArturMarkov0_1@outlook.com



 The study aimed to justify an approach to the planning and completion of Bayesian A/B tests, within which interval certainty criteria are combined with threshold requirements for a practically significant effect and an assessment of the economic feasibility of further data collection. The methodology was based on methods of analytical synthesis, structural-logical classification, logical-algorithmic formalisation, and illustrative case analysis. The study established that in Bayesian A/B tests, “interval precision” has two dimensions: informativeness (the narrowness and stability of the credible interval for supporting an engineering decision) and calibratedness (the proximity of actual coverage to the nominal level, which may deviate for small samples). The study demonstrated that uncertainty decreases with increasing n but with diminishing returns; therefore, further narrowing of intervals requires disproportionately greater expenditure of time and computational or experimental resources and is justified only when such costs are warranted, with the required n depending on the metric and the scale of the effect. The volume of data can be reduced by incorporating prior information into the prior distribution or by defining “sufficiency” based on criteria of evidence or the probability of achieving the objective, whereas high variability of engineering metrics, rare events, low baseline event rates, strict accuracy requirements, and a small effect increase the required n and the duration of the test. To reduce the required volume of data, a sequential framework for concluding a Bayesian A/B test has been proposed, combining an information component, a target component, and an economic feasibility check: “update estimates  →  check threshold  →  check interval → weigh up the benefits and costs of continuing”. The practical applicability of this approach was confirmed by engineering studies, where Bayesian methods reduced resource consumption by 66.6%, 36.7%, and 26.9% in stream computing and achieved up to 25% energy savings, 15% greater energy efficiency, and about 10 ms tighter latency guarantees in edge computing. The practical significance of the results lies in their applicability to researchers and analysts involved in engineering experiments when making decisions regarding the termination of the experiment and the implementation of the proposed changes

solution; experiment; data; sufficiency; certainty
55-66
Markov, A. (2026). A sequential decision framework for Bayesian A/B Tests: Reconciling interval precision, practical significance, and resource constraints. Information Technologies and Computer Engineering, 23(2), 55-66. https://doi.org/10.31649/vitce/2.2026.55

References

  1. Ali, I. (2025). Probabilistic A/B testing with rstanarm. Retrieved from https://cran.r-project.org.
  2. Chennu, S., Maher, A., Pangerl, C., Prabanantham, S., Bae, J.H., Martin, J., & Goswami, B. (2023). Rapid and scalable Bayesian AB testing. arXiv. doi: 10.48550/arXiv.2307.14628.
  3. de Paula Castro, T., Paulino, C.D., & Singer, J.M. (2022). A fair comparison of credible and confidence intervals: An example with binomial proportions. METRON, 80, 371-382. doi: 10.1007/s40300-021-00225-6.
  4. De Santis, F., & Gubbiotti, S. (2021). Sample size requirements for calibrated approximate credible intervals for proportions in clinical trials. International Journal of Environmental Research and Public Health, 18(2), article number 595. doi: 10.3390/ijerph18020595.
  5. Douven, I. (2023). Bayesian stopping. Journal of Mathematical Psychology, 116, article number 102794. doi: 10.1016/j.jmp.2023.102794.
  6. Fu, Q., Moerbeek, M., & Hoijtink, H. (2022). Sample size determination for Bayesian ANOVAs with informative hypotheses. Frontiers in Psychology, 13, article number 947768. doi: 10.3389/fpsyg.2022.947768.
  7. Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2022). Always valid inference: Continuous monitoring of A/B tests. Operations Research, 70(3), 1806-1821. doi: 10.1287/opre.2021.2135.
  8. Kamalbasha, S., & Eugster, M.J.A. (2021). Bayesian A/B testing for business decisions. In P. Haber, T. Lampoltshammer, M. Mayr & K. Plankensteiner (Eds.), Proceedings of the 3rd international data science conference: Data science – analytics and applications (pp. 50-57). Vieweg Wiesbaden: Springer. doi: 10.1007/978-3-658-32182-6_9.
  9. Kelter, R., & Pawel, S. (2025). Bayesian power and sample size calculations for Bayes factors in the binomial setting. arXiv. doi: 10.48550/arXiv.2502.02914.
  10. Kunzmann, K., Grayling, M.J., Lee, K.M., Robertson, D.S., Rufibach, K., & Wason, J.M.S. (2021). A review of Bayesian perspectives on sample size derivation for confirmatory trials. American Statistician, 75(4), 424-432. doi: 10.1080/00031305.2021.1901782.
  11. Liao, J.G., Midya, V., & Berg, A. (2021). Connecting and contrasting the bayes factor and a modified ROPE procedure for testing interval null hypotheses. American Statistician, 75(3), 256-264. doi: 10.1080/00031305.2019.1701550.
  12. Markov, A., & Zaritskyi, O. (2025). Improvements to sequential approaches in A/B testing. In Proceedings of the IEEE 5th international conference on smart information systems and technologies (pp. 1-7). Astana: IEEE. doi: 10.1109/SIST61657.2025.11139233.
  13. Mezzetti, M., Ryan, C.P., Balestrucci, P., Lacquaniti, F., & Moscatelli, A. (2023). Bayesian hierarchical models and prior elicitation for fitting psychometric functions. Frontiers in Computational Neuroscience, 17, article number 1108311. doi: 10.3389/fncom.2023.1108311.
  14. Moerbeek, M. (2021). Bayesian updating: Increasing sample size during the course of a study. BMC Medical Research Methodology, 21, article number 137. doi: 10.1186/s12874-021-01334-6.
  15. Mulier, G., Lin, R., Aparicio, T., & Biard, L. (2024). Bayesian sequential monitoring strategies for trials of digestive cancer therapeutics. BMC Medical Research Methodology, 24, article number 154. doi: 10.1186/s12874-024-02278-3.
  16. Pan, J., & Banerjee, S. (2023). Bayesassurance: An R package for calculating sample size and Bayesian assurance. The R Journal, 15(3), 138-158. doi: 10.32614/RJ-2023-066.
  17. Pananos, D. (2024). Calibration of mind changing for Bayesian AB tests. Retrieved from https://dpananos.github.io.
  18. Quin, F., Weyns, D., Galster, M., & Silva, C.C. (2024). A/B testing: A systematic literature review. Journal of Systems and Software, 211, article number 112011. doi: 10.1016/j.jss.2024.112011.
  19. Rainforth, T., Foster, A., Ivanova, D.R., & Smith, F.B. (2024). Modern Bayesian experimental design. Statistical Science, 39(1), 100-114. doi: 10.1214/23-STS915.
  20. Richardson, T.S., Liu, Y., McQueen, J., & Hains, D. (2022). A Bayesian model for online activity sample sizes. In Proceedings of the 25th international conference on artificial intelligence and statistics (pp. 1775-1785). Valencia: PMLR.
  21. Sahu, D., Nidhi, Chaturvedi, R., Prakash, S., Yang, T., Rathore, R.S., & Alsolbi, I. (2025). Optimizing energy and latency in edge computing through a Boltzmann driven Bayesian framework for adaptive resource scheduling. Scientific Reports, 15, article number 30452. doi: 10.1038/s41598-025-16317-6.
  22. Shen, Y., Psioda, M.A., & Ibrahim, J.G. (2023). BayesPPD: An R package for Bayesian sample size determination using the power and normalized power prior for generalized linear models. The R Journal, 14(4), 335-351. doi: 10.32614/RJ-2023-016.
  23. Stefan, A.M., Schönbrodt, F.D., Evans, N.J., & Wagenmakers, E.-J. (2022). Efficiency in sequential testing: Comparing the sequential probability ratio test and the sequential Bayes factor test. Behavior Research Methods, 54(6), 3100-3117. doi: 10.3758/s13428-021-01754-8.
  24. Stucchio, C. (2015). Bayesian A/B testing at VWO. Retrieved from https://www.chrisstucchio.com.
  25. Sambucini, V. (2024). Hybrid classical-Bayesian approach to sample size determination for two-arm superiority clinical trials. The International Journal of Biostatistics, 20(2), 553-570. doi: 10.1515/ijb-2023-0050.
  26. Tkachuk, V.O., & Shestakova, A.V. (2022). Prospects for the use of mobile applications for digitalization of the main business processes in the restaurant business. Economics, Management and Administration, 4(102), 28-34. doi: 10.26642/ema-2022-4(102)-28-34.
  27. Turner, R.M., Clements, M.N., Quartagno, M., Cornelius, V., Cro, S., Ford, D., Tweed, C.D., Walker, A.S., & White, I.R. (2023). Practical approaches to Bayesian sample size determination in non-inferiority trials with binary outcomes. Statistics in Medicine, 42(8), 1127-1138. doi: 10.1002/sim.9661.
  28. Vasishth, S., Yadav, H., Schad, D.J., & Nicenboim, B. (2023). Sample size determination for Bayesian hierarchical models commonly used in psycholinguistics. Computational Brain & Behavior, 6, 102-126. doi: 10.1007/s42113-021-00125-y.
  29. Williamson, S.F., Williams, C.J., Lendrem, B.C., & Wilson, K.J. (2023). Sample size determination for point-of-care COVID-19 diagnostic tests: A Bayesian approach. Diagnostic and Prognostic Research, 7, article number 17. doi: 10.1186/s41512-023-00153-1.
  30. Wong, T.K., & Tendeiro, J.N. (2025). On a generalizable approach for sample size determination in Bayesian t tests. Behavior Research Methods, 57, article number 130. doi: 10.3758/s13428-025-02654-x.
  31. Wright, A., Ding, S., Philips, S.J., Matthew, R.A., & Ma, X. (2026). Model-based constrained Bayesian optimization of IEEE 802.11 VANET safety messaging. Wireless Networks, 32, 751-766. doi: 10.1007/s11276-025-04080-5.
  32. Zhang, L., Zheng, W., Zheng, K., Zhu, H., Li, C., & Guo, M. (2024). Bayesian-driven automated scaling in stream computing with multiple QoS targets. IEEE Transactions on Parallel and Distributed Systems, 35(7), 1251-1267.
  33. Zheng, H., Jaki, T., & Wason, J.M.S. (2023). Bayesian sample size determination using commensurate priors to leverage preexperimental data. Biometrics, 79(2), 669-683. doi: 10.1111/biom.13649.
  34. Zhou, T., & Ji, Y. (2024). On Bayesian sequential clinical trial designs. The New England Journal of Statistics in Data Science, 2(1), 136-151. doi: 10.51387/23-NEJSDS24.