Shrinkage priors for isotonic probability vectors and binary data modeling, with applications to dose–response modeling

• Published in: Pharmaceutical statistics : the journal of the pharmaceutical industry Vol. 23; no. 4; pp. 540 - 556
• Main Authors: Boonstra, Philip S., Owen, Daniel R., Kang, Jian
• Format: Journal Article
• Language: English
• Published: Chichester, UK John Wiley & Sons, Inc 01.07.2024; Wiley Subscription Services, Inc
• Subjects: Bayes Theorem; Clinical Trials as Topic - methods; Computer Simulation; Dirichlet distribution; Dose-Response Relationship, Drug; gamma distribution; horseshoe distribution; Humans; Lung cancer; Lung Neoplasms - drug therapy; Main Paper; Models, Statistical; monotonic relationship; Probability; Random variables; Regression Analysis; Toxicity; underflow; horseshoe distribution; underflow; monotonic relationship; gamma distribution; Dirichlet distribution
• ISSN: 1539-1604; 1539-1612; 1539-1612
• DOI: 10.1002/pst.2372

Motivated by the need to model dose–response or dose‐toxicity curves in clinical trials, we develop a new horseshoe‐based prior for Bayesian isotonic regression modeling a binary outcome against an ordered categorical predictor, where the probability of the outcome is assumed to be monotonically non‐decreasing with the predictor. The set of differences between outcome probabilities in consecutive categories of the predictor is equipped with a multivariate prior having support over simplex. The Dirichlet distribution, which can be derived from a normalized sum of independent gamma‐distributed random variables, is a natural choice of prior, but using mathematical and simulation‐based arguments, we show that the resulting posterior is prone to underflow and other numerical instabilities, even under simple data configurations. We propose an alternative prior based on horseshoe‐type shrinkage that is numerically more stable. We show that this horseshoe‐based prior is not subject to the numerical instability seen in the Dirichlet/gamma‐based prior and that the horseshoe‐based posterior can estimate the underlying true curve more efficiently than the Dirichlet‐based one. We demonstrate the use of this prior in a model predicting the occurrence of radiation‐induced lung toxicity in lung cancer patients as a function of dose delivered to normal lung tissue. Our methodology is implemented in the R package isotonicBayes and therefore suitable for use in the design of dose‐finding studies or other dose–response modeling contexts.