Data-driven decision making involves estimating the value of each potential option and selecting the one with the highest estimated efficacy. This approach underpins a wide array of modern marketing and AI applications. However, several papers have shown that the estimated effectiveness of the chosen options will be systematically over-optimistic, even when the estimated outcomes are themselves unbiased. Using simulations calibrated to realistic parameters from marketing papers, we first demonstrate that the magnitude of the winner's curse is often high in real-life contexts, and that the severity of the winner's curse depends on the true performance difference between options relative to the noise level in the data, the number of candidate options, and the number of observations. We further show that using machine learning to target treatments to different consumers can lead to extremely high levels of winner's curse, especially if the machine learning model is very flexible. We propose a correction method based on an $m$-out-of-$n$ bootstrap and demonstrate that our bootstrap usually outperforms the solutions that have been previously proposed in the literature. Finally, using over 32,000 real-life A/B tests from the Upworthy Research Archive, we document substantial winner’s curse that accounts for over 70\% of the estimated lift.