The biophysical modeling efforts in diffusion MRI have grown considerably over the past 25 years. In this review, we dwell on the various challenges along the journey of bringing a biophysical model from initial design to clinical implementation, identifying both hurdles that have been already overcome and outstanding issues. First, we describe the critical initial task of selecting which features of tissue microstructure can be estimated using a model and which acquisition protocol needs to be implemented to make the estimation possible. The model performance should necessarily be tested in realistic numerical simulations and in experimental data – adapting the fitting strategy accordingly, and parameter estimates should be validated against complementary techniques, when/if available. Secondly, the model performance and validity should be explored in pathological conditions, and, if appropriate, dedicated models for pathology should be developed. We build on examples from tumors, ischemia and demyelinating diseases. We then discuss the challenges associated with clinical translation and added value. Finally, we single out four major unresolved challenges that are related to: the availability of a microstructural ground truth, the validation of model parameters which cannot be accessed with complementary techniques, the development of a generalized standard model for any brain region and pathology, and the seamless communication between different parties involved in the development and application of biophysical models of diffusion.