Sub-Challenge #2

Challenge name

All models are wrong: are yours useful? 

Purpose and relevance of the challenge

With this challenge, we aim to understand the current ability of the MRI field in modelling white matter (WM) tissue microstructure. The challenge will consist of a number of simulated WM digital environments (or “substrates”), on the scale of individual voxels, generated by changing in a controlled fashion a range of microstructural features, including:

  • self-diffusivity (D, um2/ms)
  • intra-fibre volume fraction (f, unitless)
  • fibre radius index [1] (a, um)
  • permeability (k, um/ms).

Participants will be given the simulated signal acquired from the acquisition protocols PGSE, DDE and DODE of sub-challenge #1 and #3. Participants are then asked to estimate any (or all) of the above microstructural features, according to their proposed model of WM microstructure. Participants are also encouraged to submit potential biomarkers that may be indicative of these features even if they do not directly estimate them. The outcome of this challenge will be an evaluation of the accuracy and sensitivity of microstructural parameter estimation.

4 winners will be determined by the most accurate estimates of each of the four indices.

Datasets

The dataset for this challenge is composed of 3D substrates of WM tissues designed following the methods in [2], which provide flexibility and control on microstructural features. The substrates are combined with the Monte Carlo based simulator in Camino [3] to provide synthetic dMRI signal.

The challenge data will be based upon 256 substrates representative of an array of white matter voxels. For each substrate, each of the 4 tissue parameters is swept through a range of physiologically realistic values (for example, D from 0.7-3 um2/ms, f from 0.4-0.7, etc.), and six different levels of signal-to-noise-ratio(SNR is hidden from participants).

The signal is simulated using the three acquisition protocols from sub-challenges #1 and #3: a PGSE acquisition with 1300 unique datapoints using a multi-shell strategy, a DDE with 2 different diffusion times, and a DODE with 5 different frequencies, 5 b-values, and 72 directions each. [In an extended version of this sub-challenge, planned for next year, we will work to make available any user-defined acquisition, in which each participant will be able to create and submit a “scheme” file describing the diffusion sensitizing gradients of any experimental protocol they would like to simulate.]

Link to Data: https://www.synapse.org/#!Team:3388304

Participation (Data given to the participants)

The task will be to estimate any (or all) of the microstructural parameters of interest.

Participants are free to use all or any subset of the generated signals from any set of acquisitions.

Participants will be provided with the following files. Sample scripts to load the data will be provided for popular working environments (MATLAB, Python, C/C++).

Note that for each acquisition type, we include files for (1) a protocol description, (2) the acquisition parameters, and (3) the MR signal. For example, the dataset for each sequence is provided in a text file and consists of M columns, where M = 1536 is the total number of simulated voxels to be analysed (256 voxels * 6 noise levels) and N rows, where N is the number of measurements. [N=3011 measurements for PGSE, N=2000 measurements for DODE, N=800 for DDE).

  1. PGSE_ProtocolDescription.txt: Description of the acquisition parameters for the PGSE sequences
  2. PGSE_AcqParams.txt: Acquisition parameters for all measurements of the PGSE shell dataset. It is a NxA matrix with N rows and A columns, where N is the number of measurements and A is the number of sequence parameters.
  3. PGSE_Simulations.txt: Dataset for PGSE shell sequence
  4. DDE_ProtocolDescription.txt: Description of the acquisition parameters for the DDE sequences
  5. DDE_AcqParams.txt: Acquisition parameters for all measurements of the DDE dataset.
  6. DDE_Simulations.txt: Dataset for DDE sequences
  7. DODE_ProtocolDescription.txt: Description of the acquisition parameters for the DODE sequences
  8. DODE_AcqParams.txt: Acquisition parameters for all measurements of the DODE dataset.
  9. DODE_Simulations.txt: Dataset for DODE sequences

Submission

Participants are asked to submit any or all of the following for each of the 256 environments:

  • self-diffusivity (D, um2/ms);
  • intra-fibre volume fraction (f, unitless);
  • fibre radius index [1] (a, um) computed as:

assuming a Gamma distribution P(a) of axonal radii a.

Screen Shot 2019-04-08 at 5.32.18 PM

  • permeability (k, um/ms).

Submission for all sub-challenges must be submitted as a single compressed folder with a unique name for each submission (we recommend TeamName_SubmissionName.zip).

Within this folder will be two (and possibly three files).

  1. {SEQ}_prediction.txt, where SEQ = {’PGSE_shells’, ’DDE’, ’DODE’}. This file should contain a 4×1536 matrix. Rows 1 through 4 represent predictions for radius index, volume fraction, residence time, and diffusivity, respectively. 1536 is the number of simulated substrates. Importantly, it is not necessary to estimate ALL indices. If only a single index is estimated, NaN should be submitted for the other three rows. If two indices are estimated, two rows will be composed of NaN, etc. Also, except for the pre-exchange residence time, the effect of all the non-estimated features on the estimated ones will NOT be excluded. For the pre-exchange residence time only, if its estimation is not provided (i.e. third column in {SEQ}_prediction.txt file all NaN), then the effect of exchange between intra- and extra- axonal water will be neglected by considering in the evaluation only the contribution of the impermeable configurations among the 256 simulated.
  1. Info.txt: this file contains relevant information for the challenge organizers. For all sub-challenges, this file must contain: the (1) submission name, (2), submission abbreviation, (3), team name, (4) team members who made meaningful contributions, (5) member affiliation, (6) brief one sentence submission description, (7) extended submission description (to enable reproducibility of methods) (8) all relevant citations, (9) observations (optional), and (9) relevant discussion points (optional). For sub-challenge #2 specifically, we also ask for: (10) number of free model parameters, (11) the type of model (signal/tissue), (12) noise assumptions, (13) model parameters estimation and algorithm optimization strategies, (14) pre-processing (outlier strategies), (15) data and subsets of data used to form predictions.
  1. (Optional). If you wish to submit a potential biomarker that is not a direct estimator of the four parameters, but wish for it to be included in evaluation (we will assess correlation of markers both with each other and with ground truth data, and assess robustness to SNR) you may submit a modelparameters.txt file. This will simply be a Nx1536 matrix, where N is the number of model parameters you would like included in analysis. Please, make sure that the relationship between each of the N column in the modelparameters.txt file and the definition of the corresponding model parameter in the Info.txt is very clear. For instance, when defining your model parameters in the Info.txt use simple names like p1 for model parameter 1, p2 for model parameter 2, and so on. Then, in the modelparameters.txt file, values in column N=1 would correspond to p1, those in N=2 to p2 and so on.

Example submissions files will be made available with the data

Evaluation

Reference “gold standard” for this sub-challenge will be ground truth values for the four specific microstructural metrics, known by design of the numerical simulation study.

We will evaluate accuracy (or bias) and precision (or statistical dispersion) of dMRI measured metrics related to the microstructural features, compared to ground truth values known by design from numerical simulations. Specifically, accuracy for each estimated microstructural feature fj|j=1..4will be quantified by mean normalized error across the Ncmicrostructural scenarios taken into account:

Precision will be evaluated as interquartile range of the normalized error:

Screen Shot 2019-04-08 at 5.32.30 PM

Overall score of accuracy and precision will be quantified by the mean squared error:

Screen Shot 2019-04-08 at 5.32.35 PM

The code for the evaluation metrics will be open source. There will be a winning submission for each microstructural measure j, defined by the submission (for that feature) with the lowest scorej.

Screen Shot 2019-04-08 at 5.32.46 PM

How to get the data

Please see “Registration and Data Access” Page.

Sub-challenge Chair

Marco Palombo <University College London>

Daniel Alexander <University College London>

[1]       Alexander, D. C., Hubbard, P. L., Hall, M. G., Moore, E. A., Ptito, M., Parker, G. J., & Dyrby, T. B. (2010). Orientationally invariant indices of axon diameter and density from diffusion MRI. Neuroimage52(4), 1374-1389.

[2] Hall, M. G., & Alexander, D. C. (2009). Convergence and parameter choice for Monte-Carlo simulations of diffusion MRI. IEEE transactions on medical imaging, 28(9), 1354-1364.

[3]       Y. B. P. A. Cook, S. Nedjati-Gilani, K. K. Seunarine, M. G. Hall, G. J. Parker, D. C. Alexander,, “Camino: Open-Source Diffusion-MRI Reconstruction and Processing,” in 14th Scientific Meeting of the International Society for Magnetic Resonance in Medicine, Seattle, WA, USA, May 2006, p. 2759.