Load algonauts videos

By Neuromatch Academy

Content creators: Kshitij Dwivedi

Produtction editors: Spiros Chavlis

---

Terms of Use

By using this colab to download the brain dataset (i.e., data including fMRI, images, labels), you agree to the following terms:

1. You will use the Datasets only for non-commercial research and educational purposes.

2. You will NOT distribute the Datasets or any parts thereof.

3. Massachusetts Institute of Technology make no representations or warranties regarding the datasets, including but not limited to warranties of non-infringement or fitness for a particular purpose.

4. You accept full responsibility for your use of the datasets and shall defend and indemnify Massachusetts Institute of Technology and International Business Machines Corporation, including its employees, officers and agents, against any and all claims arising from your use of the datasets, including but not limited to your use of any copyrighted images that you may create from the datasets.

5. You will treat people appearing in this data with respect and dignity.

6. This data comes with no warranty or guarantee of any kind, and you accept full liability.

Algonauts2021 dataset

Here we will load fMRI responses (whole brain + 9 ROIs) to 1000 video clips of 3 seconds and visualize in glass brain using nilearn.

P.S. The dataset provided is part of Algonauts 2021 Challenge. Please check the details here if you are interested in participating.

Install dependencies

Install dependencies

Install dependencies

Install dependencies

Install dependencies

# @title Install dependencies
!pip install nilearn --quiet
!pip install decord --quiet

# Imports
import os
import glob
import pickle

import numpy as np
import matplotlib.pyplot as plt
import nibabel as nib

from nilearn import datasets
from nilearn import plotting

fsaverage = datasets.fetch_surf_fsaverage()

Get the dropbox link to download dataset by filling this google form.

Important: You MUST set dropbox_link parameter dl=1, e.g., https://www.dropbox.com/myurl/participants_data.zip?dl=1

Enter the dropbox link

Enter the dropbox link

Enter the dropbox link

Enter the dropbox link

Enter the dropbox link

# @title Enter the dropbox link
dropbox_link = '' # @param {type:"string"}

Run the cell

Run the cell

Run the cell

Run the cell

Run the cell

# @title Run the cell
import requests, zipfile, io

# Use the dropbox link to download the data
print(f"Dropbox link: {dropbox_link}")

if dropbox_link:
  fname1 = 'participants_data_v2021'
  fname2 = 'AlgonautsVideos268_All_30fpsmax'
  if not os.path.exists(fname1) or not os.path.exists(fname2):
    print('Data downloading...')
    r = requests.get(dropbox_link)
    z = zipfile.ZipFile(io.BytesIO(r.content))
    z.extractall()
    print('Data download is completed.')
  else:
    print('Data are already downloaded.')


  url = 'https://github.com/Neural-Dynamics-of-Visual-Cognition-FUB/Algonauts2021_devkit/raw/main/example.nii'
  fname = 'example.nii'
  if not os.path.exists(fname):
    r = requests.get(url, allow_redirects=True)
    with open(fname, 'wb') as fh:
      fh.write(r.content)
  else:
    print(f"{fname} file is already downloaded.")

else:
  print('You need to submit the form and get the dropbox link')

Dropbox link: 
You need to submit the form and get the dropbox link

#fMRI Data

The Algonauts dataset provides human brain responses to a set of 1,102 3-s long video clips of everyday events. The brain responses are measured with functional magnetic resonance imaging (fMRI). fMRI is a widely used brain imaging technique with high spatial resolution that measures blood flow changes associated with neural responses.

The training set consists of 1,000 video clips and the associated brain responses. The brain responses are provided here in two tracks corresponding to two independent tracks in the Algonauts challenge.

In the first track, brain responses provided are from selected voxels across the whole brain showing reliable responses to videos. The figure below shows the reliability values of different voxels in the brain.


In the second track, brain responses provided are from a set of specific regions of interest (ROIs) known to play a key role in visual perception. These ROIs start in early and mid-level visual cortex (V1, V2, V3, and V4) and extend into higher-level cortex that responds preferentially to all objects or particular categories (Body- EBA; Face - FFA, STS; Object - LOC; Scene - PPA). In the figure below we show the masks of the above mentioned ROIs for an example subject.


#Loading fMRI data The dataset contains 1,000 3-second videos + fMRI human brain data of 10 subjects in response to viewing videos from this set.

The ROI data is provided for 9 ROIs of the visual brain (V1, V2, V3, V4, LOC, EBA, FFA, STS, PPA) in a Pickle file (e.g. V1.pkl) that contains a num_videos x num_repetitions x num_voxels matrix. For each ROI, we selected voxels that showed significant split-half reliability.

The whole brain data is provided for selected voxels across the whole brain showing reliable responses to videos in a Pickle file (e.g. WB.pkl) that contains a num_videos x num_repetitions x num_voxels matrix.

In this section, we demonstrate how to load fMRI data for a given ROI.

Utility functions for data loading

Utility functions for data loading

Utility functions for data loading

Utility functions for data loading

Utility functions for data loading

# @title Utility functions for data loading
def save_dict(di_, filename_):
  with open(filename_, 'wb') as f:
      pickle.dump(di_, f)

def load_dict(filename_):
  with open(filename_, 'rb') as f:
    u = pickle._Unpickler(f)
    u.encoding = 'latin1'
    ret_di = u.load()
    # print(p)
    # ret_di = pickle.load(f)
  return ret_di


def visualize_activity(vid_id,sub):
  # Setting up the paths for whole brain data
  fmri_dir = './participants_data_v2021'
  track = "full_track"

  # get the right track directory depending on whole brain/ROI choice
  track_dir = os.path.join(fmri_dir, track)

  # get the selected subject's directory
  sub_fmri_dir = os.path.join(track_dir, sub)

  #result directory to store nifti file
  results_dir = '/content/'

  # mapping the data to voxels and storing in a nifti file
  fmri_train_all,voxel_mask = get_fmri(sub_fmri_dir,"WB")
  visual_mask_3D = np.zeros((78,93,71))
  visual_mask_3D[voxel_mask==1]= fmri_train_all[vid_id,:]
  brain_mask = './example.nii'
  nii_save_path =  os.path.join(results_dir, 'vid_activity.nii')
  saveasnii(brain_mask,nii_save_path,visual_mask_3D)

  # visualizing saved nifti file
  plotting.plot_glass_brain(nii_save_path,
                          title='fMRI response',plot_abs=False,
                          display_mode='lyr',colorbar=True)


def get_fmri(fmri_dir, ROI):
  """This function loads fMRI data into a numpy array for to a given ROI.
  Parameters
  ----------
  fmri_dir : str
    path to fMRI data.
  ROI : str
    name of ROI.

  Returns
  -------
  np.array
    matrix of dimensions #train_vids x #repetitions x #voxels
    containing fMRI responses to train videos of a given ROI
  """

  # Loading ROI data
  ROI_file = os.path.join(fmri_dir, ROI + ".pkl")
  ROI_data = load_dict(ROI_file)
  # averaging ROI data across repetitions
  ROI_data_train = np.mean(ROI_data["train"], axis=1)
  if ROI == "WB":
    voxel_mask = ROI_data['voxel_mask']

    return ROI_data_train, voxel_mask

  return ROI_data_train

def saveasnii(brain_mask,nii_save_path,nii_data):
  img = nib.load(brain_mask)
  nii_img = nib.Nifti1Image(nii_data, img.affine, img.header)
  nib.save(nii_img, nii_save_path)

Loading fMRI data and inspecting dimensions

Loading fMRI data and inspecting dimensions

Loading fMRI data and inspecting dimensions

Loading fMRI data and inspecting dimensions

Loading fMRI data and inspecting dimensions

# @title Loading fMRI data and inspecting dimensions
# Select Subject
sub = 'sub05'  # @param ["sub01","sub02","sub03","sub04","sub05","sub06","sub07","sub08","sub09","sub10"]

# Select ROI
ROI = 'V1'  # @param ["WB", "V1", "V2","V3", "V4", "LOC", "EBA", "FFA","STS", "PPA"]

######## fMRI data loader wrapper code ###################################
fmri_dir = './participants_data_v2021'
if ROI == "WB":  # Loading whole brain data
  track = "full_track"  # stored in full_track directory
else:  # Loading ROI data
  track = "mini_track" # stored in mini_track directory

# get the right track directory depending on whole brain/ROI choice
track_dir = os.path.join(fmri_dir, track)

# get the selected subject's directory
sub_fmri_dir = os.path.join(track_dir, sub)

# Load the fMRI data for the selected subject and ROI
if track == "full_track":
  fmri_train_all,_ = get_fmri(sub_fmri_dir, ROI)
else:
  fmri_train_all = get_fmri(sub_fmri_dir, ROI)

######## fMRI data loader wrapper code ###################################

# Visualize the fMRI responses in a heatmap
f, ax = plt.subplots(figsize=(12, 5))
ax.set(xlabel="Voxel", ylabel="Stimulus")
heatmap = ax.imshow(fmri_train_all, aspect="auto", cmap='jet', vmin=-1, vmax=1)
f.colorbar(heatmap, shrink=.5, label="Response amplitude (Z)")
f.tight_layout()

The above matrix shows each individual voxel’s (columns) response to 1000 videos (rows). To find out more details about how the fMRI responses were preprocessed please click here.

Visualize a given video (with id from 0-999)) and its corresponding brain activity

Visualize video

Visualize video

Visualize video

Visualize video

Visualize video

# @title Visualize video
from IPython.display import HTML
from base64 import b64encode

vid_id = 1  # @param {type: "integer"}
video_dir = './AlgonautsVideos268_All_30fpsmax'

########### Video display code #################################################
video_list = glob.glob(video_dir + '/*.mp4')
video_list.sort()

mp4 = open(video_list[vid_id],'rb').read()
data_url = "data:video/mp4;base64," + b64encode(mp4).decode()
HTML("""
<video width=400 controls>
  <source src="%s" type="video/mp4">
</video>
""" % data_url)
########### Video display code #################################################

Visualize corresponding brain response

Visualize corresponding brain response

Visualize corresponding brain response

Visualize corresponding brain response

Visualize corresponding brain response

# @title Visualize corresponding brain response
visualize_activity(vid_id, sub)

The values in the fMRI response above correspond to beta values from 5s to 9s after stimulus onset calculated using an Finite Impulse Response (FIR) model. The beta values were averaged across time and z-scored across videos.

---

References

1. Radoslaw Martin Cichy, Kshitij Dwivedi, Benjamin Lahner, Alex Lascelles, Polina Iamshchinina, Monika Graumann, Alex Andonian, Apurva Ratan Murty, Kendrick Kay, Gemma Roig, Aude Oliva. (2021). The Algonauts Project 2021 Challenge: How the Human Brain Makes Sense of a World in Motion. arXiv, arXiv:2104.13714v1.

2. Radoslaw Martin Cichy, Gemma Roig, Alex Andonian, Kshitij Dwivedi, Benjamin Lahner, Alex Lascelles, Yalda Mohsenzadeh, Kandan Ramakrishnan, and Aude Oliva. (2019). The Algonauts Project: A Platform for Communication between the Sciences of Biological and Artificial Intelligence. arXiv, arXiv:1905.05675.