How To Upload And Save Large Data To Google Colaboratory From Local Drive?

January 15, 2024 Post a Comment

I have downloaded large image training data as zip from this Kaggle link https://www.kaggle.com/c/yelp-restaurant-photo-classification/data How do I efficiently achieve the follow

Solution 1:

!pip install kaggle
api_token = {"username":"USERNAME","key":"API_KEY"}
import json
import zipfile
import os
with open('/content/.kaggle/kaggle.json', 'w') as file:
    json.dump(api_token, file)
!chmod 600 /content/.kaggle/kaggle.json
!kaggle config set -n path -v /content
!kaggle competitions download -c jigsaw-toxic-comment-classification-challenge
os.chdir('/content/competitions/jigsaw-toxic-comment-classification-challenge')
for file inos.listdir():
    zip_ref = zipfile.ZipFile(file, 'r')
    zip_ref.extractall()
    zip_ref.close()

There is minor change on line 9, without which was encountering error. source: https://gist.github.com/jayspeidell/d10b84b8d3da52df723beacc5b15cb27 couldn't add as comment cause rep.

Solution 2:

You may refer with these threads:

Also check out the I/O example notebook. Example, for access to xls files, you'll want to upload the file to Google Sheets. Then, you can use the gspread recipes in the same I/O example notebook.

Solution 3:

You may need to use kaggle-cli module to help with the download.

It’s discussed in this fast.ai thread.

Solution 4:

I just wrote this script that downloads and extracts data from the Kaggle API to a Colab notebook. You just need to paste in your username, API key, and competition name.

The manual upload function in Colab is kind of buggy now, and it's better to download files via wget or an API service anyway because you start with a fresh VM each time you open the notebook. This way the data will download automatically.

Solution 5:

Another option is to upload the data to dropbox (if it can fit), get a download link. Then in the notebook do

!wget link -0 new-name && ls

Python Guru