How To Download Amazon S3 Files On To Local Machine In Folder Using Python And Boto3?

I am trying to download a file from Amazon S3 to a predefined folder in the local machine. This is the code and it works fine. But when the file is saved, it saves with lastname of

Solution 1:

Here is my test code.

import boto3
import os

s3 = boto3.resource('s3')
bucket = 'your bucket'
response = s3.Bucket(bucket).objects.all()
# If you want to search only specific path of bucket,
#response = s3.Bucket(bucket).objects.filter(Prefix='path')

path = 'your path'ifnotos.path.exists(path):
    os.makedirs(path)

for item in response:
    filename = item.key.rsplit('/', 1)[-1]
    if filename.endswith('.csv'):
        s3.Object(bucket, item.key).download_file(path + filename)
        print("success")

I have tested the code and it gives a correct name.

---

What is wrong?

I think, there is a missing / in your code for the path.

print(os.path.join('..', 'data', 'lz', 'test_sample', 'sample_file'))

The code gives the result:

../data/lz/test_sample/sample_file

So, in the below step,

S3_Object.download_file(BUCKET_NAME, f, download_path + file_name)

the download_path + file_name will be wrong and it should be:

S3_Object.download_file(BUCKET_NAME, f, download_path + '/' + file_name)

Solution 2:

the following function downloadS recursively the files.

The directories are created locally only if they contain files.

import boto3
import os

def download_dir(client, resource, dist, local='/tmp', bucket='your_bucket'):
paginator = client.get_paginator('list_objects')
for result in paginator.paginate(Bucket=bucket, Delimiter='/', Prefix=dist):
    if result.get('CommonPrefixes') is not None:
        for subdir in result.get('CommonPrefixes'):
            download_dir(client, resource, subdir.get('Prefix'), local, bucket)
    for file in result.get('Contents', []):
        dest_pathname = os.path.join(local, file.get('Key'))
        ifnotos.path.exists(os.path.dirname(dest_pathname)):
            os.makedirs(os.path.dirname(dest_pathname))
        resource.meta.client.download_file(bucket, file.get('Key'), dest_pathname)

The function is called that way:

def _start():
    client = boto3.client('s3')
    resource = boto3.resource('s3')
    download_dir(client, resource, 'clientconf/', '/tmp', bucket='my-bucket')