Programming

How to store Django Static and Media files in Amazon S3

Django is a free and open-source, Python-based web framework that follows the model–template–views architectural pattern. Django advertises itself as “the web framework for perfectionists with deadlines” and “Django makes it easier to build better Web apps more quickly and with less code”.  Django is known for the speed at which you can develop apps without compromising on robustness.

Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance.

In this guide, we will learn how to use the Amazon S3 service to handle static assets and the user uploaded files, that is, the media assets.

Related content:

# Amazon S3 Bucket

Before beginning, you will need an AWS account. If you’re new to AWS, Amazon provides a free tier with 5GB of S3 storage.

In this guide we will use aws cli to manage aws resources but you can achieve the same with UI. Give the bucket a unique, DNS-compliant name and select a region:

aws s3api create-bucket \
    --bucket citizix-djuploads \
    --region eu-west-1 \
    --create-bucket-configuration LocationConstraint=eu-west-1

Turn off “Block all public access”:

aws s3api put-public-access-block \
    --bucket citizix-djuploads \
    --public-access-block-configuration "BlockPublicAcls=false,IgnorePublicAcls=false,BlockPublicPolicy=false,RestrictPublicBuckets=false"

To confirm

aws s3api get-public-access-block \
    --bucket citizix-djuploads

You should now see your bucket back on the main S3 page.

# IAM Access

Although you could use the AWS root user, it’s best for security to create an IAM user that only has access to S3 or to a specific S3 bucket. What’s more, by setting up a group, it makes it much easier to assign (and remove) access to the bucket. So, we’ll start by setting up a group with limited permissions and then create a user and assign that user to the group.

# Iam Policy

Let us create a policy that will only allow access to our s3 bucker. Save this as AllowDJUploadsS3Access.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::citizix-djuploads",
                "arn:aws:s3:::citizix-djuploads/*"
            ]
        }
    ]
}

Then create the policy called AllowDJUploadsS3Access using this command:

aws iam create-policy --policy-name <span style="font-size: calc(11px + 0.2em);">AllowDJUploadsS3Access --policy-document file://AllowDJUploadsS3Access.json</span>

# Iam User

Finally let us create an Iam user that we can use from our application.

aws iam create-user --user-name djuploads-django-s3-access

To attach the policy we created to the user:

arn:aws:iam::648921681300:policy/AllowDJUploadsS3Access --user-name djuploads-django-s3-access

Finally we can generate aws access key and secret:

aws iam create-access-key --user-name djuploads-django-s3-access

# Setting up the Project

Ensure that you have python 3 installed. Consult your OS installation guide. I can confirm that it is installed in my machine using this command:

➜ python --version
Python 3.9.13

# Create virtualenv

Although not strictly required, it is highly recommended to start your project in a “virtual environment.” A virtual environment is a container (a directory) that holds a specific version of Python and a set of modules (dependencies), and which does not interfere with the operating system’s native Python or other projects on the same computer. By setting up a different virtual environment for each project you work on, various Django projects can run on different versions of Python, and can maintain their own sets of dependencies, without risk of conflict.

To create the Virtual environment:

python3 -m pip install virtualenv
python3 -m venv <env-folder>

The virtual environment is now set up. In order to use it, it must be activated in the terminal you want to use it.

To ‘activate’ the virtual environment (any Python version):

source /<path to venv>/bin/activate

This changes your prompt to indicate the virtual environment is active. (<env-folder>) $.

From now on, everything installed using pip will be installed to your virtual env folder, not system-wide.

To leave the virtual environment use deactivate :

(<env-folder>) $ deactivate

# Install and setting up django

We can now install Django in our virtual env. Use this command to install the latest version of django:

pip install django

Next, we can create a django project. A Django project is a Python codebase that contains a Django settings file. A project can be created by the Django admin through the command django-admin startproject NAME . The project typically has a file called manage.py at the top level and a root URL file called urls.py . manage.py is a project specific version of django-admin , and lets you run management commands on that project. For example, to run your project locally, use python manage.py runserver . A project is made up of Django apps.

Use this command to create a project. I am calling mine djuploads.

django-admin startproject djuploads

A Django app is a Python package that contains a models file (models.py by default) and other files such as app-specific urls and views. An app can be created through the command django-admin startapp NAME (this command should be run from inside your project directory). For an app to be part of a project, it must be included in the INSTALLED_APPS list in settings.py . If you used the standard configuration, Django comes with several apps of it’s own apps preinstalled which will handle things like authentication for you. Apps can be used in multiple Django projects.

cd djuploads
python manage.py startapp uploads

That will create a folder named uploads which will contain the django app files. 

Now add the app to installed apps in djuploads/settings.py:

INSTALLED_APPS = [
    ...
    '<span style="font-size: calc(11px + 0.2em);">uploads',</span>
]

Update the djuploads/urls.py file with the following content to include the uploads.urls:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('', include('uploads.urls')),
    path('admin/', admin.site.urls),
]

# Database connection

By default django comes set up to work with sqlite

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': BASE_DIR / 'db.sqlite3',
    }
}

You can also set up another database of your choice from the list of supported databases. In my case I am using postgres. First ensure that you have a postgres database set up and running then create credentials to use with django. Next install python postgres dependency:

python3 -m pip install psycopg2-binary

Next update the settings file with Postgres connection properties:

DATABASES = {  
    'default': {  
        'ENGINE': 'django.db.backends.postgresql',  
        'NAME': 'djuploads',  
        'USER':'djuploads',  
        'PASSWORD':'Python@123',  
        'HOST':'localhost',  
        'PORT':'5432'  
    }  
}

# Django Storages

django-storages is a collection of custom storage backends for Django.

Install django-storages, to use S3 as the main Django storage backend, and boto3, to interact with the AWS API.

pip install django-storages
pip install boto3

The boto3 library is a public API client to access the Amazon Web Services (AWS) resources, such as the Amazon S3. It’s an official distribution maintained by Amazon.

The django-storages is an open-source library to manage storage backends like Dropbox, OneDrive and Amazon S3. It’s very convenient, as it plugs in the built-in Django storage backend API. In other words, it will make you life easier, as it won’t drastically change how you interact with the static/media assets. We will only need to add a few configuration parameters and it will do all the hard work for us.

Add storages to the INSTALLED_APPS in settings.py:

INSTALLED_APPS = [
    ...
    'storages',
]

Next, we need to update the handling of static files in settings.py:

if os.getenv("USE_S3") == "True":
    AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
    AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")
    AWS_STORAGE_BUCKET_NAME = os.getenv("AWS_STORAGE_BUCKET_NAME")
    AWS_S3_OBJECT_PARAMETERS = {
        'CacheControl': 'max-age=86400',
    }
    AWS_STATIC_LOCATION = 'static'

    STATICFILES_DIRS = [
        os.path.join(BASE_DIR, 'static'),
    ]
    # AWS_S3_ENDPOINT_URL = os.getenv("AWS_S3_ENDPOINT_URL")
    AWS_DEFAULT_ACL = None
    AWS_QUERYSTRING_AUTH = False
    AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'

    # STATIC_URL = f'{AWS_S3_ENDPOINT_URL}/{AWS_STATIC_LOCATION}/'
    STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{AWS_STATIC_LOCATION}/'

    STATICFILES_STORAGE = 'storages.backends.s3boto3.S3StaticStorage'

    AWS_PUBLIC_MEDIA_LOCATION = 'media/public'
    # MEDIA_URL = f'{AWS_S3_ENDPOINT_URL}/{AWS_PUBLIC_MEDIA_LOCATION}/'
    DEFAULT_FILE_STORAGE = 'djfiles.storage_backends.PublicMediaStorage'

    AWS_PRIVATE_MEDIA_LOCATION = 'media/private'
    PRIVATE_FILE_STORAGE = 'djfiles.storage_backends.PrivateMediaStorage'
else:
    STATICFILES_DIRS = []

    STATIC_ROOT = BASE_DIR / "static/"

    STATIC_URL = 'static/'

    MEDIA_ROOT = BASE_DIR / "media/"

    MEDIA_URL = 'media/'

Take note of USE_S3 and STATICFILES_STORAGE:

  1. The USE_S3 environment variable is used to turn the S3 storage on (value is True) and off (value is False). So, you could configure two Docker compose files: one for development with S3 off and the other for production with S3 on.
  2. The STATICFILES_STORAGE setting configures Django to automatically add static files to the S3 bucket when the collectstatic command is run.

Remember to add the appropriate environment variables.

To prevent users from overwriting existing static files, media file uploads should be placed in a different subfolder in the bucket. We’ll handle this by creating custom storage classes for each type of storage.

Add a new file called storage_backends.py to the djuploads folder:

from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage


class StaticStorage(S3Boto3Storage):
    location = settings.AWS_STATIC_LOCATION
    default_acl = 'public-read'

class PublicMediaStorage(S3Boto3Storage):
    location = settings.AWS_PUBLIC_MEDIA_LOCATION
    file_overwrite = False
    default_acl = 'public-read'

class PrivateMediaStorage(S3Boto3Storage):
    location = settings.AWS_PRIVATE_MEDIA_LOCATION
    default_acl = 'private'
    file_overwrite = False
    custom_domain = False

With the DEFAULT_FILE_STORAGE setting now set, all FileFields will upload their content to the S3 bucket.

# Django Model

A model is the single, definitive source of information about your data. It contains the essential fields and behaviors of the data you’re storing. Generally, each model maps to a single database table.

Here we create a post model with the fields title, post, created_at and updated_at.

Create uploads/models.py

from django.db import models


class Upload(models.Model):
    caption = models.CharField(max_length = 100)
    upload = models.FileField(upload_to='file_uploads')
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    def __str__(self):
        return self.caption

    class Meta:
        db_table = "uploads"

This is how you use the private and public storages

from django.db import models
from djuploads.storage_backends import PublicMediaStorage, PrivateMediaStorage

class UploadPublic(models.Model):
    upload = models.FileField(upload_to='file_uploads', storage=PublicMediaStorage())

class UploadPrivate(models.Model):
    upload = models.FileField(upload_to='file_uploads', storage=PrivateMediaStorage())

# Django Forms

Django’s form functionality can simplify and automate vast portions of this work, and can also do it more securely than most programmers would be able to do in code they wrote themselves.

Django handles three distinct parts of the work involved in forms:

  • preparing and restructuring data to make it ready for rendering
  • creating HTML forms for the data
  • receiving and processing submitted forms and data from the client

It is possible to write code that does all of this manually, but Django can take care of it all for you.

Save this in uploads/forms.py:

from django import forms

from .models import Upload

class UploadsForm(forms.ModelForm):
    class Meta:
        model = Upload

        fields = [
            "caption",
            "upload",
        ]

# Django Views

A view function, or view for short, is a Python function that takes a web request and returns a web response. This response can be the HTML contents of a web page, or a redirect, or a 404 error, or an XML document, or an image . . . or anything, really. The view itself contains whatever arbitrary logic is necessary to return that response. This code can live anywhere you want, as long as it’s on your Python path. There’s no other requirement–no “magic,” so to speak. For the sake of putting the code somewhere, the convention is to put views in a file called views.py, placed in your project or application directory.

Add this content to the uploads/views.py&#xA0;file:

from django.shortcuts import render, redirect

from .forms import UploadsForm
from .models import Upload

from logging import getLogger
log = getLogger(__name__)


def index(request):
    files = Upload.objects.all()

    if request.method == "POST":
        form = UploadsForm(request.POST, request.FILES)

        if form.is_valid():
            form.save()
            return redirect('uploads:index')
    else:
        form = UploadsForm()

    return render(request, 'index.html', {'files': files, 'form': form})

# Urls

A clean, elegant URL scheme is an important detail in a high-quality web application. Django lets you design URLs however you want, with no framework limitations.

To design URLs for an app, you create a Python module informally called a URLconf (URL configuration). This module is pure Python code and is a mapping between URL path expressions to Python functions (your views).

This mapping can be as short or as long as needed. It can reference other mappings. And, because it’s pure Python code, it can be constructed dynamically.

Django also provides a way to translate URLs according to the active language.

Save this in uploads/urls.py:

from django.urls import path

from django.urls import re_path as url
from . import views

app_name = 'uploads'


urlpatterns = [
    path('', views.index, name='index'),
]

# Add Templates

Add this to uploads/templates/index.html

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta name="description" content="">
    <meta name="author" content="Eutychus Towett">
    <title>Citizix - Django file upload example</title>

    <link rel="canonical" href="https://citizix.com/">
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.2.0/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-gH2yIJqKdNHPEq0n4Mqa/HGKIhSkIHeL5AyhkYV8i59U5AR6csBvApHHNl/vI1Bx" crossorigin="anonymous">

    <meta name="theme-color" content="#712cf9">

    <style>
      .main-content {
        padding: 70px;
        min-height: 400px;
      }
    </style>
  </head>
  <body>

    <nav class="navbar navbar-expand-md navbar-dark fixed-top bg-dark">
      <div class="container">
        <a class="navbar-brand" href="/">Dj Files</a>
        <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarCollapse" aria-controls="navbarCollapse" aria-expanded="false" aria-label="Toggle navigation">
        <span class="navbar-toggler-icon"></span>
        </button>
      </div>
    </nav>

    <main class="container main-content">
        <h1>Dj Files</h1>
        <form class="form-horizontal" action="{% url 'uploads:index' %}" method="post" enctype="multipart/form-data">
          {% csrf_token %}
          <div class="form-group row">
            <label class="col-sm-2 col-form-label" for="caption">Caption</label>
            <div class="col-sm-8">
              <input type="text" id="caption" name="caption" class="form-control" placeholder="Caption"/>
              <span class="help-block text-danger"></span>
            </div>
          </div>
          <div class="form-group row">
            <label class="col-sm-2 col-form-label" for="upload">Upload</label>
            <div class="col-sm-8">
              <input type="file" id="upload" name="upload" class="form-control" placeholder="Upload" />
              <span class="help-block text-danger"></span>
            </div>
          </div>
          <div class="form-group row">
            <div class="col-sm-8">
              <button type="submit" class="btn btn-primary">Upload</button>
            </div>
          </div>
        </form>

        <hr>

        <div class="files">
          {% if files %}
            {% for file in files %}
              <p><a href="{{file.upload.url}}" target="_blank">{{file.caption}}</a> - {{file.upload}}</p>
            {% endfor %}
          {% else %}
            <p>No uploads</p>
          {% endif %}
        </div>
    </main>

    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.2.0/dist/js/bootstrap.bundle.min.js" integrity="sha384-A3rJD856KowSb7dwlZdYEkO39Gagi7vIsF0jrRAoQmDKKtQBHUuLZ9AsSv4jD4Xa" crossorigin="anonymous"></script>

  </body>
</html>

# Running The Django Project

Migrations are Django’s way of propagating changes you make to your models (adding a field, deleting a model, etc.) into your database schema. They’re designed to be mostly automatic, but you’ll need to know when to make migrations, when to run them, and the common problems you might run into.

First we will need to make migrations for our app:

➜ python manage.py makemigrations
Migrations for 'uploads':
  uploads/migrations/0001_initial.py
    - Create model Upload

Then we can apply the migrations to the database:

➜ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions, uploads
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying sessions.0001_initial... OK
  Applying uploads.0001_initial... OK

Finally we can run server to check the content we just created:

➜ python manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
July 28, 2022 - 02:35:02
Django version 4.0.6, using settings 'djuploads.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

From here you can visit http://127.0.0.1:8000/ then navigate the pages to test the file upload functionality that we just added.

# Conclusion

In this guide we learnt how to work with django to Upload files to S3 using django storages. We walked through how to create a bucket on Amazon S3, configure an IAM user and group, and set up Django to upload and serve static files and media uploads to and from S3.

Checkout this related content:

comments powered by Disqus
© 2020 - 2024 Citizix, Inc.
Citizix Ltd
Built with Hugo
Theme Stack designed by Jimmy