https://shaunchuah.github.io/tags/django/ Recent content in Django on Shaun Chuah https://shaunchuah.github.io/profile_photo.jpg https://shaunchuah.github.io/profile_photo.jpg Hugo -- 0.151.0 en-us Sun, 10 Dec 2023 18:59:15 +0000 https://shaunchuah.github.io/posts/how-to-handle-big-data-uploads-in-django/ Sun, 10 Dec 2023 18:59:15 +0000 https://shaunchuah.github.io/posts/how-to-handle-big-data-uploads-in-django/ <h2 id="traditional-file-uploads-in-django">Traditional File Uploads in Django</h2> <p>Handling non-trivial big data file uploads (think &gt;5gb/file) in Django can be challenging. A typical file upload storage strategy in Django is to use django-storages and an Amazon S3 backend. In the traditional way, your browser uploads the file directly to Django which then transmits the data to S3.</p> <p>This creates two problems:</p> <ol> <li>Django needs to have enough memory to hold your uploaded file for retransmission.</li> <li>The Django worker receiving the file upload is blocked until the upload is finished.</li> <li>Streaming the upload onwards to S3 doubles the amount of bandwidth used.</li> </ol> <h2 id="what-if-we-could-instead-upload-directly-into-aws-s3-while-registering-the-upload-with-django">What if we could instead upload directly into AWS S3 while registering the upload with Django?</h2> <p>I stumbled across this excellent article by Radoslav Georgiev of HackSoft: <a href="https://www.hacksoft.io/blog/direct-to-s3-file-upload-with-django">https://www.hacksoft.io/blog/direct-to-s3-file-upload-with-django</a></p>