Data-Engineering

A while ago I wrote about the challenge of orchestrating multi-omic data in translational research. That post described the problem. This one is about what we did next - we built Foundry120, a research operating system for translational science. The Problem, Restated In modern translational research, the hard part is no longer generating data. Sequencing is cheap. Assays are routine. You can now do 6000 genes at subcellular resolution and pin their x,y coordinates?! Most groups are drowning in data, not starved of it. ...

Over the last few years, one of the key challenges we have faced in our multi-omic translational studies is the orchestration of multi-omic datasets. In this blog post, I summarise some of the strategies we have employed to manage this complexity, after dedicating significant time to considering this problem. Industry Data Warehousing I recommend the Kimball book on data warehousing (The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modelling) as a starting guide to see what other people have done in tackling ‘big data’. ...

Traditional File Uploads in Django Handling non-trivial big data file uploads (think >5gb/file) in Django can be challenging. A typical file upload storage strategy in Django is to use django-storages and an Amazon S3 backend. In the traditional way, your browser uploads the file directly to Django which then transmits the data to S3. This creates two problems: Django needs to have enough memory to hold your uploaded file for retransmission. The Django worker receiving the file upload is blocked until the upload is finished. Streaming the upload onwards to S3 doubles the amount of bandwidth used. What if we could instead upload directly into AWS S3 while registering the upload with Django? I stumbled across this excellent article by Radoslav Georgiev of HackSoft: https://www.hacksoft.io/blog/direct-to-s3-file-upload-with-django ...