How to ETL API data to AWS S3 Bucket using Apache Airflow?
Reading Time: 7 minutes 2.5 quintillion bytes of data are produced every day with 90% of it generated solely in the last 2 years (Source: Forbes). Data is pulled, cleaned, transfigured & then presented for analytical purposes & put to use in thousands of applications to fulfill consumer needs & more. While generating insights from the data is important, […]
Easily build ETL Pipeline using Python and Airflow
Reading Time: 5 minutes Apache Airflow is an open-source workflow management platform for authoring, scheduling, and monitoring workflows or data pipelines programmatically. Python is used to write Airflow, and Python scripts are used to create workflows. It was created by Airbnb. In this blog, we will show how to configure airflow on our machine as well as write a […]
How to build a serverless event-driven workflow with AWS Glue and Amazon EventBridge(Part 1)?
Reading Time: 9 minutes Have you ever wondered how huge IT companies construct their ETL pipelines for production? Are you curious about how TBs and ZBs of data are effortlessly captured and rapidly processed to a database or other storage for data scientists and analysts to use? The answer is the serverless data integration service, AWS Glue. It makes […]