This blog is an improved version of this AWS blog post by Will Kruse, In his blog post, he walks us through how to set up CloudWatch alarms on IAM configuration changes. Here’s a quick overview of the setup that was suggested on how AWS usage ends up triggering a CloudWatch alarm. The users of... » read more
It is common when organizations have large workloads based on on a multitude of AWS services, they may lose track of how resources are being used. In a nutshell, identifying resources can take rigorous effort. On AWS, utilization and cost go hand in hand and tagging helps ensure that the resources are managed efficiently. In... » read more
In an earlier post, we had described the need for automating the Data Engineering pipeline for Machine Learning based systems. Today, we will expand the scope to setup a fully automated MLOps pipeline using Google Cloud Composer. Cloud Composer Cloud Composer is official defined as a fully managed workflow orchestration service that empowers you to... » read more
A common use case in Machine Learning life cycle is to have access to the latest training data so as to prevent model deterioration. A lot of times data scientists find it cumbersome to manually export data from data sources such as relational databases or NoSQL data stores or even distributed data. This necessitates automating... » read more
Airflow is used to create code pipeline where we can schedule and monitor our workflows. A workflow can be a collection of tasks to be executed like a flowchart. It is like an orchestra conductor that controls all different data processing tools/tasks under one roof. Why Airflow? Open source. Can handle upstream/downstream in an elegant... » read more