Real Time Data Engineering Pipeline for Machine Learning
Real Time Data Engineering Pipeline for Machine Learning

Our focus in this post is to leverage Google Cloud Platform’s Big Data Services to build an end to end Data Engineering pipeline for streaming processes. So what is Data Engineering?Data Engineering is associated with data specifically around data delivery, storage and processing. The main goal is to provide a reliable infrastructure for data which... » read more

Processing Kinesis Data Streams with Spark Streaming

Solution Overview : In this blog, we are going to build a real time anomaly detection solution using Spark Streaming. Kinesis Data Streams would act as the input streaming source and the anomalous records would be written as Data Streams in DynamoDB. Amazon Kinesis Data Streams (KDS) is a massively scalable and durable real-time data streaming... » read more

Introduction and Installation of Logstash

So far in this series on ELK (Elasticsearch-Logstash-Kibana) stack, we have talked about Elasticsearch and Kibana(here and here). It’s about time we look at the third pillar of this stack – Logstash. So, what is Logstash? Logstash is an open source data collection engine with real-time pipelining capabilities.  This implies that Logstash can dynamically put... » read more

Introduction to Visualizations using Kibana with Elasticsearch

We talked about Kibana installation on Ubuntu in the previous blog post, let’s look at how to setup some basic visualizations using Kibana with Elasticsearch. Log into Kibana using  http://127.0.0.1:5601. Once the Kibana page opens up, from the left side Menu, click on “Management” and then choose Index Patterns->create Index Pattern and fill in the... » read more

Kibana Installation and Troubleshooting Tips

In the previous blog post, we gave a bird’s eye introduction to Elasticsearch, which is a highly scalable open-source full-text search and analytics engine.  But, everyone does not get a kick out of typing commands for querying data using Elasticsearch. Is there another way around? What is Kibana? Why is it required when I already... » read more