Big Data and Hadoop overview

-
In a nut shell – “Big Data is the problem and Hadoop (FW) is one of the solutions to solve the problem using HDFS, MapReduce”.
 

Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. But it’s not the amount of data that’s important. It’s what organizations do with the data that matters. Big data is a large set of the data that is generated through the FACEBOOK,YOUTUBE, and many social sites. That is in the form of video, image, text, graphics, sensors, and many other things. Big data has divided in to 3 types Structured, Semi Structured, Unstructured data.
 
 


Big Data has also been defined by the four “V”s: Volume, Velocity, Variety, and Value.

a)      Volume: The amount of data.

b)      Velocity: A fast rate that data is received and perhaps acted upon

c)       Variety: New unstructured data types. Unstructured and semi-structured data types, such as text, audio, and video require additional processing to both derive meaning and the supporting metadata.
d)      Value: Data has intrinsic value—but it must be discovered.

Big Data capabilities:

·        For real-time, key-value data stores, such as NoSQL, allow for high performance, index-based retrieval.
·        For real-time, key-value data stores, such as NoSQL, allow for high performance, index-based retrieval.
·        For batch processing, a technique known as “Map Reduce,” filters data according to a specific data discovery strategy. After the filtered data is discovered, it can be analyzed directly, loaded into other unstructured or semi-structured databases, sent to mobile devices, or merged into traditional data warehousing environment and correlated to structured data.
Hadoop (Open Source FW) is one of the tools designed to handle big data. Hadoop and other software products work to interpret or parse the results of big data searches through specific proprietary algorithms and methods. Hadoop is an open-source program under the Apache license that is maintained by a global community of users. It includes various main components, including a MapReduce set of functions and a Hadoop distributed file system (HDFS).

The idea behind MapReduce is that Hadoop can first map a large data set, and then perform a reduction on that content for specific results.The HDFS system then acts to distribute data across a network or migrate it as necessary. Database administrators, developers and others can use the various features of Hadoop to deal with big data in any number of ways. For example, Hadoop can be used to pursue data strategies like clustering and targeting with non-uniform data, or data that doesn't fit neatly into a traditional table or respond well to simple queries. Various steps in accessing/visualizing the Big Data.

Steps in accessing/visualizing the Big Data:

High level architecture of Apache Hadoop:



 Spark is a framework for performing general data analytics on distributed computing cluster like Hadoop. Spark can be accessed through "putty" after logging into Hadoop server by using the below command.

$spark -shell --name

We can use Pega to access Big Data (API’s or direct integration using HBase/HDFS in PRPC V7). For more details, check PDN site.
References:
1) www.google.com
2) www.youtube.com
3) www.tutorialspoint.com
4) https://people.cs.kuleuven.be
5) https://www.sics.se 
     

Popular posts from this blog

Connecting Claude to Pega Infinity 25.1.3 via MCP — Step-by-Step

-
Image
Enable AI-powered access to your Pega Investigative Case Management data directly from Claude prompts using the Model Context Protocol (MCP) and OAuth 2.0. What is MCP and why connect Pega to Claude? The Model Context Protocol (MCP) is an open standard that lets AI assistants like Claude securely call tools and read data from external systems — all from a natural-language prompt. Think of it as a universal adapter between your enterprise systems and AI. Pega Infinity V25.1.3 ships with a first-class MCP service built on its existing REST framework. Once enabled, Claude can: Query and summarize investigative cases without switching apps Retrieve assignment details and case status on demand Trigger Pega workflows from conversational prompts Version requirement: This guide is based on Pega Infinity V25.1.3 (Community Edition) with the Rule-Service-MCP rule type. Earlier versions may not include native MCP support. Architecture overview The connection uses three layers: Claude as the AI c...
Read more

itextpdf API to generate PDF doc from an image file using Pega PE

-
Image
Probably some of the techies might have an idea on an opensource called " itextpdf " (earlier name is lowagie ) to convert an image file(png,jpeg,jpg,tiff,gif etc.) to a PDF file. It is very simple to use this API in Pega function to build the logic using java code. Moreover, this API comes in one single jar file and we need to import it in Pega before we start using it.  I see the conversion comes good in PDF file but not sure if any limitations of this API in terms of conversion is like formatting, font, images etc. So far so good and may be we can keep this API as one of our solutions while working on the requirements accordingly.  Heard that same API can be used to convert the text file to PDF but I didn't explore on it. Those who are interested to work on text -> PDF can take a look at the documentation. Don't hesitate to contact me offline for any further discussion in this context :) :) Prerequisites for doing this exercise: 1) itextpdf 5.4.0 jar file 2) Peg...
Read more

Understanding of Hugging Face platform for AI/ML platform

-
Image
Hugging Face is a platform that provides tools and libraries for natural language processing (NLP), machine learning, and artificial intelligence. It is known for its Transformers   library, which offers pre-trained models for tasks such as text classification, translation, question answering, and more. Hugging Face also hosts a model hub where users can share and discover pre-trained models. Hugging Face is useful in various domains, including: Natural language processing (NLP): Provides tools and pre-trained models for tasks like text classification, translation, sentiment analysis, and question answering. Machine learning research: Offers state-of-the-art models and datasets for researchers to experiment and innovate. Data science and analytics: Helps data scientists analyze and interpret large volumes of text data. AI application development: Enables developers to build AI-powered applications with ease using pre-trained models and APIs. A very simple exercise using the google...
Read more