Big Data Full Course

Big Data Hadoop/ Spark/ Scala/ Kafka Developer Course

Teklabs delivers you the highest quality of training with hands-on approach and industry recognized curriculum. Training will be provided by our experienced and certified professionals in the respective fields. Training will include exposure to real time work environments and will also prepare you to attend interviews confidently. You do not require prior knowledge for any of the courses listed. We give you an excellent opportunity for enhancing your career in course of your choice. Each training module in IT consulting and training is specially designed so that by the end of the course you will be ready to face client interviews.

Course Duration

• 60 hrs Training
• 5-6 weeks course
• 20 hours Project Work
• Free - Demo Training available for interested students

Hadoop/ Spark/ Scala/ Kafka Development Course Content

Module 1 – Introduction to Big Data & Hadoop

• What is Big data?
• Sources of Big data
• Categories of Big data
• Characteristics of Big data
• Use-cases of Big data
• Traditional RDBMS vs Hadoop
• What is Hadoop?
• History of Hadoop
• Understanding Hadoop Architecture
• Fundamental of HDFS (Blocks, Name Node, Data Node, Secondary Name Node)
• Block Placement &Rack Awareness
• HDFS Read/Write
• Drawback with 1.X Hadoop
• Introduction to 2.X Hadoop
• High Availability

Module 2 – Linux

• Making/creating directories
• Removing/deleting directories
• Print working directory
• Change directory
• Manual pages
• Help
• Vi editor
• Creating empty files
• Creating file contents
• Copying file
• Renaming files
• Removing files
• Moving files
• Listing files and directories
• Displaying file contents

Module 3 – HDFS

• Understanding Hadoop configuration files
• Hadoop Components- HDFS, MapReduce
• Overview of Hadoop Processes
• Overview of Hadoop Distributed File System
• The building blocks of Hadoop
• Hands-On Exercise: Using HDFS commands

Module 4– Map Reduce

• Map Reduce 1(MRv1)
• Map Reduce Introduction
• How Map Reduce works?
• Communication between JobTracker and TaskTracker
• Anatomy of a Map Reduce Job Submission
• MapReduce-2(YARN)
• Limitations of Current Architecture
• YARN Architecture
• Node Manager & Resource Manager

Module 5-SQL

• DDL Commands
• DML Commands
• SQL Constraints
• Aggregate functions
• Scalar functions
• Joins
• Views
• Indexes

Module 6-Scala

• Setup Java and JDK
• Install Scala with IntelliJ IDE
• Develop Hello World Program using Scala
• Introduction to Scala
• REPL Overview
• Declaring Variables
• Programming Constructs
• Code Blocks
• Scala Functions - Getting Started
• Scala Functions - Higher Order and Anonymous Functions
• Scala Functions - Operators
• Object Oriented Constructs - Getting Started
• Object Oriented Constructs - Objects
• Object Oriented Constructs - Classes
• Object Oriented Constructs - Companion Objects and Case Class
• Operators and Functions on Classes
• External Dependencies and Import
• Scala Collections - Getting Started
• Mutable and Immutable Collections
• Sequence (Seq) - Getting Started
• Linear Seq vs. Indexed Seq
• Scala Collections - Primitive Operations
• Scala Collections - Sorting Data
• Scala Collections - Grouping Data
• Scala Collections - Set
• Scala Collections - Map
• Tuples in Scala
• Development Cycle - Developing Source code
• Development Cycle - Compile source code to jar using SBT
• Development Cycle - Setup SBT on Windows
• Development Cycle - Compile changes and run jar with arguments
• Development Cycle - Setup IntelliJ with Scala
• Development Cycle - Develop Scala application using SBT in IntelliJ

Module 7-Getting started with Spark

• What is Apache Spark & Why Spark?
• Spark History
• Unification in Spark
• Spark ecosystem Vs Hadoop
• Spark with Hadoop
• Introduction to Spark’s Python and Scala Shells
• Spark Standalone Cluster Architecture and its application flow

Module 8-Programming with RDDS, DFs & DSs

• RDD Basics and its characteristics, Creating RDDs
• RDD Operations
• Transformations
• Actions
• RDD Types
• Lazy Evaluation
• Persistence (Caching)
• Module-Advanced spark programming
• Accumulators and Fault Tolerance
• Broadcast Variables
• Custom Partitioning
• Dealing with different file formats
• Hadoop Input and Output Formats
• Connecting to diverse Data Sources
• Module-Spark SQL
• Linking with Spark SQL
• Initializing Spark SQL
• Data Frames & Caching
• Case Classes, Inferred Schema
• Loading and Saving Data
• Apache Hive
• Data Sources/Parquet
• JSON
• Spark SQL User Defined Functions (UDFs)

Module9-KAFKA & Spark Streaming

• Getting started with Kafka
• Understanding Kafka Producer and Consumer APIs
• Deep dive into producer and consumer APIs
• Ingesting Web Server logs into Kafka
• Getting started with Spark Streaming
• Getting started with HBASE
• Integrating Kafka-Spark Streaming-HBASE

Module 10-Spark on Amazon Web Services (AWS)

• Introduction
• Sign up for AWS account
• Setup Cygwin on Windows
• Quick Preview of Cygwin
• Understand Pricing
• Create first EC2 Instance
• Connecting to EC2 Instance
• Understanding EC2 dashboard left menu
• Different EC2 Instance states
• Describing EC2 Instance
• Using elastic IPs to connect to EC2 Instance
• Using security groups to provide security to EC2 Instance
• Understanding the concept of bastion server
• Terminating EC2 Instance and relieving all the resources
• Create security credentials for AWS account
• Setting up AWS CLI in Windows
• Creating s3 bucket
• Deleting root access keys
• Enable MFA for root account
• Introduction to IAM users and customizing sign in link
• Create first IAM user
• Create group and add user
• Configure IAM password policy
• Understanding IAM best practices
• AWS managed policies and creating custom policies
• Assign policy to entities (user and/or group)
• Creating role for EC2 trusted entity with permissions on s3
• Assigning role to EC2 instance
• Introduction to EMR
• EMR concepts
• Pre-requisites before setting up EMR cluster
• Setting up data sets
• Setup EMR with Spark cluster using quick options
• Connecting to EMR cluster
• Submitting spark job on EMR cluster
• Validating the results
• Terminating EMR Cluster

Placement Assistance Provided

• Resume Preparation
• 1-O-1 Feedback Sessions
• Mock Interviews
• Marketing Profile