Welcoem to posts!!
in the future - u will be able to do some more stuff here,,,!! like pat catgirl- i mean um yeah... for now u can only see others's posts :c
Cracked 4 Data Engineering Interviews and got a 200% Hike π
I will start a podcast series of people who were able to crack interviews or get jobs because of my content and courses.
If my content or courses helped you in any way to crack interviews then reach out to me at info@datavidhya.com
I'd love to host a podcast and share your story with the world, let's inspire people and help them
228 - 2
π’ FREE! Fundamentals of Data Engineering Masterclass - Exact 3-Hour Video is LIVE!
Topics will be covered:
- What is Data Engineering?
- Data Engineering Lifecycle
- Data Generation & Storage
- DBMS System
- Data Modelling
- NoSQL Databases
- SQL vs NoSQL
- Data Storage Processing - OLAP vs OLTP
- Extract Transform Load
- Data Undercurrents
- Data Architecture Complete Guide
- Data Warehouse
- Dimension Modelling
- Slowly Changing Dimensions
- Data Marts
- Data Lake
- Data Lake vs Data Warehouse
- Big Data Landscape
- Data Engineering on Cloud
- AWS Data Services
- Real-World Case Study Architecture on AWS
- GCP Data Services
- Real-World Case Study Architecture on GCP
- Azure Data Services
- Real-world case Study Architecture on Azure
- Modern Data Stack
- Important Tools for Data Engineering
- Python and SQL for Data Engineering
- Understand Data Warehouse Tools like Snowflake, BigQuery
- Understand Apache Spark with Databricks
- Understand Apache Airflow
- Understand Apache Kafka
And many more
88 - 3
Going to upload Fundamentals of Data Engineering Masterclass 3 Hour Video TOMORROW!
301 - 20
Fundamentals of Data Engineering Masterclass
Here are topics I am planning to include, suggest something that you want to learn!
Note: These are just fundamental concepts, for hands-on we will have different video
This is going to be one shot video on YouTube
Topics will be covered:
- What is Data Engineering?
- Data Engineering Lifecycle
- Data Generation & Storage
- DBMS System
- Data Modelling
- NoSQL Databases
- SQL vs NoSQL
- Data Storage Processing - OLAP vs OLTP
- Extract Transform Load
- Data Undercurrents
- Data Architecture Complete Guide
- Data Warehouse
- Dimension Modelling
- Slowly Changing Dimensions
- Data Marts
- Data Lake
- Data Lake vs Data Warehouse
- Big Data Landscape
- Data Engineering on Cloud
- AWS Data Services
- Real-World Case Study Architecture on AWS
- GCP Data Services
- Real-World Case Study Architecture on GCP
- Azure Data Services
- Real-world case Study Architecture on Azure
- Modern Data Stack
- Important Tools for Data Engineering
- Python and SQL for Data Engineering
- Understand Data Warehouse Tools like Snowflake, BigQuery
- Understand Apache Spark with Databricks
- Understand Apache Airflow
- Understand Apache Kafka
What more topics do you suggest? Let me know in the comments ππ»
327 - 51
NEXT VIDEO
54 - 31
When you purchase any product from Amazon you need every notification on a real-time basis, you order some food, you want to track where the driver has reached
Real-time data Streaming is everywhere, these days you want everything in real time!
But we didnβt start with this :)
The Internet started booming in the early 2000s, A lot of new people started getting access to the Internet.
The data volume and the speed at which the data was getting generated were low but due to the rise of social media and more and more people getting access to the internet.
Data started growing at a rapid pace and it was not just about the size of the data but the speed at which it was getting generated.
The business wanted to use these data to understand their customer, provide better services
We used batch processing for this, they used to get chunks of data and process it in batches like once every day or weeks
This is good for analytics but what if there are cases where you need to take action immediately as the event happens?
For example, if there is a fraud transaction from your credit card, you need a real-time system that processes data right away so that you can get a notification about it
There was a need for real-time data processing and at that time we had a system that made it possible
Companies used message brokers like RabbitMQ and traditional databases. These tools worked well for small-scale applications, but as the amount of data grew, they struggled to keep up with the real-time demands of modern, large-scale systems.
Latency and bottlenecks were constant challenges. The team spent countless hours optimizing and scaling these systems, but there were many limitations.
The industry needed something more robust, scalable, and capable of handling streaming data.
Enters Apache Kafka
60 - 5
We have more than 8+ FREE End-To-End Data Engineering Projects Available on this channel
Kick-start your career in Data Engineering with these projects, you will learn more than any paid courses for FREE!
Spend your weekend doing these amazing projects ππ»
1. IPL Data Analysis (End-To-End Apache Spark Databricks Project)-
https://youtu.be/0iNJPKheQqM
What will you learn?
β
Python and PySpark
β
SQL
β
Apache Spark Basics and Databricks
β
Writing transformation logic
β
Visualizing data for insights
2. YouTube Data Analysis (End-To-End Data Engineering Project) - https://youtu.be/yZKJFKu49Dk?si=hMsSB...
What will you learn?
β
Python and PySpark
β
SQL
β
How to understand the business problem
β
AWS Services - Athena, Glue, Redshift, S3, IAM
β
Building Data Pipeline and Scheduling it
3. Twitter Data Pipeline using Airflow - https://youtu.be/q8q3OFFfY6c?si=FFKMI...
What will you learn?
β
Python
β
Basics of Airflow
β
Working with Twitter Data and Package - Tweepy
β
Python Package - Pandas
β
Writing ETL job and storing data on S3
4. Stock Market Real-Time Data Analysis using Kafka, AWS, and Python - https://youtu.be/KerNf0NANMo?si=u4dYZ...
What will you learn?
β
Build a Real-Time app using Python
β
Understand the basics of Kafka
β
Install Kafka on EC2
β
Generate a real-time pipeline and
β
Analyze Data in Real-Time
5. Uber Data Analytics Project On GCP
Video Link -
https://youtu.be/WpQECq5Hx9g?si=TIFz_...
Here's what you will learn:
β
How to understand raw data
β
Building Data Model (Lucid Chart)
β
Writing ETL Script (Python)
β
Modern Data Pipeline Tool (mage)
β
SQL queries for analysis
6. Olympic Data Analytics | End-To-End Azure Data Engineering Project
Video Link - https://youtu.be/IaA9YNlg5hM?si=6Ez3J...
Here's what you will learn:
β
Extract Data from APIs
β
Learn Azure Services DataBricks, DataFactory, and Synapse Analytics
β
Writing Spark Code
β
SQL queries for analysis
Have you ever done any of these projects? Let me know!
393 - 30
Shout to all channel members for supporting my work β€οΈ
β’ Venkat Rayudu
β’ BαΊ£o Phan Gia
β’ Burak Γzalp
β’ Hasan Saberedowo
β’ Prabhjot Kaur
β’ Naresh M
β’ Catalina HernΓ‘ndez
β’ Balaraj Kakumanu
β’ mallikharjuna pamulapati
β’ Rahul Das
β’ Shashi Kant Chaturvedi
β’ Sneha Challa
β’ SOULFUL READER
β’ Pradeep Dhanawade
β’ Gustavo Navarro Lema
β’ Suhas Chadar
β’ hongbo sun
β’ Ahmet Emin SarΔ±caoΔlu
β’ Kumaraguru Mani
β’ JAY ITANKAR
β’ deepak arora
β’ Aman AnalytiX
β’ Kolawole Samuel Aina
β’ SKR Talks
I have planned some good vidoes and projects for YouTube, will provide update soon!
66 - 0
Understand Airflow For Data Engineering (Quick Guideπ)
π What is Apache Airflow?
As a Data Engineer, one of the tasks you perform is to build a data pipeline now for that you can write simple Python scripts or use enterprise tools.
Simple Python script with cron job is enough for a few pipelines but what if you have 100s of them? Hard to manage!!!
This is where Airflow Comes into the picture
π‘ It's an open-source tool for managing data pipelines, you can build, schedule, and monitor workflows.
In one place you can manage everything!
There are a few components you need to understand.
π DAG (Directed Acyclic Graph):
At the heart of Airflow is the DAG, which defines a collection of tasks and their dependencies in a specific order.
This is a core computer science concept.
Think of it as a blueprint of your workflow, ensuring that tasks run in the sequence.
ππ» Directed: Tasks move in a certain direction.
ππ» Acyclic: No loops! Tasks don't run in circles.
ππ» Graph: A visual representation of the tasks.
π What is a Task?
It is basically where you write your logic, such as reading data, transforming it, or writing it. Each task runs independently, in its own process.
To create a task we need to use Operators.
π What are Operators?
There are many different operators you can use for a specific task.
They determine WHAT gets done.
ππ» BashOperator: Executes a bash command.
ππ» PythonOperator: Executes a Python function.
ππ» PostgresOperator: Executes SQL on a Postgres database.
and many more!
π Executor:
Determines HOW tasks are run.
There are several types:
ππ» SequentialExecutor: Runs tasks sequentially.
ππ» LocalExecutor: Runs tasks in parallel on a single machine.
ππ» CeleryExecutor: Distributes tasks across multiple machines.
π Scheduler: The brain behind when your tasks run. It checks the DAGs and sends them to the executor to see if they have tasks to run.
π Web Server: A friendly UI to monitor and manage your DAGs. You can check task logs, rerun tasks, and visualize task dependencies.
Let me know if you found this helpful ππ»
You can watch an overview of Airflow here - https://youtu.be/5peQThvQmQk
161 - 8
The data engineering market will cross the $100 billion mark by 2028 ππ€©
This is the best time to get started in data engineering!
Keep reading and you'll know why ππ»
According to the Analytics India Magazine Research report from last year, the data engineering market was expected to surge at a Compound Annual Growth Rate (CAGR) of 33.8% over the next five years, increasing from USD 29.1 billion in 2023 to an estimated USD 124.7 billion by 2028
One of the reasons is Generetive AI or LLM, we need to covert raw data that can provide actionable insights
The report presented many different areas of Data Engineering
β
Number of jobs available
β
What is the median salary earned by data engineers (based on company & industry types)
β
Attrition rate for data engineers
β
Skills required based on years of experience
and many more...
As more and more people get internet access, a significantly large volume of data is being generated. The data is generated through sensors, transactions, social media, and others.
Data engineers play a vital role in managing the data and ensuring its quality and integrity.
Watch this video and you'll understand everything!
24 - 0
Freelance Data Engineer and Solution Architect.
I love learning about technology and teaching thing I know to other people. The goal of creating this channel was to teach everything I know to the world, I have spent most of my time learning things online and I want to share all my learning on this youtube channel.
I will be posting content related to data engineering, architecting systems, data science, machine learning, and interview preparation.
Happy Learning :)