Skip to main content
Loading Explore guides…
SQL Full Course Tutorial · Acedra
Home
Explore
Computer Science
SQL Full Course Tutorial
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
1,832 segments
Copy
Hide video
0:00
Data nerds, welcome to this full course tutorial on SQL. This is the video I wish I would have had when I first got into data engineering. To master this tool, we'll break down concepts into short five to 10 minute topics. You'll work right alongside me completing real world exercises to learn not only SQL, but also other in-demand skills like terminal Git and GitHub. After each short lesson, you'll have the option to tackle interview level practice problems that prep you
0:24
for the job and reinforce your learnings. And by the end of the course, you'll have built two fully customizable portfolio projects that showcase your skills in building and analyzing a production level data warehouse pipeline. Now, SQL is by far the most popular tool for data engineers. If we look at recent job postings, we can see that it's in two out of every three postings for data engineers. And it only increases demand for those senior roles,
0:51
making it even more of a vital skill to learn first. Now, SQL or SQL is the language used to communicate between you and a database. It's by far my most used tool as a data nerd, starting with my first job working for a global fortune 500 company all the way to my most recent job working with Mr. Beast. Yeah, even Jimmy uses it. More recently, I've been using it in my data pipeline for my app data nerd tech, but more on that in a bit. So over the years, I've cataloged
1:18
everything that's essential in using SQL and bundled it all up into this course. Now, to be clear, I've made this for beginners. There's no coding terminal or even engineering experience required. I'll be teaching you everything you need to know step by step for this. We're going to break up the course into two halves with the first half focusing on the fundamentals you need to learn to feel confident writing SQL in a real data engineering workflow. We'll start by getting
1:45
your environment set up and making sure you can run SQL both locally and in the cloud. Then we'll introduce core query building blocks such as filtering grouping aggregation and sorting data. From there, we'll shift into understanding how data should be structured in warehouses with dimensional modeling. And throughout the sections, you'll use the same tools real teams use of VS code, the terminal and get in GitHub. At the end of this first half, you'll build a
2:08
portfolio ready project where you explore a real data warehouse. In it, you'll write SQL scripts to answer stakeholder style questions and your analysis will be published to GitHub. And this will ultimately help you start thinking like the stakeholders will be building data pipelines for. Now, in the second half of the course, we'll shift into writing SQL the way data engineers use it in production. We'll start with how they build and evolve warehouse tables using create, alter, and drop.
2:35
We'll then cover how pipelines load and maintain data with insert, update, and delete. From there, we'll focus on data modeling techniques, especially star schema design of fact and dimension tables. So your warehouse is built for analytics and downstream use. And finally, you'll level up your SQL with CTE and sub queries for clean modular scripts, plus window functions for real warehouse analytics and pipeline logic. At the end of this half, you'll build a production style end-to-end
3:00
SQL pipeline, designing the tables with a correct schema, and then loading the raw data into it, and then producing data marks for specialized use cases that can be accessed by a multitude of tools and prove your job ready data engineering skills. Now, I believe in open sourcing education. So this entire course is completely free. You'll have access here on YouTube to every video lesson and can follow along with exactly what I do. And everything we're using to run SQL is completely
3:27
free and open source. Oh, and I've even made both final projects publicly available so you can compare your work to it. Now, for those that want to support the course, I have some perks that add structure, practice, and certification. First, you'll get access to your own course dashboard that tracks your progress. In here, it lets you watch each lesson individually in short bite-sized segments. Then after each lesson, you'll have access to additional SQL interview level practice
3:54
problems that provide a detailed solution. However, you get stuck, you can leave a comment below and get help from other students. On top of this, I'm giving you all my course notes that walk through the video lessons step by step, and you'll get access to my custom cheat sheets breaking down key concepts. When you complete the course, you'll receive a certificate of completion that you can post directly to LinkedIn. So if you want to support and bring to life more courses like this,
4:20
use the link in my description to get access to the supporter resources. Now, I have two SQL courses, SQL for Data Analytics, my first SQL course. It's four hours long and ends with an analytics focus project. And then this course, it's 14 hours long, and it has a similar analytics project, but then an engineering focus project as well. So which one should you take? The simple answer is data analysts should take the first course and data engineers should take this one.
4:43
Now, neither course is a prerequisite for the other. Both courses teach the same SQL language, but they implement it differently. SQL for Data Analytics focuses on querying existing data to answer business questions. And then this course starts similarly, but it ultimately focuses on building production level pipelines, framing this all from an engineering perspective. If you've taken SQL for Data Analytics and want to take this one, go for it. You'll still benefit early
5:08
sections may feel familiar. So skim ahead if those concepts are already solid. Now, this course was a behemoth to me. So for 14 hours long, it took me the greater part of last year to make. And it took a team of four to put it all together. So I'd be remiss if I didn't give a shout out to these teammates. For this, we're going to do it the office style. Ricky, my content developer, did a lot of work behind the scenes putting together those SQL interview practice problems, ran in my
5:36
video editor help condense down about a hundred hours of video footage into 14 hours. And finally, Kelly was the producer of the course, helping out with both lesson and interview practice problems. So shout out to my team for helping me bring this together and make this course possible. All right, one last thing before we jump into the course, if you get stuck along the way, I highly recommend you use any popular chatbot as they're more than proficient in SQL and a
6:00
heck of a lot faster than waiting on me to reply. Anyway, enough of me, Evan, let's get into this course diving into our first concept to understand what SQL actually is. All right, in this lesson, we're going to get you up to speed in understanding everything you need to know behind this powerful tool for data engineers. To understand this, we need to grasp two major concepts. First, where do you write and execute your SQL queries? And secondly, your database, where your data's
6:29
actually stored. So let's dive into that first. A database is a collection of data. And these things were made to hold a massive amount of data. An Excel file spreadsheet can only hold around a million rows, but a database can easily hold way more than that nine quintillion rows. Now about those databases, there are two main types of relational and non-relational. We'll explore each type more in a bit. But first, let's go back to where we're actually running SQL.
6:57
When you write this code to request data from a database, that's called a query. And SQL can do a whole heck of a lot. A common acronym associated with this is CRUD. And it stands for create, read, update, and delete. With SQL keywords like create and insert into, you can add new records. Then with keywords like select, you can retrieve specific data you want. Next, with keywords like update, you can modify existing records. And then finally, the scariest of all, you need to remove
7:24
records, you can delete it. Now these commands fall under three types of SQL commands. DQL or data query language is used for the reading aspect of CRUD. And is what we'll focus on on the first portion of the course. Next is DML or data manipulation. And this is used for creating, updating, and deleting new data, specifically in tables that already exist. Then we have DDL, which is data definition language. And this deals with creating or deleting the schemas and
7:52
tables we need from scratch. So understanding this, we can now understand how SQL actually works. From your computer, you write a SQL query and send it to a database. The database then performs your CRUD operation and then provides you a SQL output, depending on what you did to your computer. But this now begs the question, where are databases even stored? Well, there are two main options. You can either run it locally or in the cloud. Local means the database runs directly on your
8:19
computer. This is ideal for data engineers to iterate quickly without affecting production databases. However, others can't access it. Now production databases are usually stored in the cloud. This enables your entire organization to connect to the data source you built. You probably heard of popular cloud providers like Google, Amazon, or Azure. Anyway, we're going to be using a combo of both local and cloud for this course. And in the final project, you'll be building out a
8:45
data warehouse and data mart locally, and then push it to the cloud to make it publicly available. So now that we understand that databases can be stored on your computer or in the cloud, what are the different databases we can store in these locations? Well, two main types of relational and non-relational relational databases store data in rows and columns, very much popular in analytical applications. Say of a database, a job posting that companies go to daily to update
9:12
on what jobs are available. You could have one table to keep track of all the different jobs, and then you have an associated table to keep track of all the different companies. There would then be things like a company ID within the two in order to relate the different tables. Hence, relational databases. Now, this course will primarily focus on using SQL to interact with relational databases. But it's important to know the other type of database of non-relational
9:39
databases, commonly known as NoSQL. But NoSQL doesn't stand forward, not SQL. It stands for not only SQL. These supports storing your data in a variety of different forms, including key value paired, column based, graph based, and even document based. Alright, so we understand the types of databases now, because of that, we can get into narrowing down what database we're actually going to use for this course. Well, I needed to meet a few criteria. First of all,
10:06
it needed to be a relational database that was also highly popular. So it has community for support. From there, I started to narrow it down to a database focused on analytics to make it support building data warehouses. After this, I narrowed it down to free and open source options so that anybody taking this course can use the database. And finally, with that, I needed this database also to run locally first, along with the cloud. And so this gave us DuckDB. DuckDB is a blazingly
10:34
fast database that satisfies all of our needs. So we're going to be using it for the entirety of this course. We're not only going to get you set up with running it locally on your own computer, but we'll also run DuckDB in the cloud by using mother duck. This is a managed cloud platform that will be completely free to use as well. Now this brings us back to the last concept we need to cover of where we're going to be writing and running these queries. Well, that's commonly
11:00
done in an editor, whether that's database provided, cloud provided, or just a standalone code editor. Now database provided ones provide you with an application to use on your computer to access their local database or cloud based one. This is a popular option from Postgres called PG Admin, and inside it even allows you to write SQL queries. For data engineering work, I'm not a fan of these. Next up are cloud provided editors. Typically, you access them in a web browser and they allow you
11:27
to interact with your database. We're going to be using mother ducks cloud provided editor in order to connect to our needed databases for the course. We'll actually do this for the first half because it's super easy to get up and running for you. Last up are code editors. And this is what we'll focus on for the majority of the course. We're going to be using a popular option of VS code along with the terminal, which allows you to build all your SQL files locally, but then interact with
11:53
databases that are either local or in the cloud. You'll be running SQL queries like a pro data engineer after this. Alright, so now we understand what SQL is and what tools we're using for the course. But it now begs the question, what data we're going to be using and what pipeline are we going to be building for this course? Well, covering that next. Welcome to this lesson. We're going to be diving into understanding what data and also what pipeline we're going to be building
12:23
for this course. But before we talk about any of that, we need to understand what problem we're trying to solve. And it's a real problem that I had. This all started a few years ago when my subscribers were asking what are the top skills they should be learning in data. And the best way to answer this analytical question is to look at where these skills are demanded in job postings. So I built out a data pipeline that solves this. It pulls online job postings from a source and
12:49
then using SQL cleans up and transforms this data. Finally, it then loads it into a data warehouse. Now I'll go into this data source and how I built the pipeline in a bit. But first, what's a data warehouse? Well, in order to qualify, it needs to be subject oriented, non volatile, integrated, and time variant. If it doesn't have these four things, it's just a database and not a data warehouse.
13:12
This is all from the father of the data warehouse, Bill Inman, and we'll be going over this definition in much more detail as we build out our projects. Anyway, for our final project, we're going to be replicating what I did by taking raw job postings and building out a data warehouse and also data marks all with SQL. So what does this data look like inside of our data warehouse? Well, here is an ERD or entity relationship diagram, and it shows the different tables in here.
13:37
We have four major tables, one fact table containing all our different job postings, then a dimension table around the different companies that are posting jobs, and finally, dimension tables that contain key information about the skills required for these jobs. Now, if you haven't heard about fact tables, they contain the core data for our analysis. They measure and record actual events. So in our case, the different job postings,
14:01
the dimension tables describe attributes or dimensions of the data. In our case, skills and company information, and this is really important to supporting filtering and grouping different sets of data. Anyway, that's all theoretical. Let's actually dive into my machine and look at the key attributes from the data. So querying this data warehouse, we can see that we have 10 different jobs available to look at job postings for data analysts,
14:25
data engineers, and data scientists are by far the most popular, but there are other options available. And with that, these postings come from around the world in over 160 different countries. So the country you live in probably is in this data set, and this now begs the final question of what platforms did we get all these job postings from? Well, a ton of popular options like LinkedIn, indeed, and even Zippercruiter. So you're probably wondering, where the heck did
14:52
this data come from? Honestly, it wasn't easy. Back in 2022, I wanted real data on what skills actually get you hired and data jobs to provide to my subscribers. Not opinions, not blog posts, but real job postings. So I did what any data nerd would do. I tried to scrape LinkedIn. So over the course of a few days, I built a scraper, learned just enough front end to make it work, stayed logged in with my LinkedIn account. And surprisingly, it worked. Well, for about a day.
15:20
Then the captures started to hit. Then my account got banned, and it became obvious real fast. Well, job posting data is extremely valuable. And these sites protected aggressively. And this is a very real data engineering problem. Scraping at scale quickly turns into an infrastructure nightmare. So at this point, I thought the project was dead. But then I happened to stumble across SERP API, and they had exactly what I was looking for. Instead of scraping LinkedIn directly,
15:54
SERP API gave me access to job postings at scale through search engines. So SERP API can pull these jobs from Google that ultimately come from different platforms like LinkedIn, Indeed, and ZipRecruiter. So with this, I wasn't fighting captures anymore, managing accounts, or building infrastructure. On a whim, I reached out to SERP I explained my problem and asked if they could support the project. You've got mail. Surprisingly, the CEO himself replied,
16:21
and they agreed to support the project. So I immediately started by jumping into their platform and building out the code in their playground area, which lets you test out different queries and see the results clicking search. I can see the search results that it gets back in this browser view. But what's actually more useful are the structured results that I get back. For example, with this first one, I can see it's a senior data engineer role in Washington DC scraped from
16:48
LinkedIn. It tells me that it's not only full time role, but also in the job description, I can see it list a host of different skills. This right here is the jackpot. Oh, and some jobs even have more information like this one with salary. Honestly, this is way more detail than I could ever have gotten by scraping on my own. From there, I can grab a code template for any popular language or tool such as curl, Python, or Google sheets, and I can run it locally or in the cloud
17:14
wherever I need it. Now, to be clear, Sir PPI has way more than just job postings. They support over 50 search engines, everything from things like Google flights to Google maps to product listings on sites like Amazon. And oh, they even have an ability to look at YouTube. If I wanted to, I could scrape all of my YouTube data and get it back in only a few seconds. Oh, and if you need responses and under a second, they also offer add-ons for that. Now back to my job data. I've
17:43
been systematically pulling these job postings on a daily basis, which is how I've collected almost five million jobs over the past two years using Sir PPI. All the data can be accessed via my site data nerd tech. And it's solved my problem of figuring out what skills data nerds actually need to get hired. It's how I figured out that SQL was the most popular skill of data engineers. And all of this job posting data that I've collected of the years will be using throughout this course
18:12
in order to learn SQL and ultimately build a SQL based data pipeline. Now to wrap up, I want to give a huge shout out to Sir PPI for sponsoring this video. Use my link in the description and tell them Luke sent you to get 20% off. They even have a free tier with 250 searches per month to try it out. Now enough of me happen. Let's get into setting up SQL on our computer. So we start playing around with it. All right, see you in the next one. Welcome to this
18:42
lesson where we're going to be not only going through setting you up with the environment in order to execute SQL, but also get you set up with the actual database that we're going to be using for the entirety of this course. And jumping into my computer to show you this, we're going to set you up in the browser in order to be able to execute SQL queries, you're going to have the results right below here. And you're going to have this environment here that we're working on in the
19:09
first portion of the course in order to investigate and learn some different SQL syntax as we go through this. Now I'm going to get you all set up to be able to execute SQL in your browser and with the data set. Before that, we need to understand some concepts of actually where we are running this SQL query. Well, as you can recall from the last lesson, we're going to be using the free open source option of DuckDB for this course. And this is a very lightweight database that you can
19:37
run right inside of your computer. Now DuckDB is relatively easy to get set up and installed on your computer, but it requires one pretty big caveat. If I go into the installation guidelines, it tells you how to install it. And specifically, it has you installing it with a terminal. And I'm assuming for a lot of you taking this course, you don't have much familiar RRT with running commands in the terminal. And there's quite a few to get set up with not only DuckDB, but also the
20:08
data set. So because of that, I made an executive decision when making this course that we're going to start out working in this web browser environment that has DuckDB on the back end. And this is called Mother Duck. Now Mother Duck, which you can visit via this link here, is the commercial SaaS or software as a service platform that supports running DuckDB in the cloud or, if you will, in your browser. DuckDB or DuckDB Labs owns a significant equity stake in Mother Duck. So,
20:43
it's very beneficial for both of them to work together and collaborate. Mainly, they keep it separate so DuckDB team can focus on improving the database and keeping this free and open source. And Mother Duck is the commercialized version that actually makes money to keep DuckDB Labs funded. And probably the most important part that is relevant to you, Mother Duck allows us to host DuckDB databases on the cloud, basically have data warehousing capabilities. And then you can use
21:13
other popular services in order to connect to these DuckDB databases inside of Mother Duck. For me coming from a data analysis background, this is pretty imperative because tools like Tableau and Power BI can connect to a single source of truth inside of Mother Duck. And there's a host of other data engineering tools that we'll eventually get into later on in the course that can connect to it as well. So to recap, DuckDB is a free open source software that you can run
21:40
either locally or on another server. Whereas Mother Duck is the platform that allows you to host these databases in the cloud, basically host data warehouses. And honestly, I wouldn't have known about either of these service. If it wasn't for Medi, you can check them out on the YouTube channel here. He heard that I was building this course on SQL for data engineering, reached out and connected me not only with the Mother Duck, but also the DuckDB team
22:06
in order to put this course together and make sure it was successful. So you'd shout out to him. Now let's get into setting up your Mother Duck account. And for this, we're going to be on the free tier. We're going to remain on the free tier for the entirety of the course. And the good thing about Mother Duck is there's no credit card required. All you got to do is just sign up for those complaining that you got to sign up for a service order to take this course.
22:32
As a dad engineer, you're going to be signing up for a lot of services. And a lot of them don't offer free tiers. So consider yourself lucky with this one and get used to it. For this, you're going to navigate to the URL on screen of loopb.co/motherduck. Now the Mother Duck team has built this special login for us and the course takers of this. So that way we can skip the normal marketing emails that you get if you logged in through motherduck.com. Anyway, you'll sign up by either
23:00
putting your email below, or I recommend using something like GitHub, or like I'm going to do Google. From there, go through the authentication process with Google and your associate Google account. From there, insert your name organization. I just put Luke's organization. For me, I'm located in North America. That's where I want the region to execute my different queries. And then from there, click continue. From there, you may be asked a few questions about your background.
23:23
Feel free to just fill it out or skip where applicable. And then from there, it has a let's get started. I'm going to go ahead and just click all of these and say that we did them because we're going to go through them in this. So now we're inside of the environment where we're going to be running SQL queries. Take note of this URL that where we're going to be at for executing this, so you can come back to it easily. Let's get into understand the UI or user interface
23:50
of this mother duck experience. Now it's broken into three major parts. Over here on the left is what you'll hear me call the left sidebar, the sidebar in the middle right here is our SQL notebook. And we can actually add different notebooks as well in that sidebar. But that's where we're going to be executing those SQL cells, which we'll get into a little bit. And then we also have on the right hand side, something called an inspector, which we can go into inspect data sets.
24:20
I'm going to leave that open for the time being so that we can get into that. We'll go back to that mother duck notebook right here. Let's dive into that left sidebar a little bit further. They have capabilities do search. So if you have a certain term, you can search for it. We can also create tables from files. Then in that notebook section, I can either hide or show it. We just created this untitled notebook. I'm actually going to rename this notebook. And we'll just name it after
24:42
what this section is specifically, we'll call this SQL and database setup. And then I'll click rename. Feel free as we go through these different lessons to create different notebooks to keep track of what we're doing underneath these notebooks are our attached databases. And right now we only have two. We have a my database and nothing inside of it. And then we have some sample data sets. And we'll explore one of these to explore a notebook here in a second. But I want to call out this
25:09
down at the bottom, you may see something this of, Hey, your trial ends in 21 days. Don't worry about that. You will stay on that free plan. And the course will or this platform will be continue to be free to the entirety of this course. You're not going to use enough resources to need to upgrade. So don't worry about that. So I'm just gonna minimize this. Before we dive into those databases, it's important that we understand how hierarchy works in a database, specifically for
25:35
tables and different data we have inside of it. So for a database, you can have one or multiple different schemas. We're just simplifying it here, but only allowed to have certain tables within that schema. So schema a in this case could have table one, two, and three, such as users, orders, and products, and then table or of schema B could have something like employees and products in a separate
25:59
one, you can keep it separate from each other. But in the same database, we'll apply this to our data set that we're going to be using for this course in a bit. But I wanted to go over that because if we look inside of here, we have something like my DB, and it has the schema of main. Similarly, underneath the database of sample data, it has 123456 different steam schemas. And then if I go to something like the stack overflow survey, it has two different tables inside of that schema. So
26:29
what I'm gonna do is I'm gonna go to that survey results of the stack overlay survey. And what I can do is click these three dots right here. And it allows me to query in the curtain notebook. I'm in the welcome to mother doc. That's fine. I'm just gonna go ahead and do it. And it puts it down here at the bottom. Now we're gonna get into actually understanding the SQL dialect here in the next lesson. But I want to show you that you don't even know, know any SQL in this program, and that you
26:54
can just query it using the actual UI. And it puts in all the relative information, all these different column names for this data set, a lot of columns in this one. And then down at the bottom, it provides a table view of this. And I can go through and see all the different values in it. And also scroll down because it returned, I can see from up here, 100 rows returned, and 374 milliseconds, Dr. B is
27:20
super fast. If I wanted to, I could copy the results to my clipboard, or even download the results to some different file formats. Now what's cool about this UI that I really like, is that over on the right hand side in this inspector window, I can go through and view all the different columns right here. And it provides some EDA or exploratory data analysis for me to actually dive into it. So looking at something like the country column, it tells me that there are 28 distinct values.
27:52
And it shows a bar chart with this, how it relates to the other columns and how many distinct values it has. And we can scroll through and see any different columns of one. This, as like I said, this is a lot of columns in it. So this is great for someone that wants to actually look through a lot of different columns. Now besides viewing columns right here on the inspector, you can also view it down below here. If I just click survey results right here, it will appear right underneath
28:17
it. And once again, I can actually in this case, we can see that there's a difference here, right? This one, because we only queried 100 rows, 100 of all the different rows that they have, it only showed 28 columns. But over here on the left, because this one actually queried all 615,000 rows, it has different results. And it says we have 264 different countries. So make sure you understand what we're actually querying whenever you're viewing this, neither one of the windows.
28:49
Anyway, I can also go into and dive into seeing a deeper dive on any one of these as well by just double clicking on it. So if you notice, when we're going through that, none of that jobs data that we're going to be using for the course is located in here, we actually need to attach that database. So we you have access to it. So what we're going to do is we're going to navigate over to this notebook
29:13
that we're going to work in for the remainder of this lesson. That's on that 1.1 SQL and database setup that we created inside of here, inside the notebook, we're going to add a cell. Now we're going to run an attach command in order to attach a database. Specifically, we have to specify this long line in there. So I'm not going to expect you to actually type that in. I'll include it in the description below for you to just copy and paste into here. And this is actually specifying
29:43
that data jobs database with a unique identifier. From there, we're going to run our first cell by pressing either this button up here, this play icon, or we can press in my case on a Mac, on press command enter. Now we can see up in the left hand corner, is my data jobs database. It has only one scheme in it of main and then four different tables. Company down, job postings fact, skills down and then skills job down. So in our case, this is what the database hierarchy looks like.
30:15
We have our schema of main and then those four tables underneath that schema. This is all within the data jobs database. Now going back to this diagram, you saw earlier in the course, this is how all the tables are related. We'll have a separate lesson on how to actually read this diagram, but the main thing to understand here is that this job postings fact table is our main fact table. And then these other three tables on skills and company provide skill and company information
30:50
and they are a dimensional table. So they provide extra attributes, but the core of the information's in our fact table. So for the next few lessons, we're gonna be spending the majority of time in that job postings fact table. Once we get some basic SQL down, we'll get into how these actually relate, how to actually join them, what the heck this PK and FK actually mean. So let's actually inspect this data. So I'm going into job postings fact, selecting these ellipses here
31:16
and then go into query in current notebook. Now some other things to note with this, right? We have up in the right-hand corner, it tells us which database we are connected to. That first one up here was connected to my DB, but that's okay. It's an attached database. It doesn't really matter which database we were to attach it 'cause it wasn't there yet. Anyway, that's really important to understand which one we actually have right here to actually execute the query
31:39
'cause it may prevent you from executing it. Anyway, from there, it also provides underneath it those 100 different results and the different information. What I want you to do, 'cause there's no practice problems for this lesson, what I want you to do from here is just spend some time inspecting this table, understanding the data set itself. Also feel free to use the table explorer in there to actually dive into and see these different columns and explore them further.
32:08
Specifically, you could do something like dive into what's this job title short column. You could see all the different job titles available and what amount they actually take up for the data set and how many rows. So that wraps up this lesson. You should have signed up for mother duck, got into the UI and attached that data set. In the next lesson, we're gonna be diving deeper into understanding the basics of SQL syntax. So you can actually understand these SQL queries
32:33
that we just ran here. With that, see you in the next one. Going to this lesson on basic keywords. By the end of this, you should be able to, well, understand the basics, but be able to read basically any SQL statement from here on out because a lot of the keywords that we're gonna be covering in this are used time and time again in SQL queries. Real quick plug before we get into it and that is to my course notes that you get in the supporter resources. Yeah, I'm first of all ready.
33:09
They actually go through and describe everything that we're gonna be doing in an associated lesson like this one and then break down the lesson step by step with even more detail. Anyway, I digress, let's get into it. And the first keyword or keywords we're gonna be covering are select and from. And this is how you actually read this. Select identifies the columns or data from a database and then the from statement or from keyword identifies the tables we are connecting to.
33:43
Now, the asterisk in this case is a special command that says, hey, we wanna select all the columns, everything. So in the case of our job postings fact table, we could do select star to select all of this from job postings fact and showing this visually. We guys do this to get the results. Let's actually do it for real. So inside of mother duck, I'm gonna go ahead and create a new notebook. I'll call this 1.2 basic keywords. I'm starting a new cell and conveniently it automatically
34:12
selects our last database that we're using of data jobs. We're gonna type in select and then star from job postings fact. From there, I can press play or use these different shortcut commands to run the cell. If you're on a Mac, you can press Command Enter on Windows. I think it's Control Enter. And this outputs all the results down below it. And it even says that it returns the first 50,000 rows in nine to six milliseconds. Now, this is not all the rows of data.
34:44
Thankfully, it caps it off to sort of limit the data. But if I scroll over here on top of job posting facts, I can see that our data set has over 1.62 million rows. One quick note in the syntax, you notice I use uppercase letters here and lowercase letters here. That's pretty standard. But if you wanted to for the keywords itself, you could use actually lowercase letters. And in this case, it's gonna still return the same result. So I can also change from press Command Enter
35:11
and bam, same results. I could also change the table name itself to be all caps and still get a run. But specifically for table names and stuff, it's very important that you stick with that syntax of the lowercase because many databases require it to be what it is, the same syntax. And for keywords, it's just best practice to use these as all caps. It also makes it a lot easier to read. Now, of our job posting table, let's say I wanna select specific columns,
35:37
such as job ID and job location. Well, instead of what I can do instead of using select star, I can list out the columns or column names and then still live that from of what the table is. So back inside of Mother Duck, I'm gonna put in that job ID. I'm gonna just run this real quick. You can select one column. If you wanna select multiple different columns, you would need to enter in a comma and then from there add in the next one that you wanna use of, let's say, job location.
36:05
And you can see we have some hints coming up as we type something. If I know, so I do know I want job location, I'm just gonna press tab and it's gonna accept that. Anyway, gonna go ahead and here and press Command Enter. And now we're getting that job ID and job location. Now, I like this a little bit more readable. So I'm actually gonna enter job ID and job location onto the next line. And then from there, I'm going to press tab to indent each one of those over.
36:32
I can also do that with the from statement here and press tab as well. Now, this does not affect the execution of the query. I can still run this, it provides the same results. I can also do multiple different indentations on different lines, just like this. And when I press play, it's still gonna run. It doesn't really matter. This is all just for my readability and also if I make a good query, it's for my coworkers readability.
36:55
Now, it's very resource-intensive to the computer itself if we were to continuously always pull all the columns from a data set. So in this case, if we wanted to just pull the top three rows, we'd need to use a keyword to limit it to that. Well, conveniently, we can use limit and limit goes at the very end of your SQL query. And you specify the number of rows that you want from this. In this case, return those three rows. So inside of our query, I'm gonna go in and put
37:28
at the very end and put a limit of, we'll just do 10, press command enter and run this and only get 10 rows returned. Anytime you're building queries on large databases, you need to be using this keyword. You can always remove it at the end after you build it and verify that it's working properly. You'll save your boss a lot of money. So going back to this diagram, where we showed the database container, its schema and the tables and the hierarchy for it,
37:52
we can specify any one of these things, assuming they're in the appropriate order. So right now we have the database and then the table. Remember, we also have something like the schema itself so you could specify the schema in here as well. When we run it, run just the same. Update the other portion of the query for this. It works just fine. Now in the more advanced lessons, we'll get into when you need to put this nomenclature
38:19
inside of here, maybe whenever we're joining different tables. But for now, I just want you to that in the back of your head. Next up, let's say we have the scenario where we wanna look at distinct values or unique values inside of a column. Take for example, our job title short column. In here, we have things like data analyst, data engineer, data scientist. Anyway, we got millions of rows, we wanna get the distinct values out of there.
38:44
Well, we can use the distinct keyword right after select and it specifies, hey, we want distinct values from job title short. So for this, I'm going to select distinct values of the job title short column. And we're gonna be getting that from that job postings fact table. Running command enter, scrolling on down, we can see we get 10 rows returned. And just so you understand the concept behind this, originally I'm actually gonna remove this distinct keyword right there.
39:15
And I'm gonna throw in the job title column press command enter. The job title column is the actual name listed on the job description and it can get pretty verbose. And so if I wanted to pull out like all data analyst or all business analyst, it could be a hassle doing this. So I created this job title short column in order to make it easy for you for analysis and building pipelines. Now with any of these queries you want to end them,
39:42
it's best practice to end them with a semicolon at the end. Running command enter, still gonna have the same results. And I can even put into something like a second query underneath this for those distinct job titles. Now whenever I run this one, it will execute both queries, at least here, but only one of them will return. And I can see that by saying, hey, 10 rows returned, but two statements ran in 3.5 seconds. When we get into building out data pipelines
40:13
towards the end of the course, the semicolons, we're gonna save your butt. Next up, let's say we want to get specifically filtered job titles. In our case, we want to get them where that you have the job title short of data engineer. Well, in our case, we can use a where keyword and we put that directly after from. And in that, we're gonna specify via an expression saying, hey, we went from the job title short column, values equal to data engineer.
40:49
And it will provide those filter result. So inside of our query, we're already selecting the job title short and the job title and we want to inspect further what are some other job description names for data engineers. I can then press enter. We'll put in that where statement and we'll say that that is for the job title short column with it equal to data engineer. Now I'm gonna go ahead and execute this and press command enter. We're gonna get an error here.
41:16
It says parser error syntax error at our near engineer. The problem as we're getting help from AI inside of mother duck is that we don't have these single quotes. You can actually just, if we want to accept and run by clicking this or pressing in my case, come in enter and it automatically updates it for this. And then inspecting it further, we can see all the different job titles for data engineer and they have a lot of different random ones.
41:44
With this syntax structure of the where statement here, it doesn't matter. I did those spaces in there for readability. This is still gonna work like this, but I like to have spaces in between. Just make it a little bit easier. Now the single quotes is not applied to everything. That's just text values. Let's say I'm gonna add in the salary year average column and go ahead and run this bad boy. And then for this, I want to filter that salary year average column
42:12
for let's say a salary value. We'll say salary year average equal to let's go with something like 100,000. Press command enter. And now scrolling in through this, we're seeing all these different jobs that come in at $100,000. In this case, we did not put any quotes around this because this is a number of numerical data. And if we were actually curious to learn what's text data, what's numerical data, we could actually go into job postings fact by double clicking on it.
42:39
And over on the side, we can see that the different values has different things. Specifically for salary year average, we can see that it has a number next to it. And also it calls out that it's double, double. So type of data type that we're gonna go into in a future lesson. But something like job title is a text character by this text symbol. We can see that it is the type var char. And we're gonna change things up a little bit and not necessarily go over a keyword in this case,
43:09
but a special marker that you may see in data sets or that you will see in data sets. I'm gonna remove this where salary your average statement from here and then press command enter to run this. And if you remember from what was returned or if you actually do it, you can see that salary your average column, it has null values. And null is a special marker where a field does not have any value in it whatsoever. It's not zero, it's not anything, it's null.
43:37
So it's very common that I want to filter for this. I'm gonna add back in that where statement that we had. And I want to filter for, in this case, I want to filter for null values. Now, you would think you would wanna do, hey, we wanna do where salary your average is equal to null. But if I go to run this, press command enter, I'm gonna get zero rows returned, this didn't work. What you actually have to use to specify that you want this null is this operator of is.
44:05
Running command enter, now we actually have the results of all the different null values here. But if you're like me, you don't really care about those null values, you actually wanna get values that are not null. So if you can use this is operator, you could also use this not operator running this. We're filtering now for salary your average that is not null. And now we have nothing but numerical values in here that we can go through.
44:33
Now, don't worry if you're getting too freaked out by these operators of this equal sign is and not, we got a whole lesson on it coming up next. Now, another concept to cover, if that's not a keyword, is commenting in code. Let's say I build this spectacular query and I wanna share it with others, but they may not understand what it is or even how to read it. I can put comment inside of the code specifically by putting these double dashes. If you're in the military, double tax.
45:03
And I can write something like this is a comment. You can even see inside of here that it's formatted differently in this italics. But if I run this, it's still executing. And this doesn't matter if it's up on the top, off to the side, or even down at the bottom right here. No matter what, it's going to work. What I really like about this too is anytime I'm troubleshooting code, I'll usually use a commenting to comment out. So I could comment out like this line right here.
45:31
Now when I run this, I'll only get the job tile short and the salary your average column. Now let's say I have a multi-line comment going on here. In this case, I have a description of what this query does, of what fields it's like, what it filters, and shows some cleaning results. Anyway, this is really annoying to have to go through and execute these comments every single time inside of here. And so what I could do instead is use a multi-line comment.
45:57
It has a forward slash and then an asterisk. And if you notice by that, everything in this block is now marked as a comment. So I need to mark when that comment actually ends by using an asterisk and then another forward slash. And that stops it. Now whenever I run this command enter, boom, I have it all below. And I don't have this all commented off and it makes it a little more readable. The last keyword that we're going to cover in this is order by.
46:26
And we can, that's shown in this example right here, we have a column of salary hour average. We could sort the entire data set or wherever we select by that column using the order by command. In this, we're going to put order by at the end of our query, but before our limit, we have a limit on there. And we can specify the column what we're going to do. And automatically it's going to do this in ascending order. So for this, I've selected the job tile short column,
46:57
job location, job via and salary or average. And we're going to be going from the job postings fact table. And we want to order by, so I use order and then buy, and then enter in that salary year average column. Running this query, we can see that the salaries, in this case, start at around 15,000 and then work their way up. So it's in that ascending order. Now conversely, we can also look at it in descending order. And we do this by at the end of the order
47:31
by specifying DESC or descending. So inside of here, I can just type in DESC. Yeah, DESC, press command enter. And it sorts it in that descending order, starting with that first and highest salary of $960,000. Now just a note, you can also specify ASC for ascending and provides the same results that we saw previously, but ASC is redundant because you should know that it automatically does it automatically, so we don't include it on there. [MUSIC PLAYING]
48:06
Now there is a certain order, or you'll get errors that you need to put these keywords in that the machine actually expects them to be in. Selecting from gets pretty easy and redundant, but when it gets to these other keywords, some of which we haven't even covered yet, such as group by and having, it takes some trial and error, but over time, you do understand what type of order this goes in. It just sort of takes trial and error. Now the order that you write commands is important.
48:35
What ends if I wanted to, in this case, take that order by, stick it right here, and then try to run it? I'm going to get this. And we get a parser error, syntax error at or near from. And even the AI auto suggestion is not actually fixing for what it needs to actually do in this case. It thinks we need to change the order by to a from. This is why it's important to understand that you need to understand that there's an order to write commands. Specifically, this order right here.
48:59
Select, then from, and where group by we haven't covered yet, or having haven't covered yet, and then order by and limit. So let's say we wanted to get the top 10 highest paying jobs for data engineer roles specifically. Well, the first keyword we know you can use is where. So we'd specify that job title short equal to data engineer. After the where, we know we can then also specify the order by. And in this case, we want to get that salary year average.
49:29
And we want it from highest to lowest. So we need to specify descending order. And then finally, we'll put in, we want the top 10 results. So we'll do limit of 10. And for good print measure, we'll put in a semicolon, press command enter. And we're not getting anything. And that's because I fat-fingered engineer. I'm not spelling it right. I need to put in. And after this, actually running this again. OK, now we're getting those top 10 jobs.
49:56
Now, we routinely switch this up and always will sometimes put the order by before the where. And you'll go to run this, and you won't get this. And it's really important that you understand the ordering because the actual syntax message isn't very helpful at all in helping you fix this. So I'm going to go ahead and just move that down below. Wanting this again, command enter. We have the results that we want. [MUSIC PLAYING] Now, throughout this course, I'm going
50:22
to refer to something called DuckDB Friendly Syntax. And basically, DuckDB is super great because it's not only fast, but they also have a lot of ways to make it a lot easier to write your code. But unfortunately, a lot of the ways that you can write your code in DuckDB, you can't actually write in other databases. So that's why I call it friendly SQL. One quick example of this is if, say, I wanted to investigate that company dim table,
50:48
and I clicked the three dots and go query in current notebook, if you dive into this query right here, you see that, in fact, well, one, they were in lower case was crazy. But two, they put the from statement up at the top. And if you remember from my order to write commands, from is clearly below select. Now, this generalization is applied so that you understand it, is applied to all databases beyond just DuckDB. But DuckDB has this friendly SQL that
51:21
allows us to actually do this and put the from before. So that's why it's a little bit different here. So from time and time again, throughout this course, you'll hear me refer to certain things as, hey, this is a friendly SQL syntax that only DuckDB allows. Because I really want you to come out of this course being able to be database agnostic and be able to work in any single database. All right, for those that supported the course, you now have inside our dashboard,
51:50
you have some problems to go through and get more familiar with operating all these different basic keywords. The last problem gets pretty spicy. So good luck with that. See you in the next one, where we get into operators.
52:07
In this lesson, we're going to be getting into operators. We're going to be using this to perform some exploratory data analysis on our data set. And we have two types of operators we're going to be using of comparison and then logical operators in order to further deepen to it. Let's get into it. So let's just briefly cover both. And first up is comparison operators. These are the foundation of data filtering in SQL, as we saw in the last lesson using the equal to operator.
52:41
Now we're going to be diving into these in this table with the exception of is null and is not null. We covered in the last lesson, and the like operator, we're going to cover in the next lesson on wildcards. The other type of operators are logical operators, and they allow you to combine multiple conditions to create more sophisticated queries. Instead of having to run multiple queries, you can combine it into one. We're going to be looking at and or a not for this one.
53:09
Now in this lesson, we're going to be using it specifically in that WHERE clause, because we covered in that previous example. And it's the easiest one, frankly, to get an understanding of the concepts of these operators. However, I want to be clear in that you can use it, and we will use it in the future in other keywords, such as having, joins, and cases. So these operators are not going away. They're going to be the foundation of what we need to use for the remainder of the course.
53:37
So for part one of this, we're going to just cover the easy ones of equal to, not equal to, greater than, less than. And for these examples, we're going to be filtering in order to meet certain conditions as going through if we were searching through these job postings from the perspective of a data engineer. Inside of Mother Duck, I'm going to go ahead and create a new notebook. We're going to call this one 1.3 operators. In here, I'm going to start a new cell.
54:03
I'm going to start with a standard query that we're going to carry out through the rest of this lesson. In this, we're going to select certain columns, specifically that job ID column. Additionally, job tile short, job location, job via, which is like the online platform that we're going to be using for this. As you'll see, that list platforms like LinkedIn or Upwork or anything like that where jobs can be posted to. And then salary or average, because we care about salary.
54:27
All of this is going to be coming from that job postings fact table. So we'll go ahead and run this, press and command enter, just to see what we're getting here. And I'm going to close out of this pane right here and also zoom in a little bit so we can see a little bit better. Anyway, what I like here is we have the inspector over on the right-hand side that we can dive into any of the different attributes with. And it looks like, yes, the query below here
54:50
only shows 50,000 results, but in the inspector, we can see that there's a total of 1.6 million. So the first thing we want to do, right, we're going to eventually get into data engineer jobs, but I want to see remote jobs. And so we could use this job location column and filter for that using a WHERE clause, specifically specifying the job location equal to, we're going to say anywhere. For this, remember, remember we need to put it as a string, we need to put it in quotes.
55:19
And this comes back and it looks like we get about 144,668 rows. Now, I want to get remote jobs, and I actually want to call out something real quick with this is why we're actually doing this, is if I copy this query and then I create a new cell, paste this in, what we do have is in this data set, we actually have another column called job work from home. And this is a Boolean value column, whether it has true or false values. And we want work from home values
55:51
that are equal to true. I'm gonna go ahead and press play. Now, this is going ahead and return the results, but I'm understanding right now that maybe you don't understand what this Boolean value column is, I'm gonna just go ahead and add it in, so I'm gonna do a job work from home, add it in, run this query again. And it now adds in this job work from home, I can actually scroll over and see it, and we see all these different true values because we filter for true,
56:16
Boolean being true or false. Anyway, this one for filtering for job work from home equal to true, which would be a remote job, we have 1,000 or 144,532. Whereas if we scroll up to this one, this one has almost 100 more, even though this job location is anywhere, thus a remote job. So we'll be able to actually identify the discrepancies of what jobs are marked as, well, anywhere, and also are not necessarily marked for remote work. This is an end condition,
56:52
and we're gonna be covering it in our logical operators coming up. Next up, let's look at the not equal to operator. And for this, we can use two different types here. You can see here it's either an exclamation point in equal sign, or there's like two Pac-Man eating each other. For those data engineers that we're eventually gonna filter for also with this, I wanna look at, let's say, I wanna look at just contractor jobs. So going back to that base query that we have already,
57:18
go ahead and run this. I'm gonna add into here the column that we're gonna be filtering for, and that is the job schedule type. Running can man enter, so we can actually see this one. We can see down and over here at the bottom of the column, it has different values inside of here, and actually can go in and inspect it further, selecting job schedule type over on the right sidebar, and we can see that the full time has 1.42 million, and contractor has 75,000.
57:47
So what I can do is I can filter in a wear condition on job schedule type equal to contractor. Running command enter, we get our results at 75,000 for that's with all of them being contractor, but we want not equal to contractor, I think I said equal earlier. Anyway, in this case, all we need to do in our query is do a exclamation point equal to for the not equal to, and we get about 1.5 million results. We could also do the two Pac-Man eating each other,
58:19
and run this as well, and say a matter of rows appears. Now, because I think you're getting the gist of this, we're gonna briefly cover greater than or less than operators. In this one, we'll start with that same base query, job ID, job tile short, job location, job via salary or average, and we're gonna filter our salary year average column, let's say we just want jobs greater than 100,000. So we do wear once again, specify that salary year average, and say greater than 100,000.
58:48
Running this below, we scroll over the column. It looks like it's about correct, but we can actually verify it's completely correct by going over this inspector panel. We can see that they actually start over at 0.10 million to 0.12 million, which actually, looking at the statistics at the bottom, we can see that it starts at $101,000 because it's greater than. Whereas if we wanna do greater than or equal to, running this now, Command Enter, inspecting this column,
59:17
we actually get a min value of 100,000. Conversely, instead of doing that greater than or equal to 100,000, we could do less than or equal to 100,000, and we could see all the results less than that. All right, we're gonna take a break from comparison operators real quick and dive into logical operators to actually start combining conditions. We have three ones we're gonna cover in this and or a not. The and operator only shows records when both conditions or all conditions are true.
59:48
In this case, we have terms A and B, and then we have this nice little fancy Venn diagram off to the side. Unless they meet both of those conditions, it's not gonna be included in it, So it's-- [AUDIO OUT] quite limiting. So we're going to be looking for data engineers, which would be term a, and then also remote positions, term B. And if they meet both those conditions, we'll get it. So starting back with that very base query, we're going to be filtering where we remember
1:00:17
we want to do two things. So first thing is for data engineers, using that job title short column, we're going to use the equal operator to set it equal to data engineer. And now we want to include another condition. So I'll do and, and for this, we want the job work from home column, that Boolean value of whether they marked it as remote or not, equal to true. And I'm going to go ahead and run this press command enter, and we can
1:00:43
scroll down and see this or I can actually dive into it off to the side, we can see all the values are data engineer, and for the job location column, all of them are anywhere. And quick note on this Boolean value, you could also do a lowercase true like this that's going to work just fine. I like to use all caps pretty standard and it makes it stand out. Additionally, with this multiple conditions, I usually indent this down to the next line
1:01:07
to make this a little bit more readable. And it still works the same. Now, because we understand now the and keyword, we actually can investigate that issue we saw previously, where we had for a job location of anywhere, we had this many values. And then for the condition of just job work from home of true, we had about a hundred less values. So I'm pretty lazy. I'm going to go ahead and copy this job location anywhere up here. And inside of our query,
1:01:31
I'm going to replace this data engineer portion that's going on. I'm going to run this because now we're going to be looking for job location of anywhere and remote jobs. And so they may be we have 144,000. Well, where's the discrepancy at? And that's with the we know that the anywhere column or the anywhere for job location has more values. So there are some values that are marked remote or that marked anywhere that actually were marked false in the remote value,
1:02:01
running this now, we get the 140 rows. So now we have a list of the 140 jobs that are maybe marked incorrectly as job work from home, because they have the job location listed as anywhere, but yet the job work from home column is false. Next up is the or operator. And this only shows records when any of the conditions or either of the conditions are true. This would be the case if I'm looking for multiple different conditions. Say I'm looking for not only data engineer roles,
1:02:36
but also senior data engineer roles, starting with that base query. And if I go into and investigate the job title short column of this 1.65 million, we can see we have about 392,000 data engineer positions and around 91,000 senior data engineers. So we can use or to get both of those. So I specify where and then for job title short, we wanted equal to that engineer. Then on the next line, we're going to enter that or operator, specify for the job title short, we want then senior
1:03:07
data engineer roles, go ahead and run this and inspecting that job title short column, we can see we have both of those options. Last up on logical operators is the not operator. This thing can be used all over the place. And it's in a way of excluding certain values that we may not want. Now we saw this in the last lesson, I didn't specifically call it out as a logical operator. But remember in the last lesson, whenever we were filtering for cellular average, no values, I'm gonna go ahead
1:03:37
and run this, we had no values in that salary year average column. And then we used is not null in order to actually get all values that are not null. And then from that, we figured out that we had values from 15,000 up to $960,000. Now not can be used in any of our previous queries that we generated. So I just scrolled back up to our previous one where we had the job work from home equal to true. And if I wanted to search for instead of putting in false in this case for job
1:04:07
work from home, I could just specify not run command enter. And it provides exactly the opposite of this. And in this case, we get 1.4 million non remote jobs. Now not can also be used in a more complicated example, going back up to that previous example that we did, where remember we were looking at basically finding those jobs are marked incorrectly. They had a job location of anywhere. My job work from home of being remote should have been true, but it was actually false. And we had 140
1:04:37
rows for that. Well, if we wanted to basically exclude those values, because we think it's bad data, we could use a not for this. But the key thing is we need to apply for both of these conditions. What I'll do is I'll wrap them inside of parentheses to put it one joint logical statement. And now when I run this, we have that 1.6 million rows, and it's about, well, it is 140 rows removed from that of those that are potentially bad data.
1:05:07
All right, only two more comparison operators to cover of between and in will cover like in the next lesson, and already covered is null and is not null for between, we're using this to get values within a given range. So starting that base query that we've been using, let's say we want to get salaries between 100,000 to 200,000. So we'll go ahead and put a wear statement. And we'll say we want for the salary or average, we want it between 100,000 and 200,000 going ahead
1:05:43
and running this pressing command enter. Looks like I mistyped between, it has syntax error between, we'll go ahead and accept and run this, and then going over to the inspector, I can see, okay, I have values between 100 and 200,000. Now this I like to use, I'm going to actually comment this out, because previously not knowing this, you would have probably written the query something like this, where salary or average is greater than 100,000. And salary or average is
1:06:11
less than 200,000. So we'll go ahead and run this pressing command enter. And I'm realizing now, I actually have too many zeros in here, I said 1,200,000, nothing appeared in this query. You can actually little shortcut that the syntax or that the engine will pick up on is you can put these dashes in here. So I'll put those dashes right inside of here. And so what you can actually see and count the zeros a little bit easier, or if I go ahead and run this, we can see from the results
1:06:40
of this, we are still getting values between 100 and 200,000. Basically, the SQL engines not picking up this tack right here. And visually, you can see a lot easier when it's 100,000 200,000. Anyway, getting back to this point, this though, as you can see, it's highly, it's a lot harder to run to ride out. And it's a little bit less readable. Additionally, look at this, this current query that we have has 28, 631 rows. I'm gonna go ahead and comment it out. And then go ahead and put this
1:07:10
one back in. This one has more has 30,677. Why? That's because this one is inclusive. It includes 100,000 and 200,000. So this, this query down here at the bottom is actually written in correctly to try to match it. It would be greater than an equal to 100,000 or less than an equal to 200,000 running this one now. We have the same results 30,677. Last one up is in, and this allows us to specify multiple values within a where clause, much more succinctly. Previously, we had a query
1:07:48
that we were looking in to find job titles that were either data engineers or senior data engineers. Now, let's say I want to add in something like data analyst, I would have to put in another or statement and put it on in here. Instead, we can use in. So we're moving this previous portion, I'm going to put in, and then we're going to create a list. And a list in SQL is in parentheses, and then you put the values separated by a comma. So we'll include that data analyst,
1:08:18
that data engineer, and then also that senior data engineer. Now go ahead and run this, where we search that job title short column, we have three values. And this makes it super easy to add or remove values a lot more quickly. And it's a lot more readable, because we know, hey, we're specific to this job title short column. Let's wrap this lesson with a final example. And I'll be doing final examples at the end of a majority of the lessons to tie all the
1:08:47
concepts together and really test your skills. If at this point, you're feeling good enough, and you have enough understanding, you can feel free to skip ahead. I won't be covering any really major new details for using SQL. Anyway, let's say we have this problem here. For this problem, we're going to be looking at getting jobs for both data engineers and data analysts. I'm going to say I have more advanced skills for a data analyst position than a data engineer. And so for data
1:09:15
engineers, I'm going to say, Hey, we want to salary between 75 and 100,000. And then for data analyst, I'm going to say I want a higher salary. So I have 100,000 to 125,000. Now I also want to filter for location. I'm currently in Ben Villarcan saw. So I want either a job here or a remote position, or if I have to move, I'm going to remove to sunny San Diego, California. For this, we're going to be sticking with that standard query of all those different attributes or all those columns
1:09:42
that we want to get from this. So we'll leave those all the same that we used. I'm going to start first solving this first portion of data engineers of 75,000 to 100,000. So I'll create that wear clause and specify the job title short equal to data engineer, along with that salary condition of finding it between 75,000 and 100,000. I'm going to just stop there. I like to execute queries as I build them to make sure I'm building them correctly. So I'll verify that the job title search all
1:10:10
them is only data engineers, and that we're meeting that condition right now for our salaries. Let's move on to now meeting also the data analyst condition of 100,000 to 125,000. So this will be an and condition. I'm going to wrap all of this for data engineer into a single parentheses, and then right and, and then underneath it, I'm going to just copy this using command C or control C, paste it in here, and then go change the values to data analyst, and then the salary of
1:10:40
between 100,000 and 125,000. Let's go ahead and execute this bad boy, make sure it works, and we get zero values returned. Oh, because silly me, we want to be this is an or condition, right? We want either that engineer jobs or data analyst jobs. I don't know why I did that crazy me. Now for this, we have even more values. We see that job title short is both data analyst and data engineers, and then I'd expect the salary between 75,000 and 125,000. Now we can move to the third part
1:11:12
of we want to meet of job locations in either Bentonville, San Diego, or remote jobs. Remember, we're going to be using for these two Ben Villarquez on San Diego, California. We're going to be using the job location column. And then for the remote jobs, we learned that job work from home is more accurate. So we're going to use a different column to filter for that. Anyway, this is going to be an and condition linking. Hey, we want either of these roles, and they need to be located in any of
1:11:43
these conditions. So let's create an and condition between these two, and we're going to use parentheses in order to do this. I'm going to actually wrap this entire whole portion that we did already on filtering for the job title short and the salary information into one giant parentheses. As you can see, this highlighted on both of these conditions. What I'm also going to do is go ahead and just
1:12:08
indent this, I highlighted it all and press tab to move it in. So this makes it a little bit more readable so you understand, okay, all of this is one giant condition. And now we want to also meet on the condition of the salary. So I've did an and, and we're going to put whatever we need inside of this parentheses. Specifically, we'll just start first with this Benville Arkansas in San Diego, California, filtering for that job location. Remember, we could do Benville Arkansas
1:12:37
or San Diego, California, but in this case, we're going to use in and make a list of the different locations because I could always add in additional locations later with this. So I'll put in Ben and Villarcon saw, and then also San Diego, California. We will add in the remote work upcoming, but I want to just double check, make sure this query runs correctly. And it goes all the way down now to,
1:13:00
based on our locations, it goes down to 16 rows. And it does look like this is correct because I can see actually inspecting the results. We have data engineers in both Bentonville and data analyst in San Diego and vice versa, along with their expected salary. So this is looking good. And remember, we want to now get remote jobs as well. So this is only one statement right here. So what I'm going to do is I'm going to go ahead and put an or in, and then I'm going to specify that we want
1:13:29
to filter the job work from home equal to true. Now this is not an or condition. So I expect when I run this, the values are going to go up, and they do, they go up to 429. So there's about 400 remote jobs, probably more likely that I'm going to find a remote job than something Ben and Villarcon or San Diego, unfortunately. The last thing I do to dress this up and prioritize where I'm going to focus my effort is actually sorting this. And I'm going to use the order by to sort our salary
1:14:01
year average column. And we want to do it in descending order, because I want it from highest to lowest add a typo, make sure you're typing correct. Thank you, AI. Oh, and I can start now. It looks like a lot of different well, obviously, that analyst is going to have the highest in our conditions that we have. And I can start with these here for where I'm going to do my job search. All right,
1:14:21
your turn. I'll give it a try for those that purchase the supporter resources. We have quite a bit of practice problems for you now to work through and get familiar with all these different logical and also comparison operators. And then the next lesson will be jumping into that last comparison operator of like in order to use wildcards to filter further on basically doing some text analysis to filter out certain keywords. All right, with that, I'll see you in the next one.
1:14:50
All right, in this lesson, we're going to be exploring two major concepts. The first is on wildcards, and we're going to be applying it inside that logical operator of like. And then additionally, we're going to be looking at aliases, aliases are how we renames columns and tables to make it basically easier on a self to make queries. At the end, we're going to wrap it up all together and put both these concepts together in a final example.
1:15:20
So to make sure we're using the right terminology, I think I just used the wrong terminology on the intro. Comparison operators, we're going to be looking at the comparison operator of like, and like is similar to equal to except we're able to do pattern matching, which we're going to demonstrate. So inside of mother duck, I'm going to create a new notebook and call it wildcards and alias. For the queries in this lesson, we're going to be focusing on these columns of job ID,
1:15:51
job title, job title short, job location, and job via. That's a quick reminder that job title column is a column that shows the original job posting title in it. And then I used some machine learning separate from this to actually clean up those columns and designated into 11 different categories, which closing out this sidebar right here, and open up the inspector on the left hand side, and we can see these all these different values here. And I said 11, because there was 11 right
1:16:22
here, but it's actually 10. This I'm hovering over and it says approximate distinct values, it has this wrong, it's actually 10. Anyway, I digress, let's get into that like operator. So let's say we want to filter for job location, and we'll just say we want to filter for Columbus, Ohio. Previously, we just type in where put in that job location, and we set it equal to, in our case, Columbus, Ohio, going ahead and running this, we get a bunch of results from Columbus, Ohio,
1:16:49
on spec and job location, it's only Columbus, Ohio. Now let's institute it with pattern matching using like instead, and it's pretty easy to swap out. What we have to do is take out the SQL sign and put in like run this command enter, bam, now we're doing pattern matching. And we have the same amount of results of 1,464. So let's get our first squad card of the underscore. And since we're using that like comparison operator, we can then put special symbols inside of our value that we're
1:17:19
searching for. In this case, inside of data engineer, we want to look for one single character. In that case, it is a space, so we could leave it a space. But this is just a showcase, it is matching on not only that data engineer, but specifically that there's one character in between there. Now let's pass through a more realistic example. Let's say I wanted to search for Columbus's throughout the United States in different states. Well, in the United States, at least we do a
1:17:48
two state or two letter abbreviation for a state. So what I can do is use two underscores here, because underscore represents one single character, and that character can be anything. Now, whenever I run this pressing command enter, I can see that I get a host of different values actually popping up the job location right here. I can see I got them from all over the United States, even a few in Minnesota, Kansas and Mississippi. Now, the second and final wild card cover is the
1:18:18
percent sign. And this one I feel is more flexible. And frankly, I use it more. The percent sign represents zero one or more characters. So for this query right here, we're searching for engineer, the word engineer, and it can have zero one or as many characters once before, or after it. And so this would be a really great way to filter out different types of engineers from our job title column. So let's try it out on our original query. I'm gonna go ahead
1:18:48
and run this bad boy. And for this, we can see that there's a lot of data analyst roles in that job title column. If I wanted to, we can match on that using this operator based on what we know now. So we'll insert that where clause, specify the job title column, use that like comparison operator, and then put inside of here the percent sign data, and then space and then analyst and another percent sign go ahead and run command enter. And with this, we're getting a bunch of
1:19:18
different data analysts roles. And inspecting the job title short column, we actually can see like this one. Oh, it is Mark business analysts, but it is in the title business data analyst. All right. So now let's say we want to either share this table or we're maybe building out a data warehouse and other people can access it. And we want to create table names that the final end user is going to actually understand what is going on there. Then they won't send me emails
1:19:46
asking what they mean by it. Well, we can use aliases to solve that. Remember, we talked about the job title column is that original job title in the job description. We can use the alias of as that's the keyword there. And then from there, you can assign it a new name. In our case, we could do something like job title original. This is just a name I made up on the spot. I'm going to go ahead and run command enter. And now we can see down here that it changed the name of the column that's
1:20:17
appearing. It's alias. I could also go through and update other columns as well. So job ID is ID, job title short as job category, job via as job posting site. Now aliases are not limited to within that select statement. We can also use them within other statements as well, such as this from portion. And I could put in as and it's common for us in that engineering to rename this is a pretty long
1:20:44
name to something like JPF, which stands for job postings fact, running this query still operates exactly the same. And as you recall from previously, you could put the table names in front of the actual column itself that you're calling and going ahead and running this same results. We're not joining multiple tables right now. And so because of that, there's only one table in here. I like to leave this off, keep it as clean as possible, and not include that naming of the table as well.
1:21:15
Not necessary. Now, what can be slightly confusing when reading other people's single queries is, is that sometimes people like to go through and not use an alias in there. And they just write the alias right next to the column itself. And as you see, whenever I run this, it still works. It reassigns it as necessary. So that as alias is not necessary, but it improves the readability of your query. Let's get into the final example. For this, we want to look for not only non senior
1:21:48
data engineer roles, but also non senior software engineer roles. This is because a lot of the skills of software engineers and data engineers overlap. So in job search, you should probably look for both to make sure you're maximizing your opportunities and landing job. Anyway, to do this, we've broken down into three steps specifically, we want to only get job titles that include either data or
1:22:12
software. But then also include those with engineer in any part of the title. And we don't want to include anything that has something like senior or maybe SR for senior. That's typically followed by like a period or maybe some other character. So we'll just make sure it's no character. For this, we want to just get the job ID job title location and job platform. So this job title short right here, we're not going to use this time, we're going to go ahead and remove that. So let's
1:22:38
forget this first condition of only get job titles that include either data or software, we'll end at a where clause. And inside of job title, we want to match the condition use in that like operator for data. And this is an or right because we also look at software. So we're going to put in an or and put for this one job title, like software, go ahead and run this. And there's not a lot of options, right? Because we didn't provide, you know, we didn't use those wild cards
1:23:08
appropriately. Right now, we're only getting 107 values, right? We want to use that percent sign operator to match zero, one or more characters on both sides of these. So added in percent signs for both of these, let's see what we actually go up to from 107 to 1.3 million, inspecting the job titles here, inspecting the values here looks like we're being our conditions, although it's kind of hard to see software. I'm not seeing any really pop up. Now, real quick, you may be tempted
1:23:36
to in this condition, right? We've sort of had to repeat what was going on here. So you may be tempted to use that in operator specifying data and software using those wild cards. However, when you go to run this, you're going to get zero rows, because the in operator is not a pattern matcher, like like is, like like is. Anyway, we can't do that. We're not going to use it. So let's just move on. I want to show that. So now we're going to be getting into, we want to also
1:24:03
include those with engineer in any part of the time, remember that engineers or software engineers. So we're going to wrap this all because we want to meet this condition. And we want to meet the condition of the engineer. So we'll add in job title. And we'll say it's like engineer running this again, commander. Now when we look at this job title column, we have a lot fewer jobs popping up. And now we are seeing that software actually pop up inside of here. Now we do want to move
1:24:31
into the last part, right? We want to remove or don't include any job titles with senior or SR followed by any character, as we can see here, we have that SR period. And we have senior down here. So we'll add an end condition. But for both of these, they want it not to be senior or SR. So I'll say and not. And then I'm going to go ahead and copy this one up here, because there's a lot of rewriting
1:24:54
going on here, add in job title like senior or job title like SR. Remember, it says followed by any character. So we can do we could do the underscore, but we'd also then want to do a percent sign. So I'm just going to do only the percent sign just for it to make sense. Now going ahead and running this, bam, inspecting the job title column, it looks like yeah, all of those senior roles are now removed, but we still have our software roles in their end that engineer roles. The last thing we
1:25:26
need to do is just rename all the columns appropriately. So I'll change job ID to just as ID, job location as just location and job via as platform. Going ahead and now running this, we have all these results with ID, job title, location and platform, one or two, I could go ahead and copy this first 100,000 to my clipboard, or download that amount to a CSV or whatever. So that wraps up wildcard and aliases, do now have some practice problems for those that purchase support resources to go
1:25:56
through and get more familiar with those wildcard and aliases. In the next lesson, we're going to be covering arithmetic operations. So very much following this theme of operators. All right, with that, I'll see you in the next one. In this lesson, we're going to continue learning more about operators. Previously, we did comparison and logical operators along with wildcards. And in this one, we're going into arithmetic operators.
1:26:28
Now, soon when you have a grade school level education, you've probably covered all of these different arithmetic operations. Well, except for the last one, where we gone over plus minus multiplication and also division. And then the final one you probably haven't heard of yet is modulus. So first, we're going to start off with that addition and subtraction. So inside a mother duck, we're going to start a new notebook. And we'll call this arithmetic
1:26:55
operators. I'm going to close out of the side panel, add a new cell. And I haven't demonstrated this before, but you can actually do very simple simple, very simple SQL with only a select statement. So I can do select. And then if I wanted to, I could do something like 11 plus two, run command enter. And it provides me the results. Similarly, I could do something like 11 minus three, rest command enter, and results as well of nine. Now, before we go further with a real
1:27:26
world example, I do want to call out real quick that we're not only limited to use these operators inside of a select clause, you could also use it in other places like a where order by or group by or having which we're going to cover eventually. The order by is least likely place we're going to carry it. The other ones are more likely where we're going to see it. So getting to that use case of operators for this, we're going to go into LinkedIn to look at job postings. Oh, by the way,
1:27:54
if you aren't following me on LinkedIn, feel free to follow me. Anyway, if I go into jobs up here in the top left hand corner and actually search for some sort of data engineer job, it's going to go through and populate this. But the thing to notice with this is whenever we look at anything that has a salary, they actually list a range. This is a little bit this is I mean, this should be criminal.
1:28:18
99,000 to 232,000 dollars for this job. Anyway, companies list this range because you could have any variable amount of pay based on your experience. They don't typically list a single value. Well, jumping into our data set, feel free to run this query if you'd like. I've queried the job ID, the job title short column, and then two salary columns specifically the salary year average and the salary hour average, just as some background to understand why that name averages on it.
1:28:53
That's because typically whenever I scrape this data and collected this data, there was a band and I collected and I've made all of these things instead of having a band made into a single value so we can compare it down the line. So in this case for this cloud data and analytics manager, whenever I scrape this, this is going to be put in as the minimum salary of 99 and the max salary of 2000 or 232,000. And what's the average of that? Well, running some SQL adding 232 to 99 divided
1:29:26
by two average run 165.5. So the salary or average for that job would be 165.5,000. Anyway, let's recreate some fictitious bands, if you will, of the min and max salary. For this, we're going to be focusing on that salary hour average column because it's a little bit smaller numbers and it'll just make this a little bit easier for us to read. We're still going to understand the same concepts. So for our starting point for this, we're going to start with the job ID, job title short,
1:29:54
and salary hour average column. And we're going to filter out any jobs that have no values in that Gallery Hour Average Column. All right, so this is what we have. Now, we wanna create a, if you will, a fictitious band of men and max. And what we're gonna say is the men band is $5 under this average band, and it's then $5 greater than this for the max band. So for this, I'm gonna create a new column. We're gonna start with that men band first, and we'll say salary hour average,
1:30:30
and we'll subtract 5 to get the men. I'm gonna add a come on the end, press command enter. Okay, automatically whenever I did this operation, it put the operation as the column title. I'm gonna go ahead and give this an alias of as salary year men. Oops, I mean hour. And now running this, the column's gonna update of salary hour men. And we can see as it goes through for each of these values, it just subtracts 5 from that associated row and provides to that value.
1:31:01
Similarly for the max, I put in that one I want to add 5, and I name this salary hour max running this. I got this error, parser syntax error, add or near plus, and I can see based on the AI, but you should be able to see this too, that it's missing a comma. At the end of that salary hour average, 'cause I added an extra row there. Anyway, gonna accept and run it. And double checking these results, it looks like that salary hour max does add that $5.
1:31:26
Now that plus and minus example, isn't really realistic of the plus or minus five. Instead, I asked chat GPT, hey, what's the typical percentage band of salaries on job posting sites? And that's to give it a percentage of the average salary. And it states that the minimum range is at about 80 to 85% of the midpoint, and the maximum is about 115 to 120% of that midpoint. So let's actually use the multiplication operator in order to carry this out and make a more fine-tuned model
1:32:03
with the min and max salaries likely work. So for this, I'm gonna take that original query that we used already, and for the min, we're just gonna multiply this times 80% or 0.80. Similarly, for the max, we're gonna multiply this times 1.2. Now running this, pressing command enter. Here are some values that are way more closer and way more likely to be what would have been the min and max of an associated salary. Now I also wanna demonstrate division,
1:32:33
so I have another scenario for you. And in this one, we previously looked at the salary year average column, and in this we're gonna filter for removing any no values. I'm gonna go ahead and run this to show it. Anyway, so we have the different job IDs, job tile shorts, and that salary year average column. Well, as you recall, we do have that salary hour average. If in the future we wanted to maybe compare the values from these two, we could make it into a similar value.
1:33:01
Meaning, let's say I wanted to get from the salary year average, I wanted to convert it to an hourly salary equivalent. Well, how can we do that? Well, we're gonna start a new line here, and I put in salary year average, and what we wanna do is, is yearly, so we wanna divide by how many hours are in a year. Well, typically people work 52 weeks in a year, if you will, and there are 40 hours in a week. Now those not necessarily realistic for most people,
1:33:36
but we're just gonna go with it, all right? So going ahead and running this, we can see how a lot of these transpose over to a hourly salary with this. Oh, I need to also give this an alias. And I'm gonna call this hourly year salary. So in this case, we now have that hourly year salary. And if I wanted to, I could then also bring in that I'll change the salary year average one to salary hour average, and we'll bring in those that our salary year average
1:34:09
is not null, and salary hour average is not null. Running this, we're not gonna be able to compare across the two, and no, we're not. It's only returning zeroes, 'cause I got my logical operators wrong. We need to do an or value, running this now. Now we have a way to look basically in apples-to-apples comparison between job descriptions with that hourly salary and that with the hourly salary of our yearly salary. In the next lesson, we're gonna actually get into
1:34:42
being able to form aggregations across entire columns. So this will be more useful than trying to go through these 50,000 results and trying to extract any insights out of what I'm trying to do here. But I think you're getting the gist of what I'm trying to do with this. Last one up is modulus. And this returns the remainder of a division operation. And many of you probably haven't seen it before, so let's just play around it using it only in select real quick.
1:35:11
So if it is something like select three modulus, which is the percent sign, one, and then run this, well, three divided by one is, well, three, and there's zero remainder. Now instead, if I were to divide three by something like three modulus, two, you would expect to divide this in there, and then when we run this, have a one as a remainder. This type of operation right here is really common, or it's a really common problem to be done to find out if a number is even or odd.
1:35:46
So you don't have to have some list of even or odd numbers. You can put something in like, oh, I could say 10, 10 to modulus two. If it's even, it's gonna return zero. In the case of, let's say I change it to 11, 11 modulus two is one. So if it ever has a remainder greater than zero, it's an odd number. So here's the scenario I have. I'm pretty picky. And as you can see, when you go through this salary year average column that I've queried here,
1:36:12
these salaries are pretty clean, thousands. But then you have every once in a while, you have something like this where it's 120,531. Let's say we wanted to filter out values that were only ended in basically three zeros or were specifying a thousand. So something like this one is gonna be filtered out, along with this one right here of 130,500 because it's not three zeros at the end. Well, we can do this with modulus. We can see what the remainder is.
1:36:42
So let's have it in a new column and we'll call this salary year average or we'll bring up salary or average and we'll form the modulus of this by 1000 or those three zeros if you will. Let's go ahead and just run it to see what it shows. Okay, in these cases that I wanted the value, the value is zero. And then for those that I want, it's basically greater than zero. So I can now filter these out. And I can do that by adding to our WHERE clause
1:37:12
and specifying that we want this condition right here, which I'm just going to go ahead and copy this and paste it here and then go ahead and press Command Enter. And it's only me, we didn't actually even specify anything right now. We just said, hey, and salary year average modulus 1000, we have to actually set it equal to something. We want to set it equal to, we want it to be equal to zero. So I'm going to go ahead and Command Enter that
1:37:38
and it returns previously, it was removing the zeros because zeros are equal to false and thus far it removes it. All right, for those that purchase the practice problems, you now have some problems that go through and get more familiar with those arithmetic operations. In the next lesson, we're going to be jumping into aggregation functions and actually putting to use what I was hinting towards in this lesson about comparing that hourly and your yearly column.
1:38:07
All right, with that, I'll see you in the next one. In this lesson, we're going to be going over aggregate functions and functions are just something that you're allowed to give into that output, some sort of result. And in our case, we're going to be forming aggregation like sum, min, max, or count. Now, in addition to those functions, we're also going to be covering two new keywords and that is group by and also having. We need those to use with our aggregate functions.
1:38:38
Now, here is a table of the different aggregate functions that we're going to be going through where we reference this multiple times throughout this lesson. But the main point, as we can see from that first column, is that we need to input into our function some variable. In this case, it's going to be our column name and that is what the aggregation will be performed on. We're going to start with the easy ones first, like count and work our ways up to a even harder one
1:39:09
that don't even have lists on this table. Now, as we'll get to in the lesson, you can do aggregate functions by themselves but they become even more powerful when you bring them by them with keywords like group by and having. Group by allows you to segment by a certain condition like job title short in this condition and then having allows you to filter. But I'm getting ahead of myself. Let's actually get into some of these functions.
1:39:35
All right, the first one we're going to cover is count. And it can be used in two different scenarios. We can either do count star or count column. Count star counts all rows in a table, including those with null values, whereas count when we're just using the column name gives only non null values in a column. So let's jump into mother duck and actually test this out. We're going to create a new notebook and we'll call it aggregate functions.
1:40:03
Now it's pretty common when exploring data set to find out how many rows are in it. So we're just going to do something like select and then call out to do a count star and then from there specify the table name of job postings fact. And so this returns back the number of rows in our data set, which we know already 1.6 million. This is pretty great because we could actually apply a condition to this as well, such as filtering for data engineer jobs only.
1:40:28
And in this case, we only have 391,000 jobs for them. Now I could in this case list a column instead of count star, such as in this case be more appropriate to use the job title short column if we were not going to use the star. So running this, this provides exactly the same amount of results. And if we go back, remember that count star counts all rows, including those with nulls, and count column name counts only non null values in a column. So if you remember, I'm query in here,
1:41:02
a salary or average column. If you remember from that column, there are a lot of null values in here and they do have a salary every now and then. Well, this would be a good case to just use count of the column name. Specifically, going back to our original query, if we wanted to find out what data engineer jobs have a yearly salary listed, we can put that in here and we're going to see the jobs go from 391,000 to 10,551. Unfortunately, a lot of job postings
1:41:31
don't provide the salary. Next up is, well, we're sticking with count, but we're also adding the word of distinct and this counts unique non null values. So if I were to query the job title short column, we can see that a lot of these values are repeating in here. I also showed you how you can actually dive in on the right hand side via this UI, but you may not necessarily have access to this all the time when you're doing SQL queries. That's where distinct comes into play
1:42:04
if you wanted to get these distinct values. So I put in that count function and inside the parentheses, I'm going to add in distinct. Then I'm going to close off those parentheses, run command letter, and it would tell me that the distinct values or the amount of distinct values is 10. And just as a refresher from one of our earlier lessons on basic keywords, this is that same distinct keyword that we use in order to filter and actually list all those different job title shorts,
1:42:35
which is as simple as removing these parentheses and then this count function right here and then running it as is to show those values. Next up is sum and this adds up all the non null values in a numeric column, which we have two numeric columns right, salary or average and salary, our average. So in our case, let's apply it to that salary or average column where we do have results eventually in here. All we're going to do is wrap salary or average in that sum function right here
1:43:07
and run command and enter, we can get the sum of this. And the sum is around 6.3 trillion dollars. Now with that, we could, I don't really find sum is actually that useful. In this case, I could get the average and based on what we learned already from our arithmetic operation, we could divide by the count of that salary or average column, which as we know, counts all those non null values. And then in this case, we get the average salary, but there's actually a function for that.
1:43:38
And that's average. It returns the average of all non null values, specifically once again, this is for a numeric column only. So underneath this column, I'm going to put in that AAVG function and insert the appropriate column, run command and enter, hopefully the same results. It is the same results. Now up to this point, we've just done aggregation functions by themselves to refine basically an aggregation of the entire data set. We can actually break this data set down
1:44:12
and actually segment our data using a group by clause in order to understand characteristics specific to certain values in a column. Basically group rows that have the same values into summary rows and provide a summary for it. So what do I mean by this? Well, what happens if we wanted to get the average salary broken down by country? I'm going to go ahead and remove this line right here because we already have average on there.
1:44:39
And remember we have in our data set a row on job country. Now if I were to go ahead and try to run this pressing command and enter, it's going to be, hey, say hey, column job country must appear in the group by clause or must be part of an aggregation function. And thankfully to this AI here within here, it actually tries to add in this group by for this to work properly. I'm going to go ahead and just reject that to show you what I would normally do.
1:45:07
So underneath the from clause, I'm going to insert in our group by and we need to specify any and all columns that are not aggregated to be all to group by. So entering in job country, I'm going to go ahead and run this and we have over 161 results. It's a little hard to read. So I'm going to actually order this in descending order. So I put in that order by and specifically, I want to order it by this portion right here, this average of the salary or average. I'm going to run it now.
1:45:42
Oh, remember we wanted in descending order. So now we can see the highest country, apparently it's Panama, $279,000 for the average salary there. Now let's say I want to take this a step further in that I want to not only categorize it or break it down by country, but also by that job title short column. I would go ahead and put in here that job title short column up in the select statement. Now if I tried to run this with now modifying the group by, I'm going to get an error saying,
1:46:12
hey, you need to add it here. It has some AI that's trying to fix it. I'm not going to let it do that. I'm going to do it myself. And under job country, I'm going to add in that job title short. Now whenever I go to run this, I have it broken down by country and job title short. And I can see that, ooh, Belarus for data analysts is at $400,000. Now one thing to note with this, some SQL dialects allow you to, if I wanted to, I can assign an,
1:46:40
well, all SQL dialects allow you to assign an alias for this column of average salary year. So I could say, hey, this is average salary, but only some will allow you to actually use that alias within the order by. Microsoft SQL Server, for some reason, doesn't allow this. But anyway, going ahead and running this, it still operates the same. And I feel this is a little bit more readable with this. (gentle music) All right, next up are max and min.
1:47:10
And they find the largest non-null value in a column or the smallest in a column. This is not specific for these two specific to numeric columns, because you could have something like date, where you want to get the min or max, or you could have something like text values, where you want to get something that's close to the As for the min or Zs for the max. For this example, we're going to stick to only our numeric column. So in here, I'm going to add in that min,
1:47:37
and we'll do that for that salary or average column, and I'll give it the alias min salary. Similarly, I'll add in that max function for the salary or average, and give the alias of max salary. Let's go ahead and run this. And it looks like we have some errors, because I forgot to also syntax error add on your average, and that's because I forgot to put in commas at the end. Let's go ahead and run this again. I also mistyped salary your average in there. I kind of like this AI,
1:48:03
but saving my butt here and there. Anyway, now we have for each of these different breakouts for this also the min and max salary. So that one that is in Belarus is very highly skeptical, because apparently the min and max was 400,000. So I'm guessing there's only one job posting in this country for that analyst. Now, one note on group I, and that is for DuckDB specifically, so that friendly DuckDB, I like it because you can actually replace, you don't have to list these columns.
1:48:35
You can just write the keyword of all, and it captures all of these keywords that are in here. What if I go to run this? It's gonna provide the same results. Once again, this is only DuckDB specific, but it looks like other SQL dialects are also picking this up as well. But I'm gonna go ahead and change this back, so I really want you practicing with things that are generic to most all SQL dialects. Next up is the median function,
1:49:04
and I wanna first execute it to show you what it looks like, and then actually get into the theory behind what median is for those that may not be as familiar with the statistical function. For this, I'm gonna clean this query up a little bit. I'm gonna remove that min and max, and we're also gonna only group by that job title, short column, to make this just easier to read. Now inside of here, I'm gonna add in that median function, that is MDIA, and we're gonna be running this
1:49:31
on that salary or average column. And we'll give it the alias of median salary. We're gonna go ahead and run this. So inside of here, we can see that we have the average on the left, median on the right, for these different job titles, and that for each of these, the median is appearing, it seems very close to this, but if you notice, the median is actually slightly less than, well, all of these values, except for the very first one. So it's almost consistently less than.
1:50:03
Let's get into the theory behind how you would calculate median. So let's say you had a list of values, such as our salary values. What you would do first, if the number of values is odd, you would put all of those values in order from lowest to highest. Then from there, it's pretty simple in that you just select the middle value, the median value, and that is what it is. So in this case, it's 100,000. But for these five values, you see how the seller goes on.
1:50:34
If you actually calculated the average for this, we can see that the average is actually much higher because look at this last value right here, it's at $300,000. So it drags that average up. If I go inside of here and put in that max value in order to find that alias of max salary, we can see that all of these different job titles have like inorbitantly high values. But honestly, I would think that's actually unrealistic to even be achieved.
1:51:07
That's why I'm really a big fan of using the median because these high values are really dragging that average up. Now, what happens if the number of values that we put in order is even, such as in this case? Well, similarly, we put it in order, and then from there, we'd select the middle two values. With those middle two values, we then do the average of those two values in order to get the median, which in this case would be 95,000. Comparing this to the total average
1:51:38
of this all the six values, the average is once again, drag even higher once again because of that outlier all the way in the end at $300,000. So whether we have an even number of odd number of values, we're basically getting about the same amount or the same value every single time for that median. And I really like median for analyzing that salary because of this issue that I've gone through. Because of that, for the remainder of the course,
1:52:03
the focus will be for looking at the median salary over the average because I feel like it provides a much more representistic value you should be aiming for. Last keyword to cover that can be used with aggregate functions is the having keyword. And this filters queries based on aggregate functions and groupings, it's used because you can't use an aggregate inside of a wear clause. So applying it in our current query that we've been going through analyzing
1:52:36
these different salaries for different job titles short. Looking at this, I can see maybe I don't wanna target roles that are less than 100,000. Sorry, business analyst and data analyst. Well, what I could do is I want to target roles with a median salary greater than 100,000. So I would use a having clause. Your having clause is gonna be used right in between your group buy before or after group buy and before your order buy. And you're specifying having.
1:53:03
In this, we need to not only specify the function, but the condition that we wanna meet, that we want it to be greater than 100,000. And I'm putting that underscore there like we talked about previously to make just this easier to read. Going ahead and running this pressing command letter. Boom, I can now see that all those different jobs of less than 100,000 median salary are filtered out. Now for DuckDB specifically, similar to group buy, you can use column alias
1:53:32
in the having, but with unlike group buy that it is common other SQL dialects, using it in the having is not common to other SQL dialects. So DuckDB specifically, I could take this alias up here a million, median salary. I'm just command C or control C it and paste it down here. It's gonna work the same in this query. So just to bring this all together, I made this fancy-dancy table that shows other popular databases and those that allow are marked yes if they allow to use an alias
1:54:03
and things like the where, group buy or having. As you can see, it's pretty consistent for DuckDB to allow this via their friendly SQL syntax. So if you're going through and learning this, I wouldn't be, I would definitely, I would encourage you just not to use alias in those. That way, if you were to switch to another SQL dialect, you're not gonna get confused on why it doesn't work. All right, you now some practice problems to go through
1:54:28
and get more familiar with these aggregate functions and a lot of fun examples made for you. In the next lesson, we're gonna be switching things up a little bit. We're gonna be getting into some other data engineering skills, specifically into setting you up with a terminal and how to use it and then installing DuckDB locally. And then we'll be running DuckDB locally for the remainder of the course. I think it's really fun and it's really helpful skills
1:54:56
are gonna help you out as a data engineer. With that, I'll see you in the next one. Welcome to this lesson on an intro to a terminal. And we're gonna be taking a short break from learning SQL commands in order to get DuckDB set up on your local computer. So you can operate SQL queries just like a data engineer would. Now I've broken this up into three lessons that we're gonna go through in order to get things set up. The first one is this, which is on the terminal
1:55:31
and you need to understand the terminal in order to go into the next lesson on installing DuckDB. 'Cause DuckDB is run from the terminal inside your local computer. And then next after that is the VS Code or Visual Studio Code intro. This is gonna be our code editor. So we're actually gonna be writing SQL queries and executing it in tandem with the terminal. We're gonna build out our entire project in the VS Code but more on that later.
1:56:02
Anyway, on this lesson on an intro to the terminal, we're gonna be going through, I'm gonna assume you have no experience with it or that you don't even have it installed or set up. So we're gonna get it all set up and installed for you and then go through some basic commands. But before we get into all that, we understand what the heck is the terminal.
1:56:23
So here we are inside my computer. I have two virtual machines running. On the left-hand side, we're running Mac OS for those Mac users out there. And on the right-hand side, we're running Windows, Windows 11 specifically. And that's for those Windows users. For this lesson primarily, I'm gonna be going through and executing in both environments so you can follow along. But in follow-on lessons, I'm not necessarily gonna do that because I think you're gonna get the gist after that.
1:56:48
Anyway, in both of these OS's or operating systems, I have up the terminal app. And this is a tech space interface where you type commands instead of just clicking around in the GUI or graphic user interface. It's how engineers interact directly with their computers and automation tools. And Mac, it's just the plain old terminal app that we're using for this. And in Windows, they use the same name, terminal or Windows Terminal. Now, both of these use different shells
1:57:18
and shells is the language behind the scene that's interpreting what you put into it. Let's say I went to list the context on Mac, I could just write LS and they would list all the different folders in here. In Windows right now, I'm using the shell of PowerShell and I would enter something like this of get child item. And it's gonna show me the same exact thing. Granted, this isn't a little bit different format, but you get that idea. Anyway, they have, we're working in
1:57:46
in these two different environments in two different shells. Now, different operating systems have different shells, but it's different, if you will, default shells because you could actually put any shell into your operating systems that you want. For example, that Mac OS that I'm working in, it is in using Z shell. And that's for most modern versions. However, previous older versions are using bash or born again shell. Windows, as we demonstrated, is using PowerShell.
1:58:13
And then Linux is using either bash or Z shell. Now Z shell and bash are very similar. Basically, anything you know about bash can be applied to Z shell. So basically, if you know bash, you know Z shell. Except Z shell has a few more extra features, which is dead engineers, not gonna really affect you. However, PowerShell is completely different. It does have some similarities, but a lot of differences. In our course notes, I've put together this fancy-dancy table and it goes through
1:58:45
and shows what are the differences between the two. Now there are in regards to commands. And there are some similar commands used, but eventually it does get complex enough that there's not really enough similarity between the two. So we need to use a consistent one. So which one we're gonna use? Well, we're gonna reference the 2025 Stack Overflow survey where they interviewed over 30,000 different developers on what is their, well, language, and then more specifically for us,
1:59:13
what is their shell of choice? And out of all of these programming languages because shell languages are a programming language or a scripting language, bash comes in at fifth on this list. Right after SQL and Python, a core skill of data engineers. PowerShell is on this list, but it's much further down and I found less likely to be used in my data engineering workflows. Now to bring this all together on why I even covered this, you need to understand if you need to install
1:59:46
something special for your computer to continue with this course and to have the necessary tools at that end year. If you're on a Mac, newer or older, whether you use Z shell or bash, either of these are gonna be fine because like I mentioned before, Z shell, L is almost identical to bash. Now, if you're on Windows 10 or 11, you're probably using default shell of PowerShell, or if you're on your older Windows, you're gonna use, you're using CMD or Command Prompt.
2:00:12
For either of these cases on Windows, we're gonna get you set up with another shell in order to work properly with bash. And if you're following along on Linux, one kudos to you, but two, you also have bash set up in there as you probably know. (gentle music) So for this, we're gonna walk through opening or installing what's needed for the Mac first, and then the Windows users after. The Mac users, we're gonna be using that terminal app using Z-Shell or bash.
2:00:43
And then for the Windows users, we're gonna be using Windows Terminal with the default shell switched to Git Bash, and we'll have to install Git Bash. So let's start with those Mac users. So on a Mac, pretty simple. On your home screen, you're just gonna press CMD and Spacebar, and it allows you to start searching for things. You're gonna search for Terminal. And that's pretty much it for this section on installing or setting up Terminal. It's ready to go.
2:01:06
If you want to make the screen bigger, you're gonna press CMD+ or CMD- to make it smaller. So I'm gonna make it a little bit bigger so we can zoom in on it. Now inside of Windows Terminal, you're gonna have this first line up here that's gonna tell you your last login. From there, it's gonna tell you whose Mac OS that you're on. You'll also see the location that you're located within your file directory. Don't worry, we'll get more in that a bit, which is this little tilde line.
2:01:32
And then the percent sign is basically signifying that's the beginning of where you can now enter in any command, something like LS. Now switching over to those window users. You're gonna come inside your search bar, and you're gonna search for Terminal. And this fancy-dancey terminal app should appear. Nope, we're not using this one of Windows PowerShell or of command prompt. We're using Terminal app. And this Terminal app, it's pretty dang good. I'm actually a big fan of it.
2:02:00
Now this one similar to Mac, you can make the words on the screen bigger or smaller by pressing Ctrl and entering either positive or minus. By default, it should have opened up to PowerShell. And I like this one also to be right because you can do different tabs and have multiple different shell environments opened up. And it has by default, not only Windows PowerShell, but also command prompt. And it looks like Azure Cloud Shell, whatever. Anyway, we, both of these,
2:02:32
we're not gonna be using for the course. Remember, we're gonna be using git bash. So we need to git git bash installed so that we can use it inside of here. So in your favorite web browser, navigate over to gittechscm.com. This is where we're gonna be installing git, which is a version control system, and has the bash environment or bash shell needed to actually execute the commands that we want inside the terminal. Also, I know we have some steps to go through this,
2:02:59
but you're actually doing some pre-work as Windows users because we will be eventually installing git in order to manage the version control of our project. But it's a whole other lesson to get into version control. So more on that in a bit, let's actually just get into installing this. So to navigate down, go to install for Windows. And it should have your appropriate one that you need for your type of Windows appearing up at the top.
2:03:21
And we go ahead and click here to start downloading. From there, it's gonna open the file up. It's asking to allow it to allow this app to make changes to my device, yes I do. Now we're gonna have to go through a little bit of setup. Through this, I would leave everything as default and accept these things on where it's setting it up. So these default things, I'm gonna leave all the same. Now I just went through and left all the default settings
2:03:43
that I had set up for me, just accepted it, and it's now installing it. It said that it's complete, and I want to launch Git Bash, 'cause I wanna actually view that. I'm gonna close this of view release notes and click finish. If it didn't pop up for you whenever you did this, you can actually just go in and type Git Bash, and you can get it that way as well. Now I make this slightly a little bit bigger by pressing command plus and making it bigger.
2:04:07
Now for this terminal inside of Git Bash, it has your username, along with the computer using, and then also your directory location, which is that tilde. And then next to the dollar sign, you can go ahead and enter any commands next to that, and actually run it. Don't worry, we're going to commands in a little bit. But this is not the terminal that we wanna run Git Bash in. If there's nothing wrong with using this terminal, I'm just not as a fan of it
2:04:33
as much as something like the Windows Terminal app, and we can get it installed right inside of here. So let's get it into here. So I'm gonna press down this down arrow, and we're gonna go into settings. I can also press Control comma. Scrolling on down on the left-hand side, I want to go down here. We see that we have the PowerShell, the Command Prompt, and the Azure Cloud Shell. I wanna add a new profile. Specifically, I don't wanna duplicate a profile, I want a new empty profile.
2:05:02
For the name, I'm gonna change this from profile four to Git Bash. For the command line, we're gonna be changing this as well. We need to browse where this is located. You should be able to follow along where I'm going, where it's located, specifically in the C drive, under Program Files, under Git, and then Bin. And then we have this bash.exe file right here. This is what we wanna actually open up. Now I'm gonna modify this slightly.
2:05:30
I'm gonna put double quotes around this file path, 'cause I'm gonna add a command onto this to automatically log in. So I'm gonna do a dash dash, and then log in, and then a dash i. This login portion, just make sure that it's basically a fresh new shell that just opened up. And this i portion, makes sure is that you're able to run it in an interactive mode, basically, type into it. Next up is the starting directory. And for this, I'm going to uncheck this,
2:05:59
use Parent Process Directory, and we're gonna use this one to, basically navigate to our user profile. The percent signs around it's a special windows way to get to your user profile. Next up is the icon we're gonna use for this right now. It's this like, lookin', see lookin', weird lookin' thing. And that's specific for command prompt. Instead, I wanna go to a file location where this icon's at. Conveniently, it's in that git location.
2:06:22
So navigating again to our C file, program files, Git. And then this one of Clangarm64, you may have MingW64 or something like that, something with 64 or something else that looks like this. You wanna go into that, then under Share, then Git. And then you have this Git for Windows icon right here. And then everything else can be left as is. I'm gonna go ahead and click Save. Now I can come and click the down arrow and see that Gish Bash appears, and it has the icon.
2:06:53
And I can locate it right in here. But the one thing you'll notice is if I close out of a Windows terminal, yeah, I wanna close all the tabs. And I try to launch it again. It's gonna go into PowerShell. I want it to go into Git Bash by default. So I'm gonna go back into Settings, and under this first tab of startup, as under the default profile, I'm gonna change this to Git Bash. I'm gonna save this as well. And now whenever I go ahead and open this up, clicking the Terminal,
2:07:22
it automatically opens up the Git Bash profile. But also on Windows and Mac, add this to your taskbar. So now let's get into some basic terminal commands. Now that we have Terminal set up appropriately on both Mac and Window machines.
2:07:40
For this, we're gonna be going over four or basically five different types of terminal commands shown here. These are all used to navigate within your different folders or where you are inside of the terminal. The first one up is PWD, or Print Working Directory. In both shells, I'm gonna go ahead and enter, press Enter for that PWD command, or Print Working Directory. It's gonna tell you where your terminal is located. Right now it's located in that users,
2:08:12
on the Mac, Loke.data Engineer, and then on the Windows one, I'm in the users/loopbrews. It's very important that you understand where you are in the terminal because depending on what commands you give, you may not have access to files depending on hierarchies in your file system. Speaking of which, let's list the contents of our files, and we'll do that with the LS command. Running that on Windows and then running that on Mac. Starting with Windows,
2:08:40
we can see that we have all these different folders in here, specifically have something like the documents, downloads, favorites, they even have our desktop. Now, I wanna note, right, these are a virtual machine. These are actually looking at two separate locations, even though this is on the same computer. So there's gonna be differences between the two, but generally they're about the same. Such as on my Mac, I still have access
2:09:04
to things like the desktop downloads and things like that. So now with the information that I gave you, you're probably like, how the heck do I change into a directory? Well, for this, we're gonna be using the CD command. And we're also gonna do it on the Windows one, but before we do that, you notice it's at the bottom, the screen, there's a whole lot of clutter. You can actually type something like clear, and it will bring you up to the top.
2:09:28
You can do the same thing on a Mac as well, I'll type clear, or bring you to the top. Anyway, if we wanna navigate into specifically like the desktop folder, I would type CD, and then desktop, press enter. Now I can see on Mac, it went from that tilde to the desktop, because that's where we're at. We'll do the same over here, and we went into the desktop location. Now say I wanted to now go back, or up one level. I would use CD, and then dot dot dot, dot dot signifies up one level.
2:10:01
And we see we're back at the tilde. Just execute on the Mac, it did the same thing, we're back at the tilde, or that home directory. Now a cool neat little thing, if you don't know how to spell properly all the time, you can do something like this. I want to go back into that desktop, right? I can start typing it, so DES, and then if I press tab, it will finish it for me, and I can navigate into it. Same thing on my windows, start typing in desktop, press tab, it updates it for that.
2:10:26
Now remember, I'm still my desktop, it is possible to go past that home directory, so if I do that, CD dot dot, I'm up in that the tilde location, which doing PWD, I can see I'm at users/loopbrews-danger. I could go up even more, and now I'm in the users profile, and I can type LS and see what's available there. Similarly, I can do the same thing on the windows over here, and get back up into there. They have inside of their users folder, it looks like a little bit different.
2:10:55
But I wanted to navigate further up, because I wanted to show that you don't have to necessarily navigate or change directory down, and only one folder increment. Instead, what I could do is, I could say CD, I know I want to go into loop dot engineer, I press tab to complete, and then I know I want to go into the desktop. So I start typing desktop, right type tab, and now I can get there that quickly. Doing the same thing on this, on the windows machine, typing in loopbrews-tab,
2:11:24
and then I'm going to go into desktop, typing tab, boom, get into it that quick.
2:11:31
So let's crank things up a notch, and we're gonna take these commands now to the next level. In order to do that, we need to actually break down the syntax of how a bash command works. So in this code block, I have a representation of how a typical bash command is formatted. You have the command first, and that is always required. And then you have two other optional parameters after that. You have the first option one, which is in this bracket, you have a flag or option.
2:12:04
And then finally, you have an optional argument of, well, an argument. It's a pretty simple format once you understand the basics of it, and that the command is the program or tool you want to run, the flag or the option is how you want to modify it. And the argument is the target that the command should act on. So let's show a few examples of what I mean by this to break that concept down. And for the remainder of this, we're gonna be just inside
2:12:31
of Windows with this terminal, because the command's gonna be all the same throughout this, and I think you got the idea. Anyway, we saw previously of LS. And this LS command lists all the contents of the directories. Well, we can add a flag to this. Specifically, there's some files in here that may be hidden. So here I'm gonna bring this up to the top of the screen by pressing clear, and type again LS. And then I'm gonna give the flag, I'm gonna do two dashes and then type in all.
2:13:02
And now, looking at this one, there's a heck of a lot of different folders in here and files in here that we didn't see before. Specifically, these dot folders and files. They're automatically because they have the dot at the beginning, they're hidden. So that all flag helps get into it. Now, clearing this, I do wanna show you don't necessarily need to always type out a flag completely. Like previously I did dash dash all. Instead, what I can do for simplicity is actually dash A,
2:13:34
and it still means the same thing. And that is still gonna show all of those different files. At the end of this, I'm gonna show you how you can find out what different flags are available to commands. So that was an example of how we can use a command with a flag. Let's go now into a command with an argument. Well, our change directory command out of CD is the command itself and the argument is where we want to go. So if we're gonna go into the desktop,
2:14:02
that's what the argument is for this. So now, running this, we're in the desktop. Similarly, if I do the CD and then dot dot, that's gonna move us back up into that home directory up one. Also say for some reason that you just go crazy and then all of a sudden you're in a place if you don't know how to get back to your home directory, all you need to do is do CD and then give it the argument of tilde and then you're navigated back into that home directory.
2:14:30
Now that we understand that command flag argument structure, we're gonna go into actually building out how you can work with files and folders, which is really common for that engineers have to do in the terminal. And frankly, anybody that's interacting with the terminal. We're gonna go through these different types of commands in order to make different directories, files, edit them, remove them and be able to view them. Now for this, I want to show this
2:14:56
like as it's happening in the file explore. So what we're gonna do and what I recommend the same to you is I would have one window to the left with your terminal and then one window to the right with the folder location you're navigated into. For me, I went into the C drive, under users, under Luke Barouse, which I know where that's where we're going and then we're gonna go into the desktop. So right now I'm navigated in the desktop in my file, file explore.
2:15:23
I need to do the same thing in here. So I'm going to change my directory into my desktop. And it says inside of here, hey, this folder's empty. Let's just double detect in the terminal. Yep, looks like it's empty. Well, let's see if it's actually empty of all files. So for that dot and double dot, as we learned earlier, the double dots mean the parent directory or one folder up. We use things like cd dot dot to navigate up a directory. Now the single dot is a new concept,
2:15:52
but that just means the current directory. Anyway, they're both just listed here as they represent where you are and what folder is above it and how you can access it via the command line. Okay, it looks like it does have some dot dot files inside of there on a big deal. So the first thing we're gonna do is we're gonna make a folder and that's what's called a directory. So we want to make a directory that's mkdir. That is the command. So we now don't need to give you an argument
2:16:20
of what we want to call this folder. We're gonna name it something real original. We're gonna call it folder one. Now I can see inside my desktop, I have folder one. Now that's for making folders. What happens if I wanted to make a file? Well, in that case, I would use the touch command and I would give it the argument of the file I want to create. So I want to create something called maybe file one dot TXT. Go ahead and run this. We now have file one dot TXT.
2:16:51
And really this could be any type of file. So it could be file two dot JSON. And it's gonna create a JSON file. Now let's say I wanted to create a file inside of my folder one. Right now if I navigate into it, this folder is empty. I could change directory into that folder one and then make that file or I'm gonna navigate back up and clear out these contents. Or instead what I'm gonna do is I'm going to create the file by specifying I want to create it in folder one.
2:17:25
And I'll name this file three dot TXT. We'll go ahead and run this. We can now check inside of it. Well, we can actually see Windows makes it pretty visually appealing that you can see there's files and that's cool. And I see I have file three dot TXT. Now let's say I want to remove files. In that case I would use the RM command for remove and I could specify hey, I want to move that file to JSON file that we created in the desktop. And it disappears.
2:17:54
Now removing a directory is a little bit more complicated. Unfortunately, so they do have a command of remove directory or RMDIR and I can specify hey, I want to remove folder one from this directory running this. It says hey, it's failed to remove folder one direct. The directory is not empty. Now I could go in and remove that file in folder one, which I'll do in this case. And then from there actually go in and remove directory of folder one.
2:18:24
But there's gonna be cases where you have multiple files in a folder and so it would be a very nuisance to have to try and delete all those files before you delete the folder itself. So I'm recreating that directory for folder one and I'm using touch to make that file three dot TXT inside of folder one. We'll go ahead and clear this out. So we cover, we can't use remove directory because there's contents in it. What happens if we try to use just remove folder one?
2:18:56
It's gonna say hey, you cannot remove folder one. It's a directory and we know we had remove directory to try to remove it. Anyway, we can instead provide a flag to this of dash R or dash dash recursive. We're just gonna go with dash R for Mac friends 'cause I think only the Mac the dash R is gonna work. And what's the saying is it's gonna recursively remove all of the file contents of folder one and also delete the folder and allow us to delete this. So let's go ahead and try.
2:19:29
Okay, and it actually deletes it. But even though remove says it can't remove directories 'cause it's directory, it can just gotta give it the right flags. So now let's get into editing the contents of a file. And right now file dot one dot TXT, remember we just created it. I'm opening it up inside of notepad just to show that, hey, nothing exists inside of it right now. I'm gonna go ahead and close it. Anyway, yes, we can go into notepad to add contents to it,
2:19:57
but we may need to programmatically add contents to a file via the terminal. So we need to learn how to do that. Well, in order to do this with commands, we need to actually learn a couple of commands. And the first one is echo. Echo is a command that helps you output text. What do I mean by that? So I'm gonna type in echo and then in parentheses, type in data nerd. So it knows that's the text I wanted to do. Okay, whatever I press enter on this, it echoes data nerd to the screen.
2:20:27
I can't just type in something like parentheses data nerd. It doesn't let you do that. There's no text output that echo allows us to output this text. Now, echo is not limited to just text strings. You could also do something like a number and I'll echo it to the screen. Or something that we haven't really covered yet is variables such as we have a variable for our home path and you could echo that to the screen. Variables are designated by that number sign at the beginning of it.
2:20:56
And by number sign, I mean, the currency symbol at the front of the dollar sign. Anyway, why did we learn this all, right? We wanna send stuff into file one dot text. And so we can do this by saying, hey, wanna echo in something like maybe data nerd and I'm gonna provide it this greater than symbol, which is the redirection operator and we're gonna put this into file one dot TXT. Now, I'm lazy, right? I don't wanna open this up and actually I have to look into here
2:21:24
to see that there's data nerd inside of there. Instead, how can I easily see what I did? Well, I can use cat to concatenate and display the file contents of file one dot TXT. And I can see it has data nerd. Now, let's say I wanted to run that again. I'm actually going to press the up arrow and when I press the up arrow, it navigates through the last command. So you can use the up arrow to navigate to previous commands and the down arrows to navigate back.
2:21:55
Anyway, let's say I wanted to change this and I wanted to insert something else such as data analyst. We'll go ahead and run this. Now, whenever I look into this file of file one dot TXT, the contents got overwritten with data analyst. So if I wanted to, let's navigate up back into data nerd. If I wanted to also append data nerd to this list, I have to put two greater than symbols. This is the append operator. And now when I press up to go to cat the file,
2:22:32
I see I have data analyst and data nerd in that file. Now, real quick just to go back to how bash commands work, we've covered commands and we've covered how to use flags with commands and then we've also covered how to use commands, the flag and even an argument. But we haven't covered options yet. Options require on top of the designation of like dash dash color. It requires some sort of value to go with it. Also, I don't know why I'm using my hand so much. So let's demo this.
2:23:06
We can actually use on top of just cat. Cat's more popular, but you can also use something like head to show the beginning of a file. So I could use head file one dot TXT, or files only two rows long, so it can show the top of this. Anyway, let's say I wanted to only show maybe the second line of this. Well, I can specify head, give it the option of N, which shows is how many number of lines to show. And I could specify two in this case.
2:23:42
And we'll go file one dot TXT and on mistype file one dot TXT. We have to actually add a one in and displace it all. Now, we could also just show one row in this. I could change it to this to one, run this as well. We get just that analyst. Anyway, I wanted to run this option just to show you options. Now you may be like, Lou, that's crazy. Anytime I want to edit a file, you're telling me I have to write this, echo what I want to put into it, and then append it into the file.
2:24:09
And then hopefully I get it in there right whenever I cat that I'm gonna type that command fully. Just a shortcut, if ever your terminal gets locked up, you can use control C and it will automatically exit out of whatever operation it's doing and you'll start fresh. Anyway, if I, you don't want to go through and just type something like cat file one dot TXT and do that every single time to make sure that you've inputted in right.
2:24:34
Instead, the terminal has an editor you can use with this. It actually has a couple. One of them is Vim and we can go into file one dot TXT. Vim though is a little bit harder to use. And so we're not gonna go into it. If you ever get yourself into Vim, you're gonna need to press shift and get the colon going. And then from there, you can see a colon in the bottom left right here. You would need to say WQ to write it and quit it. Anyway, Vim's really hard as a text editor to use.
2:25:06
We're not gonna go into it in this course. What I prefer is the nano text editor. We're gonna use this on file one dot TXT. Whenever I go into this one, it gives me all the options at the bottom. So if I wanted to add something else like data scientist, I can. And then now I want to write it out. So I'm going to do control O. That's basically saving it. Press enter. We wrote these four lines. Now I want to exit it. So I'm gonna do control X.
2:25:39
And now when I take a look at the file of file one dot TXT, I can see we have all of those things. Nano in general is just a good text editor to know everything's at the bottom for cutting, pasting and moving things around. Really a fan of it. So we're gonna be using it predominantly in this course, anytime we need to edit stuff. And we go ahead and exit. So just as a recap, we covered how to make not only directories, but also files and then also how to delete
2:26:05
associated files and directories. We also got into how to do text editing from the command line and get an into text editors themselves. Now you may be like loop. How the heck do you know all these different commands and what type of flags and whatnot you can use with it? Well, my number one source that I'm going to is either something like chat GPT or Gemini. And it's really quick at actually going through and helping you out and learning how to use the terminal,
2:26:33
what commands to not to use and what not to use. Now you may get so efficient with running different commands in the terminal. You want to investigate them because you don't want to hop over to chat GPT. Well, here's a tip. Unfortunately, they're both specific to you, whether you're on Mac or on a Windows. On a Mac, you're gonna run the man command, which is short for manual. And if I wanted to investigate something like LS, I provide that as the argument.
2:26:59
And then inside of here, I can scroll around to see what the name of this is. LS is list directory contents. It gives a description and then scrolling down, I can see all the different flags in here. We can even see that dash A command where that includes directory entries whose names begin with a dot, like we learned, along with a whole host of other options you can look at. Now in order to exit out of here, I can see there's a colon down at the bottom.
2:27:24
I would just press Q and it exits out of it. Now on a Windows machine, if I try to run man LS, that's not gonna work. Instead, what I'll run is the command itself of LS and then provide the flag of help. This provides all the things we have. That's not really formatted that great because of how zoomed in for this. But if I zoomed out, you can see it actually does look cleaned up, just the way I have it on split screen. Anyway, I do wanna show I can't run LS-help on a Mac.
2:27:53
It's not recognized, it doesn't work. In either case, they provide the same results and you'll be able to go through this and actually investigate further what type of flags and what type of arguments you could provide to a command. All right, you now have some practice problems to go through and get more familiar with operating inside of a terminal. And it's really a skill that's gonna pay dividends as a data engineer. So I don't recommend skimping on that.
2:28:20
In the next lesson, now that we have the base zone terminals, we're gonna get into setting up local duck DB inside your own computer to run from. All right, with that, I'll see you in the next one. In this lesson, we're gonna be going through how to install both for those Windows users and Mac users how to install duck DB. And with this, we'll be walking through all the commands with it and by the end of it, we'll be connecting to our attached databases in mother duck.
2:28:52
But you know what's this whole thing? Because we need to install the correct version of duck DB in order to connect to mother duck. Now, before we jump into the install, we need to understand how duck DB locally is connecting to mother duck. And what the heck each of those two is, just in case you forgot, duck DB is the database engine that's running behind the scene whenever you're doing any SQL queries. So previously, we were running in mother duck
2:29:23
and that has duck DB databases in there and it manages it through mother duck. And in this lesson, we're gonna be actually installing locally on your computer duck DB. Now, just to show this simply, we've been accessing mother duck where it has our different databases on there, specifically our data jobs database. And we've been accessing it in your computer via your browser. We've been just using it via data jobs connection. Now, with this setup with mother duck,
2:29:56
this has allowed us and all the other students in the course. or to connect to the same exact database. Pretty dang cool. But now we're gonna be shifting gears and instead of using the browser, we're gonna be using the terminal 'cause that's very popular data engineers in order to automate your different scripts. And in order to do this with the terminal, we need to install DuckDB locally on your computer. And this will allow us also to create local databases
2:30:24
as necessary that we can then operate on and do whatever we want to and not affect anything in Mother Duck. This is very common for data engineers for you to work on a database locally and then once it's fine tuned, you could push it up to your data warehouse like Mother Duck. And this is automated not only through the terminal, but specifically using different SQL files and something like code editor like VS code, which we're gonna install in the next lesson.
2:30:50
And we'll be using VS code to interact directly with the terminal. And this, as I mentioned, is a real life production setup. Because now you would work like you would as a data engineer with DuckDB locally on computer, connecting also to your data warehouse and Mother Duck. And then other people on your team, whether data engineers, analysts or scientists, could connect to those databases in Mother Duck via their preferred tool. So we're now gonna be getting into installing DuckDB.
2:31:23
And for this, we're gonna be, at least for one of the versions, we're gonna be installing it here from this location. So I put a URL on the screen, that thing's heck along. I would actually instead just Google Mother DuckDB CLI and you should get directed to this page. Now we need to pay special attention to this note up here in that it says Mother Duck currently supports DuckDB 1.4.2 in the US and then 1.2 to 1.4.2 in Europe. And then it closes by saying, Mother Duck supports client
2:31:57
version 1.4.1 through 1.4.2. You're like, "Yeah, what does that even matter?" Well, if I go over to DuckDB.org, they're having, or they're the recent version they have is 1.4.3, basically one version ahead. And so this version is not supported by Mother Duck. And what do I mean by that? Well, as I found out the hard way, whenever I installed the wrong version of DuckDB of 1.4.3 when I filmed this lesson the first time and now I'm having to re-record it right now,
2:32:29
is that when I went to connect DuckDB locally to Mother Duck, I got this error message and it says, "Hey, you're currently installed version of DuckDB 1.4.3 "is not supported for connecting to Mother Duck." And unfortunately, just changing the version of DuckDB is not super easy. So that's why I wanna make sure you install it correctly the first time. Now, this type of issue of versions is pretty common in the data engineering world in that you have another company
2:33:02
that builds their service on top of a certain product, such as in this case, Mother Duck builds their product on top of DuckDB. And then DuckDB, because it updates, Mother Duck needs to then go through some different verifications and testing to verify that DuckDB actually works with or is acceptable and safe enough to work with Mother Duck before implementing that new version. So complete standard and something you get used to in the data engineering world.
2:33:28
One last thing on this note, you're gonna be watching this at a way different time than as I'm filming this in December of 2025. So these versions may be completely different. Heck, you may even have it to wear the version on DuckDB is actually supported by that of Mother Duck. Anyway, because of that, the safest and easiest option is just to always come to this if you want DuckDB in order to install it. Now for both Windows and Mac users,
2:33:55
I'm gonna walk you through this install separately. And for Windows, it's as simple as installing a .exe file or binary file and installing it. And then for Mac OS, it's as simple as running this curl command in your script. Although there is actually another step you have to do after that to make sure it gets added to your path. We'll get to it when we get to that. Anyway, that will install the correct version. So if you just want a simple option,
2:34:18
you can just stick with that that I'm gonna give you. Now I mentioned this all to the Mother Duck team about all these different issues. And they had a pro option that actually I've implemented in managing and installing DuckDB. And so if you don't wanna do or you can watch that first portion that I do on the easier option, this will be the pro option. And with this, this right here is via this GitHub repository. This is DuckMan and it manages your DuckDB CLI with ease.
2:34:48
Scrolling down to the documentation, what it is is this is a version manager. So unlike the easy option where we just install a certain version and then that's it, you can't really update it that easy if you wanna change to a different version. Well with this tool, you can. So we're gonna walk through this on both Mac machines and also window machines is the pro option. And once it's installed, it makes it super simple to then go through and install DuckDB for a specific version,
2:35:17
setting it as your default, and then just running with DuckDB with that version. So for both Mac users and window users, whenever I go through this install process, I recommend you just watch the easy option and then the pro option and then whatever you feel your comfort level is with, go with that one. So let's get into installing DuckDB via the mother duck option here. We're gonna go and this is the easy option. It has that we want to download this binary here
2:35:47
and then we're going to extract out this sip once it's installed. Well, I did pretty quickly 'cause it's pretty small. And from there, I'm gonna just double click on DuckDB to launch the application. And what this shows is that this is the DuckDB executable. This actually ran it inside of, it looks like command prompt. But this shows that DuckDB is installed, if you will. But if I were to open up a new tab, such as where GitBash is, GitBash is,
2:36:14
and I tried to use the command of DuckDB, it's gonna say, hey, that command's not found. So yeah, anytime that I'd want to use DuckDB, I'd have to navigate to this file location and then click on DuckDB in order to use it. Sort of a pain in the butt. So right now, this application executable's inside of our downloads folder and inside of this, basically zip file for it. Not a great location for it to be because sometimes I like to clear out my downloads folder.
2:36:44
So what I'm gonna do is this. I opened up another file explorer, and we're gonna move this specifically. I'm gonna navigate into my C file or procedure file and then into program files. This is where it would commonly be, and it would be in a DuckDB folder. So I'm creating a new folder. And so what that's gonna do, I wanna confirm this. I do, 'cause I do wanna create one here, and we're gonna call this DuckDB. Go ahead and press enter to create it, and then go into that file,
2:37:14
and I'm gonna drag the DuckDB application into here. So that's gonna be again, if I wanna do that, yeah, I wanna do that. Now, I do wanna show something that you could technically now run DuckDB from the terminal. What I'm gonna do is I'm gonna go ahead and I'm going to right click it, and I'm gonna say, "Hey, copy as path." And I'm gonna go ahead and paste it in. And so now, I'm gonna go ahead and press enter. Boom, it can run. So you could, technically, any time you run it
2:37:44
to run DuckDB, you could just enter that in and run it. But that's a lot of work. We don't wanna do that. We wanna be able to use the command just DuckDB, and then it run. So how do we do this? Well, we need to update our environment variables, environments or variables inside of our terminal window right here. So I'm gonna go in here and type in var for variables, and it says, "Hey, I wanna edit the system variables, "environment system variables, I do."
2:38:11
In this pop-up, I'm gonna select environment variables, and then specifically, we want to update our path variable. And this is, every time you run a command, it looks through this path. Actually, I'll just go into it and I'll click it. It goes through and looks in these locations for different executables that you may need to run. Specifically, like our DuckDB one. So we'll add it by doing new. We need to get the actual location of this,
2:38:39
not necessarily this one, but this file path up here. I'll right-click it, and I'll say, "Hey, copy address as text," click new again, and then I'm gonna paste that in by pressing Ctrl V. And then from there, click okay. Finally, click okay. Close out of this, and close out of this. Now, if I try to run DuckDB, not gonna work just yet, I have to actually restart the terminal altogether. So I'm gonna close out of this. And then I'm going to restart it up,
2:39:06
and now whenever I run DuckDB, it automatically goes to DuckDB. This is supposed to be the easy option. I don't know. (laughs) (gentle music) For the pro version option of installing DuckDB and able to manage the different versions, we're actually not gonna install this DuckMan. I tried to install it just now using Windows and ran into a whole bunch of issues. I did get it to work on Mac, so Mac user is still gonna use this. Anyway, we're gonna use an option for the pro option
2:39:40
of WinGet. What the heck is WinGet? Well, it's a command line tool enabling users to discover, install, upgrade, remove, and configure applications on Windows 10, 11, and Windows Server 2025. Basically, it's a command line interface package management tool. It's short for Windows get. Now, you can see if you have installed, and you should for Windows 10 or 11 by typing in WinGet/version. And it will list the version. For those that don't have WinGet installed
2:40:10
for an older Windows version, I'm not supporting that. Feel free to use chat GBT to figure out how to get installed. So now we can use WinGet to install a specific version of DuckDB. So I'm gonna type in WinGet, install DuckDB.CLR command line interface, and then we're gonna use the option of version. And for this, we need to make sure that we're specifying the version that's supported by you currently right now. And for me, as time filming this, it's 1.4.2.
2:40:40
And so I'll just type that in here, 1.4.2, run enter, and for the pro option, this is actually a lot easier. It says, hey, the command line alias added DuckDB, running DuckDBD, and then checking the version, we can see that 1.4.2 is installed. So, I don't know, maybe the pro version is actually easier than the original version. Now, the only issue with this is, or not really even an issue, if you want to install a different version of DuckDB,
2:41:13
you need to first go through and uninstall this version. By running WinGet, uninstall DuckDB, and then the CLI. It'll successfully uninstall it. Now running DuckDB, I can see that it's no longer in my path. And then from there, running the install on the newer version, I'm going to keep it 1.4.2 for me. We're now going to get into installing DuckDB from other Duck, for the correct version on a Mac. And this will be doing the easy option. I'll put easy option, because actually,
2:41:48
I think the pro option's a little bit easier, but it's a little bit more complicated with the technologies we have to use with it. Anyway, let's just start with the easy version first. We're going to be installing it via this URL that's on the screen here. And it may be easier just to input in mother DuckDB, CLI, and Google to get to this. Anyway, in macOS, it says to do this, we need to install with Bash using this script. So I'm going to go ahead and copy this.
2:42:14
It's a matter of terminal. I'm going to go ahead and paste this in, and then run this bad boy. It looks like it's going through a setup. And it's always important to read documentation. So it's said that it's successfully installed DuckDB, but we need to append the following path to our shell profile, specifically this right here. And I'm going to show you what that actually means. Anyway, we can also see that it's installed the correct version, but it could not find a shell RC file
2:42:41
to add the mother Duck token to exiting out. Anyway, the key thing here is we need to understand that if I wanted to launch DuckDB now, I'd have to type out this full command right here. So I'm going to go ahead and copy this with command C, paste this in and press enter. Okay, I'm now in DuckDB. I could run something like select 42 as answer, bam. But running.quit, that's not. I don't want to have to pressing the up arrow. I don't want to have to type out this entire thing
2:43:12
every single time. I need to add this location of DuckDB to my path variable. So that way I can just put into here DuckDB and run it. And it won't give me this Zish command or Z shell command not found. So how do we do this? You run this command, but we need to specifically add it to our shell profile. I'm actually going to just show you real quick. So we can actually run this. I'm going to go command C and copy this file and then paste it down underneath here.
2:43:43
All this is going to do is exports, I think it's very similar to echo and that it's going to add this to our path. But it only does it for this terminal right here. So now whenever I run DuckDB, I can get into it and I'm going to do dot quit. We'll cover dot commands in a little bit just to get out of this. But if I were to close and exit this terminal and then try to run DuckDB, it's not going to find the command. I still have to go through that full on command this
2:44:15
in order to get into DuckDB 'cause it resets every time. So we can fix this by adding this to our path or script that manages or runs every time we start up a terminal. So we need to add this audition to our path variable that I just showed to a certain script in order that whenever you start up your terminal, it automatically gets added to your path and you can just run DuckDB. Anyway, it's going to depend on what shell that you're in in your terminal on your Mac.
2:44:51
As I discussed in one of the earlier videos, newer Macs are using Z-Shell and you can see that up at the top of the terminal. If you see Bash, then you're using the shell of Bash. Anyway, for whether you're using Z-Shell or Bash, there's two type of files and you either have the Z-Profile, a dot Z-Profile file, it's a hidden one or a dot Z-S-H-R-C. And our C stands for run configuration. Similarly, Bash does the same name and convention instead of using Z, it uses Bash.
2:45:22
But concepts are still the same. Now, the dot Z-Profile only runs once and that's at login, whereas the dot Z-Shell RC file runs every new session. Bash profile and Bash RC follow the same convention. Anyway, inside of here, I'm going to see if I have those files because they should be located in your home directory. So I'm going to do LS-All. Now, they're not in there, so I'm going to go ahead and add both of them. There's going to be blank files, it's okay.
2:45:52
I recommend you do this regardless if you're going to do this install method or not, you should have these files. Making sure I'm in the home directory, that told of directory, I'm going to do touch to create the file and then dot Z-Profile. And then I'm also going to do touch of dot Z-Shell RC or run configuration. Now, when I run LSA, I can see I have the Z-Profile and the Z-Shell RC file. So now, pressing up, we need to add this right here, this previous command that we ran.
2:46:23
I'm going to go ahead and just copy it, command C and then exit out of it. And then from there, we're going to use the nano editor on dot Z-S-H-R-C, I'm going to tab over, select it, press enter and I want to paste it inside of here. Remember that Z-Shell RC or run configuration is going to run every new terminal session, this is where I want to add it. All right, I want to save it. So I'm going to do control O and press enter. And then I want to exit out of it.
2:46:50
And so I'm going to press control X. Now, DuckDB is not going to work right now, right? Because we need to reset this terminal session. So now starting a new window, I'm going to run DuckDB and goes into DuckDB, just fine. So now getting into the pro version, which I actually feel it's going to be a little bit simpler. But with this, we need to go in and it says hey, for this, the scrolling down to the actual documentation on it. It says before proceeding with the installation
2:47:22
is advisable to uninstall, remove any existing DuckDB CLI. So if you installed this and you want to do this version, make sure that DuckDB is uninstalled. Specifically, if I do a L-S-All, I can see that the DuckDB file or the DuckDB files in there. So I'm going to do a remove. And then I'm going to do a flag of R and F to make sure that it deletes it recursively. And then the F portion to force it. So recursively and force. And then I'll put in there.DuckDB for it to delete.
2:47:52
Now, whenever I list it all out, DuckDB is not in there. If I try to run DuckDB, it's going to be, well, come in, I found, but I need to go into that file that we added in previously. So I'm going to nano into Z shell RC file. And then I'm going to just control K this file to cut this out, control O to save it. And I want to save it to that name and then control X to get out of there. I guess I didn't save it last time, so I'll put yes. Go ahead and put Enter.
2:48:24
All right, so now it's completely removed. We can go through this install process. For good measure, I'm just going to close out of this terminal and start a new terminal. Okay, in here, it says this is the command that we need to run inside of our terminal. Go go ahead and copy it. And then in that new terminal that I started up, I'm going to paste that right into here. And it says, do you want Duckman set up to autocomplete and add the path for you? Yes, I do.
2:48:48
I don't want to have to go through that again. Now, it's important that you, before you ran this, that you had run that touch.zhrc and also that touch.z profile. So just looking at that.zhrc file, we can see that inside of it, it added in this portion right here to add Duckman to our path. But we run Duckman right now. It's going to say not found because we need to restart our terminal. Remember, it's every single session that it runs that. Now, whenever I run Duckman.zhrc version,
2:49:24
it tells me what the version is. All right, so now scrolling on down to usage. From here, we need to tell it which version or which version of DuckDB to install. So you need to go into the mother duck documentation for this DuckDB CLI install and actually read this note right here on what's applicable to you right now. 'Cause as I'm filming this right now, I need to do 1.4.2, maybe different for you. So I'm going to do Duckman install 1.4.2 and didn't get any issues, okay.
2:49:57
Now, if I go to run DuckDB, not going to work, the command is not found. And that's because part of the documentation, we need to set a version of DuckDB as the default one to use. So we're going to run this command of Duckman default. And then we want to use 1.4.2, okay. Now, whenever I run DuckDB, bam, get into a session and it's using 1.4.2. All right, looking good. I can exit out of this by doing dot quit.
2:50:30
All right, so let's just go through a quick, simple demo, showing you how to get into DuckDB and also run a query and then exit out of it. For all this, I'm going to be staying inside of my Mac machine, but Windows users are going to do exactly the same thing. Anyway, we're going to run DuckDB. And this opens up, one, it shows, it shows what version we are. And it says, hey, we're connected to a transient in memory database. Basically, anything we put into this database
2:50:59
that we create here, as soon as we close out of the session, it's going to be gone. Anyway, if I wanted to write a SQL command, such as this of select 42 as answer, I can, but notice this, if I press enter, nothing's happening here, it's like, what's going on here? I can actually press backspace and navigate through, add in a colon, 'cause you need a colon at the end of a statement and it will actually run. Now notice up here, it says enter.help for usage hints.
2:51:24
Dot help is a dot command. And there's a whole bunch of dot commands that you can actually run with this. Specifically, I could run things like dot databases to see the name and files of each attached database. Dot tables to list the name of tables or even something like dot schema to show the create statements matching patterns for how a certain table is created. Anyway, the main one I think you should know about is this one up here of dot quit and that exits the program.
2:51:54
They also have this one up here of dot exit and that exits this program with a return code. Anyway, for our sakes, I mean, we kind of do something like dot tables. Once again, there's nothing in here, so nothing's gonna be shown, even if I did dot databases, nothing's gonna be shown. So we're gonna just exit out of here by just doing dot quit. Now to be clear, those dot commands are specific to the DuckDB CLI. So I can't necessarily inside of here run something like dot databases
2:52:26
and expect the databases to be listed just not gonna work. So actually recommend in what actual command you should run if you want to see that. So running just DuckDB by itself is something I rarely do and use a transit in memory database. I don't really necessarily do that alone. I'm gonna quit out of this. Instead, I like to actually use a database or either attach to one or create one locally that I can work with. How the heck do we do that?
2:52:57
Well, let's inspect that DuckDB CLI command by running dash help. Now this allows us to see where you should be able to run help with most CLI commands and see what they do. And specifically up at the top, it says the usage. It has the command of DuckDB, any options. These are the options down here below and they explain it all. And then it has file name and then finally SQL. Notice like these square brackets right here that means that parameter is optional.
2:53:30
I'm not sure why their file name doesn't have it also 'cause technically a file name is optional. Anyway, with file name, it's the name of a DuckDB database. A new DuckDB database is created if the file does not previously exist. All right, we'll do that in a second, but I wanna show this real quick, these options to show how you could also run this with DuckDB. Specifically, one of the ones that we're gonna be running a lot to eventually is this one right here
2:53:57
where we do a dash C and then you input a command. And this runs the command and then exits. So let's go ahead and try this out. So I can run DuckDB and do dash C. And then inside of quotes, put something like select 42 as answer, put a semicolon at the end and double quote. Run this and then bam, it runs it and exits. So you don't actually even have to go into DuckDB. Now, let's actually create that database file. Remember, we can list DuckDB and then the file name
2:54:29
of the database to create it if it doesn't previously exist. For this though, it's gonna create it wherever we say to create it. So right now, we're in the home directory, not necessarily the best one. I'm gonna navigate into my desktop. So we can also see it on your desktop when we create it. And I'm gonna call out DuckDB. And then for the database, we're just gonna let jobs.duckdb. So we're gonna make a fictitious job.set. Okay, I'm gonna go ahead and run this.
2:54:55
And nothing's appearing on my desktop just yet. It's gonna take a little bit of time before it does appear. And manly, we gotta put something in it if we wanna actually even create the database. 'Cause right now, if I run that command of dot tables, there's no tables in there. Oh, it did look like it did create it whenever I ran that command. And now it's right there off to the side. Anyway, I think you're smart enough at this point that we're gonna actually use some more advanced SQL
2:55:21
that we're covering in an upcoming lesson on DDL and DML in order to create a table and insert data into it. So let's just try this out so you can keep up. We're gonna use the command create table. And we're gonna give that name of the table jobs. We're gonna then put into parentheses, I'll enter down. We're gonna put in the column names for this. We need to create that first. So we're gonna do with an ID column and that's gonna be of the data type integer.
2:55:50
And then also a job, which is like a text value. So there's data analyst, data scientist, whatnot. This is gonna be of the data type of our car. Don't worry, data types, I have a whole lesson on that. So just press the I believe button right now. And from there, I'm going to close out that parentheses and put a semicolon at the end. So let's create this table. And now when I run dot tables, and this shows that we do have a jobs table, I can actually go in and query that table.
2:56:21
I can just do select star from jobs, semicolon. And I see that we have an ID and job column. And notice it says what the data types are for each of these. And there's zero rows of data into it 'cause we haven't inserted into it. Now, if you noticed, I had this DuckDB file dot WAL, which is right ahead log. And this is a safety mechanism that DuckDB implements in order to make sure that if you ever write any data, it's getting updated appropriately in the final database.
2:56:53
Anyway, don't delete that file anytime when you're in this session right here. You can, if it doesn't delete automatically, delete it afterwards, but don't delete it anytime, you're actually working in a database. All right, so we've created this table. Now we need to insert into it for the jobs table. We need to insert values. We're gonna insert a few values. First, we're gonna do the ID. And then since the next value is a var car or a text value, we need to put it into quotes.
2:57:23
Single or double quotes is fine. I'm gonna do single quotes and put something like data analysts. Put close parentheses and then a comma. And then now we can insert other values as well. In my case, I did two more on data scientists, that engineer. And then finally, I just close this all out with a semicolon. I'm gonna go ahead and run this. And now pressing up, I can scroll through different commands. I'm gonna go to select star from jobs. And I can see all those different values.
2:57:50
We just inserted into the table. Now let's see if these files or these values stay inside this DuckDB file. So I'm gonna actually exit out of this by doing dot exit. And if you notice that right ahead log disappeared during that, so it should have for you as well. And now pressing the up power to go back into run this command of DuckDB jobs dot DuckDB. We're inside of here. And then I press up again. I wanna navigate to our last command
2:58:13
to query that jobs database and our data's still in there. So it's inside of this file, pretty cool. All right, I'm gonna exit out of this. So the DuckDB team recently implemented a feature earlier this year that now the UI that you saw in mother duck can actually be run locally via DuckDB dash UI. Let's try this out. So I'm gonna run DuckDB and then dash UI. And what this is gonna do is pop open a new browser and running on local host. So locally on your computer,
2:58:49
this server is now showing to you the DuckDB UI. So this has the same structure that we previously saw in mother duck and that we have all of our notebooks located here and then any attached databases. Specifically we have an in memory transient database with nothing inside of it. And then they have an option to show me the basics but I'm just gonna go to an empty notebook. Anyway, I do wanna show this with that database that we actually just created how you could investigate that.
2:59:21
So I'm going to go back to the terminal and type dot quit or exit. It's fine either one. And if I navigate back to my browser, it says, hey, connection to DuckDB lost and that's fine. I know what that's what I wanted to do. Anyway, I want to actually not only specify that that UI flag or UI option but also that I want it to connect to the jobs.DuckDB file. So now running this, it pop up into the UI. And instead of having that in memory database, it has our database,
2:59:55
which has those columns of ID and job. I'll go ahead and start a blank. Notebook and inside of here, I can query this now because I see I collected two jobs and I so do select star from jobs I run this query and I see all the values below this query currently is limited in the UI as I can And I actually investigate any of the different columns like we can in Moddeduck anyway One of the cool things about this UI is I can via the UI
3:00:23
Sign into mother duck and we can get all of our notebooks and attach databases So I clicked it just now and now I'm going to go through the login process as a reminder If you installed a version that's not compatible as version of duck to be this not compatible mother duck You're gonna get this that's why we went through that whole rigamarole So make sure that we installed the correct version anyway going through the sign-up process
3:00:47
I'm gonna use Google and now I have all of my notebooks brought in here along with slowly my databases are loading And I now have my data jobs attached to that base along with those other ones as well and the local database So all in one environment. So now we have this somewhat hybrid hybrid approach, right? We have our jobs at DuckDB. That's locally and so whenever I run this this is run Well locally and then if I wanted to I could also do something like where my data jobs and for this
3:01:19
I'm just gonna do a simple query of select count star from data jobs And then running this pressing command-tensor. So me I don't know. I've apparently been working on this course too long I should have done job postings back to the table. I'll accept and run this anyway This is running not locally, but this is running in the cloud So this is a very much a hybrid approach and what needs about this I actually updated this query to just query the job tile short column just for fun
3:01:44
But in this one whenever I actually inspect into the columns I can actually see that same type of format that we'd see up in mother duck All right, so you see why it's cool, but we're gonna get now back into the terminal Because I want to show how you can connect to mother duck programmatically using the terminal So for this I want to get out of having duck DB set up locally before we get in next port
3:02:08
So I'm just gonna just quit out of this and then I can see inside of my web browser. It's closed out So how the heck do we connect to our database programmatically well I'm gonna recommend this right here for the easiest and that's you can connect to mother duck directly from the duck DB DCLI by running the following command and that's duck DB and then then MD colon The parentheses aren't necessary unless they're well
3:02:35
We can add databases on to it and if your database has spaces in it, you want to have quotes Anyway, the MD what I said is mother duck We're gonna specify that and the database we want to connect to so instead of my terminal I'm gonna type in duck DB and then mother duck and we want to connect to I want to connect to the data jobs Database now I'm gonna go ahead and press run and you may be like well You're connected because it doesn't say in transit memory
3:03:00
But if you try to do query to see what are the tables inside of here it's gonna pop this up to go through and Confirm that you've actually or who you say you are and want to actually connect to mother duck It will then go through the authentication process process and says I previously entered in my password I didn't have to do any of that and so it says hey the token successfully retrieved You can display the token and store it as an environment variable to avoid having to log in again
3:03:30
So every time we want to get into this we're gonna have to log in via those Credentials we just went through which is the pain but in the next lesson we will go through how to set up this environment variable We're not gonna do this method. We're gonna do another method anyway We're connected so I can run something like select distinct job title short And that's from the job postings fact table put a semicolon at the end run this bad boy and bam
3:03:56
We get the results back immediately these 10 rows and so this is how these results are gonna be output to us This isn't as friendly as the duck DB UI So if you're ever getting into complex tables and views you may want to shift from the terminal to the UI So now that we're connected to that database I can run that dot help command again Just a reminder we can run things to inspect the tables like we did and we can also think run things like dot databases
3:04:21
To see what databases were in so running dot databases We can see that although I did type in MD and then data jobs We have those other databases as well You just if you wanted to connect to a Table or another database you'd need to actually specify it By saying there's something like sample data and then looking at something like hacker news And we're just doing a select count star from this. Oops. I didn't do a colon
3:04:50
So and that semi colon at the end and silly me. It says hey, did you mean this one? I didn't specify also the schema I'll press up one enter in the schema of hn Now with running this okay 3.87 million rows So I do have access to this other mother duck as well I just like well, I'm gonna quit out of this to show you. I Just like the ability that whenever I connect to duck DB Specify this of data jobs because now whenever I query that job postings fact table or any table
3:05:24
I don't have to specify the database and the scheme every single time It knows what's the main database? I want to point at and actually look at oh looks like I got once again confirm my authenticity I'm all set and to provide the results and that's a great segue into what they're going to be covered in the next lesson of Actually setting up your environment variable to avoid having to log in again again again All right
3:05:48
So you now have some practice problems for those that bought the support of resources and to go through and practice getting familiar With working with a local duck DB database and also trying out a few different dot commands some of which we didn't cover in this video with that. I'll see you in the next lesson where we're going to be getting into Installing our code editor
3:06:10
We're going to be using for this which is VS code and it's going to make it a lot easier than typing SQL directly into the terminal And we'll also be able to keep all our different SQL files for building our project with that. I'll see you in that one
3:06:26
Alright this lesson is going to be all about VS code or Visual Studio code our code editor That we're going to be using for the remainder of the course and to build our project Now we're going to be walking through it all from how to install it how to actually operate in it If you're not familiar with it and then also Setting it up to where you can work with duck DB and mother duck within VS code
3:06:50
And I does a hint it towards in the last lesson actually setting up that environment variable with your token So you can automatically log in to mother duck to connect to our data set But for moving forward with installing VS code you understand why the heck we're actually even installing it well according to the 2025 stack overflow survey of over 25,000 different developers
3:07:15
Visual studio code or VS code was by far the number one choice with three out of four developers using it now personally I use this option right here cursor, which is like sixth on the list and it's basically is a VS code It's a fork of it and it is slightly different in that it's AI enabled and it has an AI coding system But we're not really going to use AI for when we're learning SQL. I don't really think it helps I think it hinders you so I'm not going to recommend that instead
3:07:49
I'm going to recommend in the vanilla option with visual studio code now one quick note on terminology over here I've been calling it a code editor and it's under here under dev ID ease or developer ID ease Well, there's actually three different ways you can run code and also in our case run SQL code That's a code editor IDE or a database client as I mentioned VS code is a code editor And its main job is to well edit code or edit text files or SQL files
3:08:20
It's super lightweight as far as interacting directly with the database You'd have to do it either via extensions or like we'll do through the terminal Debugging is limited and you can use many languages for ID ease a popular choice right now is data grip And I've tried in the past pretty good, but this is like a full development environment It's heavy duty. It has built in everything. You can navigate inside of a database natively all inside of here
3:08:48
Super strong debugging, but it only supports one language in that in this case for data grip SQL The last option is a database client and this allows you to also run SQL similar to an IDE And it's around a medium weight. It has native database browsing built into it and similar to an IDE It only normally supports one language and like a database client's well It only does support one language and that's of SQL
3:09:15
So I do want you to be aware that there are three different options as you progress after this course You may want to investigate these other ones of data grip and debiver as far as investigating ID ease and database clients Personally for me, I'm sticking with a code editor and I'm using cursor Which is a fork of VS code basically as VS code with an AI agent in it So let's get into installing it if you're on a Windows machine
3:09:42
You're gonna just go into the Microsoft store and then install visual studio code from there Once installed you can go then and go search for it should pop up as it's appearing I'm gonna go ahead and pin this to the taskbar and then open this bad boy up and we're there Now let's install for Mac navigate over to code.visualstudio.com and inside of here We're gonna go ahead and download it. We'll need a select that we're on a Mac operating system
3:10:09
It put it inside my downloads folder since it downloaded it and I'm gonna drag it over and put into the applications folder Now we'll go ahead and click on this bad boy and pop it up All right, we're gonna continue now But we're just gonna stay in this Mac environment because the operation of the app is the same whether on a macro Windows First up you can zoom in or zoom out of this editor by pressing command positive or command negative
3:10:32
Control if you're on a Windows. I'm gonna zoom in a little bit anyway Let's go through a quick tour over here on the left-hand side is our activity bar We can see things like it or explore whenever we open a folder. We'll see the folder contents here We can search our files inside of here. We can manage source control Which we're gonna do in an upcoming lesson when we go over git and github you can do things like debugging And then finally you can add special extensions to basically
3:11:03
Upgrade your code editor be more closer to an IDE like installing a Python interpreter to actually be able to run Python Down in here in the bottom is how you manage your accounts And then you also have a settings option as well I'm gonna close out of this and minimize this I was mentioned. This is the activity bar and then when it pops open That's the sidebar and then inside of here is where we're gonna edit it and over on the right hand side is our chat
3:11:28
Slash AI. I'm gonna go ahead and just minimize that for the time being. So right now We have nothing inside of a VS code I'm going to show you my current VS code setup for this project and walk you through how I can do a sequel Before we actually do this. So this portion is just a demo. You're not doing any of this I'm just showing you what we're gonna be able to do by the end of this lesson. All right So here we are in my VS code currently. I have a heck of a lot more files
3:11:56
I opened this file explorer and inside my folder of sequel data engineering course and I'm building for this course I built this course and the projects under this one underscore basics folder and I have the lessons in here Along with the projects. Now, what's cool about this is like, you know, we're gonna create two projects in here So here's our first one do an EDA and the different sequel files with it
3:12:21
And so the sequel file appears in the text editor off to the right hand side does all the syntax highlighting and has the sequel query right here But how the heck did we run this so I'm gonna open up a terminal by pressing control tilde And then I want to connect to our data jobs database. So I'm gonna type in duck DB and then MD data jobs We're now connected and so what's neat about this, right? I have a key binding shortcut
3:12:48
So I'm gonna highlight the sequel that I want to put into the terminal and what we're gonna set for you all as well I'm gonna press shift enter and it inserted it in I'm gonna actually make this a little bigger So we can see it so it inserted in this sequel query and then I have the results below it So anytime I want to enter or run any different sequel files All I got to do is just highlight them run shift enter and it runs it in the terminal right below for me to go ahead and inspect it
3:13:17
pretty cool So I'm back in my other Mac environment where I haven't set up any project folder yet or anything like that Let's walk through Setting up the project folder or the folder We're gonna be making the rest of the contents of this course in and then setting up those key bindings so that we can actually Excuse sequel inside of here in the terminal. So I'm gonna open this file explorer and click open folder
3:13:45
I'm just gonna create it here in my document of a new folder and call it sequel underscore data underscore engineering Underscore projects will be very specific of what this is because we're gonna put this on github eventually and share it So you want recruiters and other people to know what this is all about So we're gonna create it and then now that the create a folder is created I'm gonna open it and it asks if I trust myself well most days I do so I trust this
3:14:12
It has this walkthrough and we're once again close out of that and minimize this AI window off to the side Anyway, we now have the sequel data engineering projects folder right here. Let's go through and create some different files and folders in it Specifically, I'm gonna create the first one and we'll call this lessons right now We're in 1.9. So I'll actually just create a folder for this I'll call this 1.9 and then in point 1.9. I'm gonna create a file So we'll call it 1.9
3:14:45
underscore VS code underscore intro dot sequel because it's gonna be a sequel file now inside the sequel file You're noticing up the top it has this of generate cult code and you can use control command I or select a language Start typing to dismiss. So basically if you start typing it goes away So I'm just going to put in some random code right now of select 42 as answer and then put a
3:15:08
chemical in now this file right here off to the right hand side I can tell that it is edited and that it needs to be Saved because it has this white dot next to it You can either come up here and click file save or the pro move is just to use something like command s if you're on Mac or Control s if you're on Windows So now let's start up a terminal to try to run this code and for this you can get the terminal up a couple ways You can go to view up here and then select the terminal or
3:15:40
Personally, I like the shortcut they have of control and then tilde now for those window users This is with whenever you go to pop open the terminal by pressing control tilde by default Like windows terminal PowerShell is gonna open now I can come up here to the down arrow next to this arrow and you can select a different profile in this place I selected git bash that's available right here go back to PowerShell and then back to bash But we want to stay in git bash
3:16:09
So what we need to do is change this to select our default profile and this is gonna come up the selection of what we want to Do I'm gonna just come up here and select get bash and now I'm going to close out of both of these and now when I run Control tilde get bash comes up notice here Also, you have this open chat control I start typing into dismiss you do have some AI features inside the terminal But once you start typing it disappears anyway back to Mac machine. We want to get it now
3:16:39
So I'm gonna just type in duck DB We want to get to where this code right here whenever we press shift enter well doesn't delete it I'm gonna command and but when we press shift enter on there it copies this and then paste it into here I'm gonna pop up and go close this and then we're able to Run it as well. So that was a sort of cumbersome to control c control v and then press enter We're gonna create a key binding for this. So we're gonna go into the command palette
3:17:09
You can either get for this go to view and then command palette But I like the shortcut of command shift P on Windows. It's gonna be control shift P So I'm pressing command shift P and we're gonna search for this one here of preferences Open keyboard shortcuts not the one with JSON. We're not gonna mess with the JSON just yet It's the same thing as this right here. I'm gonna close out of this window quick It's the same thing as this it's just in a JSON format
3:17:35
Anyway, we want to type to search in for the key binding that we're gonna be editing for this So I'm gonna go to type in terminal and I want this one right here of terminal run selected text in Active terminal. So it's this one right here. We've noticed right here. There's no key binding assigned to it So I'm gonna just click inside of here Or I double click inside of there and we want the key binding to be shift enter that's inserted into there
3:18:03
I'm just gonna press enter and it's inserted into here So now opening back up my terminal I have an active connection duck DB If you don't set one up right now now all I have to do is highlight that test went text whenever I press shift enter It sends it into the terminal below So now we're almost at where we need to be set up for the remainder of this course We've got VS code set up with our terminal to run sequel
3:18:34
Locally and now we need to set it up to where it's automatically we can connect a mother duck And we don't have to log in with those credentials as you recall I'm gonna log out of this little session right here If I go to log into duck DB Specifically if I try to attach the database of data jobs whenever I do this Yeah, it pops up, but whenever I try to query it makes me go through this authorization I can confirm it and then I'm in and then it gets me actually connected to it
3:19:04
And finally getting the tables back, but this is annoying to every single time you do this and Informatically, you don't really want to do this now in order to do this and connect securely We need an access token and we can go into app dot mother duck comm and Underneath settings. They have a section here on access tokens Now I'm gonna go ahead and create this token And I'm just gonna use this default name of lewd read Luke read right token
3:19:36
And I'm not gonna have it automatically expire and then I have it create the token Now this is the value. I'm gonna end up deleting it right after this lesson So please don't try to steal it and use it and not gonna be it's gonna be useless Anyway, I'm gonna go ahead and copy this bad boy now back inside of VS code. We're gonna use the Terminal for this. I'm actually gonna maximize it make a little bit bigger so we can see and I'm gonna exit out of here
3:20:01
So with this token what I can do is I can save it to my Terminal environment. I can do that by at least in this session running this command of Export and then it has to have a very specific name that mother duck knows and that's mother duck Underscore token and then I'm gonna set that equal to and then inside of parentheses I'm going to paste in that value and then close the parentheses. All right, I'm gonna run enter
3:20:34
Okay, and now that variable is saved in the environment. This isn't a long-term solution This is just me demoing this so if we were to run that echo command that we've learned previously And then if you recall for variables you put a question or a dollar sign at the front of it and you put the name of this So mother duck token and I run this it outputs the token right here now Whenever I go to log into duck DB Specifically into mother duck into the data jobs
3:21:10
It's not going to ask me anything when I run tables. There was no pop-up for me to Validate micro dentials or anything like that. It just showed me the tables that I've connected to so super cool But there's one problem with this. I just exported this mother duck token into the current shell session So if I were to go ahead and we'll crash this shell session And I'll use control tilde to start another one If I were to do duck DB and then try to log into mother duck again
3:21:44
When if I run a command like that tables It's gonna make me go through the authentication again because that variable is not saved permanently in there well as the Mac users learned about in that previous lesson There are actually certain login and shell configuration files that wherever you launch a terminal It loads these files and goes through like a startup script to load whatever in those files into your environment Mac users are using Z shell and
3:22:15
Windows users are obviously using bash and then potentially older Mac users using bash. Anyway, there's two main files for each of these One is your login configuration file and this runs only at log in This is such as the dot Z profile or the dot bash profile And then there's another one of dot Z shell RC or dot bash RC And this is your configuration and this runs every time there's a new terminal session
3:22:46
So we want to put it into this file into your appropriate one whether you have Z shell or bash Then we'll go ahead and quit out of here and clear this now for the Mac users We need to get that home directory. I like where we're at right now So I'm gonna create a new terminal and then in that I'm going to Change directory to the home directory now. They should have LS - a They should have created a Z profile or a Z shell RC file and RC stands for run config
3:23:19
Similarly, I want now Windows users to also create your requested or your required fire of that bash RC and bash profile So I'm gonna create a new terminal window inside of that second bash I'm gonna come in here. I'm gonna change directory to that home directory I'm gonna do an LS - a and I can see that there's no bash Profile or bash RC script So I'm gonna create those by running the touch command and adding bash profile
3:23:46
And then also I'm gonna add that bash RC as well now when I do LSA with a space in it I can verify that it's inside of there now I want you to create both of these because it's just good practice to have both of them. They're both necessary We'll eventually need both of them depending what you're gonna be doing But we're only gonna be adding to the bash RC because it runs every single session same with those Mac users
3:24:09
They're using the Z shell RC and that runs every single session anytime every time you start a new terminal So we're gonna now edit that that bash RC file or if you're on a Mac You're gonna now edit your ZSH RC file and in here We're gonna put that command that we previously had To export the mother duck token So I'm put export and then mother duck token the equal sign and then it's very important you put inside of parentheses We're gonna be putting the rest of this code
3:24:40
Specifically that access token that we generated and then I'll paste it inside of here by pressing command V or control V All right, it's inside of here. I'm going to now Write it out by doing control O and then also control X thing out of here Okay, but we haven't Run this profile this script yet So if I was to write the currency symbol or dollar sign mother duck underscore token Nothing's gonna come out of there. So I need to either I can restart my terminal or another trick to do
3:25:11
Is just run this command of source and then the file itself So Zh RC Now it ran that and so now whenever I'm gonna press up when I run this mother duck token The mother duck token actually appeared right here similarly for windows users just going through that I'm going to go into the editor for the bash RC Put that export command with a mother duck token paste in that token by pressing control V Adding that end parentheses Writing this out
3:25:41
exiting out of this and then running source bash RC out of problem with previously And now it refreshes that and so now whenever I run mother duck token Boom it prints out right below it Anyway back on the mac if you do the same thing on windows this should work now You should type in duck DB and now whenever you go to md data jobs It should log you in and you'll should be able to get right into it without doing any verification And load just like this for you. All right, let's move on
3:26:11
All right, do you want to wrap this lesson up talking quickly about github Co-pilot and their new integration of bringing an AI to it This may change more and they're rapidly changing it over time So I don't want to go too in-depth with a lot of the features because you may be watching it Maybe completely completely different. I do encourage you to take use of it No matter what I do think they're going to make you sign in like if I said explain this file
3:26:40
And I clearly have the 1.9 vs code intro sequel file attached to this it automatically did that by the way And I say hey explain this file. It's going to have you log in if you want to you can now go through and Set up a github profile by continuing with github You'll need to then come down here and create an account Like I said, I'll walk through creating an account and an upcoming lesson So we're not going to go through much of that right now
3:27:09
On how to actually create an account and all that because I like to cover it later Anyway, it went through and now executed this because I logged in And explained a summary of this sequel file So this is pretty cool as much this chat off to the side if you're getting stuck along the way That's a good way to just go there's something directly and you don't have to jump into chat gbt I'm gonna close out this welcome message One thing to note though with this
3:27:36
If I toggle go down here to this bottom little icon of it if I've logged in now Right now it has inline suggestions turned on So if I were to come in here and start typing in select It's going to want to go ahead and auto complete this with some AI generated stuff While we're learning this this is can be very inhibiting and can slow down your progress because you're not relying on your own memory You're going to start relying on the AI too much
3:28:04
So I recommend if you turn this on to turn off at all for all files In this little thing it's pretty neat too because you can see how much percentage of used They have right now i'm using the free plan and it looks like it's pretty generous in the amount of Prompts you can give to it and the amount of suggestion you get with it All right So we now have gone through taking a little bit of a break from sequel to be able to learn a lot about
3:28:29
The terminal setting up duck db locally and now getting you set up with vs code in the next lesson We're going to be jumping right back into sequel and specifically we're going to be focused on data modeling some little theory behind sequel With that i'll see you in the next one. Oh, by the way, there's no practice problems For those that purchase the supporter resources. It's a little christmas present for you. All right, and that see in the next one
3:28:54
In this lesson, we're going to be more theoretical in that we're going to be diving into our first part on data modeling We're going to start out first by understanding and breaking down databases and schemas and also tables You understand how and what they're made up of and what you have access to then from there We're going to be diving further into this diagram on our data warehouse. That is our erd or entity relationship diagram
3:29:26
Specifically, we're going to be diving into understanding what those tables are what those lines are for relationships and how they're related with keys And then finally we're going to wrap it up by Using some sequel in order to investigate these different type of things within a database. All right with that. Let's jump in First up just a quick recap on this hierarchy that we've seen before We have a database and that's our top level container
3:29:54
Inside of it. You can have one or more schemas and this is a logical group grouping inside of a database, and then underneath this, or underneath each schema are tables. And they show the actual data in some sort of a row in column format. Now this specifically is just showing related to the tables. Databases are a little bit more complicated than this that we're actually gonna get into. But this is directly related to how we have our database
3:30:23
modeled with our main schema inside that data jobs database, and then all those different tables underneath it. So let's break each one of these concepts down a little bit further so we understand how they're actually made up. First up is a table, and we're gonna start at the bottom most layer, and then work our way up. Anyway, tables as you've got by now are made up of columns, and then also rows. Inside of there, an individual unit in the column and row is a value.
3:30:51
But what you may not see on the surface is that a table also can have something called a primary key. And this is a unique identifier number for a column. It shouldn't be repeating, at least in that specific table that is the primary key for. Now related to that is a foreign key. A foreign key is a relationship to another table's primary key. So in our case, that company ID is related to the company ID and the company dim table. And just for fun, let's investigate real quick.
3:31:26
Inside of VS Code, I create a new folder for 1.10, and I'm gonna create a new file. And I'm gonna call it 1.10, data modeling dot SQL. We wanna make it a SQL file. For this, I'm gonna quit the job ID, job tile short, salary column, and that company ID column from job posting packs, and only get 10 rows back. In order to do this, we need to start up a terminal, and then connect to our mother duck data jobs database. So for me, all I have to do is select this,
3:31:53
and then with our key bindings, enter it right into the terminal. With this, we can see those IDs seeing that they are unique actual values for each, and then our company ID. Now there's no repeating values in this one right now, but there are technically repeating values in this table. Anyway, also wanna look at that company dim table, specifically at company ID and name. And so with that table, we have that company ID, and so this is the primary key in the company dim table.
3:32:24
So tables are really the building blocks and what we're working with inside of a database. Let's move up now to the next layer of schemas. So as we've learned already inside of a schema, we can have things like tables, which are a logical object. Listen underneath it are some examples, specifically in our case, we have job posting facts, company dim, and skills dim. Now schemas can actually have other things as well besides tables. That's why I said that hierarchy diagram
3:32:51
was sort of a simplified view. One other thing in a cab is views, and these are virtual tables. So I could use a SQL query to find out company info and make it into a view, and others could go to this view and actually, well, view it. And this is made from our tables. Now the other thing we could have, and I've sort of grouped it all together, is something called a semantic layer. Basically think of this like logic and also metrics. In here, I could define functions or macros.
3:33:21
And if you've worked with macros and Microsoft Excel, it's very similar. I could create some sort of function and then use it within a SQL query. I also could create things like metrics that go through and define how we're going to evaluate things like our salary. And then finally things like business logic that allows us to have reusable meanings to create things like flags or buckets. Anyway, both views and semantic layers are built on top of tables.
3:33:47
We can use them and are built from tables. So we need the tables as the hard core under core of the structure. And the main point I want you to get out of this is that database schemas aren't just related to tables because you may work in other databases and there's other things in there that you need to be aware of. All right, moving up last to the highest layer and that is the database. As we've learned already inside of a database, you could have one or multiple different schemas
3:34:14
that has your tables and views and everything else. But once again, there's a little bit more in there that you need to know about. One area is security and this defines things like users, roles, and more specifically the permissions for those different users and roles. These type of things can be managed at a larger company by a database administrator, but if you're a smaller one, you may be managing it yourself. Next up is the metadata.
3:34:41
And the purpose of this is to describe the database, the schemas, and the tables. They describe how to actually go through and build and the structure for all of our records. And at the end of this video, we're gonna be going through how to look at things like information schema in order to investigate further about our data warehouse. So if we have the metadata on how a table is built, well, we have to have the data that goes in that table and that is in storage.
3:35:13
Now, in our case, we would be taking that data would be inside table spaces, but you could also have physical files inside of a database and they'd be stored here. So now that we have the basics of database schemas and tables, we're gonna get into breaking this diagram down. And this is an ERD or entity relationship diagram. And now I'm reading this is like repeating diagram twice, like saying ATM machine. Anyway, this is a map of our data warehouse.
3:35:47
And there's a lot of key information that you can extract out of it. As we already learned about our job postings fact table, job ID is our primary key, doesn't mean by that peaky, and company ID is our foreign key, doesn't mean by FK. And then we have these lines linking between these other three tables, which has information on things like skills and also on our company, which may get you to thinking, look, why the heck do we split up this data across different tables?
3:36:18
Why make it complicated and have to use this ERD? Well, going back into VS code, if you recall, whenever we queried that job postings fact table for an associate job posting, we just had, well, in this case, a four digit number describing the company. And that four digit number doesn't really take up a lot of space in there. Now, if I go ahead and query that company dim table and show all the different columns associated with it, remember we have four characters
3:36:47
in our original job posting fact table, but now see for each company, we have a lot of different data associated with it. Not only its name, but also links to the company itself and Google information and whatnot. Anyway, breaking it up into tables helps solve this problem and allows us to basically make a more concise data model. And it makes querying it a lot easier. And an added benefit of this is improve data quality. So say in our case, look at this,
3:37:17
I queried our company dim table, looking for a company named Facebook and a company named Meta, I write it renamed. I now have a separate table that keeps all this company information. So it allows me to keep older companies and prevents me from having to overwrite those with those new companies. So now we're gonna break down three main components of this entity relationship diagram. And the first one is the entities which are our tables. We're then gonna go into keys
3:37:44
and then followed by the relationships to understand what these lines actually mean. So our tables as we previously discussed look like this, the range in column and rows, but that's not how they look in the ERD. In there, they look like this where we have our table name up at the top and then from there underneath it, you have your column names. And then there's a designator off to the right-hand side typically that would say something whether it's a primary key,
3:38:08
foreign key or both. So as we mentioned in job postings fact, our primary key is job ID and we have one foreign key of company ID. And this is related to the primary key of company dim. Now job ID can also be used as a foreign key in other tables. In our case, the skills job dim, which is our bridge table that bridges to our skills dim table that has the actual skills in it. So I queried in VS code, both the skills job dim table and the skills dim table.
3:38:39
Now let's start with that skills dim table first. As a refresher, this has a column of skill ID, the skills and then what type of skill it is. In the case of Python, it's a programming skill. Anyway, our bridge table that bridge together our job postings fact table and our skills dim table has two different columns in it. And that skill ID and job ID, both of these are the foreign keys because they associate to those two opposite tables. So job ID to job posting facts
3:39:07
and skills ID to the skill dim table. But also when we inspect the ERD, we can see that this skills job dim is not only the foreign keys for those associate tails, but also the primary key. Basically, it's a dual primary key. And this is called a composite primary key because the combination of the two columns actually makes a unique value. And remember a primary key needs to be a unique value for each row. Going back to actually look at that skills job dim,
3:39:39
you can see that for each one of these lines, if we were to combine those four digits on one side and the one on the other, those are gonna create a unique identifier for that row. So we understand the entities are tables and we understand the keys. What the heck now is going along with these lines connecting between the two? If you notice, they have these special symbols around where they actually connect to the tables. Well, there's three major types of lines that you can find.
3:40:07
There's a little bit more, but we're only gonna stick to the main ones that you're gonna encounter. You can have a one to one, one to many, or many to many. The diagram line of one to one, you can see this in here based on the fact that it has for one, it has just one single line on a side. If we look at the one to many, it has the single line on one side, but then on the other side, it has a crow's foot. And this shows that it's a many relationship. And then for the many to many,
3:40:35
you have crow's feet on both sides. Now, what the heck does that actually even mean? Well, we're gonna show it visually. For a one to one relationship, one record links to exactly one other. So in our case for one, two, three, four, five, you would have exactly one match in the other table of one, two, three, four, five. This one is like probably the second most common one you'll see. And the most common type is one to many. And it's where one record links to many others
3:41:03
or just one other one as well. But the main thing here is it can lead to many different ones. In our case, we do have a direct link between one to one, but for two in table one, there's three different values of two in table two. And that's the common relationship or the same type of relationship we have in every single one of our tables in our ERD. Specifically going to that company table, we have only one unique ID for a company,
3:41:32
but then there could be multiple associated job postings in the job posting fact table. So one company to many job postings. Now, the last one to cover is the least common that you'll ever encounter. And that's many to many. You could have many records linked both ways. This is cause is a lot of different confusion and it just makes clearing your database super difficult. We're not dealing with this. And if you see this in the real world, run. For us to duplicate this,
3:42:04
we would, relating it to our company example that we just did, we'd have to have many duplicated companies and then connect those to many different job postings. All right, so we went through basically all that theory. How the heck do we actually investigate this? Specifically with like this database here, how do we get in and inspect this metadata? Well, we can use information schema. Information schema is basically a collection of read-only views that tells us information
3:42:39
about the metadata inside of our database. We can look at things like tables, columns, views or table constraints. And there may be like, what the heck is a view? Don't worry, we're gonna be getting into that and viewing it, but it basically looks like a table. And so we're gonna be covering some different functions from information schema. And then you could check out the docs here for DuckDB on what you could actually investigate with this.
3:42:59
But the main purpose of this is this information schema is not only applied to DuckDB. Other databases implement this view or catalog of metadata that you can go in and query. So what you learn from information schema can be applied to other databases. So how can we query this? Well, feel free to follow along with me in VS code for this, we're gonna do, I'm gonna do something simple right now of just selecting all the columns. And we're gonna do this from information schema.
3:43:33
And this is a collection of views. So we have to actually use the dot notation to then go in and access different things in it. Specifically, we want to look at something like the tables and what it's type in dot tables and then add a colon. Remember, in the terminal, we need to connect to our database inside of mother Duck on what we can do. We can just connect right to the data jobs database. So this first one, we're gonna look at the tables. I'm gonna run shift enter.
3:44:01
Now this thing is a behemoth because it's including not only our data jobs database, right? It's including other ones as well. We're gonna make this simple for all our future queries using this. And that we're gonna query or filter for the table catalog of only data jobs. So in here, gonna add a where statement 'cause it works just like SQL and this table catalog. And for that, we wanna look at data jobs. Now data jobs is a text value. So you need to put it in quotes.
3:44:33
So I'm gonna select this all and then run shift enter. And now we're querying just data jobs. We can see the schema belongs to the table name and then the table type. There's also some other columns that hidden here. As I can see from down at the bottom, it says there's, hey, there's 13 columns, but only eight are shown. But those are the main ones I wanna show anyway. So let's dive into some other things as well about information schema. We can also look at things like the columns,
3:45:00
selecting this all and running it. And then for this one, it shows key information about the columns. Now, one thing, unfortunately, especially in a terminal view, you're limited by how wide your terminal is. Once again, we got truncated to only 40. There's 45 columns, but only seven columns are showing. So if I wanted to view pertinent information from there, I'd wanna go into the documentation here. I can go directly to on information schema, go into columns,
3:45:28
and then inspect which ones I've wanted from this. And then update my query for what I'm actually looking for. In this case, I could put in, hey, I wanna look for table name, column name, and then also the data type. The other option is use it in a more friendlier program for looking at very wide tables, such as my mother duck's UI or duck DB's UI. Now, there's only a couple more I wanna look at to show you for this. And that is regards to, hey, what else have you wanted
3:45:54
to look up things like the keys, where are the primary and foreign keys? Well, we could use something like the view on table constraints. Running this bad boy, we can see that we have this one entry here for on the company dim table that the company ID is the primary key and scrolling on over as well. We can see that, hey, the constraint type is primary key. And then conversely, related to that, in the job postings fact table, we have company ID, and that is a foreign key.
3:46:24
And the constraint for this is foreign key. Now, this table constraints also checks if there's not any nulls in certain ID specifically, make sure that there's no null values in a company ID or a job ID or whatnot. So this is a pretty useful table. But if you only cared about keys, maybe something like key column usage, that would be a better table to look at, as it simplifies this a lot more into identifying what's the primary and foreign keys in a very similar type format.
3:46:52
Now, another command to inspect metadata that is slightly easier, in my opinion, is Pragma. Now, Pragma is the keyword that you'll use, and then you could do things like, hey, list all tables by saying show tables. Anyway, it's a lot simpler and easier, right? The only thing about this is, is that Pragma is a method that was adopted from SQ Lite. And so what you learn with this can't necessarily be applied to any other databases, except for SQ Lite of course, in this case.
3:47:23
So for this, I'm gonna just type in the command of Pragma, and then we need to know that we can use show tables. I like this all, sending it down there. It will show us the tables that remember we connected into the data jobs database. So it's only showing those associated ones. We can also do something like this of show tables under store expanded. And this goes into breaking down, not only data jobs database, but the other ones as well, and a lot more information about the tables.
3:47:50
Now, besides Pragma, the other common one I find myself using specific to DuckDB is describe. And this is a great way to just quickly describe tables. So for this, I'm gonna just put in the command of describe. And then after it, you can put in any table, I'll put in job postings fact. Running this below, we get information about the different columns in that associated table. Such as the data type, whether it's null or not, if it's a primary key or not.
3:48:19
So a lot of quick information at a quick glance. And when you're working in DuckDB, it can be a lot quicker than writing out this entire statement, trying to go into the information schema to look at the columns. But I do wanna end on this, remember that information schema is very common across all databases. So I have some practice problems now for you to go through and get familiar with querying information schema in order to look at different attributes in there.
3:48:47
So now that we have this knowledge understanding how our tables are connected with relationships and how we're using primary keys and foreign keys, in the next lesson, we're gonna now be going over joins and how to actually join these tables together and perform analysis with that. All right, with that, I'll see you in the next one. Now that you've mastered basics of data modeling and understand why our data is split across multiple tables, we're now gonna jump into joins,
3:49:20
specifically these four types of joins and understanding how different joins affect the data returned with it. We'll be going one by one to cover these. So there are four major types of joins. A left join, right join, inner join and outer join. These Venn diagrams that are associated with each of these joins are showing you based on a table, assuming we have a table A and table B, what are the records returned from each of these? We'll be going into the specifics of each
3:49:55
as we break each one of these joins down. Now, one note is that these are only the four major types of joins. There's also one more type of join called a cross join and this produces a Cartesian product of two tables. And that's fancy talk for, that's super complicated and beyond the scope of this for right now. Or if you do see it, know what is an advanced concept and it is available. For each of these joins, we're gonna be walking through
3:50:20
how we can connect the different tables in our data warehouse. And then by the end of it, you'll have a better idea of what join you should actually be using to connect this data. First up is left join and left join and both inner join or two of the most common joins that you're gonna be seeing and using, so you should probably get used to those two. Anyway, left join returns all records from the leftmost table, table A, and then the matching rows from the right table, table B.
3:50:54
So for this example, we're gonna be walking through connecting the job postings fact table, which will be our A table. And we're gonna be connecting it to the company dim table as our table B. We'll be joining these using that company ID column. The basic syntax of this is we use a select statement to select the appropriate columns from each table. Notice we do have an alias for each table and that is defined below the from statement in that left join.
3:51:25
Specifically, we would start it with left join, specify the table and then you can give the table an alias and then we use the keyword on to specify how we want to connect this. In this case, connecting job postings fact ID column with the company dim ID column. Also, as noted in the from statement, we did have an alias as well for this table. All right, here I am in VS code. Let's go ahead and type this query out for this. I've created a new SQL file where we actually can go in,
3:51:52
ooh, actually didn't actually create it as a SQL file. So make sure that you're creating a SQL by ending it into SQL. Okay, that's working better. For the layout of this, try something different so we can actually see the output of the queries and put the terminal on the right hand side. You can actually change it yourself. You want to by right clicking terminal, going to panel position and then changing it to right instead of bottom.
3:52:12
As always, we need to connect to the database itself. So I'll switch this by DuckDD and then connect to our data source of data jobs. First thing I'm gonna do is write that select statement and I'm gonna fill in the columns after because what is the aliases I'm gonna do? I don't know just, well, I do know, but I wanna make a point that sometimes it helps filling in the next portions first with the table alias so that way you know what it is. So specifically, we're coming from from
3:52:40
and that's gonna be the job postings fact table. And we're gonna alias as J-P-F, basically the starting letter of each one of these. Pretty common in the data engineering, we're able to do it like this. And then, directly underneath the from statement, that's where you're going to include your join. In this case, it's going to be a left join. And we're left joining that company dim table. And at this point, we're defining that table so we can define it as an alias
3:53:05
and we're gonna define the alias of CD. And then I like to put what we're joining on, indented on the next line. And specifically, we need to join the company ID from each of those tables. So J-P-F dot company ID is equal to CD dot company ID. All right, and then for this, we're gonna just query all the different columns right now, just and we'll sort them down in a second, but I'm going to indent in and to specify J-P-F, period I want star and CD dot star
3:53:38
as I want the star from basically all columns from each. Also, there's a lot of different data. So I'm going to, initially what we're building this query, I'm just gonna limit it to 10, put a semicolon in there. All right, I select it all and they're on shift enter. And this is good now, we can test what is going on with the query if we have any errors. And we did because I can see it says CD dot period, it got confused by this period base here
3:54:01
and I looked around it and I say, oh shoot, I didn't include a comment at the end of that. Well, let's try again. All right, this time it actually worked. Now I'm gonna show more columns, so I'm gonna slide it over here and try it again. All right, all right, now that columns are actually fitting, so let's now actually fine tune this down to the columns of interest. Now I'm gonna do something first to just make an example, we're gonna correct after this.
3:54:24
I'm gonna go through and list the columns without any alias. Specifically, we're gonna get in the job ID column, job title short, company name, which is just name and we'll give it the alias of company name, and then also job location. Okay, when we run this, it should run just fine, let's go ahead and try. Okay, it does all of these return back. Now, this is very risky, so I wouldn't recommend it. Say in the case of our company ID, right, we have company ID in both of these tables.
3:54:56
If I were to put company ID in here and then run this query, I'm gonna get this error right here. Ambiguous reference to column name company ID. And it will specifically talk to me these pretty good, it actually tells you, hey, use the alias with that one because it's ambiguous, you don't know which table it's from, whereas something like job ID or job title short, it is specific to that table. But as good practice, you should just go through and list it for every single table,
3:55:25
even if you know it's not ambiguous. Here I did the company ID from the company dim table, but it didn't matter, I could do that one. And now running this, I went a little too quick, and then specify the correct table to take the job location from. Now running this, okay, it works just fine. So we've now demonstrated how to connect the left table table A to the right table table B, using something like a left join. But what the heck is actually going on with this left join?
3:55:50
So we've made this fancy, dancy diagram for this. And one note, this is fictitious data in here. Specifically, I've made it to where there are in table A of job postings fact, there are company IDs that don't have an associated company ID in the company dim column. So if we look here, you can see that company ID is zero, and there's nothing in the company dim column for that. And vice versa, we have this one on three that's going to experience,
3:56:20
and we don't have an associated job with this. This is fictitious, this is not what actually happens in the data set, but I did this to illustrate an example to show what happens when you do a left join with this type of data, 'cause that's come in the real world. Anyway, let's go through it step by step. Remember, remember left join takes all the data from that table A, that first table, and preserves that. And then table B, it only brings over those that are related.
3:56:47
So in our case, table A is preserved, and you can see job IDs 1000 and 1004 are preserved in there. However, there's no associated company for company zero, so therefore the company name is null because there's no associated one. In situations like this job ID where the company ID is repeated, it just lists it twice. And then in the case of Experian for company ID three from the company dim table, it is just not brought over with a left join.
3:57:16
So going back to VS code, I wanna actually see how many jobs are returned whenever we do this left join. Now just to recall, if I counted all of the columns in our job postings fact table, we would see that there's 1.62 million jobs. And then whenever we do this one now, without our limit statement, should have the same value in it. 'Cause as I said, right, there's no jobs without an associated company ID in it. And actually, that doesn't matter.
3:57:47
The more important thing is in table A as preserved. So in this case, we have 1,6,1,5,9,3,0,3 and the same of 1,6,1,5,9, well, three zero, I don't know why I've said zero, three. The same amount of values are preserved in each. It shows that the left join is preserving all those values. Next up is a right join. And this returns all records from the right table, table B in this case, and the matching records from the left table, table A.
3:58:19
This is so much less common than left join and inner joins, but I wanna cover it just that way you're aware of it. Anything you can do with a right join, you could basically, if you think about it and see how the diagram works, you could just flip the tables and do this with a left join. Anyway, for the syntax of this, it remains exactly the same. The only thing we're gonna do is change that left to a right. And then running this query here in VS code,
3:58:43
we have one, six, one, five, nine, three, zero, which scrolling up is the same exact values from the left join. And this is just because for every company in the company dim table, there's an associated job posting. So I expect them to be the same. So let's walk through this right join using some more, well, unclean data, if you will. And remember, for the job posting side table, we have this company idea zero with no associated company in it.
3:59:08
And along with that, we have the company dim table with this three of experience that's no associated job with it. So different from our real data set. Remember with this in the right join, we're keeping all records, we're preserving all records from table B and only the associated ones from table A. So starting with table B in this case, we have Netflix go over and then notice this. You would think there's only gonna have one entry for meta,
3:59:34
but remember, meta has two associated job entries in it. So in this case, it's going to duplicate down and you're gonna have meta from that main table duplicated twice. This is called fanning out. Next up from there, we have Google, which has a job posting and it appears. Now, in this case, I mentioned experience, but also Uber does not have an associated job posting. but that's in table B, and we want to preserve all records from it.
4:00:02
So it does get brought over, but we have only null values for the job ID and job title short. Once again, just to reiterate, right joins are less common. If you can rewrite with left join, well, you most absolutely should. And if you see this in the wild, run. Next up is inner join, and this returns rows when there's a match in both tables being joined. Applying this in our previous query, I'm going to change this right now. All we're going to have to do is change this to inner.
4:00:35
And whenever I run this bad boy by selecting all shift enter, we once again get the same amount of rows of one, six, one, five, nine, three, zero. It's a reminder that's because the job postings fact table has associated companies and company has all associating job postings with it. Now one thing to note from this and what you'll see also is that inner join was the original default join in SQL. So because of that, you may see some people write this out as just join.
4:01:04
And this is going to do exactly the same thing as an inner join. I'm not a fan of not specifying that this is an inner join. So I would always specify it just for best practice. So let's show this on the fictitious data set that has some jobs and some companies that are not associated. Remember, inner join is only going to return the rows when there's a match in both tables being joined. On the job postings fact table, we can see that yes, there's no company
4:01:31
for zero, but there is one for one, two, and also four, which are Netflix, meta, and Google. And then over here, the two remaining experience and Uber, they don't have any job postings associated with this. So from our results table, we can see we just have job postings from Netflix, meta, and Google, because that's where overlaps. The last type of join up is a full join or a full outer join. And this returns all rows from both tables. If keys match, rows are combined, if keys
4:02:05
don't have an associated value, then the missing side is null. This is great for checking for data completeness and checking for basically orphan data data that doesn't have an associated value in another table. Anyway, let's demo in VS code. So I'm gonna change that inner that we were previously doing to a full outer join. And then running this bad boy, we get once again, all the same results similar to all the last tables, because the data cleanup that I did on building these tables was
4:02:32
so good. So unfortunately, we have to resort to the diagrams to better explain this. Um, what one note before getting into the diagrams is this. So full outer join works, but also a full join only writing that full join syntax can also be used. I would continue to write or I would write full outer join that's pretty common. Using just full join is pretty common. And so I would say it's acceptable to write this as is. Now the reason why that outer is optional is because
4:03:04
things that we did previously, like our left join, I'm gonna go ahead and run this bad boy just to show that it does work. The left and both right joins are actually technically left outer joins and left right or right outer joins. Anyway, these produce the same amount of things, but it's very common as well with this, not to write the outer in there. So if you don't want to write the outer, completely understand in the full. So what's going to happen whenever we combine those two
4:03:33
fictitious tables in our results table, remember in this one, this full join, it's going to return both values and both tables, whether it has an associated record or not. So our case over here, we can see that it includes the 1000 all the way to 1000 for job IDs, because it's in the results table. And then additionally, we know that we used one, two and four in there. So one, two and four included, along with three and five. So we have Experian and Uber down here with null values.
4:04:06
And then one that I sort of glossed over is that company idea of zero that doesn't have a company dim, that one's going to have a null value in it. So we just worked through the four different types of joins and specifically with that company dim table. Although that one wasn't really exciting because it returned the same results on each one. Now we're going to get into looking at going through the most popular joins with bringing in our skills table, in order to show associated
4:04:35
skills for job postings. Now to do this, we're going to have to do two joins, we're going to have to join into our that bridge table of skills job dim. And then from there, connect into our skills dim table. Let's look at these tables, just the top 10 rows of each just to, in case you haven't looked at it just yet. In our skills dim table, which that bridge table, we have associated skill ID and an associated job ID. And then our in our final skills dim table, we have for each one of
4:05:07
those skill IDs, the skill itself, so something like SQL Python and R, and then what type it is, programming analytics, whatever. So from our job postings fact, we need to connect to the skills dim table and then from the skills dim table into our skills job dim table. Now, one note with this one, unlike the company dim table, there are some jobs without an associated skill, there's no skills required for it, probably a job I want to apply for anyway, let's work through figuring out what
4:05:38
type of join we're going to be using for this for this, we're going to start with a blank query, we're going to keep it pretty simple, we want to get the job ID, additionally from that job postings fact table, we want to get the job title short. And then for fun, we're going to bring in from the skills dim table, which we're going to give the alias of SD, we're going to bring in that skill ID. And then also from this, we're going to get the skills from it. Now the skill ID doesn't matter,
4:06:04
we could have gotten that from the skills job dim or the skills dim table, it doesn't really matter. I think I just said we're gonna get this from skills job dim, doesn't matter. Anyway, for this, we always want to start with our leftmost table in this case, is our source of truth is going to be our job postings fact table. And we're going to give this the alias of JPF. Now we're just going to start first with a left join just to see how this does. So we're
4:06:28
going to do this, this left join of the skills job dim table, alias as SJD, although we don't even use it in this. And then we need to specify the on. And that's the job ID from the job postings fact table. And the job ID of skills job dim, which actually we do use the alias. Anyway, I like testing queries out as I build them. So actually, I'm going to just go ahead and just comment this out right
4:06:50
now. And also put a limit on there of 10. Let's go ahead and run this bad boy. And so good that we are testing this out because SD is not defined yet. All right, so we got the J in there, we can now go ahead and run this run, shift enter. All right, so this is good. It's working. We can see that for something like 4 5 9 4. This is a it's got three skills associated with this job posting. And then 4 5 9 8, which was skipped in between here. This has well, apparently no associated
4:07:22
skills right we're doing a left join. Now it doesn't mean 4 5 9 8 isn't included here. It's probably the way the query runs. It does the null values at the bottom. So that's why it's missing, at least in the view for this limit 10. Anyway, let's continue on with getting the other left join. Specifically, we want to get that skills dim table so we can get those skill names. We're going to give this the alias of SD. And this one, we're doing on the skill ID of the skills job
4:07:49
dim table and the skills dim table. Oops, I got a typo right there. Okay, I'm going to also untie on comment out that skill. So we can actually start seeing those skills. All right, let's run this. Okay, good. This query is working and based on and putting in those different skills that actually now reorders those job IDs. So now let's think about this and what case we want to use this. We are doing left joins and left joins are used anytime we want to preserve all of the values
4:08:19
from the job postings fact table, regardless if they have an associated skill. So I'm going to go ahead and remove this limit 10. We're going to see how many values we actually get for this. We're going to experience that it's going to fan out because there's a bunch of skills associated for job postings. Now we're at 7.48 million rows. And like I mentioned, the null values are all at the bottom. Now remember that I said the two most common types of joins
4:08:49
that you're going to be encountering are left joins and then also inner joins. Inner joins are only going to return matching values that exist in both table A and table B. So in this case, job postings that don't have an associated skill won't appear. So I'm going to copy this query right here, paste it underneath here and I'm going to change this left to an inner join in both places. All right, let's see how much this query changes. We should expect the number of job
4:09:21
postings to go down. And they do went from 7.48 million to 7.19 million rows. So basically 200,000 jobs don't have an associated skill associated with it. Now, so we're typically always going to want to preserve all jobs regardless if they have a skill or not. Because of that, left is going to be the most common one they use. But just to see what ends have we tried instead of this inner right here? What ends have we tried full? And remember, that's going to return and preserve all
4:09:56
the contents from table A and for table B. In our case, when we run it, I would actually expect to get that 7.48 million job postings. Yep, same one here. So this yields the same results as our left join. So that was just for funsies. I would however recommend anytime we're wanting to do an analysis and we're wanting to preserve the job postings fact, we're going to be sticking with a left join going back up to our company dim table as well. We're going to be maintaining the same
4:10:26
practice as well if we're ever joining with this one in order to preserve all of the job postings in it, we're going to be doing a left join going forward. And this running this query would preserve all the records of 1.62 million jobs. All right, so up to this point, I would say that we've covered all the basic SQL syntax to perform analytical analysis because of that, we're going to be jumping into the project after the next lesson on order of execution understanding how actually the SQL
4:11:01
database engine processes queries, so you can get some background on what's going on behind the scenes. It's going to be a little bit technical and theoretical, but it'll give you background and really strengthen your knowledge on understanding order of execution. For those that purchase the course resources, you now have some practice problems to go through and test out when you would use different types of joins in different scenarios, depending on what your final need is for your
4:11:28
engineering query. All right, with that, I'll see you in the next one. In this lesson, we're going to be diving deeper into understanding the theory behind how your SQL queries are executing. And the main purpose of this is to be able to understand why also you have to write certain queries in a certain order, and how it actually gets transferred within the database engine itself.
4:11:59
So first, let's dive deeper into how a query processes, because this is going to be important for understanding where order of execution actually gets carried out. So for recall from this diagram, we went out in the very beginning of this course, SQL works as follows, we first go to our computer and we type in in our favorite text editor, a SQL query. From there, we press enter and it sends it to the database engine, whether that's locally or up in the cloud. Now the database does some
4:12:30
work that we'll go into in a second. But the results are then output to a table, which is then sent to your computer to actually view and save these results. Now let's build on this slightly more, and we're going to be going into the query processing architecture. Inside of a database engine, there's typically three, sometimes more components, but we're going to focus on the three main components of a parser and optimizer and the executor. So let's break each one of these down,
4:12:59
and we're going to start first with the parser. And this reads your SQL text and checks to make sure that it's valid. Every time I forgot a comma, or didn't name a column correctly, that was done by the parser. Now it only doesn't only just check for that, it also builds out an internal tree structure called the parse tree. This is a structured representation of what you meant and is then send into the next thing of the optimizer. This, as the name implies,
4:13:27
figures out the most efficient way to run your query. It can reorder your joins or even sort out different strategies for scanning and hashing. This is rewriting your SQL query to make sure it runs as fast as possible. All right, last up is the executor. And this actually runs that optimize plan step by step. It reads your data, applies any filters and joins groups, aggregates, and returns the rows necessary. After it goes through that, you have your final result set.
4:13:56
Now as a quick refresh, there is a certain clause order that you have to implement when writing SQL queries. And this is what's checked in that first step of the parser is not only checking to make sure that you got those commas and names spelled correctly, but also ensuring that you type them in the correct order. If you write them in the wrong order, the parser is the component that will reject it and throw a syntax error. For example, here in
4:14:24
VS code, I'm connected to our database. And if I were right, select star, and then put a limit statement above our from which we know is out of order. If I tried to run this, one, it's going to take a second reply because it has to figure out, oh, there's a mistake here. So that's one thing, you know, you know that you've probably messed up if it's taken a long time for the query to come back. And then we finally get this of parser error syntax error at our new from. And so it gives you
4:14:49
the glue parser error. That's where it's done in the parser. So now we understand that the order right queries is important because the parser is going to check it. But how the heck do the actual commands or the keywords get executed after this point in things like the optimizer and executor? Well, for theirs, there's a very similar order that we're going to flow through in showing the step by step process for this. First, it looks at any froms and joins to collect all the tables we
4:15:23
need. Next, filtering happens with the where expression to get the number of rows down. From there, we group any rows that share any properties. Then because we did that aggregation with group by, we can then do a having statement to then filter those groups based on those aggregates. We can then select which columns we may need for the final data set that may have been used up above, but now they're not used below. And then from there, getting to order the results in a
4:15:50
certain order and finally limiting to the number of results we want. This order of execution is really similar to that SQL clause audit order. I don't want you to confuse because they are carried out in different parts of the database engine, but they do follow a similar the only one that is out of order here, if you noticed, is step five of what columns you're selecting in that we would name
4:16:12
that in the SQL clause order up at the top. Besides that, how you actually write those clauses in that order is pretty similar to how it gets executed in the query engine. Pretty cool. Well, the order of execution is carried out in technically two areas, specifically first, the optimizer implements that order of execution and puts it in the correct order and then sends that over to the executor that
4:16:39
executes that correct order. So let's go through building a SQL query and showing an understanding how this order of execution actually comes to our advantage and how this all works together. For this, we want to find the top 10 companies for posting jobs. And in order to make this list, they have to have greater than 3000 jobs. So the first thing we're going to do is get into selecting the columns that we want specifically. I know I went from the company dim table. I want the name,
4:17:14
and we're going to name this company name. And then from there, what the count of job posting. So I'm going to put inside of here, JPF, and then period star to count the rows in that column for the count of the job postings. I'm going to give this an alias of posting count. Next up is the from statement. And what the job postings fact table as our table a, and we're given the alias of JPF. And we're going to we want to preserve all the job postings, right? So because of this, we need to
4:17:44
do a left join. And we do that on the company dim table as CD. And for the idea of this, we want to match on the company ID of each of those tables. Okay, this is enough query. Anytime I get this many lines in, it makes me nervous that there's a mistake. So I'm going to go ahead and just run this bad boy. And as we can see, we got into our first error, right column name must appear in the
4:18:05
group by clause or must be part of an aggregation function. And so let's put that group by in specifically we want to group by that name. Now we're going to list CD dot name and run this now. And now we're getting the counts of those companies. So the query is working. And it looks like we have around 215,000 unique companies. Something I want to call out with this going through that order of execution,
4:18:28
we did the from and the joins that was step one. And then next we did step three of that group by the column. And then underneath that, so step five is that select statement. And the reason why I'm calling this out is because technically, whenever you do this in most SQL dialects, you cannot put the alias so you could not put the company name inside of here. If I was in Postgres, not going to work, at least as a filming this. Now, however, in duck DB, they've understood that
4:19:03
hey, probably who people want to do this and make it a lot easier in our lives, you can do column aliously aliases in here. But don't get too used to this because other databases don't always support this. So in this case, right, if I run it, it's still going to provide the same results. So I'm going to change this back to what I would expect to see across all SQL dialects. Now notice one thing, right, I said that alias of the column names. But we could use the alias of the join
4:19:28
because the alias of the join is specified or the table alias is specified in step one, and would be available to the group by. All right, next on, let's move on to this list. To get to our final answer, I'm going to add another caveat to this one of our question. We want to look in the US only. So I'm going to limit this only to US jobs. Now, if you remember from the SQL query order that we need to do this in, we can put the where between the join and the
4:19:57
group by. And for this, we would specify that job country. And in our case, we're going to do it equal to the United States. Okay, first, we're going to just operate, go ahead and execute this to make sure this works. Okay, it is working. Now I want to call out a few things about this. First, we have the table alias here. And we can do that because the where clause is after that from that step one where it executes first, additionally, this where clause, well, right now we are doing
4:20:28
aggregation. But let's actually, we're doing aggregation on all columns. So you may be like, oh, well, all columns are there. So that's why we can see job country. Let's actually change this where job country isn't in the select statement. So I'll change this to job ID. And now whenever I run this, okay, getting the same results, it looks like I'm getting the same results. And the reason why I'm
4:20:50
showing this is because if we go back to our order of execution, we haven't sold it, we haven't got to step five yet where we're actually filtering down to certain columns. So whenever we do that where clause, it's up above right after step one, when we have all of the different columns available. So that's why we can list a column in a where clause that isn't necessarily in our select statement down in step five. All right, moving on to the next condition. And I think
4:21:19
that's well, almost the final one, we need to get the job postings that have greater than 3000 job postings. So we're going to do a having clause, we'll do a count, we'll just keep it that job ID listed above. And we want it greater than 3000 running this bad boy. All right, we're getting the results here. Now, one thing to note about this, once again, in other SQL dialects, you can't list an alias, in this case, posting counts inside of the having clause. However, duct DB
4:21:51
makes it super friendly, and you can list alias inside of here. But if we were looking at that order of execution, that step four for the having clause is above our select clause, where we actually define those aliases. So that's why in most databases, you can use alias and having. All right, moving on, this is just for me, I want to actually order these results. So I'm going to put an order by, and when this one though, I can put in the alias of posting count, of course, in descending
4:22:21
order, and then running shift under, we have them now in order that we want it. Notice that we go back to the duct DB with that friendly database, it says, Hey, it allows column aliases and where group by and having, that's because order by already allowed aliasing. Because if we look at the order of execution, it is step six, after step five, where we're selecting the columns and assigning aliases. So regardless of your database, you can put aliases in the order by, all right, and then
4:22:51
moving into the last portion, we just want to limit this to 10 results. But if you notice by this, right now we're only at eight. So even when I run this to a limit 10, I didn't select the whole query, that's why I'm getting that error there. So even with selecting it to a limit 10, by the time it got to that limit 10, it was already at only eight values. So being the last step, it becomes basically obsolete and not really even necessary. So we just walk through how this query operates
4:23:25
step by step by step using this order of execution. But we can actually use one or even two keywords of explain and explain analyze to actually see what the steps are. So diving into the documentation start with that first keyword of explain, explain and specs query plans. You use the explain keyword right before a query, it can be a searchable query, or it can even be DDL and DML used with. Anyway, it breaks it down that the explain statement displays the physical plan, i.e. the
4:23:57
query plan that will get executed as enabled by propending the query with explain. And so what happens whenever you execute this with something like explain is it goes through in this block like diagram and explains what's going on. Anyway, let's demonstrate that for this up above it, I'm going to just put in explain. And then from there, we're going to now run this entire query. So the query ran and the plan is here. Now, an important note about reading this, you would
4:24:25
think you read this from the top down, but that's not correct. We've been going through with this thing explaining it from the top down of step one down to step seven. But really, what it actually looks like is this step one is at the bottom and then all the way up to the top is step seven in this case. So we need to read from bottom to top. So let's go through this and we're just going to look at some key turns from this that you should be able to notice. So I'm at the very bottom. And
4:24:51
for this, we have a sequential saying that's looking at the tables, it's looking at the company Dem table, and then also the job postings fact table. Now with this job postings fact table, remember, we did a filter on job country of United States. So during this scan, it goes as in filters at underneath here, it has an estimate of how many rows it's doing for this. It's not actually running this query in explain. That's why this is an estimate. And it has these like little
4:25:17
correlate notation in front of it, the tilde. So all these roles up here are just estimates. Anyway, going ahead from here from the sequential scan about these, it then performs a hash join, which we can see that in this case, it switched our left join to a right join. Like I said, you can just switch the side depending on what you're doing to switch from left right. So not a big deal here. It even tells it what ID it hooks it on. Next from there, we get into the projection.
4:25:41
And what it's doing is trimming down the column names that it wants to keep for the rest of the queries. This is like a duck DB specific thing, because it makes it more efficient by removing the columns. Next, it moves into our hash group by, which is effectively our group by where we're analyzing by company name to get the different counts or the aggregate here that it specifies. Then from there, it moves into our filter of having where we wanted to
4:26:07
get the count or get those counts that were less than or greater than 3000. From there, it moves into that top N or that limit statement to limit by 10. And then from there, it moves into two other operations that are not really specific to order of execution that are duck DB specific. Anyway, that is the physical plan that is explained. Now with this, we're going to be very clear about it. Explain here the query or it says note that the query
4:26:35
is not actually executed. That's why those row counts were estimates. Well, that's where explain analyze comes into play now. As from this, we can see that prependent a query with explain analyze both pretty prints the query plan and also execute it, providing runtime performance numbers for every operator. All right, similar to explain, we just put explain analyze at the front of a query and scrolling down, we can see that it gives a similar execution plan, but now it
4:27:04
includes some additional information specifically time. All right, so in here, I'm going to update our query to explain explain analyze. And then from here, selecting it all and running it. Now the overall plan compared to explain has remained the same of the different steps that it performed. There are a few more up at the top, specifically these extensions here. And these are just specific to once a duck DB doing some stuff behind the scene. That's not
4:27:33
really important to our order of execution. Anyway, up at the top, we can actually see now what was the total time to run this query, which was 0.778 seconds. And in addition to that, scrolling up back to the bottom where it starts, it will actually provide per step. What is the amount of time for that step with the exact number of rows for it? Now, one thing to note, because you may be confused, especially this one right here, it says, Hey, it was 3.35 seconds
4:28:01
just for this table scan. But then up at the top, it said, Hey, the total time was 0.778 seconds. Less what's going on? Well, remember, this is these can work, uh, sequentially or can work across multiple different threads or multiple CPUs. So this is the total time across all the different CPUs added up. But overall, in time that's relative to our time, this total time would only took 0.77 seconds. And this time down here is just the total labor time across all the different CPUs.
4:28:32
Anyway, explain analyze. I'm not going to go through all of this again, because it's still, it's, it's the same. We can see that the row counts did update and the time did update for each. But overall, it's now providing us, I find this more beneficial, especially for doing some sort of analysis queries. Now comparing the two with explain like we learned, it doesn't execute the query. So because of this, it's only having estimates about what is the row count. And it
4:29:02
doesn't even provide an estimate for the runtime in DuckDB. Now for both explain and explain analyze, they do show the plans. And for explain, they show estimates and depending on your database, explain analyze will also show estimates. But in our case, DuckDB, it doesn't do that. Anyway, the last key thing that I do once you get out of this is explain is safe to run on any type of query. If you're running a right query, something that may be creating or dropping a table,
4:29:31
it won't actually do that. So it's safe to run it. Whereas if you use explain analyze with creating or dropping a table, it's probably going to run it. So use this with caution. But overall, if it's just this type of query here, what we're doing, some sort of analysis or aggregation, I find myself gravitating towards explain analyze because it's more accurate and it tells us what's actually happening through executing it. Now in our case, this query, Time was, what, less than a second.
4:30:01
So overall, it looks like this query executed just fine. But if you find yourself in situations where it's taking minutes or hours to execute a query, you should be using this Explain Analyze and analyzing to see how you could make your query more efficient. All right, now it's your turn to give it a try. I have a quiz and some practice problems to get more familiar with how the query operates and also what you could actually do in your different queries.
4:30:29
Not gonna lie, the quiz, I made this one a little bit spicy. But it does test on every single concept that we've covered in this video and also previous videos. With that, I'll see you in the next one and we're gonna be diving now into building out our very first data engineering project. All right, I'll see you in the next one. All right, welcome to our first project. And when we complete this, we'll be halfway through the course. Now, the main purpose of this project is,
4:31:02
as a data engineer, you need to start thinking of who you're building your products for. Specifically, you're more likely gonna be building something like a data warehouse and it's gonna be used by a data analyst or business analyst. And if you don't understand how data analysts and business analysts are working with the products you build, how the heck can you even build them yourself? So for this, we're gonna go into the project goals and scope.
4:31:32
But first, we're gonna dive a little bit deeper and understanding on what exactly a data warehouse is. So for this project, we're gonna be continuing to work with that same data warehouse that we've been using in the last few lessons with our job postings fact table and our different dimensional tables that describe company and skills. Now, I've told you time and time again, this is a data warehouse, but what the heck makes this a data warehouse and not just a regular old database?
4:32:03
Well, for that, we're gonna reference this dude, which considered the father of data warehousing. His name's Bill Inman. He wrote the original book that defines this data warehouse in the 1990s and he defines this follows. That is a subject oriented, non-volatile, integrated time variant collection of data in support of management's decisions. So that's four key attributes, subject oriented, non-volatile, integrated and time variant. And if you're missing any four of these attributes,
4:32:34
you're no longer working with the data warehouse, you're working with just a plain old database. So let's define each one of these attributes versus subject oriented. This means it's organized around key business subjects rather than just an application. So for us, this is around the area of job postings. In a business setting, you'd see this more around things like customers, products or even sales. Next is non-volatile. This means the data is stable
4:33:00
and doesn't change once it's entered into the system. Data is appended to this, not overwritten. Next is that it's integrated. It combines data from multiple sources into a consistent format. For us, we have taken these job postings from a variety of different platforms such as LinkedIn, Indeed or Google and putting it all in here. And mainly, making it into a consistent format that we can easily query. And lastly, it's time variant then that it stores historical data
4:33:30
across different time periods for trend analysis. So you typically want to find different databases for different years. You should be able to go to one location to get all the recent years that are needed to do your analysis. Now as a data engineer, it's important to understand this definition because there's a lot of different tools out there that are advertised as a data warehouse. But if you're not implementing this approach of it, you're not actually building a data warehouse.
4:33:58
You're building a database. So for this exploratory data analysis project, we're gonna be first extracting or working with the data inside of our data warehouse, which we're gonna be using of DuckDB and MotherDuck. And so with this, you're gonna take the role of a job market analyst looking to analyze the top paying roles and skills in the data science industry. You're gonna be creating SQL queries, specifically three SQL queries, to explore this large data set specific to,
4:34:30
well, specific to you if you want to. And I'll get into how we can adapt these queries to your specific circumstance. But the end results are gonna be very similar in what benefits they actually provide. And that we're gonna be uncovering data insights around what are the top-dominated skills, highest-paying skills, and then my favorite, the most optimal skills. Now for this, if we recall, if we look at the distinct values of the job title short column,
4:34:59
you have 10 different roles you can select from. I'm gonna be analyzing it, assuming you're gonna wanna become a data engineer. That is what the analysis I'm gonna be forming with. But feel free to adapt it to any of these other ones here. Additionally, what I won't be doing, but you can do and further adapt this, is you could change it to what country you're doing. And there's over 161 different countries you can choose from for it. Now, as I mentioned in intro,
4:35:25
EDA is typically a task of something that a data analyst or business analyst can perform. But if you think about it, data analysts and business analysts are basically your customers. And so this is gonna help you understand what type of questions the analysts start to think of when they have to analyze data. And also with that, it helps you understand how data should be structured and also organized so that way those customers can do more efficient queries.
4:35:52
When you start understanding these end goals, it's gonna provide you with better understanding to build a better data architecture for your customers. So navigate over to the URL on the screen here. And this is gonna direct you to GitHub, on to where my final repo or repository is for this project. Now we have a whole lesson on understanding Git and GitHub. So you'll be understanding better of how to get things up to there and how to navigate this.
4:36:24
But first, here's just a quick little overview of the project and how to read what's going on in here. Up here at the top is the structure of our files. So for this, we're gonna be creating three different SQL files. If I navigate into this first one by clicking on it, I can see that inside of here, I have up at the top the question for that first query, the SQL query itself. And then underneath this, the results, along with some insights that will be extracting out
4:36:54
and put in here as well. Navigating back to that top folder, it's the same format for the other two SQL squeals. And then finally we have what's called a readme.md or markdown file. That's what's underneath here, auto appearing. You could also technically just click on it to go into it. Now this bad boy, we're gonna go into how to actually build this and then a lesson after our SQL queries. And it's gonna go through and explaining what our project is.
4:37:23
And this is really good for others or even hiring managers to come in and see your work and see what skills you've demonstrated. For example, with this readme, we're gonna give a high level overview with an executive summary, telling them what queries they can check out with navigatable links to get to a specific query. We'll go into the problem and context next of what we're trying to solve, what data warehouse we're using for this.
4:37:49
We'll explain the tech stack of all the different skills you've built up and utilized during this. And this includes not only SQL, but also your other skills that you're gonna learn such as VS code and Git and GitHub. Then from there, we'll get into an analysis overview of what we extracted from a high level from those three queries. And then finally, putting ports on the paper of what skills you actually used for this. Really helpful for hiring managers to look at this
4:38:13
and be like, oh, okay, these are the skills I'm looking for. These are the skills he has. Or she, so for the next three videos, we're gonna be diving into each of these questions for it. Starting with the next one, investigating what is the most in demand skills. And remember for all of these, I'll be doing it from the perspective of the data engineer that wants to look for remote jobs. You can do the same or modify it as I showed earlier.
4:38:37
All right, there's no practice problems for this lesson. I would spend some time going through that repo that I gave you and understanding what generally we're gonna be doing with this project. With that, I'll see you in the first query in the next video. (upbeat music) All right, let's jump straight into this. I'm starting here in VS code. I'm doing it in Mac, but Windows is gonna be the same regardless. Remember with this, we're trying to solve that first query
4:39:05
of what is the most in demand skills. We'll get a further problem statement for you. But first, let's set up the structure that we need for our project. Right now, inside of our SQL data engineering project folder, we have a lessons folder. And I wanna create another folder as well. So make sure when we go to create a new folder, which we select right here, that it actually inserts it here and not. If you have lessons selected and you were to insert a folder,
4:39:31
inserts it underneath there. Totally annoying. Anyway, so with our new folder, we're gonna call this, this is our first project, one underscore EDA. I like putting a one at the front because it helps organize this. Mainly numbers go first and then followed by letters. All right, so with our one underscore EDA or first project for EDA, we're gonna stick our first file in here. And I like to do a similar numbering approach for a query. So I'm just gonna put zero one
4:40:00
'cause technically we probably maybe could have more than 10 and so I want double digits. And for this one, our top demanded skills. And this of course is a SQL file. Okay, this bad boy is open. I'm gonna go ahead and close the Explorer. So let's get into our problem statement. And for that, it's the question of what are the most in-demand skills for data engineers? In this, we're gonna identify the top 10 in-demand skills for that job role.
4:40:26
And I'm going to be focusing on remote job positions. Like I said, you could do something like focus it towards your specific country. And then underneath that I have the importance of that it provides insights into the most valuable skills for data engineers seeking remote work. So anytime I'm working with a problem statement, I like to include it up at the top. I'm gonna use a multi-line comment to insert it in. So I'm gonna put this in here on what we're trying to focus on
4:40:52
and what we should have the outcome to be. So let's start building our query out. But in order to do that, we need to connect to our data warehouse in Mother Duck. So I'm running DuckDB command to connect to that data jobs database. So for this, we want the top 10 most demanded skills by count. And that's in our, the skills name is in our skills dim, but we wanna count of our fact table, which has the actual postings in it. Now because it's in two separate tables,
4:41:20
we're obviously gonna have to do a join, but we need to think about what type of join we're gonna be doing for this. I like to keep job postings fact as the A table and skills dim as the B table. If we did a left join, yes, that's good. We can preserve all the facts from the A table. But remember, there's some jobs in the A table that don't have an associated skill. So we also have that in there. And I don't really wanna count values of no skills required, sort of not necessary.
4:41:51
Now we could technically do a right join to connect to those job posts in fact table, but right join isn't widely used and if you're using the right join, you should probably use the left join, but I have a better solution for this. And that we're gonna use an inner join in that from table A on that job postings fact. If there's jobs that don't have an associated skill, we're just gonna weed them out. And conversely for table B of skills,
4:42:13
if there's skills that don't have an associated job, but spoiler alert, all the skills do have an associated job, they would then be filtered out. So inner join is our best bet in this. So let's start building this out. We're gonna first just do a select, and then I'm just gonna throw a star in here, 'cause I don't know which column is just yet, we wanna get or the name of the alias of the tables. We're gonna be going from our job postings fact table,
4:42:35
and we're gonna be forming an inner join. Specifically, when you get to do two, right, we have to connect to that bridge table of skills job dim and then to our final table of skills dim. So we'll specify that skills job dim with the alias of SJD, and we'll do this on the job postings fact table, which we haven't given an alias for on job ID. And that's to the skills job dim job ID. Now I need to assign the alias for job postings fact up here. Okay, we've got one table joined.
4:43:02
The next table that we're gonna get is our skills dim, and we're defining this as just SD, and we're doing this on our skills job dim table skill ID with our skills dim skills ID. Looks like I got a typo here, fix that. All right, I do wanna test this query, but I don't wanna run this query on a bunch of different rows, so I'm gonna go ahead and put a limit 10 on here. This limit 10 also is gonna be helping us in understanding the top 10 in demand skills.
4:43:32
So we're sort of getting two birds of one stone. So I'm gonna go ahead and select this, run shift enter. All right, and we're getting results back, not too bad. All right, let's fine tune this further, right? We wanna get the most in demand skills for data engineers. Let's focus on that first thing, first of most in demand skills. So we need to, in order to get that, we need to do an aggregation of the count of jobs based on a skill. So we'll define in our select statement
4:44:04
that we want from the skills dim table, we wanna get that skills. And then next we wanna do an aggregation. Now for this, you could do either the job postings fact table, or you could do the skills job dim. We're gonna do job postings fact table, and I'm gonna just do all columns in that. We're gonna give this the alias of demand count. Now anytime we do some sort of aggregation, right? We need to do a group buy. In this case, we're gonna be doing the group buy on the skills column.
4:44:36
Okay, let's go ahead and run this bad boy to see how it's doing. Looking good, although it's not an order, right? I got our skills and our demand count. I would want this in order so I can see those top 10. So I'm gonna put an order buy in here, and I wanna order it by. We can use the alias here, 'cause we learned in a previous lesson because the alias is now available for order buys. And we want this in descending order. Running this, looking good.
4:45:05
All right, now this is for all jobs and regardless of where you're working, right now we're seeing Python SQL, AWS and Azure up at the top. Let's actually get into filtering this and remember where statements or where clauses are going right underneath your joins. And specifically, we want that job title short equal to data engineer. Along with, in my case, I wanna look for remote work. So we're gonna do once again the job postings fact table, that job work from home column.
4:45:32
Specifically, we wanna specify that to true. All right, let's go ahead and run this, see what we got. All right, not too bad. SQL and Python are still topping the list. Along with AWS, but now Azure Spark and Airflow are up there all in the top six. Just going back to that problem statement and checking it over to make sure I get all the different attributes. Yup, got the top 10 in-demand skills, folks on data engineers and remote job postings. Now, what I like to do after verifying
4:46:00
that I do have the correct query, I just go ahead and copy this right here and do control C on Windows or Command C on WAT MAC. And then down here underneath it, so that way somebody reading this SQL script afterwards, they can see what the results are also. I'm gonna just go ahead and paste this in here with Command V on MAC or Control V on Windows. Now, take some time to look at this and see of any trends that you're noticing with this,
4:46:26
especially if you did something different for me, not to an engineer or in a different country. For me, I just went through and did a breakdown of the most demanded skills, specifically the top five, and then did any key takeaways out of this top 10 as a whole list. You don't even necessarily be this verbose and how you go through and define it, but this will pay off on what you're seeing here for when we go forward with building out our final readme.
4:46:52
Because in that readme, not only we're going over how the project is defined and what we were trying to do with it, but we have a section at the bottom talking about our analysis overview and what we found out overall. All right, so not so bad. That was our first query of three knocked out. And the next one, we're gonna be getting into analyzing those top paying skills. Like some time now to do look into those skills and what results you got and start putting together
4:47:17
what are some key insights, and you're gonna see some trends as we go through all this. All right, with that, I'll see you in the next one. Now that we've found out what are the most in demand skills, we now need to take this a step further and find out what are the highest paying skills for data engineers. Now to do this, we're gonna be focusing on, as we learned about previously, we learned over aggregation functions, we're gonna be using this using the median function.
4:47:46
This is gonna be the most accurate representation because we can be skewed by those high salaries. We're gonna continue for me at least, gonna be focused on remote positions and of data engineer positions. With this as a bonus, we'll just also include the frequency to not only just see like, okay, if it has a high salary, is there actually a demand with how many counts of a job it has? So inside VS Code in our one underscore EDA folder, I'm gonna go ahead and create a new file.
4:48:12
Looks like it's creating the wrong place. Okay, I'll start for now. We'll specify it's the 02, top paying skills, and it's a SQL file. I'm gonna go ahead and just grab this and then move it into the correct folder. If that's do I wanna move it? Don't show this again, yeah, move it. Now in here at the top, I wanna do the same thing and that I wanted to find in a multi-line comment what we're trying to solve. So I'll put in here what are the highest paying skills,
4:48:36
along with those three different focus areas, along with that Y. All right, now I don't like starting from scratch because this query that we're gonna be building is gonna be very similar to our last query, which is conveniently right next to it in the terminal. Anyway, I'm gonna go ahead and go back into that file, copy it, and then paste it into here. Now this previous query already aggregates by our skills and in this case does account,
4:49:02
but conveniently I've written this question in a way that we include the skill frequency to identify both salary and demand. So we're gonna end actually keeping this. We need to just add in a column to go in and find the median salary. Just as a quick refresher for the rest of this, we're gonna be using that same from statement of the job posting facts, doing two inner joins to connect our job postings fact to our skills. We're filtering it currently for data engineers
4:49:30
and job for work from home true. We may have to add another one with this and we're grouping by only by skills. And right now we're ordering by demand count. We may have to change it for that. And we're limiting it to 10. For this one, I'm gonna change it up a little bit. We don't have anything listed. So I'm gonna just say, let's go with 25. Okay, so let's add in first that median salary calculation. We're gonna be using that median function
4:49:56
and then Jape from our job posting facts table. We wanna use that salary year average column. For this, we're gonna specify this as the median salary. Let's get the terminal backup. Let's try to run this bad boy. And we get an error parser error. We know what parser is now. It's at our near account. And I can see following next to this. Oh, we're missing a comma at the end of this. Run this again. All right, not bad. Noticing with this.
4:50:23
We do have like the skills are organized by demand count. So the highest salary isn't at top. But looking at these values, they're looking pretty good. I'm not liking the formatting right for AWS and Scala. So let's actually fix this so they're all one consistent value. Well, anytime you need something like this, highly encourage you to dive into the documentation. 'Cause not only way you find what you want, but also something even better. Anyway, we know already that we do have
4:50:52
some numeric functions available. We went over operators very early on, and they follow under this of numeric functions. And then scrolling on down, we also have these that allow you to do other type of math operations. Now, with this, you could use something like add for adding X and Y values. It's an alias for basically the plus operator. But they have other things like trigonometry functions. And then this one here is what we're gonna be using
4:51:20
of round and it rounds to S decimal places. So with that round, we need to put this around in our entire other function of median, so I'll do round. And this does take an argument. So I'll do a comma and then I'm gonna specify zero for the amount of significant digits for this. And just double checking the documentation. Okay, yes, it takes the numeric. That's the Y value is what we wanna give the column. And then S the integer is the round to S decimal places.
4:51:53
Can't, and apparently negative values aren't allowed. So that's what we're doing, zero. All right, let's go ahead and run this bad boy. And not bad. Now, you notice there is, I did round it to zero. And it still does have one significant digit after the decimal place. And that's because it's of the type double. So it's not an integer, but it is a decimal. And so it's always gonna have this unless we change it to another type of data type, which we haven't covered yet.
4:52:23
But that's coming up in part two of this course. Anyway, just to show that I'm not crazy. And that's what I actually mean. If you put in one in here and then run this same query, shift enter, you notice now it's not all zeros anymore. We actually have, you can see this has three and this has five. I think that's just too many. I don't really care too much about that. So I'm gonna change this back to a zero and then run this. All right, so we're almost there.
4:52:48
But we haven't actually calculated that first one of what are the highest paying skills for each deferred data engineers. Everything else though, we have hit on for what we need to do. So in order to do this, we need to change what we're actually ordering by. And that's gonna be by median salary. Pulling back up the terminal by pressing control tilde and then running this again, pressing shift enter, bam. All right, not bad. This is looking good.
4:53:15
It looks like Rust is up there at $210,000. That's been blowing up more recently, especially on social media. So I can definitely appreciate that one. However, as I'm going down the list, right, I get down to like this one right here and it only has a demand of one. And then down here even further, we have these that are like, oh, five and nine. So like not very popular skills at all. I think this list, I would be more comfortable removing values, just me looking at this
4:53:50
and interpreting it, seeing values that had greater than a hundred as a demand. So we need to do an aggregate filter. In order to do that, we have to do a halving clause. And for this, we'll say that the count needs to be greater than 100. Yes induct DB, you can list the alias there, but right in this generic for any database to use to demonstrate your skills. So we're gonna leave it like this. All right, let's go ahead and run this. All right, like in this list, it has skills
4:54:22
that have a lot more demand to it. So I think that the salaries mean a lot more with this. Anyway, we can take this a step further if you wanted to. Specifically, I'm gonna just copy this here, doing Command C or Control C if you're on Windows. I'm gonna go to something like chat GPT and I'm gonna say, hey, plot the top 10 skills by salary. And then I'm gonna just paste in that table here. Looks a mess in here, but chat GPT will figure it out. So this graphed it for us and we can see
4:54:50
that the highest, well, the top 10 at least, are ranging from around 160,000 up to over 200,000. And with this, it also gave us some takeaways with this. Specifically, a lot of these skills are niche and also hard to learn, but because of that, there's more money there. And interesting enough, there are two most popular skills on the top 25 list of airflow and Kubernetes, very valuable skills for data engineering to know. Have massive demand, but don't even crack the top 10.
4:55:22
So now back in VS Code, I'm gonna go in and put together some insights that I've extracted out of this. And I suggest for you to do the same. And not only am I gonna put my insights in, but also I'm gonna put those values in here as well. So I'll copy this and I'll put this down at the bottom. All right, so overall, this is looking good. Got my question up at the top. My query now built, and also that results said and how my analysis goes.
4:55:48
I'll go ahead and save this by pressing command us or control us if you're on Windows. All right, only one more query to go and this one's gonna get even more complex. With that, I'll see you in the next one. Now that we've identified the most highest paying skills, the most highest paying, I don't know if that's right. Anyway, we're not gonna need to get into our third query, which is gonna be what are the most optimal skills for data engineers, balancing both demand and salary.
4:56:21
Now previously in our results, we've gotten both the skill with its associated median salary, and then also its demand count. In that query, we ordered it by the median salary, but we really need to figure out a way to order this that takes into account both the median salary and the demand count. So we're gonna develop sort of like a ranking column. I'm not really sure how we're gonna do this just yet, so hopefully we can figure out something as we go along.
4:56:49
And for this, we're gonna be focusing on remote data engineer positions like mine, like I wanna do, feel free to adapt it to your situation. All right, so inside VS Code, we need two, in our one underscore EDA folder, we need to create a new file. We'll call this 03, optimal skills and make sure that it's a SQL file. Up at the top, I wanna carry over this question that we're gonna be answering with this. So I'll point that question along with those other two attributes
4:57:14
that we wanna meet with this query. Now this one is gonna be very similar to our last query, right? Because we're using the skills and the median salary. So I'm just gonna go in and select that last query and we're gonna use that as a starting point for this query. So jumping into this, we need to create a ranking column that combines demand count and median salary to identify the most valuable skills. Now, one disclaimer real quick about how we've been calculating this demand count.
4:57:45
Well, let's go ahead and run it in our terminal just to show and I'm going to run this query. Okay, so we've been doing that demand count and this has been all of the rows because we're doing star. Now, if you recall, we actually, we're analyzing the median salary based on a skill but not all skills necessarily. Not every job posting has a median salary with it. And we can show this by, I'm gonna insert another column right here. I'm gonna be using the count function
4:58:18
from that job postings fact table. We only wanna count the salary or average column which remember account on that column is gonna count all non-null values and I'll call this as corrected demand count. So selecting this all and then running it just came out of hot mess. Let's, okay, not too bad. So what we can see is, right, that these values go down and actually, I mean, Rust and Golang demonstrate this pretty well, right? We went from 232 to 912
4:58:51
which is a pretty significant jump proportionally wise. I mean, it's almost five times bigger. But when we sort it for those with actually that have a job postings that have a salary value listed, we're going from 23 to 39, not even double what it is. So I don't, whatever multiplier we use for this, I don't want to inflate some of these skills that may just have a bunch of postings but they're not backing it up with a actual and actual salary.
4:59:27
So what we're gonna do is we're gonna stick now, we're gonna stick with this corrected demand count and we're just gonna replace that demand count up here with that and I'm just gonna rename this to demand count and then just double checking to make sure that this query can work properly. All right, it's still working. I'm gonna show it all of our correct values. All right, so now we need to create a ranking column and something that we could do is we could create
4:59:54
a multiplier column, and what do I mean by this? Multiply column, it's basically we're going to multiply median salary, time the demand count, and hopefully maybe we'll see how this actually goes. We'll rank it according to that column and we'll see what kind of, what we're getting out of this. This is just an experimentation. All right. So I'm going to put a comma new line and we're going to take that median value and multiply
5:00:21
it times the demand count and we'll give this the alias of optimal score, all right. And for order by, we'll change this down here. We can list alias and order by. So we'll list optimal score in the setting order. Okay. Let's just check this out, see what we get, not guaranteed it's our solution. All right. Cool. What we're getting is, well, optimal score over here, so absorbently big number. Dude, you notice any trends with this?
5:00:49
Well, the key thing I'm noticing is, is that demand count is overwhelmingly driving this ranking right now, because if I was even looked down this list, it looks like, except for a few that are out of order, that the demand count is really just, we're really, what we're doing is we're ranked in by demand. And that's not really what I want to do. I want to get something that's like a bridge between demand and median and helps clean this up a little bit.
5:01:18
Quick note, not related to demand count. We could also list a using duct as you're going to find out. I could also list the aliases in here in the select statement, and then pulling up the terminal running shift enter. It still works the same, but this is only allowed in certain databases. DuckDB, of course, loves being friendly, but like MySQL and SQL light also allows it, but most other ones don't allow it because that we're not going to use that in here. Get that out of here.
5:01:49
All right. So rerunning this query, let's get back to on hand, this demand count. Now if we actually plot this, it has a really interesting characteristic. So I had chat GBT plot this, and if you look at it, we can see that, especially when we start getting up to the higher up skills, it's not necessarily like a vertical, a smooth line going up to SQL Python instead, it's almost like these skills go exponentially. So here's what it did.
5:02:20
I had chat GBT plot those top 25 skills by the main count from lowest to high. And it basically does show this, right, it's a almost an exponential growth with this getting to that Python and SQL. Well, there's a function we could use to help smooth this line out. And that function is natural log. So on the x axis at the bottom is basically like a raw count going from zero to a thousand. Now it does have this sharp up swing, but then after we get to around 100 to 200, it
5:03:02
then steadies off and goes at a more steady value. Natural log is great at compressing large values. So extreme numbers like that of Python and SQL don't dominate. Now it's important to know this that the scale is going to change when we apply this, but the order is not going to. So I think this using natural log is going to be a better option for us. And conveniently, if I go to DuckDB to look at the numeric functions available, we can
5:03:32
see we have this one that computes the natural logarithm of X by just using LN. So let's just first convert that demand count into the natural log of it. So I'll define the natural log function and then inside of it, put the count of those non null values of the salary, basically job postings with salary listed for the associated skill. And we'll call this LN demand count. Now something's going to happen when we go to run this, well foreshadowing here, when
5:04:03
if I run this, I'm going to get this error out of range error, cannot take the logarithm of zero. And the problem is this, I'll actually show it in a query underneath here. I'm just going to run a simple select statement and then I'm going to do a natural log of zero. Whenever I try to do this, I get this error once again out of range, can I take logarithm of zero, basically there's a zero value within our count at one point for one of these skills, and so it's causing it to do that.
5:04:32
You have to have values of one or greater whenever you put them into the natural log here, I'm running it with that. And whenever I even use the value of one in the natural log, it gives the value of zero. And after that, it only increases for there, you're doing it for the natural log of 100, that's 4.6. All right. Enough of that history lesson, so the problem is right now is that some of these counts
5:04:57
that we're doing in order by for the skills, some of these skills have a count of zero. Well, we need to just remove those skills altogether before we even stick it into the natural log. And so because that we know because of order of execution, specifically step two is our way or condition to filter, and then three is our group by that performs that aggregation. So we know if we filter those skills that don't have any associated salary associated with it, we will therefore get it out.
5:05:29
And then by the time it gets to step five for that performing the natural log of it in the select statement, that's not going to be an issue anymore. They're probably like loop. You're crazy. I don't know what you're saying. Well, let's actually demo it. And I think it will make sense. So inside of here, instead of doing salary or average for this one, I'm just going to do star and that's going to be selecting all the columns of the job posting fact.
5:05:52
But I'm going to filter it here for the salary or average column of the job postings fact table to remove all null values. Now, whenever I run this, we actually get values back because inside of this natural log, there's no skills with a without a associated salary. So there's no skills with count of less than zero or no skills with the count of zero. Anyway, I like to maintain the same format throughout. So anytime we're doing this in the count, I'm going to replace it here.
5:06:29
And then also here in our optimal score. So just moving all the sides so you can see it a little bit easier. I updated it in three different places. But now, now that we have the natural log demand count, I need to take this portion and we're going to replace this in our optimal score. All right. Now I'm going ahead and opening terminal and then shift entering this into here. All the values aren't showing. So apparently I got to move this over more running this again. All right.
5:06:59
We're getting everything. I do want to do some data cleanup before we get into this actually evaluating what's up at the top. First up is there's just too many decimals and too much going on here for this. First of all, that natural log demand count. We only really need one decimal place. Also we no longer need just this demand count anymore. So I'm going to go ahead and just coming out the main count. If you want to leave the main count, you can.
5:07:25
And then for the natural log demand count, I'm going to do a round function as we've done previously. And I just want to have one decimal place for this one. As always, I like to just make sure that it's operating correctly. So I'm going to shift enter and I did route whoopsies, selecting again. All right. So this is looking better now into the optimal score. Right now these values are, oh my goodness, it's up into the hundred thousand almost close to a million.
5:07:53
These numbers are really arbitrary. So I'm just going to convert it to a decimal and make it as small as possible. So this is just me, you could do whatever you want, but I'm going to wrap this entire thing in parentheses and it was getting up to a million. So I'm going to divide by a million. Remember we can use underscores between the zeros and ones to help show where the separators are when it actually goes to evaluate this query is not going to look at it at all.
5:08:22
Now doing this, just to show it, we're on shift enter, all right, looking good. They're all basically decimal places because of that, I'm going to just do a round on this and then comma and then we'll do two decimal places. Now running this, I did round again, try this again, shift enter, all right, this is looking good and I can actually bring this over closer to here. So now actually getting into evaluating what is that final output. We see that Python and SQL aren't on top of the list.
5:08:59
Instead we have Terraform, which is a popular infrastructure as code tool appearing higher on the list. And so, I mean, look at that salary for that, it's at 184,000 compared to what Python and SQL is 135,000. So in my mind, that makes sense of an optimal skill to learn. Other ones that are appearing on this list as well, just looking down this natural log domain count with this 5.7 is Kafka, which would have been lower down and also air flow.
5:09:32
I'm realizing now taking off that count of just the regular count is sort of confusing. So I'm going to just bring it back for a second to show what I mean. Anyway, Terraform right was only mean two, almost 200 jobs had it compared to almost 1100 for Python and SQL and still got up to the top. And then air flow was only around 400 and Kafka is around 300. So pretty not a bad multiplier that we just came up with for an optimal score in order
5:10:01
to uncover some skills that we may not have noticed if we hadn't have done this type of manipulation. I'm actually going to leave this for the final results with that demand count index. I think that'll be good. So now as always, I'm going to go ahead and copy this down and put this into a multi-line comment and then also take some time to go and actually break down how, what this means for these scores and maybe going by those top 10 skills.
5:10:28
That's what I did here and get some insights from this. We're going to be using this to build our read me out for final projects for the cover page for this. All right. So we just wrapped up our three main queries for our data project. Our next step is to get into building out that read me. So we have a cohesive story to share about this project that we just built. All right. With that, I'll see you in the next one. All right.
5:10:59
For this build out of our read me, I'm going to assume you have no prior experience with building this out. So the first half is going to be with the importance of a read me and also some basics of building it out. And then from there in the second half, we'll get into actually building out for our project. So going into GitHub to just go briefly over what our read me is again. What's cool about it here on GitHub is that if you have a read me and you navigate into
5:11:29
a folder or whatever, the read me will automatically display below here. Anyway, in our case, a read me is a great way of showing an overview of what our projects are about and what it does and what the heck it even exists. I made this for the perspective of a hiring manager coming in to maybe verify your skills. And so that's why I have up at the top executive summary for hiring managers and have some key areas in there for where they may be looking at.
5:12:02
Additionally, I go into our problem, adding context about the data warehouse that we are working with and then more into what hiring managers care about the tech stack we're working with, where they can find all their queries in the repo and an analysis overview and then their skills demonstrated. Read me's are everywhere. The projects, if you remember, at least for those Mac users, we installed Duck Man, which was a duck DB version manager inside of Mac, it also windows.
5:12:30
Anyway, scrolling on down inside of this project, we have the read me at the bottom and it has a brief overview of what's going on here, how to install it, its different usages and then references. And then I have a read me for everything, such as my Python for data analytics course. It goes through and explains what this repo is about for the course and how you can actually navigate it. So read me's are super helpful and is like a front page of your project or wherever you're
5:12:58
working on. Now let's get into some basics of markdown and what do I mean by that? Well, whenever you're actually building out these files, as you can see in here, I have some different images. I also have things like links, I'll link me to the queries. Anyway, this is actually an HTML rendering of what is in the markdown file that dot MD file. I did a side by side. This is what it actually looks like. This is the actual text that I can actually edit inside of here.
5:13:33
For example, if I want to change the title to just EDA, I could do that and it's going to change in that HTML rendering of what the preview is of the read me whenever we uploaded to something like GitHub. So we have all these special characters and basically it's like a coding markup language that we can use to format this read me. So let's now go through in our project, create this read me and start getting familiar with
5:13:58
some different syntax we can use while building out this read me to try some different things in it. All right. So inside of our one underscore EDA folder, I'm going to create a file and in this it's going to be a read me in all caps and then a dot MD and then MD stands for markdown file. So you can have markdown files that aren't necessarily read me's like I could rename this to something like help dot MD and that's going to work just fine too.
5:14:26
But for the purpose of this, we're going to name it read me because also GitHub picks that up. I'm going to close this off to the side and I'm going to open over on the right hand side. You see this open preview to the side and we're going to try different things. The first syntax is this using a hashtag. You can write a heading in this case one hashtag is heading one note. You have to have a space in between there. It's not going to work.
5:14:49
I also have things like heading two and heading three, if I just write in here, this is normal text. However, if I want to bold it, I need to put two asterisks around it and then two more to close it off and it's now bold. Notice it's on the same line and this is because this is a weird little thing with markdown that you have to do. At the end of a line, you have to space over two times for it to go down and pop on the next line.
5:15:15
Once it's bold, you can do italics with only one asterisk. Once again, I can't forget the two spaces, so it keeps on doing that period. Why does that keep doing this? Come on, play a stop. Then also, if we wanted to maybe put in code, we would use a tilde and then write this is code. Now, if we wanted to clue things in bullet points, we could just do attack and say this is bullet one and then this is bullet two. Or we can do numbering of number one and number two.
5:15:45
If I want to include a link to something, I would do square brackets for the link text and then in parentheses, I would put what I wanted to go to. So I'd do HTTPBS and then something like Google.com. Now there's a link going on right there where I click it, I get directed to Google. It's Christmas time, so nice little logo. Now also besides links, we can do images as well and that is a similar formula as links
5:16:12
except we're going to do an exclamation point in square brackets is the alt text and then in parentheses is the location of the image. Now this image can be located online, which let's done with that first specifically if we navigate over to our project, I have this up here, this diagram of our what's going on here. This we can actually use. If I right click this image, I can copy image address and then inside of VS code, I'm going
5:16:44
to go ahead and just paste this with command V and I can see that it lists the GitHub address that it's located at and this image now is inside of here. So you're free to use this image, however, you don't know if I'm going to go crazy and move with this image and replace it. And so you need to be able to, I would have this image and just download it instead. So I'm just going to right click this and go to save images and then from there, I'm
5:17:08
going to navigate into my data engineering project and I'm actually I'm going to create a new folder inside of here and I'm going to call this images. So go ahead and create it. So images is inside of here and then we'll just keep this name right now of project one EDA save and now navigating into here, I open up the explore. We have this folder now of images of this right here of the actual image. What I'm going to do is I'm going to right click it and I have two options here.
5:17:42
I'm going to copy the path or copy the relative path. Let's go with copy the relative path first inside of here. I'm actually going to remove this whole hyperlink thing and I'm going to write, we can paste and link to a file in here and the relative pass is just the images. However, if I go in here and I right click this and select copy path and say the paste that in instead, that's going to go all the way up into that main directory. That's not necessarily what we want.
5:18:09
So that's why we went with that copy relative path. Anyway, as we can see over to the left hand side, this isn't working and that's because sorry, going back into this folder right here, right? This read me is located in the one underscore EDA folder where right now the image is located in the images folder. So it's not seeing, the read me is not seeing this because it's trying to navigate from here into the images. Also I'm noticing right now that I have the wrong path inside of there.
5:18:41
So I'm going to copy that relative path again and paste it into there. So it's not, it's not located in the right location, right? Because it's trying to go from here into images. What it needs to do is it needs to go back a directory and then in the images. We can do this similar to going to like a CD dot dot, we're going to do a dot dot and then a slash. Whenever I do that and press the dot dot slash, it goes back up and then VS code does a school
5:19:06
thing where it then tells you, oh, you can actually navigate into these three different folders now that you have access it. Anyway, we already have the rest of the path and the correct location of the image. I would then also do something like change the salt text to some like project one overview. One last thing that I find myself using from time to time is code blocks. So previously we talked about, we can do inline code like this.
5:19:30
But also I like to have physical or big old code blocks. So if in the case, if I wanted to put in this query into a read me, I could copy it. And then inside of our read me, I'm going to close this out of the way. If I were to only do this one dash and then paste it in and then another dash, it makes it look, well, okay, this is, this is hideous. Instead, what you do, you do three tildas, and then I move this down the next line and
5:20:00
to close this out, I'm going to do three tildas to close this out. Now right now, there's no syntax highlighting on this indicating what the different keywords and everything are. I can actually specify what type of code this is right next to these three tildas by saying SQL and bam, now it colors it. Now there's a bunch of other stuff you can do with Markdown as well. So I highly encourage you to check out this Markdown cheat sheet that has a lot of things
5:20:31
that we've just covered along with some other features as well. So you can get some ideas from it. So let's get into building this read me out and for any of these different sections, you can feel free if you don't feel as applicable, you can skip it and you do whatever you want. I mean, you're reading me. This is what I would recommend, especially for somebody that's a job seeker in data engineering. The one thing we are going to keep on this is this image that we previously did.
5:20:57
So I'm going to delete all the way up to heading one and then also delete this SQL code underneath here. I'm going to give it the title of exploratory data analysis with SQL and this is a job market analysis. Then we're going to have the image there, but we need to now go and give a brief intro. So I say this is a SQL project analyzing the data engineering job market using real world job postings. It demonstrates my ability to write production quality analytical queries, design efficient
5:21:25
queries and turn business questions into data driven insights. Now next, let's get into the major sections. We have our executive summary, our problem in context, our tech stack and analysis overview, and then finally SQL skills demonstrated. So now we're going to go through and just build these out. I'm not going to tell you word for word and read every little thing. I really encourage you to go through that and do that, but these are the main sections
5:21:52
I do highly recommend that you do include. So for this, I go into four main areas, first is the scope analyzing or specifying that we only did three queries. Next is the data modeling involved and specifically how I joined tables. Then from there, the analytics, what aggregations, filtering and sorting we had to do. And then finally, the outcomes, what I'm going to give at the end of this of what we found out from the analysis.
5:22:17
Now, I also like including right in this section links directly to the queries that they just want to dive right in does make sure that you can write SQL. So if we wanted to link to something like the top demanded skill, well, we're located at the read me right now. So we just need to select that right there. So it's pretty simple, right? You put in square brackets, the text that you want to see such as top demanded skill query.
5:22:44
And then in parentheses right next to it, I'm going to put the file itself. So just typing in or you can copy and paste this path or whatever you want to do. Now it has that link in there. And so whenever I go to click on that, it navigates me to the file. Anyway, so I went through and listed these out, ended up changing to the SQL file itself so they know what they're clicking on. And it gives the SQL file along with a quick little snapshot of what we did in that query. All right.
5:23:12
Next, I want to focus on is problems and context. In this, I specify that, hey, job market analyst, the thing that we did the job of. You got to enter questions like most in demand skills, highest paid skills, and best trade off skills. And then I want to go in to demonstrate that we have a familiarity with data warehouses. I wanted to them then provide them with a diagram of the data warehouse. Now you don't have an image of it, but feel free to still mine for my GitHub repo.
5:23:39
And so just scrolling on down in my repo to that data warehouse image, I'm going to go to save images and just save it right in that images folder once again. And then to include the image, remember we need to use an exclamation point, square brackets for the alt text, give it the name of data warehouse. And then we need to go to the image. Remember, we're in this Rimi right here. We need to go up a folder into back into that SQL data and a legit SQL data engineering projects folder.
5:24:07
So inside of parentheses, I'm going to do a dot dot and then a slash. Now I have the images folder. I'll use the down arrow to select images. And then I'll use the down arrow, select one, two data warehouse, PNG, and voila. Underneath this image, I then go into a little bit more detail explaining what the fact table is the dimension tables and then the bridge tables. I used whenever calling out these tables, I want to make it into a code block.
5:24:34
So I do put in line or sorry, tilde characters around each one of the tables to make me make it really stand out. All right. So that is the tech stack that we use and we really need to call out all the different skills we've used so far and that we're going to use. So I call out things like our query engine of DuckDB, the language of SQL, our data model, how we actually had to go into a data warehouse. And then even the things that we're using now, right, so things for development.
5:25:03
So how we're using VS code for SQL editing, how we use terminal to work with all of this. And then how we're what we're going to do actually in the next lesson on version control, using Git and GitHub. All right. Next up is the analysis overview. And this is really going to pay off for all that homework you already did looking at all those different queries and what we got from it. With this, I first started with the query structure.
5:25:30
If they didn't notice it up above in that hiring manager portion, I re put all the queries into here and link directly to it and then what we did in each one of those three queers. Anytime you build out any of this stuff, it's good practice to make sure that the link actually works and navigate you to that file. And then underneath this, I did key insights and this is really extracting out all the things that we've found out previously and this not only includes those insights about
5:25:56
how high of a demand SQL and Python are, but also whenever we got into our optimal skills finding out about Terraform about how it was a optimal skill. All right. Last section on SQL skills demonstrated and this project is primarily focused on SQL. So that's where I wanted to call out specific stuff that we did. I broke it up into two sections. The first one, which is a feel a little bit more applicable to data engineers on query, design and optimization.
5:26:22
We go over not only complex joins, aggregations, filtering, but also sorting and limiting. And then after that, we go into data analysis techniques, talking about grouping control lock the JIC, the different functions we've used, calculated metrics, the having clause and how we handled null values. So all very important attributes. All right. Now just scrolling this over, making sure everything appears like it should. It's looking good. I'm liking this. All right.
5:26:51
So if you're liking it so far, make sure that you command S or control lesson. If you're on Windows to save it. All right. Now that we have this read me built and we have our three sec queries done, we're pretty much done with this project. We're going to take a light segue from this project and get into learning about Git and GitHub because we need to get built up this requisite knowledge on how this tool actually
5:27:13
works or tools work before we even upload our project into there to share this project with the world. All right. With that, I'll see you in the next one. All right. So we have our project all built. And now we need to actually get this project onto GitHub to share it with others. So this lesson here is going to be going over two important concepts of Git and GitHub. Now Git is the version control system that we're going to be interacting with through
5:27:46
the terminal and GitHub is where we're actually going to share in our project. The main focus of this lesson will be on Git because there's a lot of different terminal commands that you need to know about in order to manage a project properly within this version control system. But the end of this lesson will have pushed our project up into GitHub and it will be available to share. Now let's get into this and we're going to be breaking Git and both GitHub down first to start.
5:28:20
So you have a background on this. To be clear, Git can be installed locally on your computer and also Git is what is behind the scenes working to manage and track all the changes within GitHub. So Git is a free and open source distributed version control system designed to handle a lot of your different coding tasks. This not only allows you to keep track of snapshots of your project as you build it out,
5:28:50
but this also enables you to collaborate with others because of this version controlling. But how the heck does this actually all work? Well here I am on my local computer and this is the folder that I'm building out this engineering course in. Specifically, I have it set up using Git. And you can't see this right because all you can see is the course and then these three different files. But there's actually other files.
5:29:16
Now you can list all hidden files on a Mac by doing LS-A, but I'm going to just show you with this shortcut of command shift period. And what this does is that shows the hidden files inside of my folder right now. Specifically, there's a dot git folder in there. That dot git folder is keeping track of all the different changes of this project within here. Now we're not going to mess with the contents of this.
5:29:40
I just want you to understand that this is how it's actually being tracked in here. So even if you're working offline on a project, it still can be tracked. Now there are other options for version control systems, but they're nowhere near as popular. Last one. time that Stack Overflow did a survey on version control systems back in 2022, get one by far with 94% of the votes. It's so popular in fact that they don't even do this
5:30:11
poll anymore and 2022 is the last time they actually pulled on version control systems. Now that we understand Git, let's dive into Git Hub. Git Hub, I like to think of it as like the Facebook for coders. Anyway, it's a hosting platform for Git repositories where we can then you can share with others but also work with others. As I mentioned and showed you look at my machine of SQL Data Engineering course, I also have it here inside of GitHub. And because it's
5:30:38
up here, it then allows me to work with others specifically like Kelly Adams has helped me out with building this course. She's contributed to it. We can collaborate on it together because it's here in GitHub and she can access it from where she's at. And like Git, this is completely free. Now going back to that Stack Overflow survey but from last year or actually this year of 2025, for code documentation and collaboration tools, GitHub is by far also number one at 80% of the
5:31:06
choices. Interesting enough, something that we just learned in the last one, markdown file, is that number four? But back to GitHub, the closest competitor to GitHub on this list would be something like GitLab. So now that we understand the basics of Git and GitHub, what are they actually managing? Well, for both their manager repository or repo, it's like a personal library for your project where you can keep manage and record every change to your documents and files.
5:31:37
On GitHub, this is my remote repository. And it's called remote because you can access it remotely like other people like Kelly can access it. And then locally managed with that. Git file, we have our local repository on my computer. Now inside your computer, it does get a little bit more complex because you can work inside of here. So besides a local repository, you have two other areas of a working directory and your staging area. The working directory is where you actually
5:32:07
edit your code. The staging area is just a temporary area where you prepare your set of changes that you're going to then commit to your local repository. Don't worry too much about those, we're going to be diving into each one of these and you get very familiar of what are those different stages of keeping your files. So now we're going to get into installing Git, but we have to do one prerequisite for those Mac users, specifically for them, whenever you go to install Mac, the top
5:32:35
option that you have to do is for using homebrew to install Git. And homebrew is a package manager inside of Mac. Windows users have something similar of WinGet. Also, Windows users have already installed Git, which you can verify in your terminal by typing in git and then dash dash version. We installed Git, remember, because we had to install git bash of this terminal to run bash inside of here. Anyway, the next section is only applicable to Mac users in order to install homebrew,
5:33:09
and then we're going to get into setting up your Git environment. So for my Mac meet users, navigate over to this URL for Git dash SCM, we're going to go into install for Mac. And in this, we need to install homebrew if you haven't done that already. Now you can check if you have homebrew installed, and then you don't need to follow this by typing in brew and dash dash version. In my case, it says command not found brew. So it's not installed. So with this, I'm going
5:33:39
to first go to this link for homebrew right here. And homebrew, as I mentioned, is a package manager for Mac OS. Think of this as like an app store, but for developer tools. I've been using it ever since I've been a Mac lover, and highly recommend installing this and actually you need it to get so you don't have an option. Anyway, to install homebrew, we're going to just copy this command right here. And then inside of our terminal, I'm just going to go ahead and paste it. Anytime you're
5:34:06
running any commands, you do need to be aware of what it's actually doing. Specifically, I can see it's using bash to then execute this command of a curl command. And this is used to make a request to a URL specifically, this is calling an install.sh script, which is actually conveniently in get up. Anyway, I know homebrew is reputable. I trust this. I'm going to go ahead and run it. If it's asked for a password, you just enter in your computer password, you'll ask if you want to confirm by
5:34:38
installing these directories, all you're going to do is press enter. Now homebrew is done installing. It's important anytime you install any of these softwares to read through here, because sometimes there are, well, as this case, next steps. And if we don't do this, right, so this says run this commands, you're trying to add homebrew to your path, we've talked about path a little before, but if I were to just run brew right now, it's still not found, it's not in our path. And so we
5:35:01
have to run these commands in order to do that. And as always, you should understand what this is doing. This is adding contents to our dot z profile file. Now, in a previous lesson, we did create this file, remember, I run LS dash a, and this has our dot z profile that I recommended that you install or create at that time, if you haven't done that yet, you need to run touch dot z profile in order to get it here in your home directory. Anyway, we do have it. So I'm going to go ahead
5:35:33
and copy both of these commands, command C, and then I'm going to paste them right in here, place enter. Hey, and even if I wanted to, I could cat dot z profile to check it out, see what it looks like. And then now whenever we run brew, it actually has or list the commands, I would have to do brew version, but it does tell you how you can use brew by just typing in brew. So home brew is installed. Cool story. Let's now install get right. And for this, all we have to do
5:36:04
is brew install get. So internal, I'm going to type in brew install get, and bam, it's installed. If you didn't watch the Windows version of this of installing it, that was a mess. There's so many things and clickers that you have to go through. And this for Mac makes it super simple. So I love home brew. Anyway, when you verify get is installed by typing get dash dash version. And it's installed
5:36:25
with 2.3, 9.5. Now we're going to be pushing our repo that's tracked via get up to GitHub. Because of that, we have to put in our credentials, we're going to be using an HTTP protocol. And it says every connection needs a username and a password. So let's go ahead and set that this is going to be the same whether you're on a Mac or a Windows, we're going to start with that get command, and we're going to type in config. And this is going to set our configuration up. And this is like
5:36:54
our settings for get. And then we want to apply this globally, basically, we want to apply this to us. Now we're going to set a few different things. The first thing is the user and then dot name, and then fill in your name with in double quotes. So I'll put in loop brews. Now we need to put in our email address. So we use get config and then specify global. And then for this will be user dot email. And then within double quotes, put in your email address, and press enter. Now one
5:37:24
more setting I want to set right now for globally. And that is what editor what text editor is going to use by default inside of terminal whenever you're doing any of these terminal commands. So I'm going to do the core dot editor. And for this, I'm going to specify it to be nano. And I go ahead and run this. If you remember, nano is that text editor that we've been using previously, to edit files like our bath bash profile. By default, it would set up you to use the
5:37:53
VIM editor, which that's a lot more trickier here. I'm in it right now. And you're like, how the heck do I get out of it? You have to remember to do semicolon and then wq. And then press enter for you to get out of there. I just like nano a little bit easier to tell you what to do at the bottom. Anyway, if we can confirm any of these changes or what we've listed so far by having get config and then list and then show origin. And we press enter and run this.
5:38:18
So now let's get into working with get using these terminal commands. And so we've already completed step one of the get install and setup. We're now moving on to the next parts we're in here where we're going to be creating a local repository. And then after we set it up locally, we're going to shift to GitHub to create our remote repository. And we're going to then sync in between them using push and pull methods from get. So for this section, we're going to be focusing
5:38:50
on working on get here and using those three different areas of a working directory staging area and local repository. So to get started, we need to run our very first get command. And in our case, this is going to create an empty get repository. You can also use this to re-initialize an existing get repository, but this is more used for maybe a corrupt files, or if you need to reset some of the default settings. For this all, I'm going to be doing
5:39:19
this from VS code. You can do it from the terminal if you want. But mainly I want to do it from here because well, I'm a little lazy in that this takes us directly to or navigates us directly to where our folder is, or I'll do pwd. I can see that we're in my correct folder for where the project is. Now with this command, we're going to be creating that hidden folder of get. So I do want to show right now that there's nothing in here, I'm going to do a list all. And inside of
5:39:44
here, I see, well, the folders over here to the left hand side of eda images lessons. Also, they have a dot and dot dot in here, and that just directs to dot is the current folder. And dot dot as we've learned previously, the parent directory. Anyway, there's no dot get file in here. So when if I run get init, because we're going to now initialize this get repository. And I can see on here that this initialize an empty get repository in my folder. If I do ls
5:40:12
dash a now inside of here, I actually see this dot get folder. Okay, I do want to talk about this real quick. This is only applicable if you have greater than get 3.0. And this is referring to branches. Now, branching is something that I'm going to cover before the second project. It gets a lot more in depth. But the core component to understand about branching is, is that well, you can create different branches within your get repository to basically work on
5:40:43
maybe things like fixes releases or develop a new feature. Basically, they have different names. But the main branch that you work on is this one up here called main. And it's the default branch within a get repo. Basically, it's the most important branch. It's the branch that everybody works off of whenever you're collaborating together. But right now in my repo, you may not be getting this message because you may be using get 3.0 or greater. Right now in my repo, because
5:41:15
I'm not, it has it as master. So that main branch is master. Anyway, to avoid confusion for future projects that I may do within get in here and maintain consistency, I want to switch this to main. So it gives me a command. These are both these commands are not commands, you need to memorize. Well, the second command, you need to memorize eventually. But the first command, you don't need to memorize. I'm going to copy this. And I'm going to paste it in here. And what it's going to do,
5:41:42
it's going to set the default branch name whenever we create a new and if we ever do get in it again, it's going to set it as main. So I type mine in and I went ahead and executed this. Now what I want to do is with our current branch, I want to change the name master to main for our current one, because we already did get a net. So we can use this command up here to change that name. So I'm just going to type get branch dash M and then change it to main. Alright, so a little bit side
5:42:14
track, but we just ran get in it. And that was the main command for you to get out of this for creating an empty get repository. Now that we have this repository initialized, we can now move on to more of the commands that are a focus of this lesson. And the main commands are these five, well technically for here, get fetch and get merge is the same thing when combined together as get pull. So that's why I say only for anyway, we're going to be walking through these as we go through
5:42:44
this. Now although we're going to be using git commands, I do also want to look, they have this area there's panel off to the side of source control we can select. Right now it's telling me to download get for Mac OS. I've already done that. I need to basically just reload this. And now the source control pane has reloaded. Also, I'm not liking this. I want to expand this more and that's making my terminal small. So I'm going to rearrange the windows real quick by right clicking
5:43:08
terminal and changing the panel position from right to bottom. And then I'm gonna go ahead and just run clear to get this all cleared out. Alright, so pretty cool right now in that this source control over the left hand side is going to be a little bit of a visual aid for us. Now I do want to stick to just using git commands, but this definitely can help us visually understand what's going on as we're moving through this. Right now, this is visually showing us that there's changes on all
5:43:37
these files. Basically, we just set up get init. So of course, there's change in all these files. And these files correlate to all these files inside of here. Also, VS code pretty neat. It does this green coloring right now to telling you that there's untracked changes on this file. And what this is saying, these untracked changes is that these things are inside of our working directory and they have changes associated with them. So let's use git add and move one of these
5:44:08
files into our staging area. So let's type git add. And then I can select any one of these. I'm going to use this read me right here that's inside of our one underscore eda folder. And then start typing read me press tab. So we auto fills it in. All right, going back over here to so we can view what's going on here whenever we do this, I'm going to press enter for this command and it gets added and notice here it gets added to the stage changes. Now, besides just adding the read me
5:44:38
itself, I could also do something like add the entire folder. So I do one underscore eda, run this, and now all the contents of that folder are in there. So now I know that there's other files still in my working directory that have untracked changes in it. But just go with me for the time being, we have a few files specifically that eda folder in the staging area. Let's now get it into our local repo by using git commit. So with this, I'll type in git commit type enter.
5:45:08
And with this, a file editor or the nano editor comes up for us to put in what is going on here. So up at the top, I'm going to type initial commit, everything underneath here that has these ampersands in the front of this is going to be ignored. But it tells you what's going on here. It tells you, Hey, you added basically all the content from that one underscore eda folder. And there's untracked file still within the images folders and the lessons folders, which we can see
5:45:38
here for the changes, we'll get them in a little bit. Now that we've made those changes, we need to write out those changes by pressing Ctrl O and enter. And then now we can exit out this by doing Ctrl X. And then I see up here at the top that for our main branch, we had a commit. This is the number of the commit. And it was our initial commit. That was at least the name we gave for it. In it, it says that four files changed with a 291 line insertions. And then it gave us all these
5:46:07
different files that we committed to it. Visually over on the left hand side, we can also see that we have our first commit as we add more commits, they're going to stack up over each others and create like a graph as we go. So we just took untracked changes that were in our working directory, use get add to put it to our staging area, and then use git commit to get it into our local repo. So right now only our eda folder is committed to our local repo. Let's get everything else inside.
5:46:36
Now it would be burdensome to have to go through and add each one of these files line by line. So there's a shortcut instead. What we're going to do is going to do git add and then period. And this period, as we talked about previously, is a representation of the current directory. So everything inside of it is added to it. Then for there, I'm going to press enter. Now we can see visually that it updated to be a stage changes. But say you want to be a hardcore dad enthusiast
5:47:05
and not use this graphical user interface. Well, you can also use git commands to check out what the status is. And it's as easy as get status. And in this, it says, Hey, we're on branch main, and we have changes to be committed. And it has it in green, all the different files that we need to add into here, or the files that are now being staged to be now committed. Now let's say I went in and say changed the file contents of this file here of 1.13 project one intro for SQL.
5:47:36
We want to change this job country to job title short. I can see the file changes visually and VS code because as this white dot, I'm going to press command s to save it or control s on windows. Notice visually this changes from a green to a yellow. And then also whenever I go into the source control pane here, I can see underneath changes, it got added to it. So it is state in the stage changes, but also it's back in our working directory with changes in it. And if I run something like
5:48:09
git status on this, it shows me all those different files that I have once again. And it says, Hey, changes not stage for commit. It says we have this new modified one. Now I do want to add that so I can just do git add. And then I could specifically specify that folder. And VS code makes it simple because it allows me to syntax highlight, which one I want, you can also do git add period, but that's no fun. Also not wrong, you can do to get add period. Anyway, we now I can see visually that
5:48:40
everything's up to date. And even when I run git status as well. Yes, all the new files are in theirs. There's no untrack changes. Now that we're in that staging area, we need to get it into our local repo by committing it. So we're going to do git commit. This time, I'm going to give it an M flag. And this M flag allows us to provide a message. So we basically don't have to go in that nano text editor. If we don't want to, we can type the message directly. In this case, I'm going to
5:49:07
type this as second commit. Now normally you would want to provide more robust type of titles with this. And actually, with good practice, we're going to stick with that. So I'm going to change this to say, hey, add in remaining files. Let me understand better what's going on here. All right, I'm going to go ahead and press enter and making this a little bit bigger. I see my message up at the top of add in remaining files. It says seven files change 99 plus insert of the manual lines
5:49:33
that we inserted into this. And then it shows all those different files. Notice inside of here, there are no untracked files inside of here. So nice clean area. Also, we can see down at the graph, we now went from the initial commit to the add in remaining files. Now previously, we change this to job title short. And yeah, we saw that the file updated. But what if we want to actually see what actually changed? So I'm going to actually change this back to job country. And I'm going
5:50:02
to save it. I can see visually that we have an untracked change, because now everything's, everything's white. But this one is this brown color. If I run git status, yes, I can see that we modified this file of 1.13 for this project intro. Well, what we can do is another command that you should know. And that is git diff or checking the differences. And with this, it calls out the file specifically that we're talking about. And it compares it to what's in the repo or what's in
5:50:32
our current local repo. Sorry, I had that backward compares what's in our local repo to what's in our working directory, which is B even shows it here for the A is the negative B is the positive of what got changed in this case. As in a we removed job title short from what was in the query. And we added in job country. So pretty cool at showing the differences. Now you could also do this visually by going into the source control pane, and then just double clicking on a specific file. And it
5:51:06
will show you a little bit more visually better, what has actually changed in the file with the red being the removal and green being the addition. But unless you need to know the commands get diff, this is basically pulling get diff to provide this to you. All right, so anytime we do changes to a file, we need to go ahead and add them. So I'll just do a git add period. And then from there, we're going to commit this with the message to something like update intro query. All right,
5:51:35
so run and get status just to do a final check. Yes, everything is cleaned up. Nothing to commit. Working tree is clean. We are good to go locally. So now we went through how to get add get commit. We now need to get into get push, but we can't do get push because we haven't set up a remote repository in GitHub yet. So you're going to navigate over to github.com. And you're going to sign up there. Personally, I prefer to do signups with things like Google so that way I can just keep all
5:52:07
my credentials in one place to have to make a new email and password. Once you're logged in, you're going to navigate over to your profile by clicking this profile icon in the right hand corner and going to profile. Now I highly recommend you clean this up because this is going to be where we're going to host our final data or final project on. And so if you have employer years or somebody else coming to check it out, you want to make sure it's presentable. Specifically,
5:52:32
I would include a picture and updated name, a little title underneath here. And then any other of your socials that you use such as LinkedIn, what we'll be able to do by the end of this is that project we're going to be hosting. You're going to be able to if you want to pin it on your personal page. So now that we have our github account and profile, we need to establish that remote repo. Up at the top, I'm going to go to create new and specifically, I'm going to do a new repository.
5:53:01
We're going to have a few data engineering projects in here. So I'm going to call this SQL data engineering projects. Now this is the same as our, the name of our folder of SQL data engineering projects, but it doesn't have to be. I like to just keep them aligned. So they're going to avoid a confusion. From there, I'm going to add a description. Well, this repo demonstrates my ability to use SQL for building data engineering pipelines. I do want this to be
5:53:25
a public repository. So I'm going to keep it public and then scrolling down. I don't want to create a read me. I don't want to add a git ignore. We'll handle those two separately. And I don't want to add a license. So I'm going to go forward with just creating this repository. Now it directs us to this page. Now that this repository is created remotely, they have this quick setup option for those that are more experienced, we're not going to use that one. They also have this other one of
5:53:50
if you're creating a new repository on the command line, basically, if you don't have it created already locally, you're going to create it using basically these commands in your terminal. But the one we fall into is this or push an existing repository from the command line. So let's now get into pushing and also pulling from this remote repository. So we're going to focus first on just using git push to get our local repo up into our remote repository. Unfortunately, we can't just run git
5:54:28
push. We need to run some commands to get it set up properly before we do this. Because that we get into our project, it doesn't really know like it doesn't, they're not connected right now, they have no idea that they are actually related and they need to be together. So let's run through these commands that they're telling us to run in order to set this up and then push. So for this, we're going to be using the git remote command. And this manages commands or manages connections
5:54:56
to remote repositories. And we need to add a remote connection. So we're going to add add to this. And we need to give this an alias similar to how we alias different columns, we need to also give an alias of what is GitHub in our case, in our case, it's typically given something like origin. And then with this, we need to tell it what is the URL of this GitHub repository. And so I'm actually going to just go ahead and copy this. I know I could have copied the whole command,
5:55:25
but I wanted to walk through that. Then in my terminal, I'm just going to go ahead and paste that in. So commit git remote at origin, go ahead and run this. Now the next command should look familiar from what we did earlier to go. And that was this git branch and dash M flag for main. If you remember from previously, we renamed it, it was previously the branch of master. If I run git branch, I can see that we are on branch main. We already did this. We don't need
5:55:54
to run that command. You can rerun the command. It doesn't have any effect or any harm. It will still stay main. The point of this is though, that GitHub automatically uses main as the default branch. And so your local repository and also your remote repository need to be aligned on what that default branch is. So both of them need to be main. So either way, we're going to have to rename it to main anyway, glad we did earlier. So now we've probably got enough to set it all up that we
5:56:23
can get now into git push. With this, we're going to use a U flag. And this is short for set upstream. Mainly, we need to tell it to go to well, GitHub, we're going to be pushing it to which is our origin. And we want it to go to that default branch of main. So that's why we have git push, tap U origin main. So I go ahead and run this, you'll then need to go through and enter your credentials for GitHub. So after you've entered in your username and also password, you should
5:56:57
then be able to just go through or it should have gone through and push this up GitHub. Note, I did have two factor authentication enabled for this because I've had GitHub for a while. If you do, you have to go through the process of generating a token and then inputting in here, I'd recommend you just use something like perplexity or chat to BT to go through the steps of that. It's not applicable to most of y'all. So I'm not going to cover it. Anyway, main point is it's been pushed
5:57:22
to GitHub and I can go to this link right here. And this is our repository on GitHub. If I navigate into our one underscore EDA folder, it has our folder with our project and our read me all in here, not looking bad. Pretty good job. So we just did git push to get our content up to GitHub. Now, let's walk through, well, git fetch and git merge, which is eventually we're just going to use a git poll, but we're going to walk through each of these individually to get some changes on our
5:57:57
remote repo that we can get into our local repo. So we need to change inside of our remote repository. I've been continually picking on this file in 1.13, specifically this project one intro SQL file. And so let's make changes here in a remote repository. I'm going to go up in the top right and go to edit this file. And I'm going to once again change job country to job title short. So I've changed that. And now I'm going to go in the right and click commit changes. It used AI
5:58:28
to write a commit message is a very good commit message I'll give by the way. I don't need to add an extended description. And I want to commit directly to the main branch. I don't want to create a new branch for this commit. We'll go over branches more in the second part of this for the second project. Okay, I'm going to go ahead and commit changes. All right, so now project one intro SQL file is updated to job title short. But on my local repo, I can see that it's still a job
5:58:56
country. Now that we have these changes in our remote repository, let's get fetch them to get them out of local repo, and then git merge them to actually start working with them. For this, I'm going to change this to the source control pane to be able to see what's going on the right hand side or left hand side visually. All right, so let's go ahead and get run git fetch. And it's important understand if you think about it from this, does it know where to fetch from? Well, it does
5:59:20
know where to fetch from, because when we ran git push earlier, we use this u flag to set the origin main, meaning we set that of GitHub as the origin to pull from that main branch. So it does know. So running this, so from this message, I could see that it did download a file and everything to seems there's no error message. So everything seems to be updated correctly. Notice here, right? So we still have it. It still says job country here locally on our
5:59:51
file, because that's in our working directory. But in our remote directory, it says job title short. So we're done. happened is when we get fetched it, it grabbed those files and put them into our local repo, but it has not yet updated our working directory. And we can see this visually in the graph on the left hand side. Origin, once again, is GitHub and that main. And so we can see that we're actually a commit behind what is happened up in GitHub on our remote repository,
6:00:24
because we're down here with this main. And that is our local repository's main. This incoming changes is basically remote has commits that we do not have just yet. So if I wanted to check what these actual differences are, I could do get diff. And then I wanted to look at what I have or where I'm looking at in the files. And this is using the nomenclature of head. And that's right here pointing directly to where main is locally for me. And then
6:00:53
we're going to use some range syntax and point it to origin's main. So we're pointing what is the difference between this one to this one? Now running this, oops, I typed or again, not or a gin. So make sure that you have the to write correct typing on this, try this again. Okay, so with this just scrolling this up a little bit more. And in this it calls out how we change this from job country to job tile short. Also, for some reason,
6:01:21
I guess I didn't catch it. We did some sort of new line addition at the end of the file or remove that new line addition, not a big deal. Nonetheless, it shows here tracks everything. So now let's get this file updated with those changes that we fetched into our local repo. For this, we're going to do get merge. And we need to specify this branch here of origin slash main, specifically our remote tracking branch that we want to merge with. And as
6:01:51
you can see from the file above, it updated to job tile short. Now that was with using get fetch and get merge. But I don't like to use those in practice. Instead, I like to use get pull instead to just get it directly my working directory. Also, I didn't point this out earlier, but our graph updated to where we now have all four commits. And our main or local repo is aligned ahead of main is located with the same commit as our origin
6:02:19
main, which origin main being our remote repo. Anyway, we're going to change, we're going to edit this file again one more time. I'm going to change job title short back to job country. I'm going to commit changes at job country is distinct job title query. I want to do that. I want to keep it to the main branch. I'm going to commit those changes. Remember, because we're going to do a get pull, it has been updated. And now in here,
6:02:43
I'm going to go ahead and just run get pull. I don't need to specify because we're like we talked about earlier, I don't need to specify what our origin is, because that's been set up already with that you flag when we ran get get pull. So we just need to run get pull, boom. And it updated to job country. And along with our graph here showing that hey, we advanced ahead one more commit. So if you take anything away from this lesson,
6:03:07
it should be those four get commands of get add, get commit, get push, and then get pull. Now I can understand this can be a tough concept to wrap your mind around. So we've come up with some practice problems from you to now go through in the support of resources. In this, you're going to be creating a remote repository from scratch and a local repository from scratch. And then doing a bunch of push and pulls and different type of commits and
6:03:32
actions in order to get more familiar with get commands. In the next lesson, we're going to just finalize our project by getting it, making sure it's up to date on GitHub and then also sharing it on LinkedIn. So with that, I'll see you in the next one. Alright, congratulations for completing your first that engineering project. In this video, we're going to be going through and just doing a quick double check and along with an addition
6:04:02
to our project to make sure it's button up and good to go to present on GitHub. And then after that, we'll then transition over to LinkedIn to work through how you can share your projects and also make a post about it. First up, if I navigate to where our GitHub project is right now, inside of EDA, yes, I have my project in here, I would go in and verify that all of the images are uploading properly. That's the biggest mistake I see
6:04:29
with other students or that whenever you click a link, it doesn't navigate you to where it needs to. So make sure all your links are up to date. Anyway, if I go back and navigate into the project itself, we have a little bit of a snafu in that we don't have on the front page, have a read me that is detailing what this repository is about and it's going to capture all of our projects that we have. Now, this is my repo that I'm using that has
6:04:56
all of our projects that we're going to build. Anyway, we're going to have multiple different projects inside of here. So what I do is I've built a read me that just showcases that I have these two projects. And then from there, they can navigate into if they want to learn more about what I did with EDA, they can click that and then they can go see what I did with EDA. Anyway, we need to make that real quick. So back in VS code, I'm just going to create
6:05:21
a file real quick. And we're going to call it read me dot MD, we give it this a heading one and call it SQL data engineering product projects. So first I put in there this, the following projects are a collection of SQL projects that I've worked on to practice and reinforce my skills with data engineering tools. Then I have a call out that says, click the project name below to view the tools I use to build these. No, we're going to be adding
6:05:45
a second project. So I'm going to title this pro the section projects. And then underneath this, I'll title this one EDA, give it I want to make it into a hyperlink. So I want to go into that one EDA folder. So I do a backslash going to one EDA. And I want it to go right there actually. And then I want to do exploratory data analysis. Then I want the image of the project underneath here. So I navigate into our other read me that has this in here, copy
6:06:11
it and then paste underneath, it's not going to work. Because it's trying to navigate up to the parent directory, which there's no parent directory where this read me is. So I'll just take that off. And now the image is working. Then I'll put a little description underneath this about what the project's about. Now this should be enough. I am going to make this into this EDA into a headings of three ampersands to make it a little bit bigger and call out.
6:06:33
That's good. I'm going to go ahead and save this. Then from there, you know what we got to do, we got to push this up to GitHub. So I need to add all these files. So we'll do git add dot, I'll do a git status to see that we've added a read me, I'll do a git commit because we want to now quit that commit this to our local repo. And I give it the message of add parent read me. Now we just got to push it. So we'll do git push, boom, then inside
6:07:00
GitHub, I'm going to verify that's up to date by refreshing. Okay, it's been added. Are all my pictures working? Yes. Are my hyperlinks working? They are. All right. Looking good on GitHub. All right. Now in LinkedIn, we need to do two things. Well, I guess three, if you haven't done already, but you should create a profile in LinkedIn, LinkedIn super powerful with being able to connect with employers and network. Even if you're not looking for
6:07:26
a job, it's a great way to build a reputation and then have opportunities come to you. Plenty of come to me. That's how I got my Mr. Beast job. No lie. They message me on LinkedIn. Anyway, I'm assuming for this you have a profile. We're not going to go through a set up a profile, but a minimum have showed out your name about you and about section and fill in all your different experience that you have previously. Now on your profile, you have a section down
6:07:49
here on projects. And this is where you want to add in this project. Now, if that section was not appearing, you need to come up to the top and go to add profile section and then go to add project. So in here, we need to put a catchy title for this. I'm going to put this of data engineer skill analysis dash exploratory data analysis. And then for the description, I just copied out of the read me what we already had for the description
6:08:12
of it for skills at a minimum, you need a list sequel, but I also recommend including terminal bash and get also through in GitHub, but you could throw in something like VS code as well. You're only limited to five. Next, I'm going to go into add media. And for this, we're going to be adding a link specifically a link to our sequel data engineering projects. And I'm actually going to go to write directly to this one. I'm going to get the link for
6:08:35
this that way it navigates right to the page that has all the details about this actual project. I'm going to place the link in right here. And it looks like that's not working right now. However, if I take off this tree main and EDA, what it's navigating into what I wanted to do, it does look like it's not a populating below. So unfortunately, you just have to do it like this. It looks like if you're encountering this error right now. Anyway,
6:08:59
in here, it already has a title and everything like this. I'm going to leave the description blank. So I already gave a description for the project so I shouldn't know what it is. Put your start and end data to the project. I've been working on this dang course since June. So I'm going to bliss this entire six months, add any contributors that helped you with it. If you work with somebody along this, and then from there, click save. Now after
6:09:18
you do this, I also highly recommend that you make a post. I get posts like this all the time where people are tag me in the projects they complete for my courses in it. They just do a quick detail about what the project is. They tag my name in it so that way I can see it. And then from there, they also list a link to the project so you can go in and actually view what they actually did. Pretty cool. And to make a little bit more interactive, they
6:09:42
usually include some sort of pictures, something like that. For this project, I highly recommend either including some code snippets in your picture, or this image right here of what we actually did. Anyway, as I mentioned, I love seeing these projects. I go to the project, check them out, and I'll comment on it to hopefully improve the reach and others can see this. Alright, congratulations again on finishing up your first project. We're now
6:10:07
going to be getting into the second half of the course, and things are going to be ramping up quite quickly as we're going to get into a lot of heavy engineering concepts around how to use SQL to build out data warehouses. All right, with that, I'll see you in the next one. In this lesson, we're going to be going over data types. And at first, it may seem like this lesson is silly and should be basic understanding that everybody should
6:10:36
already know. But it's actually really important that we understand the basics and the reasoning behind data types because we're going to be building out SQL pipelines that populate different tables and views from scratch. And so we need to have that understanding of data types so we can build out tables with those data types. Now, I've hinted quite a few times towards how our columns have certain data types sent already. If I'm in Mother Duck, we've seen
6:11:06
this as follows. If you pull the job postings fact table, it populates to the bottom. And then when I click those ellipses, I can go into show types. And it goes through and shows the different types that we have here of integers, characters, Boolean or timestamp. So why the heck do these data types are setting a column as a certain data type even matter? Well, I have four main reasons. First is data integrity. And that deals with actually downstream of
6:11:34
you. If people are building queries or models off of your tables, if you don't have the right data types or change the data types or aren't consistent with it, you can break their logic. Conversely, it also works with data validation having a set data type in a column. This prevents any bad records from even getting into your table. Now with that, there's also performance and storage gains. You're going to get to the point where you're working with tables with billions
6:12:01
or trillions of data. And at that stage, you need to be able to make sure that you're using the most succinct way to carry that data. Something like a Boolean that is either a zero or one or true or false takes up a lot less memory than something like a Varkar or a character that can take up tons of characters. And then finally, with this type specific behavior. And that's that we can do certain SQL functions to a column dependent on its data type. Which brings us into the more
6:12:35
important topic of what the heck are the different data types? Well, I've booked them or put them into five general bins based on what I use on a daily basis. The first four are the most common and that fifth one is just an other or common types of data types that I just sort of lumped all together. They don't necessarily all relate. But the first four are ones that I do use and that's numeric, which are used for numbers like integers, decimals, characters used for text or
6:13:04
strings, booleens that record, whether it's a true or false value, and then date and times, which are used for things like timestamps and time based data. Now, for numeric, there's generally about five different ways that you can define what a number is going to be. First one's pretty simple that it's int or integer. It's a whole number. Also, some systems have a small int or big int, where small n is whole numbers within a smaller range and big n is in bigger numbers
6:13:31
between a bigger range. Right now, we're going to keep it simple to int. Next are float and double. Both of these are approximate decimal numbers, whereas float is for lower precision. As you can see, there's a little bit more rounding here, whereas double is much longer. The science behind it is that floats using 32 bit to evaluate this and double is using up to or often around 64 bit to evaluate it. Double is commonly used for analytics and statistics calculations. That's why I've
6:13:59
ingrained it to where our salary, your average and salary, our average are both double. Next are decimal and numeric and functionally, depending on the database, they actually can be the same thing. It could just be an alias. Anyway, in parentheses, you can put things like p comma s and this specifies your total digits and then your digits after the decimal. So in the case of the 12.1, the total
6:14:24
digits is three and the decimals after the decimal place are one. Next is character, which is used for text or strings. If it's a super short amount of characters, you would use care with a parentheses of n to specify how many digits typically is like two or three. Now the key thing about this is this is a fixed length. So whatever number you put at put for n for max characters, that's how many it
6:14:47
will be and it will cut it off or it will fill in with blanks if it's too short. Think of like a state or a country code, something like that that you know it's going to be set, you'd use that. Now if the text is going to vary a little bit, you're going to use varkar, but varkar means variable length character. For the majority of our columns in here, that is what I coded them as. Now with that comes the caveat that that's not necessarily the most efficient way to store data,
6:15:13
but there's no really no other choice for this. And at least I didn't do the last option of text, which is a long unbounded text that you could include. There's no length limit on this. Next step is boolean. And that's recording whether a value is true or false. Like we discussed before, these values are behind the scenes held as something as either a zero or a one. In some databases, it may even be written as bool B O O L. We use this in our work from home, no degree mentioned
6:15:42
in our health insurance columns. Next up is date and time. And we have a whole lesson on how to use date functions. We're going to dive deeper into these. But for the time being, I think you can understand that some of these are dates that record a date time because of time and then time stamp, or you could see it in some databases as date time. And this has a date and time object to it. Now you can also see it with time stand Z and the Z stands for zone. And this has the time
6:16:10
zone at the end of it. Right now this one, the time zone is just UTC or coordinated universal time zone. Basically what's at the prime already? And finally, our last bucket of other types. And this, we're not going to get to any of these in this course. But I just want you to be aware that there are other types available such as arrays, things to handle JSON data, UUID, so long identification numbers, or even something like binary. And that list is not in all encompassing. There's many,
6:16:40
many more. Now, how should you be checking what data type a column is? Well, as we learned about before, you can use something like information schema that provides this based on the metadata that's stored inside of a database. So let's demo it inside of VS code. I'm inside of my project here. First thing we need to do is connect to our database inside of mother duck of that data jobs database. I'm gonna start a new file for working in here for running different commands. Before
6:17:12
this, I'm going to select all for right now. And this is going to be from information schema and specifically columns, which information schema is well, the schema and then columns is the table. So reason that dot notation to access it, we'll go ahead and see what we get back with this. Oops, got my colon or semicolon. And yeah, a lot of stuff in here. So I need to actually filter down based on the table name. So I'm going to do a where table name is equal to job postings fact.
6:17:44
And then I'm going to clamp the columns to because there's a lot of columns in here. I want that table name. I want the column name. And then also want to call them called data type. This time, let's not forget the semicolon. We're going to go ahead and run this bad boy. All right, this is a lot more manageable to read. And we can actually see all the different data types associated with our columns. The same as what we would have gotten our mother duck view. Now,
6:18:09
the other option you can do is using the describe command. And if I run this on job postings fact, sending this down below, this provides very similar data that we saw previously and a little bit additional. Now, the one problem with this is is describe isn't universal to all databases. It is ductie bees part of their friendly sequel. But some databases don't have this like SQL Server and Postgres. Now describe is actually pretty cool, because you can use describe on top of
6:18:41
another query. Specifically, I have this query right here. I'll just run that query real quick. And this provides back a column on job tile short and salary or average. Now, it does say that data types in here. But if I wanted to, I could just run this and then provide a column back with not only column types, but also any other information about it, such as if it's keys or if it allows nulls or not. Now, let's put this knowledge to test of understanding the basics
6:19:09
behind data types and actually converting data within queries to a different data type. And we can do this within select clauses by casting, which refers to the operation converting a value in a particular data type to the course writing value and another data type. We're going to show this first with using the function cast. And inside of here, you have your column or value. In our case, let's say it's the integer of 123. And we want to convert this to something like a string
6:19:38
character. So I'll just convert this to a var car. So let's go ahead and run this bad boy. And we can see we get 123. And it is tough. And it's of the type bar car. Now you can't convert everything to just any old data type. Let's say for example, this 123 was a string instead. And it also had some digits in it, such as like def. And I wanted to convert it now to something like an integer. Well, whenever I try to run this, I'm going to get a conversion error. Could not convert string
6:20:09
123 def two in 32 or an integer, because it's got these letters in it. But if I were to actually remove these letters from here, and then run this, it's gonna work just fine. Because all those are numbers. So let's get an example with this. So we have the following columns that we're trying to query job ID, job work from home job posted date and salary or average getting this from the job thought table, I'm only going to limit to 10 to not overload the database. Anyway, I have
6:20:33
comments next to each on what we want to do to that particular particular column job ID is by far the most complex. So we're gonna start right down here with job work from home. And this one, we want to convert convert from Boolean to numeric value. Let's actually even look at all these values first. And we can see that job ID is then fact in work from home's Boolean job posted date is a timestamp and salary or average is double. Anyway, we want to convert job work from home
6:21:00
from Boolean to numeric value. Well, anytime we do any conversions, I'd recommend consulting with the source documentation linked here. And we just want to go to a numeric value. Specifically, I'm thinking we want to go to an integer. And for this, it has the name, but also there's other aliases. So I can use any one of these to get a an integer. So I'm going to use cast here. And then inside of here, I'm going to specify that we're gonna cast it as an int for the enclosing
6:21:27
parentheses. Now running all this, I can see that it is now int. And all these are zeros. And that's because previously they are all false values, false values correlate to zero true values would correlate to one. Anyway, I'm also give this an alias. So that way we keep the same name of job work from home. All right, next one, we want to convert that job posted date from a timestamp, which it has a date and time to actually reflect its name of job posted date. So I run the cast
6:21:56
function on this, specify we're gonna do this as the date, and then give it the alias job posted date, running this, we now have only days knows column, salary or average, we want to convert from double to no decimal places. Well, going back to our documentation, we can see we have this one here for decimal. And it takes the two arguments of precision and scale, which are the full amount of numbers for the precision and the scale is the number of decimal places after. So we'll do cast
6:22:22
one salary, your average as decimal. And we're gonna specify 10, along with a digit of virtual node digits, so zero, so no decimal places. And I'm only going to do 10, because I would expect all these values to be less than, well, a million dollars. And if they are, they're outrageous, anyway, we probably need to get them out of here. So this would be a good filtering mechanism. Running this, well, we can't see any values, because they're all null right now. So I'm at a
6:22:48
where statement in here of where salary, your average is not null. All right, let's go ahead and run this bad boy again. All right, and now we're getting nice clean values for this. I will give it an alias to now this top one and we're gonna do job ID and we want a more unique identifier. With this, what I'm thinking is, is we're gonna combine it with company ID, because it's a pretty unique identifier as well. Let's go ahead and run this to show you what I mean. Anyway, what I
6:23:18
want to do is I want to combine these to this job ID and this company ID column to one. I don't want to add them together, because right now they're both integers. I want to basically append one to the other. In order to do that, though, I'll need to convert them to strings and then from there, append them. So I'm going to cast that job ID as var car, and I'm going to do the same thing with that company ID. I'm going to run this real quick just to see what I get.
6:23:47
Okay, I have both of these columns now and they're basically their var cars. Also, I can see that they're a text value because they're offset to the left side of the column, whereas numerical values like the salary or average are to the right side of the column and just like this job worked from home. That's how you can quickly check whether it's a text or a numeric value. Well, I'm going to use a concatenator operator and this concatenates two strings,
6:24:16
list or blobs. In our case, we have two strings. They're basically going to put them together. Could also use the concat function, but for the operators, that makes it more readable, I feel. So I'm going to take this one up here, put on those two concat operators, not two concat operators, just one, but I'm putting them right next to each other. And now running this bad boy. All right, I'm getting them combined. Looks a little looks a little strange to me. Now I could
6:24:41
wrap this all in another cast and then convert as an int to get it back to an integer. But actually, what I'm going to do is I'm going to put it between these as a hyphen. And I'm actually, we have to stick another concat operator in there as well. And now whenever running this, we get this nice little hyphenated, unique identifier. Pretty cool. Now besides using the cast function, DuckDB supports a popular option that's used in other databases like Postgres,
6:25:12
and instead uses an operator. So I can use this double colon to cast this eye here as a double. So I went through and updated what our previous query was using cast and just replace it out with this double colon. Personally, I feel that the latter option is a little bit more readable, and it doesn't have cast all over the place. You can just follow along and read it through of what's going through. Anyway, whenever I go to run this bad boy, this provides the same exact
6:25:45
results that we saw before. So just as efficient. Now one last thing, you should be explicit when you're casting anything that there may be ambiguity between two columns when you're combining them. So in this case, I was adding something like three, which is an integer to some like 5.5. Whenever I run this, I can see that I do get back a decimal of 8.5. But remember, three is an integer. 5.5 is a, well, in this case, a decimal of 12.1. If for some reason that you wanted to maintain
6:26:16
it as an integer, this would be a good use of that casting operator in that you could get that out of this. But it really depends on what you're trying to do and what your end goal is. So casting is going to save your button data pipelines. It's not going to necessarily hurt you now. It's going to hurt you down the road later, because some junior data engineers probably going to put in some values that they shouldn't into a column, and you need to have the code in place to check
6:26:43
for this. All right, it's your turn to give it a shot. We got some practice problems for you to now go through and get more familiar with now using that cast function, but also the cast operator. And now that we have this level of knowledge on different data types, we're going to be shifting next into using a different type of SQL, DDL and DML, which we'll go into. But this uses our knowledge on data types to build out future tables and pipelines. With that, I'll see you in the next one.
6:27:16
Welcome to this lesson on DML and DDL commands. DML stands for data manipulation language and DDL stands for data definition language. Now these type of commands are imperative in order to build out and create different databases that are warehouses and even a data mart, which we're going to do. So over the course of these next three lessons that we're going to be doing on DDL and DML, we're going to be building out this job mart. And for it, we're going to be exploring all the
6:27:48
different commands or the most important commands that you need to know within DDL and DML. Now in this lesson, we're going to focus on just the basics. We're going to start out simple and look at how to use things like a create and drop command to create and drop databases, schemas, and tables. After we know how to create these objects, we're then going to move into inserting data into a database specifically into a table and updating records within a table. Merge is a
6:28:20
more of an advanced concept. And so we'll be covering that in our third lesson on this topic. Finally, we'll get into altering tables, looking at things like adding columns, dropping columns, renaming them, renaming tables, and some other popular features with it. And this lesson does cover quite a bit of different keywords, but I don't think it's anything behind your capacity, none of them are that complicated that you can't really wrap your mind around of what it's actually
6:28:44
doing. Now, up to this point in this course, we've really only covered one type of SQL command, and that is DQL or data query language. These are commands that query and retrieve data from tables, things like using select and from, and then filtering as necessary, may be using where, and then maybe even aggregating it using something like a group by. Now, what we'll be jumping into as first is DML or data manipulation language. This is used to, well, manipulate data within
6:29:17
tables. We're going to be using keywords like insert, update, delete, and merge in order to change or update data within an existing table. But that's for existing tables. What the heck do we do to actually even create a table? Well, that's where DDL commands come in or data definition language. These define and modify database structure and schema. These use keywords like create, alter, drop, or truncate. We're creating and destroying complete objects with this language. Now, I will
6:29:50
add a disclaimer. Technically, there's five different types of SQL language. We're not going to be covering the two other ones of DC. DL data control language, which is used to control and access permissions within databases. That's for more database engineers. And then also TCL or transaction control language, which manages transactions and ensure data and charity. That's even more complex that we're going to need to get into for this point.
6:30:15
Now, what the heck are we going to be doing with this DML and DQL language? Well, we're going to be working with creating our own database. Now, as you know, this is our existing data warehouse that we have. And this is the data jobs where we have that main schema and there are four main tables that relate to each other and has all of our job posting data. This is a database that is a data warehouse. Well, we're going to build something that would be downstream of a data warehouse and
6:30:44
going to be based off of a data warehouse. That is a job marked. Now this is a data mark and it's subject specific. So it's curated subset of data designed for a specific business function or analytical use case. If you had something like a sales team, they want to look at something like revenue or orders. And then if you have like a marketing team, they'd want to mark that would look at campaigns, clicks and conversions.
6:31:11
So over the course of these next three lessons, we're going to be building out different components inside of here to better understand that DML and DQDQL language. Now, just a sneak peek, we will be building out not only the data warehouse, but also a few other data marks in our final project at the end of this course. So if your mind's not wrapping around completely what data marks is, don't worry. We're going to have plenty of practice coming up even after this, but the main thing to
6:31:42
understand is that data marks are downstream of a data warehouse and they have a curated set of data that's coming from a data warehouse designed for a specific business function or analytical use case. But typically smaller, that's why I made them smaller in this picture. Now where are we going to be executing these SQL queries or we're going to be running this locally with local databases or in mother duck?
6:32:06
Well, we're going to be using the latter in mother duck and this is going to give you practice with creating and setting up different databases in the cloud. Now if you want to, you can do all of this locally as well. Some of the commands will change from time to time and I won't necessarily be going in depth about how the differences are if you were to execute this locally. I'm going to be focusing on specifically mother duck executing it in the cloud.
6:32:34
All right, our first set of commands that we're going to get into are create and drop. For this, we're going to focus on creating our database and then inside that database create a schema and then inside a schema create a table. Now drop is a very dangerous keyword. You can ruin people's lives if you run drop on the wrong database schema or table. So make sure anytime you're running it, you know exactly what you're doing and think twice. Now let's jump in and start programming for this.
6:33:06
I'm going to be operating all of this on a Mac, but I've done this before in Windows as well. Everything's completely the same. Anyway, I ran ahead and create a file called 1.21 DDLDML part one and this is where we're going to be executing queries from in the terminal. I'm going to go ahead and create our connect to mother duck. Note if you're running this locally, you just go out and list that duck DB file that you want to create.
6:33:32
Go to our job mark dot duck DB and when you press enter, run that, that would be the create warehouse or create database that you're creating for this. That's the extent of the sport I'm going to give if you're going to do this locally. Anyway, I'm going to connect, I'm going to connect specifically to the data jobs warehouse and mother duck. First thing I'm going to do is just see what databases we have listed currently inside of here.
6:33:55
Now right now it has four different databases, information schema to get information about the different databases, sample data data jobs and also my DB. Anyway, none of these are what we want to create. We want to create another one. So we're going to use this syntax of create database and then list the name of the database. If not exist, we'll get to that in a bit. I'm going to add in create database and then the name we want to use for this and I'm going to call this jobs marked.
6:34:24
Don't forget that semi colon. Now we go ahead and run this. Bam. You just created your first database. Now let's see if it exists. I'm not going to run that database instead. I'm going to run some that's more dialect friendly, bit of a league command show databases. Okay. We got five now. We can see that jobs Mart has been added to this. Too cool. Now back to that. If not exist portion that can be added to this. Let's say for some reason we had a script.
6:34:50
We were trying to run an automation script to create databases, tables and schemas. Really common for data engineers to do. If I were to try to rerun this command right here, I'm going to go ahead and do it. I'm going to get this error right here, binder error. It says fail to create database database with the name job mount or jobs Mart already exists. We can add this condition of if not exist, therefore it can attempt to create a database
6:35:15
that already exists without throwing an error and stopping execution. So I'm going to add in if not exists. Now whenever we run this, we don't get an error. And checking the show databases, it still exists. Now that we have created a database, let's drop it. We'll be very careful with this one though. For this, I'm going to use the command or keywords drop database. And then be very careful. We're listening the actual one.
6:35:40
So jobs Mart is the one we want to drop, I'm going to go ahead and run this. Okay. And now I'm going to select show databases to see what's available and it's deleted. Now once again, if we were running sort of automated script with this, if we tried to run this again, now that we've dropped it, we're going to get catalog error. And that's a database with the name jobs, more does not exist. Do you mean my DB, it tries to relate to maybe a similar one anyway with this one, instead
6:36:08
of writing if not exists, we want to check if it exists, then you can try to drop the the database. So I'm going to add in if exists into here. Now let's try to run this bad boy, boom, don't get in there. We do want this database though, so I'm going to go ahead and comment out this line. I'll just keep it there for the time being. And we'll rerun this create database so that way we have it and just double checking it, running show databases, it's in there. All right.
6:36:37
Next up, now that we have our database created, we need to now create a schema for this. We're going to be using the keywords create schema and it has very similar in that you can use an if not exist optional keyword inside of there for if it exists or not. And then you list the name of it. Now for the data mark that we're going to be building, we're going to have two different schemas. We're going to have a staging schema where we basically stage our data before we send
6:37:04
it into the other schema of the main schema. So before we actually even create the schema, let's check what schemas are inside of this database. For this, we're going to be using information schema, specifically checking the scamata. This is diet database agnostic. Basically, you can run this in most all databases. Let's go ahead and check this. I had a typo in information schema. Let's try this again.
6:37:31
And inside of here, we can see that we have our jobs marked and specifically, there's only one schema of main by default, ductDB create your database with one schema of main already. So this main one's complete. We just need create the staging schema. Oh, also with this main, you can't delete it. It's there by default is there permanently. So let's create that schema by using that create schema keyword. And then now we can't just specify staging because what database are we connected to
6:38:02
well, that scroll up and see, we connected initially via ductDB to the data jobs schema or the data jobs database. So it would try to create the schema there. One option is we can list the database name in front of this. So in our case, we're going to list that jobs, Mart in front of it using that period notation. And then I can go ahead and then putting that semi colon in there, running this and then checking it the information schema.
6:38:30
We can see now that jobs, Mart has this staging inside of it. Now we're going to continue to use jobs, Mart throughout the remainder of this lesson in the next few lessons. So I don't want to have to keep on writing jobs, Mart every single time it gets annoying. So above this, I'm going to add the command of use and then specify the data, the database I want to use of jobs, Mart this use keyword is only allowed in some other database, like
6:38:58
SQL Server and MySQL and also ductDB, obviously, but anyway, let's go ahead and run this and show what I mean by this. So we got was able to use that, then I can go ahead and remove that now. And whenever I create the schema of staging, well, it's a schema myth name staging already exist. So I can add in if not exists and then run this again. And so now one, I don't have to list jobs, Mart, because it knows this.
6:39:24
And two, it create our schema without any issues, I can just double check it by going here and check in the information schema. And did I see? Okay, staging is inside of jobs, Mart are we've created schema? Now we need to delete it or better said, drop. And we're going to be dropping the schema, specifically that of staging. Okay, let's go ahead and run this bad boy looks like the command run just ran just fine. Now let's go ahead and check information schema.
6:39:50
And inside of here for jobs, Mart, it's no longer there. Like dropped database, you can also include the optional command of if exists, that you get the point of that, we're not going to run that again. So I'm going to actually just comment this out, because I want to be very careful not to drop the database or drop schema. All right, next up is tables, we're going to be creating and dropping a table, creating and dropping are the same commands that we've seen before.
6:40:16
But for this, we're going to be creating a table called priority roles. And this is going to be used by the business to track which job postings they are marking as a priority to track. Now we're going to end up renaming this. And let's say we accidentally started out with creating this, that we wanted to name it preferred roles, not priority roles. Now this keyword is a little bit more involved than that for schema and database sort of gloss over that.
6:40:47
But we need to focus more on what is the syntax of this. Well, first of all, we start with that create table, if not exist is an optional command. So I'll put that square brackets to make that more clear. Next after that, it's the table name that you're going to give to it. And then underneath it in parentheses, or actually you can go off to the side. This is the once again, indentation doesn't matter in SQL, it's just if it's more readable.
6:41:09
But underneath this, what do we have on each different line is the column. But that first column is optional, but we want to build nice databases that have a lot of similarities to how data warehouses are built. And people expect an ID column, and this is typically of the data type integer. So we'd specify that data type of integer. And then next to it, because we want this to be our primary key, we'd put that out right next to integer.
6:41:38
Then from there underneath this, you would list any associated column names that you want in there, and their associated data types, if you had any foreign key columns, you list their data type. So for this, we're going to start with that create create table command. And this is going to be called preferred roles. No, we're going to be changing the name of this preferred roles later to priority roles, the main name of the table.
6:42:03
But I'm just doing this for demonstration purposes that we can change the name letter. Anyway, inside of here, I'm going to go ahead and open parentheses and then dent down underneath it. And we're going to keep it simple with only two columns. We're going to give one of role ID, because we're going to have also roles associated with this. Anyway, we'll get to that. We're going to give it the data type of integer, and then the role name will be buttoned in
6:42:26
things here like data scientists, data analysts, that engineers. So this is going to be of that data type var car. Once again, I don't know if our car is approved to say for this, but it's fun to say. Now I used integer and var car, but remember, there's a bunch of other different data types that we can use to assign the data type of a column and recover that in the last lesson. Anyway, let's go ahead and run this bad boy.
6:42:51
I don't know if anybody knows this, but we haven't we're going to have an issue with this. I'll go ahead and run it. Well, the first is that there's a comma that was missing in here. So I'll go ahead and run this again. All right, it ran with no problems, but let's go inspect it specifically. We're going to be using information schema again. We're going to be coming from well, information schema dot tables.
6:43:11
And I'm going to just filter this down for the table catalogs that way we can make it a lot easier to read and that of job mark, because that's what we created for semi colon on the end. Go ahead and run this. And nothing returned because I'm silly. I have job. Mark. It's actually jobs. Mark. Let's run again. And okay. It's inside here. But what the heck, it's inside the main schema. Well, the problem with this is anytime you run this or run to create a table, you have
6:43:41
to specify the schema in our case, we want to go into the staging schema, and we go ahead and run this now again, and I ran into an error there actually. I don't know why, but my schema, maybe I did run drop to the staging schema. Anyway, I had to recreate the stage, but this time I ran it boom. And we created preferred roles in staging, we can go ahead and check it by once again, checking information schema, filtering for the table catalog for jobs, Mark. Okay.
6:44:08
Now we have preferred roles, but we have it in two places, main and staging. So we need to drop it. Now this one doesn't have any additional syntax to memorize with this or to learn with this. It's just drop table and you can use that additional application if it exists. So we want a drop table and specifically I want to delete the one inside of the main, I can just write preferred roles and drop this one.
6:44:32
So running like this, and then inspecting inside of there to see if it in fact, delete you looking at information scheme, I can see that only staging is inside there. If I wanted to write this drop table with main dot preferred roles, that would also be acceptable and probably more preferred just to make sure you're more precise. In our case, this is already deleted. If I were to run this again, I'd have to include if exist, run this bad boy, and it would actually work.
6:44:56
So now to that is the basic keywords of create and drop, whether using it with database schema or table. Now as a data engineer, you're going to be assigned to build scripts that do item potent operations. Item potent as defined by Google in here is denoting an element of a set, which is unchanged value when multiplied or otherwise operated on by itself. Basically, if we build some sort of script that we want to be safely rerun without any
6:45:27
side effects or errors, we want to build it in a way to do this. So right now, this is our script. I'm going to go ahead and save it by pressing command S or control S on Windows. Let's say we're building this and we want to maybe put it somewhere in the cloud to rerun to build this from scratch. And by running the whole script, what I mean is is, well, I have everything highlighted here. I'm going to go ahead and press shift enter to send that into the terminal.
6:45:51
But anyway, as it tried to go through this whenever I gave it the command that starts up here, I ended up getting an error halfway through here because of preferred roles that table already exist. So in our case, I would have to add, if not exist, okay, now when actually running this entire script right here, I don't get any errors with it. So this is pretty good that we built this. Now, I will say an easier way to do this instead of having to highlight this entire
6:46:20
file.db has a dot command for this specifically, it's read. And for this, you have to remember, we're right now for our terminal inside the sequel data engineering project folder. We want to run this folder right here. So what I can do is right click it and go copy relative path. And then inside of here next to read, I can go ahead and paste this in here. And so it will navigate down into the where I want it to run the script. Okay, now press enter.
6:46:51
And as we can see underneath here, it went through and ran the script. Notice name with preferred role already exists. I guess I was silly earlier when I said it worked just fine. I guess I missed it. I didn't save it. Anyway, let's try to run that again using read. Okay. Now when I'm going through this, I can see it lists the databases, list the catalog, another thing. There's no error messages anywhere in there. So it worked just fine. It's item potent.
6:47:17
Anyway, I'm going to want to do this again. So I'm going to copy this command and I'm going to go ahead and actually insert it up at the top. But key thing here is I'm not going to because that'd be really strange. I'm going to comment it out. It's just going to be up here for if I want to run that. All I have to do is just highlight this. I don't want it to run this command and run that command and it just be in this never ending loop. But crazy.
6:47:43
One last tweak that I'm going to make to this script as we're going through this is I do want it to start from scratch whenever it's doing this. I don't want to every one of these has this if not exist in it for the create schema database and table. Basically, it's checks that it exists and it doesn't run it. I do want it to do it during this. And that's just the operation that I want that I'm just particular about.
6:48:08
So this drop database, I'm going to go ahead and actually stick this up at the top. So I know it's going to drop the database and these items then shouldn't exist anyway and we're going to be creating them from scratch. We go ahead and save this and then copy this command up at the top that the one then paste into the terminal to do this entire script, including dropping. And I get this error. I tend not to touch job smart because it is the default database, select a different
6:48:39
database using use to allow detaching this data database. Basically I'm trying to drop a database that in our script here, we have use job marked. So I need to use a different database first before I actually drop it. So I'm going to use data jobs, which is a different database, then I'm going to drop job smart, and then go through the creation script, which will then switch back to job smart, saving this and then pressing up to refresh that last command. We're going to try it again.
6:49:13
Hopefully we're right. And we did it with no errors. Everything happened just fine. And we can see at the end that we did create it. Pretty cool. All right. Next up is insert into. And this is grouped, I'll put it into this diagram with two other ones of insert, update and merge. Merge is just a combination of insert and update. And it's more advanced. We'll be covering it in the third lesson when we go through DDL and DML.
6:49:42
Anyway, in this lesson, we're only going to be focused on insert and update. Insert. It's pretty simple. It's just a row level addition. If the row already exists, it will fail on duplication. And this is best for things like new data ingestion into a table. So we just created that preferred roles table right here. I'm going to go ahead and actually remove this drop table if exists main preferred roles because that's not a problem anymore. That shouldn't be in our script.
6:50:09
Anyway, we need to fill in these preferred roles with a role ID and role name. For this, use the keyword insert into you list the table name and then you specify in parentheses the columns you want to fill in and the particular order you want to fill it in. And then underneath this, we would list the values and what values we want to put into those specific columns. So let's go ahead and try this out.
6:50:33
I'm going to do an insert into and we want to go into in the staging schema into preferred roles. Now inside of here, I would list the first column of role ID and the second column of role name, then underneath here, I would insert in values, I'm going to create a new line for this. And then each set we want to put inside its own set of parentheses. So we'll do the job or the role ID of one and we'll put in that first one of just data engineer.
6:51:01
Then outside the parentheses, I'll put a comma. I'll enter the next row that we're going to insert into here. And it will be number two of senior data engineer, that's all we're going to insert for right now. We'll go ahead and close this with the semicolon. Now let's go ahead and run this. Okay. No issues. Let's actually query this table to make sure that it got inserted into there and I'll do a select star from staging dot preferred roles.
6:51:28
Running this, we can see we have inside of here that role ID of one and two of that engineering senior data engineer worked. Now we can insert in more roles. I'm going to just go ahead and copy this for brevity. And let's say in this case, we wanted to add a third role of software engineer. Now real quick, remember, I didn't add it up here. And that's because remember, if we try to run this with these roles already made, well, whenever we run this, yeah, we get no errors.
6:51:54
And if we do this, look into it, well, we have actually duplicates. Now what the heck is going on? Oh, I made an error. It shouldn't have allowed us to do this. And I realized what I did now. We never specified that this role ID was a primary key. That was the issue because then it wouldn't prevent the duplicate. So good thing we're testing this. So let's go ahead and add this key already here of primary key and that would go after the integer in this case.
6:52:20
Now we've already created this table. Just for simplicity, we're going to go ahead and just drop this table. So I'm going to type this down here because I don't want to save this because this was just sort of an impromptu thing that we had to do. And this will be drop table staging preferred roles. Okay. And now we're going to recreate this time assigning that primary key, good. And then querying it to make sure that the table exists, yep, it exists in there.
6:52:44
And then actually querying inside of the preferred roles table, the values didn't get inserted because, well, we have an inserted values yet, right? So I'll go ahead and insert them in. And then now checking it, all right, we have those two that are already in there. If I try to reinsert it, I'm going to get an error because we have duplicate here because we already have a one and two and it's checking based on assigning this primary key that we don't have any repeating.
6:53:14
So just rechecking this of querying that preferred roles table. Okay, we have data engineer and senior down here. Okay. And now getting to that final point that I was trying to get to. Now we can add a third role if we wanted to in here and querying this table. We can see we have all three of these roles inside of here. So a little bit of learning experience going through that. Now I want this script to be, like I said, out in potent.
6:53:39
So what we're going to do is I'm going to just add this up here. I know I want these three values. I'm going to take that and I'm going to insert it right here. Make sure you have that comma for that second line. And then I go ahead and delete this right here. I'm going to go ahead and save this file command S or control last double check it real quick. Make sure everything looks right. Yep. I'm going to copy this up top here, paste it in. Let's try to run this entire script. Bam.
6:54:06
And for our final output, we can see that we have the correct table. So our script's working great. Now that we've created our table and inserted data into it, let's go through a few alter table commands that we can do. And this is these commands are stacked on one top of the other. Basically we have alter table and then you specify an additional thing that you want to do to alter the table. Specifically you could do something like an add column, drop column, rename a column
6:54:34
or table, or you can alter the column itself such as the data type it is. So for our table, we want to add an additional column and this column is going to be called preferred roles. Basically we want to give it a Boolean value of true or false. If it is preferred in our case, that engineers and senior data engineers preferred. And this will be true software engineers, not so preferred, we're going to give it false. So the syntax for this is we list first alter table.
6:55:04
We specify the name of the table so staging dot preferred roles. And then underneath this, I'm going to do the next part of the command of something like add column. In this case, we want to be a preferred roles. And note for this column name, it's preferred role, not preferred roles like the name of the table. It's very similar. We're going to end up changing both of these, but just for demonstration purposes.
6:55:26
Now anytime you create a column, just like we did above whenever we did that create table, we need to specify the data type. In our case, we want to be Boolean. So inside of here, I'll add Boolean. Let's go ahead and run this. So we add her a column. We're going to now run our query to check it and we can see that a preferred role column is created and that there are no values or no values inside of it.
6:55:53
Now along with add column, I'm going to go ahead and just copy this right here, paste it underneath. We also have drop column and it's just changing add to drop. And I can remove that Boolean value because I don't need to specify that when I'm dropping the column. Now whenever I run that to drop the column, querying the database, I can see that it's going. But we want it in there, right? So I'm going to go ahead and add it back, check to make sure that it went into there.
6:56:20
And then from there, remove this because we don't want to repeat this in our script. Now that we've created this new column, we need to add data to the column. Now insert is a row level addition. Update is a row level modification. So we can pinpoint that new column and add data specifically to the null values in that new column. Now it needs a wear condition or it will update all the rows, particular to whatever column we're calling out for this.
6:56:55
This is best for corrections and fixes and not inserting an entire row. For this, we needed to use that minimum three keywords. The first one is update to say what the table name we want to update is. Next is set and it defines the new value that we're going to be putting or inserting into here or updating in here and then finally the wear condition. What do we need to meet for that particular row that we want to modify?
6:57:21
So we'll start with that update command specifying our table of preferred roles with an S for this. We'll next use set and we'll be setting that preferred role, particularly remember we want to set data engineer and senior data engineer to Cheeru and software engineer to false. We'll specify in here that we want the preferred role to be true, wear role ID equals to one or role ID equal to two. Okay, let's go ahead and run this and it looked like it. We didn't have any errors.
6:57:53
So now I'm going to go ahead and query the table to make sure they got inserted in. All right, we got two true values in there for our software engineer one. I'm going to go ahead and just copy this because I'm a little lazy. Change the role ID to three and remove the rest of this and then also update this instead of true to be false. Running this query, I've got no errors and now querying the table, we should have it all filled in. Yep, true, true, false. Nice. Now one last on this.
6:58:20
We just went through insert and update. We will in the third lesson cover merge and that's going to combine both insert and update all within one keyword command. All right, last portion of this, let's wrap this up. We're going to be finishing up with the altar table specifically, we're going to be covering how to rename a table and a column and then also how to alter a column. Now as I hinted towards before the preferred roles, that's not the preferable name of this.
6:58:52
Also I'm not a fan of this preferred roles column that we're using. I want to change now in the name of the table but also the name of the column and what's inside that column. What I want it to be instead is a priority level. So instead of true or false, I want to like rank it on a scale of one to three where one is the highest priority and three is the lowest priority. So let's first start by changing the name of this from preferred roles to priority roles for the table.
6:59:24
We use that altar table command and specify the table of preferred roles. In this we're going to rename to, that's the keyword, so we're renaming to what we want to now name it. In this case, we don't need to specify the schema, we can just specify what we want the name of the table to be and we're changing this to priority roles. So running this, it looks like it runs quite correctly. If I try to query our previous preferred roles, I'm going to get an error saying, hey, that
6:59:53
that doesn't exist anymore. So we'll go ahead and copy this. and then put this underneath right here, and then change this to be priority roles instead. Let's go ahead and query this, and we get it back. Okay, we've re-changed the name of the table. Now let's rename the name of the column. For this, we're still using that alter table command as the top one. Specifying now that staging.priority roles could really rename the table. And then we're gonna use the keyword of rename column.
7:00:24
And we need to specify the old column first of preferred role, and we're changing this to priority level. I like this better to this rename because now priority roles and priority levels, they're not as close as was preferred role and prefer roles the name of the table. Super confusing. Anyway, let's go ahead and put a semicolon in, run this bad boy, and then query the table to see if it's updated. Yup, priority level is updated. So we got the table changed, column changed,
7:00:54
let's actually change the values in here. Now we're gonna be using the alter column command for this in addition, we'll alter table, alter column. And you can do multiple different things in altering a column. We can specify to either change the type, such as if we wanna change the data type. We can specify what we wanna set default values to whenever we add new roles. So you don't have to necessarily automatically set it to null.
7:01:20
We can also drop default values from a particular column. And we can also add or drop specific constraints of whether it's null or not. The last two are literally beyond this, the scope of this. We're just gonna focus on changing the type of data that's inside of that column. Specifically, right, we wanna be a priority level. And it's gonna be either one, two, or three. So that's an integer. So let's start by typing this out. We're gonna start with alter table.
7:01:45
And we're doing this on the staging.priority roles table. And we're gonna be altering the column. We need to specify the column, which is priority level. And then what we're doing to it, we're changing the types. We're gonna put type in now as the keyword. And we need to specify the new data type for this. Now, anytime you select the data type, you're going to, you have to be very specific. Not all data types can be transferred to other data types.
7:02:11
In our case, Boolean, which is true or false, can be transferred to a numeric value. Yes, it has true or false in here. But behind the scenes, it's encoded as a zero or a one. So, transferring it over to an integer. So I'm gonna type in integer. It's not gonna be a problem. We'll actually see what it gets. So if I run this, I've transferred it to an integer. And now, whenever querying that table, we can see inside of here, the trues were replaced with a one,
7:02:39
and the false was replaced with a zero. But going back to that point, I was trying to make, there are some things you can't transfer to, such as if it was a character value, you can't transfer it to numeric. For example, this is just an example, you don't have to necessarily run this. I'm gonna paste this underneath here. We're gonna do the priorities role. Let's say I wanted to change the role name, which is of the type varcar to an integer.
7:03:01
If I tried to run this, I'm gonna get a conversion error, could not convert string to type in 32. So there's some data types, you can't do this too. Anyway, we need to now go in and fill out what our priorities levels are for this. Honestly, data engineer and senior data engineer, I was gonna make a priority one, so therefore perfectly find the way they are. However, role three, I need to update. So I'll use the update command on staging that priority roles.
7:03:28
We're gonna be setting priority level equal to three, and we need to list a wear condition of the role ID equal to three. Okay, let's go ahead and run this. Looks like no issues. I'm gonna take this query that we have right here, and I'm gonna insert it at the very end, and now run this. Boom, it's updated. All right, so not bad at all. Just taking a role, scroll through this script. It's looking like it's working pretty good. Let's actually, I'm gonna save this
7:03:58
by pressing command S, control S if you're on Windows, and then from there, taking this command and running the entire creation script scene if we have any errors. And inspecting this, it goes through, tells us the schemas, the tables, everything like that, and it gives us our final updated table. No error message, this is an idempotent script. So this was a little bit of a longer lesson, but I don't feel any of the keywords and commands that we went over,
7:04:28
really that intense or tough. So we now have some practice problems for you to go through and get familiar with all of these different keywords. In the next lesson, we're gonna be jumping into using these DML and DDL languages to create different types of tables, and also get into some more scary stuff of how you can delete or truncate tables. With that, I'll see you in the next one. All right, welcome to part two of three where we're going into DDL and DML.
7:05:02
This one's gonna be heavy on the DDL. In the first half, we're gonna go over three very common commands used to create different types of tables. We're gonna go over CTAs or create tables as, and you can provide a query after that and create whatever you want. A view, which technically not a table, it's a stored query definition, and then finally a temporary table, something that only exists for a session. Now besides just table creation, we're also gonna be going into
7:05:33
how to remove rows from a table. We're going over the delete keyword to just remove one or more rows or truncate to remove all rows.
7:05:44
There's just a quick refresher on DDL or data definition language and DML, data manipulation language. The majority of what we're covering today or in this lesson is DDL, and it defines and modifies the database structure. Specifically, we're gonna be creating different types of tables, and this affects the schema. Now, we do have one command today that is DML or data manipulation language, and this manages and manipulates data within the tables, and delete is this,
7:06:16
because it actually affects records in a table. Delete is the opposite of insert in that it can delete rows based on a condition. Now there's one other way that I like to think of how to separate these two. It's in their usage frequency. So DDL, ones that you use to, you know, affect the schema, create tables, create databases. That's used during database setup, whereas DML is more used on a daily basis with working and interacting with data to insert or maybe remove rows.
7:06:47
Now for this first section, we're gonna be doing or creating CTAs, a view, and a temporary table. But before we actually dive into that, like what are we gonna be actually creating inside of our data mart? Well, for this, we've already created inside of our data mart. We've created that main schema and the staging schema, along with priority roles table in our staging schema. The first example, we're gonna be using CTAs to create the job postings flat table,
7:07:16
which is just a flat version of our job postings, the list of the job postings and the company associated with it. From there, we're going to use that table combined with our priority roles to create a view inside of our main schema called our priority jobs flat view. And then finally, we're going to wrap up this table section by creating a temporary table where we're only looking at the senior jobs and we're gonna create a senior jobs flat temporary table.
7:07:43
Anyway, you don't need to memorize this. I'm gonna remind you. But first, let's get a general idea about these three different objects that we can create. CTAs means create table as select. This is a physical table that we create inside of a schema because we actually create it, it allows for really fast reads. Next up are views and these aren't technically tables. Instead, what it is is a virtual table. It's a query and then whenever a user goes to this view,
7:08:14
the query executes and then provides this virtual table. No data is stored with this and if you use this in a query, it's more likely than not going to be slower. And then finally, we have a temp table. This is just materialized data that is stored in a table but only for the session that you're logged into. So as soon as you log out of mother duck and you're no longer in there, temporary table doesn't exist anymore. All of these have a very simple syntax
7:08:43
that follows the same pattern where we're gonna be writing something like create or replace table and then the associated name that you're gonna give it and then from there as. And then following this as, usually we include some sort of DQL language that queries and provides what we wanna put into that table view or temp table. All right, so let's jump in and start coding this CTA's. For this, I've created a new file of 1.22 DDL, DML part two SQL file. So into the last lesson,
7:09:17
we're gonna be building it out on here and we're gonna be creating it to be idempotent so that way we can execute the script as a whole. And we can rerun it over again and it doesn't cause errors to happen. First thing I need to do is connect to mother duck and I'm gonna connect right to our jobs mart. So for this example, we're gonna be creating a job postings flat table. What this is gonna do is it's just gonna combine our job postings fact table with our company's dim table,
7:09:46
put it into one table. We're not gonna join the skills for now. So the first thing I'm gonna do because it can create table as select, I'm gonna first just build the query, make sure it's correct, and then use it to create the table. So with our select statement, I'm gonna just select all for right now, while specifically from our job postings fact table and from our company dim table. We're gonna find it, tone it down after this. And so I'm gonna do from job postings fact as JPF
7:10:15
and we're gonna do a left join because we wanna maintain all of those job postings in that job postings fact table to the company dim table as CD. And we're gonna do this on the company ID of both tables. All right, let's just go ahead and run this and see, well, you know me, I like to make sure we limit this first. I'm gonna just limit it to 10 values. And going ahead and run this, got an error in here. I'd have forgot to put a comma after our first portion in the select.
7:10:44
We'll try to run it again. Oh, silly me. It says hey table with name job postings fact does not exist. Did you mean data jobs? I did. Remember, whenever we logged into DuckDB, we specified that jobs marked. So anytime we're connecting to anything, we need to specify, well, anything outside that database, we need to specify it. So when both these, I'm gonna add data jobs. All right, let's hope there's no more embarrassments with this. All right, not bad.
7:11:11
I wanna see what columns are going on here. As we learned previously, we can just do, this is DuckDB specific, but I can run describe above this. And then running this, I can see all the different column names in here. Now, I want to pick out specifically what company name or what column names I wanna use in this. And conveniently, it's in this list right here. So I can copy this and just reuse this as necessary. Now, I could paste this inside of here and then try to clean it up.
7:11:46
This is a mess. This is honestly a job for AI. And I highly encourage you especially for this kind of case. I can pop up in the AI pane over on the right-hand side. And I can see in here that it has the file as context, including lines one through 27. I can say, hey, clean up this query. And it looks like it fixed it. I'm gonna close out of this chat so we can see whatever we did. And it has read for what it removed and green for what it included.
7:12:14
And it did all those different column names that I wanna do. This is really good. I'm gonna go ahead and keep all this. Now, if you don't wanna log in to GitHub in order to use that, probably fine, feel free to just use chat GPT. Anyway, that's cleaned up. And now I can clean up this list of what's going on here. Specifically, I'm gonna remove this company ID, along with down here for a company. I'm gonna remove the link, link Google and the thumbnail. Everything else can stay.
7:12:40
But the main point is we wanted to join in to get that company name in there. So let's go ahead and make sure that this query works. A Ranshift Enter. And it looks like it's working well. Okay, I'm liking it. So now that we have this operating query, we can use this to create a CTA's. So I'm gonna remove this limit 10 on the end here first. And then up at the top where the AI input this comment, I'm gonna do create table. And then we're gonna give it the name.
7:13:10
And we're going to put this into the staging schema. We're gonna give it the name job postings flat. And we're gonna do this as this entire query underneath here. Now it's a very important right. Remember we, whenever we logged in, I logged into the mother duck connected to the jobs mart. So I know I'm connected to that. So when if I go to run this query, it's going to insert it into there. Right now it's running through the query. It took about four seconds in mother duck to do.
7:13:41
Now it's done. So now let's query it to make sure that it's there. Remember we need to specify it using select star from the staging job postings flat. And I'm just gonna limit it to 10 values. Running this bad boy, boom, it's all there. And something that I actually should have checked before is what is the count of this? Probably the most important thing. Make sure we got all 1.62 million rows in there. So running this, well, it's in there, 1.62 million rows.
7:14:14
Now that we create the CTA's, we're gonna move on to creating a view. Once again, not a physical table, it's a virtual table and it's gonna be a query stored so that way other users go to it and see a fresh set of data. Because whenever they query it, it's in real time pulling whatever data is in those associated tables. That's like the one key benefit of views is that it's always updating. Whereas if you use CTA's to create a table,
7:14:42
whatever that snapshot is that you use of the table, that's locked in place of whatever that time it was. Now for this, we listed previously, or we created previously that priority roles table and now we just created that job postings flat table. We're gonna combine these two to create a priority jobs flat view and we're gonna put this inside of our main schema. So basically it's our job postings data warehouse that is filtered for job postings that the business defined as priority
7:15:14
in the priority roles table. Now similar to last time, we need to build up the query first that we're gonna then be putting into the view. So for this, we're gonna select right now, we're just gonna select star, select all and we're gonna be coming from the staging of that job postings flat. We're gonna give this the alias of JPF. Slightly confusing with job postings fact. And we're gonna join this to our priority roles in staging and we'll give it the alias of just R.
7:15:46
And we're gonna do this on that job title short column 'cause technically we don't have all the different job IDs in our priority roles. So we're gonna do that job title short and match it up to the priority roles role name. Now let's go ahead and execute this. We'd have to fine tune this query a little bit more and it came back and here's what we need to fine tune. And mainly, I don't really care about including well, anything from the priority roles table.
7:16:16
So role ID, role name or priority level. Because of that, I'm gonna just specify JPF.star. Now we do have one more condition and that's that we only want those with a priority level of one, only the most important data. So we're gonna put a where condition on this. So we'll add an aware priority level is equal to one. All right, let's go ahead and run this bad boy. And it looks like we got about 480,000 roles back, not too bad. And it looks like also all the columns
7:16:51
are filtered as necessary. All right, now we can go ahead and build out our view. So above this select statement, we're gonna go ahead and do create view. And then we need to give it a name. With the name staging dot priority jobs flat view. And then we always need to leave that or as at the end. All right, now let's go ahead and create this view, run shift enter. All right, and like before, we need to query it. So I'm gonna copy this query above here
7:17:18
that would be queried for the other table. And then change the table name to priority jobs flat view. I'm gonna go ahead and run this. Okay, looking good, there's less counts in there, but that's not the actual verification I wanna do. Let's take this query of this view a step further. Specifically, I wanna filter this for the job title short column. And then with each one of these counts, we can label this as the job count. Now, because we did an aggregation,
7:17:44
we need to do a group by specifying that job title short column and also order it. And that's by the job count in descending order. Okay, let's go ahead and run this, make sure it's running correctly. All right, good. So we have senior data engineers and data engineers. The priority level ones in there and those values for the amount of jobs that are associated with it matches to what I expect. Now I'm gonna go ahead and save this. Remember, we wanted to create this script
7:18:11
to be audio important. So that way we could run this again. But if we were to try to run this again by doing that dot read command and then putting in the name of where this is located, I'm gonna right click this and go to copy relative path and then paste that into here. Now, if we were to do this, we're gonna get an error. We're gonna get two errors with this. And that's because when we went to create it, it says, hey, table with the name job hosting flat
7:18:38
already exists and the priority flat view, job flat view already exists. Now for both of these, they have an optional parameter we can add where after the create statement, you can say or replace. Remember and when we did create or drop a database, we could have if exists or if not exists. Well, this is very similar. In that you can either create this new object or replace it if you're running this again. So for our first query, I'll go ahead and put or replace in here.
7:19:07
And then for our view, I'll do the same thing and do replace or replace. Now saving this and now running this again, this takes a little bit more 'cause now it has to create every single time, created the tail about 3.44 seconds. We see the count of the table and then we visualize that view. All right, so now moving into the last table that we can create as a temp table or temporary table. This is a session scope table. So as soon as we log out of mother duck,
7:19:39
it's gonna disappear forever. It's really just should be used if you need to basically have some complex query that you need to get some data and you just wanna stick with that data to further manipulate more. I do it all the time when I'm running analytics on a table and I have this complex query that I need to go a little bit further. Well, I can save it as a temp table and then do analytics on that temp table. So for our example, we're gonna create a senior jobs flat temp table,
7:20:06
which is just our senior data engineering role inside of a temporary table that is aggregated or has all the results of the job postings flat. So let's first start by just creating our query. For this, we're gonna be doing select star of all of our data. And this one, since we already filtered down for our priority jobs flat view of just the priority jobs, I'm gonna use that view. So I'm gonna say from, and then I'm gonna paste it into here the staging priority jobs flat view.
7:20:35
And then it's gonna be where that job title short equals to senior data engineer. I know this query isn't that complex, but use your imagination here and that you may have even more complex queries that you wanna save into a table to use. All right, let's just run this bad boy to see if we get the correct results and looks like I have some errors in here and had a little bit of issues. I was using double quotes around senior data engineer.
7:21:03
We're gonna show you been using senior single quotes. Anyway, this was throwing issues. Great, change it to single quotes and it's working just fine now. When you use double quotes in DuckDB, it treats it like column or table names. So you don't wanna confuse it. So anytime it's text values, use single quotes. Anyway, this is populating just fine. We want to now create this as a temporary table. So we're gonna do create and you can do temp table,
7:21:29
but it's more acceptable and more databases just do temporary, I'll type it in all caps. Now this one, we're not gonna specify a schema because it doesn't live in any sort of schema. Actually, you know what, just to show it, I'm gonna put staging in here and put an as on here. If I try to run this with staging on there, it's gonna give me an error. Actually even tells me that schema doesn't exist, but that's not the real issue here.
7:21:55
The real issue is we don't list a schema for temporary tables 'cause it doesn't go into any schema. So now running this, boom. Now that's available. Now what I can do, I'm gonna copy this query above and I'm gonna take that name of that temporary table 'cause that's what we want to use instead for the from and I'll replace that in here. So now whenever I run this, I should only see senior data engineers, which I do and it's the correct count. So real quick refresher on this, CTA's
7:22:25
or create tables as select store rows. Views, don't they store the query? And a temporary does, table does store the rows, but it's temporary, it's only for that session. CTA's and views can be viewed by everyone, whereas the temp table is for you, it's a linear session. Views are really great because they are fresh by default and that whenever they run the view, it automatically runs the query and connects to the appropriate tables and queries it. However, that also makes it slower.
7:22:52
Compared to something like a CTA's, that is not fresh by default, but hopefully your tables are built in a way that they are maintaining that freshness into it. And there's the same four temp tables. So here's how I like to think about it. If you need the latest data for every query, use a view. If you need fast reads and stable results, use CTA's. If you're just testing or debugging, then use a temp table. And last after this, we're gonna cover two commands of delete and truncate.
7:23:25
Delete is, I like to go at the opposite of insert in that it removes specific rows. This can be slower on bigger tables, but it's best for whenever we wanna do targeted deletes within a table. The other command is truncate and this would be in a case where we wanna keep that schema of the table and then maybe refresh it with some new data. This is a little bit of a faster way to empty a table. Now, I put this also along drop table because drop table is another way to delete data
7:24:02
and that is just removing the schema and table and data all together. So you have three different options depending on what you need to do. The syntax for this is pretty simple. For delete at least, it's delete from, you list the table name and then where as the condition. What condition are we trying to meet with this? Well, the business unit came back to us and they told us inside of our job mart, they only want data from 2024 and beyond. Anything before 2024, they don't wanna have
7:24:37
inside of their job mart. Not my choice, it was their choice. So for this, we need to modify our source table or that staging table of job postings flat that we brought in and we just wanna remove that from any values before 2024. Now, it's important to monitor what's going on before and after this. So I'm gonna run some queries on looking at the row count for each of the three tables that we just made. Specifically, we're gonna look at the count from that job postings flat table,
7:25:10
which is the one we're gonna alter. Also from our priority jobs flat view. And then finally from our senior jobs flat temp. Let me put semicolons all this. Let's go ahead and run these, make sure they're running correctly. All right, it looks like they are in that we have 1.62 million error of fact tape or the flat table, 483,000 in our view and then finally 91,000 in our temporary table that's only senior roles. Let's get into deleting this data. So we're gonna do delete from
7:25:40
and that's the staging job postings flat table. And that is where the job posted date is less than put in parentheses, single parentheses, January 1st, 2024 and I'll put a semicolon after this. Okay, and then after this I want to run this as well. I'm gonna just put this here. Now, anytime we're about to run a delete and you double check it, looks like it's correct. All right, I'm gonna run all of this at once. So with this initially, we had 1.62 million
7:26:10
in that flat table and then going down after this, it went to 828,000 jobs. With our view, we had originally 483,000. After this right, a view is a fresh snapshot so it went down to 251. Now, however, for our temp table right here, right, we started at 91,000 and we ended at 91,000 because it's a temporary table and we didn't update it. Now, let's get into Trunkate. For this, if the business unit came back to us and one of those jobs that were more fresh, that were 2024 and greater,
7:26:49
but they had concerns that maybe some of the job postings in the original data warehouse that updated, this would be a good use case of Trunkate in that we can wipe the entire table of this and then rebuild it back up with the contents because we know the scheme is gonna remain the same. This all depends on delete or Trunkate, depends on how many rows you're moving, how long this is taken, is the data upstream, fresh or not. You have to take this all into account,
7:27:19
it's really gonna depend on the scenario you're in. Anyway, let's demo this. To do that, we need to get back in that 2024 data and pretend that it's in there. So what I'm going to do is I'm gonna just come up to the top and run that CTAs that we use to create that job postings flat table. It loaded, took about four seconds and now running the count on it. I can see that it's 1.62 million. Okay, we have a good starting point for this.
7:27:47
Now for this, the syntax is pretty simple of Trunkate table in that you write Trunkate table and then the name of the table, so job postings flat. Now, whenever you run this, this is going to take everything out of the table. Let's just actually demo it. I'm gonna go ahead and run this shift, enter. And now whenever we query this staging.job postings flat table, there is, I guess a little bigger, there's zero rows inside of it. So now we need to fill it in.
7:28:17
And we know the structure is there 'cause if I do a select star from that job postings flat, all of the columns are still in there. So we just need to insert into it. So for this, we're gonna use the insert into command. And we're going into that job postings flat table. Now previously, whenever we did this in the last lesson, we went through and we specified all the different columns and then the values that went into it. And especially, and like I said,
7:28:46
these names of these columns can be optional, but you have to make sure that it'll end up properly. Anyway, we're gonna be a little lazy with this and we're gonna insert into this this entire select statement that we used for the CTAs up above it. So we'll go ahead and select this all. I'm gonna close this off to the side, put this into here. And remember why we even wanted to do this from the first place in that we wanted job postings
7:29:13
that were more fresh, that were 2024 and greater. So for this, I'm gonna put a where statement filtering our job posted date for those that are greater than or equal to January 1st, 2024. And then after this, I'm going to also post this down below underneath it to make sure that we have everything updated. All right, let's go ahead and run this up from truncate table, let's find a run truncate table again. And boom, we have that job postings flat, updated that 828,000 like we expected,
7:29:47
that view back to 251,000 and like usual, the temp table not affected by this. Now, delete or truncate are also gonna depend on how many rosers? going to removing or rows you're going to be adding. If you're removing almost all the rows, you're probably going to want to truncate. If you're only removing a few rows, you're probably going to delete really going to be situation dependent. All right, it's your
7:30:08
shot to give it a try now. We got some practice problems for you to go through creating different tables, views using delete and truncate. We only have one more lesson next on DDL and DML. And it's going to get even more spicy from here. All right, with that, I'll see you in the next one. All right, in this lesson, we're going to be going over sub queries and CTEs or common table expressions. For this, we're going to do a brief intro to explain
7:30:37
what those are sub queries and CTEs, and then do an example of each along with a more complicated final example. And one quick note, we're taking a short little break from our DDL and DML lessons. Remember, we have three lessons for that. We've completed two. The third one will be following that. We have a concept in here of CTEs that we need to master before we can actually go to that third part. So what the heck are sub queries and common table expressions?
7:31:09
Well, they're used for organizing and simplifying complex queries. Here in this image, we're showing the syntax differences between a subquery on your left and a CTE on the right. Sub queries as the name implies are queries nested inside a larger query, it's sub query. And this is done by putting the query inside of parentheses. And you can also provide it an alias specifying it as in this case, that engineer jobs, common table expressions are CTEs, define a temporary
7:31:43
result set that you can reference. And a temporary result set is just a fancy way of saying that it's a intermediate table of the rows of data that we're going to use for a query and then get rid of it after. This is the same general concept also behind CTEs as well is that they're a temporary result set. Anyway, we're going to quickly demonstrate each using a simple example. First thing you need to do is start a new file. And we're going to need to connect
7:32:09
to our database in mother duck. All right, we're connected to that data jobs data warehouse in mother duck. All right, first up is our sub query. Remember, that's a query nested inside a larger query. For this, we're going to do a very simple example. We just want to filter down our job postings fact table to only provide job postings with salary data. So we're going to select star of all the columns. And then in our from clause, we're
7:32:37
going to open up parentheses. And this is where our seat or our sub query is going to be in here, we'll do a select star from job postings fact. And we'll filter down in a second, but I just want to make sure that this query first works. So I'm going to do a limit of 10. And then I'm going to run this query. All right, so looking good, right, we can see that we did this. Now we want to inside the sub query, filters where we can
7:33:06
actually see values for salary, your average and salary, our average. So for this inner sub query, we're going to add a where clause where salary or average is not null or salary our average is not null either. All right, let's go ahead and run this bad boy. And we get this back, we get all this where we have actually salary down and filter down in our sub query. Now, real quick, this is just a simple demo. Obviously, you probably can
7:33:33
just, well, you can, you can just rewrite it with the query inside of it. But I wanted to demonstrate, I'll simple a sub curious. Anyway, let's now rewrite this using a CTE. Now for a CTE, we define that temporary result set at the beginning of the query using the with keyword. After that, we need to give it an alias. So we'll say for this valid salaries. And then we use as, and then in parentheses, similar to a sub query, we're going to define
7:34:02
the query itself. It's the same one from up here up above. So I'm not going to go ahead and just copy this all above and paste it in here. Now we need to actually display this temporary result set. So we can now provide it in the name of that temporary result set of valid salaries. We'll say select star from valid salaries, put a semicolon in the end, copy this all, and then go ahead and run this. I didn't do a limit 10 on this one probably
7:34:32
should have as well. But looking at the results from this query, we can see that we get it back. And we've actually filtered down to include only job postings with an associated salary. Okay, so that was a quick demo of CTEs and salaries. One minor note in that CTEs, I use this valid salaries, I define this alias of the CTE right here. You can also in the sub query, give it an alias, such as as valid salaries. And then whenever I go ahead and
7:35:02
run this still provides the same results, doesn't really change this one in any way. But it's typically a good practice to provide an alias so people understand what's going on. We're now going to dive deeper into sub queries. Previously, we're just demonstrating using the sub query of a temporary result set inside of the from clause. But we can actually use it in multiple different locations, such as the select statement, the from statement,
7:35:30
where and also have it. So we're going to tackle this first one once, first one first of using a sub query in the select statement to display what is the overall market median salary based on well, all the job postings and display it next to each job title short. So we'll start with the select statement. We'll query query that job title short column. And then we'll next provide our sub query in here. We're going to open parentheses.
7:35:57
And in here, we're going to select our median salary or average. We're only going to select one column for this. That's the key to making it work. And then we're going to be doing it from that job postings fact table. For this, we're going to give it an alias of market median salary. And now getting back into that outer query, where is the outer query from? Well, it's from that same table of job postings fact. Right now, we're going to just limit
7:36:23
the results to 10. Let's go ahead and run this and see it in here. All right, so pretty cool. We have here the results. Now, this isn't really that helpful. What I do want to display also with this is we have the market median salary, we can see that all of the media are the same as we would expect. So that's pretty cool. But I went next to this the actual salary of what it is for each one of those job postings. So we'll add in that
7:36:48
salary or average column. And for this, I don't want to see any null values. So I'm going to add in a where clause of where the salary or average is not null. Let's go ahead and run this. All right, not too bad. This is pretty cool. Because we now can see next to each one of these job postings and their correlated salary for that posting what the median salary is for the entire data set. Now let's move into scenario two, where we're
7:37:14
going to build on this query further, in that now we want to only stage jobs that are remote before aggregating it. So we're basically going to be using a sub query in the from clause to do this. So I'm going to start with that previous query that we had, and go ahead and paste it into here. And sorry, I think I should be more clear, stage only jobs that are remote before aggregating. And that's to determine the remote median salary per job. So instead
7:37:42
of just looking at that salary or average column, we're going to instead now do a median. And since we're doing this aggregation, right, we need to now do a group by, and we're doing that by that job title short, this market median salary is technically an aggregation. So we don't need to do it for that. So I'm going to go ahead and run this. All right. So now based on the query they've placed in, we're getting based on the job title short
7:38:08
column, we're getting their associated median salaries. And next to it, we have the market median size. This is even better at what I would probably want compared to that last query. Also, I noticed here I actually didn't alias that median salary. So we need to do that. So add an alias of median salary. And also, we need to get back to that main thing that we wanted to do of adding a sub query in our from to ultimately to determine that
7:38:33
remote mean salary per job. So we're going to replace this job postings fact with parentheses to put our sub query in. And I'm going to minimize this so we can actually see this all. We need to make sure that we query all the relevant columns that we need to provide to this upper query up here. So you could do a select star, but I don't really like that, we're going to do job title short. And that salary or average is that's better practice
7:39:01
to name the specific columns. And we're going to be doing this from the job postings fact table. And we want to do it where the jobs are remote. So we're going to do the job work from home has a true value. We're going to give this the name of as clean jobs. But as you can see, this alias isn't really getting used anywhere else in here. All right, this should be working as is let's go ahead and run this. And then taking a look at the query
7:39:29
results, looking good. They are different, right? Because initially senior data engineers was 147 500. But now senior data engineers are 145, well, 1000, I don't know, I say 100 previously. But look at this, this market median salary is still 116950. And when I scroll up here, it's 116950. So let's go back to that query. And the problem is this is this why I wanted to do this example where we combine sub queries. This query, this sub query inside
7:40:05
of here that we're using the select statement is its own query. And it's not affected by any of the filters for the outer query. So if I want this query to also provide the market, or in our case, the market remote median salary, well, I'd have to add a where clause in here as well for where job work from home is equal to true. Okay, let's go ahead and run this. See now, our market remote median salary is 130,000. We're previously was that 116950.
7:40:42
So the market remote median salary is slightly higher. Pretty interesting. Now one quick note on this original query that we have here, we're doing aggregations of, well, we're doing aggregations inside here. So this whole way of salary or average is not null, it's not necessary anymore. I'm gonna go ahead and remove this. And then if I go ahead and check it by running it, I'm still getting those same results that I got above. So anytime you
7:41:07
can simplify queries, do it. So now, let's get into the third scenario in where we want to use a sub query in our having clause. And now for this, we're going to use the having clause to filter to ensure we keep only job titles where the median salary is above the overall median. This will be for all that remote jobs that we did previously. This is going to be building on our previous query. So I'm going to just go ahead and copy this
7:41:34
from above, paste it down in here. And then as we know, the having clause goes after that group by, and we're going to be using this for to make sure that we're above the market median, or the market median remote salary. So we're going to run the median function again on salary or average. And we want those results provided to be greater than what's inside of our sub query. And conveniently, we provided it up here as the market remote
7:42:04
median salary. So what I'm going to do is I'm going to just go ahead and copy this all, come and see, and then paste it in. All right, let's go ahead and operate on this to make sure we're operating correctly. Okay, we got the results. And as we can see now, that market remote median salary was 130,000. And only results greater than that are being shown now. So not bad, we've demonstrated how to use a sub query in the select statement from
7:42:31
and also having, we'll go into it the final example, how to use a sub query within a where clause. Now, I do want to point out something real quick with all of these different scenarios that we did. Whenever you're building these out, I like to sometimes even start with the base query or that sub query, if you will, first. So if I were to just run this first one right where we were going through, and we want to get the market median salary within
7:42:59
the select statement, I can just run it by doing shift enter. And then it puts it down into the terminal at a semicolon, and then run it. Anyway, it provides only one value as a result. This 116950. And that is what is being put into this entire query, which we'll go ahead and select it all, and then run it. That's what's getting put into here. So any of these queries, if you'd like to dive into more about thinking about how to
7:43:27
build it, it's as simple as making it into a state that is being that's able to be used. And what I mean by that? Well, for that select statement, well, moving into scenario three, remember, we only had one value for that select statement, because we just want to put one value per row. Whereas for our from, right, we had an entire table with the twitch of I run this right, adding a semicolon on to this and running it, this output more of a table
7:44:00
or a multiple values that you would expect to use as a table in a from statement. And then the having statement in this case was what we use previously, right? Whenever I'm going to run this, putting a semicolon on the end, it's just a single value. So that's what I mean. It has to be in the required state for it. For example, let's actually make an error in here. Let's say for this select statement, right, we had it needs to
7:44:28
be a single value to be put into the column. If I were to get rid of this meet this aggregation function, and basically provided a list of values, or so I'm going to run this, and I'll put the results below. If I run this right, it's a it's multiple values that it has that will be providing this query within the select statement. So if we tried to run this in this form within here, it's going to give me an invalid input error, because as it explains
7:44:59
more than one row returned by a sub query, you use as an expression, scalar sub curatories can only record return a single row. They have an option down here to change the behavior by studying a certain variable. So we're not doing that basically, you need to just write the proper query for what our intended use is. So I'm going to go ahead and add median back from here, and hope you take a learning lesson out of that, I sort of glossed over
7:45:25
checking these. But anytime you're building these sub queries, feel free to execute it first, and then put it within the main query. Let's move into CTEs now or common table expressions. For this is a similar and that we're creating a temporary result set in parentheses, but it's defined above the query, we're going to use it with a will with the with keyword defining the alias and then what's next in parentheses. Now, a few short notes on this,
7:45:56
similar to the last sub query, it's using a lot of different places, right? So so can CTEs, what is defined as the alias, we could use it in a from in joins, you can use it in other CTEs, and that's actually pretty common, as you can have multiple CTEs on stacked on top of each other. And then you can use it inside any main statements, such as select, insert, update, or delete. It's used with that with keyword at the beginning to define
7:46:21
that CTE, and it exists only during the execution of the query. I don't think I mentioned it with sub queries, but you can also use sub queries, this is a CTEs, you can use sub queries, you can use join with a sub query as well. So moving into our CTE example, for this we're going to compare how much more or less remote paying roles compared to on side role on side roles for each job title. So for this, we're going to use a CTE to first create this temporary
7:46:53
result set where we have a job title, median salary, and whether they are job work from home or not. And that's for all the different job postings. Then with those CTEs, we're going to be able to make it into a table that compares one line by line, how it looks per job title. Well, let me just show you the solution. In this, we have the job title short column that we aggregated it by. And then we're going to have the remote median salary right next to the
7:47:23
onsite median salary. And then because they're on the same line, we can also get what is the difference between the two that we're going to call the remote premium. And this is all possible, we can do this because of CTEs. So we're going to be building a CTE, but I'd like to typically just write the query that's going to go into the CTE and then wrap it in a CTE. So remember, we need three different columns. We need the job title short, job work from home, and then that median
7:47:51
salary, we're going to give it the alias of median salary. This is all from that job postings fact table. And then because we did aggregation, we need to do a group buy of job title short and of job work from home. I put a semicolon on this, let's go ahead and execute this, make sure it executes. And it does. And notice for this, right, it's aggregated. So it has each of the job title shorts, and then it has for a false value in the true value, the market median salary.
7:48:18
Now, I'm going to modify this question slightly. I like to narrow it down to a region. So I'm going to change this or I'm going to add on to this a where statement in that specifying that the job country is specific to me of the United States, you can also modify it to your home country as well. And then going and doing this one. All right, not too bad. This is looking good. Comparing it to that table above, I can see there are some slight differences with this. It looks
7:48:46
like, well, for the United States, it's it's slightly higher. Anyway, one thing I don't like about this is this has some crazy, that look, I mean, look at that long decimal place. I'm not a fan of that. So what I'm going to do is I'm actually going to cast this market median salary as an integer to just make this a little bit easier to read. And then after running this, I can see, okay, everything's cleaned up. It's looking a lot more readable. I don't need down to the decimal
7:49:11
place with yearly salary. All right, so going back to that table that we're working towards getting, we now need to wrap what we just did into a CTE. So we can get this displayed in this manner, where we have the remote remote median salary right next to the onsite median salary, and it's broken down by job title for each line. And then after that, we'll focus on that remote premium. So I'm going to do first, I'm going to dent this over by highlighting it all and pressing
7:49:41
tab, and then giving a with keyword. I'm going to call this the tidy title median. We'll do as, now open parentheses, and then we'll wrap this all within parentheses. Now I'm not going to put a semicolon on the end of this because I want the query to continue. All right, so the first thing I'm going to do just to make sure that I'm building correctly, I'm going to just do a select, and we're going to look at the job title short, and the median salary, and this is going to be from our
7:50:07
CTE, which that title median. Okay, I'm going to put a semicolon on the end of this. All right, let's go ahead and run this all. And it has a, I have a parser syntax error at or near that parentheses. I never took off that semicolon on the end of here, as I should have, you can't have a semicolon in that CTE, because then it tries to end it. Okay, let's try it again. I also have an issue goodness gracious, I have a little issues today. I named this incorrectly, specifically
7:50:34
I named it the market median salary. I'm actually not sure why I named it market median salary. It should just be median salary. So that was a little bit of a goof up. Okay, let's try to run it again. All right, we got it. Okay, so this good. But remember, we want to aggregate this or just have, you know, let's do first, we're going to focus on just remote median salaries. How are we going to do this? Well, here's what we're going to do. What we're going to do instead is we're
7:50:59
going to provide a where clause, and where job work from home is equal to true. So the remote. Now, what we're going to do is, you'll follow along with me for a second, we're going to assign the alias for this table of title median as our basically remote, short for remote. Okay, I'm going to define this as r dot, this is our dot, and then this as our dot, because this is all aliased here. If I go ahead and run this query, I'm going to have, I'm going to have an error because I misspelled
7:51:32
where, when I run it, it runs just fine. And it's now filtering just for those remote jobs. Well, how can we bring in now those onsite jobs? Well, we can do a join. So what I'll do underneath the from is add an inner join, and we'll add join on that same table. But we'll give it in this case, a different alias of Oh, for onsite. And we're just going to be matching it up on that job title short column, because that's what we have to similar between the same table basically. So we've
7:52:05
added that job title, sure, we want to do it. And we need to filter those now that we're bringing in with Oh, because we want to bring an Oh dot median salary. So these median salaries, we need to filter that are coming from the Oh table. So I might go ahead and put an or in here for Oh dot job work from home equal to false. And I've been really bad about given aliases here. I need to give aliases
7:52:33
specifically, this is the remote median salary. And this is the onsite median salary. All right, so now we have both of these defined now of what this is, we're filtering the appropriate table. Let's go ahead and run this bad boy. Well, we got to run all of it, including the CTE. And I have a sent a parser syntax error at our near our basically I forgot to do specify on for the keyword. I said, interjoint title median as Oh, and then you say on this. All right, let's go
7:53:05
ahead and try to rerun this. Okay, getting a little bit of an error here. And that's, well, no, not an error. Again, issue with the table, right, we're seeing that these job titles are duplicated multiple different times. Specifically, the problem is with our filtering on remote and on site. Right now I messed up that query. I did where the remote jobs are true, or the on site are false. And this basically allows it to do every different combination, because everything's going to be
7:53:41
true or false. Anyway, I need to change this or to an end. Now going ahead, rerunning this. Bam, we got it. All right, so looking good, we now have it in the same table. Now what we can do is we can calculate that remote premium by subtracting the on site median salary from the remote median salary. So let's go ahead and add another line in here. And we're going to do the r dot median salary from the remote table, and we subtract from the on site table median salary. And we're
7:54:12
going to give this alias of remote premium. Also, I want to order it by this. So I'll add an order by and add in remote premium and put it in descending order. All right, let's go ahead and do execute this final query. Not too bad. It looks like things like machine learning engineers have the biggest premium along with data scientist that engineers and senior data scientist. Not too bad. Okay, so
7:54:41
it's a great example of how you can use CTEs to manipulate data and basically compare it within a single table. All right, let's move into a final example. And for this, we're going to be focusing on sub queries. Specifically, we're going to be using existence filtering in order to use a sub query within a where clause to filter the data in a table that we want when comparing two tables. Now for this, we're going to be looking at two tables. Previously, when we did joins, we did
7:55:17
like an A and B table. For this, we're going to use the nomenclature of a source table and then a target table. This is very common in data engineering to have a source table that you're taking data from and a target table that you want to, well, targeting get it into. And there's two different existing existence filterings that we're going to go through. The first one or both of them are ones where exist. And that's going to be filtering where the data in the source table is also in the
7:55:45
target table. And then we'll also look at conversely where not exist, where the data is in the source table, but not in the target table. The thing about this is just has it uses the not keyword in front of exists, not that special. So let's break this concept down. Let's pretend we have a source table with three job postings in it and a target table with two job postings in it. The source table has a job idea of 101, 102, and three. And the target has 101 and 104. So we can see that 101
7:56:16
exists in both of these tables. So how do we get this result via a query? Well, we first query the source table with this, we're doing a select star from the source table. And we want to filter it with a where clause where exists. And we put our sub query inside of here. Now this sub query has a common syntax that you'll see with anybody that uses existence filtering, in that it starts with a select one. And that really that one value can be any value, it's just going to return a value.
7:56:49
If we're going through it, if the target table has where a job ID matches up, it's going to go through each one of those job IDs in that target table, compare it to that source table. And if in the case of 101, it returns a match, the select one will actually return something. Whereas if there's no match, like in the case of 104, it returns nothing. And we don't get anything for 104 in that return statement for the where statement. So therefore, we just get our return table that
7:57:20
has 101 in it. Conversely, we can use something like where not exists to keep the rows with no match in the target. In this case, we see from our source table, we have 102 and 103 in here, that do not exist in the target table. So with this new query of select star from the source table, where not exist, that sub query will go through and first in that target table, we'll say, hey, does match up with a 101 job ID in both tables, but we're using a where not exist. So we will
7:57:52
tell it not to include it in that final query. And then 104 doesn't have an associated job posting. So therefore, there's no match or anything to return with that. So our final results table only has 102 and 103. So I want to just walk through a very simple example, using a new function that we haven't seen before, specifically of select and then the will not select. But range, and I'm just going to put a number in here of range of 10. We'll go ahead and run this. I forgot to add
7:58:20
a stomach colon, I did. Anyway, what it returns is a list of values or 10 values. And it doesn't return 10 because the number 10 is exclusive. It's not going to do it. But there are 10 values there. Really, this could be anything I can make this even range three, put a semicolon on this time and run it. And this will be 012 and not three. Anyway, what I can also do with this, I can manipulate this a little bit further to make this into a table that we can use as a source and a target
7:58:47
table. So I'll say select star right now from range three, go ahead and run this. Okay, now it's providing me a table of 0123. I can then give it an alias. I'm going to give this the alias as source, because this will be our source table. However, when I run this, we can see that, okay, you don't really see anything that the table's name source. But I do want to name this column right here, because we're going to do this existence filtering, right, we need to be able to compare
7:59:15
the values in here. So what you can do is you then if you want to name the values of a table you've created through something like this dysfunction, you put in parentheses, and I'll give it the column name of key. All right, let's go ahead and run this bad boy. All right, we can see it replaced up at the top. Now it is key for that column name. And so we'll say this is our source table. And then I'll go ahead and create just copy it right below this. And I'm going to name
7:59:42
source for this one to we'll make this our target table. And this one will just be of the range to go ahead and run this just for safekeeping. Yep, 201, no two. So let's use existing filters now to keep. It grows with a match that are in the target that we're in both the source table and in the target table for this, we're going to use where exists. So start with, I'm going to just copy this query up at the top, paste it down underneath. So we're selecting star from, on the source table.
8:00:20
And now we want to apply a filter for it, the where filter where exists. Then we're going to start our sub query in parentheses for this one. This one's going to start with select one. And we're going to try different values in a little bit. We're going from this target table right here. So I'm going to just go ahead and copy it right here, remove that semi colon. And we're going to do this where, and we had to name those columns, right?
8:00:44
Cause now we're going to compare where the key of the target is equal to the key of the source. Okay. Now what we should see right as we compared previously, this should keep the zero and one from the both tables whenever we go to compare this. So let's run it. Oh, so let me kind of select all of this. Okay. It does return that zero and one conversely, we can use where not exist to keep rows with no matches in the target. So in our case, that source table has zero, one, two,
8:01:18
the target has zero, one, two is the value that doesn't appear in the target. So if I change this to where not exists and then go ahead and run this bad boy, two is what is returned. Like I mentioned before, that one in the select one is a common way of sharing what, or what symbolizing this is existence filtering. You could do any other number. I could do some like one, two, three, whenever I run this, still get the same result. Heck,
8:01:47
I could even do data nerd or I can even do some like select star and that works as well. So it doesn't really matter what's in there. One is the simplest query to do. And then when we have that matching key return result, and so we're using the least amount of data, so I would use select one and not something like select star. All right. So now that we've gone through that entire thing, let's actually get into the final example of identifying the job postings
8:02:12
that have no associated skills before loading them into, let's say a data more, this is a very common thing to do using this existence filter filtering. Remember from our entity relationship diagram, we have that job postings fact table, which has all those different job postings and then related through the skills job dim and then to the skills dim table, they have the associated skills in there. So if I were to just query it,
8:02:39
this is just a simple query. I'm querying the job postings fact table and then ordered it by the job ID to show the first 10. And we get back this table. Notice that the numbers here, what I'm trying to prove with this is the numbers are all sequential. There's no missing values in between your foot. It starts at four, five, nine, three. And in this case, just goes to four, zero, six, zero, two. Now also with this, let's then now look at what I want to show is look at the
8:03:05
skills job dim table. I also reward by job ID. I'm going to go ahead and query it. And the thing I want to show with this, that query that's now returned is if there's not an associated or if there's no skills associated with a job ID, there's just going to skip over that job ID. So if I scroll down looking through this, what I can see is we get to four, six, zero, one, and then skip over four, six, zero, two, because there's no skills associated with that.
8:03:30
And it goes to four, six, zero, three. Similarly, four, six, zero, five also doesn't have any skills. So things like four, six, zero, two and four, six, zero, five are jobs that we want to basically capture in jobs that don't have associated skill. So we're going to use existence filtering specifically where not exist to keep the rows with no match in the target, where in our case, the source table is our job postings fact table. And our target table is our
8:04:00
skills job dim table. For this, I'm going to take this query that we used up here and just paste underneath the start with, we'll put a wear clause. And for naming sake, I'm going to just keep this as we normally didn't do in the alias of as JP app, we're going to do target. So TGT, we're just doing it for this example, just to make sure that we wrap our hand, wrap our minds around target and source. And then we're going to do a where not exists. Start our sub query. And for this,
8:04:30
we're going to do the select star from skills job dim. So I'm going to paste this in underneath here, making sure that it's indented over. We're going to give this skills job dim the alias of source. And we're going to be doing this where the target job ID matches up with the sources job ID. Also, so let me I have select star in here. Remember, that's not good practice, we're going to do select one. And I'm going to keep that order by in there still, because I want to be able
8:04:58
to find those two jobs that we are identified should not have any associated skills with it. Let's go ahead and run this bad boy. And this query ran took about two seconds to run. And the first two postings on it are the four six zero two and four six zero five, which we previously identified. And it has a total of 285,000, not bad. Now, remember, we have figured what 1.6 million job postings totally. So I could just to double check, I could remove this not from
8:05:29
here. And also see how many job postings are returned with this. And in that case, they have 1.33 million rows. So this plus what we've got last of about 300,000 rows gets us to that 1.6 million jobs. Okay, this makes sense. And it's checking out that our query was built correctly. All right, you get some practice problems. And now go through and get more familiar with CTE sub queries. And also through in some existence filterings. With that, we're going to be moving back
8:05:58
into DDL and DML part three to do a pretty interesting and fun data pipeline that I'm excited to share with you. All right, with that, I'll see you in the next one. And this lesson, we're in our third and final lesson on DDL and DML. And specifically in this one, we're only going to be covering really one new keyword. And that is of merge. This DML or data manipulation language keyword allows us to wrap up within one statement, an ability to perform an
8:06:34
insert update and delete. Now, before we move into this and actually build the query or the table that we're going to get into, which again to the table in a second, we're going to go through a brief refresher to build out that same query that we're going to do with merge, but instead using just insert update and delete. And we'll be doing this using that existence filtering that we learned about in the last lesson.
8:07:00
So what the heck are even going to be performing this merge operation on? And why the heck do we even need it? Well, let's get into it. Well, for this, we need to understand that there's two major types of processing that's batch processing and continuous processing batch processing is by far the most common type of processing. So that's what we're going to be going over and using merge four. It's used for incremental loads or ETL or even with DBT
8:07:28
models, it's scheduled to run on fixed intervals, processes, data's in groups, and only updates or changes new rows. Now, all the hype nowadays, you'll hear about this continuous processing. But to be honest, it's not as frequently used. This is for streaming and real time data pipelines. And this processes data in real time as it arrives, each record that triggers this new logic, it's event driven. So every time there's a new record, it processes in to go
8:07:59
into where it needs to go. And overall, it's built for live systems. And the reason why I say continuous processing is in that common is most times we are working in data teams where you don't need to see analytics in a real time. You could have some sort of day delay and it's going to work just fine with management to make their decisions. So for our problem today, we're going to be focusing on batch processing. So what the heck are we building? Well, as a quick
8:08:26
refresher, remember, we have our data jobs, data warehouse, and that's the same data warehouse been using the same course. Now over the course of our DDL and DML lessons, we've been building out this job marked inside of it. We've had two schemas of a main and staging. And we've built things like temporary tables, regular tables, and even views. Now we're going to create one last table. And this is going to be built with our merge statement. And that's the priority
8:08:54
jobs snapshot table. Now this priority job snapshot table is a combination of multiple tables. Specifically, we're going to be using that job postings fact table and company dim table from our original data warehouse. And this is an engineer own table. We're also going to be using priority roles. Remember, priority roles has what the business designates as a priority and a level of one to three. We're going to combine these tables into one called the priority
8:09:24
job snapshot that has key information from our data warehouse, along with priority jobs and their priority level. Now the issue or what we're going to be demonstrating in this is that the business own priority rules tables is going to be updated by the business. You know, it could be updated as frequently as daily. And so there's going to be updates to this. And that's going to affect our downstream table of our priority jobs snapshot. So if they
8:09:52
do things like change a priority level of something like software engineer or add a new job like data scientist, we'll need to be able to update update that in that priority jobs snapshot table. So what's going to be the data pipelines for these tables that can get changed or updated? Well, for priority roles, we're just going to assume that every single time the business updated, they're updating the entire table. All it wants is a pretty small table. So creating a
8:10:21
new table isn't a big deal. Because of this, we're going to assume that the business just has a simple script for creating and replacing a table and inserting into it the values they want to change those values. They're going to just adjust that simple SQL script. Pretty simple for them to handle on their own and do I would assume a business has somebody trained enough in SQL to do this. Now the priority job snapshot is a little bit more complex. So this is
8:10:46
going to be engineering out. The first one I do is create a script for an initial load. So it's going to create our table for that snapshot table. And then therefore load all that data into it that needs to be there. There's only going to be one run once. Now on a daily basis, we'll then need to go through and perform a batch load. And for our first version of this, we're going to be using what we previously learned of an update insert and a delete statement in
8:11:17
order to maintain this final snapshots table up to date. We're then going to update this batch loading into our version two that relies on using that merge keyword that will combine all of those different things we've done with update insert and delete into one statement alone. And this will do exactly the same thing as our version one. So the first thing we're going to do is create the SQL script that is going to be used to automatically for an initial
8:11:49
initial load and update of the priority roles table. I create a new folder of 1.24. I'm going to go ahead and add a script inside of here and call it priority roles dot SQL. I'm just going to go ahead and create this from scratch. You could go back to lesson 1.22 if you want to. Anyway, we start with create table and we specify it's in the staging schema of priority roles for the table name. Then inside parentheses, I specify the role ID, the role name and
8:12:16
the priority level. Role ID is an integer, role names of our car and priority level is an integer. We'll also establish the role ID is the primary key. Now that we have this table created, we need to insert into it in that staging schema of priority roles, those three columns of role ID, role name and priority level. We're going to insert into it the values and specifically that of one of that engineer level one, two of senior data engineer of one and three of software
8:12:43
engineer of level three. Now this is the script we're going to be running. So at the end of this, I'm going to put a select statement in here to just see what's inside of here anytime we run this script. Okay, I'm gonna go ahead and save it by pressing command S and I'm going to log into DuckDB, specifically connecting to mother duck and then going into our jobs mark directly. So I want to run this script directly. I don't want to have to select it all and then press shift
8:13:07
enter. So we're going to be using for this is the dot command of read. And then we need to specify this file. So what we can do is we can right click priority roles. And we can do copy relative path. And then I'll go ahead and paste that in here. And it says it's under the lessons 1.24 priority roles sequel. Yep, seems right. Let's run it. Now we've gotten air with it. Let's inspect it. And there's a catalog error table with name priority roles already exists. If we go back and
8:13:35
inspect our script, right now we didn't design it to be unimportant. Instead, we need to add into here create or replace table. So it knows to do that. Okay, any time I update my script, I have to press command S and then opening up the terminal, I can just press up and it runs that last command. That ran pretty quick. No errors. That fixed it. Our priority roles table is up to date. If we need to make any changes to it, I would expect the business unit, which they
8:14:04
have control this script will say they can just come in here and they could say, hey, update software engineer to one, save it. And then inside of terminal, they can rerun it by running it and in that we can see that software engineer is one. Once again, this is such a small script and small table that I'm not going to do some sort of batch processing to update the table. It's just simpler for me to create or replace it from scratch. Let's get into now the initial
8:14:36
load of getting that table built for our priority job snapshot table. For this, we need to do something very simpler and simpler, simple as in we need to first create our table. And then from there, insert into it the values we want. As a reminder, this takes use of or is built using three other tables, job postings, fact company DIMM and that priority roles joined in a way that provides us that priority job snapshot. And that priority job snapshot table only includes roles
8:15:09
listed in that priority roles table. So under 1.24, I'm going to create a new table and call it priority jobs snapshot initial and then laymen at a SQL file. So we'll start out with this of a create table statement. So it's buying the main priority jobs snapshot. Now I would don't expect to run this again. So you don't necessarily need to conclude or replace but for a good measure, I'm just going to include it. Inside of here, I'm going to include seven different columns. We're
8:15:37
going to use the job ID, job title short, job post-to-date, and salary or average. That's from the job postings fact table. The company name, which is from our company DIMM table, and then the priority level, which is from our priority roles table, and finally an updated at column that has a timestamp of when the row was last updated. We're going to be able to control this updated at timestamp as we go along and build out that update script. Anyway, I want to keep it as simple as
8:16:08
possible with the amount of columns. You can include more from the job postings fact, but I want to keep it simple because it's going to make our other columns a little bit or it's going to make our other keywords even more robust. Next, we're going to insert in to that main that priority job snapshot. We're going to specify that job ID, job title short, company name, job post-to-date, salary or average, priority level, and updated at. Now we're going to build this using a
8:16:30
query. So I'm going to begin with a select statement and then we're going to go ahead and add in from what we know we're going to give an alias of the job post-to-fact table for job ID and job title short. Oops, looks like a duplicate. Next, from the company dim table, which were alias as CD, we need to get the name or what we're going to alias as as company name. We'll then get that job post-to-date and salary or average from that job postings fact table. And then
8:16:54
finally, from our priority rules table, we're going to get the priority level, which we're going to give the alias for that table of just R. And we have one more attribute we need to apply a pry for this and this is the updated at. For this, we're going to use a formula to fill this in specifically of current timestamp. Now if you haven't seen this before, it's actually pretty simple. Inside my terminal, I'm just going to go ahead and type this in. Now I'll run select current timestamp,
8:17:21
put a semicolon at the end and bam, it provides me the current timestamp. So we're going to mark all the rows with the current timestamp and then if we update insert or delete it, that timestamp will get updated appropriately. Now where are we getting this from? Well, we're getting it from the data jobs data warehouse where the job postings fact table is, we're going to alias ads jpf. And we're going to left join from the data jobs data warehouse, that company
8:17:48
dim table as CD. And we're going to do this on the company ID of both tables. Now we have one more we got to bring in right when you get that priority level from that priority rules table. Now previously, we just did a left join with job postings facts and company dim because we want to preserve all those job postings that are in that job postings fact table or that company or that table A. However, for our priority roles, we only want to keep those roles
8:18:15
that are in the priority roles table. So in this case, we want to do an inner join. So I specify inner join and we're connected to our jobs, smart database. So I don't need to specify the table. So I specify the schema of staging that priority roles. Just a reminder, we didn't have to put the schema in these up here because they're in the main schema. And main is the default one. So you don't have to specify it. But here, we're using a separate schema, we're not the main
8:18:40
schemas is why I have to specify staging. Okay, we're going to give it the alias of just R. And we're doing this on the job title short of the job postings fact table. And that of role name, going back to that priority roles table, right, we don't have a like a job ID with this. So we're not going to link on the job ID. The most common attribute is the role name itself. All right, so with this, I'll put a semicolon at the end. Let's go ahead and save this. Oh, when there is
8:19:06
one more thing anytime I run this initial load, I do want to see what is on what is the final table results. So what I'm going to do is I'm going to query our final table. And I want to aggregate it by that job title short column. So that way we can see to make sure did they use have an update for what I would expect to have updated. One thing I want to see is did the count get updated? Did the numbers go up or did it go down? And then the other thing I want to check is the
8:19:36
priority level and the updated at time for this, right? Anytime we're doing some sort of aggregation, we need to wrap it. So we're going to wrap priority level and also updated at in a min function. And I would expect them. Well, they are going to be all the same. So it's just going to provide a single value back using that min function. This is going to be from the priority job snapshot table, which we created right above. And we're going to group by the job title short
8:20:06
column as expected by doing all this aggregation. And then finally, I want to put the most important up at top. So I'll put the order by job count and descending order. All right, there's looking good. Let's save this now. Same thing. I'm going to come over to that table. I'm going to right click it and I'm going to copy the relative path. And then inside my terminal, I'm going to do dot read command and then paste in the location of that script. Let's run it.
8:20:29
Hopefully don't have any errors. And I got an error because I had a typo. It says, Hey, table name job postings facts does not exist. Did you mean job postings fact? I did. I'm going to go ahead and update it up here, save it by pressing command S going back in the terminal, pressing up, and then running it. Alright, that took a total of three seconds. And I can see that the final table that we have has looks like almost 500,000 jobs in it of data engineer software and
8:20:56
engineer and senior data engineer is what I expect. And the prior level of one, and it was updated. Well, as of looking at this, well, just a few seconds ago. So that is our initial load script. We're only running that for the initial load. So we're not going to be using anymore. We're going to be transition to the batch load process now. And I'll be using the update insert and delete statements. Because of that inside of here, just to keep things simple, I'm going to close out of
8:21:22
the priority job snapshot initial script, because we technically shouldn't have to run this again. I am going to keep that priority rules open because we'll be updating this to make sure that our batch processing actually is working correctly. So let's get into building that data pipeline for updating our snapshots final table in this, you know, a lot of Venn diagrams. And so with this Venn diagram, what we have is we have a source table, which is
8:21:54
represented by the blue color, and then our target table. And what I want to go through this is what we're going to be doing for different sets of data that may be in the source and on the target or in the target and on the source. First up, let's talk about those right here in the center where our source and our target table overlapped. And that's where we have matched rows. If they're matched rows, they're either likely OK, and we don't need to update them. Or they
8:22:24
may be happy change within the row. And in that case, we need to update them. Next up are those that are unmatched in the incoming rows of that source table. In that case, if they're in the source table and not our target table, we need to insert them in and conversely, if they are unmatched existing rows in our target table, but are no longer in the source, we need to delete them. So we're going to be walking through how to do this update, insert and delete with our
8:22:55
tables. Specifically, we already know that our target table is going to be that priority jobs snapshot, our source table, which we've shown previously in the initial build, if you will, that that is just right now. It's only a SQL query for what we can get. So what we're going to do is we're going to build a temporary table within our script to create this source table. And then as necessary, load it or updated insert or delete it into our target table. So
8:23:25
inside of 1.24, I'm going to go ahead and create a new file. And for simplicity, we already have the other one called the prior job snapshot. We're just going to call this one priority jobs snapshot, that's equal. Now inside of this script to start with for this data pipeline, we're going to create a temporary table that has our query that creates our source table. And the reason why we're creating this temporary source table is because we needed to then reference it
8:23:51
in our other statements below specifically, we have to reference it in our update insert and delete. So let's get into creating this source table. First thing we're doing is use create or replace temp table or temporary. We're going to give it the alias of source priority jobs. And I have that backwards. The as needs to go at the end of that, because we're assigned that name as, and we now need to use that query that we previously did. Now I said we could close out the
8:24:20
initial, I'm gonna go ahead and open it up. And in here, we're going to reuse this insert into query right here with the select statement. So I'm going to go ahead to paste that into here. Now I want to test this as a goal along. So let's just run this because it's going to do no harm just to run this and create this temporary table. So I'll right click that priority job snapshot in order to get that copy that relative path. And then inside of our terminal, I use a dot
8:24:46
read command pasting in that snapshot SQL file and ran with no issues. Now the first thing we're going to write a query for is an update statement to capture those match shows that may have chains that are different between the source and target table. Now as a quick refresher on update, this is used to in situations where maybe a value within a column changes and we need to update it can change with our single column or multiple columns. In our case, what we're
8:25:18
trying to update for or the only situation we try to update for is if that priority level changes for a role, not if a new role gets added to that priority role is gone. But if I priority level changes, such as software engineer changes from like three to one, the update syntax is pretty simple. We're going to use update specify the table, you set to set what column we want to the new value. And then we use a where condition where condition we're going to
8:25:44
be using existence filtering for this. So we'll start with that keyword of update. And we're going to be using the main that priority jobs snapshot as our target table, we're going to give it the alias, keep it simple in our mind in the right spot of target. In this, we want to set from the target, the priority level equal to our sources priority level, which our sources appear is that temporary table that we previously established. So we'll define it in our from statement,
8:26:17
that source priority jobs as source as the alias of source as RC. Okay, we don't only want to just update the priority level. Remember, we also created that column that has the current timestamp at it. Specifically, it's noted as the updated at, and I'm noticing this temporary table doesn't have that even specified at so I'll do updated at. So here I'll add in a line now to have the updated at equal to the sources updated at which the script's going to run all at
8:26:44
one. So it's fine if we want to do it like this, I could also replace this source that updated at with just that current timestamp formula. So now we need to include a where statement to how we're going to match up this priority level or know what to update it on. Specifically, we need to do it to where when the target job ID is equal to that sources job ID. Now currently, as this is written, this will go through and update every single priority level in the target target
8:27:12
table, whether it updates are not. So we need to include an end condition to also check if it actually changed. Specifically, we want to check if that target priority level is distinct from our source priority level. And this is distinct from is a new type of way we've seen distinct before and using distinct for columns. This is distinct from is a new way to check line by line or row by row, if it actually changed. So let's go ahead and test this now to
8:27:49
make sure that our update statement is correct. Remember, previously, we ran this priority roles already, and we saved software engineer as a one. So I want to see what happens if we were to change that software engineer back to something like a three, I'm gonna go ahead and save this from any time we do anything that's filed, we save it, I'm also save this snapshots file. So I'm gonna press up to get back to that one dot read statement that looks at that
8:28:13
priority roles sequel file, because we updated now software engineer to three I'm gonna go ahead and run this. Okay, it's updated to three. Now I want to run the priority jobs snapshot. So I'm gonna press up twice because we just rain it. And nothing is shown because we didn't put a sequel query at the end. If you remember from our initial build of priority job snapshot, we include the sequels query at the end. I'm gonna go ahead and just copy this. And then
8:28:39
underneath this update statement all the way at the very end, I'm gonna go ahead and paste this in press command S. All right, let's see if it did update this time I'll press up in terminal and then run this and then it updated software engineer is now three. And what we can also see from this is that it only updated that software engineer because we can check that updated at time stand. And we can see that software engineer is updated an hour later. It was
8:29:07
an hour later because I took a little bit of a lunch prank when between film and this. So we've now handled those match rows or those that update with priority level. Now let's move on. For this, we're gonna be looking at unmatched incoming rows. Basically, did we add or did the business add jobs to that priority roles table? If so, we need to insert those new rows into our target table. So let's start by building this by putting in our insert into
8:29:35
statement, specifying our main target table of that priority job snapshot. In it, we want all seven columns of that replaced or inserted into if it's difficult to insert into. And we're going to insert into it using a query underneath it. And for this, just for consistency, I'm going to go through what they're doing to select star. I want to make sure that the column names do line up properly. So I'm gonna list out every single column name,
8:30:04
job ID, job title, short company name, job post-a-date, salary or average priority level, and updated at. And we're gonna be getting this from that source priority jobs, the temporary table. We're gonna alias this as source. Now there's a reason why we assigned it and alias, and that comes to the fact that, well, we don't wanna insert every single value from the source table into our target table. Remember, we only wanna include those unmatched
8:30:31
incoming roles that aren't in our target table. So this one, we're actually gonna do existence filtering. I think I said on the last one, we're gonna do existence filtering. This is the one we're actually gonna do existence filtering for. Specifically, if you recall back, we wanna do where not exist to find those values in the source tables and insert them into target. In our case, or in case of the where not exists, keep rows with no match in the target.
8:30:59
So we'll do a where not exist, open parentheses, then start with that select one. We're gonna be coming from our main priority job snapshot, which is our target table. And that's where our target job ID is equal to our source job ID. All right, so that's this query. I'm gonna go ahead and now save this. And now we need to test it. So we're gonna go back to priority roles, and we're gonna insert a new line here, and we're gonna add in a number four of data scientists
8:31:31
with the level of three. I'm gonna go ahead and save this as well. So let's first run this priority roles script. So I pressed up with the arrow twice. I have a syntax error because I forgot to put a comma after this. Make sure you have the correct syntax. Let's try again. All right, so it's updated to have data scientists in there. Now let's run the priority job snapshot, specifically testing our insert statement to see if data scientist gets inserted in.
8:31:57
Press up price to get to the priority job snapshot, and then run this, and it updated. We see that data scientist is added to the list now. It previously, if I scroll back up here, we can see that we previously only had three values in it. Well, now we have all four. We have the data scientist three, and we can see that updated at time was updated for it. About 16 minutes last after we did that insert into. I mean, 16 minutes after we just did the insert into after that last update.
8:32:29
So now we only have one last statement to create, and that is to delete unmatched existing rows in our target table. Basically, if we have jobs that were removed from our priority roles table and effects that snapshot table. For this, we're gonna use the basic syntax of a delete from statement, and then specify a wear condition. Once again, we're gonna do existence filtering for this as well to get the correct rows we need to delete. So let's get into building this delete statement.
8:33:02
The first thing we're gonna do is use delete from specifying the main dot priority job snapshot as the target table, and then we're gonna use wear not exist. Now, previously for this wear not exist, the main table is source, but if you've noticed, we're actually using in our query, we're using target. So that source and target are flipped here in the diagram, and we can see that delete from the target table is the main table here, whereas up above here for our insert into, right,
8:33:34
our main table is the source, and our target is that other one that we're trying to filter. Anyway, I want to clarify that on why we can still write this wear not exist, because we've sort of flip flopped it. Anyway, we're going to then put the query or a sub query inside of here, a select one, from our source priority jobs table as source, and that is where our source job ID is equal to our target job ID. All right, so now let's test this. I'm gonna go ahead and save this file.
8:34:02
Inside of our priority roles, I'm going to go ahead and just delete out this data scientist and save this. Now let's run our priority roles by pressing up twice. Okay, we only have three roles in here. Then I'll press up twice again to get that priority jobs snapshot, and we're now going to run this, and bam! We can see that previously. We had data scientist in this table, but now in this new one, we have data scientist removed. So this priority job snapshot SQL file
8:34:35
is a full data pipeline in order to perform batch processing. And we do this by loading in a temporary table, performing an update, then an insert, and a delete where necessary, and then having that final check query at the end to double check our results. So technically, we could stop right here. We have our initial load script that we built for our snapshots table, and then our batch loading v1. But I want to now go further.
8:35:04
Specifically, I wanna combine these three different update, insert, and delete statements into one single merge statement.
8:35:15
So let's quickly go over the syntax for merge, and this starts by using merge into, specifying the target, and then what you're using as the source. And then from there, you specify what you're doing it online, similar to like a join, in this case a key column. In our case, we'll be using that job ID. The first thing in the merge into what we're doing is an update. For those matched rows, we want to update them, and we'll set them appropriate to their columns
8:35:47
based on what's in the target, to what's in the source. Next up, we'll do those rows or unmatched incoming rows in the source table that we need to put in the target, so we need to insert into. We'll use the keyword in our syntax of when not matched, then, and then from there, our insert syntax to insert it in. And then finally, we need to account for that last condition, those unmatched existing rows in our target table. And this is a single line we can just put in there
8:36:13
of when not matched by source, then delete. So let's go ahead and do this. For this, I'm gonna go ahead and select all the way here from the update down to the delete, and I'm gonna press command slash to then comment, then, all out. And then underneath this all, we'll put our merge into to where we'll be running this now, this new query that we're building out for our priority job snapshot SQL file. First thing we're gonna do is build out that merge into using
8:36:41
and on, so we'll type merge into our main that priority job snapshot table. And we'll specify this as our target table. Next, we're gonna be using our source jobs, or source priority jobs table as our source. And we're doing this on the target job ID equal to the source job ID. Now we need to include our update statement using that when matched then to include those match rows where the priority level may have changed. So I'm gonna start with a when match then,
8:37:12
we wanna update and set the priority level of our target table equal to the priority level of our source table. Similarly, we also wanna update that updated at for the updated at time in our source table. Now, we did up above, we scroll up here, whenever we did that previous updated at, we added this condition in here. And that was when the target priority level is distinct from the source priority level, basically only updating it if the priority level changed.
8:37:44
And that's also what we wanna do here. So I'm gonna go ahead and just copy this. And then inside of here, we can go when matched, well, it should be when matched, when matched. And this condition of the target priority level is distinct from the source priority level. Then we wanna update the set. So it's not only matched, but it also, it needs to meet this condition to be also distinct match being the job IDs are equal. The next condition we now need to get to
8:38:16
is doing our insert into specifically unmatched incoming rows in our source table that we need to insert into our target table. This starts with the keyword when not matched then and then with our insert statement. So in our merge statement, I'm gonna do when not matched, then we want to insert into, and I'm gonna come up here. I'm gonna cheat a little bit. I'm gonna copy this portion of this insert into. We're gonna modify it slightly.
8:38:44
I'm gonna select all the way from the insert into all the way to right before the from. And I'm gonna paste this into here. I'm gonna do command slash to remove the comments and then tab it over. Now, what we wanna remove is we don't wanna say insert into 'cause we don't need to specify the snapshots table again because we're already doing this in this merge statement right above here. So we do want insert with those columns. And then from there, we wanna specify not using a select,
8:39:11
but inserting into using values. And we put an opening and then a closing parentheses around this all. Specifying we're inserting into that target table, these values from the source. All right, the last thing we now need to match up on is the target table's unmatched existing rows. Basically, we need to delete from it. And we only need to include this keyword phrase right here of when not matched by source, then delete. So include this of now when not matched by source, then delete.
8:39:43
So now let's go ahead and test if this works. Remember this updated or this update insert and delete that we created before is coming it out. So we're actually testing the merge into. I'm gonna go ahead and save this. Now the first thing we wanna check or test is a change in this. So I'm gonna change this data engineer to a level or a priority level of two and go ahead and save this. And then I'm gonna first start by running the priority roles to make sure that it got updated to a two.
8:40:11
Yep, data engineers too. Now I'm gonna run our priority job snapshot. There's the first time, hopefully there's no errors. All right, so I've been sitting here for a second. This query didn't execute. That's because I got an error. I quickly or went through this. I went through this and saw that when I did an on and I had target dot job ID up at the top of that merge into the very beginning. It should say equal to source dot job ID. I thought I typed that, but I guess I didn't.
8:40:36
Anyway, if you didn't do it, yeah, I need to fix that. Anyway, this query's stuck up because it can't do that just yet because the query's wrong. So a good way to exit out of this. I'm gonna just go and press control C and I'm going to interrupt. Now we're gonna try this again. This priority job snapshot. I'm gonna go ahead and save it, command S, now for those new changes and we're gonna try again. And bam, I did it. So now we have this data engineer
8:41:01
and it updated it for that engineer of priority level two. And we can see that's the most recent time stand of 1842, so an hour after our last edition. All right, the next thing to test for is let's test adding a line. I'll test in adding data scientists with a level of three. I'll run first our priority role script to make sure data scientist is added, it is. And then from there, test the priority job snapshot script to see if it will update or insert those data scientist things.
8:41:34
And it does, we now have data scientists added to this and we can see that it's done a minute after our data engineer. All right, last one to test for is a delete. What happens if we delete data scientists from this? Go ahead and save that priority roles. Run our priority roles again, okay, I see data scientists removed. And then from there, run our priority job snapshot. And bam, data scientists is removed. So this merge into statement captures it all
8:42:01
compared to what we were previously using with our update insert and delete, along with some existence filtering. I feel this merge into is slightly easier to read and simpler to use. But as we saw, it took a little bit of an understanding of why this is necessary. So that's why we walked through you building that update, insert and delete to start with. Now as a quick reminder, right, we went through this example to demonstrate with both the update, insert, delete
8:42:34
and then also the merge to demonstrate how to go through and perform batch processing. And typically something like we're doing would probably run on a daily basis, maybe even as quickly as hourly. Now you may be wondering this, or at least I did after building this, why wouldn't we, instead of building out that complex merge into that we did, why wouldn't we use something like a CTA or create table as and just create that priority job snapshot table
8:43:05
from our original SQL or original SQL query where we queried all those different tables? And that would be a very viable option, especially in our case with how small our tables are. But the problem runs into when you start getting into bigger and bigger tables. Anyway, throw together this table that compares some different scenarios on when you would wanna use a CTA over something like a merge. This is very general and don't take it as hard rules,
8:43:38
but if it is a small table that you're building less than 10 million rows, a CTA would be perfectly fine. And this would accomplish that batch loading that we would need to do by just rebuilding the entire table instead. You may also want to do a CTA when it's a very large table and more than 50% of the table needs to change. Basically, you maybe have some deletes or inserts and it's gonna be a lot of rows, then you may need to build it from scratch. But I would say in most situations
8:44:08
where you have a very large dataset, greater than 10 million rows, and you're just doing incremental changes, so less than like 10% or so. That's when a merge would be possible or when you'd wanna use a merge. Other times you'd wanna do it these last three where you need an atomicity, some live traffic, or even incremental or streaming data. These are a little bit more nuanced than I wouldn't necessarily think that you need to commit those to memory though.
8:44:35
All right, it's your turn now to go through and we have some practice problems for you to now get more familiar with this merge into. In the next lesson, we're gonna be going deep as you saw we did some intro into data modeling, but we're gonna be continuing this with data modeling in our part two lesson. With that, I'll see you in the next one. (upbeat music) Welcome to this second lesson on data modeling. We covered the first part before project one
8:45:05
and this one we're gonna build on it further. Now, don't worry if you don't remember much from that part one. I'm gonna do a quick refresher at the beginning of this, but the majority of this lesson is gonna be focusing on understanding design behaviors and building different databases, specifically data warehouses. We're gonna go through seven core design patterns with the key one being a star schemas that's most common seen in data warehouses.
8:45:33
Now, we're not gonna just jump straight into design patterns because we first need to understand where are we building this design pattern or are we building it in a source system or an analytical system. Previously, we've been working a lot in analytical systems, specifically data warehouses, but we have to be able to get this data from somewhere, a source system and put them in an analytical system. Don't worry, we'll cover more on that when we get there.
8:46:00
Also, because we're gonna be covering the source and analytical systems that will support the different design patterns, I'd be remiss if I didn't cover the different types of databases that you can use for these source and analytical systems. Anyway, I'm getting ahead of myself. Let's actually get into this with a quick refresh of what we covered in the last lesson. Previously, we went over the database hierarchy where the database is the overall container
8:46:28
that keeps all of our data inside of it, specifically we have different schemas, and then under that, we have the sub-level of tables. We then transitioned into understanding how our data warehouse, specifically that data jobs, data warehouse, was modeled. And this modeled after a star schema using fact tables and associated dim tables. And with that, we learned the importance of inside of tables, the importance of things like primary and former foreign keys
8:46:58
in order to relate to different tables. And then the relationships that you could have to between tables, specifically the most common of either a one-to-one relationship or a one-to-many relationship. And then finally, we wrapped it all up in that part one by using actual SQL to query that information schema to look at things like tables, columns, views, and table constraints. Now in that lesson, that lesson had some actual SQL exercises that this lesson
8:47:28
is gonna be mostly all theoretical. And by mostly I mean entirely. Now why the heck does data modeling even matter? Well, in order to understand that, we probably should look first at the problem. And here's the problem I had whenever I was first trying to collect these job postings in order to make a data warehouse that had all these postings throughout the internet. So the first website that I targeted was LinkedIn. Now LinkedIn probably uses some database,
8:48:03
we'll say in this case, they use SQL Server. And they probably have some sort of design pattern to keep all of their job postings organized in the data model that they like for their own data team. And with that, they have their own schema, so therefore they have their own table designs as well. Now let's say I did get access to that LinkedIn SQL Server database and I was able to get that in. That's great. But what happens when I go to the next company?
8:48:30
For example, let's say I go to Indeed Next, they may use a different database, such as MySQL, they may use a completely different schema design as well, along with table design. Oh, and then I go to say even one more thing, like Upwork, and they're using Postgres with a different schema and table design as well. Well, how the heck, if I have this data from these three different sources, how as a data analyst, could I go in and actually query this data? It would be a mess.
8:48:59
So data modeling helps with this by bringing in structure and consistency. If we design our schema to have a known design pattern that our analysts are gonna be using, they can go into it and access the data a lot quicker. Additionally, we could use a table design as well with certain column attributes that makes it a lot easier to analyze and aggregate. Well, for the solution of what we could do for this, is we could combine these three different databases
8:49:32
of SQL Server MySQL and Postgres from LinkedIn, Upwork, and Indeed, and combine this data into a singular location, specifically a data warehouse. And in our case, we're using DuckDB in order to drill in and analyze this data from all the different sources. This is why data modeling is so important. As all sources are standardized into one analytical model, an analyst can query data from a single consistent place. So this was my use case. What would be a real world use case of this?
8:50:10
Well, typically as a data engineer, what we're gonna be doing is we're gonna be taking data from some sort of source system. In my case, that was the example that was given with LinkedIn, Indeed, and Upwork. Then from there, perform some sort of ELT, like an extract, load, and transform. These are just example scripts right here. And then from there, loading it into analytical systems. In our case, our data warehouse of that job postings fact table and the other dimensional tables.
8:50:43
Now, analytical systems, we've seen data warehouses, and we've also seen data marks that we just built, and that last lesson. Another common one is data lake houses, which is basically a combination of a data lake and a data warehouse. That's gonna be on the scope of this right now, understanding it's more of an advanced concept, but it is an analytical system where we wanna store that in. Anyway, so what are some common source systems? Now, that would be things like ERP systems
8:51:13
or enterprise resource planning, CRM, or customer resource management systems, app backends, or normalized tables. Now, let's quickly break each one of these down because I don't think you may be familiar with them. Enterprise resource planning is a system that is used by an entire company, and it's used to manage, well, everything within the company. You can use it to manage the supply chain, your shipping, your manufacturing, HR can work in it, finance, everybody can work in it.
8:51:43
Some common enterprise resource planning solutions that are out there right now are things like Oracle, Odo, or even SAP HANA. Next up is CRM or Customer Relationship Management. This is a solution that allows businesses to, well, talk to other businesses or customers. It's basically the front office interaction. This is very common for teams like sales, marketing, or support. Popular CRM solutions would be things like Salesforce, HubSpot, or even Zoho.
8:52:17
Now, there's two other types of common source systems and they'd be more custom built. Imagine if you have some sort of app, you could have the app backend with its own database that would be the source or system of truth. Additionally, you may have a robust data team that builds out your own internal ERP or CRM systems, and they have their own normalized tables. We'll go over more into normalized tables in a bit, so just hold on to that for a second.
8:52:45
Now, all of these solutions have to store their data in some sort of database. For example, Oracle's NetSuite is gonna be using an Oracle database. Odo uses something like Postgres. An SAP HANA uses, well, their custom solution of SAP HANA. Similarly, for our CRM solutions, something like Salesforce uses also an Oracle database, HubSpot uses Postgres, and Zoho uses MySQL. Anyway, there's actually a commonality between all of these different databases.
8:53:17
And I don't understand that we need to understand that databases are made with a certain design pattern in mind. Specifically, there's two major types of relational databases that we can have. One is OLTP, which is online transaction processing, and the other is OLAP or OLAP, online analytical processing. And I might say OLAP. All of these are spoken out via their abbreviation. Anyway, OLTP are for transactions, and they operate apps and capture live transactions.
8:53:59
This is optimized for high-velocity rights. So it's very common, like if you were in an ERP or CRM system, where we're getting lots of data updated all the time, it's constantly having to write to the database. Now, the structure of this is more normalized, and once again, we'll get into a normalized data schema in a little bit. The other type is that OLAP are online analytical processing, and it's our analytical type databases. This is optimized for large scans,
8:54:29
aggregations, filters, basically any type of data analysis. And this is typically stored in a structure that we saw previously of fact and dimension tables, or let's star schema. So let's look at some common OLTP and OLAP, or transactional and analytical databases. We're gonna start with the transactional first. Some popular options include things like SQ Lite, Postgres MySQL, also other ones like Microsoft SQL Server, Oracle's own database, or MariaDB.
8:55:00
If you recall, when we looked at the databases of ERP systems, all of those, well, we didn't look at SAP HANA, but that Oracle and Postgres were transactional processing databases. Additionally, for those CRM solutions, they were also using transactional processing databases, and that's what's typically used in any type of source system. Now, moving on to those OLAP or analytical databases. We've already been working with one, which is DuckDB, but they also have other popular options
8:55:31
like Google's BigQuery or Amazon's Redshift, and they also have popular options that you've probably heard of, Databricks, Snowflake, and even something like Clickhouse. So for our pipeline of what you would typically see of a data analyst from those source systems to analytical systems, you're typically gonna see those source systems are typically OLTP and analytical systems or typically OLAP. These aren't hard and fast rules that you have to use these type of databases,
8:55:59
like you could see a Postgres database as an analytical system, but it's just not common. Also, it's just not designed for it, although Postgres is getting a lot faster. (gentle music) All right, so we're now gonna shift into going into the seven core design patterns, and the main one that you're gonna take from this that I want you to take from is the star schema that we're going over first. From there, we're gonna see that a lot of these
8:56:24
different design patterns are based off of the star schema or a modification of the star schema. Our first up is star schema in that it has one central fact table, and it's surrounded by dimension tables. The fact table is a central table that stores measurable quantitative data about business fence or transactions. The dimension tables are supporting tables that provide descriptive context about the facts. Facts answer how many or how much
8:56:55
in the dimensions answer who, what, when, and where. For example, an e-commerce company may have a sales fact table, and then as dimensions around it, it has things like the customer, product, store, and date dimensional tables. Our job posting table, or our job posting at the warehouse, really has two dimensions in that. We have a job postings fact table, and then a company dam and a skills dimension table. Now, this star schema is the most popular design pattern
8:57:27
used within analytical systems, and because we typically use OLP databases for this, you're gonna see star schemas in those OLP databases. So what is the most common design pattern in source systems, which are typically OLTP? Well, that's where we get into a normalized schema, and this is a database design, where data is split into multiple related tables. The goal here is to reduce duplication and to maintain data integrity by only installing the instance exactly once.
8:58:05
In the real world in ERP systems, I've seen this where it's hundreds of different tables linked together. It's a nightmare as a data analyst trying to go through it, trying to find the correct table. But it's great for maintaining transactions and actually storing all of our results. So when we compare this normalized schema, that's usually a source system to something like a star schema that would be in a data warehouse or mart, this star schema is a lot less intimidating
8:58:32
as we have the data in a known form, the star schema that an analyst would expect for them to go to and analyze and extract data from. By having to go into a normal schema and try to figure out where in the hundreds of tables something may be. So in a real world scenario, we would have to develop either some sort of ELT or ETL pipeline to get maybe some order data into and created into a star schema within a sales fact table, along with its dimensional tables.
8:59:03
And building each one of the tables would be as simple as for the customers, just moving the customers table over, but then maybe for the sales table, it would be extracting out and joining different tables such as the orders, payments and order items. So this normalized schema to a star schema would be exactly what we would be trying to do in this analogy here that I have with a source to analysis of pulling something from a source system and getting it into an analytical system.
8:59:30
All right, next up is a constellation schema, also formerly known as a galaxy schema. Constellations more frequently used nowadays. Anyway, if we can see from this, this is a rendition of a star schema in that it's a collection of multiple different stars. As you can see, you could have multiple fact tables connected by multiple different dimension tables. With how it's arranged, you can see how it sort of looks like a constellation. This would be used in very large companies.
9:00:02
like enterprises that have maybe different teams such as a sales team and orders team a marketing team or whatnot and they would have their associated tables and be easy for them each to get into the data they need. So this is a typical schema you would see in a real enterprise data warehouse. Next up is a snowflake schema and this one is also a rendition of a star schema in that you have your fact table surrounded by a dimensional table and your dimensional
9:00:33
tables could have an additional things that's fun off of it to have like sub dimension tables. This is more typically used in legacy data warehouses you're not finding this used as often but you could find yourself on a data team that has this implemented still. A simple example of this would be say we had a sales fact table and then from there we had our product dimensional table and each product may go into a category and then each category may go into a
9:01:04
subcategory. So this is going into a snowflake type schema. You could also see it broken down to something like this we're not going to go through every attribute but maybe in the upper left-hand corner you could see sales broken down by customers and then customers broken down further by their customer type and maybe region. Next up is a factless fact table which once again is a rendition of a star schema but in this paid close attention to this fact table
9:01:31
in it it has well it's factless. The only thing inside of here are IDs to associate it to its associated dimension table. So no numeric measures are recorded in the fact table rendering it a factless fact table. Our job postings fact table does not fall into this for our data warehouse we actually have numerical attributes in there like salary whether work from home and what not. However we do have this design pattern of a bridge table. A bridge
9:02:03
table is a table that resolves a many many relationship between two other tables by acting as an intermediary. So between our job postings fact table and our skills dim table we have well what's actually named as skill jobs dim but it would be more appropriate to name it skill jobs bridge because technically it's a bridge table. That skills job bridge table includes those foreign keys of the job ID from the job postings fact table and the foreign key of that
9:02:33
skill ID from the skills dim table. If that table didn't exist we'd have to shift what not only the skill ID into the job postings fact table but also the job ID into the skills dim table and then we'd have to join them using the job ID and skill ID and that's going to be creating a many to many relationship. That'd be awful. Bridge tables helps all that. At the last design pattern that we're going to cover is a flat or known as a wide table. In this we would take something or
9:03:02
data that's in a data warehouse in a star schema perform some sort of ETL or ELT and get it into a single table with all of our column and attributes in it in one. Flat tables are sort of more used to it as an analyst and that's historically used in reporting in Excel error BI tools. You're probably more commonly here in the data engineering world referred to as wide tables and this wide name emphasizes that all the relevant attributes are joined into one
9:03:32
wide table with many columns. I would expect to see a flat or wide table in a scenario where maybe you have a more junior analytical team and they're not used to using a star schema database or data warehouse to query out their data. Now there's one last thing that I want to cover and it revolves around maintaining and updating data models and that comes through a very well-known practice of slowly changing dimensions or SED. Now if you think about it if you make a
9:04:06
data warehouse and it maintains for over 10 years without updating your probably doing something wrong. SED is a method used to track and update changes in a star schema or data warehouse. So like in the case of where I said hey skills job dim probably needs to be updated to skills job bridge I could use this type of method to update this table so that way I have it tracked properly and change properly. Now slowly changing dimensions is only one
9:04:39
method to actually track changes. There are a few others but SEDs are by far the most common to track changes and typically you could see anywhere from zero from type zero to type six in no way we're gonna go through a few of these in no way do you need to memorize them at all they're mainly just there for you to understand what the purpose of this is. Now for implementing these you could use any one of these types or in the case of something like type five it's
9:05:09
a hybrid of type one and two or in the case of type six is a hybrid of type one two three but what the heck are these types let's look at some examples. Well let's say in our company dim table we needed to update a record and we had a company called Facebook but it got renamed to something like meta. Oh and also the link for it changes well well with type zero it's fixed so you can never change the value and it's immutable has to stay what it is. That's actually
9:05:38
an SED kind of strange. The next type or type one is an overwrite up above we have dim company before the update where it's Facebook and we have Facebook's link and then after the update you would find that it got updated to the name of meta and then the link. Now one thing to point out in this is you would typically find in this type of environment that you would have a company sk or surrogate key and the company ID the surrogate key would be technically the primary key and it
9:06:08
would be assigned by the data warehouse and the company ID would be the unique ID inside of our maybe the normalized schema inside of our source system. So a little bit different from the company ID that we have in our table. Next up is type two where we would just add a new row. In that we would create another line where we'd create that new surrogate key for that company and we did add in the information of meta the new link
9:06:34
and when it was valid from or valid to and anything else we need to associate with it. Type three would be just be to add a new attribute or attribute we'd add new columns. Previously we had name and link we'd probably change them to previous name previous link and then add the new names and new link under current name and current link. In this case this only tracks the last change not the full history. The last one we'll cover is type four of a history table.
9:07:02
So our current or new table would be that dim company table with our meta information in it and then we'd probably savor old previous one as something like dim company history. In it we could also not only have that previous name of Facebook but we may store in it as well that history version what our new current name is as well. The key thing here is that main table stays clean and the history moves elsewhere and then just to wrap this up like I mentioned type five is a hybrid of type
9:07:33
one two and type six is a hybrid of one two and three. Anyway the key thing to stay in here is not to memorize these different types but to keep in the back of your mind what would be a way to keep track of design patterns if they need to change. All right so I have some practice problems for you now go through and test your knowledge on all of these different design patterns that you could have within the different types of databases. With that I'll see you in the next one
9:08:02
we'll be getting into case expressions. All right see you there. All right in this lesson we're going to be going over case expressions. These apply conditional logic in SQL transformations in order to categorize clean and standardized data. A really powerful tool for data engineers. So after we go through the basics of the syntax of case expressions we're going to then go in through some multiple quick use cases along with a final example.
9:08:37
So getting straight into the syntax case expressions begin with well case as the expression and it they end with end. Inside of there we can specify multiple conditions that we want to meet or when we meet a condition then we do what that condition is. There's no limit on how many when and then we can then perform underneath this. That's why the when and then the second one is in square brackets. Now additionally we can do a catch all
9:09:06
at the end of an else which is optional because it's in square brackets and it provides an output if none of the when conditions are met. Now today we'll be going through examples using case expressions inside of a select statement but really they can be used anywhere where a value is expected in SQL. So you could use them in wares group buys or even order buys. So let's go through a simple explanation or example to showcase how this case expression would be used
9:09:39
in a scenario like this. In this we're going to be using the salary hour average column and we want to basically say hey if there are any values that are greater than greater than 40 dollars we want to classify it as high salary otherwise classified as low salary. This low salary uses an else condition as a catch all for any values that are not greater than 40 so basically less than 40. Anyway we wrap this all in a case and then also end and as you can see we provided an
9:10:08
alias using as and specified to be salary category. So let's walk through this step by step. We're going to have in our job title short column we have data analysts and we have our first value of 10 for salary hour average. It's going to check whether that 10 is greater than 40 basically that first condition. In our case this is false so we're going to now meet the else condition to classify this as a low salary. Similarly we'd have some like data
9:10:36
scientists maybe that's a 30 as well. The 30 is going to be less than that 40 so it returns a false and therefore once again we meet that else condition of low salary. Finally though we will get a result like dead engineers at 50 conveniently greater than our threshold. In that case the returns it's true and therefore we will meet our condition to be marked as high salary. Now we're going to be walking through using case expressions in
9:11:05
four popular use cases that I've found for data engineers. This first one of bucketing data will build off that example that we just went through using salary data in order to bucket it. We're just going to make it slightly more complex. After this we'll move into our second case of handling null data specifically with that salary data. Then we'll shift gears and we'll perform categorization of values using text analysis. Then finally we'll wrap it up with our
9:11:36
fourth case using conditional aggregation and this is more of an advanced case in order to perform analytical analysis using case expressions. We're going to start with this first one first of categorizing data or more specifically bucketing data. We're going to bucket salaries based on the following bins. This is all for that salary hour average column from our job postings fact table. So let's start writing this query. For this we're going to write a select statement.
9:12:04
I'm going to add in that job title short. Also I want to check as we go through with our case expression so I'm going to list also the salary hour average column. Next we're going to start our case expression. I'm going to write a case and then also I'm going to write an end. Inside of here I'm going to put our when and we want to meet our first condition of salary hour average less than two five. So that's the entire condition right there.
9:12:30
Then we want to categorize this as low. For this make sure we're using single quotes for this. The next condition we want to meet is when salary hour average is well we can do between 25 and 50 but I'm actually not a fan of this. Instead I'm more of a fan of just explicitly saying hey less than 50 because you've already once you go through that first condition of filtering out all those that are less than 25 you could then go to the next one of less than 50
9:12:59
and it's understood that it's between 25 and 50. In this case we're going to classify it with the then statement of medium. Now I could do another when statement but we'll just right now we'll just wrap it up with an else and all others are high. We'll give this an alias as salary category and we're going to be getting this from the job postings fact table. Now one thing right we don't all not all values in our table are filled. There's a lot of null values in here
9:13:28
so I'm going to filter them out by giving a where statement of where salary hour average is not null. Also I don't want to do this to the entire data set so we're just going to put a limit 10 on there and close this out with a semicolon. Okay let's go ahead and run this. For this I'm going to open up turnable terminal and connect to our data jobs database. Selecting our query and throwing it in here BAM we get back these values and I can now go through this
9:13:54
and far off and see that okay it is matching our logic everything's matching as expected. All right the next type of method or that is used for case expressions is in handling missing data specifically those nulls. We're going to build on that previous example and instead of just filtering out null data using a way we're actually going to use this within our case expression to handle this. So I'm going to come back to our previous query
9:14:19
and copy it and then paste it into here. Now let's see how null is handled if I were just to remove this where statement right here and then run this. Basically those nulls get captured by the else statement and I'm pointing this out because this first when of salary average of less than 25 like null is not a value so it's not getting bucketed into that low of less than 25 instead it's going in getting captured by that catch all of the
9:14:48
else high. So I like to be explicit with if I want to capture or handle null values so I'm going to add a statement or a when statement to capture this. Specifically specifying when salary average is null then I want to categorize it as missing. Let's go ahead and run this and now there's null values that are marked as missing. The next thing we can use case expressions for are standardizing categorical values. Let's look at this use case where we want to use the job title call
9:15:21
another job title short column but just the job title column and classify jobs as either data analyst, data engineer, and data scientist. Let's just take a look at these values first specifically looking at the job title and job title short and that's from job postings fact and we'll limit this to well we'll limit it to 20. We'll go ahead and run this. All right looking at this this is not necessarily providing what I want as most of these are well data analyst or senior data
9:15:47
analyst. I got this one business analyst. I want to look at some that are data scientist engineers and data analyst to see if we can categorize them. So I want to randomize this sample a little bit more. So I'm going to modify this query that we have and I'm going to put an order by and then throw in here a random function. Random function it is it's not just duck DB specific you could also use it in Postgres and SQ Lite but it's not used in
9:16:12
every single SQL dialect. Anyway let's go ahead and run this. All right so this is getting more random results of what I want. All right so looking at this I think the plan of attack I'm going to do for this is so if like things like data engineer I can see that there's data and engineer in there in this case it has data engineers with a NAS so we'll need to make sure we capture for that. But even the other ones like data analyst they do have I mean data and
9:16:39
analyst in it and data scientist has things like data and scientist in it along with other things. So I think we're filtering for those specific two words in each we could maybe get you know maybe 90-80 percent of it. All right so after the job title I'm going to throw in that case and then automatically put in that end and I'm going to give it an alias now of job title category. All right so let's go for the first condition
9:17:03
and that's when job title contains data we can't do that equal sign. Remember we're you talking with text character right now so we're going to use like and then inside of single quotes I'm going to use two wild card characters of that percent sign and then data and so this is helping those percent sign if you remember I can match zero or more characters so data could be at the front or it could be at the very end it'll match no matter what.
9:17:30
Now this is just the first of one condition we can actually we can plug multiple conditions into this and I can do this by putting an and in here and then specifying that job title now we want to add in that portion let's go for analyst next. So I'll use like and then inside of single quotes those two wild cards surrounding analyst. All right so now we have these two conditions in here if we meet those two conditions then we want to categorize it
9:17:58
as data analyst. Okay this is actually enough for now to go ahead and just run this right because remember you only need one when statement inside of a case expression so we'll go ahead and run this and I have an issue because I forgot to include a comma at the end specifically we say hey end as job title category I need to include a comma right here. All right let's try this again. All right not bad we can see our first one it does match on that data analyst
9:18:27
doesn't specify as the analyst also this list doesn't really have a lot of data analyst in here and this one right here does have data analyst in it but this case sensitivity messes us up in that the analyst is lower case. Luckily we're going to be covering different text functions to help with these type of situations but that's for an upcoming lesson right now we're not gonna be able to handle it. So let's build out the remaining of those scenarios we want to meet.
9:18:56
So I'm going to go ahead and just copy this paste it twice and then update that line for data engineer and then the third line for data scientist. Then we'll also include a catch all using an else specifying it as other. All right let's go ahead and run this bad boy and bam not bad. This goes through and as we can see I mean this doesn't get 80-90 percent like I thought this gets about probably about 70 percent but pretty good in categorizing these data
9:19:24
analysts data scientists and data engineers just using that job title column. It's not really related to this but just as background if you're curious that job title short column I used a machine learning model specifically it's called BERT you can google it. It went through and read all these job titles then categorized it into these 10 different ones here. So I didn't necessarily use a case expression for this but this is a situation where you could use a case expression
9:19:51
if you didn't have access to machine learning models. The last use case we're going to look at is more specific to those data analysts and I do cover this more in depth in my intermediate sequel course. This is going to be a brief intro you don't even have to follow along if you don't want to for this specific example. Anyway it's on conditional aggregation. Let's say we want to calculate the median salaries for different buckets. Specifically we'll say one of the
9:20:16
buckets is a yearly salary of less than $100,000 and a yearly salary of greater than $100,000. What is the median salary for each of those buckets? Once again there's a more advanced case so don't expect that engineers to have to do this all the time but you should be aware of it. So for this we're going to do a select and we're going to aggregate this based on that job title short column so I'm going to use that first. We're going to just do a simple count
9:20:41
first just to get the hey what are the number of job postings there. This is not part of the conditional aggregation using case it's just a simple total postings that we're trying to find. So let's get into this actual conditional aggregation for this we're going to be using the median function and inside of here we're going to use that case expression and we're going to do that first bucket first of when salary your average is less than $100,000.
9:21:08
And in that case well we want to provide that salary your average. Now technically the else basically if it's not any of those conditions we're going to provide it back as null and so you could do else null but that's sort of that's not necessary it's redundant it's already assumed so I'm not even going to include that. Anyway we'll give this all the alias of median low salary put a comma. All right so I'm going to just copy this now and paste this underneath here and then
9:21:37
change this to greater than or equal to 100,000 and the alias to median high salary. Remove this comma from here and then we're going to be from the job postings fact table specifically where that salary your average is not null. Obviously I just don't want to see them in results and then from there put a limit 10 in there for to see only those first 10 results. And actually limit 10 is not necessary I don't know why I said that
9:22:03
we're doing aggregation here right counts medians so I need to do a group by that job tile short column. Now we'll go ahead and put a semicolon a limit's not necessary because we only have 10 job postings and go ahead and run this and bam we'll get this back to where we see for each of our job postings where the different total postings they have associated with it and then for salaries less than 100,000 what is the overall median salaries for that bucket
9:22:31
compared to those greater than 100,000 what are the different what's the median salary for those once again this advanced concept I don't necessarily expect that engineers to know this you just recognize when this is being used in case you find it in queries out there all right let's move into our final example and for this we're going to look at one other common method used for case expressions and that is for conditional calculations specifically that first part of this
9:23:00
problem where we're going to compute a standardized salary using our yearly salary and then adjusting also our hourly salary basically getting our hourly salary to a yearly data and then making it all into one column from there we're going to go also and then bucket this data into tiers of low or medium or high based on where it falls into the provided ranges so let's build a query first to handle that first condition
9:23:31
of getting that standardized salary for this I want that job tile short column and then we want to check the work as we go so I'm going to put in that salary our average along with salary year average then we're going to get into building our case expression which we're going to give the alias of standardized salary so for this we're we're making this column right and it's using yearly salary when yearly salary is available or then our salary adjusted whenever that's available
9:24:01
so the first thing we need to check for is well is yearly salary available and we can do this by checking if it is not null so I'll specify salary average is not null in that case I do want to provide it oops sorry I read this hour I don't know why right hour and we're talking about year like right now I think that's a salary year average so in that case then we want to provide that salary year average column because it's not null conversely when salary hour average is not null
9:24:32
then we want to provide salary hour average but with a twist we want to multiply this times two thousand and eighty basically there's two thousand and eighty or typically two thousand and eighty work hours in a year so I'm going to multiply this times two thousand eighty now in either those cases we don't meet this I want to provide it back as null but as I discussed earlier that's sort of by default anyway so I'm not going to include that all right let's wrap up this screen now
9:25:02
we're going to be doing this from the job postings fact table and I just want to see the salary or average in salary hour average data right now so we're going to say where salary or average is not null and that salary hour average is not null we'll go ahead and limit this to 10 let's go ahead and run this bad boy and I have a syntax error basically I included a comma at the end of my case expression or by when that's not right you don't need to include a
9:25:31
comma at the end of that I went ahead and removed it and we got zero rollbacks because I'm silly and didn't even think about this this shouldn't be an and for this condition of where this should be an or condition to actually get something back because it back gonna be never both a salary hour average and salary your average in a single column or in a single row okay we finally got results back anyway pretty cool we have back this standardized column
9:25:58
where we have that adjusted salary average provided in a yearly format and this is all within a single column that we now can aggregate with or like we want to do actually bucket so what we're going to do now is we have this we have this available we need to move on to categorize these salaries into tiers or buckets of low medium high so because of this I'm going to wrap this all into a CTE so I'm going to dent this over remove this semicolon and I'm going to go and start with that
9:26:30
statement of width and we'll give it the alias salaries as and then open the parentheses and then at the end of this close the parentheses now I didn't also remove this limit 10 at the end because we're wanting to well bucket all the salaries I can if I want to I'll put the limit on the next one so now we're going to add a select and I want to get well I want all of these columns from that salaries in addition to we're going to be
9:26:56
bucketing so I'm going to throw in that case expression and we're going to give this case expression that we're doing as a salary bucket now our first condition we need to meet right up here of less than 75,000 of low so coming in here in that case expression I'll do when standardized salary is less than 75,000 then low then when standardized salary is less than 150,000 we'll do medium else we'll do high now remember we're filtering out
9:27:27
null values up here in the CTE so if we didn't have this next I'm going to just go ahead and remove it and say we don't have that there to remove out those nulls we would need to have a condition to handle those nulls so we'll say hey when standardized salary is null then it's missing all right so not bad let's go ahead and wrap this up I'm doing from all the our CTE of salaries and for this one I do want to see the values right remember we have no
9:27:56
values in there so I'm just going to do an order by of standardized salaries and do descending we'll just do limit 10 for right now okay let's go ahead and run this hopefully we don't have any errors in this all right not too bad we're seeing well all these are marked as high because yeah they are greater than 150,000 conversely I can come in here in this order by and remove this descending portion and then get it ordered in ascending order
9:28:24
and now we can see that the low ones are actually marked I can also go as far as just to remove this order by all together and then run this and we can see that the null values are all marked as missing so looks like our calculations or case expressions are working out just fine all right you have some practice problems now go through and get more familiar with using these case expressions in the next lesson we're going to be jumping into function specifically
9:28:51
date functions with that see there welcome to this lesson on date functions and for this there's a whole host of date functions we go over but we're going to be focusing on the three most common that I find that I use on a daily basis when working with SQL first is extract and this is used to pull out specific parts from a date such as like a year month or day next is date trunk or date truncate and we can do things like well truncate a date so our date time
9:29:25
data we can make into maybe just only a date and then finally we'll look at at time zone in converting date time data into different time zones and before we get on that let's do a quick refresher on what we know up to this point on dates where we focus on that job posted date column so let's query that column in that job postings fact table for this we only need 10 results in the terminal i'm going to go ahead and connect some other duck
9:29:56
to that data jobs data warehouse all right so let's inspect these columns And, that's all I mean. misspelled job postings fact not facts. All right so inspecting this we can see that the job posted date column is of the data type time stamp which means it has not only a date part but also a time part. Now let's use the casting operator to convert this to a few different data types that we saw previously in our data types lesson of date time time stamp and time stamp
9:30:29
Z where that includes the zone of the time stamp. More specifically time stamp with a time zone so I'm going to go ahead and create a new column with a job posted date I'm going to use that casting operator to first convert it to a date and then give it the alias I'm just gonna give it simple for right now of date. Next we'll cast it as time and provided the alias of time then cast it as time stamp name it as time stamp and then finally cast it as time stamp with time zone as time
9:30:59
stamp Z. All right let's go ahead and run this all. All right so inspecting these these are all pretty self-explanatory right the date is a date only the time is well time only time stamp there was no change because it's already time stamp and then time stamp with time zone for this one look it kept the same time and then it added this negative zero six what the heck's going on here so typically what databases do whatever you convert it to a time stamp with a time zone it converts
9:31:32
it to your local time zone right now I'm in central time zone the United States and that is negative six and what this means is this is six hours behind well where there's a zero and that's known as UTC time or universal time coordinated or more commonly spoken coordinated universal time this is strange enough to find is the time at the lunch to zero or the prime meridian strangely enough not
9:32:05
with the international date line I'm not sure why but it's no it's located or based off of this time and it's a central location that most people base their clocks off of and by most people I mean all so why do I provide you that history lesson well up above whenever we converted this time stamp z or time zone zone it converted it to this negative six or my time zone and actually this I know whenever I collected this data I actually stored this as UTC so it's right now it's converted to
9:32:37
the wrong time zone of what is keeping track of anyway that's a future problem we're going to be fixing with the at time zone function so a little spoiler alert so let's get into our first function and that is extract it follows this syntax for using in a select statement we can use another statement as well but for using select for simplicity we would use the extract and then we're specifying a part from the column name which the column name would be some sort of date time or date object
9:33:10
so going back to that base query that we had where we're selecting from the job post to date from our job postings fact table we're going to go ahead and extract out we're just going to give example first of extracting out the year from the job posted date column and then we can also provide it as a column of job posted year let's go ahead and run this bad boy all right so for all of these we can see that we got this new column called job posted year we got it in an integer
9:33:39
data type format because it's just well integers for the year itself so now you're probably wondering look what are different parts that I can put into here besides just year well jumping into the documentation induct dv on date part functions we can see first of all that can be used for time stamps time stamp z date and also interval intervals real quick or periods of time that we added or subtracted from date time stamps or time stamps with zones we're not going to go into intervals at
9:34:09
all with this but it is important that you're aware that you could do intervals such as like one year or a month and one day or whatnot anyway we can use the parts in the extract function in this list right here and you can see we could use century day decade hour microseconds and what not and this is for date part specifiers and in intervals along with this down here for date part specifiers exactly and so you could go as far as saying things like day of the week day of the
9:34:37
year or even something like week number anyway common ones that I find myself doing are specifying things like the month and then in this case I gave you the earliest job posted month or even something like day in this case given the earliest job posted day running this query we can see that these also are converted to their appropriate columns and each of these are represented as integers as well now I find this highly useful especially whenever I want to do aggregation
9:35:05
let's say in our case we want to aggregate the total number of job postings for data engineers on a monthly basis so let's modify this I only need the month and year I don't need this day so I'm going to go ahead and remove that I also don't need that job posted date and remember we want to aggregate on a monthly basis so I'm going to throw an aggregation in here specifically we'll count the job ID and we'll give it the alias job count for this we'll do this where job title short
9:35:35
is equal to data engineer and then we'll need to remember we're doing aggregation so anytime we do an aggregation we got to do a group buy and I'm going to just use the actual names themselves remember DuckDB doesn't allow you or DuckDB allows you to use the alias but some other databases like Postgres don't let you use the aliases so just for continuity I'm going to use this I'm going to remove this limit 10 in here let's go ahead and run this and see what we got so far selecting it all
9:36:05
pressing shift enter okay so this executed the query inside the terminal right now it's a humble jumble mess but it looks like it's pretty good except for okay it's I want to get it in actual year and month order so I'm going to do an order buy and in this case we can use the alias because that order buy is done after the select statement in our order of execution so we'll organize by the job posted year first and then job posted month let's go ahead and run this looks like I misspelled
9:36:37
order I don't know what's going on here I don't know why I highlighted it blue still oh sorry I didn't misspelled order I put a semicolon at the end of this that's why it got confused all right let's go ahead and run this all right so pretty cool it's now plotting it in order of year and month and now this is much more beneficial and taken it so I can analyze it anyway I threw this into chat GBT to do the dirty work for me and we can actually see how those job postings are
9:37:05
going down over time specifically there's a dip back in october of last year and actually I somewhat suspect that towards the end of a year they sort of slow down on hiring and then rip it back up at the beginning of the year anyway pretty cool insight with this next up is date trunk or date truncate for this inside of our formula for date date trunk we're going to specify a precision and then from there inside of there provide the column of a date time column and more precisely
9:37:38
it can be date or timestamp or timestamp with zone anyway this one is precision because this returns the value rounded down to the start of the specified unit what I mean by this so let's go back to that core query that we were working on previously a job posted date from job postings back table for 10 entries and here we're going to use that date trunk formula we're going to specify the precision now this time it has to be in single quotes and provide it and I'm going to
9:38:08
just make it month for right now and then do a comma and we'll do job posted date we'll give this the alias of job posted month all right let's inspect this see what we got here and in this case as we see we ran the formula what it did was it rounded it down so these are all January first postings right here and so what you're not saying is it actually rounded it down to all be January I think it'd be better if I actually inside of here do an order by and then do random so we have
9:38:43
random dates in there now whenever we run this okay this one's showing better right here is an entry for June 18th of 2024 but it rounds down so it's only June 1st 2024 and look at this the column is of the date of the data type date now once again what can we use for well precision in this case it's the same values in here there's just this time we do have to use them in single quotes when specifying them within this function and they're typically written in our case in lower case so
9:39:20
you can do anything from century day decade hour whatnot so I went into here and I just went and plugged in we're going to look at year quarter month week day and hour other common things that I find with date trunk we're going to go ahead and run this so with the year it rounds up to when this case 2023 all of these start with January 1st be the rounding the next is quarter and the same here it's going to round to that first month of the quarter and that first date similarly we saw
9:39:50
a month already where it rounds that first date you could even round to a weekly basis a daily basis or even an hourly basis so let's simplify our previous query that we were working with now using this date trunk form function remember we were using extract and pulling out the year and then the month then we're having a group by this and also order by that it's a whole mess because we have to deal with two columns anyway instead we're going to delete this use date trunk specify that
9:40:20
we're going to do the month as precision and then we'll specify the job posted date column and give it the alias job posted month we're still going to go with the same best practice of listing this entirely inside of the group by but in the order by we only have to list job posted month all right let's go ahead and run this bad boy and we're getting now the same results back only with now two columns now i like this so what happens if we wanted to filter this data let's say we wanted to
9:40:51
look specifically at 2024 well in that case we can filter it i'm going to add an an condition in our where and to do this i'm going to use the extract function so if we're going to extract the year from job post-a-date and set it equal to 2024 now whenever we go ahead and run this we get only the 2024 values back now i find this is very common use cases of combining both date trunk and also extract and then i'm finding i use like date trunk in the select statement and then
9:41:29
extract in the where now i could have used instead of using extract here i could have used date trunk specifying this as year and then of job post-a-date and setting it equal to 2024 0101 i think that's right let's go ahead and run it yeah that's right so i could do it this way but it's a little bit more robust and it's not as easy to easy to read as using the extract so there's cost and benefits
9:41:56
of when you would want to use either all right last concept to cover is using at time zone and we're going to be converting between different time zones throughout the world now for this the syntax is pretty simple well that's pretty simple if we're dealing with a timestamp with a time zone already associated with it in that case what we're going to do is with our an associated column we're going
9:42:24
to use the keywords at time zone and then specify in single quotes the time zone we want to go to now in order to know which time zone you can use for this you need to go into the documentation itself we're going to be using really simple examples in our case specifically all that utc or universal time coordinated and then my time zone that we're currently in of central standard time or cst and we're also used es t or eastern standard time zone anyway just as a
9:42:57
reminder utc is that zero time zone so it's right there at the prime already and in the middle central is that of the united states at that negative five or that blue portion and then eastern standard time zone is that negative five so one hour ahead of central or negative five behind utc all right so let's start with a simple example first and we're not even going to query our database we're just going to make a single data set so in quotes i'm going to specify a time and date of january
9:43:30
first 2026 at exactly midnight with the time zone of utc so zero zero now right now this is a string because this is in single quotes so i'm going to cast this as timestamp and then add a tz to specify that it's time zone anyway let's see what we get with this first so i'm going to go ahead and run this remember to have that semicolon in there so what this did is it returns back well data type of time state with time zone and if you notice with this it actually once again it converts it to
9:44:08
my local time zone so it's going to convert it to your local time zone and remember i'm in central standard time zone so that's negative six and so we originally started with this at midnight on january first six hours back would be eighteen hundred at night but for at that negative six for that central standard time zone so this is effectively of doing the same thing of at time zone and then specifying in this case central standard time you'll need to put in what your
9:44:41
appropriate time zone is and then running this we get what it is of eighteen hundred for that time zone so we just demonstrated how to convert from utc to central standard time zone once we want to go one hour ahead to eastern standard time zone well i would just adjust our query to be eastern standard time and then run this and it has no problems with this it adjusts it one hours ahead to nineteen hundred so this is not that hard of a concept especially when you're dealing with
9:45:11
time zone data but unfortunately for us our job posted date data well let's actually just query it and then someone look at the job posted date column from our job postings fact table and just limit this to ten results querying it okay what i want to show with this is that this is just of the time stamp well it's all is of the data type time stamp only so the syntax we need to use for this is a little bit different so when you have time stamps without time zones our situation that
9:45:42
we're in it's automatically treated as a local time in duck db and most major databases so because of that we need to use at time zone twice first we need to use it to specify what is the time zone of the data itself and then you use it a second time to convert it to whatever time zone you needed it so let's get back to that job posted date this data i know as we recorded it we recorded in real
9:46:13
time and i specifically recorded it in utc so what we need to do let's say we want to convert it to eastern standard time zone so what i'll do is i'll add at time zone specifying utc and actually let's just run it seeing what it's going to do with that so press shift enter with that we can see that it applies this negative six so it could provide it provides it to our local time it converts it to
9:46:38
our local time zone but if we scroll up we can see that this didn't actually mess up even though I converted it into our local time zone for the results here when comparing these entries i can see that okay previously it was zero zero zero zero zero zero four and that was utc so i would expect whenever we go we do that negative six yeah that's six hours behind that the day before on december 31st of 2022 so the core of the data is correct but it's not in the form of the one
9:47:12
mainly we wanted to go to eastern standard time zone and so going back to our query we can just add at time zone es t now whenever we go to run this we get to that first entry is at well one hour ahead scrolling up here we're going to go over previously it was 1800 at 0004 for central standard time zone well eastern is one hour ahead of us so that's 1900 so it did do the conversion correctly as i would
9:47:40
expect to be 1900 on december 31st for this value but i just want to show this it wouldn't be correct specifically for our data to not include this at utc but we're to take this away and then now try to run this this does not go to the right time at all so once again it converts that local time zone of negative six we're in central but this time it says that it's only one hour behind that initial
9:48:10
entry which scrolling back up we can see what the initial entry was of january 1st 2023 at midnight 004 so this is converted completely wrong moral the story is that we need to if you're dealing with date time data you need to specify what the time zone is for that data and then also what the time zone is you want to get to so this type of time conversion is really useful let's say that i was you know we did all this work to get it into eastern standard time zone well we could
9:48:41
filter data where jobs are located here specifically we'll do it on the job location column and we'll look at values that are from new york so we specify these values i'll also go ahead inside of the select statement include the job title short and job location then from there i'm going to remove that limit 10 from here i want to just see everything that's there in new york now inside of terminal we can see we ran the terminal ran the command properly and we start
9:49:11
getting back all the different hours and times that we got all these different results back from new york job postings and it looks like we got a total of around 20 000 rows now this is actually pretty cool because we could do some analysis to see what times locally they are actually posting to sort of game when the most job postings are so what i'm going to do is copy this original query that we have because that's pretty useful information and now i'm going to modify this one
9:49:41
to where i want to extract out the hour from this job posted date for that eastern time hour that was posted so i do extract hour from and then that previous column that we had and for this i then want to do an aggregation specifically account of we'll do account of the job ID we did an aggregation so we need to do a group buy and i'll go ahead and just copy this all and paste it into the group buy also i realized up above i didn't give this an alias so i'll call it as job
9:50:13
posted hour okay let's go ahead oh we need an order buy in here so i'll do an order buy in this case we can list the alias of job posted hour all right there's a lot of stuff i'm not sure if this is going to work oh it did work all right so we have this not bad i'm going to take this and post this into chachi bt and what we can see from this is is that this is plotted an hour per day it looks like we have a very high spike at four in the morning and then it sort of slowly goes
9:50:48
down throughout the day and then dies down after eight until about midnight so this could be pretty useful in being able to identify trends to a specific region because we've converted it to the applicable time zone all right so now your turn to give it a try testing out those three different popular functions i've found for manipulating date time values in the next lesson we're going to be jumping into set operators all right with that i'll see you next one
9:51:18
in this lesson we're going to be going over set operations and this allows queries you could have two or more queries to be combined according to set operation semantics and that's some mathematical terms that will actually go through each specifically for these six terms yeah there's three listed here but you have sectorly of union union all intersect intersect all accept and accept all anyway enter deep into it let's jump into the computer
9:51:52
so before we even break down those six different operations let's first understand the problem that we would probably be trying to solve that we need these set operators for here i have two tables one on the left is from 2023 and has job postings and then also we have the next one to the right of it two job postings from 2024 now between the two they have some unique job postings that don't get repeated but some of them do specifically highlighted in red is that of a data
9:52:27
engineer in the United States for a hundred thousand that one got repeated between the years similarly we have one for data analyst and we noticed in 2024 now it got repeated once but got repeated twice well there may be certain scenarios that we want certain values such as maybe one other repeating values or non repeating values and set operations are going to solve it we'll actually go through solving some different scenarios with this for the final example now as mentioned
9:52:56
there are six different set operators that we're going to be walking through today we're going to walk through each one of these with just simple examples first and then we'll move on to actually applying it to our job posting data set first up we're going to start with the most common of union and union all in this what we're going to be working with is the same table so table A and table B they only have one column in our case but as shown in that earlier example of job postings fact you
9:53:23
could imagine that you could also do tables that are well more than one column anyway table A is represented by red table B's blue where they intersect is in purple in the case of union this returns all rows from table A and B and it removes any duplicates now union all does something very similar to just union in that all values are preserved or duplicates are preserved so in the case of table A and table B everything is just appended together so let's actually demonstrate this
9:53:58
instead of just looking at a diagram and we can recreate those diagrams with a sequel query first thing I'm going to do is go ahead and connect to mother duck to that data jobs data warehouse now what we're going to do is we're going to recreate this table A and then also this table B right there so we're going to do something we haven't done before we're going to first create a list of those values you create a list and sequel by using square brackets and then from there we'll place
9:54:25
the values inside of here if I go ahead and execute this query I can see that I get back this main list values one one two and it's of the data type int of a list anyway it's inside of here what I need to do now is I need to use a function to get these this list value into rows so for that I'm going to use the unnest function running this I can see that I get all the values in an actual table with their different rows I can also create another query for that table B and this one now
9:55:01
has table B's contents anyway what I can do is well I'm going to remove this semicolon here we're going to now perform a union between these two so I'll do a union right the operator and all it is is is just right in between the two queries now when I go ahead and run this boom I get back those values now union doesn't do any type of sorting it's just going to perform the union operation and some things may get out of order in this case they did two one three but it's unique values
9:55:32
if I change this now to union all and then run it I expect to get both values from both lists and yes we get both of them all seven values next up is intersect and this is going to return rows that exist in both queries basically their intersection so this returned rows common to A and B with duplicates removed in our case one is preserved in the results and the other duplicate one is removed similarly with intersect all this returns rows common to A and B with duplicates preserved
9:56:09
note that this one in this case right for table A that has one three times and table B has one two times so this is a row by row comparison of maintaining those duplicates so that third one in table A is not preserved anyway let's test it out here I might go ahead and just put an intersect it uses pretty much the same syntax as union go ahead and run this and only one is returned similarly I'll add intersect all running this just one and one is preserved does that row
9:56:45
by row comparison to only preserve those values next up is accept it's also known as minus and some database like oracle but accept is most common this combined results from two queries to compare or filter data in our case this is going to return rows in A but not in B and duplicates are removed so for us to is preserved from table A as that's the only value not in table B now accept all is going to be slightly different that we've seen previously from other all's in that
9:57:23
we're going to return rows in A minus rows in B but duplicates are removed one for one so in the case of table A containing three ones and table B containing two ones it does a one for one duplicate removal and one of those ones that third one in the A is going to stay in the results along with that two that's not in table B anyway let's demo it in VS Go so trying out accept first I expect only two to be preserved yep two is only preserved and then using accept all one and two are both
9:57:56
preserved in that case now these six different separate set operators are pretty simple right union is just combining all the rows from table A and B intersect is their intersection and accept is basically getting that first table rows that aren't in table B but applying this in a real world scenario can get tricky so that's why we're going to move into our final example to actually put these to a real world scenario so let's set up this final example as I demonstrated previously
9:58:28
we're going to be using the job postings from 2023 and 2024 we don't have any tables right now with that so we're going to have to create some temporary tables for both 2023 and 2024 once we do that we're going to then walk through six different scenarios on maybe questions we would have about these two years and how we can use these set operators to handle it so let's first get into creating those temporary tables for 2023 and 2024 to do this analysis so for this I'm going to start
9:59:03
by creating temp table and I'm going to call it jobs 2023 and I'll use that as then we're going to use a query basically want to select well most all the columns from job postings fact now I don't want to compare all columns between 2023 and 2024 and here's why in 2023 we're going to have unique job IDs that's just unique to basically the row that it's on so that's not going to matter we don't need to compare that value to 2024 also our job posted date column is going to be unique to
9:59:39
the date and time they posted it and so that's not necessarily unique to the job posting itself so I need to just go ahead and remove those rows and get into a table without those rows now I'm a little lazy and I don't want to have to go through and write all that from different columns just to remove job ID and job posted date anyway DuckDB and some other popular databases not all of them have this feature where you can see the image.
10:00:01
You can write select star and then exclude the college you want to exclude. Once again, there's DuckDB specific, not necessarily all database sport this. Anyway, what I can do in our query here is I can do select star exclude. And then in parentheses, write job ID along with job posted date. Go on ahead and then ran this. I can see that the query executes properly and job ID is no longer in the front. I can't really see it. So that's actually the easiest method.
10:00:33
So what I'm going to do is I'm going to just put a describe up at the front of this and then run this with describe. And this then outputs what columns are there. Now I have these columns left. I've gone through it and I want all of these columns that are unique to a job posting and there's something that would trigger me in my mind that. OK, this job posting is duplicated by this company because it has this location and this amount of pay anyway.
10:00:59
I need to get rid of this describe now and go up here. I know this query works just fine. So I can go ahead and well, I can't run it yet because this isn't filtered for 2023. So we need to do a where clause and we need to filter it for the year of 2023. So we can use the extract function specifying year from job posted date equal to 2023. All right, let's go ahead and run this bad boy. All right, not bad. I'm going to then just go ahead and query it.
10:01:30
See what a sample the data is from this job's 2023 data set. Looks like 787,000 rows. All right, this is looking good. Let's do the same thing now for 2024. I'm going to just copy this, create table, paste it below, change 23 to 24 in two locations and then go ahead and run this. Then similar to 2023, I'm going to also make sure that it is, you know, available. So I'm going to query it as well, and it looks like this one has 480 30,959. So let's get into our first question.
10:02:02
And it's this, what unique job postings appeared in either 2023 or 2024? What do you think? Feel free to pause after each one of these questions to give you time to think about it, but I'm going to jump right in. In this case, we'd want to use the union operator. Is this going to return all those unique values with duplicates removed? In order to create this, I'm going to copy this query up above, root the semi colon, add a union and then copy this 2024 one and paste it as well.
10:02:33
Now running this bad boy, we get a total of 1.08 million rows. But what was our starting rose to begin with for 2023 and 2024? Well, what I actually like doing with the union operator is what I can do is I do a count of each of these. So I added the count function of both whenever I run this with this union, which pretty neat is it puts it into a table with the values of each. And I can actually modify this table further to make it a little bit easier to
10:03:09
even read to know which one's 2023 and which one's 2024. So I'm going to break this query down by formatting it more appropriately. Anyway, I'm going to create two columns. Now, these columns have to match. And for this, I'm going to give a text value of jobs 2023 for this first value. And we're going to give it the alias for the column name as the, well, table name. Now, real quick, I do want to demonstrate something right. These are these two queries aren't matched up.
10:03:37
I have two different types of I have two columns in this first one, one column in the second one. If I tried to run this, I'm going to get an error. That operations can only apply to expressions with the same number of results columns. Also, they need to be the same data type. So I can take this up from above, paste this name underneath here, change this 2023 to 2024. And then now whenever I run this, it works.
10:04:02
And it has the table name is job 2023, job 2024, along with their associated values. Also, I probably want to give this count an alias specifying that it's maybe the record count or the row count running this again. All right, now we have it. It's nice and cleaned up. I can actually see this. So originally we had 787,000 in 2023 and 483,024. When we did the union operator with this, we got back 1.80 million. What did I say? 1.08 million rows.
10:04:36
So now let's say we wanted to ask the question, what job postings appeared across both years in 2023 and 2024, counting the duplicates? If you said union all, you'd be correct. It returns all rows from tables A and B, preserving the duplicates. So I just copied the query above modifying union and we're going to change this to union all. Originally, we had 1 million job postings in this. In this one, we have 1.27 million job poses. This makes sense.
10:05:07
There are more job postings because duplicates are preserved. Next question up, we're going to switch it up on you. Which job postings appeared in 2023, but not in 2024? This would be except in that it returns rows in A, but not in B. So those in 2023, but 924. So copying the query from above and changing union all to accept. So I'm going to run this. I expect this to be one of the smallest query results. Well, smaller, at least than union.
10:05:38
And it is in the fact that it's down to 709,716. And going up to execute that query above that does the count of each one of these. How does that compare to that? We can see that 2023 has 787,000. So in this case, there's almost 80,000 duplicates that aren't in this one. All right. Next up, which job postings from 2023 remain after subtracting those that match from 2024? Doing a one for one comparison. Well, that'd be except all.
10:06:12
It returns rows and A's minus rows and B's with duplicates removed one by one. Copying the query above, adding an all. Previously, this query was at 709,000. Now, as expected, it's going to go up to 766,000. So make sense. All right, second, last question, which job postings appeared in both 2023 and 2024? In this case, it would be intersect in that it returns rows coming to a and B with duplicates removed, making a little bit of an assumption in this case that they want
10:06:48
the duplicates removed. So I'll copy the query above and replace it with intersect. Let's go ahead and run this and expect this one to actually be the smallest of all queries in that yes, only 18,761 job postings are repeated. Now, that gets us into our final question of what job postings appeared in both years, preserving duplicate counts. In this case, we're using intersect all and that returns rows common to a and B with duplicates preserved.
10:07:17
So I'm copying the query above, adding intersect and then adding all. I do expect this to be greater than that 18,000 previously. And it goes up to 21,000. So almost 3,000 postings were even duplicated on top of repeating between the years. That's a quick crash course on set operations. Once you understand the basics of what union intersect and accept are doing, you can better apply it to different business situations that you need it for.
10:07:47
Are you have some practice problems to now go through and get more familiar with this set operators in the next lesson, we're going to be moving into some functions, specifically some text and also functions to handle no values. With that, see you in there. All right, welcome to this lesson on text and no functions. And we're going to be going through, well, two sections. First, we're going to go through some common text functions that I find myself
10:08:16
in with the final example and then also with no functions with its associated own final example. Now, real quick, this lesson and actually this entire course is not, well, all encompassing, right? So like text functions, there's hundreds and well, not hundreds. There's tons of different text functions that you can have and access. And then over here, looking here on the left hand side, inside of DuckDB's documentation of functions, there's a variety of different types of functions.
10:08:47
We did and we will have two more lessons on functions after this. We've covered the majority of this. You're going to have a good understanding already of what function should be used for what. But I really can't stress it enough that you should use some sort of AI like ChaiChaPT, if you've ever get into a situation that you're curious about. Well, is there a function for that or not? You can simply query this and it will provide you back on what you can do.
10:09:13
So actually moving into text functions, I find that there are six major types. We're going to first go through just some simple examples using only a select statement to basically test out each one of these and then we'll move into that final example. For this, I'm going to be starting in a new SQL file. Additionally, I might go ahead and connect to mother duck data jobs data warehouse. First up is length and count. And for this, we can count the number of characters in a string.
10:09:40
Now, I have to listen here of length and car length or char length. But going to the documentation, what we can find is is a lot of these that exist in other languages. They'll sometimes give them, in this case, like car length right here. It has the alias for length. And they'll have it's the same function. It's just alias differently. So that way, if you come from another language, you can just stick with your same syntax that you used to.
10:10:07
Anyway, in a select clause, I'm going to enter that length function and then just put any old text phrase in there. I'll stick with SQL. Whenever we're going to run this, we get, as expected, three SQL changing this to char length. This also works and gives us the three because it's the same function. It's just aliased. Next up are case conversions. We can use this to upper and lower case different letters within a text or all of them specifically.
10:10:33
So let's trust out that lower running lower. You can see that gives us an all lower case. And then I also want to try upper, but I want to provide it. Actual lower case values will do capital S and then QL. Then running this bad boy, we get all SQL in upper case. Next is substring extraction. And this is really good if we need to extract out maybe something at the beginning or left of our text, the right of our text.
10:11:01
Or if we need to pinpoint something specifically based on a number of characters, we use substring anyway, we're going to start with the left function first. And this is the syntax as you provided the string or the column, if you will. And then the number of characters you want to extract from the left side. So in our case, we're just going to do two and it provides as cute. Similarly, I can try out right whenever I run this, I get QL. Next, we'll do stub string.
10:11:28
Now with this, this is going to take multiple more than two arguments. Take not only your the column or the texture providing, then from there, the number of characters over that you want to start the extraction. So in our case, we'll say, hey, we'll keep it at two. So it's going to start at Q and how many characters we want to extract. In this case, let's say we want to attract two more. So whenever we run this, we'd expect to get QL.
10:11:53
OK, and we do and we can even change it up if we want. I can extract just one character if I want, starting in that second position. In that case, I'm only going to extract Q. Next up is concatenation. And we've seen this before using the concatenation operator. So this should just be a simple review in this. This is used to concatenate text functions and put them together in this. I specify I concat and then I could do like SQL and then maybe do a dash and then add in functions.
10:12:26
If I go to run this, I get SQL functions. Now, I personally like to operate this with the concat operator, no pond intended. And it makes it I feel it makes it a little bit more readable as you don't have to have this function and you just see the symbols right there. Anyway, whenever we run this, we're going to get the same results of SQL functions. Next is trimming functions and we can use commonly, I would just stick to trim
10:12:51
as it trims white space that is on the left or right side of a statement. In this case, there's an extra space to the left is SQL and also to the right. You could also use something like L trim or left trim and art, right trim. And this will remove the extra white space on a certain side. But typically for me, I find just trim is good enough. Anyway, first, let's just provide the select statement with the SQL in it itself. And it's just the string so we can see where it is.
10:13:20
All right, so now we have reference. I mainly do this because there's no quotes around here. You can't really see the empty space that it's getting rid of. Whenever we go to do now, the trim function, which is going to wrap that all in. Now running this, we can see that it did trim it because SQL had this extra space right there and now it's all the way over. Last up for text functions are replacement and they have two options here.
10:13:46
You have like your standard replace and then you also have regex replace. Anyway, let's just go over replace first and then we'll get into regex. Anyway, in here, I can run the replace function and we run it on what we want to run it on. We're going to stick with SQL and we'll say, hey, I want to replace the Q with an underscore. So when running this, we get S underscore L. Now, this is just this replace function.
10:14:14
You give it a text in this case to replace with and it tells you where you tell it, then what you wanted to replace with. It's very standard. Now that brings us to using regex now for a place and this one's more dynamic. You give it the regex or the regular expression in order to pick up on, which is sort of a little bit of coding, if you will, and then what you want to replace it with. Now, regex is beyond the scope of this course and if you don't know regex already,
10:14:44
I recommend just use chat GPT. In this case, I could say, hey, give me the regex replace formula in SQL to extract out any email domains such as gmail.com. I also had this assume the input column is valid email addresses. So it then goes through and provides this formula from us. We can see this formula right here uses regular expression in order to pick up on that, well, usually an at symbol that is the domain. And then it fills it in at the end with well a blank character.
10:15:17
Anyway, let's test out. And so if you're here, I place it and I'll just replace email with an actual value of data.nerd@gmail.com. Now, let's go ahead and run this boom. And this extracted out the email domain. I wanted what's forward anyway. You get the point you use chat GPT for this. So let's get into our final example. If you remember back to our lesson on case expressions, we got to this portion where we were going through and using case expressions in order to match up based
10:15:52
on a job title, looking for certain keywords and then providing whether it was data analyst, data scientist, data engineer. So running this, this does do the same problem that we saw before and that's that some of these are categorized as other right because they don't have data or analyst or engineer scientist. That's fine. The one that does bother me though, that I want to fix is like right here. We have data analyst, right, but it's marking it as other.
10:16:18
Clearly, that one is a data analyst role and that has to do with this text is doing like more of a perfect match. So what we want to do is we want to do a lower case of this job title. Now, I'm not a fan of going in and putting lower into was one, two, three, four, five, six different times just to then try to detect this instead. I'm going to recommend is or what I would do in practices. I would first start with a CTE with this.
10:16:48
I'll call this with title lower and then inside of the CTE, I would extract out the columns that I'd want, specifically a job title and then also job title, but in lower case, so we can do lower job title. Now, also for good practice, right, in order to make sure that we're providing the most succinct data possible to this, I would just throw in a trim function to remove any white space to hopefully speed up the computation if necessary. So I'm going to wrap inside of the lower.
10:17:20
I'm going to write job title in trim and we'll alias this job title clean. This is from job postings fact down here. I need to update this. We're no longer coming in our normal query from there. We're coming from title lower and I now need to update this because I use job title here and needs to be job title clean. So I'm going to copy this and then paste it into there. The last thing to do is we don't want to detect anymore of these uppercase data and analysts.
10:17:49
So I need to go through and change all these to just lower case all the way through. All right, it's updated. Let's now run this query and this ended up working in this one right here, right? We had this full time role data analysts with SQL New Jersey's locals only. We did match up on it and we're able to mark it as that analyst. So these text functions can come in handy, especially for bucketing and categorizing data.
10:18:16
All right, last type of functions to discuss are null functions and we're going to be going over two of the most common that I find that I'm using for null functions and that's null if and coalesce. Now, this category of null functions I do want to give a caveat of is under the bucket, this duck DB gives it utility functions because these functions, which coalesce and null fall into below are difficult to categorize into specific function types and are broadly useful.
10:18:44
Anyway, starting with the first one in its null if and this takes two arguments, expression one, expression two, which could be values or in our case, that we'll get to columns. This returns null if the values are equal. So in the case of putting in 10 or 10 gives null in the case of 10 or 20, it gives 10. So let's go ahead and test it out inside of here. I'll put 10 and 10 and then run it. We can see that they are equal. So therefore it gives null. We make it not equal.
10:19:13
They provide the value with that value being that first expression. And it does say expression because you could technically put something like a mathematical or something in here. And then that case is going to evaluate and still give us that 10. So what's the use case for this null if function? Well, let's take a look at our first, well, let's just take a look at this data right here. We're querying our salary, your average column and our salary, our average column
10:19:40
from job postings fact where both of these are not null. Basically, I don't want to have all those null values pop up. Anyway, here it is below. And what we want to do is say there was a case that we had zeros in these values, meaning let's say that we had some data in here, such a salary or average, and they listed the salary as zero. Well, what we could do is is we could do null if and then provide that column of salary or average along with an zero and then close
10:20:17
it and then we'll also do the same for the salary, our average column, adding that null if checking if it has zero. And in any case, if there were, I executed this below, in any case that it has zero in this column, it's going to replace it with null. And this is extremely valuable if, say, we were getting, you know, like the median of these columns that we wanted to make sure that all the zeros were removed from this, I can do the median from this column as well.
10:20:44
In fact, I will, so I don't have to do a group by. And then when I run it, I know that it's using values without zeros in it. Now, disclaimer on this example is that I know I rewrote this query to look at the lowest salary, our average. In this case, the lowest is eight. If I reword this to see the lowest salary, your average. Whenever I run this, the lowest is 15,000. So there's no zero values in our data because,
10:21:11
you know, myself, but I cleaned up the data to remove any of those zeros. But if you find yourself in cases with this, this is the great use of the function for that. Next up is coalesce. And this returns the first non-null value from, well, this is a list of expressions. So in that first case, if we're providing it a null, A or B, he's going to get a return first. In the second case, 100 will get returned because null, null is, well, nulls, and then it'll be the 100.
10:21:41
So trying this out, I'm going to just put coalesce and then 0, 1, 2. Running this, we would expect to write the first non-null value. So it'd be 0, changing the 0 to a null. I would expect now the 1 and 1 it is. And then finally, changing that 1 to a null. I would expect just the 2 to return because the first non-null value. Now, how is this typically used in the real world? Well, we've seen this before, right? Here, I'm once again querying that salary, your average and the salary,
10:22:11
salary, our average from job posting facts. Oh, and I added in a where clause to basically show me the data so I can see the salary, our average and the salary, your average. And when it's not null. Anyway, let's go ahead and run this. What we can do is we can use coalesce and we could use, maybe, salary or average as that first expression to check and then salary, our average as the second expression to check. And while we want to do this, well, if you remember previously, we did a,
10:22:38
we were trying to convert to a standardized salary. Remember we wanted that salary, our average converted to a yearly salary. So this is a great use case for this. So in here, I'm going to put in that coalesce. I'll let salary your average first and then salary, our average next and just multiply this times 20 or 20 80 or two thousand and 80 hours, which are in a year. And then, right? Normally it lists a third expression.
10:23:06
If it's not, if you wanted to place it with something like you could do, we could do missing salary, but we want to do text because this is an integer call or this is a double column. So we wouldn't want to do that. We also wouldn't or it's not necessary in our case to just list null because null is default already. So we're not even going to list a third expression in this case. So whenever I go to run this, I get an error because I have a period right here and not a comma.
10:23:34
So I change that after salary, our average, let's run it again. OK, pretty cool. Right now it's going through and it's doing this operation by getting the first nine null value from these columns and then doing either in the case of the 20 multiplying at times 20 80 or in the case of the 110,000 just using that value first. Now, if you recall back to the case expression lesson, we ended up drafting or providing this for the final example.
10:24:09
And what we did was in the CTE, we used a case expression to identify whether it's not and then in that case, make this into a single column with now using that formula for salary, our average to multiply times 20 80. Anyway, basically, I'm trying to say this entire case expression is now a coalesce function and we did this CTE because we didn't want to have to write this over again, inside of this other case expression down below here, where we then went
10:24:44
through and bucketed the data into missing low, mid or high, but scrolling up, we can see that this OK, so previously we were using this entire case expression. Now this has all been limited to this single function right here of coalesce. So I can just go ahead and copy this and then I'm going to remove this entire CTE because we're no longer going to need it. And instead, I'm going to pray standardized salary with this. I'm also going to replace it here, here and here.
10:25:14
I'll give this the alias though of standardized salary so we can use it inside the order by and then I just need to update the from to job postings fact. OK, I think I got everything we needed for this. Let's go ahead and run it. All right, not bad. It went through and did what we wanted to do of combining the standardized salary. We see that we did have one hourly right here and it converted it into that yearly salary and then we got the appropriate buckets for each.
10:25:41
And then we're necessary for the no values. It filled it in the bucket as missing. So this simpleized our query a lot further with this function and makes it a lot more useful. All right, so that is null and also text functions. We now have some practice problems for you to go through and get more familiar with these functions in the next lesson. We're going to be diving into some other functions, specifically window functions,
10:26:09
a really popular topic when it comes to using SQL for aggregation. All right, with that, I'll see you in the next one. All right, welcome to this lesson on window functions and window functions. Let you with names given because they let you look around like a window at other rows while you keep your current row intact. And we're going to get to a demo here shortly. And these functions are very important to data engineers because they allow us to
10:26:42
keep our data modeling intact while performing or adding in advanced analysis. Anyway, let me happen. Let's get into this. We're going to be looking at what window functions are, the different type of functions we can use with it and some real world use cases.
10:27:01
So let's go back to the analogy I was just given where window functions let you look around at other rows while keeping your current row intact. Previously, we learned about aggregate functions and that is using group by. But the results of this aggregate the results into a single row, typically, or multiple, depending on what you're grouping by window functions, though, keep this all intact and provide this aggregation at that row level.
10:27:31
And because of this, you can do a lot of powerful things with it. Let's go ahead and demonstrate this using a simple example where we're going to be forming aggregation only to count the rows and then doing a window function to count the rows open up terminal and connect to our mother duck database for data jobs. So the first thing I'm going to do is just a simple aggregation of count the rows. We've seen this before, right, we run a select and we'll run a count star to count
10:27:56
all the rows and this is going to be from the job postings fact table. Make sure to end in semicolon. All right, let's go ahead and run this as expected. We got back 1.62 million rows now. Let's use this with a window function where we preserve all the different rows. Now, for this, I want to I want to preserve all the different rows. In order to see all the rows, we're going to list the job ID and this is from job postings fact table.
10:28:24
Let's go ahead and just run this to see what's shown. So with this query, right, we're getting every single job ID. What I want now is to be able to next to on each of those rows provide that total count. Well, it's pretty simple. We'll enter this in. We're going to run a count and then star and then from there, this is the over keyword with an opening and closing parentheses. This keyword right here is the most important as it signals that this is a
10:28:53
window function and whatever you put in parentheses, I can put anything right now defines the window we want to look at right now. We didn't define anything, so we're going to look at the entire data set. So because that, let's go ahead and run this. And with this one, right, we have the query up at the top. We can see that for each job ID. We then have the count of 1.6 million. And this repeats all the way down. And so this allows us to get any number of different aggregations,
10:29:21
not just that count star and preserve that row level data. So why are window functions important for that engineers? Well, they let you add context without destroying all those roles, which is basically your job. You're typically going to be building data models and these pipelines need this little low level level data and window functions allow you to keep this along with adding additional insights. So let's break down this syntax and we're going to be using it where it's
10:29:53
primarily used within that select statement. The first portion is the window function, what we want to actually do. actually use as the function for this. Now we're gonna dive in just a second into this, but there's three major types of buckets I find for window functions, and that's aggregates, row and rank, and navigation. As far as the aggregate goes, you've seen a lot of aggregate we've done already whenever we did our group by lesson,
10:30:21
we're just doing it in the context of window functions. Anyway, more on this in bet. So after we define what is our window function and the function we're gonna use, we then start by creating our window function by using this over keyword and then in parentheses after this. And that makes it a window function that's all you need, that's the bare bones for it. The partition by is an optional next parameter that divides the rows into groups or partitions.
10:30:52
And then this function runs separately within each of those groups. For this, we're creating basically separate windows for each unique value. Next up is the order by, and this one is also optional, but often required, in that it defines the sequence of rows within each partition. This is especially important when we get into those ranking functions and running calculation that required to be in a certain order. Now more advanced window functions
10:31:23
may even use this additional addition inside of window functions where you can specify the rows between or the range, basically selecting what number of rows up or down around a row. Anyway, that's gonna get too complex. We're not getting that, let's be on the scope of this. Just know that it's available to modify window functions even further if you want to. So getting back to those three types of window functions. You're already, like I said,
10:31:50
pretty familiar with aggregate functions. We've run through all of these of average max men sum and count within a group by context and applying to window functions I feel will be pretty intuitive. Now other areas where it shine are things like row and rank so we can use window functions to give the rows row numbers or even rank values such as highest to lowest salary. And the third other type of function that's useful is navigation. In that, especially those lag and lead,
10:32:20
you can actually use window functions to pinpoint a row above or below it in order to maybe do some sort of comparative analysis to see how salary is changing over time. Anyway, getting ahead of myself let's actually get into demonstrating this. So let's get into breaking down partition by. We're gonna start with this example. Let's say that we wanted to not only keep the value of what is the salary hour average for a job postings
10:32:56
but we also want to look at what was the average of that for all job title shorts. Well, partition by is gonna help us solve this because it helps divide the rows into groups or partitions specifically here on the job title short column. And then the function will run separately with each women, each of those job title shorts. So what we can get at the results of this is we preserve our row level because we do window functions. And then it goes through and calculates that
10:33:32
in this case data analyst has an average hour salary of $50 an hour whereas data engineers have an average of around 100 and it preserves it at the row level. So let's do this. We're gonna go ahead and add a select. I'll add in the job ID column, job title short, salary hour average. And then we'll get into defining our window function. We're gonna be using the average function for this. Now we want to perform an average on a specific column so we can do that in this case.
10:34:01
So we add salary hour average into here. And then we go ahead and add an over and over defines the window function itself. So now it is a window function. Actually, we're gonna keep this over in here to demonstrate what would you would get in this case. And then from there we'll go and add enough from and the job postings fact table. Okay, let's go ahead and run this. I just ran this, took almost two seconds. Probably should put a limit in there so it's not causing to be so much.
10:34:31
But anyway, what we can see is we do at least have one value here. Well, what's happening though is all these values are output in the same thing of 47.39. Anyway, just to prove that this is the average over the entire data set. I'm gonna go ahead and just run a select segment on just the average, not using window function, the job postings fact. And as shown, it is that 47.39. So what that window function is doing, specifically with this over and then parentheses,
10:35:01
we're looking at the entire data set. And then from there, it's performing the aggregation on the appropriate column. So just over by itself is already powerful, but now let's actually get into getting that average salary but on a job title short basis. And we're gonna do this by adding in a partition by and specifying a job title short. All right, let's go ahead and run this now. All right, not bad. We're seeing that for the results for this,
10:35:32
for software engineers, their average is around $51.17. And then scrolling down, we can see that senior data scientist is different at $56.96. Anyway, I wanna actually see a little bit more to see the variation and check out this data set more. So I'm gonna add in a group by and then do random to randomize this data. And then I'll also add in a limit 10. All right, running this now. And I got an error because I don't know why I was doing group by with randomness.
10:36:05
Now what I wanna do, I wanna do order by random. So I'm gonna just change this group to order and then running this again. All right, sweet, this demonstrates it better. And we can actually see the difference between this where senior data engineers are at around $60 and data engineers are at $56. This makes sense. Now I'm gonna take this one step further. I'm actually gonna delete this query down here. We're not using this at all. Let's say we were also wanting to not only get
10:36:31
that salary by that job title short, but also by companies. So I could specify in here just to demonstrate it better, the company ID and then inside of the partition by add in that company ID. Now, whenever I run this, it's gonna partition by the job title short and company ID. And in that case, some of these companies don't have any salaries associated by it. So only this one here from company ID 634 has an associated salary. Anyway, I wanna just look at ones
10:37:04
that do have salaries with this. So I'm gonna add in a where salary average hour is not null. All right, with that added in, I'm gonna go ahead and now run this. All right, so pretty cool here. We can now see based on a job title short value and a company ID, what is that average salary? And then we could compare it to what is the salary of a posting. Very powerful analytics that we can provide are end users with all with just providing or with using a window function.
10:37:42
Let's now get into order buy. And we're gonna be looking at order buy just by itself without in addition to partition buy and then order buy. And then we'll move on to a more complex case combined with anyway. Order buy, I find most simply is best to explain how it's used with ranking functions. If you recall, there are three major types of window functions I've found of aggregates, row and rank, and then also navigation. We can just focus on a one example real quick
10:38:12
of row and rank. So let's say we have this scenario. We have our job postings here with job ID, job title short, and then the salary, our average. And we wanna rank these values in order of their salary. Where the highest salary as job 105 is $105. And that would be the highest ranking one of one. So in this case, we're gonna use order buy within our window function with a window function of rank in order to define the sequence of the rows within this particular partition.
10:38:49
In our case, the partition is the entire data set. And so for the final results for this, we'll have that rank our salary column and it will provide that numerical rank all the way through based on what it is on that salary, our average column. So for this, we're gonna go ahead and jump into VS code. I'm gonna copy our previous query that I had above and just gonna paste down here for what is to work with. And I'm just gonna preemptively change this
10:39:17
to salary, our average and descending order. 'Cause that's what we're gonna do. We're gonna be ranking this. So let's go into here. We'll keep all the things of job ID, job title short, company ID, we don't really need it. So I'm gonna delete it, keep that salary, our average. Then from there, we're gonna redo this window function. In this case, we're using the rank function and the rank function doesn't require any arguments inside the parentheses because it's really specific
10:39:41
to the window function that has everything in it to specify it. We then from there have our over and inside of here, we're not gonna do partition by just yet, right? We're gonna do just order by and we'll specify that salary, our average. We'll give this an alias of rank hourly salary. All right, so let's go ahead and run this bad boy. All right, so inside of here, we did that over by so we have the highest hour salaries up top. Notice here that our ranks though started 26059
10:40:11
and that's because right, we didn't specify to order by the salary, our average in descending order. So inside here, I'll update our query to do that in descending order, then we'll run it again. And then now we have it to where we have a rank based on the order of the salary, our average. Now one quick note, this order by portion right here, technically you could, I'm gonna comment it out real quick by pressing command slash and then running this without the order by in there.
10:40:44
In that case, it is still that window functions because as an order by in there, it is going to order it but you can't always rely on it, especially when you have more complicated window functions or maybe you have certain limits in there. So it's always good practice to still list that order by because we don't necessarily know how it's gonna output. So now let's put these two concepts together in order to look at how we can perform not only a partition by but also order by.
10:41:18
Now for this example, what we're gonna do is we're gonna get a running average of our hourly salary and that running average is gonna be based on that order by that job posted date. However, we're gonna break it into a subset or partition by based on that job title short column. Just a reminder, right? Partition by groups rows for these window calculations and then order by defines the sequence of rows within each of those partitions.
10:41:52
So in this case, what will happen is we'll have, yes, the jobs put in an order for the window function in order to have this average function run. Specifically, if we look at data analysts, that first entry is $60 an hour. That is the average for that case 'cause there's none before it. Now when we moved into the second entry for data analysts, we now have $40 introduced. So 60, the average of 60 and 40 is 50. And then moving on to the third data analyst, it's 50.
10:42:26
So therefore the average stays 50. Similarly, we can see for the data engineers, it starts at 95 with only one entry. So the average is 95. And then when we add that second in of 105, that average comes out to 100. So it's partitioned by the job title short and then ordered by the date in order to perform this running average. So let's go ahead and do this. I'm gonna take what we have up here and copy this original partition fly formula.
10:42:53
And we'll go ahead and paste it right into here. For this, I'm gonna change out that job ID with the job posted date column. We'll keep that job title short. We don't need the company ID for this. And we'll also keep that salary hour average. So we're still gonna be forming that hour average of the salary hour average. Over our partition by, we're gonna do the job title short. We're not gonna do company ID in this case, we're just gonna put it simple.
10:43:16
And then now we'll add in that order by. Specifying for that, we wanna do job posted date. We'll give this the alias of running average hourly by title. And we're getting this from the job postings fact table. We're gonna keep our where in here to filter by, to remove any non-null values. And then for the order by, we'll delete this random. And then I'm gonna do it first by the partition of job title short. And then next for that job posted date of the final order by.
10:43:47
All right, let's go ahead and run this. All right, and first up we have business analysts 'cause B is one of the first job titles. And we can see that they are in fact ordered in that job posted date column. Now going through this, just to double check, right? That first entry of 17, I would expect. Yeah, the average at that time is 17. Then whenever we get a second entry of 20, 17, the average of 17 to 20 is 18.5. Then when we get that third entry of 35, we now have three entries.
10:44:18
And so the average of all three of those is 24. And it goes on so on for this. Now if I wanted to, I could also add an end condition here where we look at just data engineer jobs. I specify the job title short equal to data engineer. Whenever I go ahead and run this, check that one. Similar thing here, starts at 64.5, that's the average. Then we add in 25.50. And then the average of those two are 45 and so on. Now I do also wanna demonstrate this
10:44:48
with our last example that we did with this order by by ranking our salary. So I'm gonna go ahead and just copy this and then paste it into here. I think a really neat thing would be to not only rank this, in this case, right? We were ranking it based on what was the our salary, but we also maybe wanna rank in different partitions based on that job title short. So we'll go ahead and add in a partition buying, specifying job title short. And then I do wanna order this correctly.
10:45:18
So I'm gonna go and ahead and press command slash to uncomment that and then add in that job title short and to comma. All right, let's go ahead and run this bad boy. And it did do it correctly. I can see that based on business analysts. I can see the what it is for one through 10, which is good, but I actually wanna see it from what is the highest salary hour average to see different job postings. So what I'm gonna do is I'm gonna change up
10:45:45
this order by right here and put salary hour average to sending order first and job title short second. Now, whenever we go to run this, we can now see data analyst that its first one is at 391 and then data scientist first one is at 250. And then correspondingly data analyst second one is at 242 and so on. So it's pretty cool now because now we have not only ranked it based on the hour salary, but also we've grouped them or partitioned by that job title short.
10:46:14
One last note, this order by inside of the window function is completely independent from this order by here at the end. So this window function will handle its calculation on its own and then after this order by that happens within the window function happens, then from there, this order by is then applied. And we know this based on what we learned in order of execution where step five has where our step or a select statement is run, which that is where our window function is.
10:46:49
So that happens first and then step six is our order by where it actually goes then and orders the columns. All right, so now we're gonna walk into well, aggregate functions, but we're actually gonna go through all three of these showing how or showing the most popular functions from each of these three different types of major window functions. Now, aggregate functions you've done before in earlier lessons. So I feel like you have a good understanding of this already.
10:47:19
So we're just gonna look at some simple examples real quick. For these examples, let's go back to that second-to-last query that we were working with, where we had both partitioned by and order by, and we were doing a running average of the hour salary. Now, I just wanna demonstrate a few of these. I wanna demonstrate it simply first. So we're going to, I'm gonna comment out the order by. And in this case, remember, if we did the average of this,
10:47:44
and this is filtered for data engineers, this provides us the average of all those hour salaries. And then partitioned by a job title short. Granted, we're only filtered to only one job title short, but they get the idea with this. Additionally, we can do things like finding the minimum salary. In this case, when I run it, we get the minimum of eight. I could also do the max. And in that case, I get that of 221, which is the highest salary. Or I could even do something like some
10:48:16
to sum up the entire column there. And the sum of all the data engineer's salaries are 337-981. That's hourly. Anyway, I could see this even more useful if you're doing some sort of running minimum, running maximum, or running count or something. So what we're gonna do is go back to this and remove that order by, or order by being commented out. So in this case, we'll just keep the sum in there so we can do a running sum of the hour salaries.
10:48:46
And we can see that as we go through for data engineers, that that hour salary gets accumulated. Similarly, I can change this to something like min. And for this, it will track what is the minimum as it goes through here until it gets replaced. So it was 64.5 and then it got to 25.5. And then it sticks with this until it gets to the next lowest of 24. And then it finally gets replaced again by 20. Just like min, we could also do max. And this would keep that max value on what it is.
10:49:16
Looks like it hit 70 and then stays with 70 for the remainder of the time there. So these aggregation functions that we've previously learned about can be even more powerful when you combine them with window functions and then can partition them or even order them by certain values to get even more specific results.
10:49:36
Next up, we're going to look at row and rank functions. Specifically, we're going to be focusing on those first three looking at rank, dunce rank, and then the row number. So for this first example, we're going to scroll on up and we're going to use that third to last example that we were demonstrating order by when we were using the rank function with it. So I'm going to go ahead and copy this and then go ahead and paste it into here.
10:50:02
All right, now previously, we were just using the rank function. I'm going to put back in this order by. Okay, so let's go ahead and run this showing that rank function where we were ranking the salary our average. Anyway, for this one, right, we weren't doing any partition buys and we were just going down and it was ranking it. Now, if you notice here, we get to this value for data scientists and senior data scientists where we have two values that are same with 200
10:50:28
and it says nine and nine. Actually, let's increase that limit a little bit to see what happens after that. So we'll change this limit 10 to a limit 20 and then rerun this. Oh my gosh, this is not what, all right. So you could see, right, we have a bunch of 200s and we have nine ranked all the way up to the 20 values. Let's go a little bit higher with the limit. Change it to 40 and that's not even enough. All right, so I ended up changing this to 140.
10:50:53
That took me a second to find it. Anyway, if we see, we get down to this value here into our values that are like 139 and 140 and we can see here that that's the rank, 139, 140 and it jumps right from nine to 139 to 140. So that's what's just happened with the rank function. So now I'm gonna go ahead and change this instead to change this to the dense rank. And whenever we run this, what we see is is that when it gets to that, well, 139th value previously,
10:51:24
it just goes from nine to 10. And in this case, the rank's rank after nine is not skipped like that last case. Personally, myself, I find myself using rank more than dense rank, but there may be situations where you'd have to use dense rank. So let's move on to row number. And this is very common I find whenever I want to develop maybe a new ID column for maybe a data set I'm building. Anyway, for this, what do I mean by this? Let's go to, let's just select star
10:51:59
from the entire job postings fact set. And for this, I'm gonna just limit this to 20 results. Oh, and I do want to actually order this by the job posted date. Okay, let's go ahead and run this. So these are ordered by that job posted date. And as you can see, even if I order this by the job ID, you'd notice the same thing, it starts at the job ID of 459.3. Let's say we wanted to give a new ID and start directly at one. Well, we could use the row number function for this.
10:52:35
So I'll keep all those columns on here. And then I'll add in row number, then we'll use over. And then from there, we're going to order by and we'll keep that same of job posted date. Now I'm gonna go ahead and run this. And what we get here is that row number based on it ordered by a job posted date. And it goes through and just starts at one and then goes through there. One note, these are out of order right here. And it probably has to do with that these are
10:53:05
probably the same date and time that they were listed. And anyway, it doesn't really affect it too much. This gets what accomplishes what I want to happen of starting at one. I've also seen this row number used as a ranking function because if you remember whenever we looked at rank and dense rank, I'll go ahead and just run that query again. Remember it assigned a specific or at least it had matching values no matter whether it's rank or dense rank
10:53:32
for those ninth place values, all of them were rank nine. So you could use row number in that case to assign unique numbers for the rank. And actually we can just demonstrate that real quick by changing this to row number and then running it. And then we can see from the results it goes one, two, three, four, five, six, seven, eight, nine. And then between that one and what was a 139 it started assigning individual numbers. Note that these aren't sorted correctly.
10:54:01
You would have to actually put this rank hourly salary into the order by to actually get it in the numbers that you've wanted in order. And so I changed that up here in the order by and we get those in the correct order. All right, last up type of functions we're gonna look at our navigation functions. For this we're just gonna focus on, well, one, well technically two. And then we're gonna look at lag and lead. So let's get into the scenario that we're gonna be using lag with.
10:54:32
And for this I'm gonna just go ahead and show you what the final table is gonna look like 'cause it'll make more sense of what or how actually we're gonna use lag. Now what I wanna do is I'm trying to analyze based on a company, how does their yearly salary change with every job posting and specifically ordered by or yeah, ordered by job posted date. So for example this, right? So company ID 4493, they have their first posting at 75,000 and then their second posting is at 150,000.
10:55:11
What we can do is we can use the lag function to get what was the previous value right here. So that's what I'm doing, I'm getting that 75,000 here. Now moving on to the next company of 4594. It has 90,000 for the first value, there's no previous value so it's null. And then when it goes to the next value, it then can put in say it does have a value above this of 90,000 and it puts it there. So this lag function is very powerful. It can allow us to compare
10:55:47
and maybe perform some time based comparison on how a company is raising or lowering salaries over time. So let's build this query. So in here, let's start adding our columns. In this we want the job ID, company ID, job title, job title short, job posted date and salary year average. We'll add in the Windows function a little bit. This is from the job postings fact table. And we want to do this only where the hour or the yearly salary is not null. So we'll add that in.
10:56:21
And then for this, I'm going to limit the results to, well, I know it needs to be 60, be able to see the different kinds of things we want to look at it. So let's now use that lag function to get the previous postings salary. So we'll put in here lag, then we'll put in over and then open the parentheses. Now how do we want to do this? Well, we want to do right partition by first. And like I mentioned, we want to look at, we're looking at companies specifically.
10:56:48
So we need to partition by that company ID. And we want to see how their postings are changing over time. So we'll order it by that job posted date. Now I do want it to be finally ordered whenever it shows it to me. So I'll put an order by at the end based on the company ID and that job posted date. Okay, let's go, well, let's give an alias real quick for this as, and we'll call this previous posting salary. Okay, let's go ahead and run this now.
10:57:16
And so let me, I got ahead of myself, lag needs one parameter. It got zero. So in this case, what are we trying to provide as the value that we want to look at previously? Well, we want to look at salary year average. So we're going to specify that inside of here. Now, let's go ahead and run this. And we get our results as we saw previously, right? We don't exceed the previous posting salary for this four, five, nine, three portion because, well, that's the first posting.
10:57:46
And then the second one, we do see the 75,000 for the first posting. And then it carries all the way in through for the next company. Then I can even scroll down to inspect further. When we go from company five, four, nine, four to five, four, nine, eight, we see for the first value, it's null. And then after that, it goes to 85,000 based on that previous salary. Anyway, that's cool. What was actually better than this is we want to see what is the change
10:58:13
with each of these postings. So what I can do here is I'm going to add another column and I'm going to take salary year average and I'm going to subtract what was that previous posting salary. So I'm just going to copy this all right here, paste it in and then change this alias right here to salary change. Okay, this is now, I feel even more informative of what happened here. And that now we can see that, okay, in that second salary of four, five, nine, three,
10:58:46
it increased by 75,000. Whereas for company four, five, nine, four, we can see the change that goes on here where it goes up 22,000, down 22,000, stays about the same and so on. Now that's lag and that we're looking at values from the previous column. We could also similarly do this to change this to lead and this would analyze instead of the next posting salary or the previous posting salary, this is going to analyze the next posting salary and then I change this lag also to lead
10:59:18
and then whenever we run this, it does just the opposite in that that first posting gets what the next salary's value is, so 150,000 and then we can provide that salary change. I find myself, especially for time-based analysis, mostly using lag and not lead, but I wanted to just demonstrate both because they're both so similar. All right, so that is window functions. You now have an understanding of how to go through and use things like partition by and order by it
10:59:48
to get even more in-depth analytical measures and provide it to your stakeholders. We have some practice problems for you to go through and then we'll move it into the next lesson after that on understanding how to use functions for nested data. All right, with that, see you in the next one. Welcome to this lesson on nested data. And nested data is also commonly called something like semi-structured data or even complex data types. Anyway, the gist of it is,
11:00:19
is this allows you to store multiple different values within a single column. So it's really powerful feature for you to know as a data engineer. In this lesson, we're gonna go over the five most common types of nested data. And then from there, we'll dive into two real world examples with our data sets, looking at arrays and also array of structs. Also real quick before we get into it, this is the last lesson before we get into our final project.
11:00:52
So super excited with this last portion we'll be covering on SQL. So you're probably like, Luke, what the heck is nested data? And well, here's an example of what nested data looks like inside of a table. Take for example, here this skills column. For this data engineer, there are multiple skills listed for its associated job. In this case, Python SQL and Excel. These are listed in what's known as an array. The other example I have here is on benefits.
11:01:27
And this is using a struct data type. And we can notice this because it has those curly brackets around it. But in here, we can specify three different attributes about this job, whether it's remote, whether it has healthcare included, or whether it has PTO. And then these values either true or false. But we'll come to find out that it's not just limited to true false values for this. Anyway, the main point of this is to showcase
11:01:53
that we can now go forth with putting in multiple values inside of a single column. So let's quickly break it down onto what these five different nested data types are. And then after we go through these five, we'll then go a little bit further, introing them, actually demoing how you can create them in yourself in VS Code. First up is an array. It's also commonly known as a list. In DuckDB, you can say anything you can do to arrays, you can do lists and vice versa.
11:02:25
Well, for the most part. Anyway, these are an ordered collection of the same type values index, meaning in this case, I'm demoing it with text values, but it could be number values, dates, or whatever as long as the same type. You know it's an array because it has square brackets around it. Next up is a struct. Also known in other databases or as records or rows, but most commonly known as a struct. This is a composite data type containing multiple name fields.
11:02:56
Now here in this demo, this we have it around in this curly brackets. And so we have skill and Python. Skill is the field name. And then Python is the value that we're trying to store within that field. Similarly, level is the field and advanced is what we're trying to store inside of level. In most databases, the data type specifically of Python and advanced has to be, what was the example here, has to be at the same type. So these are both probably var cars or text values.
11:03:32
And if I wanted to use numbers, both of them would also have to be numbers. We want to use dates, both would have to be dates. Next up is an array of structs. And quite simply, it is physically an array with structs inside of it. Of the data types we're gonna cover today, arrays and then also this array of structs are the most common I've found to be used in data engineering in order to store data. And that's specifically for storing nested data. All right, next up is a map.
11:04:02
Also commonly called an object in dictionary, but most commonly known as a map. And it's known as a map because it stores key and value pairs. So the keys are mapped to the values. Unlike the previous one we went over of array of structs, this is an unordered collection. So they're not maintained in a certain order. Anyway, this data type is, I find less common use than an array of structs. So we're only going to spend a little bit of time
11:04:35
just for you to understand the basics of this. Last up is JSON and this is a text based serialization format for nested data. Now the thing about JSON is, is it's not native to SQL, in that you'll find JSON out in the real world. If you work in web development, you'll see it all over the place. And JSON stands for JavaScript Object Notation, which JavaScript once again is well, plenty used in web development space. Anyway, the reason why I'm including here
11:05:07
in SQL is it's common for you to receive data, if you're accessing it via an API or something like that in a JSON format and then you have to unpack it, or maybe convert it to something like an array of structs, which we'll do. This does have a similar format in structure, whereas it does use the curly brackets and square brackets in order to contain data, but doesn't necessarily always follow the same rules. So now what I want to do, I want to jump into VS Code
11:05:37
and I want to walk through actually building out each one of these data types. First up is an array and a reminder, it's an ordered collection of the same type values and it's index being that we can access the value inside this array by a number that identifies where it is in this array. Gonna hand myself, let's jump into VS Code. So inside VS Code, I'm gonna go ahead and attach to our database and mother duck. So the first thing we do, demo those array,
11:06:10
I'm gonna create a select statement and to create the array, we're just gonna enter in the square brackets and then you're gonna enter in, we'll just enter in three values, one, two and three. I'll add a semicolon then, go ahead and run this. So it says underneath here, main list values, like I said in DuckDB, arrays or sort of synonymous with lists, but we can see this data type underneath here and it says in 32 and then square brackets, this shows that it's an array
11:06:41
with those square brackets next to it. Anyway, let's actually change these two skills instead. So I change this to Python, SQL and R. Whenever we run this, we can see that the data type changed to varcar with square brackets, I need this an array. And then I'll name, I'll also give this an alias of skills array. And now a lot more readable. So let's dive into creating an array, but by using a function and going to the documentation for DuckDB, we can see that all list functions
11:07:13
work with the array data type. And that's important because we're gonna have to use a list function for this, well, sort of. Anyway, I'm gonna navigate over here to aggregate functions and in here, there's a function with lists where you provide orgs or arguments. And it returns a list containing all the values of a column. This also has the alias conveniently of arrayAG. That's why actually we're gonna be using this case instead of using list, we're gonna use arrayAG.
11:07:44
So let's create columns full of data of our skills. So first, I'm gonna add Python and our column name is gonna be skill. Then from there, I'm gonna do an union all and then I'll use another select add in SQL. And then finally, one more union all to perform the last select of R. All right, I'm gonna go ahead and run this and just show what's going on here. And in here, I created a table and with the column name of skill and then it has those skills of Python, SQL, and R.
11:08:18
What we wanna do now is take that column and convert it into an array. So I'm gonna do is now convert this all into a CTA. So I'm gonna say with skills as then open parentheses, indent this all over, remove the semi colon, and then put the closing parentheses. We're then going to query this. And first, I'll just query skill from skills just to demonstrate that we did do this correctly so far. All right, we run it, run just fine, provides us the same table with Python, SQL, and R.
11:08:55
But now what I wanna do is I wanna use that function we just learned about of array ag. And then I'm gonna wrap that with skills and give this the alias skills array. All right, now whenever I run this bad boy, we get all of our skills in an array. We can tell that this because it has that type R car with the square brackets. Now, if you remember, we could also use list if you haven't seen this before with aliases. I do wanna demonstrate this real quick
11:09:25
in that instead of using array ag, I can just put in here list instead. And whenever I go ahead and run this, it's gonna provide exactly the same results, does exactly the same thing. But I'm gonna go ahead and change this back. Anyway, I'm gonna get to this last point here where it says this is an order collection of same type values that are indexed. How do we access this via an index? Well, we have an array now. I wanna access via an index. So I'm gonna wrap this also in a CT.
11:09:55
We'll call this one. Well, I'm gonna call it skills array. I'm gonna have to do some changing around here. We'll say as, we'll open that parentheses, take away the semicolon and those final parentheses and dent this over. And then I'll change this skills array to not be that as well. I'll just change it to skills. So now I'm going to select skills and that's gonna be from skills array. Let's go ahead and run this to make sure that I actually built this correctly.
11:10:20
All right, so it's working. It's still providing that array of skills where we have our SQL Python. Now let's go in and actually access them using the index. So we can do that by after our skills right here. What I'm gonna do is I'm gonna put a bracket and then specify what skill I wanna get. We'll put one. We'll say, hey, this is gonna be our first skill. So let's go ahead and run this and see what we get. So with this, our query returns SQL as the first skill.
11:10:54
But hold up real quick. Let's actually run this a few times. Selecting again, pressing shift enter. This time we get Python. Let's try it again. Now we get R. See, I'm running this a few times. We can see the skill is constantly changing. And you may be like, look, I thought you said that an array is an ordered collection of same type values. Keyword here being ordered. Well, if we go into the documentation on aggregation, let's look at this array ag function.
11:11:21
As you recall, right? It's under that list 'cause that's the alias. Anyway, it returns a list containing all the values of column. This function is affected by ordering. Whenever I click this, it specifies that in order to specify an order, we have to use, well, we have an aggregate function, your argument, and then from there you use an order by. So what's going on here is we're using this array ag function, which is an aggregation function. And yes, it's putting it into an array
11:11:49
that is technically ordered, but we didn't specify a consistent order to follow. That's why every time we run it, it runs different value, SQL R, Python. So what we can do is what the documentation said is we can add inside of here order by, specifying what we're gonna order by, what order by, well, there's only one thing to order by, the skill. And in that case, it's gonna do it in alphabetical order. Now, whenever we run this, every time we run it, it's gonna start with Python.
11:12:16
Additionally, let's modify this to add in the second skill and third skill. So I'll add that in. Now let's go ahead and run this bad boy. And this consistently will stick with Python R SQL, which is alphabetical order. Here I am running it over and over again. It's maintaining consistent. It's maintaining ordered. All right, next up is a struct. And this is a composite data type containing multiple named fill fields. So let's get into creating it. I'll create a select statement.
11:12:47
And then from there, we're gonna do squirrely or a squirrely, curly brackets. We'll provide the first field name of skill and then give it inside of it the value of Python. And then from there, we'll give the next field name of type and we'll classify Python as, well, programming. We'll give this the alias skill struct. We'll go ahead and run this. Inside of here, we can see that it is of the data type skill struct, where the skill name is of Varkar and the type is also of Varkar.
11:13:23
Type is a reserved keyword in SQL. So it puts it in double quotes to make sure that we're using it in the proper context. You can also see it's a keyword because it turns blue here when it should be white. Anyway, not really best practice for this, but just for example, purposes, we're gonna keep it as type. Anyway, I wanna do something similar that we did with our arrays. I wanna use a function in order to build this. For that, we're gonna navigate over the documentation
11:13:53
for struct functions and go to struct pack. This creates a structure containing the argument values. The entry will be the bound variable name. So in this, we'll specify as in parentheses, the name of the field, and then after a colon and an equal sign, you give it the value one for it. So let's do it. I'm gonna put in a select statement. Then from there, specify struct pack. Open our parentheses. I'm gonna break this down into lines to make it a little bit easier to read.
11:14:27
We'll specify the field name first of skill, then that colon and equal sign, and then the value we wanna put inside of it. And then we'll just go ahead and put the next one in. Specifying type for the field name and programming. We'll give this the alias of just S right now, 'cause we're gonna expand upon this in a little bit. Anyway, running this, we get underneath here that yes, it is of type struct, skill is varcar, type is also varcar, and then it displayed underneath here.
11:14:55
Now, similar to what we did with arrays, I wanna show now how you can access things like Python or programming inside of a struct. So for this, I'm gonna wrap this all in a CTE. So we'll say with skill struct, and we'll open our parentheses. I'll indent this all over, and then put that closing parentheses, and then from there, we'll select. Right now, we'll just do a select star, and that's from skill struct. All right, selecting it all in order to run it. And I got an error,
11:15:30
mainly because I forgot to define my CTE with that as keyword. All right, as is in there now, we'll go ahead and run this now. Okay, so we're getting back everything that we saw before, but now we wanna go in and access values like Python and programming. What we can do is we can do this with dot notation. So specifically, we saw from that table, let me open back up, that the column name is S, and then skill is the field name. So what we do is we would do S dot skill,
11:16:05
and then we can also do something like S dot type. So now whenever I go ahead and run this, I get basically unpacking that struct by having Python, and then programming under what it is. Now, based on you seeing it like this, I do wanna take this a little bit further, in that right now, you could imagine we could have a table with multiple skills and multiple types. So let's go ahead and create this. So as a starting point, I'm gonna scroll up though,
11:16:35
and I'm gonna select this portion where we created this table with Python SQL and R. I'm gonna go ahead and copy it, and then come down here underneath and paste it, and then you shift tab to move it back in. So right now we have Python SQL and R characterized under skill, what I can do now is add in something like programming as type. For SQL, I'll specify its query language, and for R, we'll make it programming. All right, now going ahead and running this,
11:17:09
now adding in a semicolon, we can see now that we have a table. So now let's get this table into a struct. So what I'll do first is I'm gonna put this all into a CTE, we'll call this with skill table as, and then open up parentheses, close it up. From there, we will select, and we'll use that struct-packed formula again, and similarly, we're gonna do that same thing where we provided the field name, skill, and then the alias up here of, it's well, it's best confusing.
11:17:45
We're gonna change this to skills, 'cause this is a skills column, and we'll call this types column. Not only that's necessarily proper English, but we're gonna go with it, make sure we can distinguish between the two. Anyway, this up here is skills, and we would specify it here as skills. And then we want the field name of type, and we want to be the values of types. And this is all from our skill table. All right, let's go ahead and run this. I forgot a semicolon,
11:18:16
and I have a syntax there at or near type. When I think I did, yep, I did it, I forgot to put a comma after this. All right, let's try to run this again. All right, pretty sweet. We see in here, we have the data type of struct, with skill var car, type var car. And then we have each one of those on their own separate row. All right, next up is an array of structs. And this is an ordered collection of structure vectors, or better said, this is an array of structs.
11:18:50
So let's start by creating this one. We'll create a select statement, and then square brackets. That's going to start our array, and now we need to insert our structs in here. So for that, we'll insert our curly brackets. So let's find that field name of skill, and then the value of Python. We'll also do the field name of type, and the value of programming. We want to add another struct into here, so I'll put a comma, and then go down to the next line.
11:19:19
And so I'll put in SQL as a skill, and the type is query language, we're just gonna do two. I'll get this alias skills array of structs. Let's go ahead and enter this, putting on a semicolon and running it. I have a typo, I used a period up here for some reason. This should have been a comma. All right, so let's try to run this again. All right, so now we have the value inside of here, and we can see from the data type, it's a struct. And then the skill, and then the type of our car,
11:19:47
and for the season, and it's right next to this, of the square brackets, symbolizing it's an array. All of this struct is inside of an array. Anyway, let's create this using formulas. You know, like I like doing that. What we're gonna do is we're gonna build on that second, or that last example we were just working on. I'm gonna go ahead and just copy this, and then paste this down here. If we recall whenever we run this, this has a multiple rows where each row is its own array.
11:20:16
Start at the back, I mean, that is its own struct. So we want to make an array of all three of this structs. So this is actually pretty simple, in that we just need to wrap this entire portion right here, in our array function. So I'll go ahead and type out array ag, open up parentheses. I'm gonna just format this by indenting it over, and then put that closing parentheses for array ag. Now, whenever I go to run this, we can see that we get all the values within one row,
11:20:51
and that it's a struct with the skill and type, and this is all within that array, so it's an array of struct. Now, we're gonna take this one step further. How do we access these items in here? How do you access things like R programming, or query language? Well, we'll put this all into a CTE, we'll call it skills array struct, and then we'll give it an as, and then open parentheses, and close parentheses at the end of here. Then from there, we're going to select,
11:21:22
we're gonna do select star from right now, to make sure we're built this correctly, and that's from skills array struct. All right, and they have a semicolon right here, caught that, gonna delete it. Now, let's go ahead and run this bad boy. Okay, we're getting the exact same results, right? It's all on one line, and let's now go and access it. Well, this is all, first of all, this is all an array. Remember to get access inside of an array, we need to use that number,
11:21:50
or that index notation to access the one, two, and three item. So, I'm realizing I didn't give this an alias up here for this array of structs that we created, so I'm just gonna call this array struct. And then here, we're gonna do array struct, and we'll just access that first item by doing one. Let's go ahead and run this, and we can see we get that first item. It's a skill of R and type of programming. I can also do it, or add to this, where we access the second and third item,
11:22:20
just to show how we can break it out, and here we can see how we've broken it out with that first item, second item, and then also I'll have that third item. Okay, how do we access now inside of here, something like SQL or query language? We remember we're gonna use the dot notation for that, after the square bracket notation for inside the array, now we're accessing the inside the struct, we can access something like the skill, we'll access the type for this one,
11:22:46
and we'll keep that last one just number three. All right, let's go ahead and run this. All right, sweet, so we can see that we got the skill back from that first item, which is R, the type back, which is query language for that of SQL, and then this one just maintains as the struct of array. Sorry, just maintains as the struct. (gentle music) Next up is map, and as I mentioned before, map is also known as things like object or dictionaries,
11:23:13
but it's not as popular as things like arrays or array of structs. So we're just gonna go over this briefly so that you understand what it does, but unlike the other ones we went over, I don't really expect you to have it memorize on what functions you can use for this. Now in order to create a map, actually we have to use a special keyword of map in order to create it. You can't just use that square bracket or curly bracket notation. And DuckDB has a few different ones for this
11:23:45
on how you can create a map. We're just gonna go with the simplest case. So for this, we're gonna do select and then specify map, and then from there, insert curly brackets. Now inside of here, we're gonna specify that key first, previously we were talking about it was a field name, but now we're using something called a key, and it's a little bit different in that we can't just type in skill by itself. We have to actually classify this as a text value
11:24:14
by putting single quotes around it. And then from there, we can add in the skill of Python. We just have one key value pair in this map to start. I'll go ahead and run this. And we can see we have in here that it's of the map data type with varcar, varcar. And then there's the key is skill and it's map to Python. If I were to run this without the single quotes on it and try to get it to run, I'm gonna get this right here, this binder reference column skill was not found
11:24:45
because the from clause missing. Mainly the problem is is skill has to be classified as a text value and the key has to. These text values will always be the key. Anyway, I can go ahead and add other things to this map as well, such as type of programming. Now, we can go ahead and run this bad boy. And we can see now that skill equals Python type equals the programming. Now it's important to understand then how we built this in this structure.
11:25:14
You can't, I can't repeat this skill again. For example, for where to add in, I just, I'm gonna copy and paste it, a skill let's say I wanted to add of something like SQL. Whenever I try to run this, I'm gonna get this where map keys must be unique. So this map that we've just built has to have unique keys of skill and type. Now, how do we access the values inside of here? Well, we'll create this as a CTE, pulling this skill map, then I'll bring up parentheses,
11:25:45
putting this all inside of here. I'll get rid of the semicolon and give this the alias of skill type. Go ahead and close parentheses. And then now inside of here, we're just gonna do a select start to make sure this is working correctly. This is from our skill map. So running this and adding in a semicolon, we get our values just like we saw before. Now, how do we access them like Python or programming? Well, we're gonna specify in here, we'll specify first that skill type.
11:26:13
That was the same as select star previously. We're gonna do very similar, if you've worked with dictionaries and pythons before, you would use a square bracket. And then inside of here, you specify the key you wanna access. So we're gonna access the skill key whenever we go ahead and run this and actually insert in a semicolon, we can see we get the skill of Python. Similarly, I can go in here and add that skill type and add in a type for this one.
11:26:44
All right, let's go ahead and run this. And then we get both back Python and programming, accessing both values within our map. Now, real quick, I wanna talk about the differences between map versus array of structs because I feel like they're both very similar in that map has that key value pairs and array of structs has that field names and then you can nest it down further. Well, both of these have methods to access the data inside of it.
11:27:16
We're gonna get to using a nest on array of structs here in a little bit, but we demonstrate with map how you can use the key to access inside of a map. Now, other differences include ordering inside of it that array of structs, there is an order once you set it inside of there. Yes, whenever we create that Python R and SQL example, it switched up, but it's going to stay like that. Whereas a map, it can change. It doesn't matter. There's no way to maintain an order inside of there.
11:27:48
Array of structs are great for things like analysis and also maintaining relationships as you can build out a really solid structure. Maps, on the other hand, are great at storing some sort of metadata or doing look-ups with it as it has a key and you can plug that key in and get it back. So it's really good at those type of values where you wanna look up a certain value. Now, inherently with how these are built, array of structs are naturally good at joins and filters,
11:28:20
whereas maps are not. And ultimately, why we're gonna be using array of structs is it is much stronger for analytical use cases. And so that's why it's come more common in data engineering world to be used because you're build stuff for data analysts. Map, on the other hand, much more limited in that it's not as good for analytical use, but it does have some other positive benefits that we discussed. Last up is JSON or JavaScript object notation.
11:28:52
And this is a text-based serialization format for nested data. It does have a very similar structure to array and also structs in how you can actually build things out. However, you can get pretty complex with what you receive back in JSON. Anyway, the main point of this and understanding is you're not going to build pipelines to put data into JSON. Instead, you're likely going to be receiving data in this JSON format, and you'll need to be able to deal with it,
11:29:24
to parse it out and clean it up. All right, so let's go ahead and create our first JSON data. In this, this is text-based. So it's gonna be all within single quotes. For this, we're gonna maintain simplicity and just create a, if you will, an array inside of JSON. Well, first, we're specifying skill, and any text value needs gonna need to be in quotes here. And for the first skill, we'll specify that is Python. For that skill, Python,
11:29:54
we're gonna have then have the type, which is programming. Now, we're gonna take all this, and we're going to cast it to JSON and we'll give this alias as skill JSON. Alright, let's go ahead and run this and we got an error because I forgot to put double quotes around this programming right here. We have a text. Alright, let's try to run this again. Okay, and we can see we have it formatted in the format of JSON. You could also see this as instead of doing this cast formula to cast it
11:30:29
as JSON. You could see the function of two JSON and then wrapping this all inside of parentheses and then running this, we're still getting that JSON. However, I'm not as big as a fan of that, especially in DuckDB because well, it has these escape characters in here because it got converted back to string, if you will. So I'm going to just change this back to casting this as JSON as much more readable. So move forward the example with this, I care less about using a function to extract
11:31:04
values out of here, if you will, but instead of maybe I want to now transfer this into another data type. So let's start by making this into a CTA, we'll call this raw skill JSON, then we'll wrap all of this in parentheses, removing this semi colon at the end. And for this, we're going to select, you know, I like to just put the value in here for the time being to make sure we built this correctly. And that's from raw skill JSON. So I was a little French. Alright, let's go ahead
11:31:39
and run this. Yep, still doing the same thing that we were doing before. So now we're going to be putting this into a struct. Now this is going to get a little complex with the formulas we're going to use here. But I don't want you to get too bugged around it. I mainly want to wrap your head around that this is possible. So first we're going to use that struct pack formula. I'm going to go ahead and remove skill JSON. We're not going to use that. If you remember, you list the field
11:32:08
name first, so it will be skill. Now for this, we need a special formula. So navigating over DuckDB for JSON processing functions, we're going to be using this JSON extract, extract string. It extracts our car strings from JSON at the given path. So you provide it to arguments, you provide the JSON, and then the path. So let's go ahead and use that function, we'll specify JSON extract string. We need to first by specifying the JSON. So we'll list skill JSON in there. And then the
11:32:44
path, what do we mean by that? Well, what we want to do is we want to access the path to skill right here. This is special JSON notation, in that we're going to specify this dollar sign or this currency symbol, it means start at the root of the JSON document itself, and then dot notation to access skill. So we're going to go to the key name skill. All right, then I'm going to close parentheses, let's just go ahead and run this to show that we got that first skill of Python out of there.
11:33:15
And we did we have it right here, the struct of skill Python. So we have this first part of this skill Python, let's now get the type programming as well. So we're going to insert here a comma, going down, put in type, and then use that JSON extract extract string function for the skill of JSON. And then using that currency symbol to start at the root of the JSON document, specifying the type. All right, let's go ahead and run this now. And bam, we've now pre or rebuilt the JSON that
11:33:48
we had up here, right? We've now rebuilt that into an appropriate format that can now be used in SQL. Now I do want to show just quick, we're not going to go over in depth. But imagine you had a JSON value that was structured similar to an array of structs. So you have multiple different structs inside of this similar array, but it is a JSON value. Well, if you remember from our previous from that we were just doing, we did that struct packed unpack that skill and type into a struct,
11:34:20
and we can wrap that into an array. And so then running this, we can see that we get this now in that struct array format that we want, or better said, sorry, array of structs like you can get that backwards. But point that is mainly to just showcase, there are methods and functions possible for you to get your JSON data into a more usable format in SQL that's much more queryable. This text, this JSON data, which is basically in almost a text format is not going to
11:34:51
be great for an analytical use case is not what we want to keep it in. All right, so that wraps up the five nested data types. We're now going to go into two real world examples, working with our data set our job postings fact table along with skills, to make them into arrays. And then from there, even go further and use it and combine it into a array of structs. Let's down get into our first of two final examples where we're first going to work with arrays. In this example, we're
11:35:24
going to be putting our skills from our skills job dim table, or actually from our skills dim table into an array. So here's our problem statement. We want to build a flat skill table for co workers to access job titles, salary info, and skills, all in one table. As a quick refresher, in order to get those skills, we're going to not only have to use the job postings fact table, we'll have to connect to the skills job dim table, and then to the skills dim table finally, to get those skills.
11:35:58
So let's get building this query. For this, I'm going to select, and then I'm going to leave these columns blank for the time being as I want to put in what tables we want to pull from first. We want to come from the job postings fact table as jpf, then we want to retain all of those jobs in the job postings fact table. So we're going to use a left join to connect that still skills job dim table. And we'll be connecting on the job ID from both of these tables. Now we need to join
11:36:26
in one more table after this, we're going to be doing a left join for this as well. And this is to the final skills dim table, which we're going to alias as SD. For this, we're going to connect on the skill ID from skills job dim table and the skills dim table. All right, so now we have our tables and what aliases we're using for this, I'll add in the job ID. Job tile short, salary or average. And then the skills themselves and the skills dim table. Now we're going to
11:36:56
eventually put the skills into an array, but I want to show you first what this actually looks like with what table we're making right now. So this table took about six seconds to load, a little bit of time, probably put a limit statement on it. Anyway, what we can see is in the case of like job ID 54 or 4594, which repeats multiple times, it's because it has the skills of SQL, Python and R. But we don't want for this final table for these skills to be duplicated, because we just
11:37:26
are not duplicated or broken out amongst multiple lines. We just want to make all the skills within one array. So in order to do this, we're going to use array ag, as we've seen before. And we're going to alias this as skills array. Now, if I go ahead and try to run this, well, I'm going to go ahead and try to run this to show you whatever we're going to get, we're going to get this a binder error in that column job ID must appear in the group by clause or must be part of an
11:37:57
aggregate function. Well, the issue is this, right? Array ag, ag means aggregation, it's performing an aggregation as it does this. So all these columns need to be listed in the group by. So I'll add a group by reinduct DB. So I'm going to just go ahead and just add all another databases, you have to have to have type up all those different columns. So now let's go ahead and run this. And I forgot
11:38:22
to remove that other semi colon was right here running again. This table took actually a little bit less time, only 3.75 seconds. And we can see that these skills are put within array for all of these different ones. And scrolling down the bottom, it retains all 1.62 million rows of those job postings. So what I want to do now is let's put this into a temporary table. And we're going to then shift roles in how you could actually go about analyzing those skills that are in an array.
11:38:58
So first we're going to do that by doing a create, I'm gonna do or a replace to make this item important. And then we'll put temp table, we'll call this job skills array. And we'll then do ass. Okay, going ahead and running this, we built the table in about three and a half seconds. All right, so let's shift gears. And now we're going to do from the perspective of data analyst, analyze the median salary per skill using that table that temporary table we just made. Now, right,
11:39:29
temporary table understanding only exist our environment, we're just doing this for demonstration purposes, you would most likely ship this up to your own data warehouse where you'd want it. I'm just gonna do temporary temple temporary table to make it easy. Well, the first thing I want to do is I want to demonstrate how we can actually extract out these skills and get them row by row so that way I could do an analysis like this to analyze the median salary. In order to do that
11:39:56
or demonstrate that, we're going to do a simple example first. I'm going to come up to the top where we were doing arrays initially. And I'm going to copy this right here. And this is where we were pulling out the different parts of an array and manipulating with it. So coming down to where we are at the bottom, I'll paste it in. And what I'm gonna do is I'm gonna delete all this portion up to skills to just show what the output is currently for this query. All right, so this
11:40:24
just has SQL Python and R in an array. How do we un nest these values? Well, for this, that was a little bit too much of a hint. We have un nesting, which going into documentation, we can see that we have an example here for un nesting a list or in our case, an array. Remember, list and arrays are sort of synonymous inductive, but arrays are generally more accepted across most databases. And we have a nest for that. And then we also have you can do a nesting on structs. So super
11:40:56
powerful function. So let's test it out on our example. All we're going to do is previously, we had skills in here, I'm going to run the a nest function on this. And then let's go ahead and run it and brain bam, it breaks out those skills on individual rows. So that's what we want to do. And specifically, that's what we want to do with this job skills array table that we just built. So we don't need this example anymore. I'm going to go ahead and delete it. For this,
11:41:24
we're going to go ahead and select the columns that we want, which are job ID, job title short, and salary or average. And then we're going to run the nest function on that skills array, and we'll give it the alias of skill. And this is all from job skills array. I'll do a good practice finally, and actually put a limit statement in here. So we'll run this. And inside of here, we can see how it goes through. And now it, well, we have duplicate job posts as expected,
11:41:52
because it's now breaking out those skills into individual rows. So this is exactly how we want our data. But remember, we have one more step we want to do, we're going to do from the perspective that analysts analyze the median salary per skill. So what I'm going to do is I'm going to put this all into a CTE, we'll say with flat skills as, and then have that CTE within there, we want to select from it that skill column, and then also the median of salary or average,
11:42:26
we'll give this the alias median salary. This is all going to be coming from our CTE of flat skills. And we want to order this by median salary in descending order. Also, I just forgot, remember, we're doing aggregation. So we need to do a group by on that skill column. All right, let's go ahead and first add a semicolon. And let's go ahead and run this and have an error, because I forgot to get rid of one of the semicolons here, specifically that and the CTE. It's like
11:42:53
I've done this multiple times wrong within this course. Also, I forgot to put a comma at the end of skill. Let's try this one more time. So we get this back, and you're probably like, look, what the heck's going on? You don't have any median salary, Valerie's values. I'm going to look up here and see what I did. Mainly, whenever I use the CT, I left that limit 20 in there. And that was limiting what was in our CT, we want to look at all jobs. So I'm going to go ahead and move that,
11:43:22
run that again. All right, so now with our query, we can then see the median salary based on these skills. Right now we have fedora, mongo and Debian up at the top spots. All right, so not bad, we just demonstrated how we could put all of the skills within a single table, like we did up here, using that array aggregation function. And then with it in this manner, the data analyst or whoever you're downstream customer is has the option to then unness these values, pretty simply,
11:43:55
if I might add, in order to do deeper insights on the skills, if they need to. All right, so let's get into our final final example. In this, we're going to be building a similar flat table like we did previously. But in this case, we're going to be using an array of structs. So for this, we're going to be building a flat skill and type table for co-workers to access job title salary and for skills and type one table. If you don't remember, we're going to just query
11:44:25
the skills dim table real quick. And in it, we can see that it has not only skills, but what type of skill it is, the goal of this is we're going to use an array of structs to have something like the type as the field, and then SQL or the skill as the value in that field. So you know, I don't like starting from scratch. So I'm going to come up here to where we created our previous job skills array table, and we'll go ahead and copy it. And then we'll paste it right on top of
11:44:56
this query, because we don't need this anymore. For the time being, we're not going to create this, because I'm going to be testing building out this query. But this has the majority of what we need, right? It has the job ID, job title, short, cellular average. Only the skills, we do need to add the type, and we need to make it into an array of structs. So scrolling up to our array of structs example on the intro, when we're going through this, I just want to recall your memory and how
11:45:24
we can create that. What would you do is in their case, we're going to create or use that array ag function, put it inside of our struct pack function, because it's a ray of struct. And then in here, we're going to specify, we'll specify the type first, and then the skill after this. So let's do that with ours, I'm gonna go ahead and remove this alias right here, and then delete this out. We're going to first add in that struct pack function. And then inside of here, we're going to
11:45:53
first testify this skill type, and assign it the type from our skills dim table. So when you give it the appropriate alias before that, we'll also do the skill name. And this is also from the skills dim table, we'll alias this as skills type. Let's go ahead and run this to see if we get what we want. And this took about six seconds to run, and we can see inside of this, that it did go through and
11:46:21
successfully create that array of structs, which you can do it by inspecting that data type it has here. But also we can just look at it and see that it is an array of structs with a skill type programming skill name, in this case, are all right, so not bad. So now some of the last time, I'm going to go ahead and uncomment this temporary table. I'm also going to rename it to job skills array struct. And this has everything I need it, go ahead and run it and build this temporary table.
11:46:49
That took about six seconds to build. So now let's getting to actually un nesting it, and seeing how we can look at this data, we're going to be doing this, analyzing it from the perspective of analysts to analyze the median salary per type of skill. So we're going to be able to use that type field name to actually analyze it, because we're going to invest it. So the first thing I want to do is just to see how to un nest this all. So we're going to list all
11:47:14
the different columns we need such as job ID, job title short, salad your average, and then skills type, which is that array of structs. All right, so we know, well, already we can use a nest on this, so we're going to go ahead and just put a nest illness. And this is all going to be coming from that Jill Scott job skills array struct temporary table that we just created. Okay, let's take a look
11:47:36
at this to see what actually we can output it out so far. All right, so with this, what we see is, yes, the jobs are getting broken down now out, because now I can see that the software engineer job has skills of go, JavaScript, TypeScript, HTML, and so on. So we've at least un nested that portion. But how are we going to now access the values inside the struct? Well, if you remember, we're going to use that dot notation. So we'll do that skill type here. And we'll give this the
11:48:08
alias skill type. Similarly below this, we need to actually get the skills out. So we'll do the dot notation on this on nest, specifying the skill name. All right, let's go ahead and run this. All right, sweet. This now un nested this, and we can now see on each of the different rows, the different skills types along with the skills name. So let's now take this further right, we want to get the median salary per skill type. For this person portion, I'm going to just
11:48:38
come up to the top and steal the query that we did previously, and paste it in below. This was the query that we use to analyze the median salary per the skills. What I'm going to do is take this whole portion that we just analyzed and found was correct here. And I'm going to paste it inside of the CTE and dent it over and remove that semicolon. Okay, in this case, right, we're trying to do the skill type. So I'm going to change the skill into skill type. And we also need to change what
11:49:10
we're grouping by of skill type. All right, cross our fingers. Let's hope this works. And it does. We can see from this that I probably should have ordered this, but it's only 11 values. So things like libraries, specifically like Python libraries or other languages libraries has the highest salary at 140,000. But we can see the things like programming is strong in the middle at 125,000. So not too bad that we went from this table that we could provide our stakeholders
11:49:41
with that had went out of a single row based on job postings, and then have that skill and type inside of it. But then again, then can go forward and still analyze attributes about the skills. All right, it's your turn to give it a try. We have some practice problems for you to go through and now get more familiar with those different type of nested data types. And the next lesson, we're going to be jumping into our final project specifically looking at get commands. And so we're
11:50:11
going to be diving deep into that and understand what we're going to be building on the project. All right, with that, I'll see you there. All right, welcome to the second part on Git and GitHub. This one is going to be focused primarily on Git and specifically on branching. Now branching, it's not only going to be imperative to us getting through this second project, but also just important for us to know because Git branching lets you create isolated lines of
11:50:40
work so you can build, test, and even fix pipelines without breaking what's in your production line. Data engineers rely on this on a daily basis in order to safely evolve schemas, ETL logic, and infrastructure while multiple changes in teammates move in parallel. Now for this branching, we're going to cover three main commands. The first one is Git branch, Lisha branch tells you also how to create a new branch, Git switch, which allows you to switch between branches.
11:51:13
This is previously known as Git checkout, and then finally Git merge in order to, well, merge your branches. So for all this, we're going to walk through some simple examples and then also discuss how we're going to be integrating it into our second project. Before we get into what is branching, I do want to do a quick recap of the Git commands that we learned last time. Previously, we will learn that there could be three areas where you keep your files.
11:51:41
You're working directory where you actually work in, type your code, your staging area where you're putting your changes in there before you get to committing it, and then there your local repo once you actually commit it. We learned commands like git add to get your working directory contents into your staging area, and then git commit to commit what you've done to that local repo. Now from there, we explored GitHub where that is our remote repo,
11:52:08
and we can use commands like git push to, well, push those that local repo up to our remote repo. And then say there are changes on the remote repo from a partner or another teammate, we can use git fetch to get them into our local repo, and then git merge to get them in our working directory. And conveniently, we learned that instead of having to use git fetch and then git merge, you can just use git pull, which is an all in one command to get it from your remote repo
11:52:37
into your working directory. Now for that previously, we were assuming you were working in all of a single branch, specifically the main branch, or master as it formerly was, we renamed it as main. And this branching in git is crucial for you as a data engineer in order to collaborate with others. This allows an independent line of development that allows you to work on features, developments, or even bug fixes without affecting your main code base. These are like isolated workspaces
11:53:06
that allow you to safely test things out and experiment and make sure your stuff works before you then push it into production. We're going to implement the use of branches in project two as we go through it. And this will get us more familiar of how it's actually used. But we'll get on to more into how we're going to be handling this in the next video on the project to intro. So let's walk through a simple example of what you'd see typically with having a use case
11:53:35
for different branches. As you saw previously, we worked in that main branch, and this is where your main project actually resides. Previously, we'll say we're on version 0.1, and we're starting now to develop version 1.0, which has multiple different features. So not only do we create a development branch for what we're developing, but every additional feature for that new version, we would start its own branch for as well to keep separate because we probably have people
11:54:06
working on different features. So let's say just for this case, you have two different features all you're working on for this new version. Now let's say feature two finishes before feature one, well, we can then merge feature two into the development branch where we're working on version one. Also during this, let's say that we get a bug that is in our main branch and we need to fix it. Well, we create a branch for hot fixes in order to fix this bug. And then we would merge
11:54:39
this into our main branch. And this case will now be at say version 0.2. But what about that develop branch, right? We're working and currently we need it to be updated with what's in the main branch. Well, we can merge that bug fix also into our development branch. And so our development branch not only has now those bug fixes, but also has feature two implemented into it. Now, at some point, let's say we get done with feature one, and it's done. Well, we can merge it into
11:55:13
our development branch. And then from there, we can implement it into our releases branch releases branch are just the final area to put it before we're going to actually merging it back into main to do any final touchups. So we may find some things that we need to fix and we fix them. Then from there, we merge them into our main branch. And now this becomes version 1.0 that we can use. Now, now that we're done with 1.0, we can start this cycle all over again and update our development
11:55:43
branch to start version 2.0 for the features we want to add for that. So in this section, we're going to be going over four new commands of git branch and then also git switch. So let's dive into VS code and start and we'll mend this. All right, so in here, I opened up the version control or source control tab. I haven't committed any to my local repository or a remote repository recently. So let's go ahead and do that to clean things up. So I'm going to add all these files by
11:56:14
doing git add and then period where that period is selecting that current directory. And now I can see that these are all stage changes. Then I'll run git commit use the M flag specifying in double quotes that we're adding the problem files. So now we all need to do is just push this up to GitHub. All right, so our repos all cleaned up, we can see right here that we're up to add problem files. Once again, as a reminder, we've been working in that main branch. So I'm going to go
11:56:45
ahead and clear this. How do we see what branches we have? Well, we can run the command git branch. In here, it specifies main main is our only branch and then it has an asterisk. This symbolizes your head. It's a special pointer and git that refers to the current commit you're working on. So not only is on that branch, but also when you make a new commit on a branch that head automatically moves
11:57:13
forward to point to that new commit on that branch. And anyway, with that, we're going to be adding inside of our SQL data engineering folder, we're going to be adding another folder for our second project. And implement or build a read me for this, or at least start building it for this. So what we're going to do is create a branch that we're going to delete and but we're going to create a branch in order to create this read me. And this is just for demo, I wouldn't expect for just to
11:57:40
read me you create a branch, but you get the point. So do this, we type git branch. And then the name of the branch, I like to start mine with either something like feature, but we could also see something like develop if I'm developing like a data warehouse, or maybe bug fix if we're doing some sort of bug fix. And that just denotes what this major overall is about. And then I'll do a slash. And then define what this is, we'll just say this is project to read me. All right, so I'm
11:58:08
going to go ahead and press enter. Now when I run git branch, I can see that we're on main branch where head is at, but we have this new branch of feature project to read me. Now if we need to get to our head over to that project to read me branch, we could use git switch. And then specify this feature branch. It tells us we switch to branch feature slash project to read me. And running git branch, I can confirm this. Now let's say we want to delete this branch because we're
11:58:39
done with it. Don't worry, we're going to recreate it after this. Well, we can use git branch and then specify this D flag. And that means delete. And then from there specify the name of project to read me. However, if you try to run this, you're not gonna be able to do this right, because you can't delete the branch that you're on. Remember, I'm gonna do run git branch, we're on that branch. So we need to switch to main first. So run git switch main. Then from there, I'll run delete of
11:59:10
that branch. And it tells me delete branch feature project to read me was basically deleted. Now when I run git branch, I can see that I have only main. Now let's create this branch again. But this time, we're going to create it and also switch to it in one command. With this, we run git switch instead of grit branch, we specify this C flag, which stands for create. So we're going to create and switch to the branch that we're naming right here of feature project to read me. And it tells
11:59:45
us we switch to that new branch. When if I run git branch, I can see that we switch to that branch. And we also created it. Now I do what you aware of this command, which we it was previously used. And that's git checkout. And this is synonymous, if you will, with git switch. I can git check out main and it switches to that main branch. If I run git branch, I can just verify this. Yep, but git check out is the traditional command and git switch is so much more powerful
12:00:17
because as you saw, we can not only switch to a branch, we'd also create it at the same time. So for our purposes, I don't really think you need to memorize git switch. Also with this, whenever you type in something like git switch feature, I can just press tab and it automatically fills in the rest of that branch. Pretty cool, you don't have to remember all that. So I go ahead and press enter. We're now switching in that, run git branch and confirm that I'm on that feature branch.
12:00:42
Now let's get into making changes on our branch. Right now, I can run git branch and confirm that I'm on that feature branch. What we wanna do right, the purpose of this was to create a read me in our project too. So inside of here, I'm gonna create a new folder and we'll do that two underscore data warehouse or DW, Mart build. I go ahead and press enter and then inside of here, I wanna create a read me. So I clicked add new file and we'll call this readme.md. That's very standard.
12:01:16
All right, I'm gonna open up also source control now and we can see that we have changes in our working directory. So this is looking right. Let's go ahead and add a title to our read me. We're gonna add a single hashtag for H1 header. And then I go ahead and add this title of data warehouse and Mart build. There's our production ETL pipeline. All right, so I know I have changes right because I see that this white dot is over the file.
12:01:40
So I press command S or control S if you're on Windows. So I know everything is saved and up to date. Now what we're gonna do is we're gonna stage and commit these changes into our current branch. So to do this, you know, we do run git add then period and this adds our readme to our stage changes. And then we'll run git commit with the em flag specifying that this is the add the project to readme. All right, so now we want to push this. Now we can't just run git push.
12:02:13
I'm gonna run this. As this says, hey, fatal. The current branch feature project to readme has no upstream branch. In order to push the current branch and set their mode as upstream, use git push and then set upstream. So I was finding the origin as where the feature branch is. So we need to run git push. And then you could do this dash dash set upstream. But honestly, the easier and more acceptable one that we've saw previously write is dash you. And that means set upstream.
12:02:43
So go ahead and run this to push it up there. And I mistyped our feature branch. Specifically, it's project to readme. And I realized I got completely confused. You have to specify origin first. And this is specifying the name of your remote repository. Remember that is origin. And then from there, the name of our remote repository. So I'll do feature. And then press tab to add in project to readme. All right, now we have it all up to date. All right, so this went through
12:03:13
and actually pushed it up into GitHub. And conveniently check this out, right? If we were doing this branch as part of a pull request, which is very common in data engineering to submit pull requests for new features, it gives you the link to now go forward and create that pull request using that branch. And clicking that link directs you right into GitHub, you can just go in and create that description of whatever your pull request is if you were submitting a pull request.
12:03:44
Anyway, we're not gonna do this. This is out of scope of what we're trying to do right now. But I do want to show you what actually updated on GitHub. So if you were following along, go into GitHub. What we can see is right here, we don't, well, one, we're on the main branch. We don't have that two underscore data warehouse folder. But you can click this down arrow and this shows us our different branches. And I can actually, if I want to switch over
12:04:13
to our feature branch or the project to readme. And I can see it inside of here while also navigate into it and seeing what we added to it. Anyway, just going back here, you probably did see this up here, right? We could go forward in the UI to compare and perform that pull request and ultimately merge what happened here. But we're reserving that for the next lesson, we're not gonna do pull request. That's just out of scope. Let's now get into merging, right?
12:04:43
We've made those additions to our feature branch, we now need to merge it in. Now, there's a few different types of merging, but we're gonna focus only on the two most common, specifically a fast forward merge and then a three way merge. We're gonna start easy with the fast forward merge and then we'll move into the three way after that. All right, so diving into this. For this type of merge, this is when in your main branch, there are no new commits on that branch.
12:05:12
As we can see, we have commit A and B, nothing ahead of it, but on the feature branch we have done C and then D. So what we can do is we can perform this fast forward merge since we're on this and this just pushes all those different commits of C and D onto our main branch. This is the scenario we're currently in with our readme that we created. So let's go ahead and get to work doing this in VS code. Now, I do want to point out this graph down here at the bottom real quick.
12:05:46
In this, this is actually somewhat showing what we saw previously visually what I was showing. Of how we have our main branch and then our feature branch that we're doing that's ahead of the main branch. Well, in this case, this feature branch is ahead of our main branch. And just visually so you can also see this, right? This is right here, this orange portion. That's what the origin or a moat main is at. Also with our local main is that as well.
12:06:15
And then when scrolling up here, we can see that the origin also has our feature project to readme branch as well up to this point. And our local one is on feature project to readme. Anyway, the whole point is that we're ahead of main. Now, in order to perform this merge, we need to get back onto that main branch. So what we're going to do is we're going to switch to main by doing git switch and then main. And then we notice right that we now have some changes
12:06:44
that are no longer there anymore because we switched to main and we're back some revisions. Now, it's always good practice to make sure because we are a main that we are up to date with our remote branch. So I will do a git pull origin main, but I know it's up to date. There's nothing that's happening. It says already up to date, we're good to go. But this is just breast practice. So now let's perform our merge. We're going to run git merge and then we'll list that feature branch
12:07:11
slash and I'm going to press tab to fill it in. And we're going to merge our feature branch into our main branch. So the first thing it does is it says it's updating these code numbers which correlate to the appropriate commits of the main branch and then of the feature branch. And then it says this, it says fast forward because we're performing a fast forward merge. And below it says it's adding the readme file and there's one file change with a one insertion
12:07:42
or one row or one line which we can see right here. So now visually, let's go and check this out right here where it says, hey, we add project to readme. Remember that was part of our feature branch but now that is part of main which that main correlates to our local repo's main. However, we are one commit behind our origin main which was we did when we added all those problem files at the beginning of this lesson. Now you do see up here this outgoing changes
12:08:14
and this is just telling us that we're right now we're out of sync of our remote repo and so we need to perform that push. So just to be clear, our local repo is up to date at least where the main has performed that fast forward for our local repo. So we're gonna run git push and although we did dispecify origin main earlier, actually we didn't have to do this because we had already set the upstream by using that U flag earlier for main that the upstream that we're pushing to always
12:08:51
are pulling from for main is origin main. So we don't necessarily have to specify it here if we don't want to. Anyway, this went through and this updated our two commits specifically that add project to read me and the add problem files. And from there it added that read me with one file change and one row insertion into it. All right, so with this git push, what it is saying here. First I'm gonna stop at the start at the bottom. This is saying, hey, we went for our old commit
12:09:18
on origin main to our new commit. Then from there it's specified that we pushed it to our remote repository of SQL engineering projects. And then this was just a fancy way of saying that no objects needed compression, everything was sent just fine. So now I'm gonna clear this all. What branches do we have left right now? So I run git branch, we have remain and we have that feature branch still. We don't need that feature branch anymore. So we can actually delete it.
12:09:46
We do this by running git branch with a D flag which stands for delete. And then we list that feature branch. Then when I run this, it deletes the branch. I can check by running git branch. And I see that only main is left. Now, however, if I go into git hub where our remote repository is right, if I refresh this, this feature branch is still here. Listen to the branches and we don't need it up here either. We're not using it anymore. We only wanna just stay on that main branch
12:10:19
and don't have any confusion. So we can delete this by running the command git push and then we're gonna tell it up in origin of in git hub to delete. So we're gonna run with a delete flag and we'll specify that we wanna delete that feature branch. All right, so with that specified, we'll go ahead and run that and it tells us that that branch was, well, deleted. Let's check it. Refreshing my browser. Now when I click down on the branches, only main is in here.
12:10:51
(gentle music) So let's now jump into performing a three way merge. How are we gonna do this? Well, what we're gonna do is quite simple. We're gonna create another feature branch and then on that we're going to perform a commit. Specifically, we're just gonna add our first file for our project. We're just gonna add one line to it. It's gonna be super simple. Anyway, once we get that done, we're gonna push that up to our remote repo.
12:11:17
So we'll have that main and then also that feature branch on our remote repo and there's gonna simulate that we're working on this feature branch. Anyway, then what we're gonna do is we're going to make changes on the main branch and this will be simulated. We'll be doing this in GitHub because this is very similar to what would happen in a real-world scenario when you're working with other teammates and they're making changes to main while you're working on the feature.
12:11:47
And so then what we're gonna be able to demonstrate by doing these changes on both these branch is then performing a three-way merge, which here in this case is demonstrating how we're getting that commit C from our feature branch and commit E from our main branch into one final merge. So let's jump into it locally and we're gonna first need to create that feature branch that we're gonna be using for this. So we'll run git, switch with the C flag for create
12:12:14
and then feature slash project two file one. All right, so we switch to that new branch of our feature branch. Now with this, all we're gonna do is go inside of our second project here, add in a file. We'll call this O1 create tables.awarehouse SQL and then we'll add a line up at the top, a comment block saying step one, DW or data warehouse create star schema tables. So we're gonna be creating a star schema tables in that first part of the project
12:12:44
or on that coming up in the next lesson. Anyway, that's not important. The main thing is we just created a new file and we have changed in this. We need to save it. So we're gonna go ahead and select command S or control as if you're on Windows. It's now I'm gonna switch over to that source control pane. So we have changes, we need to stage them. So I'm gonna do git add and then period to do the current directory. They are now stage changes.
12:13:07
Then we'll run git commit with a message of add file one. Keep it real simple. And from here we have one file created, one insertion because we added one line to it. So over here visually we can see that we're ahead of our main branch on both origin and on main. And we now have this feature branch that is also not in our origin or in GitHub. So we need to push it. So now we'll run git push. We will need to add that U flag because this is a new branch
12:13:38
and we don't have the upstream branch specified for this. And then from there we'll specify origin and then that feature branch itself. Okay, I'll go ahead and run this. So like we saw previously, we have our data up here that says that we basically pushed our changes that we wanted to. If we wanted to create a pull request, we could click this link right here and go into creating pull requests. We're not doing that. So damn, it looks like everything is good to go with this.
12:14:04
So where are we right now? Well, we did that commit onto our feature branch. Now we're gonna go into GitHub and we're gonna add a commit there to our main branch so that we can now get into after that three-way merge. All right, so I updated our refresh GitHub. In here I can see main and I see our feature project to file one. So I know it has the changes that we have. We're gonna stay once again on that main branch. We're gonna be editing inside of our two data warehouse.
12:14:32
Mark build, we're gonna edit this, read me. I'm gonna go up here to the right-hand corner and go edit this file. And I'll enter two new lines. We're gonna enter in a to-do and we'll say, hey, fill this in. We're gonna keep it pretty simple. Okay, I want to commit these changes to this main. And this is once again, simulating like a coworker went in and changed this file. It automatically auto-generated this from co-pilot for the commit message and the extended description.
12:15:00
Looks good to me. We do wanna commit directly to the main branch. We'll be keeping that. You could select this if you were trying to create a pull request of it would actually go through on here, create a new branch for this commit and start a pull request. We're not doing that out of scope. So we're gonna keep the other one selected and select commit changes. All right, so now we have those two different changes ahead and they're separate.
12:15:23
So now we're in a situation that we can get into actually performing that three-way merge. Before any merge, we want to switch to the appropriate branch. So right now if I run git branch, I'm on that feature branch. I'm gonna switch over to main by going to git switch main. Now notice whenever I do this, well, we actually had some changes that happen. First thing is this, we wanna switch to main, right? This file that we created is not in main right now.
12:15:50
So that's why it has this line through it. I don't wanna confuse anybody. So I'm gonna go ahead and just close that. But it doesn't exist in main now. It's still in the feature branch. Also during that, I don't know if you noticed this, but this, this add to do section, is what we did in readme. And this got auto-fetched or got fetch for us automatically. I had it set up in VS code to perform git fetches from time to time to make sure that I'm always up with origin.
12:16:20
Remember, normally we're performing git pull because that's a git fetch and a git merge. Anyway, the git fetch is what is the portion that moves what we have up with the remote repo to our local repo, but not necessarily in our working directory. That's for git merge. Anyway, if you don't have that auto git fetch set up, it's not a problem. You can just run git fetch and then run it and it'll update. In my case, nothing happened. So now before we perform this merge,
12:16:48
we need to get both of these main branches in sync mainly. We have our origin main, which is one ahead of our local main. So we're gonna actually do this by running git pull. And once again, we don't have to use that use specifier because we have already done it for our main branch already locally. All right, so in this case, it said it did updating. It did in that case also, it performed a, if you will, a fast forward merge by updating that main branch to the origin main branch.
12:17:24
And it specifies what files were changed specifically that read me with basically the couple of lines that we entered into it. Anyway, we want to, and we're gonna get to it, we wanna perform that merge. I do wanna show visually though what it looks like before. If you click auto right here, you can then go to change it from the current history item reference to view all history items reference. I'm gonna click okay. And in this, we can see specifically
12:17:53
that feature to project two file one edition, which also they have it in the origin or the remote repo as well. Okay, so we can see visually in this case that, okay, this is perfect. This is setting us up for our merge. So enough already, let's go ahead and run git merge. And then we'll do that on our feature branch for project two file one. Now over on the left hand side, I could go through the UI to do this. We're not going to do it. Anyway, this does show us right now
12:18:22
that we have those stage changes for our SQL file that we added locally. What this did here in the terminal is it opened up an editor. Remember, we set nano as our editor. So if you're editor or if you didn't do that, you're gonna have film and operating with that. Anyway, in our case, what we can do is we can just enter a message into here. And I just said, hey, perform merge, adding file one for project two. And then down to the bottom, I can see first we need to write it out.
12:18:51
So we'll press control O and it says file name to write, basically to our merge message. That's what I wanna do. So I'm gonna press enter. And then now that we have it saved, I'm gonna press, I wanna exit out of it. So we're gonna do control X. If you're in the VIM editor in order to exit out of it, you're gonna just press colon and then WQ, which means write and then quit. So now we can see we have outgoing changes and that we're performing this merge locally.
12:19:19
I'm actually gonna even change this back. We're on all, I'm gonna go back to auto and select okay. Not much change in the view. But anyway, locally I can see we're one ahead of origin main. So I need to now push this up to GitHub. So I'm gonna run git push. Don't need to specify that U flag, 'cause we know what it is already for origin. And bam, it ran through and updated all the objects as it needed and it updated our specific git repository. Everything's looking good with this.
12:19:47
And we can see visually now in our graph that we've performed this three way merge based on what happened here with our yellow little thing that goes out for adding file one. Pretty cool. Anyway, like last time, when you clean this up, right now I'm gonna run git branch. We can see we have that feature branch. I no longer need it. So I run git branch and then they'd run the D flag and then specifying the feature branch to delete. And that is only locally that it was deleted.
12:20:16
So in order to delete the remote branch, we'll run git push and then we're specifying that origin and inside of it delete. And then that feature branch of project two file one. All right, and it says that it was deleted appropriately as well. Going back to GitHub and making sure that's refreshed, checking inside of here, we only have main inside of here. So with that, we successfully demonstrated a fast forward merge along with a three way merge.
12:20:46
Both of these are two of the most common type of merges you're going to encounter when performing merges. Now that we have merging down, we're gonna go over conceptually what a pull request is because a pull request involves merging. And specifically involves using branches. So a pull request or PR is a way to propose changes to repository. What it's doing or what we're trying to do is requesting to pull your changes into another branch typically from something like develop
12:21:20
or bug fix into that main branch. Now we've already discussed the benefits of branches and how they allow you to perform tests and develop on a separate branch without affecting your main pipeline or whatever it may be. But with this PR, we can do additional things and that were provided an area to one perform code reviews where team members can review what changes you're actually trying to do with this PR. Then from there, they can comment on specific lines
12:21:49
and suggest improvements based on that PR. You can run automated tests before performing the merge and then you can also do some sort of documentation if you want to explain what changes are happening and why. Anyway, this is all bundled, if you will, within a PR. So let's go over a typical process for a pull request. First, this starts by creating a new branch for what we want to do. And this is branched off of main. From there, let's say we add feature one
12:22:16
and we merge that stuff into that develop. And then we develop feature two, which we also merge into there. Well, now at this point, we'll say, hey, all of our features are complete that we want with this developed branch. We want to now get this implemented into main. So let this point submit this pull request or PR. And typically, this would go out and notify the appropriate stakeholder or owner so they can start through their review process to approve it.
12:22:45
And so what you can do inside of something like GitHub is you can chat about it. And so usually your PR is probably not gonna get septed on the first round. So you may have some proposed changes such as maybe implement change number one. And then there may be more chats involved and change number two maybe. All right, so now it's finally ready to go and to get merged back into main. At that point, the person that's in charge of managing main would close the PR
12:23:13
and merge your develop branch now into main to implement those different features that we've developed. So you have all the requisite knowledge for performing pull requests, now that we've gone through an understanding branches and how to perform different merges. So we're not gonna walk through another example with implementing a PR. But we do have the practice problems implement you walking through a PR. And so you'll get practice with not only branches
12:23:45
but also pull requests during that point. Last thing I wanna cover in this lesson is around git ignore. Now this dot git ignore file is something you should always be including and all the different repositories that you build on GitHub. And specifically use git to track. Anyway, this file tells git which files and directories to ignore and not to track. And there's a lot of different use cases on why you may not want to track different files.
12:24:15
Specifically, you could have files with sensitive credentials. This may have things like API keys or passwords within the files, those type of things. Definitely don't wanna go public on GitHub or tracked in git. Additionally, large data files such as CSV files or even database files. GitHub actually has a limit on the file size that you can upload into it. I think it's up to like a hundred gigabytes. Well, just Google that actually it's a hundred megabytes. So way smaller.
12:24:46
Anyway, I commonly found that my data files are much larger than this. And so you will get into errors if you try to even push these files that you have that are that large. Additionally, you may have other files like environment specific files, ID settings or even log files that you just don't want other people to have and that may be specific to you and your testing and tracking. So let's demonstrate this.gitignore file. What we're gonna do is we're gonna add it
12:25:13
in the root of our directory. I'm gonna add new file and I'm gonna call it.gitignore 'cause it ignores. Now let's say we went through and created a database file that was like super large and we're gonna create a DuckDB database. Anyway, I'm gonna go ahead and create that file right here and call it largefile.duckdb and that's what the database file is named. Anyway, right now, whenever I added it, you can see it went green, it's untracked. I can't even scroll over to source control.
12:25:43
It's right there and we call it a large file and it's untracked. Now, I don't want git to track this. So what I'm gonna do is I'm gonna go back to my gitignore file and inside of here, we're gonna use special syntax to be able to track files. In our case, I'm gonna just for the first portion, I'm just gonna write largefile.duckdb and I'm just writing explicitly what that file name is. Now, whenever I go to save this, watch what happens to largefile.db.
12:26:15
It updates and it no longer is tracked in here. Even I can go into source control and it's no larger tracked. Now you can imagine though, say I have another file or another database file that I create such as evenlargerfile.duckdb. Well, that one become untracked now too as well. And it would be a pain to have to go to this gut.getignore and just have to add in every single one. So we can use the special syntax for this. Specifically, I'm gonna remove this largefile
12:26:43
and I'm gonna put this asterisk here. And then I go ahead and save it. And now this does some special matching in order to match through and find just any file within anywhere in here to not track.duckdb. So this.duckdb file here, I'll just go ahead and drag it into our Data Warehouse Mart build right here. Even inside of here, it's getting picked up not to track. You can see it's a lighter gray. And even on source control, you could see only the.getignore is there.
12:27:15
Now I'm gonna go ahead and drag this out of here. And then let's say that we actually, we were keeping our stuff maybe in a folder called data, pretty common for it to be called data. And, you know, I put these folders inside of here or these items inside of here. And then for some reason, maybe I add another file. We'll call this super large CSV. And with this, I don't wanna charge it. Now I can go in and you probably should add CSV to this,
12:27:42
but why can't we can, why don't we just go through and actually mark out that we don't wanna keep this data folder? Well, for this, I'm gonna just go into here and type data and then do a slash command after this and tell this that it's a directory. Now whenever I save this, this entire data folder goes gray and even that CSV no longer is being tracked. Now it's important to understand though, one little thing about the fact that you need to add
12:28:10
this.getignore file early on in your project because let's say, oh, now I don't want to include this one underscore EDA project folder right here. And so I could come into here and I could do one underscore EDA and then do the slash and then save it by pressing command S. Well, if you noticed, these things are still this lighter color gray and they're not getting removed from Git because we already committed them in the past. I can even go to source control and see that, oh, scrap.
12:28:44
It's not in there for actually removing it from Git. It's still up there. In that case, we'd actually have to go to the terminal and I'm not gonna run this command 'cause I don't want to remove it. But we'd have to run git rm or git remove and then dash r to specify we want to do recursively for this whole directory. And then from there, specify the directory of one underscore EDA. And then you could remove it from git and then make sure it's saved and not get ignore
12:29:13
and it won't be there. Not gonna do that, but I just want you to know there are options for this. And then after you did this git remove, you'd have to then push these changes or well commit these changes and then push this up to GitHub to get it into your remote repo as well. So what files should you add to your git ignore? Well, conveniently GitHub, their GitHub account, has this git ignore repo, which has a bunch of different dot git ignore files
12:29:42
depending on what programming language or what you're using. So if you're using something like Python, you'd go to their python dot git ignore and then you could just copy and paste all the contents from this into your dot git ignore. And this would be my standard approach for any programming language. Unfortunately, there's nothing in here for SQL files or SQL projects in general, but I have a generic one that I've put together based on what I saw in the GitHub one.
12:30:14
And you can get to it via our GitHub project for this course. What I'm going to do is now just navigate into the dot git ignore file. And this has everything we want into it. All we're going to do is just copy it all coming in C or Control C if you're on Windows and then inside of here, I'm just going to paste it command V. And then once it in here, I'm going to command S to save it. Also, I'm going to do a little bit of cleanup real quick.
12:30:38
I'm going to remove this data folder because we don't need it anymore. Anyway, what's going on here? Well, first of all, this is broken down into sections for different files you may need. This hashtag is similar to a comment in SQL, but it's a comment in a dot git ignore file. So anything after a hashtag line is completely ignored. Anyway, I have a section for local database files. This dot wall is something that happens with duck DB whenever it's running and building a database.
12:31:08
So I'd like to make sure that that isn't tracked. If I accidentally try to commit during that, I also have raw data files in here. Any secrets that you use dot ENV files are very popular for this. Also have things like system files, log files, tooling. And then I included also my own files down there. You could probably just go ahead and delete this. One thing that you may encounter while going through this course is this dot VS code file.
12:31:36
This happens to appear and it has the settings that you may have depending on where you did in VS code and set during the project. Here I am in my other editor for SQL data engineering courses. Anyway, I want to just show this real quick. I have this dot VS code file with a settings dot JSON in it. This is specific settings specific to setting up duck DB for VS code that we may have implemented. Anyway, I don't need this to going up to GitHub or other people seeing it.
12:32:05
And so that's why dot VS code is tracked in there. All right. So that was quite a bit of stuff, but I feel that wraps up what you need to know as a data engineer when it comes to Git and GitHub. We've covered branching now how to do PR or pull request and also this dot get ignore file. And practice problems for you to go through and it's our last set of practice problems before we get into our final project, which recovering the details of that in the next video. All right.
12:32:34
See you there. All right. In this video, we're going to go over a brief synopsis of our second and final project in building out a data pipeline and I'm not going to lie. I'm quite proud of this project, we've spent a lot of time developing it because I wanted to implement everything we've learned up to this point and utilize it in building out a real world scenario case where we're building out this type of data pipeline for this.
12:33:06
We're not only going to be using those skills of SQL and tools like duck DB and mother duck, but also those other skills we learned with get GitHub and terminal to make this all happen. And this is going to be turning CSV files that are in the cloud into data warehouses and data marks. But real quick, let's have a breakdown of what data warehouse are and data marks are because it's really important we understand the differences.
12:33:35
So first data warehouses, rumor bill, M and define this and it has to have these four things, one that it's subject oriented, two that is non volatile, three that is integrated and four that it's a time variant collection and combining all of these four things. We put data in a manner that supports management management's decisions and this is a data warehouse. Now for this project, we're not only going to be creating our data warehouse.
12:34:06
We've been using this for scratch. We're also going to be creating three and possibly four different types of data marks. These data marks are going to be extracted from our data warehouse. I like to make them smaller visually because imagine it like a smaller subset of the data. But smaller is not necessarily the correct way to describe the difference. Anyway, I'll put together this table that better describes the differences between a data warehouse and data mark that a warehouse.
12:34:34
The scope of it is more enterprise wide. All business data would rely on it, whereas a data mark would have specific departments or even specific use cases. For the schema design, typically you're going to find a data warehouse in what we've seen for of a star schema, whereas data marks can have those star schemas, but also they could have denormalized or dimensional tables as well. So we're going to be integrating NARS flat tables.
12:35:00
Now the purpose based on that scope and schema is going to be a little bit different. Data warehouses are centralized level levels of knowledge. So you have these reusable data storage areas for this, whereas data marks are more well specific to departments so they're optimized for specific analytical questions that departments or even individual people need answered. Users for this would be, well, the enterprise wide for a data warehouse.
12:35:26
So anybody in the company would want to access this or multiple teams or even data engineers. Data marks, on the other hand, are going to be specific teams or specific people that have a use case for it. Data warehouses should have the complete historical data that's necessary for them, whereas data marks are usually aggregated or even filtered subsets of that data. So moving into the project goals for this, we're going to be building a production ready
12:35:54
data warehouse using star schema design patterns. For this, we're going to be getting our data from CSV files that have everything that we need for this, and we're going to be building out our job posting warehouse. This will be using the same star schema that we used to, but we're going to be building it from scratch so you'll get familiarization with that. And that's going to help us with our second goal of implementing proper data modeling techniques
12:36:23
with fact tables, dimension tables, and bridge tables. From there, we're going to also implement this proper data modeling in building out our different data marks. We have three specific ones that we'll go through and break down even further in a bit. And this will satisfy our third goal of creating specialized data marks optimized for different analytical use cases. Now putting this all together, this allows us to get that data from data storage into
12:36:52
data warehouses, and then from there, pushed into data marks, and in a manner that is now available to be served to others that need this data, whether through Excel, Power BI, Tableau, or Python. And I want to be able to do this controlled in an automated manner. And so our fourth goal is that we're going to develop this using automated build scripts that can rebuild your entire pipeline with only one command, going to be pretty cool.
12:37:23
Just as a sneak peek, this is the file right here, and it goes through and reads the seven different files or six different files that we're going to be putting together for the scripts for building out each one of those pieces. And so by running this file, we can build our data warehouse and data marks from scratch. So let's now get into the project scope, breaking into how we're going to build this data pipeline out over the next four lessons.
12:37:53
If you want, you can navigate to project two in GitHub, and you can see what we're going to be building out there in total, we'll be building a total six, possibly seven files if you decide to, and each one of these control all the different aspects of the warehouse and data mark build. Additionally, we'll be building out with this read me, which is located below that goes over the different problems we're encountering and what skills we're demonstrating and building out with this.
12:38:22
Oh, and also, like I showed you earlier, that script that we actually use to maintain or run this entire pipeline with only one line of code. So has everything we need in it. So in the first lesson, what we're going to be doing is, is we're going to be taking our CSVs, which are located in Google Cloud, and from there, we're going to be creating our tables needed in our data warehouse, and then importing that data into it.
12:38:50
And this data warehouse is going to be, well, we're going to do it in mother duck or I'm going to actually develop it locally as we go along to simulate more of what we do in a real world scenario. So for this first lesson, we're going to be building out these first two strips. This first one is where we go through and actually create all the different tables we need. And then from there, the second script is to load our schema into the data warehouse as
12:39:17
a few different insert into commands in order to do this, along with some data checking towards the bottom. In lesson two, we're going to start building out our data marks. The first one we're going to be building out is a flat table mart. This one's pretty simple in that we're going to be transforming our data warehouse into a flat table. And this is not only going to integrate all that company data, but also using what we learned on nested data to also incorporate our skills and type.
12:39:46
We'll do this all within a single script in order to create the table and then insert it in from that data warehouse. In the third lesson, we're going to get into building out our skills mart. Now our skills mart is designed in a star schema pattern, but it aggregates our data from our data warehouse to better showcase aggregations of special context. We want to get around skills. So based on a skill, we'll be able to tell how many postings it has and other key statistics
12:40:18
that it may have with it. This will also be done in its own single script where we build out those three different tables and then insert into it the required data. The fourth and final lesson for our project build will be around building out our priority mart. And we've done a lot of work with that already in previous lessons. If you're recalled, we built out that priority jobs snapshot table and that was built using
12:40:46
our data warehouse combined with priority roles, which could get updated daily. And we built SQL in a manner that supported incremental loading and updating all of that priority job snapshot, which is super important as a data engineer to know. And so inside out of our priority mart, we're going to build out our priority roles along with that priority job snapshot. For this, we're going to break this down into two different scripts, the first script being
12:41:14
the initial load, which is showcased here where we create an insert into it, all the different things that we need from it. And then in our next script, we're going to demonstrate how we can update our priority roles table and then go through and incrementally update it as well, specifically how we use that merge into for the priority jobs snapshot table. Now you'll also have the option to go through and if you want to build the company mart,
12:41:42
and that's just an extra mart that you'll go and go through and build and get practice building out star schema designed patterns. This mart is a lot more complex than the skills mart. So it's really going to test your skills if you're interested in going through this. The SQL code for this company mart is included in GitHub and it has everything you need in order to investigate how to go through this.
12:42:04
Now for those that purchase the course practice problems in your notes, you have even more details on how to build out this company mart. And it breaks it down step by step on what you need to do to go through and build this out. So great option for those that maybe want to beef up this project. One last thing to include from this, we are going to be implementing the use of branching when building this project out.
12:42:32
Specifically, we're going to get a develop branch for an entire project to to work along along with individual feature branches for each one of the major aspects, especially every lesson is going to have its own feature. I told you we're going to be implementing everything we learned throughout this entire course. And so all that work is going to pay off. All right. So with that, there's no more practice problems for this lesson or for the remainder of the course.
12:42:59
We're now going to be jumping straight into the project starting with the next lesson on building out our data warehouse. All right. See you there. All right. In this video, we're going to be building out our data warehouse. We're going to break it into bite size steps to do this though. First, I'm going to go over what our Git workflow is. Then from there, we'll get into implementing it using a script to create our tables and then another script to insert our tables.
12:43:29
And then finally, we'll perform some data validation to make sure that it got loaded properly into the data warehouse. As a quick reminder, this is the data warehouse right there we're going to be doing. We have plenty of familiar to do with this. But the main reason why I'm showing this is because it's important to understand we have four different tables that we're going to be loading into it. And we need to make sure that we get all these tables created and inserted in properly.
12:43:56
So for this, I've developed a Git workflow that we're going to use to build out this entire project. For this, it consists of, well, it consists of five other branches besides main. We'll have our develop branch and that will be the main branch for developing all of project two. And then we'll have individual feature branches as we build out each one of these things. Each lesson of each of these four core lessons will have their own feature branch and this
12:44:26
one will have the feature branch of data warehouse. And in that, we're going to be doing three different commits along the way. That's going to be the commit for our first script for creating our tables or second one for loading the data and then our third and final one for building our automated script that's actually going to be controlling both of those steps to run it with one single line of code. So let's get into creating these different branches and before we do that, we need to
12:44:51
make sure that our Git changes are all caught up into main. And last lecture, we did Git ignore. So we need to go ahead and get that updated in main before we even jump into this. So I'm going to do git add and then period to do everything in the directory. And then from there, we're going to commit it with the message of add, git ignore. Now that everything is committed, we'll go ahead and push those changes up to our remote GitHub main branch.
12:45:20
And it looks like everything was updated successfully. No issues. All right. So now let's actually get into creating those branches. We're going to create our develop branch first for this. We're going to run git switch with the C flag and we'll create a develop branch slash project to from there, I can run good branch and see that we are on the develop project to branch. So now that we have this develop branch, we can now create our feature branch from this.
12:45:46
In here, I have an initial commit, commit, we don't have necessarily anything to commit right now. So we're not going to do any commits. We're just going to create the feature branch. So once again, I'm going to run git switch with that C flag. And then we're going to name this feature data warehouse, I'll then run git branch to make sure that we are on that branch, we're on the feature branch and our develop branches there as well. All right.
12:46:08
So for this, it's going to be pretty simple and that we're going to be using create table statements. By the way, this is the file that will be created on GitHub. Anyway, we're going to be using create table statements to create each one of those four tables in the data warehouse. So that's when we do in in this section. Now we do have to do this in a correct order because if you remember from our create statements
12:46:32
that we've used, we go ahead when we define these primaries and specifically these foreign keys, they have to reference for the foreign keys, the other table that we're in. So we have to do it in a certain order. So for example, the skills job dim has both requires foreign keys for the job ID of the job postings fact and the skills dim table. So this table, we're going to create last. And then the job posting facts table has a foreign key of company ID.
12:47:00
So therefore, we're going to create this one second to last. And then as far as what we create first, could be either company dim or skills down. And I guess it should be more specific. The company dim table just has to be created before the job postings fact table. And then the skills down just has to be created for the skills job dim table. All right. So we've already created our script that we're going to be using for this.
12:47:22
And that's that 01 create tables data warehouse sequel, the time being, I'm going to close out of this terminal. So let's start creating tables and we'll start with that company dim table inside parentheses will specify the two columns of company ID and also name. Now with all of these, we also need to specify the data type for any ID. It's going to be integer for the name, it's going to be var car. In addition to data types, we need to also specify if it is a key specifically, if it's
12:47:50
a primary key, so we'll include that as well. And then between each of the columns, right, every line, you need to include a comma to break up all of this, along with a semi colon because that's our full command for this. All right. Next up, we can do job postings fact, but I'm going to do the skills down because it's easy. In here, we have our skill ID column, which is an integer, and it's our primary key.
12:48:12
We have the skill itself, which is a var car, and then the type, which is also a var car. If you're wondering how I'm getting this to all to line up, I'm just using tab to do the spaces. I like this because it keeps everything aligned and keeps it formatted nice and easy to read. Anyway, at the end of this, going to do a semi colon. Next up is our job postings fact table. This thing's a beast. It's got quite a few values in it.
12:48:36
The job ID of integer, it's a primary key, the company ID with also an integer, and it's a foreign key, but we'll make note to that at the bottom. Job title is short of our car, job title of our car, job location of our car, job via our car, job schedule type of our car, job work from home of Boolean, search location of our car, job post-to-date of timestamp, job no degree mention of Boolean, job health insurance of Boolean, make sure you spell it right.
12:49:07
Job country of our car, salary rate of our car, salary year average of double, salary hour average of double. So now we need to get into what we say to the best class of the company ID, right? The company ID from this company dim table, which is a primary key, we need to be able to reference that. So we'll start by defining the foreign key, and then we're going to put it into parentheses of company ID. And the reason is you could do a comma and then add other IDs in here.
12:49:40
We'll have that, we'll do that in another and actually in our next table. And then from there we need to go and identify what table this actually references. So we're going to use the reference and keyword and specify, we're talking about the company dim table and specifically the column of company ID. All right, so this is good for this create table statement. We only have one more create table statement to do. This is for skills, job, dim.
12:50:07
Now in this one, we have two columns right, we have our skill ID and integer type, and then also our job ID of the integer type. Now both of these are not only the primary keys of this, right? Because you have, they need to be unique for this, but they're also both the foreign keys to reference the other tables. So we're going to define this a little bit differently. Don't forget about your comments at the end of this.
12:50:35
We're going to call out the primary key, and then in those parentheses, we'll call out the skill ID and then also the job ID. And then we'll do the same thing for foreign key, specifying that skill ID and job ID. Sorry, I actually had that wrong. We just specified the skill ID because we need to reference where it goes to into that to that skills, dim table that we created above with the skill ID column. Similarly, we run also foreign key again for job ID.
12:51:08
And this references the job postings fact table, job ID. Okay. And then with this, also make sure we have a semicolon at the end. So now after creating these tables, right, we did all four of those create tables. I want to verify that they actually created those tables in duck DB and a few other databases you can just run show tables. However, for a more database, agnostic approach, you do something like this of select table
12:51:35
name from an information schema, we should learn about before specifically the tables and where the table scheme schema is main because we're adding these all into the main schema. All right. So I already did way too much and we should probably be testing along this way. Let's actually go ahead and test this. So I'm going to save it by pressing command S open up terminal. All right. So I want to run this file from the command line, honestly, the easiest thing
12:52:04
right great right now, we're on the terminal, we're at SQL data engineering projects, what would be easier just to navigate into our two data warehouse, smart build folder. So we can do that by CD ing it into it or changing directory into it. Okay. So we're inside of there. Now I can do an LS to see what files we have access to. Okay. We have that create tables. Right now we're going to run a duck DB command to run this.
12:52:29
Now normally, right, we just run duck DB and I can and connect to mother duck and connect to that data jobs data warehouse in the cloud. I don't want to do that. We're developing this all locally. I want to test it locally. So because of that, we're going to create our own data warehouse locally. We're going to call this data warehouse Mart dot duck DB. This is going to create this duck DB file inside of our data warehouse Mart build. All right. But I do want to run this script.
12:52:59
Well, if you remember, we can use the C flag in order to do this. And this executes commands for us specifically, we have to put those commands in double quotes, and we're going to use a dot command of dot read, because we're going to read this one create tables data warehouse sequel file, so I'll do dot read, and then, oh one, create tables data warehouse dot sequel, think I spelled it correctly. All right, let's go ahead and run this. All right, couple few errors in here.
12:53:31
First one is this a syntax error at or near cellular average, I can see that I forgot a comma right here after this one. And then I also have foreign key skill ID syntax error. And that is due to also not having a comma as well. Sorry, not syntax error parse error. All right. Let's get these changes going to do command s. And then from there, I'm going to press up and run this again. And well, first of all, this showed us all the different tables that we created, which is great.
12:54:00
But previously, when we ran this command, we got this company dim and skills dim table in here. Then when we tried to rerun it again, we got this catalog error, table names with company dim and skills dim already exist. Now, we want these scripts to be idempotent, be able to run multiple times again and again. And this is really important as data engineers to be able to do jobs, we can do it multiple times while expecting the same results every single time.
12:54:32
So in order to fix this, we're going to scroll up to the top up here and we're going to drop each one of these tables if they exist, starting with that first table of company dim and skills dim job postings fact and skills job dim. All right. I'm going to go ahead and run this now whenever I come up here, press up and run this file again. All right. So I got some errors with this, but I don't expect us to have these errors. You may not have got this.
12:54:59
What I'm going to do is I'm going to go ahead and delete both of these duck DB files. This dot wall file should not still be here. This is just a temporary file while you're building out a database. Anyway, I'm going to go ahead and delete this and delete this other duck DB file. Let's try running this again. All right. It works fine that time. Now let's run it again to make sure we don't get any errors.
12:55:22
And so like me, I probably should actually read this message better says could not drop the table because this table is main key table of the table job postings fact and going to our drop table statement. This is due to have in company dim first, basically it this company dim table is relying another table. So what we need to do is drop tables that don't rely on another table first, specifically the skills job dim, we're going to drop that first, or sorry, I said that wrong skills
12:55:53
job dim does have references to other table, but we're going to drop it first so that way other children of it, such as the skills dim can then be dropped and aren't doing some sort of check to make sure they're not there. Anyway, those make sense once we run it. So I'm going to take also job postings fact because that's a main one that references foreign key in it. And then underneath this, we're going to list that company dim and also the skills dim.
12:56:19
Okay, I'm going to go ahead and save this. I'm going to delete both these files again for this duck DB. So we're going to run the file first. Okay, those tables are created. Now whenever I run it again, no errors. Thank goodness. All right. So we got this script built because of that. We need to go ahead and now commit it. So we're going to run get add period. This stage is all our changes. Also it's probably good practice.
12:56:42
You do run good branch before that to make sure you are on the right branch. I knew I was on the right branch. Anyway, we'll now run get commit with the message of create star schema tables for our data warehouse. All right, sweet. We can go ahead if you want to push it. You can. I'm not going to. I'm just going to keep on going through as with this. So now that we've built those tables in our data warehouse, we need to insert that data in.
12:57:11
As I mentioned, we're going to be using CSV files and each of these CSV files correlate to a table and they are on Google Cloud and I've made these publicly available to use the way you can go and access them. Now real quick, this use of Google Cloud Storage is very popular in data engineering. This is using something called like object storage. So in Amazon, you have what is known as S3 in Azure. You have blob storage and then in something like Cloudflare, you have R2 anyway, these
12:57:41
have cloud storage so you can keep your files in a cloud location and then as necessary, take them and maybe move them into a database similar to what we're doing. Anyway, just to actually show you, here I am inside of my Google Cloud account. And here are the different CSV files. I'll actually have an extra one here of just the job postings flat, but we're not going to be using that. But anyway, these are all the different CSV files.
12:58:05
And with them, I can do something like for company Dan, I can click this and go to copy the public URL. And then in the web browser, put in this URL of storage at Google APIs.com/SQL, underscore the data engineering company div dot CSV. And this goes through and well, you can't see it right here, but it's actually downloading to my desktop right now. Anyway, the reason why I gave that backstory for this is because now with DuckDB, we can
12:58:36
actually not only can we import data in with CSV, we can actually query it directly. And it DuckDB supports reading and querying a bunch of different data sources. So as I mentioned before, about S3 cloud flare as your blob. And then for us, we're just going to use this direct CSV be able to read. They also support tons of other different things. Anyway, clicking on CSV. And so with this, we can see that for CSV import, that inside of a from statement, you
12:59:09
can write directly not only a table, but you could write a CSV file to a CSV file that you have. So you're all just using select star from flights dot CSV. Anyway, I took that company dim table that I just downloaded and put it into VS code. You don't have to do this if you don't want, but because this is just demonstration purposes only, but I put this company dim CSV file inside of here. And as you can see, it has the column names of company ID, name, link, link, Google and thumbnail.
12:59:38
So the column names, if you're not familiar with CSV, these are the column names and then everything below it is the actual values and everything. It's a CSV or comma separated values, and thus it has those comma separated. Anyway, the point I'm trying to demonstrate with this is I can just query this file directly. So inside of our main project where this is company div dot CSV, I'm sure I create a test.sql file and then in here we're going to run select star from in single quotes
13:00:07
company down.csv. Now we're going to open up a terminal. I have a terminal open but it's open to the two data warehouse smart build folder. I want to be in this parent directory so I'm going to actually just go up one and now a minute then from there I'm going to just start up DuckDB locally by running DuckDB and now with this I can go ahead and run this. Actually I don't want to do select. Well we're just going to run all. Bam and we actually they we queried a CSV with DuckDB
13:00:43
and that was that was really fast. We can see it returns actually in columns the company ID name middle columns link google and thumbnail. Anyway the point of that is to demonstrate that we can use DuckDB to query a CSV directly and then we can use that to insert into our tables. Now for this we're going to be using the read CSV function because it allows us to not only specify the source but also provide additional parameters. Specifically scrolling
13:01:14
down to the parameters we're going to use this one of auto detect we're going to set it to true in this auto detect CSV parameters. Actually clicking on this says that it detects the dialect of the CSV file such as the delimiters detects the types of the columns and detects whether or not the file has a header row. So this should handle everything we need for it. So this test file was just for demonstration purposes only. I'm gonna go ahead and close this out. No I don't want to save
13:01:41
it and I'm going to delete this along with this company dem CSV. I don't want to add this to the project. Additionally we have our create tables data warehouse. I'm going to save this. I want to now create our second file that we're going to be building this out. And that's going to be that O2 loads. Well that scheme is spelled wrong. Schema data warehouse sequel. So first let's just query that company dim table together using that URL to get to the CSV. So I'll do select star and
13:02:11
then from read CSV from there you need to put this in and type it exactly as written. So this is HTTP HTTPS colon slash slash storage dot Google API's dot com slash sequel underscore DE slash company down dot CSV. The only thing that's going to change for the rest of these links is this slash portion here referencing what CSV we actually need for this. Now we did say we're going to specify another parameter right and that is of auto detect. And we're going to set that equal to
13:02:46
true. Also I notice I have the spelling mistake up here of from also we're doing a test of this. I'm going to just put in a limit of 10. So now I'm going to open up this terminal. I'm going actually if you didn't fall along I'm just going to start over again in that you need to run DuckDB. And this is just using an in term in memory transient database. And I'm going to go ahead I'm going to try to run this. Now sort of surprised by this but this actually worked for me running
13:03:13
it may not work from some of you but it we did query that company damn CSV file. And I was good confused until well I read the documentation which I recommend you do too. Anyway this system to where DuckDB can read these remote files in the cloud. It is possible to do that because it will auto loads this extension HTTPFS. Anyway it says that this extension is an auto loadable extension implementing a file system that allows reading remote remote files. Anyway the point
13:03:44
here is it auto loads into here. So we don't need to previously I thought we had to write this code we don't need to do that in order to install load this extension because it's auto done. Anyway let's now get into actually inserting into these tables. We're going to first insert into that first table of company dim. Now in parentheses we need to specify the columns one two actually import into if you recall from our query there's actually more actual columns in this table. So if
13:04:18
you wanted to you could but going back to our create table table statement we've only listed company ID and name. That's all I feel we really need for this. We don't need these other links so I'm not going to but you can if you want. So inside of here I'm going to list that company ID and name. Also I might go ahead and just copy this and put that into the select statement as well remove this limit 10 from it and then make sure we have a semicolon on the end. Now the next table
13:04:46
that we're going to insert into it's really important we get this order correct is the skills dim table. Now why is it important that we get this order correct. Well as you remember from our fiasco with the drop tables that we have to drop tables that don't depend on other tables we need to be able to insert into this that don't depend on other tables. Specifically something for like the company dim table. Okay it has a primary key in it. It doesn't require other like other tables to check
13:05:16
for foreign key constraints. Specifically right if I get to job postings fact what's going to happen when I go to actually upload or insert the data into job posting facts. It's going to do foreign key constraint checks on the company ID column. And what it's going to do is make sure all the values that it has in job postings fact company ID match up to what exists in the company dim table. And if you haven't inserted into the company dim table yet that's going to be a problem because
13:05:48
it's not going to be able to make sure anything's matched up and it's going to throw an error then. So that's why we have to do this in order. Anyway skills dim doesn't require or doesn't depend on any other tables as well it doesn't have any foreign keys. So I'll list the columns of skill ID skill and also type. From there we're going to select into it. We'll select these three columns. And then for the from statement I'm going to just go ahead and copy this from the company dim table
13:06:13
and change company dim to skills dim. All right so next up we need to insert into the job postings fact table. We can't do the skills job dim table inspecting that skills job dim table right where we created it does a referential foreign ID check on the job ID column of the job posting facts. So if the job posting postings fact table isn't populated with all the job IDs it's going to throw an error whenever we try to do this. So we have to do the skills job dim table last inside parentheses
13:06:45
I'm going to list all the columns and then from there I'm going to get into writing the select statement for this. I'm just going to once again copy all these columns and paste them into here once again copy that from statement from up here paste it in here and then change this name right here to job postings fact that CSV. All right the last table we need to insert into is the skills job dim table and this has that skill ID and job ID. For this we're going to select the skill ID and
13:07:14
the job ID and then I'm going to copy this from statement paste it in here and change the table itself to skills job dim. All right so we have all of our insert into statements I'm going to go ahead and just put a comment up at the top that this is step two and this is the low data from cse files into our tables. All right go ahead and press command s now let's go ahead and run this file in order to make sure that it's going to work properly. I think mine's going to work properly
13:07:44
but the thing is this if you get an error during this we're going to need to do some troubleshooting and also fix it so we need to develop a path forward to be able to one you can't run the script over again because once you insert into a table you try to insert again it's going to be a mess it's going to just keep doubling the data not good but not what we want. Instead what we want to do is every time we run this we want to create our automated script to actually run step one where
13:08:11
we drop all these tables then create the tables and then from there insert into it so if we have errors in this as we're building out we can easily fix it and rerun it to where we start from scratch and run it again. So I'm going to go ahead inside of this data warehouse spark mart build I'm going to create another folder and I'm going to call this build dwmart.sql this is a SQL file and for some
13:08:36
reason I made this into a folder so I'm going to delete this and try again all right so that worked this time I'm going to move it all the way over to the left hand side. Now if you remember previously in SQL I'm going to go ahead and exit out of this DuckDB right now. I'm going to scroll up here and go to the command. We would run this command DuckDB data warehouse smarts DuckDB-c read01 create tables
13:08:59
data warehouse SQL. What we're going to do is we're actually going to take this command right here this dot command that can be run inside of SQL files and I'm going to just paste it right here and then from there put a comment above it that this is step one. Additionally we need to fill in step two. So I added in a comment and then that dot read command of the SQL file we will run. Now this is pretty cool right this is a SQL script we can then call this SQL script within which then
13:09:28
calls our first SQL script and then calls our second SQL script making this an automatable process. So I'm going to go ahead and save this and now we're going to be basically testing this second file. So first thing though we need to make sure our terminal is pointed to this two data warehouse smarts build folder. So I'm going to cd into it by selecting it. All right now we're in it. Then from there I'm going to run DuckDB then specify that DuckDB file of data warehouse smarts
13:09:59
then the C flag and we're going to run a dot read command of our build smarts script. I actually meant this to be build data warehouse smarts dot SQL so we'll update this then. Okay we got our command let's test it out this is going to be testing we already know the first script works but does the second script work. All right and we got an error so there's actually good for demonstration purposes. In it we say binder error reference column skill not found in from
13:10:26
clause candidate bindings skills skills ID. So this is actually a typo on my part. In here I said when we create our skills dim table I gave it the name skill and then also in the load schema I also gave it the name skill but if I actually take this URL and then go to navigate it to it in a browser you can do this as well. Previously I was in another browser I'm now in safari you can it won't necessarily doesn't have to download it it can potentially just show it to you so I
13:10:55
just found a workaround. Anyway the point of me doing that is because I can see and I can access those column values and I can confirm that that column name is incorrect. So inside of here we're going to change not only our create table statement where we reference skills right here I'm going to command ask that one but then also we're going to change our skills dim table where we reference skills as well and command ask it we also have one other binder error down here and it says
13:11:24
table job posting site does not have a column with name job health insurance. Anyway I went to that first table of create tables just calling that a job posting fact I misspelled insurance. Anyway we typed it incorrectly command dusted and then this last error this constraint error was due to these other columns not loading properly so we're just going to ignore that for now. All right so let's run this build warehouse marks SQL script again I'm going to press up and run.
13:11:53
All right not bad looks like it loaded all of those different files however that did take me that was quite a few minutes that it took to load so and I'm working it with about 16 gigabytes of RAM. Now I was curious during the process like hey what table is running now I would like some updates as it's doing this. So what I can do is in our second file right before we insert into like the company dim table which is our first table I can run a select statement and then in single
13:12:22
quotes put loading company dim table and we'll do this as info basically it's just going to perform this query let us know that's loading it and then it's going to reform it. I also do this for the skills dim table as well job postings fact table and then skills job dim all right I'll go ahead and save this I'm going to just run this just to show what it looks like and it starts out by telling me hey first it's loading that company dim table so it goes through and it was getting
13:12:45
the company dim table so it took eight seconds took a little bit longer than that. Skills dim got the job postings fact table and then unfortunately I got an error because I had an F right here after my semicolon. Anyway you get the point there's now markers along the way that tell us what table it's downloading. Note this skills dim table I think it loads because it loads so fast it's not showing a progress bore but we're going to be validating the data next to make sure we actually
13:13:12
got it loaded. All right so we're going to get into data validation I do want to well I re-ran that script also and just showed that everything loaded correctly and it has those tables for it. Anyway we've made a lot of progress with all of these different sequel code that we've generated. I want to and now at this point I want to commit these changes so I do a git add period to add all the files so I'm going to run git commit with the message of load schemas and create master build
13:13:44
script. All right so those changes committed let's move into the data validation. Now for this this is nice that we have this master build script but I don't want to I mean that took a few minutes every single time I'm running this data warehouse and this data validation script we're going to just be putting it in at the bottom. So anyway I'm not going to run this data validation or run this master build script until the very end instead what we're going to be doing is I'm going to enter
13:14:13
in the data validation queries in here but what I'm going to do is I'm going to run it directly on our database so as you know we can run DuckDB and then specify the database of data warehouse marks of the dot DuckDB file. So I can do things like select count from company dim all right let's see how many files in there 215,000 looking good. Next time I select the count from the skills dim running this we have 262 skills. Okay I actually want to put these we have to do two more tables
13:14:50
but I want to put this actually in just all one table in where I have a row for what table it is and then a row with the count of the rows for that table what I mean by that let's let me just look at this query that we're going to build out. So modify this top query right here and say company dim as table name and that will be the first column next will be count and that will be as record count from our company dim table. I'm going to remove this semi colon and I'm going to do a union all
13:15:25
to our skills dim table right here but in this case I'm going to add those two columns will do skills dim and then add a comma and then we want to count. I don't need to again give it aliases because during the union all it will automatically inherit the alias of your top most table. Anyway let me just show you what this looks like whenever we run it. We get this table where it says hey the table name and record count has company dim skills dim and it has
13:15:51
the information on it and a single table. You know it's pretty sweet. Next we'll get the job postings fact table it's count from the job postings fact table. I'm going to remove this semi colon up here so it doesn't throw me off and then finally throw in the last union all looking at that skills job dim table and getting the count for that as well. All right we'll go ahead and then run this all have a typo should be skills. All right we'll go ahead and run it again and I can tell from the
13:16:18
results of this what I know about the number of rows this all of these have the appropriate number of values in it specifically look we're going to look at job postings facts remember we saw before it had 1.6 million rows and that's where it is here. Now also I do like to look at the first few rows of the data whenever I'm importing in to make sure that it did import incorrectly. So normally I would do something like a select star and then from company dim and then limit this to just
13:16:45
five results. Also right above this I'll actually just specify what this table is of the company dimension table sample. All right let me go ahead and run this. All right and so then we get this returned whenever we run this so we'll say the company dimension sample and then it'll give us just those first five rows from it. When it also added in sampling the first five rows of skills dim job postings fact and skills job dim. I'm going to run this to make sure this
13:17:11
runs just fine and throwing through this we have everything from our company dim table looks good our skills dim table looks good also job postings fact is looking good and our skills job bridge or skills job dim table. All right so that is everything for the data validation that I want to do. What I'm going to do now is go ahead and save this file and then just for a good measure I'm going to quit out of here and then I want to run this now script from start to finish
13:17:43
to make sure that I import incorrectly and I do all that data validation correctly before I then go ahead and submit it. Now I pressed up and got this command right here I'm going to go ahead inside of my build out of warehouse smart script don't insert it into here but well you can insert into it but it has to be behind a comment otherwise it's just going to loop itself and your computer will probably blow up. Anyway I like to have a reference in here so if I need it I can just
13:18:09
copy it at a future time. Anyway we have it in CLI let's go ahead and run it. All right so it's complete we loaded all the table names in loaded all the different tables and then did our basically those five query checks or the limit five checks on all the different tables everything's checking out good so our build scripts are looking good. All right so let's get into committing all these changes I'm going to go through and make sure all of our files are saved open up
13:18:38
the source control pane click get add or type in get add period and then we stage all our changes from there we're going to get commit the M flag of data validation check set. So now we've just gone through and did those three commits we did that for our tables creating the tables loading tables and then previously I always said we were going to do it for a master build script we actually did it for the data check for the data validation. Now what we need to do is actually merge our
13:19:08
feature branch for the data warehouse into our develop project to branch. So inside of terminal I'm going to run get branch just to check where we're at right we're on the feature branch I want to switch to our develop branch because we're about to merge to it so I'll use get switch and then call out the develop project to branch run get branch for good measure it's there now we need to do the
13:19:32
get merge where we're merging that feature branch for our data warehouse. All right from this we can see that it is a fast forward merge just like we want and then it tells me that these three file additions have been added to it we know those three files we can see them up above. All right so not bad and then we can see over in the graphical user interface for the graph that we were on the develop project branch our commits are up to date with this where we're we are ahead of our main and
13:20:03
also origin main which is on github. So now we no longer need that feature branch anymore so I'm going to go ahead and that was only locally so we only need to delete it here so I use get branch with the deflag and specify feature press tab data warehouse we don't need it anymore running get branch just check what we have everything's looking good I'll clean up with this portion. All right so you should be proud of yourself because we just built a simple data pipeline that
13:20:33
implemented a schema design loaded our data in from a cloud storage device and then once we got it in perform data validation to ensure we lowered it properly we did a lot of stuff and that's a lot of things we've implemented in this course so far so you should be super proud of it. All right in the next lesson we're going to be getting into building out our first data mark with that. I'll see you in the next one. All right in this lesson we're going to be building our flat mark or our
13:21:05
flat table mark. This is going to be denormalizing our star schema of a data warehouse into a single table. So for this video we're going to go over why we're even building this flat table mark and then we'll actually get into building it along with what we need to do to validate it and then from that we're getting it updated into our master build script and implementing it into the appropriate branches. All right with that let's jump into it. All right so this is the first lesson
13:21:36
that we're actually going in and building our data mark and this data mark is going to be already called a flat mark and in this this is going to include all the relevant columns from our data warehouse. So other thing for we see from here it's going to be integrated into it including our skills. Specifically for this one right because most job posts will all job posts can only have one company per job posting but job posts is going to have multiple different skills. So we're
13:22:07
going to be doing or using that array of structs in order to put all of the skills into our flat mark. Anyway just as a refresher right that data warehouse is made for like enterprise wide we're all business data relies in it. Data mark is used for a specific department or use case. In our case we have a specific UK use case we want a flat denormalized table in order to be able to query quickly without having to join other tables. So why even build this flat table mark? Well in our
13:22:43
case query speed is going to be increased because there's no joins required. It's also a lot simpler especially if our maybe some like a business analyst that isn't as familiar with performing joins and aggregations. Additionally we're not going to do it in this portion but you could optimize this and maybe tailor it to a specific need such as you could tailor down to a business analyst or maybe senior data analyst or you can even roll it up to where you perform certain
13:23:10
aggregations within it all within a flat table. Now with this flat table there are some trade-offs with it. You could see some data duplication depending how you do it. In our case we're not going to have well actually we will have data duplication especially when it comes to the companies and then something like the skills and specifically the skill types. And because of this data duplication this is also going to cause storage to go up slightly. Overall for our table though
13:23:36
with the size of it of one and a half million rows I'm not too worried about it. We also have issues around updates and flexibility right so now we're having an additional mark on top of our warehouse so that's going to add extra time anytime we need to rebuild this or maybe investigate it due to issues in it. All right so let's get into building this bad boy and for this right we're going to be extracting this data out of the job postings warehouse and putting it into its own
13:24:05
data mart of a flat table mart. Technically we're going to just use a schema for this but we'll explain where we get there. Now as far as our branch workflow is going to go for this we're going to create a new feature branch for our flat mart and we'll perform a couple commits during this process. Once we have the full flat table built we'll then merge it into our develop branch for project two. All right so let's jump in we'll start by running git branch to see what
13:24:33
branch is available we have develop and main. So I'm going to run git switch with the C flag to create a new feature branch for the flat mart. Running git branch just check I switched yep on it. Inside of our data warehouse mart build folder we're going to add a new file for that third script called create flat mart. Now for this right we're not going to create a separate database for this although you can if you want instead what I prefer to do is we're going to keep the data warehouse
13:25:05
on the main schema and then all the marts in their own separate schemas. So because of that I'm going to use create schema and then call this flat mart and that's what we're going to be putting all of it or well the only table for this inside of here. Oh and before I get forget we're going to put a comment up here on what this script is actually doing. Now I do want to clarify something real quick. Say you didn't want to create a schema but you did want the flat mart to be in its own
13:25:34
separate database. Well in that case we want to create our own schema we would just use the main schema of a database and whenever we run this file we specifically call out to use DuckDB and then maybe we'd name it something like flat mart. DuckDB for the database we want to put into here and then similar to four we use that C flag and then we use dot read specifying this file. Now that is a completely valid option to go forward with this. I'm just trying to keep it simple by
13:26:06
putting the different marts into different schemas and also keep it to where we don't have a bunch of DuckDB database files at the end of this. Anyway on comment of that let's continue on. Let's start by querying and actually well performing a query to generate this flat table. If you're remembering from our data modeling so the goal from this is just to combine all of our dimension tables into our fact table to make this flat table. So we need to select all the different columns.
13:26:34
I'm a little lazy so I'm actually going to come into the load schema data warehouse and we query the job postings fact table in that so I'm going to go ahead and select that and then paste that into here. Now we're going to have multiple tables in here so I'm going to go ahead and start putting in the tables so we can start giving them aliases for what we need to know what the alias is for these columns. So first we'll be coming from job postings fact as JPF. I need to add JPF to the
13:27:00
front of all this. Next up we need the company dimension fields. So for this I'm going to join the table first. We're going to do a left join with company dim as CD and we're going to be joining it on the company ID of each of those. Now one thing I didn't include that company ID from the job postings fact table. So we are going to include it from the company dim table. Specifically I added in well company ID and then we took name and I gave it the alias of company name. All right so
13:27:29
last column we need I have in here is as skills and type. If you remember from earlier I said we're going to get this and collect this as an array of structs. And so in order to get this in we need to do left joins to get not only the skill job dim table but also that skills dim table. So let's start by doing the each of those left joins. First let's start with skills job dim and we do this on the job ID from both of those tables. Next up we do the skills dim table and we do it on the skill
13:28:00
ID from most of those tables. So now we need to bring in from that skills dim table the skills and the type. We're going to do this in an array of structs. So we're going to run the array ag function. And then inside of that we're going to need to run the struct packed function to make this into structs. And then we'll have two things in here we're going to have type. And this is from the skills dim type column along with name. And this is from the SD skills column. I'm only just
13:28:32
naming this name because technically this entire portion well that technically I'm going to show you the alias as skills and types. So it's intuitive that the name is referring to the skill. All right now remember right this is an aggregation function. So because of that we need to do a group by underneath this. Now you need to list all of these columns in here depending on what database you're using except if you're using duck DB conveniently we can write group by all we're going to do
13:29:05
that because we're using duck DB. So now let's go ahead and we're going to save this first. Let's go ahead and actually test this bad boy with the terminal make sure that you've navigated to the data warehouse Mart build you would do that by doing CD into that appropriate directory we're already there so we're good to go. And then from there we're going to run duck DB and then list our data warehouse Mart duck DB file that we want to connect to to perform this query on. All right so
13:29:33
I'm going to go ahead and copy this all. And then I know this is going to query all the rows but I want to make sure that it works so that's what we're going to just do it all and I'll press shift enter and it comes back with 1.62 million rows which is is expected and looking at the query there were no errors and it looked like it ran just fine and everything's in there so this is really good. So now this brings us to the question of What are we going to put this query into?
13:30:02
And what do I mean by that? Well, previously, right? We used, when we were creating our data warehouse, we use these create table statements to create the tables, list the columns, define the different data types, along with their primary key and foreign key. And then we had to go after that and perform insert intos with that appropriate query that we generated to query our CSV. So do we want to do that? Or, if you remember from our DDL and DML lessons,
13:30:35
we had three different types of tables, if you will, of CTA's, views, and temp tables. Well, what we could do instead is use the CTA's or create table as in order to build this table, and it takes a lot less code and is a lot easier. Well, I would argue that this case of building a flat mart is a perfect use case of a CTA's. It's a very clean implementation as we can just list the query and then use the create tables as, super simple.
13:31:10
It's also a lot faster than creating this whole table and inserting into. And for this use case of the mart, it fits its use case specifically. Additionally, just drilling down slightly further with this, this flat mart doesn't have, if you will, doesn't need to assign any primary keys or foreign keys or specify data types 'cause it's gonna be inheriting it from the job postings fact, DIMS and bridge tables. So because of that, CTA's is perfect use case for this.
13:31:40
Now, that doesn't mean we're gonna be using CTA's for all of our different data marts. There are use cases for using create and insert when you want production level quality for your table behavior. Specifically, as I mentioned before, if you need to enforce constraints or schema control or like we'll be doing in our third one, performing incremental loads where we need to have and ensure that our data quality is on par. This is important that we actually go through the extra effort
13:32:10
of creating the create and insert. So for all the marts after this, we're going to be using this create table and insert method. All right, so let's go ahead and do this, right? We create our schema of flat mart. From there, we want to create table. We'll specify the schema of flat mart and then we'll give it the name job postings. Then we need to give that as 'cause it's a CTA's. All right, so now I'm gonna save the file. Let's go ahead and actually run this this time.
13:32:38
And this is actually gonna be building this table in our DuckDB database. And so to do this, we're already in here. So I'm just going to stay within this command line right here inside the DuckDB database. And I'm gonna just run the read command. Specifying we're gonna be doing this from the '03 create flat mart SQL file. Gotta be saved. All right, let's go ahead and run it. And looks like I was run with no issues. I'm gonna just check it real quick
13:33:06
by running a select count star from flatmart.jobhostings. And we see that we have 1.62 million rows. All right, remember, we want this script to be idempotent to be able to run again whenever we rebuild it. If we're able to try to go ahead and I'm gonna click up and navigate to this and try to run this file again, I'm gonna get two different errors. I'm gonna get the calculator where the schema with name flatmart already exist.
13:33:35
And then also the table with name job postings already exist. So in order to fix this, I'll add above the create schema, drop schema, if exists. And specify that flatmart. And then for the create table, we'll do create or replace table. All right, gonna go ahead and save this. Let's try to run this again, press up. Now we got a few different errors with this and somehow the table also loaded with this. Anyway, it says, hey, cannot drop entry flatmart the schema
13:34:12
because there are entries that depend on the table job postings depend on schema flatmart. Basically what it's saying is, hey, there's a table inside of flatmart. I can't drop it because it's not empty. So what we're gonna do is we can use drop and then dot dot dot cascade to drop all dependents. So we'll put cascade on here. And then I'm gonna go ahead and save this. Now, let's try to rerun this. And when we do that, we get no issues. Also, I'm gonna come up here
13:34:45
and where we have this select count from this table, I'm gonna go ahead and paste this now at the bottom 'cause that's a good data validation script. And that actually brings us into our next portion of performing some data validation to make sure that we have everything in this table that we want. We already did this the count, right? We want this automated process, whenever we load this flat table to happen automatically. And because this I like to have,
13:35:17
right now you can see this table doesn't really tell us, like, hey, what is going on here? What table is it? 'Cause you're gonna imagine we're gonna be populating a lot of tables. We wanna know what it is. So I'm gonna modify this script up here. I'm gonna put a string of flatmart job postings and we'll give it the alias of, that's the table name. And then we'll do count and we'll do this as the record count. Now running this, we can see that we have
13:35:43
the flatmart job postings and it has 1.61 million rows in it. Additionally with this, I want to do a sample of maybe the first five rows of the table. So I'll first start with this string of, hey, this is a flatmart sample as info and then get into the query itself where we're gonna just select all the different columns from flatmart.job postings and then just limit this to 10 values. All right, let's go ahead and just run all this and see how this looks together.
13:36:11
So with this, we get the total job postings, 1.6 million. Then from there, it says, hey, we're getting to the flatmart sample and it shows us our 10 results for this. Now I do wanna add one other thing and that's up at the top to tell us that we're loading this flatmart 'cause it did take a few seconds to load. And I'm gonna put that right above where we're creating and replacing this flatmart table. And I'll say, hey, we're loading the flatmart and then we'll put this as info.
13:36:39
All right, so let's test this whole script out by pressing up and we're gonna run this.read on this file. Say I'm loading my flatmart. It then displayed the progress bar. It got into the different postings or how many it is, 1.6 million, along with that 10 row count for this. Now there's only one other thing left to do, right? That's update our master build script. So I'm gonna go ahead and put a third entry in here and that's for the step three of Creator Flatmart
13:37:11
where we do use the dot read command on that number three file. Okay, go ahead and save this. Now in terminal, I'm gonna go ahead and get out of here. And although I've tested this three or this third script, it should work just fine, but I always like to just go through and test all of these again, using this master build script to make sure everything gets built perfectly. Now if you, it took like 10, 20 minutes to load this, you feel free to skip this. It should continue to work,
13:37:39
but as best practice, you should test as much as possible. So I'm gonna go up here and copy this command that we put in here earlier for reading this script and then let's get to testing. All right, so I just got done today for some reason. I did restart my computer. It ran a lot quicker. It only ran in like a couple of minutes, but I loaded all the tables, did the initial data analysis for the quality checks of the data warehouse. And then it got into that of our flat table,
13:38:08
down below and everything checked out. So everything's working great. All right, so you know what we need to do now? We are, well, let's check our branch. We need to update and commit and then also merge. So right now we're on that feature branch as we wanted. We need to commit to that feature branch. So I'm gonna run a git add dot to get all the files in our directory. And then from there, run git commit with the M flag saying we're gonna, we built and validated that flat mark.
13:38:40
All right, sweet. Everything's looking like good with this. Now let's go ahead and merge this. Specifically, we're gonna be merging into our develop branch. So we need to switch to that. So run git switch. And then that's to our develop branch. And then I'll just double check that I am on the right branch. We're on the develop. So we'll then run git merge from that feature branch pressing tab to add that flat mark. Go ahead and do that. And so we can see with this
13:39:07
we added that three create flat mark file along with adding additions to our build data warehouse mart master build script. All right. And we do still have running git branch proper this time. We still have that feature branch. We don't need it anymore. So we need to clean it up. So we're gonna run grit git branch with a D flag, specify that feature branch and delete it. And then run and get branch just to check. Yep, all cleaned up, just got main and develop.
13:39:35
All right, so great work on this. You did an excellent job of denormalizing our star schema table into a single flat table. And now having it available in this new data mart. In the next lesson, we're gonna be kicking things up a little bit with the data mart that we're gonna be building. And specifically, we're gonna be building more of a star schema framed data mart. And I'm excited about that one. All right, with that, I'll see you in there. All right, in this video,
13:40:05
we're gonna be building further on our data mart. Specifically, we're gonna be building one around skills demand. And this will be, well, last time was right, denormalized, there's a flat table. This is gonna be a dimensional mart. So it's gonna have a fact table and two dimension tables. So for this, we're going over not only why we're building this skills mart, but then we'll also get into how to actually build it, along with making sure that we're updating
13:40:31
that master build script and sticking with our plan for maintaining branches. With that, let's jump into it. So first off, before we dive into the why, let's look at what we're actually building. So we have a singular fact table and two dimensional tables. The fact table is centered around understanding analytics for demand of this skill. Now, this fact table is an aggregation of our bigger data warehouse in that we're aggregating it not only on a monthly basis by skill,
13:41:08
but also by that job title short column. And then from there, we get some different counts based on those aggregation when to do it by. So we'll get things like the postings count, remote postings count, the health insurance posting count, and the no degree posting count. Now, because we have these values, we need to have dimensional tables for it. For the skills, we're gonna have a very similar table to our skills dim table in our data warehouse where we have skills and type.
13:41:36
And then for the monthly start date, we're gonna have a dim date table, which is actually pretty common in SQL. And we're gonna be able to look at based on a month start date, things like the year, month, quarter, quarter name, and year quarter. So why the heck are we building this skills mart? I mean, we already have this data warehouse already, which has all this information in it. Well, a few different purposes. As you can imagine, our data warehouse is a little small,
13:42:07
or I would consider it small, of 1.6 million rows. You may have data warehouses that are extreme, like millions or billions of rows. In our case, we're aggregating skills. So this is gonna be a lot faster to be able to deform time series analysis and really any analysis on it whenever we're trying to dive down to the skill level that we've done. And because we've aggregated it down at the skill level, this makes our data mart, in our case,
13:42:38
much smaller than that row level data in our data warehouse. And then the way we're gonna build this is done in a way that we're doing additive measures so they're safe to actually re-aggregate because we're just gonna be adding to each other. Specifically, we're gonna be doing counts. If we started adding averages or medians to our dimensional mart, it's gonna cause all sorts of problems. We're not getting into that. All right, anyway, I just wanna demonstrate real quick
13:43:03
the power of this specifically for the end user. So I went through and I already built this skills mark because, well, I'm an instructor and have all the notes and I can do that. Anyway, let's say we have this query. We're not gonna break it down completely. I'm just gonna show you, break down the gist of it. But let's say we have a query where we wanna find out, say for a senior data engineer, what are the top, what is the top skill per quarter?
13:43:30
Well, we can do this pretty simply with this data mart. Anyway, let's go ahead and run this on this skills data mart. And so based on the aggregation we performed, we can see on a quarterly basis, what is the different or what is the top skill for each of those quarters? Along with whatever associated count we want to go with this. Anyway, a mart like this would be really powerful to somebody that maybe would have the job of ensuring they stay on top of skills
13:44:04
and are analyzing it on a quarterly basis. All right, so let's walk through the Git workflow for this. We're gonna create a new feature branch for this skills mart. Then from there, we'll perform either one or two commits for the different feature. After we've tested it and updated that master build script, we're then gonna merge it into our develop branch. So let's get into VS code and start doing this. Inside of here, I'm gonna run git branch first
13:44:33
and see that, okay, we only have the develop branch that we're on and then main. This is where we wanna be. We're gonna now run git switch with the C flag. So then we run create this feature one with skills mart. We'll run git branch for good measure. We're on the feature branch. All right, good. Let's now create our SQL file inside of our data warehouse mart build project two folder. Well, this will be the fourth file and we'll name it create skills mart SQL.
13:45:00
And I like doing this number convention because well, it keeps it in order. And I put that zero in the front because it could build up higher than 10. In that case, I still wanted to be sorted. Similar to last time, we're gonna create a schema for this data mart. And this one would just be called skills mart. Also, if we remember, if we go back to that last one, remember we also did drop schema if exist along with that cascade. So we're gonna go ahead and just add that in.
13:45:29
So I put that up the top and then just change this to skills mart. Also put a comment at the top for step four, mart create skills demand, mart not mart. All right, so what's the plan for building this mart? Well, remember when we were building our data warehouse, you have to build it in a certain order, specifically that fact table has those foreign keys that depend on those dimensional tables checking them in order to then build out the fact table.
13:45:58
So because of that, we have to build our dimensional tables first. Now also, we are gonna be enforcing primary keys and foreign keys. And so we're gonna be doing this referential integrity check with this as we go through. Because of that, unlike last time, where remember we could use something like a CTAs in order to build our table, we're not gonna be able to do that in this case. We're gonna actually need to use create table and insert in order to build these.
13:46:28
So for all three of these tables, we're gonna, like I said, we're gonna do the create table and insert, but I find it easiest if we just build the query first, like for dim skills, build the key query that collects the skill ID skills and type, and then from there, we'll do the create table and insert into. All right, so starting with that query, we wanna get three different columns, specifically our skill ID, skills, and type. And that's all from the skills dim table.
13:46:59
Now let's go ahead and test this query out. So to do that first thing, right, we're right now we're in the SQL Data Engineering Projects folder, I need to navigate to into our second project folder, which is a Data Warehouse Mart Build folder. So now I'm inside of there. Now I can run DuckDB, specifically running it on our Data Warehouse Mart for DuckDB database. So now let's query this and the query runs successfully. We have all the different skills that are appearing
13:47:27
and it's 262 as expected. So now let's create that create table statement and that's for our skills Mart schema. And we're naming this, you can name it either skills dim or it's also a common convention to name something with dim first and so dim skills. We're gonna switch it up and do that for this one. For the values here, I'm gonna take these three right here and paste it in. And then we're gonna go through and just specify the data type along if it's a primary key.
13:47:57
Skills is a var car, type is also a var car. I'll put a semicolon at the end and now we'll get into the insert into. This is gonna be for the skills Mart schema in that dim skills table. This is gonna be using these three columns right here. So what I'll do, I'll just enter down and paste them right here, make sure they're formatted. And then it's using this select statement. All right, I'm gonna go ahead and try and run this. And so me, I got this schema with name skills Mart
13:48:26
does not exist. I never did that first portion right here. Anyway, what I think is I think the safest option is just will you do a dot read on this right here. So I'm gonna save this and we're on dot read and then use that create skills Mart SQL file that we're trying to reference. And it looks like I have a typo table with the name dim decills. And I can see I do have an error right here. I have decills and it wants to be actually skills. And I'll go ahead and save that.
13:48:54
Let's try this again. And ramming the issues. Now that we have our skills dim table, we're now gonna get our date month dim table. For this, we're gonna be basing it off of our job posted date column in our job postings fact table. And then we'll get these, well, if you will, aggregations from it, things like year month quarter based on the job posted date. So for this, we're gonna start this query with select. We wanna get the month start date.
13:49:24
So we're gonna use the function that we learned about last of date trunk. And in it, we're gonna specify that month, along with the column we want of job posted date. We're gonna do this as month start date. Next, we want the year month and quarter. So we can use extract for that. And if we specify extract, we specify year from job posted date. And we gotta make sure we spell that correctly. And we're doing this as year. Then I added in month for job posted date and finally quarter.
13:49:57
And as I finally quarter, I wanna test it now. We do have a few others that we want to run still. So I'm gonna do this from job postings fact. All right, so let's go ahead and run this. I'm gonna run shift enter. All right, so not bad. It looks like we're getting back what we want from this with the month start date, year, month and quarter. If you do note the month start date, it's repeating, right? Because you have all those different rows.
13:50:20
So this is like a job posting somewhere in January, another job posting somewhere in January. We want distinct values to just limit it down. So I'm gonna change that to distinct. And now we're gonna run that again. And now we're coming back with only 30 rows and scrolling up, okay, this is looking a lot better. I do want to order by though, that month start date. So I'll throw an order by in here and then put month start date. Running this, boom.
13:50:50
We can see that we have, it starts in 2023 and then goes up from here to 2025. All right, there's a few more columns I want to add to this. Specifically, I want to get the quarter name. So right now our quarter table just has the one, I want it to where it says Q dash one. And so I'll put single quotes Q dash. Then we're gonna do the concatenate operator. And then from there, we want to put in this quarter up here. By the way, I forgot to come up here.
13:51:21
We want to put this quarter here. But right now, if I look at this type, this is an integer. So I'll need to put this in here, and then we'll need to cast this as a var car. And then from there, we'll give this the name or alias quarter name. All right, running this to test it. It's looking good. That has the quarter now, the name now of Q1, Q2, Q3. All right, last one we're gonna do is a year and quarter combination. So the first thing we're gonna list is the year.
13:51:49
So I'm gonna go ahead and copy this right here. We're gonna have to concatenate this all. So I'm going to automatically cast it as a var car and then add in that concatenate operator. We'll then do a dash 'cause we're gonna do the quarter. We'll do another concatenate operator. And then we need the quarter. So I'll actually just copy it from up here in the quarter name and this one cast to var car. So we'll give this all the alias year quarter.
13:52:18
All right, make sure we have all of our commas as appropriate. Let's go ahead and run this bad boy. And this comes back with what I want. It has the year at the front and then quarter next after that. Now, I wanted the year first because I wanna make sure that whenever I go to, if I were to sort it by this quarter year thing, that it would sort properly. And so that's why I put year first. All right, so similar to above, we need to do the create table
13:52:42
and also the insert into for this. So I'll start with creating the table with our skillsmart schema for dim date month. I'm gonna name again once again, dim at the front of this. Then we're gonna go in and start filming the columns. First is that month start date and that is of the date type and it's also our primary key. Next is year, month, quarter, and all those are integer. Quarter name of our car, year quarter of our car. Now getting into the insert into.
13:53:12
This is gonna be for the skillsmart dim date, dim date month table. And then I'm gonna go in and list all the different columns for this to make sure that we put it into the correct order when we insert into it. All right, so not bad. I'm gonna go ahead and save this file. Let's now test this. We're gonna run that read create skillsmart SQL script. And it looks like I forgot a semicolon after this create table. We'll go ahead and add that on, save it, try it one more time.
13:53:37
And the range is fine, no errors. All right, last up is our fact table. And this is gonna be the most complicated of all the queries we have to build in order to get this. This is also gonna take a CTE because we're gonna have to get our counts for whether they're a remote job, whether they have a health insurance or no degree mentioned. 'Cause right now they're Boolean values. We need to get the count for that. And so because that we have to convert it to either a one or zero.
13:54:04
And then we'll do that in the first CT. And then in the outer one, we can then add those ones if it's true and zero falses in order to get those final counts for those. All right, so let's build that query. We'll start with our select statement. For this, we're not gonna be using the skill name, right? We're gonna be using that skill ID. And now I could do this from the job postings fact table. I don't wanna have the duplicates and have to narrow it down.
13:54:28
So I think what's gonna be better instead is, first I'll add in that job postings fact table. But then from there, I'll do that left join to make sure we conserve all those job postings, right? And we'll join in the skills job dim with the alias of SJD. Okay, so we're gonna take it from there. We'll say SJD.skillid. Next, we need that month start date. So we're gonna use that date trunk function again. Actually, I'm just gonna come up here and copy this and then paste it down here.
13:54:58
The only thing we need to pay attention to is job post-a-date is from JPF alias. Next, we need the job title short. We add that in. And now we need to convert all of our bullying flags from true or false to one or zero. So for this, we're gonna use a case when statement. So when job work from home is something like true, then we want it to be one else. If it's false, we want it to be zero. And then we'll add this as is removed as for the alias.
13:55:32
Also did this case when for job health insurance, if true. And we mark this or give this alias of has health insurance. And then finally, for the job no-degree mention of true, they're set out to one and I set it with alias no-degree required. All right, so this is looking pretty good. Let's go ahead and run this to test it. And I need to add a semicolon. I forgot to add a comma after skill ID, whoopsies. So running it again. I got a parser error after the as.
13:56:03
I realized, remember when you do a case when you need to have an end at the end of it. And I didn't have an end right there. Oh my goodness, a few mistakes now. Anyway, trying to run it again. I got quite a few mistakes. And this one, first of all, I got a parser error syntax error at near this left joint. So first of all, it's skills job dim on as or as SGD. And I never did an on where job ID is equal to the job postings fact job ID.
13:56:33
All right, for real this time, let's see if we're going to work. All right. And it did finally work. We have our table back aggregated at a monthly basis for each of these. And then it goes through and it performs the counts for this. This is-- yeah, these numbers are low for marking if it's remote health, health insurance, or whatever. It's not the total count. So a lot of these are false values. That's why it's returning 0. So I think we're fine to go now to build that outer one
13:57:03
to actually start getting counts for all this. So we're going to put this into a CTE called job postings prep as and then put it into parentheses. And then from there, we're going to use this CTE. So we're going to select the skill ID, month start date, job title short. Next up, our first aggregation, we want to actually get a count of all the job postings. So we're going to do a count function. And this is going to be doing a count star. And I think I've actually going back
13:57:32
to what I said earlier about this, the 0 or 1 right here. Now, it's important to understand with this, right? This is at the bottom, when we look at this, this is 7.48 million, our fact table of job postings fact had around 1.62. But with this 7.48 million, I'm actually realizing that this is too high or higher than what I would expect. Mainly, I went back and inspected the query up here. And right now, we're doing a left join on the job postings fact table.
13:58:03
And that's preserving, like I said, all the job postings. Well, the problem with that is we only want to look at skills or jobs with skills. And there's some jobs without skills. So technically, we need to perform it in this case, an inner join. So whenever I run this, we go from that 7.48 million rows down to 7.19 million rows. And that makes sense, because about 300,000 jobs didn't have any associated skills from what I recall. So that's more in line without one.
13:58:41
Anyway, I guess we're getting on a tangent. But the main point I was getting to this count star is going to work out appropriately, based on how we're going to aggregate this, because yes, we have 7.5 or 7.2 million jobs in there, or 2 million skills in there, that we did an aggregation for. But we're going to be aggregating it down, like I said, on these three things of skill ID, month start date, and job title short. So we'll tuck the numbers whenever we get there.
13:59:08
Anyway, from there, let's move on until we want to do a sum of is remote to get the counts of this. So we'll call this remote postings count. Sum of has health insurance. And this is as health insurance postings count. And then sum of the no degree required. And this is as no degree postings count. Also, one little minor thing I'm pretty particular about this. The column name is actually job no degree mentioned, like it doesn't mention degree.
13:59:37
So I'm going to change this from required to mentioned. And then change this one also to mentioned. And then update this column to no degree mentioned post things count. Do you want to make sure that whoever's downstream of this is sure of what data they're actually analyzing? Anyway, we're going to be doing this all from our CTE job postings prep and then we did a aggregation. So we need to do group by we can list the columns, but we're inductee beast. We're going to go group all.
14:00:06
All right, let's go ahead and run this bad boy. All right. And looking good as we can see up here at the top, we need to actually organize this by a month start date. But we can see something like senior data analyst had 312 job postings. And then out of those 32 were remote and 81 were health. And based on percentages that I know just off the top of my head, those numbers make sense. So I know that that count star is correct, which says count star right here.
14:00:33
I didn't give an alias to this name. I think I got a tangent. So I need to add that of as hosting's count. And then also I want to add that order by, we'll do skill ID month start date and then job title short. All right, let's go ahead and run this, see what this looks like. All right, this looks a lot more readable and percentable and what's going on here. It's ordered correctly and then has all the appropriate values in here. All right, looking good.
14:00:58
Now we need to do the create an insert into for this. So above this query, we'll start with our create table. This will be the skillsmark schema and our fact skill demand monthly. We'll do this as and getting ahead of myself. We're not doing a CTAs. We're just doing a create table statement. And so inside of the parentheses after this, we need to now list the columns. So went through and put all these in of skill ID as integer, month start date, as date, job title short, as var car,
14:01:30
posting count, remote posting count, health insurance postings count, and no degree mentioned posting count all as integer. Now we need to do the insert into our skillsmark, fact skills, demand monthly table. Once again, we want to specify the columns as good practice. And it's all the columns above, except with their data types. Okay, I did forget something for this create table. Remember, we wanted to set up primary keys and foreign keys appropriately. And I don't think, yeah,
14:01:59
I don't have them added to any of our tables. We actually need to, oh, sorry, I do have them up here. I have them up here for their primary keys for those two tables. So for this table, our primary keys, the skill ID is repeating, month start date is repeating, and job title short is repeating. Because of that, the unique primary key we can make is a combination of all three of those. I'm gonna go ahead and just copy these right here and then paste them in.
14:02:25
Our next up is we wanna do the foreign key. We'll start with our skill ID first. And this references our skillsmark dim skills table and that skill ID. Then our next one up is our foreign key for our month start date, where we reference our dim date month table with month start date as the column. All right, so this is looking good. Let's go ahead and save this. And now try to run it. We're gonna do in that dot read command. Go and create skillsmark.sql.
14:02:57
I already have my first error in that I can see where our create table statement. We never put a semicolon at the end of this. So I'm gonna add that in. And then I'm checking my insert into. Yeah, we did have it at the end of this. So we're gonna go ahead and save this. Let's try to run this again. And brand just fine. All right, so now that we have all of our tables created and inserted into, we're now gonna get in my favorite part of performing data validation.
14:03:25
Since we have three tables, I'm gonna take a similar approach that I did. Whenever we load our schema, remember we did this portion right here at the bottom where we use union all and basically got the count for each of those tables. And then after this, we do basically a five row sample of each of those tables. So first I start by selecting the skill dimension in quotes as table name, the count as record count, and then of our dim skills table.
14:03:53
I do a union all and next we get to our dim table or a date table, look at the count for this and we're doing this from dim date month. And then finally our skill demands fact tables last, doing the count of this. All right, let's go ahead and run this. And to have a typo in this, it says table with name dim skill does not exist. That's because I misspelled it. If you have to add an s on the end, let's go ahead and try again. All right, we can see these values.
14:04:18
Yup, this makes sense, all of these values, 262 in the skills dimension table, 30 in the date month and about 52,000 in the skill demand fact table. Next up I query the first five rows out of the skills dimensional table. I do the same type of query for the date month dimension table. And then finally from the fact skills demand monthly. Okay, let's go ahead and query these three. And they all came back good. We saw the skills down, date dims looking good and also our fact table.
14:04:47
All good. Now, similar to what we did for those other tables, I'm gonna just put these other messages when we're loading each one of the tables. So in this case, we're loading the skills dim table. This one added for the date dim table and then this one for the skill fact table. All right, all buttoned up and clean. It's part of good measure. We're gonna run read on this file. And it goes through, tells me it's loading all the different tables
14:05:10
and then provides the key statistics about the table along with those final queries we're looking at. Everything looking good. All right, so as always, we're getting towards the end now. We need to update our master build script. We're gonna add in that fourth line or that fourth step. In here, I put the step four for creating the mark and then added the file in using that dot read command. All right, go ahead and save this. Once again, I'm pretty OCD
14:05:39
and I like to test every single part of the way. So I'm going to test it one more time for this entire build script. So I'm gonna quit out of DuckDB right here and then come up to the top and grab this line that we've been saving, paste it into our terminal, get to building our data warehouse and data marks. All right, so that took about a minute to run, scrolling through this all. We're not gonna walk through everything again. Scrolling through it all.
14:06:05
I didn't get any error messages. So I know that it built correctly. All right, looking good. Also, not just looking for error messages. Also looking to make sure that any time you have rows that you're querying, that they actually return rows, along with their associated accounts. Anyway, first thing we need to do is now get into committing these changes to our feature branch. So we're gonna do get add to add all these two files in their additions.
14:06:32
Then we'll run git commit with the message of build skills demand mark. Okay, that's updated good. We'll run git branch and we are on that feature branch. So we're gonna now get switch to our develop branch and run git branch for good measure. Before we do this, okay, we are undeveloped. So now let's do our git merge of that feature branch into our develop branch. Not bad, it looks like everything was added. These two file changes were added into it.
14:07:01
All right, so you know what we gotta do. We gotta clean this up. We no longer need that feature branch. So I'm gonna run git branch and then a deflag to delete our feature branch. Running git branch again. Yup, it's deleted. We have main and only develop. All right, great job on building this skill mark. This is a great exercising in demonstrating your abilities to perform dimensional modeling. Just hopefully get your head around
14:07:27
how you may have to build different types of data marks or even data warehouses for your stakeholders. All right, in the next video, we're getting into our third and final data mark. With that, I'll see you there. (gentle music) All right, welcome to the final mark to build. And this is gonna be similar to or based off of the priority mark that we built back in that DDL and DML lesson. This mark is gonna be an excellent demonstration
14:07:59
of your skills in performing things like batch loading or incremental loading, as I've also called it. Anyway, for this lesson, we're gonna be going over why we're building this mark. Then get into actually creating the mark and then we'll create another script that actually performs those incremental updates to the mark. And then finally from there, we'll update our master build script, commit and then merge all our branches. (gentle music)
14:08:25
So as a quick refresher back in that lesson on DDL and DML, we built out a priority roles table and also a priority jobs snapshot table, which is based on merging, well, initially combining job posting as fast company dev and priority roles into our snapshot table. And then from there performing incremental updates with merges. So for this lesson, we're gonna be building out that priority mark and it's gonna contain those two tables, priority roles and that priority jobs snapshot
14:08:58
which the prior job snapshot is obviously based off of from our data warehouse and that priority roles table. After we've created those two tables, we'll create another script that will go through and simulate updating priority roles and then performing that incremental update to priority job snapshot using merge. Now these incremental updates or what's also known as batch loading is super popular in the industry. As you normally have something like batch loading
14:09:25
or you have streaming, which instantly puts the data in. Anyway, data analytic pipelines need to be updated at most daily for stakeholders or business leaders to make decisions, they don't need streaming data. So batch uploads are super frequent in data engineering. So that's why I think it's super worthwhile for you to include this in the project. All right, so what's gonna be the Git workflow for this? Well, first we're gonna create a new feature branch for this priority mark.
14:09:58
Then from there we'll go through the process of building out the scripts for it. And when we're done to everything, including the data validation, we're then ready to merge it back into our develop branch. All right, so let's start by creating that feature branch. We use git switch with the C flag and then name that new branch feature slash priority mark. We'll run git branch, double check. And yep, we're on that feature priority mark branch.
14:10:23
Now for this one, we're gonna add two files. Both the files we're gonna need for this. First, we'll be adding the create script and then I'll start with O5 and then say create priority mark dot SQL. So that's gonna be our create script. We also need one for our incremental updates. So that will be O6 and now the update priority mark dot SQL. All right, so let's go ahead and get started. We're starting in that number five file for creating the priority mark.
14:10:48
We'll start with a comment that this is step five for our mark create priority rules mark. Like all of these, we're going to be using a schema for the mark. So we need the create schema for priority mark. And then we want it to be adding a potent. So we'll also use that drop schema. It exists priority mark cascade and that cascade is to drop the schema if there's tables inside of it, which there will be. All right, so the first table to create is our priority roles table.
14:11:15
And we can go in and dig into our notes for this one, but this one's honestly so simple and we also did a lot of manipulation with it that I'm just gonna build this from scratch. So we're gonna create table first and this is in the priority mark schema called priority roles. We'll have our role ID, it's an integer and it's a primary key. Role name is next is a Varkar. Priority level is last and it's an integer. From there, we're gonna just insert into it directly
14:11:43
the values we want to put for this. And this is for the priority mark for our rules table. And as best practice, we'll go ahead and list out the columns of role ID, role name and priority level. Then we'll list out the values we wanna put into this. First, we'll be data engineer and we'll give it a priority level of two. Second, we'll be a senior data engineer with a priority level of one. And last, we'll be that software engineer with a priority level of three.
14:12:08
Up ahead here, I'm gonna go ahead and create that query that displays that we're loading this table of loading roles for priority mark. Also, I'm gonna just go ahead and do the data validation for this of a select star from priority mark priority roles. All right, this is looking good. Let's at least test our script. There's enough lines in here. So in terminal, we'll start up DuckDB, also make sure that we're navigated into that project two folder to start with for this.
14:12:34
But we're gonna just go with the command DuckDB and then specify our data warehouse DuckDB file. Then with our error, I'm gonna run a .read command of our O5 create priority mark.sequel script. And to have, well, two errors. First off is the insert into, well, mainly I forgot to put a semicolon after our create statement. And then I think that caused our second error of table name with priority roles doesn't exist because I never got executed.
14:13:02
All right, make sure we save that file and press up. We run this. All right, looking good, loading roles for priority mark and then all of them appear right here. Good. Next, we need to build out the priority jobs snapshot. For this, we're gonna go into our notes that we previously had and I'm just gonna copy and paste and adapt as necessary. For this, I'm going into, I named it priority job snapshot.initial under my 1.2.4. And this has the create statement.
14:13:29
It also has the insert into and conveniently has the little data validation at the end. So I'm gonna just copy this all. Inside of here, I'm gonna go ahead and paste it. And now we need to go in and what's called as refactor it because right now the schema is wrong on here. It's meant to the main schema and that's not correct. We wanna actually call that the priority mark schema. So I'm gonna go ahead and copy this. We're gonna place the main in our create or replace table statement.
14:13:58
We'll also do it in the insert into. And then down here, the job postings fact is in the main schema. So we don't need to specify it. So I'm gonna remove this data jobs. Also, I'm gonna remove it for the company dim. And our priority roles is no longer in a staging schema. We're gonna update that to also be in the priority mark. The only thing last thing is down here for our data validation. We're gonna go in and update this to say, hey, from priority mark.
14:14:27
Okay, one other thing to do. I wanna get a table that says we're loading this priority job snapshot. So I'm gonna paste this here and say, hey, loading snapshots for priority mark. Okay, I think we're good. We're gonna test it out. Inside of terminal, I'm gonna press up. We're gonna read this file after we saved it. And it looks like it went through. We got data engineer, senior data engineer and software engineer. And then for some reason in our table,
14:14:51
we only got data engineer and software engineer. So now there's some investigating, found out what the issue is. Issues here. We're using, so the inner name in here, I misspelled senior data engineer. It's supposed to be, well, I have a good near. I'm gonna add the end. Now, this is a problem with the database design to be completely transparent with you. In that we're using this priority roles table, right? And so we're relying on whoever manages this,
14:15:20
they spell the names exactly right. 'Cause then in our priority jobs snapshot, in order to get the correct roles, we're matching on a name. We're literally doing job title short from job posting facts to the role name. I would say this is not necessarily best practice. Normally what you would expect is some sort of, like I said, ID here, some sort of integer numerical value that there'd be less likely of an issue of somebody fat fingering it.
14:15:51
Unfortunately for us, our job title short column does not, we didn't do a dimensional table for it like we did for the companies. So if we did, that was how we would probably, that's actually his, how we would link the role names. But since we don't relying on this method, this is not necessarily the best, but definitely something you should be aware of. Anyway, thanks for coming to my TED talk. Let's actually, now that I've saved that, I'm going to read this file
14:16:16
and whenever we rerun this now, it has all the appropriate roles in it. So we have our create table script built. We're now going to move into our incremental updates. For this, we're going to first start by in our script simulating that the priority roles table gets updated. As we've talked about in that lesson, it could get updated as much as daily. So we'll do a couple of different changes in there to that priority roles table. And then from there, we're going to modify
14:16:47
or refactor our merge code in order to incrementally update our priority jobs snapshot table. So let's get started in that six update priority mark SQL file. We'll first start with adding a comment. This is step six for our mark update priority roles. And then we're going to be doing two different things here. In that we're going to update the data engineer to priority one in a priority roles table. And then we're going to add data scientists
14:17:13
as level two of now, let's make it level three. So for this, we're going to run an update. And this is going to be in our priority mark schema for priority roles. We want to set priority level equal to one. And that's where role name equals to data engineer. That'll be good enough for that one. From there, we're going to insert into that data scientist to our priority roles table, specifying those columns of role ID, role name and priority level.
14:17:42
And then we need to specify the values. That'll be four data scientists and of the level three. After this, I want to do some data validation to make sure that I have the correct priority roles table. So I'm going to just come into that five script and I'm going to go ahead and just take this select star from priority roles. Also up above here, just as a statement, I'm going to modify the statement up here in that it's saying updating roles for priority mark as info.
14:18:10
And then after that, I perform the query. Now, important thing to understand with this, right? This script, it will be, if you will, idempotent in that whenever we're building our master build script, as long as we run five before six, you know, we can form this update and insert into. Now, if we try to run six alone over and over again, it's not necessarily going to be true because we're going to be doing that update and insert into. And so those will have already been performed.
14:18:37
But that's not the point of our idempotency. We're trying to make it to where we can run it with the master build script. So that's just something you need to be aware of right now while we're going through this design phase of it. Anyway, let's test this. So in here, I'm going to go ahead, so we're going to run a dot read and we're going to do this on 06, update priority mark dot SQL. Go ahead and run this and we have our first error and that I forgot to put a semicolon
14:19:02
after our insert into, okay. So now if we try to run this again, it works. And then if I try to run it one more time, I'm going to get an error, right? Roll ID for violates primary key constraints because that's a primary key. And we can't add data scientists again. Anyway, not a big deal, right? We just need to remember to run 05, create priority mark and then run 06, update priority mark, every time we do this to maintain that idempotency.
14:19:35
All right, so now we want to add in that script we use to incrementally update priority jobs, snapshot, performing the merge. We also used a temporary table in addition that merge. So let's go find that. So for you, that should be in under 1.2.4 priority jobs, snapshot file, and I'm going to get this temp table. We're not going to use that update insert or delete statement that we generate, but instead we're going to get this merge into, and then also this final check query.
14:20:08
We discussed about the use of CTAs, but we're not using CTAs for this, we're using that merge into. So I'm going to go ahead and just copy this all, paste this in, and then remove this commented out portion, 'cause that's just going to confuse people on what was happening here. All right, so now we need to go in and refactor this. We'll start with our temp table first. Job postings fact is in that main schema, so we're going to remove it here, and also next to that company, Dim.
14:20:36
The last place to update this is right, we don't want staging, so I'm actually going to just copy priority mark up here, 'cause we're going to do this a few times, and we'll place staging priority roles with priority mark, priority roles. Okay, temp table is updated. Moving into the merge into, we're not merging into the main schema, we're doing it into the priority mark schema, and scrolling through, I don't think there's anything else actually to update,
14:20:59
except getting down here to our final check query, or data validation, we need to be the priority mark, priority job snapshot. Okay, not bad, I do want to dress this up a little bit. I had this line here above the temp table for creating temp source table for priority mark, and then above our merge into, I put a statement of batch updating priority job snapshot for priority mark. Then from here underneath, we have this final check query. All right, let's go ahead and save this,
14:21:28
and let's test it out. First thing we need to do is run that number five script of creating our priority mark, and then we're gonna run our sixth script of updating the priority mark. And so this looks like it performed just fine. We said, updating roles, priority mark, it added data scientist, update data engineer, it then created temp table, and then it performed the batch update, and then we can see that the software engineer and the senior data engineer were performed
14:21:55
at different times than the data engineer and the data scientist, which were after the fact, because we had to update the level and update, we'll add the data scientist. So this checks out. All right, so both these files are looking good. Make sure they are saved. There's only one last thing we need to do now is update our master build script. In here, we're gonna update step five for our create priority mark, and then our step six for updating our priority mark.
14:22:27
Go ahead and save this, and then just for good measure for me, I like to just run this entire script. So in terminal, I'm gonna quit out of here, and then I'm gonna run the command that's in the top of our script right here, 'cause I saved it up here, paste it in, maximize this. Now let's run this. All right, so not bad. This took only a couple minutes for me to load through, scrolling through the logs. Everything looks like it's adding up, so I don't have any issues with this.
14:22:56
Pretty impressive what we built so far, actually what we built. All right, so you know what we gotta do now, we gotta commit these changes, and then merge the branch. So I'm gonna do git add dot to add the current directory. There are stages, the changes are all staged now. So we're gonna commit them with the message, build priority mark, and update mark. All right, so this adds all our files. I'll just double check, if I should have done this before,
14:23:18
what branch run, we're on the feature branch still. We need to merge this into the develop branch, so we're going to switch to that branch. So I'll switch up to develop, then from there we'll run git merge with the feature branch. All right, and we can check our little graph over here and see that we're build priority mark and update martyrs in the develop branch, and that is in line with that feature branch. We don't need the feature branch anymore,
14:23:45
so we're gonna run git branch with the deflag for delete, and then list that appropriate feature branch. Go ahead and delete it. It was deleted, run git branch for good measure. Now, we're actually not really done because, so we did go through a good job so far. We built this data warehouse from our CSVs, and then we built our three data marks. If you notice from our diagram, right? We said we're gonna get our job posting warehouse and our marks onto mother duck,
14:24:17
so that way we can serve this to other customers like I then now access this in mother duck via Excel, Power BI, Tableau, or Python. Anyway, how the heck did we do this? Well, actually, it's pretty simple. So, previously we run this DuckDB command, and we've been specifying our data warehouse, smarts.duckdv, that we wanna run it locally. All we need to do now is change this to mother duck on where we want to build this. So, for this, I'm gonna run DuckDB,
14:24:45
and then do mother duck is where we wanna run it, and I'm gonna call this data warehouse, smarts. Then once again, we'll run that C flag, and do that .read command on build data warehouse, smarts.sequel. Okay, let's go ahead and run it up here. And so in me, I got this error, unable to open database data warehouse, smarts. Basically, the gist of this is that the database is not created, we haven't created it yet. Anyway, if I were to go to DuckDB,
14:25:14
and just do MD colon, and then run.databases, we have MD information, sample data, data jobs, jobs, smart, which we created at one point in my DB. We haven't created this location of this data warehouse, smart that we wanna name here. So because that, we just need to run a command of create database, and then what we wanna name, and in our case, data warehouse, smarts. Now when I run .databases, we can see that we have at the bottom of this, data warehouse, smarts underneath it.
14:25:47
All right, so I'm gonna quit out of this. And now pressing up to go back into that command, we're now gonna run this. And right now it's going through the loading information. I'm gonna go ahead and maximize this as it's going through and keeping track. All right, so this went through, and just skimming through all these different commands. Everything worked exactly as it was supposed to. It looks like there's no errors. So completely good to go with this.
14:26:13
And so if we navigate on to Mother Duck, we can go in, inside of your login here, I can go to attach databases right here, and I can see the data warehouse smarts. In here, I see our flat mart, our main mart, which is our actual data warehouse, our priority mart, and our skills mart. If I want to, I can create a new notebook and inside of here, we can query our data warehouse smarts table, query in there maybe even that job postings fact table, and everything appears just fine.
14:26:47
Anyway, with the main point of showing you inside of here is this, we can go here to the data warehouse smarts, click these three dots right here, and we can go in and actually share this. First, you need to specify who has access. You can do anybody in your organization, if you're working on an organization. Anyone with the Share Link, that's how I shared the initial database at the very beginning of this course, or you can even do specified users with the Share Link.
14:27:13
From there, in this case, I selected anyone at the Share Link, I have it to where it's auto updating. So if you were to have new data populate, and then you were to refresh this data inside of here, your end users will get that update. Then from there, you can change the share name as you want, I'm just gonna keep it as data warehouse smarts, and then from there, create share. Now, all you have to do to get it out to your stakeholders
14:27:40
is give down this attach command similar to what you had, wherever I started at the beginning of this course, they give us a attach command, they run it inside of mother doc, wherever they're at, and they'll have that database on their attached databases. You can also navigate to any of your shares later, down here at the bottom by going to shares I've created, and right now I'm here to have data warehouse smarts, click on this, and then you go to Copy Share Link
14:28:03
and re-copy that link whenever you need it. So we have actually finally satisfied this entire data pipeline, building it from CSVs and their data warehouse, and then our three data smarts. Congratulations on this. Now, if you're feeling spicy and you wanna get more experience building another dimensional mart, I've put together a company mart that's a little bit even, well, a little bit more complicated than our skills mart.
14:28:35
You can access the final script to see what to base it off of and how to actually build it out inside of my project to folder in GitHub. Inside of here in the files, you just navigate to this number seven one of create company mart, and this has all the different code needed in order to generate this bad boy. This thing is a behemoth. Now, for those that purchase the course practice problems and notes, I have a specific lesson page on this. In this, it breaks down what the heck
14:29:04
this company prospecting mart is, explains all the different tables of it, how we're gonna build it out, and then walks you through step-by-step on what you need to do, what commands you need to run in order to build this out and why we're actually doing that. All right, so awesome job on this pipeline. Now that we've had this built, we need to get it out and get it documented properly and to read me and then from there, share it. So the next step we're gonna be jumping into
14:29:31
is building out the read me for this project. With that, I'll see you there.
14:29:39
In this video, we're gonna be putting together our read me for our project too. We're also gonna need to update our read me that's on our entire repo of that sequel that engineering project or I mean, projects. Anyway, remember we have two projects in there and many of them have two projects. the purpose of this entire repo is so you can add additional projects as you build them to this repo. Anyway, for this, I'm not going to handhold you as much as I did in the first project is
14:30:11
you should have the basics of how to work with Markdown and write in and build out or read me. I will be walking you through the sections that I do recommend though and I'll briefly cover some core topics or core things that you should cover in this. But I can't trust this enough, you need to build this, read me and adapt it to your own personal need. If you're say looking for a specific job like an analytics engineer role, you need to frame it more towards analytics engineering.
14:30:45
I'm going to be just going at a general approach, something as a data engineer.
14:30:52
So like I said, we're not going to be going through and actually going through any of the syntax again for how you should be updating or writing in Markdown. I did include this link for this Markdown cheat sheet, which I find is a great resource in how to build this bad boy out. Additionally for this read me build, I'm going to be using specifically in this one a lot of different pictures to not only show the pipeline, but also for the data warehouse or the flat
14:31:17
mark and all these other different pictures. Well, you can steal, I mean, use those pictures if you want. If you navigate to the main repo inside of here under the resources slash images folder, I have all the different images that you need within here. Right here is the project to data pipeline. And all you got to do is come up here and download the raw file. And then from there, just take it and put it into your image folder on your repo. I ended up just moving all those
14:31:46
different images into here. All right, for this also, I do want to call out we are on the develop branch, which I'm fine with now, right, we've done all of our features. Now we're just updating through a little documentation. So perfectly acceptable to do this on develop branch. We're going to merge it into main at the end. So let's get into building this out, I'm going to navigate into our project to folder and update or open our read me conveniently. And here we left this to do
14:32:10
of fill this in. All right, so there's a few core sections that I want to do for this and it's going to be very similar to what we did in the last project. Actually, I'm just going to take that last project. And I'm going to, I know it started here at this executive summary, take all of it the contents, paste them in, and then delete all the different sections within it, keeping the headers. All right, so this gives us an idea of the sections we're going to need for
14:32:35
this, and we just walk through briefly what I've included in each. First thing I did was a one-line sentence, basically explaining what we did with this pipeline, and then included an image of the pipeline. So it really captures their attention of what's going on here. Next, I got into the executive summary, focusing on key things that I found are most important for data engineering, such as pipeline scope, data modeling, ETL development, and then the mark, mark architecture,
14:33:03
how we use to like a flat mark, a dimensional mark, and also a automatic updating mark, or better said incremental updating mark. Next, I got into the problem and context. I started with the challenge first, explaining why we needed the data warehouse, and then why we need each one of those marks. And then from there, I talked about the solution of building this end-to-end pipeline that extracts the data from our CSVs into those locations. Next is the tech stack. And this is
14:33:32
very similar to what we did last, because our tech stack hasn't really changed since project one. But I did add in here for the language that we have, we're using DDL for schema design and DML for data loading and transformation. Oh, and then also these other two of the automation of a master SQL script for pipeline orchestration, and that we use Google Cloud Storage for source CSV files. Now, instead of analysis overview, we're going to change this a bit because we're not doing an
14:33:59
analysis here. Instead, we're going to go in depth with the pipeline architecture that we did. I start this section with the overall architecture itself, the image we used above, and basically explain it in a couple sentences of what we did. Next, I go into each individual database or if you will data warehouse a mark that we built starting first with the data warehouse. And with this one, I link to the appropriate SQL file that we did to, in this case, create
14:34:28
the tables and load the schema. And then from there, explain the purpose and the granularity that this schema goes to. Next up is our flat mark, which demonstrated our stability to denormalize dimensions. And I included attachment to the file we used for this along with purpose and grain. I did the same with that skill mark, which performed that time series skill analysis with additive measures, defining the file purpose and grain. And then finally, that priority mark,
14:34:57
which demonstrates our ability to perform incremental updates or batch processing. For this one, remember this one had two files, so you need to link those appropriate ones. And also went into purpose and grain. Now, if you went ahead and did that company mark, go ahead and also list that here, along with the appropriate SQL file and purpose and grain. Last up is getting into the SQL skills demonstrated for this. And I'm actually going to reformat or reframe the title of this
14:35:27
section to data engineering skills demonstrate, we're going to actually change these sub sections as well, because we did more than just SQL in this, we did a lot of other things as well. For me, I broke it down into three main sections. The first was around ETL pipeline development, talking about how we did ETL, along with that demonstration of the incremental updates and how we used our master master build script to orchestrate it all. I then focused on our ability to perform
14:35:55
dimensional modeling, because we not only did that with our data warehouse, but also with our skills mark that we built. And then from there, moved into advanced SQL techniques that we did, we covered a lot of SQL concepts and a little second half of this course, specifically around DDL and DML. So we need to make sure that we capture that here. And sorry, we did cover a lot of stuff. So we actually, I made a fourth section around data quality and production
14:36:22
practices, and that how we actually designed all of our scripts to be idiom potent, to make sure that we perform our referential integrity checks, we're ensuring that our data is meeting a specific type, want to capture that all here. All right, and that's it for this read me here. So I'm going to go ahead and save this. This is now good to go. And we just have that one more script to update
14:36:46
our one more read me file. Remember, we have our project one and our project two, and then we have our main read me. And this read me directs them to the appropriate project, depending on what skills you're trying to showcase. Remember that first project was around eda exploratory analysis. They know here to go to that one, we needed to do the same thing for a second project. So for this,
14:37:07
I left the same structure and where it's a clickable link up at the top to direct them to the appropriate folder. Make sure I click this. Oh, it's not even it's not working. And it's not working because I spelled the file path wrong for this should be data warehouse Mart build. Let's go ahead and try this again. Click it. All right. And it's navigating me to the explore pains opening up and it's navigating time. So it's going to work fine and get up. Make sure all your links are working
14:37:35
appropriate. Nothing worse than some future employer comes to take a look and your links are broken. It shows that you don't have attention to detail. Anyway, besides that include a picture to try and dice them to go and read into this more. And I just conclude a simple description underneath it along with the skills demonstrated. So I'm going to keep it simple with this one. You can feel free to dress this section up more if you want to include more appropriate skills or whatever on
14:38:02
this you can. I really just leave it up to you. All right, so most exciting part we're going to now get into actually committing all these changes and merging this into main and getting this on to GitHub. Make sure that you've gone ahead and saved all your files, specifically those readme's that we recently just did for this. I'm going to go ahead and list good branch to show a branch run. We're on the develop branch. We need to get this into the main branch after committing it. So we
14:38:30
run git commit and then m oops getting ahead of myself. We're going to run git add period for add all files in this directory. They're all staged. Oh, that's why I'm not in. So I added only that in that this directory. And it only added the readme from our project two folder because in the terminal, I'm in the project two folder. I need to actually go up to the main directory and now run git add period for the current directory. Everything gets added in. Now we can get into
14:39:03
committing our changes with the message of add readme with project documentation. All right, seven files changed added all these in is looking good. We're going to run git switch to main. All right. Anytime we're about to merge in new changes, we need to make sure that we have everything appropriately pulled in from our origin up in GitHub. So we'll run git pull. And it says already up to date. Remember, we could run git pull origin main, but we've already specified at
14:39:34
the beginning. And it is not necessary to specify again. Anyway, we're going to run git merge. And this is from our develop branch for project two, exciting times. All right, so this added in all of our different SQL files for building out the smart along and their data warehouse, along with our readme files. And then those images, if you stole, I mean, took them to use in here. Now, because of that, you know, what I like to do for cleanup, we need to get rid of that develop branch.
14:40:03
So we're going to run git branch with the D flag, specifying to delete the develop project to branch. All right, and looking good running git branch, just check. Yep, it's all up to date. Now we're ready to push. So we can just go get push and bam, GitHub now should be updated. But you should always go check it, make sure it is key things. I like to check with this. Here, I'm verifying, okay, everything looks like it's up to date. I do go through and still double check clicking
14:40:31
the different links. Does it navigate to where I want to? Are all of the images also appearing? And it's looking like everything is good. Nice. All right, so we have it updated on GitHub. Now we only have one step left. And that's to get your project shared properly on LinkedIn. That's what we'll be doing in the next next lesson. All right, with that, I'll see you there. All right, congratulations. This has been nothing short of your hard work put into building out
14:41:07
this entire data engineering pipeline. And so we need to showcase it. In this lesson, we're going to be going over, well, this is our final lesson actually. In this lesson, we'll be going over how to add your certificate. If you purchase the course practice problems, you'll be getting a certificate. And then also updating your project session and a suggestion of making a post to showcase that you have both these things. All right, first thing
14:41:34
primary secure is about is that certificate of completion. After you complete the end of course survey for those that purchase the course practice problem, there's a survey at the very end. After you complete that, I'll email you your certificate with your name on it. So you'll get it that way. This will have a link associated with it along with you can download it for this. You're going to navigate to your profile on LinkedIn and then go to your licenses and certificates section to go
14:42:00
ahead and add this in. Note if you don't have this section, you're just going to come to the top, go to add profile section, add licenses and certificates. Then from there, you're going to go ahead and fill this out for it for this. You have the name of SQL for data engineering. List me as the issue in organization. When the issue date is there's no expiration date. Then from there, paste in that credential ID with the certificate along with the credential URL that links to the
14:42:23
certificate. Then under skills, I've listed five of the most important I found of SQL data engineering Git terminal or GitHub. Then from there, you can also do add media if you want. You can add a picture of the certificate just as a nice to have. Anyway, this looks good to me and we'll go ahead and click save. Next up, we're going to add the project itself. So under project section on your profile, we're going to go ahead and click add. For this, we need to fill out all these sections.
14:42:50
Don't reinvent the wheel here. Use anything that you've used from your read me. Just repurpose it for this. In this, I include the project name, the description or the one line sentence that we use at the front of it, along with those five core skills. Next up, it has add any media. For this, I'm going to navigate into our original projects folder. Then I specifically want to link to our second project. So I'll navigate to that. Oh, I want this one. I'll copy the link here. And then
14:43:18
in LinkedIn, I'll go to add media, add a link, paste in our GitHub here, click add. And it looked like in my case, it couldn't do it. So I just went to the root of this SQL data engineering project. And then in the description said, hey, navigate to project two in the read in this repository, click apply. And then from there, include your start date and finish date. I started this almost six months ago building this. So I hope you found it worthwhile. Hopefully it didn't take you six months
14:43:44
to actually complete though. And then if you've worked with anybody along this, feel free to add contributors. Okay, everything looks good. I'm going to go ahead and click safe. Now, the last thing to do is to create a post. And this is to broadcast out to the community that you've completed this and also start getting eyes on your projects. Anyway, Carol did a great job of this. And that Carol recently completed it and not only posted their certificate here on LinkedIn, but also
14:44:11
walked through the different parts of that Power BI course and what they did. And so you can do the same calling out those key skills. And then from there at the bottom included a link to the project. Now also don't forget to tag me in it. As I love to be able to get notified when you're doing this and be able to comment on it and check out your projects. Alright, congratulations again for building out this behemoth of a project that has been nothing short of your hard work. So
14:44:41
congratulations. I'm planning on building out a lot more data engineering courses. This is the first one in my series. I'm hoping to go next in things like dbt Python and then more narrow it down further to such as airflow and spark once I have those. But this all depends on if people are interested in it and if they're taking the course, so make sure that you're sharing this in order to get the word out and actually help with funding future courses. Alright, and with that,
14:45:08
I'll see you in the next course.
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat