Skip to main content
Loading Explore guides…
Big Data Analytics Overview · Acedra
Home
Explore
Computer Science
Big Data Analytics Overview
Sign in to save
Chat
If YouTube asks you to sign in, open the video on YouTube.
Watch on YouTube
Reload player
Transcript
1,410 segments
Copy
Hide video
0:00
Big Data Analytics helps organizations harness their data and use it to identify new opportunities Dad in tone leads to smarter business booms more efficient operations higher profits and happier customers Understanding this we have come for this tutorial on big Data Analytics Now before we go ahead with session I'd like to inform you guys that we have launched a completely free platform called US Great Learning Academy But you have access to free courses such as
0:28
the iCloud and they're still marketing You can check out the details in the description below Now let's have a quick glance of the agenda we started by understanding the concept off Big Data and Hadoop Then they will learn about hype and then finally we will learn about spark So today we will be covering big data What exactly is big data on Then we will look at something called a Hadoop ecosystem and Hadoop architecture Basically So these things are the foundation
1:07
for all your machine learning statistics spark anything that you're learning The foundation is this So today we will be understanding our big data and Hadoop ecosystem will ensure that we go through all the basics and all So my name Is that a good ramen You can call me I have been in the night industry for about 12 years Says off now And I have been working on big data platforms for about seven years 7 to 8 years from 2011 onwards I started on big data I trained people on ha Do spark
1:38
You know the big data Typical tours that we use I trained a lot of people from cooperate Background I'm a regular trainer Flip kart G e M c on and many other companies the definition of big data If you Google you will get first thing You can just go to Google and say What is big data You will get turns off articles beauty of some presentations That is not what I'm going to teach you Alright What I'm going Going toe tissue Is that Let's take a practical use case Right Let's take
2:07
a practical use case and try to understand how big data actually makes sense Right So first of all big data is related to i t So you need some level of fighting Norris to be fair right This has something to do with fighting I will share my personal experience Okay So when I started working in 7 4008 and dollars Okay I was working with a company in Bangui on we had a creator and application A sales application So basically what we do if you installed this application there was a retail company
2:41
The customers will import all their sales They time that you know So they will say that today this much quantity eight I'm soared
2:47
soared and soared and soared on this application will capture or the data Okay and start it now That was a very simple word We did not have mobile phones We didn't work hard Facebook WhatsApp Nothing Was there 2005 right on vehicle using an rdm a system to store this You know what is an RTV amiss Okay at least you are good at that Right So this is your culprit We're not going to use this game Big data Okay so our devia MERS is what we very using two traditionally stole the data
3:17
right on What is an RTV Um a system It has a row corland format You have a table you insert the data Now if somebody is teaching you are Libya must they will say that Al Arabiya Mrs Great It can store everything and that's all fine But unless you start their data starts increasing Right So originally arguments can store everything I don't have any I'm not against arguments or something Even for transsexual management We use RGB amiss There is no doubt about it My sequel Oracle
3:47
We were using my sequel actually So whatever the customer is inserting this my secret will capture it End off the year you do some processing You will understand how many items were sold Everybody is happy Happy This is what was happening in 2005 Fast forward to 2000 Lane 11 I am working with I C s a bank I see I see a bank want to migrate to big data platform So 2011 we run a project in which we my greater
4:12
the entire ICSC Bank into big data platform Because I was part of that project I was not doing everything in that project So what is the problem It icsc a bank right So that is a question Why I say say bang want to go to big data right now This example will also help you to understand the I p world right So for people who are not coming from that background they might need to understand how things are working Right So we've even place the a c a bag We spoke with them He said Hey what do
4:40
you want They said you know what We're facing a lot of problems We want to go to big data We want to goto How do we'll see What is Hadoop later So we asked what is a big deal What is your problem So there were a lot of problems which they were facing were not The problem is that if you look at this rgb um a system right Your typical rtb on my sister What is our WMA systems doing They will store the data in the form offer table right now If your table sizes very very small
5:10
it's not a big deal Even if you look at my sequel or if you go to enterprise scale oracular and are you know they can store gigabytes and sometimes upto terabytes of data But has this size becomes big So let's say I ask you okay You want you have to store the data You know that the data is keep on increasing every week You are not ready to migrate anywhere or actual If you're talking about or aka they have stored it's separate and B B M s engine separate I don't know how many of you have worked
5:40
extensively with Oracle Your ankle What I can do I can purchase Oracle on I can add stories boxes as many as I want
5:50
you know So these boxes will be the network So my database engine will be connecting Tow them and storing the data whatever daytime giving So you can argue saying that I can increase the stories capacity on then All right I'll can store as much data as you want There is no doubt about it right But there is also a limitation for the stories One problem second problem is that as this box increases the storage in Oracle increases You have to divide your data you have to
6:20
partition your data So let's say for example you know you are working with Oracle Okay so let's say you are working with Oracle Onda Let's say we're looking at
6:30
iPhone saves I formed right So you are selling iPhone left So you're working with apple right So one of the example is that if you are creating tables to store iPhone data typical if you look at the schema and all what we will do is that there will be one table OK will be one table wherein you will be having properties like you know how many items were purchased at what rate never purchased How many people are barred so on and so forth So this will be a right um stable soaking
7:04
the items table You will have all their data later toe the iPhones who were selling right You will also have another table but they will be the user data
7:14
You would also have that user later User data means which customer board from birth keyboard or this data this Iscar normalization You know what It's normalization in our DBM A systems in orderto avoid duplication off data What we do is that we normalize the data We always keep their data in different different tables on whenever you want to get the full data What you do joints right So if I want to get both the data from this I have to do something called the Joy Inquiry I I fired a joy inquiry
7:48
on all the data from here Welcome And join and produce me the output This is good if you're having small amount of data What if you have terabytes of data here and here My join query will fail right Or the joint could He has to wait Now Another point that people say is that if your idea Mrs becoming bigger and bigger why don't you partition your data That is possible I can have like four boxes My one table Okay If this is my table right it can be logically partition and kept
8:18
on four missions If it is beginning bigger and bigger Okay this is possible Let us something card
8:23
Logical partitioning Look at iPhone data I can say all the data
8:29
where people from India are buying the iPhone goto this machine That's for logical partitioning based on the country column So here I have all the India later where people are buying high for here I have the U S Data that people are buying the iPhone But the problem is here I'll have the berm art eight hours So in the our data Melville excepto terabyte china data will be treated by the Sylvie Cave is actually So the problem is that if a table is becoming sole big Okay I can't cluster it I
8:57
can take four machines in my Libya miss or date of arousing architecture And I can say partition the later Because if a table is so big if I fire a query the query will take a lot of time So what I do I say that OK don't stole the full table Divide the favorite table for partition the table to partition I will say that you have to use some logic for partitioning I cannot physically divide the data That's not possible in our debut Mus I cannot say that take a table and
9:24
cut it into four doesn't work so logically Partition Sorry to say that there is a country column How many countries are there in the world Maximum 155 right I'm not much of a geography person but I think some Carol 155
9:37
right But at least people who buy I 454 countries are there so I will take the country column So everybody who is buying an iPhone we'll have a country cooler So based on the country quality partition or data Right now I have four books says this box has India greater US data bear market and something later again same story If you're learning the join query this box has to process all the India later for joint query This box will complete very fast This guy has to wait So again you're so the
10:06
problem is that traditionally people were using our devia most systems and data warehouses to store the data Now what is the drawback born off The drawback is that as the size off the later increases your processing will become very very slow right You can't do it now The problem with the classes and big data is that there are a lot of technologies So when you start talking about them why don't you d normalize the later why don't you de normalize the data For example all the
10:41
transaction data You sir data everything will keep in one table Is that possible Uh one duplicates Second what is the maximum number of columns you can have in a table Are Libya must able 1000
10:57
Or I could say 1000 What if my data has more than 1000 columns If I keep all the data I can't go So that is the reason they don't provide the de normalize solutions If you look at our WMA systems they would always say normalize your data so that it will be representing different different papers And whenever you need the answer you tojoin and you get the results second row back This is only structured
11:22
eight off Meaning anything with you can represent in roll Call them I can dump here like an exception Data right Can I dump images and audio files in the stable Yes or no can be stored on structure later in Arabia Ms Yeah Yes you can That's club a blob and all objects right But can it process and structure Not possible So in the more than day if you look at the modern day what is our problem So if you look at a typical bank like so let's first complete this part So
11:59
we're not the drawbacks off our CBMs is that as the size off the data increases it cannot cope a bit Processing one off the problem like second problem is the scheme up meaning anything this you can represent in a roll call in format I can dump in an RGB amiss Anything apart from that I have to figure out the solution So what is the actual solution If I have two story majors and retrieve them in real time Just how does flip card stores the data I was working with Flip card for
12:31
sometime Flip card has around 10 million predicts each product has 10 images That is one billion images
12:38
So the nuclear corner for iPhone X picture on flip card Philip Carvel say that come after the Minnesota immediately Visibly Have you ever thought how it is happening Good question night how it is happening right So do you think they're dumping in an RGB MSN picking it from there One billion in majors No no secret later basis
13:01
There is a whole arena off data basis cold No sequence fortunately or unfortunately it is not in my course so I don't have to repeat the right But if you're going to So your end goal is to become data scientist And I think that is what you're learning the scores right You're learning spark and other things like so I would suggest you to Google and learn a little bit about something called No sequel Dicta Bases If you have heard about Mongo db Cassandra each Space New
13:31
York for J Voldemort These are all popular vendors off no sequel databases So no sequel later basis will help you to store and retrieve unstructured data at any spirit you want So what amazon amazon dot com actually does Amazon has something called dynamodb Dynamodb is no secret data peas So in Dynamodb I don't have a row column I have key value pairs So what they do owe the key will be the product name So you're clicking on iPhone X iPhone X is the key The value will be all the data you
14:04
have So the moment you play boob your pages there so no secret later This is a really faster or so it helps you to store on structure Later How are you booking a cab using Ola right I go out I didn't get a cab illustrated by the ancient night But if I tried to get a cab using Ola right I open a polar
14:24
How many customers Ola has millions in India on All of them are looking at the same time right Do you think all your requests are taken by an rd vehemence and searching the table and give you a cab Does it work Can In Arabia Ms System handled 30,000,040 million concurrent sessions on the someone off later calculating route Mongo db No secret later he's a lot Anderson among Gordy I was working with a lot for one of their design problems actually So I didn't design anything but
14:53
they told me that they're using Mongo db So the problem is that really the new world that we see today does not fit our description When I even for my first job interview in 2006
15:06
my technical guy who interviewed me Oslo because your strongest skill I said Java He said You're selected
15:13
It was job I was like you know Java You will get a job without an interview Now if I said Java he will not allow me to sit in front after being was nobody I'm in Java especially good But the new World demands new skills Actually that is my whole idea So they said tendered raw tobacco for your idea Mrs Scale abilities There Okay third is price Price is a problem right I mean obviously if you calculate the cost if you look at the oracle and or they're very very costly solutions
15:44
I si s a band was playing around 30 crawls on lean support feast or record 30 Close Not even any writing Three months they were paying 30 cross for people's Onley for support for a nickel because it is an enterprise Great product right here to pay money There is no other Be right now If you switch to no sequel platforms they're all open source You don't have to pay this much amount of money but that's not our discussion I'm just saying So all this problems means
16:11
traditionally storing the data or an RTV amiss and retrieving is not good for the modern day in the modern day What is happening is that every company want to collect and analyze the data beat banking industry e commerce industry or any industry that you want They want to collect and analyze the data So to give you an example if you visit the amazon dot com or flip car Darko what is happening They start tracking from which appear Does your visiting the website How long you are
16:44
clicking on a page which items you're adding to cart which images you're picking which offers were clicking All this data is analyzed to understand how well customer will behave
16:56
That is how they do marketing right So the more you start visiting the e commerce companies you start seeing recommendations and all the more the more you visit Amazon they will start recommending products to you asking you to buy so they have to collect all this data So when I was working with flip card flip cards and are they collect data from the APS that you're using from the website that you're having Even from social media Flipkart is really active in social media like Facebook Twitter
17:24
all these places they are active So if anybody tweets about flip card But they said their Facebook place all this data is being collected now this So this is actually called a big data So the technical definition off big data is this huge amount off data which you cannot process using your traditional methods That is called big data Right So why that is off in plus to us We are going to figure out how to store this data He was amount of data right on how to process this straight up So
17:59
you're saying that traditional techniques you will not be able to store it or process it that it's not possible So we will be seeing how you can use how do pass a solution toe to store or this gator on process or the state Right So Internet me anytime If if you want to ask anything I'll keep on talking So when I say no secret later basis out there it does not mean you're rgb mrs Gone So many people ask this question Okay How did you miss Is the only solution for transaction
18:28
management So even if you're buying something from flip card who will uh complete your transaction oracle probably Or any transaction system for transaction management And why is that Because our WMA systems are the only ones which can ensure the acid properties right before transaction goes or nor goes you bore something from flip card tomorrow Flip card cannot come back to and say that I don't really think I don't know whether the transaction happened or not Probably you got the money
18:57
It's not possible Right Then you will start complaining again Flip card All right So Flipkart comes to and say that you know what we're using Cassandra Asper Cassandra your transaction this we don't know will come back in another one week Is that possible No So transaction management Very good question is still handled by our Libyans Technically it is called poorly glowered Persistence So I'm not a person who use a lot of technical words but
19:23
poorly Lord persistence that means coexistence softs traditional databases on do you know the other ones Even if you look at Amazon not any company they will have their trailer transactional data on a bunch off No Sequels to handle other pipe off things Okay poorly north Persistence Well you don't have to Really I mean I'm not a person who really use a lot off
19:48
what you say Poor league lord Persistence Persistence is your air Don't
19:58
even my English skills are at stake when I'm training people assistances e a thing right What do you think take of work for that Probably No no no no Yeah So that's what we have not come to her dope so far I will explain Hadoop is not a database So teaching big data in 16 hours itself is a challenge actually because it's a waas topic actually In the world of big data there is a separate category off system score no secretly services And they are data vests So what does the data
20:36
business real time you know queries Right on In that area we have mongo DB Cassandra Space and this case Now these guys are used in real time If you want to fish the later say for example you're opening an e commerce website Probably It is given by you know see related These right are millions Ofpeople are booking flights through clear trip dot com on a clear trip Has to identify You know how many people are booking in a particular route so that you can give a better offer This is probably done
21:08
by reading their time data from a no secret system Other area It's not no secret No secret is that db actually extradicted these other areas not be no secret We will come to her do right now So this is the technical term for going systems off db sand No sequel and or so now let's understand So he established the area that are Libya must It's not good for us at least in the world of dictator But don't say that our debut missus nor good because transaction management is still handled by arguments
21:41
So you cannot say that Get rid off rtv amiss Right now there is a very important even though it is not directly related to your syllabus But I think it makes a lot of sense How Maney if you are aware off this down place your hand I've always old order Haiti always good to know that So you know what is any deal right And some people do not know where this any period that's fine
22:08
So 80 year stands for extract transform on Lord So what does the idea here is that See if I'm working with the company right the company will have our WMA systems like so these are all our DBM A system So probably this guy's using my sequel
22:29
on A He's also using Oracle
22:33
right A lot off Our review systems are being used by a company right now These are over serving the customers So if you take the example off ICSC Bank all their core banking data isn't on record All their core banking data isn't a record So if you're making an online transaction the data hits a record or their CRM data he said My secret What do you mean by Seattle
23:00
customer relationship Your name address blah blah blah So they have these two type of NATO now Is this the only type of data that they have No I say say Vang Harold's or so acquired a lot off other companies when we were working with them So they're subsidiary companies were giving them data in the form off example Fires examine Right now I see it's a bank or so had Ah Facebook page Okay From there
23:29
they were getting around 10,000 people visiting every that Facebook place and they're used to click on some items so they have to do this cross selling up selling If I use that goes toe Facebook page and say that I really like your lord take more long right So that is your cross selling and upsetting right So what they do they collect all this data from social media that comes in the form it off work Jason JavaScript Object notation Jason Key value pairs Basically So this is
23:59
where their core banking data This is where there are CR indicator
24:04
This is from their partner companies This is from the social media data They are getting okay Andi They had customer K this one in the social media They want to do analytics So every day 10,000 people are visiting the best place they want to know Ah person click Don't let's say alone offer And how much time he stayed on the page on what is group he belongs to so that they can recommend predicts recommendations Actually a lot of recommendations So social media data was being used
24:31
They did not reveal the all the things that they're doing because that were internal but basically using the social media data they were doing the recommendations So they will download all this later So if a customer is clicking on the Facebook page They will get the access off this guy if you are clicking on I see the same banks Facebook page They will know your age your occupation everything you shared in Facebook That is by any forgiven Tillman In most of the cases you can restrict
24:56
the information but master of the public customers they will share this So by downloading this data today they will understand 10,000 people clicked on after of them fighters and very youngsters on they clicked on this loan That means the trend is towards a particular bike alone if we given offer on bike alone youngsters and actually buy it So this is analytics right So all the social media data they can use in multiple ways actually So this was getting in the Jason for might Actually on this
25:23
type of data was that they also were collecting their customer care log files So that is a in flat fight It was coming in a format off flat file
25:34
flat file means text file So customer care log actually means the chat logs that they were having also the call logs their own record They don't get the actual car But how long the call happened and what you pressed So when you call customer care you press right This this this Ah that is actually captured And then they want to figure out how many people are called on what category how much time they spend by So if you're click don't complain button and you talk for one ever Chances are very high
26:04
You are a bad customer rating Me were never your schooling Probably right So they want to figure it out Life So that is the call record that burst coming in flat file this farm last night on At that point in time I say say bank also wonderful offer a solution So previously what they used to do their agents were the people who go to your house and make you buy alone and or so they used to go to a very place So this Asian will get a list and he will go to that place and he will
26:33
go to your house and say that Serve Please take a car loan You said no Here said by he will get money He's getting salary right So this technique was not working because if I get a list okay One customer will be in here This is what place would open any No
26:52
no I am a right one day will be No I am are another day will be some other remote place in Chennai So by the time he travels from here to that place they will be over So he's able to cover only so customers So what I see is a bang had in mind They will track his location He has a device ok on give him customers in real time So if he's in Canon Javadi hell it all the customers nearby him So this guy has to go right He cannot so I can go So keep the customer application It was a tablet
27:24
Will trying this guy on where this guy goes The nearby customers will be provided to him so he can cover more customers by And this has a lot of him back Because why It was practical Because sometimes the customers will call and say that Please come I'll give you a loan So what you do from here You might travel toe There is a European city Patties Intimate Yeah So you'll go toe Patties right on because the customer called Right So you go to Patties and meet the customer Now you don't know
27:54
how many customers are around Patties that this application track and send it to you So these guys were collecting that data hours from all the paps and that was quite some bigger more enough data They were collecting so many customers they met on whether they closed it or nor closed Okay that was coming in I think Some other format Some for my right I forgot Actually how we do it's coming So this is the whole day they have right And some more data was there on this data is lying on different
28:24
different systems It's not in one place the core banking date eyes and oracle Some other day ties in some other place Right on If you want to analyze this data one thing is that I cannot go toe Oracle I'm say that I want unless you're later
28:39
Why Because this tractor DB is already busy serving customers Critic This Oracle DB is the core banking database So if you're making a transaction when I police handling that right So I cannot go to America and say that Let me run some analytics on Eureka Doesn't work that way because they know activists Jordan I can't do that So I have to take all this data from all these places on dump it into a central place So that is what issue corners your data warehouse Wait that
29:14
discord it How many of you know what is a data warehouse So a date other house is nothing but a database only in one way It understands the language of sequel but the maser differences that it is not facing the customer Right So you will do something called E P L Okay so there are eight deal the worst which can get their data from or these places right on this guy I will dump it Indoor data Right So Data Warehouse is also called as your old lap online analytical platform sword and
29:51
so forth And what is the purpose off this guy So this guy will have all the later from here Okay And you can transform the data as well So that is why this world e t l extract transform and Lord you don't just take the data and dump it here than it is useless Right I can read the later from a flat file and give some structure and then dump it here like a proper table format So now all my data is here from this Right on I can run business intelligence here Be a toes Pablo Right
30:23
If you have heard about MicroStrategy
30:25
Pablo what I can do I can connect a visualization tool here and it will show me in charts and diagrams What is happening
30:34
I mean this is traditional enterprise architectures This is how people used to work forever I mean every company you go this will be their buyers before they will have these These desperate are discrete data sources on CTL tours So if you have heard about Informatica in for Matic Eyes one of the biggest et a little informatica talent
30:57
spend a hobby I society is so these are all the theaters So these tours where they do they will connected here and take over the data and apply some scheme and push it to a data warehouse Now the data will be here So now all the day ties in one place So if I'm Chandra Kochhar I wish I would be I can go here I see a safe Thanks See here Who is that place So I can go here Such and therefore sermon or no sequel right You can't expect that right If you're a CEO of the company I can't ask you Do
31:30
you know Java I'll be fired next moment Right So But to see you want to know what is happening right by getting over this date only you would understand what's happening in the company But you don't have time to see 10 learned Who are this language or anything So if I am Chander Kochar I'll go here I'll connect my tableau on I lost my tableau Show me the last months Ah transaction this quarter this Predict this area Show me it will show a nice bar chart I'm happy now Why This is
32:01
very very important If you make any mistake in any off this this end our stuff will be changed right on What she sees will be different right She has no idea how this is happening right Only you know how this is happening Like so you're you tear store Sort of improper The official big capture on dumped here and then analyzed So for see you sent sitios They use visualization tours to connect with a data warehouse Then analyze the data So all the data is inside this
32:30
data warehouse Right And this is not accessible by public Public has no access to this Onley internal company has access Like maybe see us off I see a say bang They can go here and visualize the data from here Others public does not have any access to here Right on Then they visualize the data they make a decision They may not make a decision that is up to them right in the whole set up There is no hard groups offer
32:56
on This is how I see I say bang was running so far Now what was the what were the challenges they faced One off the challenge Was that this eat eel tool Okay I don't know how many if you have Arnone pl tours Okay They typically around on a single machine Okay so they get installed on a server So there is a limit as to how much this they can pull toe our data warehouse
33:19
Right How much amount of data at this time put it from here toward Data House There is a limitation Actually You can't have that Others off Data being pulled Point number one point number two None off this Israel time Not enough This is real time I goto Danish store punished Essentially stuff I go to a Spanish store
33:44
I buy something from there for ₹20,000 I use ICSC credit card I should immediately get a message saying that again Your shop tornadoes 20% discount that Israel time I can't do it here because all this a separate data on these India jobs run of the night
34:02
Saturday morning If I'm shopping tomorrow get Dennis's 20% off by the time I'm out off than if shop right I may not use it at all So I say say banks First problem was that since the data was increasing you know they were not ableto run You feel properly Second problem None of this is real time They wonder toe push customized offers Their requirement was that whenever any transaction is made they want to capture it analyze it right and based on that they have to recommend
34:29
something to the user right Say for example a using credit card or debit card or even your clicking on some link immediately Something should happen now that it's not so easy right Because your data maybe somewhere in here Oracle Andi I cannot black each end of your transaction here because this is my transaction or DB I can't I can't say that practicing each and everything from here right That's not possible Third problem This data warehouse this data their houses a
34:59
very costly affair I have worked with the company court that our data they're the leader in data warehousing market I had a U S client There are They were using a product court in our data for data warehouse They were playing around $5 million every year
35:16
only for this set up Now it is good you can analyze the data and or that's OK but look being a lot of money actually so data warehousing is actually a very costly affair It's not so cheap if you want implement But companies like I c a c bank they must use it right so they can't say that we will not use it So So they wanted a cheaper solution They don't want all their data in the state of Eto's They wonder Alternate solution should be real time Cheaper Scalable right on
35:47
anything they want to integrate any any type of data that comes They wanted to integrate And that is how they came to us and asked this like can be migrated some big data solution And we recommended Hadoop And then the migration took a lot of time That is a different case altogether But if you talk about Hado Hado actually comes here in a way I can say it is sorry Right Right All right Exactly
36:25
Right Exactly right Exactly Not plus of it Not busted Exactly So here you have your in dimensional cool cubes and other things like Internet our house and you will have your what you call schema What's came on you car starts came a snowflake schema right on day So this is actually or lap is actually not designed for speed That is true It is only analytical It is bad job actually Right So with this architecture
37:02
I can never give a real time solution Exactly Exactly So a lot of people are You're saying that you can actually beat it if you do Parallel processing There is something called him creepy Right
37:20
There is something called MPP MPP means massive parallel persistent So a lot of people came saying that so these systems are actually there are great and all really faster Actually not like our baby a member there Faster Ok so a lot of people came up and say said that OK if your data warehouse is very slow why don't you do parallel processing Okay You install it on four or five machines and you can do battle that it's possible But still the same problem exists You cannot
37:48
physically divide your table's only logical division is possible Like I said current rebates you can divide or month base you can divide end off the day If you want tojoin operation or something same questions will happen so I can never propose This s a real time solution That is my cracks off what I'm trying to tell you and this is how even I say say bank was running on cost was another problem They have lost off Boston ward So And obviously since they wanted to collect
38:16
a lot off in structure later they wanted to go to a better solution Okay And that is how hard look came into the picture Not really Mostly here you have structure leader Mostly structurally They are used as the final place where you get the data from their somebody can visualize on in here You very rarely You modify the later you don't mortify the data because this is the place that the final data is available now from Eric and Connect And Michelle is the data That is
38:48
all you do what Probably you can run a query and get out That is all you do It is analysis family You're not doing an extra instruction transaction management All you're doing here right now So now the point is that now let's discuss a little bit about her Do so what I thought us like today probably we can cover the architecture a little bit off theory so that tomorrow
39:14
So if you look at your syllabus that I'm supposed to teach we have something called Hadoop and Haru pocket picture That is anyway required right And then you have a map Reduce Now there is a catch here My produce is the default programming framework on top off hurdle So if anybody want to analyze the data on top of her do they use something called my produce Point number one Now map reduce came long back Okay on duh today that a lot of this to analyze the data not only my produce Now
39:48
ask for us syllabus the curriculum say speechmaker dues and then some conception members and I will be doing that But one other thing is that if you want to learn my produce ideally ah you should be an expert in Java because I'm apparatus programs are written in Java or any other programming language So you can write and by Thorn and other things but natively people right now apparatus programs in Java Now I know that not all off you're familiar with Java and comfortable with java So
40:19
if I talk a lot about Java it may not make much sense out off you But then why you are learning my produce is to help you in spark
40:27
So next you will be learning something called spark Right 16 hours of spot content will be there Spark also uses similar Can success My produce So even if even though you're not able to understand
40:39
each and every line off court that I am writing that's OK But you need to understand the logic how it is working because you have a program start Everybody will be ableto follow and understanding deeper so that is not expected or so from you But if we discuss my producer I write a map produced program and big random apparatus program you should be able to understand what is happening inside that logically not like syntax vice line by line twice right So once we complete Hadoop architecture
41:09
like tomorrow I will be discussing my produce theory and my apparatus programs will be running on Then in the next eight over syllabus we will be completing bit off my apparatus I'm going to something called haIf but a high so high this your data warehouse in hairdo so that it's more off sequel site so you can understand that point Yes somebody had a question So someone has to question them Realizes said I will complete Who asked Nobody asked Okay I mean so I just love this
41:47
right now What I'm going to do and just going to believe this part for the time he
41:53
just remove this part for the time being that's what So now what is happening The traditional data warehouses are there like there are great and all but they're not so popular on in the market Like people used to say If you cannot convince what you do
42:20
you confuse people and it is very easy to do that right Because why am I saying this Because I see a lot of people getting confused because this seven Hadoop king had who became a very popular solution for data warehousing
42:32
right And since how does this open source and you can read the McKinsey report It will be only one by fourth off cost for traditional late our host So what will happen Everybody will use hard over obviously Why should I pay extra money But then what is the drawback The traditional data warehousing companies they cannot sell right So if if today you goto be website off Informatica or even there are data they will say that we are selling our there are data plus for loop previously
43:01
you had to give me $10 million Now you give me 9.5 Take it It's useless actually Because why should you buy Torretta But they're saying Bill State our data has a name in the market
43:12
It is a different thing right So glad it I will say that by other data warehouse We'll also give you her do that cheaper right Previously you pay Rs 10 million now pay 9.5 You're like OK great solution I'm getting to her dope Or so you don't actually need to You started out You can build a custom solution But the problem Couple of problems are that one The name off the company back There are daytime support or so from them Second it takes a lot of time to my great of these
43:39
things One fine morning You cannot go and say that OK From two more on worse We're using only her Do doesn't work like that right In a company you need timeto migrations and processes and or so people stick with their traditional tools So what I have been saying in some of the companies that I then for consulting
43:57
they have completely my greater too
43:59
big gate apart from slicker Do And they keep that our data and all in a very less level Like the whole data vehicle Right Hold data cold data of UN data we have So the immediately accessible hold data will be there in terrorist handle And you have the cold one data which you don't need immediately analyze You will dump it into her dope or something and keep it so you can reduce the overall cost in this architecture Right So now speaking off how do you have an architectural is here Study I
44:29
just changed a little bit Okay
44:31
I was just that I did here The question is when all this happened right So a lot of people So I ended a lot of trainings in her Duke and spark So a lot of people ask me that Okay so we are learning a completely new technology called her Do on I will say that you're wrong because Hado came in 2005 So almost 13 years now So it's almost like very old but for four people it is still very new for those who do not understand her Do So in 2005 this framework called Hadoop waas creator belong
45:10
back right on There were two guys called our debt cutting and my Catholic who created this framework They took the original idea from Google So Google was already doing a lot off distributed computing on Google of us are dealing with a lot of big data right So they published couple off white papers and this decorating and my Katharina They took the idea from Google and then created a framework called Hadoop on Later they gave it to Apache as not an open source project So hard Hope
45:40
is a open source project actually which means it is with Apache right now You can download it for free install it for free You don't have to pay any money But if something is really free you have drawbacks or so right Get it For example you have android phones I have an iPhone I used to use android phones But you know the difference right The differences not show off That is not the difference The real differences that on droit form might get a lot of books which men are coming an apple
46:17
from If you have used both you will understand The problem is android is open source right on open source means every company will mortify according toa requirement as Samson formula have their own android right So accordingly a lot of bucks make him the platform may not be so stable as it is If it is an apple phone only apple produces it right And if there is a bug crucify Carli Pro Somebody called operative will fix it because there's only one Russian off it available and they
46:48
know what bucks may come May not come the same story here Originally it was Apache Hadoop on Even Still you can goto the Apache Web site and download her do but soaring People realize that if they're using Apache how do it is Open source But if something does not work something crashes Nobody's there to fix it because it is open Source on Accordingly a company called Cloud Attacking There's a company called Clarinet You might have probably heard Claudia is the first company which created ah ah
47:24
commercial distribution for her and related tours So today you can goto Clodagh on as them to give give you her do what is are doing days you will be getting the staying Hadoop almost similar What You find an Apache Web site But if you install it and if something is not working
47:42
they will fix it You have a support Since you are paying money you get support just like a vendor Right So one of the companies which is very popular
47:53
in the world off big data is clouded So they sell herto spark everything as a bundle Okay even your lab actually has clouded our distribution The lab we're going to use actually run secluded our distribution Andi I can also use the this thing right If internet is there because I just want to show you this So if you just go here and simply type not now probably you can do it later Simply type Florida You can see Cloud Atlas website Right Onda clouded came in 2008
48:28
So today if you're buying any big data platform the leading wondrous Clodagh so they will give you everything packaged in 11 product like Hadoop will be the no sequel It's race will be there Other related towards will be their data Science tools will be there so commercially they are actually the number one not provided Florida right Onda Somewhere in 2002 others we will discuss her dope in depth Okay I'm just giving you some idea We'll discuss the architecture and or
48:56
that is also one more company core Horton books Harden box came in somewhere in 2012 Okay on Horton boxes Also the next major competitors took loaded up The major difference is that Horton book says Original Haru Apache There don't mortified
49:15
cloud I said some mortified wish which they say it is better
49:20
you No No So it is actually Apache Open source license So what about Texas is that anybody can modify it and sell it on anything They should also provide a free version and they do it So if you go to Claudia you can download the club address product for free You don't have to pay any money but he will not get support So what people do They will buy only with support Right So if I want to sell Hadoop I can start a company I can download from Apache Mortify The source scored I can sell it to
49:51
you guys but two things One I should give a free Osho for Or so no support Seconding that is the story with every open source right You have opened early Next you have read Hatley Next What is the difference But I had uh no Yes not open source Whatever their perfect because Horton works is open source How do and you can download from their predict Clouded as it's not 100% is open source It's a custom product but you can download it for free from their website and install it No asme animations
50:25
as you want but you will not get technical support So I really people will not do it right because you need technical support so you will pay money and get it actually right But these are the companies Thea Horton votes in Florida as the leading vendors in the big data world They were so poor wife certification So if you want to write the certification exam probably after the spot Plus you can try that Cloudera provide certification exams in the big data Would you can write it
50:53
and gets There is also one more company called map three Are there there are many companies actually the most common one Start three There is also somebody called map
51:03
happened to listen to it right It's open source community It's not a company It is like you and me so I can get it for free You cannot serve it for freaking but he will say something crashes or zero You post a quest in the community Hey yesterday my or service crashed and fired from the company Please help me to fix City and Euromonitor dude and end the price world Right So the real thing is support to be very fair with you Why still people are interesting All these companies Microsoft or
51:32
Apple or Oracle they get support right Something doesn't work You cannot be able to fix radio report on my neck Nitpicks So map Artist the third company which is sort off like popular This company if my information is correct was creator by a couple of people who could Clodagh important words Indian certainly right And they found this company called map map are is very popular these days Actually very very popular vendor the highest pay the addition is or some apple
52:00
cost ways They are bigger than clothes are important books Cloud and harder Most licences cheaper than map map US License cost is very high actually Map ours architects It is also very different not like traditional Hadoop architecture So for that reason like I haven t g e I went to flip card They're all using either Horton books or Claudia Their own use map are much They also downloaded the open source How do modified it and selling it Same same story but they made a lot of
52:33
change to the architectural off her group Actually apart from this IBM has their own flavor scored Big insights Microsoft has their own for flavor This correlation insight How actually inside is that Amazon has Every company has an addition off their own Ah Hadoop and big data platform But most popular answer Clodagh Horton Books and maps These guys have the most popular ones actually
52:57
So So for a developer there is not much difference For example if you're writing um apparatus program it will run the state we're in or three platforms There is not much of a difference That difference actually is that heart and looks a starkly open source They still the same water Patrick gives Cloudera mortify sit a bitch on They say that with the mortification How does this more stable It is their clean We don't know It's all marketing gimmick it is saying like buying an android of
53:32
Samsung Mobile for the next The same about four What is the difference End off the day internally this android But Samsung will say that our Andrew it is better more bug free or something It's Tracy will say this formal Never hang So only Ashley's your times and know what they have done inside the phone But end off the day If I downloaded say what's that It'll run on both the forms right Because it is Android So same thing only it's just a commercial our distribution same
54:00
story with me Next you can buy from red hat That is also one more provider right So say right happily next to sailing next internally of living the same If I happen unless command it'll work Both These colonel might be slightly different but end off the way They're being open source original inexorably so same story here but considering the platforms cloudy that seems to be more more Pull Pull it because they got a head start Toto Sunday they started I long back This structure Horton looks
54:31
is also gaining a lot of popularity these days I've been seeing that So the idea is included in our lab We have a
54:42
Claudia distribution OK so But we have not We have not art or around about how do place So we have been talking a lot about the company But when did we start the session I didn't check to 30 So they knew how Breaks and or I think 3 30 right now See I don't have any control on that Brakes are controlled by uh
55:09
right I don't have any control
55:12
She has to approve I don't have any control plus me CE trees talking What Should your center right on any off this Okay I can teach I can stop you Say stop Stop Okay By the way if you don't like my class you can walk out or so But he will not do it with the same vigor Money right If it was a free class half of them could have worked Note for since you played money will stay I know that Okay Yes sir Please ask Ask as many questions as you want It is yes Critic No no I'm apparatus nobody is using
55:54
in the under priced map produces Gone on this park is that That is what I'm saying in your curriculum that isn operatives And why you are learning my produce Two reasons Oneness migration Ah Lord of companies will already have a lot of my produce programs and they want to go to spot So if you want to go you should know where this map produced Second reason even for learning spark the concepts off my producer Important spark or so was built The basic idea off my produce
56:19
So how my produce works how the program runs Is this basic concepts African understand that will help you to learn sparking a better That is the reason we're learning Spot map reduce Actually in the enterprise illegal There is nothing going my produce so far no spark uses spy turn in your course
56:43
That is the reason I'm not teaching spot So initially Great Lakes asked me Can you spot I said yes And then there was a problem because I did spark using Skylar huh I'm a scholar developer You are You're learning Spark using python spark can be programmed in four languages Scallop I thought our Angela four languages are there and you learn with bite on I'm not a fighter own guy So that citizen I'm not teaching spot I actually like to stay spot because most of my training
57:11
is these days Air on spark actually But I know bite on our side champions but not immediately I said We'll pick it up later Probably on peace Sorry Scale is similar to Java No not like similar to Java Eight if you know Java Eight It's a glorified version of Java Actually Java It was developed based on Skylar
57:39
So skylights another programming language It's a very nice language actually but I don't know People don't really like it Okay So like it in the sense not as popular as job All right Java is very popular but sky lies not soap opening but it's ah it's a very interesting program language I used Qala a lot in my programs and training sandal Anyway So any other questions before we go toe to toe on architecture and our not many questions So if there are no questions I really
58:12
either people under sort of very thing Well they understood nothing Write the book Yeah you can consider But it is not just a data warehouse That is a different So you will see So what is the difference Because they find buying a traditional later warehouse It can do only one thing It can become a data warehouse I can't do anything else Hope is not like that It can become a data warehouse and much more And we will see that you up in now This still eso your architecture is Huh Huh Huh
58:54
See now the situation is very complicated actually because if you are starting a new company and I really you don't have much money what you you go to cloud right Every solution is offered in the cloud and typically companies will go for this hard and are not available in cloud also So they will go for cloud distributions
59:17
And so all of these you can or say that hard up is 100% replacement off for data warehouse Okay It can work as a data warehouse but performance considerations are there so sometimes you might need some reports within milliseconds Second two seconds which probably Hadoop may not be able to do her job is a batch processing system It's not real time So if I have an idea or or that x Arctic desert now these guys are really fast right MPP So what companies do they keep
59:48
the whole data there Probably My company has 100 out by data in that do then other data I need is really very fast I'll keep it Interrogator 90 A terrible Dump it in her Do I know that once in a while and even if it takes some time it is fine Okay
1:00:04
Same story with the flower So most off the organizations are now deploying everything over Amazon web services So Amazon or first fully managed Hadoop Solutions on the cloud like you can go to Amazon ask it Give me a hard Oh Plus in 10 minutes you will have fully functional heard of solution everything installed up and running sofa separating automation Right So there They also give you pretty or data warehouses They have something called or did their date of a house
1:00:31
That shift there is a solution called Red Shift that shifted online Cloud based data warehouse So Amazon her first red shift You if you want for data to be stored So like I said everything cannot be given in one solution You need a cheaper You need the same speed You know everything you have the package and one solution It may not be possible Right So like someone was suggesting right You can compromise speed either right or ah real time access like so same like that So
1:01:03
how does this a solution Most off the people are not But not everything can be guaranteed at the same time Okay So that takes us to the question What is Hadoop and why this is so popular or what is the architecture off Duke It's fairly easy to understand actually because the idea is very simple I have a single machine How much amount of data can be stored in a single machine What do you think I'll give you a server How much will you store Based on what Okay so I'm That is
1:01:42
based on words Yeah So stories capacity How do you decide if I give us over How many hard disk or how much space you can dump Yeah so I'll give you a server How do you decide how many hard disk I can add in that mother board Right Motherboards for decides So the mother border to say I support in hard this you can have been hardest So the idea is very simple So people initially thought I can use one machine
1:02:17
And when that mission was not sufficient where they did No no
1:02:25
no no Distributor says computing the North I came so early right
1:02:31
They added this the restores disk So one machine is always what you do Uh I'm going to be enough No I don't want God Kate Supercomputing Ah no I want cheaper I can't sell my house Another mission is what I'm going to talk But in the doing something came That's external stories NASA and all right nettle stories Probably You guys are not much from stories Background There was something called network stories Very very popular Have you heard over TMC and or a company called TMC Square
1:03:11
This is a very Liberace of these things So you have something or network attest stories Nass books You can buy your box in Store the box anywhere on You can down the data there so the system can done band process from there that is holding us So that is supporting more stories actually but it is not within the machine When asked us Not me no came sand stories area network Sansa even popular these days Okay so Ah four room for love hard disk You'll get a fibre channel connection So if I
1:03:42
can Dumb board the heart is that I can start as much as they want But the problem is even if I'm using a sign This is my son Does this give me unlimited storage Okay Problem is it gives me only stories No processing If I want to process the data Can one settler process or the data understand No not possible Right So processing cannot happen Stories can happen in a sack on the sands are very costly Clumsy to managers union separate people to manage and or so people got fed up with
1:04:19
Sanders San asked was there Sand was there So And that is where you started thinking about distributed computing Distributed computing is not a new area It was already there Okay so in how do What we're doing is very very simple
1:04:34
Rather than one machine You take a bunch off machines Okay So I will say the Bunches just for sake of argument four Okay so I take four machines 12
1:04:50
three for these four machines are let's say limits machines To be fair they can live in those missions But the support for windows is very limited in the open source community Typically the recommended the boxes to be UNIX or clinics So I take a bunch of boxes say four boxes force or worse Right And I can download and install this Cloudera or Horton whilst any off their do on this while installing it will ask me which mission is master Which mission ist Lee The idea is
1:05:26
one mission has to be master The rest of them can be sleeps So in this architecture I come up with an idea One guy iss master and three guys up sleeps fine So we have not discussed What is the master doing What is a slave doing that's later Okay But you install her dough in a sector there when there is one Master and let sit three slaves Right now the idea here is that if I want to store anything Okay how do will help me restore among these three boxes right So I can use the story space
1:05:59
off all three boxes So if each off them are having a two terabyte hard disk I will get a total capacity off six terabyte in my heart Do cluster This is called a Hadoop cluster A cluster means a group of computers heard obeisance Yeah So how deep is a framework Actually it is not a software So one confusion people have is that is how do a software No it is a platform Okay And I will show you the different different components we have in the Hadoop platform So if you install how
1:06:32
do let's say you don't know anything about her do But let's avian starlet in ah four system set up while installing it will ask you to select one s master remaining asleep So I have one master and Preece leaves right on What How do CanDo It can take the story space off all these three machines and projected as a single six terabytes stories box right So the beauty off this architecture is that Let's say you start dumping the data and the data gets stored here and it's a or six Terabyte is
1:07:07
over I can add machines owned the flight from three I can expand to 30,000 with or charting on the clusters Meaning your stories problem is sold You don't have to ever worry about stories And I heard a cluster huh Grube allows you re sizing without any down pain so I can add one of machines What I can remove machines up to me to decide right I didn't tell you how the data is stored I will I will come to so as off Now we have no discussed storage and processing I'm just
1:07:41
giving you are 10,000 ft over you like what is happening in a Hadoop cluster So one of the advantages that you can start with left three machines and expand to 3300 or 3000 machines more
1:08:03
Yes You can also add when you have your data here and you're analyzing the data Okay I can keep on adding machines without shutting down You have No Yes
1:08:17
you can remove but if you're removing will have a problem For example if the program is earning terrified more with the program might crash because this machine is needed That is not idea If you have one more machine here where nothing is turning I can remove it gracefully Nothing will happen So you are getting something course scaling out this architectural scores scaling out meaning you keep on adding more machines so that you know you can get unlimited scalability Now I
1:08:42
will tell you how stories happens
1:08:44
So if I did not discuss how stories happens which one This sizing it is not possible Sequels are designed in that fashion If I want to modify anything I have to first stop reading and writing from our database The normally I can alter my table or increase anything or decrease anything probably will be able to add a machine extra But it doesn't make any sense in a sequel world actually and in a secret world I cannot come up with a 30,000 service sector that's impossible cost
1:09:18
mr for scalability Like I said the major difference here is that in the sequel were logical Division Off later you cannot physically divide the later so then logically dividing the later Like I said country in there took off people on If one machine is taking that it'll have more data on a little less data I don't have anywhere to manage here that that will be handled I'll show you how that is happening So the so this is called a Hadoop cluster So the technical term for
1:09:44
this is the cluster Plus there is nothing but a group off machines right So that is available in our dope luster Now if beginning If you're goto companies and check with them like I was with this journey for about seven Company called 24 by seven There are core center company Onda their own Hadoop clusters for analytics and they had a 50 machine cluster 15 or cluster Each note has around 2 56 GB around and 10 terabyte stories So 50 under 10 years work Sorry Notched 10
1:10:22
What I'm missing 100 average stories So 15 200 his watch Five better bite off stories they had on blue 56 times 50 That is that I am capacity of the cluster That is a very minimal harder cluster cheap Hadoop cluster for typically will have 500 or 1000 north Hadoop clusters Because the amount of data you're storing isn't petabytes 1000 tell a very different matter But actually some arse off the company they'll have huge amount of data for dumping this So the size of the cluster will
1:10:50
be in that that level actually
1:10:54
And how do piss distributor meaning if you install it It is not getting in sorted one machine it isn't started nor the machines and the components Mint dog So it runs in a cluster Not in a single machine or two machines actually Right So So the basic idea in storage is that you can add more machines on get more stories All right No no no I go to a question What is The master goes down right I'll come back No slave will be acting on less asleep actually Yeah So good question
1:11:28
Another thing is that let me ask you this question So I'm asking you to build Ah ha Do cluster with the 50 machines
1:11:35
they can examine for the slow missions that is off this hours
1:11:40
Do you really think it is a cheaper solution Meantime self hard with Like you buy 50 servers or 500 servers is in that religion per solution or do Yeah So let them say uh get it No My point is if let's say you're buying 500 Silver's eat several cost you some money right So if I say if I under times that money right it is really a huge amount of investment for me That is the reason we call this machine's commodity hardware What do you mean by commodity hardware Commodity hardware means
1:12:28
any machine without any label you will never buy your Dennis Alberta Bilardo cluster You will build build it with cheap throwaway servers Right You are not going to buy a Volvo bus You are going to buy your PM What is that The Bernardo I'm not blaming number No don't take it regionally okay Even care lover You're not going to buy your Volvo bus to accommodate all passengers You're buying a bus You know if it's can break down at any point in time See you want to carry all your citizens Will you
1:13:00
be buying only work purposes No right If you look at most of the buses that are here What is that idea It doesn't seem to travel from 1.2 Another point on the bus should be there That's all You are not investing cross and buying more votes that everywhere you by just some buzz Same idea here in a herd of plaster Typical hurdle cluster The machines that we use our core commodity hardware commodity hard moments assembled servers so you can get it Actually so you're not actually
1:13:28
buying an IBM so over because I was also working as a design architect for one of the group's solution there We got servers for around ₹35,000 ₹40,000 ₹40,000 You can buy us over if it is an IBM Several appear to lack three lack So this first like somebody will ask Timberland give you the only thing it'll work It will crash Or so it will definitely crash 100% So then you will be thinking if I'm having a 500 North Hadoop cluster if my slaves are crashing what is the reliability that I will
1:14:02
answer that question Just just part of the question for the can't be no So that is just a big stories Both right So really Then processing Yeah so So that is what a service So this is your desktop Normally stops like so normally have something or to stop it for men Or so there is something which can so much different lines In one way it is a bigger box with bigger storage and processing So if I want to buy yourself and I have my people option one is that I go and tell IBM or Dell gin by
1:14:31
missile so they will give me a brander So So they will charge a lot of money They would also give me support and her but I have to pay to factory life or so So normally when I say so over This is where people buy a Hadoop cluster You never do that You buy the cheapest service possible because you can afford to have failures in a Hadoop cluster I show you how failures will be handled in her plaster But you are not investing a lot of money on the hardware that his
1:14:57
idea because otherwise the solution is not feasible If I'm buying 500 IBM servers I can take the same money and give it to Teradata So I was gonna say anything right So I don't need support or anything I just need some normal machines And how do poor So does not say what kind of a mission you need You can even build a Hadoop cluster using their stops If you want performance will be less little beautifully work or laptop So anything that you want it doesn't say that Right So that is the
1:15:26
silent Keep this Dagenham because I need this diagram again That is I'm not really moving it actually yeah id hot I got it
1:15:42
Right So new businesses might take hodoud more because after five years I mentor bite that are later right If this is a situation I might buy a heart absolution probably I'll tell you what is the biggest challenge in my experience
1:15:58
in my consulting experience The biggest challenges Knowledge See all the companies want a better solution right No company want don't necessarily pay money But one thing is that if it is a terrorist a solution in the market If you go there a lot of people who understand that are hosting data that everything is a new not a new concept If I go to the market I can get any number of people who know what is They have a housing and they can easily install it at our data for May I just
1:16:26
have to pay money But building a team off Hadoop expertise is not easy but it's what you're sitting here right If I go to a company or so if all of them have turned this and how hard it is working on how dope is not like Click Click install that a lot of things here to configure and tune so people don't want to spend that much of a risk
1:16:45
So one reason is that the knowledge curve of recall right the learning curve is a bit high You can't learn it in a day and say that Okay I'll implement her do It's not possible right So most of the companies are stuck in this position I'll tell you real story since you asked this question So when I went to one off the company's for training Okay the company said that we want to learn huddle basics I said OK no big deal that you heard of basics I've entered the company I started
1:17:12
the training They started asking me like name nor block size which are all related to her Do I said Hey I'm teaching you how to basics You are asking me all these questions They said we already know her do Then why am I here They said their company got a project related to her Do they implemented Haru They're working on her do for two years Okay They learned everything on their own Now they want some training to clarify Everything is correct No it's actually something really
1:17:40
happening You check with any of your friends who are in any company who are working in big data They will tell the same story because the manager will say okay tomorrow Big data physical start Who Those who are here can join half off the mill run away Okay Half of the mistake
1:17:55
Okay And these guys have to learn own their own I'm able to share this experience because I go to many companies on most off them are self learners There are no resources or training programs like this to learn they were sitting down on their own and on and then learn over a period of time So that is another problem Because if I want to learn Oracle I can get official trainings a lot And I have a lot of people who know a record they can teach me right So the learning curve
1:18:22
actually makes a problem in this case is actually So I think you guys are better because you will all this right If you're just joining in a company you can say that here I am aware off over these things probably you can get an immediate present Anyway You are learning to get a job I believe Right End of the day Your idea is to get a job right I'll show you I'll try the architecture So now what You need to understand So since we were looking at the high level definition suddenly
1:18:54
let's diver bit deeper Hope has three major components There is something called GH DFS That is something called the young I can share this life's and media If you want You can make a note that's all It's not mandated 30 Score my produce So then you download and install how do only her do these three things will come by default SPF asi on an apparatus These three components will be in started by and you don't have to do anything They will get instructor right in this
1:19:28
This guy is this storage This diet is handling storage Okay this die is resource management Is it a source Nine Agent This is processing the Bennis Not replacing so Ashley efforts is the guy who is responsible for storage inside So if you're if you're installing her do it'll have three components Actually you can see them actually Actually Office You can take and see I'll show you OK so if the office is the it's clear First stands for Hadoop distributed file system It's a fire system Actually
1:20:18
it is called Hadoop distributed file system I'll write it here Probably
1:20:24
How do distributed five system that is SPF office Now from your course point off view DFS is important because this starts like how the our date I store yon is not that much important But we will cover young map produce We will learn for sure some operatives and pushing for tomorrow So the processing part we will do tomorrow we're not going to learn how to process the data Actually
1:20:55
because if I teacher of living also end of the day or to go back and re collect all these things right you will learn Get it So this part I'll push for tomorrow on how many off you are from Java Back down or some off us are here at least Okay so because map it is actually in Java Okay So even look at the court and many people are not from the Java background or so so And like I said in the industry it will never happen that you're right inoperative School was map Reading is going from
1:21:24
the industry Migrations only happened So even if you're able to understand the logic that's more than sufficient actually So first next look at this guy called Ashley office What is the office is doing So I will probably push yarn and map produced for tomorrow because once I complete actually a first I just want to show you something called Hadoop ecosystem Which will happen Draw here So we completed steer for storage Yanis a simple piece of apparatus innovate Tomorrow will be
1:21:56
covering right So this is the guy who is handling the stories And Lord off you were asking How do you store the data in her do right Okay Is this been going Actually we have X trapping They're not writing Actually
1:22:22
This one Mr you Spence ating or so that red and green
1:22:28
we'll get black This is okay I think Right So let's take a hypothetical situation where you have installed hard Open one master Okay on In my example we have six slave missions Now This picture will pose a lot of questions Usually Okay on that Sochi So there is one master and six slave machines and we're talking only birds stories now processing the master will have a process Gore's name Nord The master machine will be running a process called a name north When you install it will
1:23:14
start running this process called name No Okay The slaves will be running a process Color data no
1:23:22
date on order So I recorded as data in order one The war Three four five and six
1:23:37
So technically we will say that there is one name Northern six daytime hours I mean that is how you say in in a Hadoop cluster right So name Nor does the stories master on date on Or is the story sleep
1:23:51
Okay So that that's fine We have understood Now the next thing is that let's say you want to store a file in the heart of cluster How do Does not care what you're storing Meaning you can't store any format can be examined Jace on images video Hadoop doesn't really care what format off data you're starting while processing You have toe make sense Operator
1:24:18
So that isn't my produce If I store ext file I have to write it in map radios that lead the specs Fine And do whatever I want to do They see a first fire system does not really border about what type of data Your story second point and probably the most important point is that once you store any file you cannot edit it
1:24:42
Modifications are not possible You start it Believed it That's it In s DFS mortification this impossible Now you have to understand this fax really well because you get confused later right So how well I did it is not possible initially of capacities So why you may be asking why it is not possible because DFS is designed as a fire system for handling huge amount of data Terabytes sent better by someone off data right on if I want Oh more defy or edit You know uh roll
1:25:22
by rope like in a transaction that is happening You know it is not possible because if you look at a hard disk right there is something called seek time
1:25:32
If a hard this want of fetch a record there is something called seek time Right on If I'm having one terabyte file I want to edit the 100 flying It's not possible for my heart to stop C can get the data in that If you're because you're using all commodity machines here they're not even faster I'm noticing solid straight rice or genetic very commodity level hardware I'm using here So how does idea is that you want to read file sequence chilly There is no random access
1:26:01
There is only sequencer access So if I store a file I can read the entire file and process it I cannot say needle in a haystack problem Needle in a haystack problem Meaning If you have a one million raw table you cannot say Take the 100 line and it nor possible in her new at least in her do it is impossible So either you can read the whole data and process it That's all you can do and the result can be stored Obviously if you process the data you get some results that can be stored But on
1:26:34
the file system editing is not possible You can't get it Hi Not possible So so But directly after it is not possible I cannot open a file and update said that is a pending as possible happening is always possible I have a file I want to upend their data That's OK because you're just hiding in the end But I'm just reading the 100 throw And I did That is not possible Now you may be wondering what if I want to do it There is a solution
1:27:07
You can do it You have MPP engines on top of her Do this Can't do it I show you how to do it I mean it's possible but considering onliest DFS Ideally if you're storing a file on you you got some modifications in the file You deleted and restore the fight Meaning you're on edited actually Right Can we take that You can delete
1:27:32
You can believe the whole fire religious possible I'm saying that my new techniques are not possible because hard up is considered to be a system There you are doing analysis off the data It is not transactional system Where will your minute edits and all come oil tp systems If I am having an rdd a most I want you to insert update insert update It's not an RTV amiss right If you are an idea must will It is possible I don't have any objection but here it is not possible by default huh
1:28:03
Uh huh Exactly Not closely Exactly
1:28:08
And also if you're bringing a transactional sort of the time transactional baby right So I don't want my nuclear It also But worst case if I want to do it it is possible So we have MPP engines on Hurdle This can do it actually Exactly So if I'm storing a one TV file if I want to process the file probably I want to process only the 100 line I have to lower the want even to rhyme with Actually Fslr so then the business use cases Welcome So that is where you need to make a decision I will
1:28:41
be I will train you on that Meaning if if my if I'm writing a group by query for an aggregate query Right Probably need the whole data I have a very big table I'm writing an aggregate query Right Which will say grew by an order and something like that It makes sense I Lord the for your file because I run over is a later forward Data need to be scanned probably But if I'm doing a select something something were named equal to something name it could get a good ramen Only one
1:29:09
line need to be processed for processing Only one line Why should I know the whole later not required for that You have an option that's calling people and generally show you how to do that
1:29:19
A massive Allah processing engines on her Do you have tools like a hawk Elope Impala So there are some guys who can do that actually and I will show you how they were contact No they are dumping in data warehouse Like from there They don't want update transaction data that happens here right This is another transactions that happening Correct So this is where customers are placing their order on this date I really want to change This data was created by customer I just want to get
1:29:49
the data I probably went to more defend get it So once it is here to make any changes I just want to analyze the data We will see that innovate Now let's take a hypothetical situation Possibles No I'll tell you what is the road OK So what is going to happen Let's say that you want the stool
1:30:09
Ah file I got a new market by the way So let's say you want to store the file off size I don't know
1:30:22
1 92 and b you have a fire ex trial Imagine And you want to store it the size off The finalists will take a break at 4 30 ducats for 10 actually So probably think will take it for the pigment
1:30:38
So the fire sizes 192 megabytes you want to store it in the huddle cluster Now there are a lot off things which actually matters to your real time experience here We're just learning the basics Because in a real company if you want to connect with a Hadoop cluster the Hadoop cluster will be some bears So if you're sitting here with a Hadoop cluster will be separate you will be separate Right So you will get something called a Hado Klein package
1:31:03
Okay so one raise that you get something going a Hadoop climb You think that you can access from your laptop Second is that you can log in tow a Nord There is something called gateway in machine This is going gateway machine So if I am sitting here this is me Very nice light So I can just connect with this machine from this machine I can connect to the Hadoop cluster you're on directly Get inside the cluster That's my point for security reasons and many other reasons
1:31:32
So companies do booting Either they will install something called a cargo plane Even that is very rare What we used to do there will be a next mission called Gateway North You will get a U sentiment password Your type you connect with this machine This mission will already know how to reach you Don't have to worry You issue the commands A little drunk here got it right But whatever way So let's say you want to push this 1 92 and b file from your machine You connected to the Gateway machine You
1:31:57
said okay upload the later What is going to happen is that first thing which is going to happen Is that your name nor machine This is the master This guy will tell you something called block size There is something called
1:32:15
blocks Ice This block size can be configured for every Hadoop cluster When you're installing her Do so I am assuming that the blood size for this cluster is 64 megabytes So what this block size block size tell you Block size will tell you what is the maximum size soft data that you can store Meaning If you're storing over 92 the file what will happen It will divide that into three blocks Same thing happens on your laptop If you're starting a
1:32:51
song on your laptop what happens
1:32:54
Well the song is stored as it is What do you think
1:33:00
Okay you start a song MP three So right Wanted laptop Where Do you think the hard this will actually take the song as it is What it will divide it It will divide it It will not start as it is I think the next uses four kilobyte off four K b block size Lennox windows Also use a similar any fire system with sharp urinate and store Same thing happens in her do poor So So I'm assuming that this Hadoop cluster has a block sex or 64 be so this name nor will get back to your client package
1:33:33
and say that Divide your filing toe three or whatever you want because the block size a 64 nb So what is going to happen This file with will get divided into three blocks You don't have to go This This will happen behind the scenes like you don't have to say that What is your block size And it will happen automatically Hydroplaned hydroplane package
1:34:00
This gator has had declined and start That is work So this hado planned on the gateway Well already communicate that name nor name Always say that Boss in my cluster the blocks I say 64 b you manage yourself So the climb will divide your data into blocks off 64 64 Sister for total is 1 98 So now you help Three blocks B one B two and B three You have three blocks now off the vicar Right on Then again you go back to the name nor
1:34:30
you ask the name nor I have three blocks Ok Now tell me where to store on the name nor what it does It has communication with already their tunnels
1:34:40
It knows where how much space is available It will tell you that Do one thing Store the first block Probably here Just an example Store the second block Probably here Be to store the third block Here be three Most off the cases the name nor will give your different different missions It will never allow you to store everything in one machine So you're divide and distribute That is how you're storing the data So now your data is divided into three blocks on They are stored in
1:35:10
three different data on ours So this is like six In reality you have to pay 500 machines So what A box is easy All right But sometimes a couple of blocks may end up in one machine also depends on the stories but ideally it gets stored like this Now you may be wondering why the blocks I says
1:35:30
it's not random It has a name that has an idea like how much story spaces available on each deck can Or so he come a bit free machines within that it just picks randomly three over and say that you know
1:35:46
Yes that has to do it processing And I will tell you so If you think logically now if I want to process this data okay the machines can process this data right If I store all the three blocks on one machine one machine has to process three blocks Right That makes a difference right So ideally you divide and distribute the data Most off the cases you don't have to do is name nor will do it for you automatically This is all inside a land This is all inside a land All these
1:36:18
machines are communicating with each other The date honors or send a heartbeat to the name not saying that they are alive or so so that the name nor can detect how many of them are alive or down or anything But uh yeah So back now we have a problem What is the problem if this machine crashes Because this is ever be MPC But it's not a Volvo bus right And it can cash at any point in time If this crashes My block one is gone I can't get my later So by the fort and had a look
1:36:52
There is a replication off three which means each block gets replicated
1:36:57
three times to other machines I just draw this than computer questions Give me a moment Be to
1:37:06
again Be too I'm just drawing randomly I mean just taking that These have the machines be three probably from here b three So if you look at each block each block is replicated three times to different different machines So now let's say this mission crashes This beetle is still available sheer and sheer I can recover it from here on now What happens is that the name nor will store the metadata meaning the name normal Right here there is a file called ABC or TXT It is
1:37:41
divided into three blocks block one is available on where three three do six So be even 3 to 6 people Where that is Beetle Huh B three pills Why
1:38:05
Yeah So this is your meta data Meaning if tomorrow you want to read the fine you have no clue is the file You just go to the name nor nor you The Hadoop line will connect with the name nor on the name nor will pass this information You goto b one b two B three So you goto late and or three for two You will get it If any of them are down it'll give the alternative So the purpose off your name nor is to actually store the metadata or the index on if the name nor crashes
1:38:36
You cannot access the cluster that is for sure whether the name nor is gone and everything will be gone toe Prevent that in most off the Hadoop clusters you will have an active name north and a path stand binding Meaning they'll be to name North These two guys will be in constant communication So if the active name nor crashes stand by we'll take over I will talk a little bit more about this later How this active standby It's working but you don't have to worry if
1:39:04
they active name nor crashes standby with immediately take over So now coming to the questions Yeah please
1:39:18
got it Now Three blocks are there at the replication half three
1:39:30
So it's not a bit other
1:39:37
very rare or the cleaners going down at the same time It's very very rare Even though we say commodity they have not really commodity these days I mean even though they say that I do because the storage and server cost is very cheap these days So all the three machines at the same time going down it's very very rare But what What is the other possibility King John fired a missile light from North Korea What will you do exactly Nobody will be there But let's say King John fire on little tonight
1:40:08
Okay on Duh So in tonight a Hadoop cluster is gone right So what do you do You are not there to run the query but the company short survey right disaster recovery will be there so typically set up a d r disaster recovery on the disaster recovery to set up that there is a company called Van Disco Disco Disco Dance Bradesco's very interesting company Sovann The school is one of the popular companies who set up disaster recovery for her Do But you don't know disaster recovery
1:40:41
like in traditional systems Meaning if I have 100 or so cluster
1:40:46
I won't set about 100 Back of cluster right That is based off money Very rare King John will fire Mr Right if he fires Gore saves So what I will go in even in the Hadoop cluster you have to classify the data you are storing under terabyte data Probably know all 100 Arab It is so important rate is it No Lord of Data is like archival and or so the 100 terabyte data I will identify probably 10 terabyte which is very critical And I can periodically back of their data to my BR center I don't set up
1:41:20
100 north cluster in Noida The backup plaster from here So the user you were called a distributed copy This city in her do that can synchronize These are two clusters Primary back of Mississippi scored distributor Copy I'm not lying So some people say I'm making up stories I'll show you Uh yeah So very excellent question I'll just come to that Just give me a moment So Goto and a school Very interesting company right I like the name who named the company or don't Bradesco's So Van
1:41:57
Disco is not only a hurdle company by the way they provide replication solutions Ah solutions backup disaster recovery fable criticise static curry Why Your internet is very very slow right See it is not even opening of the beach Yeah this Australia Cody 100% is hard of availability Right See this is what I'm talking about The used band is going or not the projects That is why you know usually people do not be aware off these things So I was working as a Hadoop administrator also for
1:42:43
some time That is why you know this because this is these are are like really times things Okay No shut study This step is that Yeah it is same thing Yeah but managing this on your own is very difficult Actually this ipic and Iran I know that even cloud their important work supports But there are a lot of things you're to consider interning BCP but straightforward resource utilization It will be a problem because you're keep on reading and writing the data Second if it fails you
1:43:20
don't know how to manage this Sleepy can fail at any point in time So if you're implementing Baron Disco they use this sippy only again But they have their own ways to handle it Actually mean they'll insure that application happens properly You don't have to bow that about it Now I'm not suggesting that you must use it or something If you want the steep you can do it or even clouded uproars their own Some tools for disaster recovery for actually But we are here right So
1:43:46
my question why the block size is 64 b not for Gaby
1:43:51
Block says can before Katie right
1:43:59
Maybe I don't know I'm just asking 69 sick time If I'm storing and four K before K B I can't read How do you read one TV you want to play huh Exactly Now this I showed you because my math is very weak 64 times three is 111 98 I know that That is why sure This example in real production Plaster the processes One condition not 64 I don't know what is three things 1 28 and beautifully honest Okay so the yellow production Lester Block sizes 1 2018 b It is configurable They used its to four or
1:44:36
so There is a parameter I show you where you can change block size Okay How do the defy uh block sizes Want to date and be You should also know where or sort of the center Noting that at three major releases off her do how does 123 How does one is the older release on Nobody is using it these days so you don't have to worry about what it is not available in production Like 2013 was the last year when this was used So no longer use Hadoop Two is what you are using I am using We
1:45:10
are using This is the current edition How do three released in 2017 December 15 Like three months back Okay Now let me ask you a question If you're storing 100 a robber data in a Hadoop cluster how much stories you need Is that true but only application considering 300 right into three
1:45:41
Is it good or bad Back
1:45:46
What is your corpse I mean see play innings are already is interaction right I might know a lot of things I might not know Lord of things you know Right So you can So what is your thought on that Don't can be fine I don't know I'm just asking it Start Ah yes or no question I'm just asking Where do you think No
1:46:13
So there are multiple approaches One thing is that some companies this replication factor can be changed By the way it's not hard Quarter Some companies safety All right So you get to cooperate or so well called being their data you can mention the application factor Very interesting thing I'm coping a file one baby file this file I can afford to lose I can say copy with replication one But it had led into a debate in the hardware world So a lot of people said that this
1:46:42
is very bad actually because three times replication means you're losing a lot of space that this fine How do three There is no replication Very interesting point I was just beta testing her 23 even I don't know completely in her Do Previ use a technical racial According it is the military raid played 10 parity How many of your other profit Parity based frayed Okay there's something called Raid nor the Akshay Kumar movie Something better Okay Redundant array off independent disks
1:47:13
That's called Raid Where this safari parliaments technique Okay so based on radar they have formulated a new strategy in her do three Where you run replicated data there is that other technical aeration Cording if you enable that you can still recover the later if it fails they say it will use only 30% of traditional stories Meaning if you're storing hundreds evey you need hundreds or PCB better right Much much better Your only 300 tv But Hado three will not make
1:47:43
it to production very soon Surround very space Still be around Who in your state they are So you want the question everyone The break Huh So you guys are too interested Actually I thought he will or sleep in the class
1:48:05
We should get the hard Oh plus in the morning and other class in the evening All right
1:48:10
I don't know that Why Like
1:48:19
status of what I actually wanted to learn statistics It's good right I can probably come and sit in the morning Probably No Statistically it is now after an you'll definitely sleep 100 Wasn't his hair people I'm not blaming the play America Don't take it That waste artistic itself is a subject which is a bit boring actually Right
1:48:37
No not tomorrow against second If I am coming is it so fast Have more Do learn our programming huh On number You Yeah What No I will love give you a solution So the processing has to be handled by a different work So we have something called map produced in my produce You have a Java class There is an abstract class in Java
1:49:21
So for those who do not know Saleh there is that there is a class There is ah technique where you tell what type of of file you're dealing with and there are only limit their types You can breathe next XML
1:49:34
sequence file key value So some limiter number off types of files are there And how do you so this is a very good question How do you handle unstructured data So it is also very confusing Kinds of many people believe that in the world of big data you are always playing with video sending images Actually no I see many people believe everyday you'll go toe office on run some facial recognition like you see in the mission Impossible movie kicking kicking kicking on Yes Yeah I got
1:50:06
a big data No you don't know that Actually you can analyze images and videos There is no It's very difficult actually to do
1:50:14
analyzing video and image It is very difficult because you can't directly analyze it You're too converted into binary Once it becomes binary in hard hope you have a former core sequence file would read it a sequence file on right the program to analyze it It's actually very difficult job to analyze these kind of fires and not everybody need to analyze it Let me give you a real a real time Example I was working with 24 7 by the customer care company What is their primary business
1:50:43
Customer care in customer care people card right Do you think they will listen to what customers calling
1:50:53
yes or no So you are running a customer care company not your company You are giving support to other companies and you are running a car center in a car sender Customers will call right
1:51:05
Do you actually listen to the what they are talking like static idea For example How many cars idea will be getting in customer care Do we really think they analyze your course Two things One it is legally not possible I think on not 100 was initial Maybe it is possible Fine fine That is just for the quality accent and or that isn or doing Big Data Analytics For example If you were angry customer you spoke for one over that call is recorded The person who took the court will go to the
1:51:39
manager and the next meeting manager will analyze that call and say that this was your mistakes I'm saying do you actually analyze the big data And it is on these kind of things Not really
1:51:50
Right You will analyze the meta data Not actually later How many people called watch was the duration where they press This is the later nor the actual audio data because you can do it I'm not saying it is impossible but then you have toe So how do you understand what he is speaking Union voice recognition Right on it may fail it man or fail You get your 100% sure if it is a U S Customer or something they can approximately match with an Indian customer How do you identify over this It's
1:52:22
not possible right He may speak any language any extent right So practically these things are not possible
1:52:29
I'm saying you can analyze unstructured data There is no doubt about it And companies like Facebook and order Facebook actually analyze video data They have toe for some reasons that they have to do right there They have written complicated algorithms to real and convert to buy an Oriental But if you are going to work in a regular project you will always see structure later CS refiles Jason data normal X data But if you're storing also so you need a class to read this type of fires
1:52:58
That's what the question Waas Yeah Sodi Oh that's an excellent question I also missed another question from this side that will stay in question No Yeah So I think we'll take a break because I need more time to explain Anyway when I say I do it is this We know that is actually a first young members but that isnot only heard of you have Lord off other tours on top of that So you know to have at least a rough idea as to what are the tools we have in the Hadoop ecosystem The your stack Right So
1:53:47
I will talk about it once we complete actually office discussion I will go to the ecosystem stack because of the ecosystem stack You will have a lot of questions How this workout like this A basic idea Because I'll take the same I see a scenario and I'll tell you how we moved
1:54:01
Then you will get a better idea Probably tonight Yeah So you don't this question site I'm sorry I didn't notice you what you
1:54:22
lost That's yeah that's an excellent question So the question was that if one machine is totally gone if I plug in traditional machine will Hadoop automatically take it off Copying the No The problem The point is that if you that's for commissioning So if you had a new north here or let's say yes Simply adding a new north said existing set of is that if I had a new no this nor will be empty
1:54:47
So you have to do something called balance Sir there is a your political balance I will run balanced So that had a bill Analyze and understand that this is underutilized is over utilized It'll move some blocks here If you don't learn violence there it will keep it as a new machine But the data will not be balanced So we run something or balancer to manage that There was another excellent question This replication How does it work so replication It works in a very interesting way Because
1:55:14
if you're writing the later the first block goes here be one right now This guy Bill 10 This guy to get a copy
1:55:25
on this day will get a copy on Then this guy will tell this guy to get a copy So third copy is God Finally this will send an acknowledgement here here and you will get an explanation Then only a block is written So this point by blind right you're not pushing everything at the same time as a pipeline It writes and gets an acknowledgement That was one question Saudi
1:55:54
Yes So this will be communicated
1:55:56
when you are doing Ah ha Do copy operation You will talk to the name nor the name nor will give you a list off date honors a chain off their turnovers They should get the replica primary underplayed This information will be passed to them Maximum storage Excellent question because the name not actually stores meta data This meta data is in the lamb when the name north is running The meta data is being stored from the time Actually it also persist in the hard disk so that
1:56:27
engaged in clashes of gifts for one trillion files 64 g b is the ram required So you should actually calculate the rhyme size nor the hard this ice One brilliant file of fire size can be anything I'm not talking about file size because for a single file to a 56 k b off my target is created What one file means How many blocks are there It's replica That is um a character Right So 64 db can handle one trillion files and Haru cluster so accordingly would increase the time size hard decisis
1:56:59
Fine You're only much harvest Katie but fight it Did you know
1:57:14
Exactly Oh you can increase that I am off that machine right And the name notice Not a commodity hardware Very important Point the name nor is a costly machine You run Wanda That's all of us right Because you run Warned that crash unnecessarily was it Uh
1:57:40
yeah exactly So if let's say this May machinist down Very good question If this machine is down this B three and B one is gone automatically It will create one more bigotry and even somewhere
1:57:54
Andi if this machine comes back online it will believe them Because it just order vis Maintain a copy off three or based on application be treated or before or two So that is automatically handed You don't have to worry It'll delete and recreate If it wants
1:58:10
make their because Gun Citizen Yes Yes it'll take It'll say data locality near a nearby consolations will be there That is eliminated Okay so I will come to that in just a moment Matar Data I have to talk about it Yeah That was because one donated his action Okay okay On top off menace huh Correct Huh What is So when you are installing it's their first Even though internally obviously next five system will talk for the next By system on the 64 m V will be further chopped Um the 64 MBs
1:59:19
on how do internally only next it is for Katie But this entire 64 they will be one point you're on You do not fragment it They're getting my point right If I'm simply dumping 64 be only next it'll divided into four KB and fragmented in many many places Yes here it will be Continue So I read and writes that fast Uh any other questions you have Eso balancer has to be I really run by the administrator Her dope administrator You have to manually do it on There is a command called How
1:59:54
do bad and said you can turn it from the jeweler so we can learn it I will show you So tomorrow If you guys can remind me about it I'll show you are this I may not be able to learn a balance that but I can show you where it is I will forget Right So tomorrow show you the hurdle cluster I can show you Where are these things Actually so they ah balance this part off for your book installation
2:00:14
The emphasis here so you can sit around balancer But normally we will run balancer only during off peak hours Like during the daytime If you turn it it has to move the plot blocks and you're processing and orbital get interrupted Want interrupter But let's slow So they say at night midnight 12 0 clock I run the bad himself I make sure that the blocks are evenly distributed So this blood says that is also a very good question This block size is just a logic Okay so my question is that if I
2:00:50
have a 200 defile how many blocks will be there let's take 64 be on But hunt 200 be file What will be the site stuff for the block So how many If you say that the fourth blocker 64 b I would say yes How many of us A four block is 12 Andy not eight and be right
2:01:27
Very lis It is eight and be start 100 the block size is valued on Leaving the file says is more than the block size Any file your divide are ultimately will end up in our 10 em bee or eight and be so hard will keep it as a deer Other rest of it is storing in a 1 28 and be peace remaining their ties unnecessarily So there's a logic Actually it is a logical even the physical division happens here Okay logically to keep a block off 12 and beyond because you have a little bit later left there
2:02:00
Got it It's not like physically m marking a space and feeling or dismiss That doesn't happen like that So yeah out of these two when you're storing or let contacts right If I'm storing a file off said 200 be it knows four blocks interview creator that I four the block size didn't say that 12 imbuing Lanier because I have little Liberator exactly given sequence something Otherwise the remaining space is wasted right on A verified will have some basterds space So why are you wasting that MySpace
2:02:38
if I saw one trillion file how much will be Easter Can you think thing A lot of file sex right That is not required actually So there's a logic internally building which the office that the blocks ice is valued as long as the file sizes bigger than the block size Once you chop chop chop and last 10 and understand them the only it's not 1 28 or 64 right
2:03:01
it will increase exactly So if you append that block size will increase That's a good question So in this 200 be file if I'm a pending the data right it will increase It will provide a room for back actually because the last block was only eight and big Now you keep upending the data That block will be number one today and it will keep on no inlay necks What it does is that it when it is creating a block is a block sizes small It will say that gives some room when
2:03:34
you're creating the next block don't keep it immediately Next to that the hardest is so big that it should not immediately created there Like most of the cases happen will happen very rarely Probably it man Or but very rarely It is almost impossible In most of the cases They will get that
2:03:53
Yeah So that is not processing I told you I will come back to that So if I'm putting all the file one big file here if I want to process it this machine really can process my fight Right You fast splitting And now three machines can apparently process my data Right Processing is faster so if I'm on
2:04:12
the same time you will not run all the processing Another miss concept that people have is that in the heart of cluster if you have 100 debut data or the 100 developers at the staying primal state run No Doesn't work like that Are you getting my point You have a lot of data and a Hadoop cluster Your order program I wrote a program here on the program We've all come to office exactly 10 0 clock I'm saying run the program Same time doesn't work like that
2:04:38
You may have a lot of data up people out and programs but you should do with the program So that is the share dual incomes I was about to talk about it So if you're voting in a Hadoop cluster resources are very important right Because each machine has processor than hard disk and network Right And each machine will be processing your files whatever you have return So we have a tool called Rosie Apache wuzzy
2:05:02
wuzzy Startled Actually that comes along with her opinion style as a story So in who's EU will be shared doing your program You wrote a spark program analyzed 100 TV later If you immediately submit the program probably the program will not work Because already somebody symbolizing the later where do you have space to fit all these things Right So what do you do You write it in woozy and you contact your How do Pittman say that I want Unless 100 Maybe later Then Can I run
2:05:30
this He will say tomorrow 11 oclock Okay submit most of this analysis Is batch not real time Real time is a different case but batch jobs you share deal it by IOC Or so you have something called Q In her do for example you are working as a developer I am a tester Can be How same access in the cluster of course Testing less totally different But let's say I'm a researcher You are a developer He's a data scientist Can we have the same access on the Hadoop cluster No probably You're
2:06:00
a developer You need more resources
2:06:03
Okay Probably I need less resources So what do you do We create something called Q Okay There is something called Q We create in those Q b I locate resources If you're coming from the developer being you will get there to personally off the resources If you're coming from other pin you get for people space by otherwise I can utilize all the resources right If I write a program I can make the cluster only for mine That's not possible in a Hadoop cluster so management is
2:06:28
entirely different How do you run these jobs that are different But to just answer the question why you are dividing is that you can use parallelism The rest of single mission has to process everything right Court assesses different Q Right Q means multiple teams will be there Each team can get you so when you submit a program you will submit to a cube You will not directly sent me to the cluster So let's say just for an example we take up cuing in times off resources
2:07:02
Say for example in your Hadoop cluster you have you have 10 date on awards each day Donald has Let's say N J b
2:07:12
right So total you have 100 GB I'm in the cluster right
2:07:16
on their safe 10 daytime hours each has four gold processor So you get what for Peak over processor Right now If if somebody want they can get a hold these resources in their program If he is getting nobody else can run a program in the cluster
2:07:34
should not be like that So what we do we create for cues Let's say or to accuse in this queue I will say 60% a year for the social Be there here I'll say 30% They should be there Here I say 10 person base would be there So the superior developer que There will be 60% off the resources that Mr Steed G B ram on What is that 30
2:07:56
60% days Right 24 24 core processor So they will never get more than 24 So my people based off they're doing is they're in the normal shave doing they will get maximum 60% only or if the other Q is empty they can clean the resources I just No no que is actually a Java class You can create something called the share dealers and her do There is something called capacity Should you will affair Should you learned all so this should do it is called it you So if I'm creating something called
2:08:26
fair share dealer within fair share you let it lay low me to create use so I can say create three cues on whenever somebody submitting a program They have to mention the Q name or otherwise the program Okay Okay You use Cut it This is not talking or the data data They will get their data I'm talking about ram and processing power there is available So if I'm running my program we will come from apparatus tomorrow Eso you family learning my program I need a ram and processor to run my program
2:09:16
right How do we run my program I need rhyme and processing power that the class special give me right so I can limit people based on cue Only this much resource you can get that it's possible that is quite sure doing in her room Oh get busted So when you submit to a cubit inside that you you can write a policy reform policies fitful first in first out But I apart from before you can also have priorities and other things You can buy your own policy
2:09:57
like that time three jobs camera These three cues Can people can submit right three or more than for example this you has Let's say 60 db ran right I need only don't easy Bagram You can also use 20 right three people depending on how much you want to process how much resource you want That has to be able to tell you how to calculate that on this Not this is important but these are all admit activities known off the side discusses actually related to your course But this may all make sense
2:10:31
when you start working See learning is one thing Okay if you if we just want to learn we just learned I produce and run some programs and go home But when you actually start working You know these these things are actually used in real life I know I don't very share doing on all who see nothing comes in your lab but in the lab part actually What is there actually offices there You will lower the later read the daytime apparatuses There high is that sort None of this actually comes But I
2:11:01
can show you at least from the club Haram an injury Why this thing What is that balance
2:11:06
on Where is the name Northerners are eternally since we can seat since just we want to look at it So are we clear about actually Fs I think at least we can say that Oh now on a basic level of your probably not everything but yeah
2:11:27
I didn't get that Of course means use huh So here it is Specs fighter flat file Any file you can select So buy it if I like I said if you're storing something honestly a first it's differs does not care What does your format you store and emotional stories Image jpg You have the makes sense off it while reading the data It is not like our baby um So that is one thing It is not like our baby a must at all because it is the emphasis A file system So if you're done say for example on
2:12:01
my windows machine can I start an MP three file Can you open in north bag Wait I don't know exactly I can start an MP three file in her do If I want to process I should write a logical process It actually that is for only queries My apparatus is not quitting Margaret is this programming language queries We're talking about haIf so high it is your data warehouse on her do high level except normal sequel queries Whatever sequel on the data Yeah So you have to create a table and or from
2:12:40
the data Exactly So high This like a your date Your date I will be initially affects your to create a table Give us Kema then it looks like an RTV in my stable Then you can run your queries No it's not an RGB Masetto Actually the data is running late Exactly It can be a C S V U can be a text file Regular five cancer Yeah Huh I will show you in haIf how to store and process the data right Yes In high you have something called indexing but it is not as efficient because indexing will
2:13:25
really work Well if your data can be managed by the are WMA system or the data bellows in system if I'm indexing the daytime haIf the data has actually managed by actually efforts Right So I can get some performance improvements But not like in your rtv Imus Never compare how do with arguments you can at the most you can compare the data warehouse because it is not transactional right It's equal to a data warehouse I can see I didn't get your asking if I copy a file can I Now you don't
2:14:14
create a record That's what I'm saying You are again talking Transaction it right You don't create a record See this is a big data platform so you don't terabytes if I You are not creating a record Very rarely You it That's what I'm saying This needle in a haystack problem right So very rarely you say that I want to create a new record You will be adding files continuous Strange that even huge in size You can afford to lose You can afford to lose the later I will talk more about that
2:14:41
in the ecosystem side Okay Huh
2:14:55
Eso A lot of companies actually run this using virtual machines It is possible but performance We cannot always depend on Williams I mean VMS are ultimately work they don't use If it is a bare metal virtual machine you can guarantee
2:15:12
there are two types off Williams right there something or bare metal like a hyper V indoor There you can get good performance or VM beer It is not a bare metal No performance will be compromised because again you're to go through one layer then talk to the stories and or Oh yeah you can be confident So I will just talk about something called Hadoop Ecosystem and then we'll are discussed for that lane Uh okay
2:15:48
Got it right Cut it No it's all a job call Actually these are all So they are dubious Actually written in Java such client is also in job Everything is a Java matter actually So you are just invoking a Java objects There is our first real object Miss you call this object will get our this in our meta data and give it to that line on the client will keep it in memory And then you go and read one by one from wherever Okay So I will just explain this ecosystem and come back your questions for that
2:16:23
I know there are a lot of questions and it is good but we should finish a couple of topics more as well right So I'm just taking this I c s a use case so you guys will understand better So traditionally we do something really appeal to bring the data right And now what we have we have this How do so I'm just writing here Ich ni office So this is your acne office five system This is a stiffness on in the world off her toe Be very rarely do appeal Why Because if he'll means you are
2:16:56
reading the data transforming the data and dumping the data if you're getting big data transforming it on the fly is very very difficult actually So in the world off How do what we Louis something called E l T Shirt off 80 l You will do something called E L P That is extract Lord and transform So first thing is extraction How do you get the data into our group system So there are multiple doors First of all if you are having structured later say you have data in Oracle you have data
2:17:31
in my secret So this is our structure data which understand sequel beers after work or scoop What is that worth Guard school school can get this data to your office Uh school It is an idiot You can call it as an 80 year tour School piss used to bring the data from any sequel system Toe Do transfer the data No it is part off Hadoop ecosystem So what is going to happen when you say that you want to do from clothing huh Clothing I will give you her do on top off that it will give you all
2:18:14
this store So shall be explaining now So Scopus part off their heart out ecosystem Our school piss usedto bring data from sequel stores It can take from lips Oracle It can take from my sequel but it cannot pay from flat file days or nothing Nothing Only sequel only places where Sequel and district So from I C S a use case we got order data from Seattle through school Okay so school prison about General it is a very lightweight doors We face the problem that I wanted to discuss that Let me ask
2:18:50
you this question This is ICSC a bank's core banking or actor database Do you really think they will allow you to touch it
2:18:58
Then how do you get the later meaning Who will go pee Even replication Any tools O r How do you replicate I mean
2:19:23
excellent CDC change data capture So if he faced the same problem there is Golden Gate Oracle has a solution called Golden Gate So see the sea change data capture Meaning they will not allow you to touch it Right So they don't have some replica solution So where they will write it into Ah replica RGB It's actually log file from their your to capture and get it They won't I love you generically touch There are actually be buying its core CDC changed their capture right No that
2:19:56
will have only the delta So what they will do They will give you the original later Backup data they have The CDC will captured only the changes in the database Got it And then you have to from seriously you can write a Java court on get the updates are dumping that is Have you get it Okay CDC will discuss later because I knew to spend half another record Explain Just understand the changes Can Decatur can be captured because it is anyway not related to your Ah big data platforms This is just
2:20:29
a business scenario Yes Yes yes yes Off course Exactly Exactly So they will not discuss CDC in detail because it will take another half another I don't want to spend half another It is not related your syllabus anyway And even I didn't go see the sea because the Olympic teams who will do that even I was not aware of it It's such a state that actually is a picnic So whatever changes are happening here really capture and dump somewhere from there You can read it Data is exactly from there So I
2:21:03
was just saying that we use illicit to get the data So Golden Gate is a product that Oracle uses for CDC Anyway where you scoop to get from here and here right now comes the place There were having a lot of debate actually OK now the real question Is that okay I know how to get there later from here through school right What about the other data Right now If you want to get the other pipe off NATO there are my people raise You can get okay One of the most popular used tool is called flu About
2:21:38
your flu flu Miss up point the point delivery toe What flume Bess It takes their data from source give to the destination The source can be anywhere and normally flew Misused for unstructured it I say for example in a folder you're getting all the Jason data or examine later or flat files I can create something called a flume Agen on this day will keep on pulling that folder Whenever there is a new data This table read it on it will send English the efforts This is what flume this Now the really
2:22:13
interesting thing is that flu can do two things One thing is that we can simply read all this data Let's say this isn't a folder okay It can simply leave this data and say send it to her Do This is fine But I see Say bank also had another requirement So desisting clear collecting the data and dumping into her do they don't want to rule that Okay so they wanted that they had a requirement in which what used to happen was that this flume bus capturing some data Okay some data of
2:22:45
us getting generated in the form of Jason here So in the social media if somebody is clicking on one off their products they wanted to send an SMS but he has been So this Jason Data will keep on coming You will keep on getting it and flew Can't directly lumber touch the office But that is useless for me because I want to analyze the data whenever the data is getting generator That is where this guy comes in to keep picture Kafka Gothika So there is a difference between flu man Kafka Okay
2:23:17
You have to understand flu mist like a point A point delivery You give me their data either dump it That is your flume does It doesn't do anything on flu Does not store the later anywhere meaning it will take the later temporarily Mr And then immediately push the later calf dies a message queue Anybody aware of Miss Excuse So CAFTA is your message queuing the big data Meaning what This they will do This day will be running in its own cluster No hado plaster This guy will run on its own cluster
2:23:51
like 30 machine 40 machine This guy flume can send the data to this day after on CAFTA is a cluster so it can store the later by the far that can store for seventies by default Okay anybody can go toe Kafka and get the data published subscript system We call it their bland ages It was stored the data so that my paper people can ask it flew Miss point to point if we don't get the later Dumpster data here Yeah So she had what we used to do if you didn't do this Okay We get big order
2:24:25
data Kafka from Kafka I can push it this year first one way So I'm getting the original copy here Okay Now in I C s a bank So the next question is that how do you analyze data in real time
2:24:40
so somebody is clicking on Facebook The data is coming as example Right on Immediately when the data comes flu will read it Push it took after So now the data has respect after But I want to analyze the data in real time That is where your spark comes into picture There is something Course park streaming in spark You will learn this I think I'm not sure Okay There is something correspond streaming So spark streaming
2:25:08
iss a utility which analyzes real time data No freedom can bring any type of data I didn't get Sorry point appliance Yeah Yeah
2:25:23
So it just makes takes from here down to look after It's a do that The sequence not on a flu and will not work with secrets ever Only school will work in the sequel later Scoop any other type of data flu now half cock and were synthetically Get this data now This is a bit confusing I know Okay because this is very new to you first awful But CAFTA can also directly get the example fire But the argument is a flu Miss Poor based poor based means If I installed flew without
2:25:56
disturbing the set up I can pour the data Castle cannot pour the data You have been stolen oil here and mortified existing set
2:26:04
FluMist Full based meaning If you start flew here it can start reading the data automatically I don't have to modify anything if I installed half correctly after can also get the example later It can breathe examine later But I need something called CAFTA Producer that has to be installed here Owned ICSC of source system That is not possible Right So since it is an existing set up I can't directly get troop After I get through flume flu will give to Kafka CAFTA will send one copy
2:26:34
to issue the efforts because I need to keep the original later second corps picket Listen to something called spark Streaming spark streaming Is the library of a level in spot where it can do Really I'm processing the moment you give the little process it real time
2:26:48
So somebody clicked on that immediately The data is here Spark streaming will process it on Say send an SMS or you can take an action Whatever you want Now what are the other tools which can do real time apart from spar tree streaming You have something called Fling You have something coarse stone These are all real time processing towards you Might have to make a note of that Yeah
2:27:13
So they said Oh yeah Uh huh Uh huh Yeah So craft can be Facebook Film can be watched That because what marks at this point To point right 1 to 1 and far Facebook is like our distributor right Anybody Uh huh Okay Watch that for so keep it right I don't know
2:27:41
Okay Okay Okay That that context Yeah I got it Correct Good I didn't understand that point See So that's their easy So in WHATS app it is push right Flu mist like what's up Okay Once flew on gets the data people sit push it to somebody cannot store the data Kafka means it'll get the rate and story for you as long as you consume it So in in I C S a bank this was the architecture being courted through flu Cindy to Kafka here One copy will go to spot streaming So this guy will do real time
2:28:11
processing and do something OK One copy will go taste the efforts So the data will remain in Kafka as long as you want You can configure by default It is seven days You can also mention the size off The rate I want to keep on 10 GB rate are twin dizzy But later there's something called topics in Kafka where you can mention how young the date I should stay here It's a class Divide the store that it it can But it can only one Then you can send here right You're getting my point Flew Can stand here
2:28:41
then This will not happen at this point The point delivery I need a copy here I need to send their oars at the same time Also probably tomorrow I want to build one more application
2:28:53
Okay This will do something on this data on that application I have a new application I built tomorrow that can also get their data from Kafka Calf dies like if you give that I took after anybody can take it from me I will start it for you Got a flu mist like point to point Got it CAFTA restored it so that anybody can come and get their eight or 10 applications can read from Kafka
2:29:20
So slow slow very slow Hardest to read I can directly push in real time Right So before you get out off panic short I want to send an SMS right after processing No this will be one golden copy will go here The original Facebook clicks and all golden copy will go here that tomorrow You can analyze Now you click Dona Pays Insurance Page and Facebook I want immediately contact you It has to be processed and understood on Then some decision has to be bacon So I'm sending one copy here One copy
2:29:55
here Here The processing will happen here Simply the original data It will be discovered because our generators anyway here you can store it also Yes Golden copy will be here School Gordon copies Um apparatus can order the anything This batch
2:30:17
Yeah Okay Uh yeah because how do you read from a record How do you read from her record Ah fluent CAFTA cannot Only school can read from a secret base Stores now cast guys a very interesting thing I teach three racecourse only for Kafka Three days It's not enough Actually diseases Kafka But it's That's a big topic Actually This Kafka can be a Mrs Q Yeah Okay Cast are right on your case type Will not come here in this example
2:30:54
Contributed indirectly you clk stack we were using That's what I was stealing in the break to somebody What we were doing is that we used to store the data in her do toe process We were using something or Splunk There is support for Splunk Splunk can actually replace a real kiss tack It does E l k both in one Cool So this visualization right You have some log files machine learning You can just stop playing Visualize We were using something called Splunk So this is the
2:31:21
big pipeline I'm talking about You can also build using lk if you want them to see Yeah okay Is a stack actually elasticsearch
2:31:31
log stash Chibana It's ah Probably can explain Like so who was working on a yellow cake Somebody was So you can explain Probably What is he okay Nitwit but probably not 400 TB 1000 baby
2:31:57
So let's say you're getting a lot of log fires You want to make sense out of it right from different different places Let's say I'm collecting all the logs from this network I want to make sense soft off it So I told this can easily collected clean it and then displaying a dashboard That is a year Okay what it is doing there is a pool which is very similar That score Splunk that that is not in the big in the Hadoop ecosystem It is not there That is a third party It'll
2:32:21
actually open Source Yeah Exactly No no no no That's what sequel Only school Exactly So that's a good question So when Schoop is getting the later right school bill I really push only to hear STF is okay But I can say school push look after hours That is also possible Why was the date eyes have a level in school Okay I can say give it to Kafka So either it can be right for life processing or school can push to something called High We have a hive here which is a data
2:33:05
warehouse From there I can analyze in life if I want but normally this transactional data is not much use for real time processing You are doing it based on this Facebook click Sanders Based on this transaction I can get it now just to answer his question
2:33:23
That's a good question actually Uh
2:33:26
so can you give me a sedition So a customer He's making a transaction Ah customer is making a transaction Using I see is a credit card He made transaction for 50,000 I have to immediately send him an offer So if he's making a transaction here I made a mistake here I want to show that if he made a transaction here 50,000 OK very little data in Iraq How do you capture the data From here
2:33:56
you will say scoop We were using flu because you're CDC will be producing a log file The long will be captured by flew north school So here scoop not coming to picture here School will come because I'm directly reading from CIA My Seattle missing my secret My secret is over the dinner table on I can read directly but in the a rattle case I was not directly reading the table If I want to read a table school But we were reading the CDC change data Capture that was a log fire that
2:34:28
was being collected by flume actually
2:34:30
so you can actually make it So this is the answer if you want to make real time Okay You need something This is the stomachs Can push the date out of flu Actually Uh uh huh Exactly Exactly Really Fine I have worked on a banking use case there so I'm talking about this site spot streaming fling stone
2:35:02
So spark streaming is what you will be learning Flink and storm or other frameworks which can do the time data processing So you will be wondering why real time data processing is so critical The all three all three aerial time You may use any off them Flinging Apache Stormy source Apache Storm is almost out of the market Okay Spot swimming and flanker popular Right now we have all done a use case that Citibank you swipe your credit card right How do you know it's a fraud
2:35:34
or not You're swiping a credit card I want to find out whether it is a fraud trying section or a real time section
2:35:43
headache No exactly No Even you don't have to be a fraud Let's say I use a credit card regularly If I one fine day if I certainly paying us dollars or something I'll get a card from customer care Did you really make it That means imagine
2:36:13
HDFC Bank How many credit cards will be in circulation And how many people will be transacting every day Do you think somebody is setting and monitoring every swipe spot streaming You are collecting over the data All the credit card transactions will get down Okay They won't come in a relational for they will come in some flat file or something Your flu will catch it the moment it comes It will push it through Kafka or directly to spark streaming or Flynn course storm And there I could
2:36:41
ever come my logic Okay match this transaction If this guy's limit is something or his last transaction made places outside in there something regatta Male customer Okay that is how you're getting this year My course or so when you make a big transaction immediately they will call and say Do you want to convert Am I right These guys are responsible for that Ah same stock market Very big example Yeah Same thing with all the time right Because tomorrow if I get in a safe for to
2:37:11
rest up close doesn't make any sense right now Since you're interested in this I think in your spark or streaming is not very back Okay But I can give you some examples because what is the difference This storm can processed one even door So meaning one credit card transactions don't can process
2:37:32
I really want critical transaction It will process minute sparks streaming can process only bunch It cannot process one meaning it will collect Probably two seconds Worth order Time section predict I mean so that's for micro batting It is creating a batch And then from that it is processing an area Exactly 100% real time is only possible It's John and Fling goes so does the same I think I know that a lot off information and passing and got off You're actually almost like dead by hearing this
2:38:03
term Sandoval But I can't really help it because I'm not making it up You can go here for that Andi Probably now It may not make sense if you learn listening to me but probably later when you start working if somebody talks about sparks streaming and the use case everything okay Okay this is what he spoke So I see the same banks Whose case was that If somebody is clicking you they are order sparks screaming court Okay now I have a question for you Huh Huh
2:38:42
And this security is we were not handling I'm not 100 a sure how it is working Very use case was only limited toe this right But now I have a question Okay The question is very much valued You are making a really card transaction on spot streaming Catches it spot screaming catches it But that is the real time crime section How does it know that this is a fraud
2:39:06
based on the previous time section
2:39:14
Fine Yeah The bass day does That s o but how It it's working machine learning They could have built a machine learning model
2:39:22
a profiling all your past habit and had openly spot spot Emily
2:39:32
Yeah they already have historical later or this customers transaction is here
2:39:37
These are on top officially or personally this is turning on top officially or personally this part of
2:39:44
the new installed spark Exactly So spark is getting installed on top of her dupe early Yes dear First is the stories was back
2:39:57
So if I'm making a credit card transaction today or my previous time sections are already here They would have already built the machine learning mortal And if you simply feel this new transaction it can predict the rest How do you compare with bashed it I cannot turn a bad job It'll take a lot of time right All the customer later So they will run I am a liberal It's one time 11 spot machine learning more ability running that seriously comparing states afford If you want to
2:40:21
compare the batch at the rice
2:40:24
are you this See now I'm just making a transaction right It has to compare with my previous transactions torn distant with him a fraud So like he said I was traveling on Leave it in tonight for the last 10 years
2:40:40
I am swiping in New York
2:40:43
So you're Swiper if you want to understand this and I normally you should compare it with previous swipe Right now each time you cannot go in complete for each customer So they build a machine learning mortal based on all this data that can if you figure date and can predict and all whether he will be a fraud
2:41:03
No no Kafka Okay so Kafka will not send spark will get because it is a publish subscribe system It is called spark Streaming is an application which will pull a Cathcart off We can get it from Kafka I get fine
2:41:27
No this is your permanent story Spark can read from here or so from here Spark and weird from a variety of sources Like one week by destroyed So this is your year time data right I'm just reading it to get my current information All my orders under the information is going here Only Write the golden corpus yet So if tomorrow I want to run Aspire job Yesterday's data I can break formation Eso this orders inner data you get from here current data and there'll be a
2:42:03
machine learning model which is already built Based on this data it will not leave the data the mortal will have already read Then learn That is so You guys have to teach me this You are learning machine learning like not me You have to tell me how it is working Like how Like I am a great engineer who are just your machine learning right Your topic is machine learning right No I'm saying in this HD a first you have all the transactions made by all the customers for last one Your credit
2:42:39
card Others are opposed You feed it to sparks machine learning algorithm It will build a model which if you give any new data can predict whether it is fraud or not It need not happen here Hi This was but this
2:42:56
back to you Have you are just in the machine learning I'm saying how Huh A bit It will not directly compare them Get this I'm talking or liver fraud detection So you are use cases Your this thing right Start market price variation I commend It's not a competition again Starting place starting
2:43:41
Okay so my point is this historical data will be huge It's been terabytes right So you could have already created a machine learning mortal by fearing this beta which if you give any new data can predict whether it is fraud or not that is happening in real time I cannot say that I won't say if somebody swipes a credit card I will tell you whether it is for tomorrow Doesn't make any sense right I could do it immediately Now the flip side is that or this requires resources Nothing is going to
2:44:11
work if you're different Half of sources So you need to give you enough for the sources for this thing to work Machine learning in order Very resource hungry CPU power ramp Right So this cluster should be capable of doing this anyway for the time being left small one from Kafka Right So we have just taking a use case right now I'll go here once your data is initially a first If you want to do batch processing that the four framework is my produce What is my produce We will see tomorrow But
2:44:41
I have my data I want to go back for processing I don't care how much time it takes One day can pay I just want to process the date I can use something called map reviews Right You also have one more guy called bigger This is almost going from the market Pig a pig The animal pig Big disaster Pinkel on her do but not very popular now because of spark So nobody is actually So even if you're not aware off big it is perfectly fine I think Okay I'm have a buddy and then you have somebody called
2:45:13
how you this is there in your syllabus so high viable teach in the next month But basically what How you allows you to do is to write sequel on top of her Do But the drawback is you say create table on sell It counts Staff on the table hit the query It will convert your query into a map produced program It's a translator
2:45:38
It is not an atheist equal in gym Yeah if I'm writing an article equity or I could really executed for me if I like the highest query map reduce well executed for So you will understand when they speak more about how you doing Buddy So huh
2:45:56
C c plus plus Exactly So my produce itself is very slow because this batch processing in high view like the sequel query and how you will not exit your vehicle it'll right um apparatus program for you Um mostly Where How the spark and we configure Yeah so But the problem is high was the origin of total It was built in Toto's and fight by Facebook Okay And at that point it was very popular because there was no other Torrey Right So people they're writing high if queries even though this low it
2:46:28
is equal Right See Which is better You can suffer some Sloan s audio to learn java You can stuff a secret light I so idiot of the minutes It is fine Yeah So Facebook developers start that in start off learning java Right We will learn We'll stick with sequel You tell your Children they created the store called Life So you found out how you develop where I don't have to learn anything It's equal I say create a real create able join But once I said join
2:47:02
it'll write a map Produce for me on the map reduce learn I may have to wait for In fact in flip card there was a hive query which took around 12 hours to run to a little fire Tomorrow they'll get Paracels 12 hours a city pickled Remember he was a one off date obviously But then they find unit and or But that's a different story But so a lot of time that's batch processing So there's our match groups These are all batch processing systems built on top of her Do right and
2:47:29
then you have spark You will learn spark later I don't want to confuse you a lot So spark is there It's on this guy's inside spark Okay when I wrote it here this gaze inside Spot Spark is an in memory execution Engine spot is again classified as a batch processing system but very very fast Then my produced only difference is that spark programs are faster than my previous programs
2:48:01
near real time Not really So spark streaming is actually really remember saying Merial point not like 100% Israel Ha ha There'll be a buffet or days like one second to second You can define that and we process like that I cannot have microsecond a real time That's not possible So yeah so m r and bigger obsolete high It is not obsolete because high yesterday ever house very you create a breeze and store the data so this game will remain But now what is going to happen
2:48:37
Spark will connect with life and spark and right Uh yeah So here we can bite us equally Okay Spark has a spot sequel library It can read papers from high even grand on top Off that it's a spark without start talking Too high so high It's not really absurdly high will be there for some time We can say no nicely if us started this SPFs A mother is biggest gorno There is nothing because almost
2:49:11
because the scripting grow so big or civil can murder career Democritus
2:49:15
obsolete Sparks Parken v not new versions Park has practically replaced pick actually So this is your batch processing frameworks right now Some other guys came and they said that So that was a typical question that I asked during the training
2:49:35
So if I want to run okay again there is a confusion If I want to learn a very big group I query Let's say I want to read the full table scan and run Equity High is great Even sparks equal with high was great because they will read the whole data and process it But if I have minute queries there say select something from the table their name equal to the ground Only one row It needs to process for that Why should I lower the fluid later and process It is useless But how dope say is Lord
2:50:07
everything and sequentially and then process That is only possible That is where you have this MPP engines
2:50:12
Impala There is a one guy court impala that it's another guy Hiker kike or hawk Hawk on L A P There is something called Presto but still there is also something card What is that
2:50:31
Really Phoenix These are all MPP engines Just just be aware of the name You don't have to be an expert on like there is no chance where your manager will ask Okay Name the top Fire Lily Where What You say
2:50:47
MPP engines or you're fired Nothing is going to happen like that Any one of them will be used Actually Impala is clouded Are specific Cloudera created in pilot so it will be mostly available only included Out What In Pilot This is very simple It has its own execution into new fire A sequel query It will run it for you Norma Produce Right So what is going to happen for these queries like their name equal to the ground And you can fire an Impala query It was can only that data right So
2:51:18
it can connect with this year First on pick only the later Because it remembers the meta data Where is for data available You can call him not a conflict their data and query But then you will be wondering Why can't we use this insert off HaIf
2:51:34
if on these days are very fast So if these guys are so fast then why don't I use them instead of high They don't have reliability They are all in memory queries even Impala queries running in my heart of cluster If anyone off the machine crashes my career will be gone If the highest court is running even any mission crash good Evil complete It is my produced built in redundancies Them apparatus will never fade
2:51:59
What's part will never fade Yeah
2:52:08
no no it is loading the whole later memory than really analysis happens right I pray that Uh huh got it I swear to that No no no before So when you say select star from this table first that table eight I will be ordered into the ground There only breeds from there Right So initially the whole data is loaded Ah exactly That's a problem But if I'm using Impala okay it has its own meta data It knows where is the data it lords Only that inquiries will be faster So if I
2:52:45
run an Impala query considering a haIf Korean pilot quit isn't much much faster but no for tolerance
2:52:51
So for this very fast queries like your queries can take like two minutes three minutes time You can afford to lose equity using palla But if I'm having are ideal job where I don't want to want the job Tofail run it using high eagle spots equal because these guys will never fade That's what you it is distributor It can turn on and number of missions We will be going because there is no I will show you tomorrow In my produced little persisted data intermediate restaurants That
2:53:21
is how it is fault tolerant Impala is all in memory Say formation crashes That data is going what was in the rhyme where even leveled so all and PP and genocide like this I think in the tradition of our door So in this are the differences Impala is on Cloudera How can they let LaPierre Horton Foote's there is a competition in Pilot was originally created by clothing so they promoted If you're in a closer a platform you will see Impala If you're a Horton Books platform
2:53:49
you will never see Impala They will promote Hawk There was a hawk is the best engine again A little It is the new Russian off hawk But I store drill and all are a similar ones but not by report available This die is very very interesting So there is something called each base I will add it here
2:54:08
Somebody was asking me What is it Space It's basis your database off her No secret later please Real time data is real time because a space will get installed on actually office That is your stories right But it candle random rear some rights not like entire block So now believe in your reading The data from HD Fosco to read the whole block right But it's space has an FBI So if you install a space on top off her do it can do random believe some plates individual records It can bring them
2:54:44
right and process because it is no secret database off her Do the default No secretly copies But the drawback is its space has its own language So if somebody has to work with this space here to learn the language it will not allow sequel queries Finex is secret on its face Phoenix and space is like 40 and paneer butter masala Deadly combination I've been very lately combination because you're writing sequel on top off North sequel but I immediately so they say that in another fool you
2:55:21
and not just low It was really fast because I space itself is fast sequiturs A conversion talk about that's already built in any sequel Query There are some limitations but most of the secret No it's not faith So Phoenix will get all the meta data on already all the queries Arkan murdered Whatever queries equally writing it knows where there's a will in this space query It s not so it doesn't just push the quitting so you just write a sequel Little knowledge base and give you
2:55:46
the research on it Kendall things like secondary indexes and or it can index Your data apart from its visits Very fast actually Now the real news is that in another two years what is predictor is that this combination The drawback is Phoenix is still in alphabetical
2:56:04
Phoenix is not production ready one Cynics becomes production ready This is oil BP So I'll be That means how do Bill provider or happened Oil pp huh It will support in future Not now So they're trying to make it Asif support so that this endeavor confidential become an oil KP system like that play So then what will happen You will get transaction management in her do So far it is not there Another four years down the line They say that So Phoenix is getting integrated with this piece or is in a
2:56:38
list This was only there The drawback is it doesn't understand sequel So Phoenix started Phoenix is a layer on top of that You can write sequel So another two years they're developing This s a system for transsexual management So you'll get through transsexual management And so Yeah So if you talk about these louder and or if I'm going to cloud it up and say that I want your predict They will give you all this in one package But if you go to a party and download you get only a steer First
2:57:10
map reduce onion Only heard all this year two separate lean start
2:57:14
So commercial distributions well packaged and you will learn everything off For example heart will not be there This press store drill Phoenix you will get its base hive week my produce Park Impala So these things will get you have a doubt I'll show you that in the cloud A page Yes There is something more spark Secretive Very trance Equal query It is really fast Actually that's what the tables you will be competing in High Spar doesn't have storage Right So whatever tables you create there it
2:57:50
can connect Denver So I'll say Clodagh CDH the It's based on its own processing engine It's on processing engine So this is clouded as predict Why I'm showing you this to tell you that I'm not blabbering Okay You may be thinking that this guy's talking about a lot off doors Do they really exist in the world Look at here Look at here Spark how you big Mac radio spark Impala Ah yarn right What You have a CFS space cafta from school This is a real product from Florida You get go to
2:58:25
cloud read This is what you get Some of them I have not even explained because I don't know guide I don't know what it is To be very honest
2:58:32
Guru is a new file system Relational Even I'm not aware of Google I don't know what it is but these are like new employees You even if you don't know them you will work without any problem on four security you have something called sentry It's a lot of people are already worried about security on her Do my little service called sentry which will handle like access management authorization or the indication all this will be handled by century We covered these things like these things we covered
2:59:01
this week over this week A word something extra Howard wrote I mean that is okay I mean he had this part right express total Phoenix and all its nor there But I just wrote them
2:59:14
one thing which is not present there but it comes is easy
2:59:21
This I told you right This is the workflow share dealer like if you want to share the lover flow and or you can do that I think we can do one thing We have half another left and you are like almost tortured I show you a cloud Lackluster However that that could be better than talking a lot So I keep on talking about this night So I show you a cloud lackluster We will not do the hands on today in the plaster via using for the lab We have only read only access from the ATM insight which
2:59:53
means you cannot change the application factor or block sites or anything These are old environments We can upload the data and then say that run the program that will work but that we will go tomorrow But let's look at the cluster Right So so far we are talking Cluster cluster and we are not seeing any cluster so far So I think clouded Horton books and map out These three are getting used a boat or the three companies are sort of famous I can see Right Okay So do you have this in
3:00:29
your elements like a lab and or just check warns Yeah so because some of the things are a bit confusing So I could tell you we have to use weapons or Web console will connect with the Lennox fire system Where hard Okay since Or we will do that later Clouded Our manager is the admin interface off the cluster so I will connect with it But we will not be able to modify but they can see stuff So if I goto Cloudera no not clouded A manager is a utility that comes with
3:01:13
Claudia Toe Administer the cluster like you want to add a service removal service at a machine removal machine So the administrator if you are a hard group administrator Leotard war these things are users Remove USA's permissions everything and build and make loaded I managed And why I did not speak about this because they will not be doing anything You send men utility But look at the cluster This guy's good So let me show you that I know that I manage
3:01:41
Now you have to install it So if I am running in a company I hope the party s let's say 20 machines physically Yeah and then I have so there is a process of installing her Do This question already has in store It is earning Callister If you're creating from scratch you have two parties slips a 20 machines and download something or Clodagh manager and run it on It will help you to set of the cluster It will tell that which is your master which is costly What is your block Science
3:02:07
What is it It is late
3:02:10
worth How cloud is also This is physical This is not even physical This cluster is actually running on AWS Amazon Web services Uh practices You will have access to this right I mean if you want to practice this set up is already available to you Right
3:02:37
Is giving Let me try to log in Just probably You don't have access I don't Yeah so just pay attention So this is your cloud at Leicester Okay this is just an admin GeoEye Okay put the cluster and once you log in this is your dashboard
3:02:57
So you are looking at a Hadoop cluster First point your gun Mr Second point What is there in the heart of plaster These are or the services installed See Actually a first Why am I talking like this I can see it here Right Ok Yes Dear First high shoe I will talk about it Impala Kafka Uzi Solar spark Yarn Zookeeper All these services are turning in the cluster You see a green right This is just to get a quick overview off the cluster Like what is running in the cluster Fine How
3:03:28
many machines are there in the cluster You click on this host menu Can you see Can you see this machines These not over machines 11 morning Right So how many missions So that I think that our nine machines say good health Nine nine missions are there in the cluster on You can also see their configuration If I scroll here can you say they see their ram and hard disk
3:03:56
What is the physical memory That is the ram The first two machines are first four machines are a G B ran next 2016 GOP ram These are their heart This can you see And you can see how my story just for land All everything right See here So this guy is having 32 0 am on 18.8 is currently used Some process will be running This guy also had a 49 g b hard disk 600 GB Hardness of busy We heard this You can oversee from a single window It's very easy So it is not as complicated as
3:04:29
you thought but right So this is just the host menu right You can Just from Florida manager this is the home page Click on host Host You will see this right now If you look at HD efforts this is actually office play Cornish dear first Okay you will see this HDR first Manu this is all the later treasury offers Initially a first What we learn Replication block size So where you can figure this There is something called configuration on Do you want to change the sea It is saying Read only I
3:05:03
cannot change it But do you want to see the replication factor off this cluster BFS Sorry BFS dark But is that application factor
3:05:16
But he click there You can change it to two or 10 or whatever you want If you are an admit you're not an argument Do you want to see the block size Bir first or block size can you see So I can click here and change if Amahoro Birdman and it'll be applicable to everybody immediately and all somebody was asking about violence That right there is balance that I can say actions Okay the problem is that some of these things I will not be ableto run
3:05:48
bad instances This is balance of it is running But I can run a comment house Saudi It's not green It is not running If it is running this process will come as green Actually um then chart commands configuration Um so that is our different figuration Okay so that is one thing we saw on duh host minorities already Right So so far we discussed only SPFs And here is Kafka If I goto cafta just goto Kafka Okay Kafka say's Kafka broker three That mystery machines are running calf guy to look after will
3:06:34
be running in a cluster Since this is a lab set up we are running Kafka inside her do properly I really run it separately because they need story spaces Toe hold your data But here CAFTA brokers are three They are running inside here only because this is only close Really Have I mean these machines If I go to these machines I will show you If you go to this host menu you can expand a whole stance See what it is doing For example if I expand this day see What is it
3:07:09
What is your name Nor right So And also I can go to H DFS and see that for example I go to a schmear Furse and Instances What do you see here Active standby Two missions to name North I told you right I even stand by there running Right So the same missions are running Calf guy here It's not a separate cluster by the way That's what I meant by that I don't have really admin access Otherwise something What I've been doing I don't want to mess up the cluster Or so by the way I saw some
3:07:45
of the minnows are actually disabled We can't run them See exactly
3:07:53
I really that should not be the case because then it issue not utilizing the proper space like so I really will set up a separate broken on that will download the data Actually uh they're all running on our Jarabulus Actually
3:08:12
I don't know I mean I don't have access to that can be anywhere It doesn't matter Hear the music and create a cluster Any of it I don't have any idea as to who created this custom And there is a third party They just give access to us Even I have only read only access like you guys have now This is to just brush for tomorrow Tomorrow will be actually doing it But I want to clarify one more thing If you go to this big data cloud lab this cloud a manager you don't really need because I've
3:08:39
started Mean interface You are not going to rain near the activity
3:08:43
The important thing that you knew this weapons sold So if you click on weapons Sword I just clicked on that For some reason it is not opening up you see But I see anything Why am I not getting Logan You can log in but if I have to show up looking like
3:09:06
so there is small honey she's not here Yeah you can log in that This that will connect you to one off the machines in the cluster and you will see Lennox you can just log onto one off the machine That is what you see that And I'm not ableto logger Uh you're getting really next show Take some time No I'm taking a long time And there is a There is a product called Hugh This is very important There is a protocol if you this Hugh is a third party application Okay but it is very very popular Why
3:09:48
do you need a hue Is that if you're a developer and you want Oh APL order your files to her Do you can use you one off the use cases for example I can click this help like you And lastly user name and password if I sign in Okay I'll just sign in if I click on this issue here first But I also see there This is my how do these are all the files in my heart So he was like a February heard It's the office file system So these are all the files and folders If you connect you will not see
3:10:22
anything here because you are having a fresh take on one more very important point I want to tell you Is that in Ha do you will have a home directory Yusor slash something I have GL faculty You will have your own directly Can you try this shooting on your PC Just click on this You type you something password See whether you can see this thing Are you able to see Another point is that sometimes the cluster will become very slow because it is not designed to accept request from 36 people Right
3:11:02
If all of us like the log and it was their timed out and other things No that
3:11:11
What is the name and password They're typing empty That's fine Because in your heart do home directory that have no face I have already around something So that is why these files are there So we will create a tomorrow Don't worry We'll create It isn't fight
3:11:30
Uh oh The same plaster so different Different users are created in the Hadoop cluster So if you log and you will get a home directly like in Lenox in the next what happens if you log on You will get a home directory So this is my home that a three initially efforts
3:11:45
and Since I already created some fires I can see them You may not be able to see because it is a fresh home directory for you Tomorrow I will show you how to upload the file around A sample Makris program explained the program and we will do that tomorrow already So there are some unanswered questions I said I will pick it up later I forgot Also Anybody had any questions I pushed Did I tell you I will show something and I forgot also some other things
3:12:18
Ah got it No I could be in a non commodity hardware like a Really So how Ah
3:12:34
the problem is I'm not able to connect with the Web Consort
3:12:38
Say it is not giving the option to commit I need to connect to that to show you something Hugh Kafka in by allies running here Yeah Okay I missed the concept I wanted to talk about it It's a very important concept right Do you know what is a black You know What is that Lack Settle back Iraq is like Iraq right How do you explain for is a vacuum and I don't know explain I saw this and or because from the point of view off her do this might be important Sorry
3:13:38
I will explain this one and then pick up your questions You may just or over dot com
3:13:44
So this is for people who do not know serve Iraq Do you know what is that like No this is a lack It's a housing Very poor Sure the service in all the high five More basically say somebody sitting something typing something in a very big data center and all right so that's Iraq Why racks are important
3:14:06
They're important for one reason Because
3:14:11
can I raise this This Bagram
3:14:16
This one even I don't know what I wrote That's fine So why blacks are important this because off one particular reason See let's say you have three racks you are having relax okay On each lack hazard a tunnel not one that our meaning So that's it Your data on on Sunday So in reality when you're installing her doping a cluster this is have you right Because in our data center we will have the axe and it will have silver's this Those are called your date annals right So I have six state
3:15:26
and orders in three racks Now there is something called Iraq awareness in her new meaning If let's say you work or being a block what you can do this first copy will go here The second copy will go here Imagine
3:15:43
on the third copy will go here Meaning if you enable this feature called rack of infamous How do we will ensure that the blocks are spread across my people Relax Why Yeah Iraq can feel rackets upper window failure Because normally you will supply this power and network everything toe through lack So if Iraq fails if I'm keeping all the three blocks in the same right if that rack face then my blocks are going So how do you can if you enable rack awareness You have to
3:16:22
tell her Don't originate on orders in the track If you do that it can place them like this But why don't it keep like all three in three different grants Why the last off The cop is not in the same rack Yeah but same location means these are all in the same data center Right So this indirect bandwidth is very important Wracked Iraq the bangle is very important So this is just replication But you will be running a lot off programs which will use this banquet
3:16:59
so it to save that bandwidth What it This is that the next replication will happen within the rack server But this is enough Now even if one rack on all the three racks going down is very remote possibility Right And my point was I can show you that here So when I saw that if you go to our host can you see this
3:17:24
So in the rack 14 other racked toward the rack 32 are there on before cracked Whether before it means I don't know They have kept something here but for two So this is actually the distribution It is actually available in your clothes A manager See that
3:17:43
No no no no Each rack It's not us A step looking Just one not one Many will be in one rack right See here all the masters will be in a separate track See if you look at this picture right in this machine some 30 years later notes will be there Each slaughter will have one server They are all our lax over So server It's not like this is like this like up is up
3:18:15
You get a bizarre How does the bazaar looks like That is have you are so well we look like not like this There is also that kind off So what Our service are there but they service to pick leaving You saw these windows I had a picture actually Yeah Who had a plaster That Yahoo How do cluster at Yahoo Huh So the biggest hurdle cluster on Earth is with Yahoo Actually Yahoo horse that a court for that They have Ah 42,000 machines and still they went bankrupt solely to some useless company
3:19:03
Hey the picture is not visible actually Where do you go Diarrhea This is No that's a good question So can I have multiple name North Yes The concept is for federation Meaning Ideally if you cross more than 5000 date on horse you need one morning or
3:19:28
like 42,000 machines I cannot have a single name nor to handle because the amount of matter it will become slow There is a problem So ideally Yahoo has done some aren t they said that if you cross Northern £5000 hordes you need one more data That's full name Now that Sport Federation Federation means my dipper active name North like you can have 10 name North All will be working their share The Lord actually Right So like that federation is guarded
3:20:01
Yeah so do you Ornament bracket actually inside the same data center Only unit All of them right Because all the data and or starting one land So the name Little socially inside lightly But they will handle the load actually Eso someday Donald Maybe not this name nor that they can handle Sorry Just a moment Files over can store but it's gonna process right
3:20:35
That's the use case Paducah is not only for story Whatever data you store the data can be processed then and there by the machine Got it Cut it So there were not using her Duke Right I we have been discussing beginning off the class No sequel databases If you don't want processing If your use cases such that I just want to dump a bunch of images they say one million images I just want to process them displaying an e commerce website I don't use her Do Her job is notice for that
3:21:21
There you have no secret later visits like Dynamodb or Mongo TV right Yeah SharePoint If it is a smaller set of SharePoint you can use it is which one
3:21:37
Haste uh haystack is built on top of this or top off this Yeah Facebook's right credit credit So it is our business use case The question is that what do I want to do with the data There are other companies who generate a rabbit Soft data They just are crazy I just did generate data I don't want to analyze I just Arcata later Who cares So it is up to me to decide What should I do with the data Right
3:22:11
far huh I was working with the very interesting use cases G right The company corgi I have a training next week there So they have an aviation department G is building this aircraft engine You know that flight engine 80% of the Camaro commercial aircraft engine self from G e and what they do so most off the planes are flying the G engine There is also a rolls Royce on one more company So are these aircraft engines are having sensors So when the flight is flying they
3:22:47
capture the sensor data like pressure and blah blah blah And they analyze it to predict when the flight will crash or turbulence will hit or something like that whether you're gonna die or something So the idea is that But so I said OK good deal You're doing this They said No it's not a good deal The problem is if a flight is flying from let's say Dubai to Bangla
3:23:08
the living two engines it takes six hours Each engine will generate one TV later How many flights are there in total and how many flights fly every day Can you imagine the amount of data they don't analyze How can you analyze I mean if I'm giving us 1000 a rabbit every hour or every minute you can't analyze it So they take us upset off that they haven't algorithm which takes a subset of the data and analyze rest their delete and dump because it is not humanly possible or so you don't need
3:23:38
that much more video sensor data any norm analyzed These are interesting because they also have locomotive engines praying plane all the Europe and us there on trains And these trains are having sensor So they get also that it's very difficult to Arkady because use amount of data They also don't know what to do with the data right And they have no People have to say it makes sense of the locomotive data issues actually so the trains will generate a lot of great actually so they get
3:24:06
it on then they take it on clean it and just get on the part off the data and analyze it in her Do you also have certain file for Mars for example Avenue you may not have heard about it after you is a file format we use on Lean Heard Barket who are see these are all used to compress and store your daytime Heard You can also have compression techniques but sometimes storing their data as it is It's not good Why don't you compress it But what is a drawback Off competition
3:24:41
you're to decompress You need a lot of processing power if you want to decompress it So I have realize the serialization for my it doesn't compress the data parquet and all will compress the data solemnly and push the rector So I will show you in high when we discuss high When you create a hive table you can say that the data has to be compressed Welcome presents ordinary faction And that is possible So my point is not everybody want to analyze all the data in the world that is also human Leaner
3:25:11
Think about Facebook How much take Adele begetting Probably They're not analyzing the whole data A subset off interesting data that is old beautiful
3:25:20
So any questions will find about 6 30 a thing But before that we can pick up huh
3:25:30
Yes Sparkles memory and then see idea He spotted a question So if you give Ramit he performed If you're doing the revenues Hartness
3:25:39
I mean rest off the space and lose hardness Performance will be slightly degraded So for streaming And are the new Lord off The problem is um for board World Cup is going to happen right
3:25:54
Yes my rent this year only Right So let's say you want to write a program which will download all the tweets on find out the most favorite player or for Barbara boss of All right so I can easily stream the data from Twitter The problem is I want to find out you know the most popular player every five minutes Imagine Probably not five minute wonder toe So I want to get five minutes or data Hold it in my memory and process What a five minute I saw a huge I cannot hold it in memory then really slow
3:26:33
Yes I'm just giving it a my imaginary situation But if five minutes or data isn't that are bites you can't hold it Action
3:26:41
Not a little any data that you have The new might run the person to disk some of the data Then processing will become slow It is not really anything that so When you're running spark streaming and oil your ram should be available Are the recipe become very slow Today's match will predict tomorrow right So things like that can happen So that's not really good actually So a lot of considerations are there where you have tow use or listening So I think they will wind up What
3:27:09
do you think Yeah so it's almost 6 30 or so yes So I will be sharing slights or so books So there there are a couple of good books around How do we should give you a lot off idea about basics Like how a city offices working or these things Actually um so I would say whatever I'm training here this enough But go through the book or so say for example actually if first you don't have to be an expert on high availability see that is an active then understand by name that they faced and ready to
3:27:52
take over how that happens That is a process So probably that topic even if you don't know it is not a big deal because they're centered Main topic If you had a hard job admin you should know how that should work But since you are a develop you don't have to go in depth into those topics Actually some are produced and how you should be more than enough and you're building this awareness to learn spark The basic idea is that you're building this so you can look at spot So I say
3:28:19
Concentrate on Spike So let's talk about hive right which you people might be mostly interested on but I'm not so usually I'm not a slice person but for high If I need some slight this has on 60 slights I will not run in all off them Don't worry I'll share it with you You can go through that I learned a couple of them to get an idea right And so that always everything out Woodrow and explain to be difficult for height right so high It is actually very simple In 2005 when
3:28:53
Duke came most of the major a vendor started using her look like Facebook yahoo and all But they all faced one common problem
3:29:02
The problem is that Yahoo started using Hadoop But yeah who had a lot of developers who did not know Java So the problem is that they had to write my produce programs on duh skill set was the problem You can't say that everybody should learn Java and start writing court So yeah who found it Very troubling Same time Facebook also started using Hadoop and they also had the same problem Facebook heard of mostly sequel developers Onda They were not comfortable with Java on way back into
3:29:35
those and five and six we had only map produced There was nothing else on her okay Playing my produce nothing else s So they had to be consistently so in the last class And I know that was available so everybody was confused So Yahoo and Facebook started to projects same time So yeah who created something called a bigger the creature Big animal pig
3:30:02
Serious Xcor pig Big dart about your daughter Org See that's a big really big creature So yeah who created this tool called It's a scripting language I teach a course called a big on Her Do only on big so very interesting language scripting language Uh when days is very simple you learn this language on the new whatever analysis you want you write in that language It's a script So it is small like four filings max or 10 lines on Once you write your big script you say run It will convert your
3:30:42
pig script in Democritus program so you don't have to learn my produce You learn big on somewhere in photos and 10 and 11 There was a heavy job in floor for big developers 80% is off her job job was handled by begin our 10 2080% is
3:30:59
it was such a successful language that everybody was dying for Pig I have taken only pick trainings People aren't because that was the effect of big actually like because it's a very easy language scripting language You just learn it on Everything will happen behind the scenes You want to do a joint operation you say join a coma B That's it Even secret code is a more complicated Enjoying your to say a heartbeat or hear nothing Take a be joining a coma Be than joining them Simple
3:31:28
inner joinem saying So people started using big extensively but at the same time your Facebook developed haIf
3:31:37
hives Idea is different You write sequel okay you create your data basis your tables and all the blah blah blah you write Sequel
3:31:47
on High will convert your sequel querian to map reviews Same concept So to companies developed one big and one haIf They didn't know that they were doing the same thing ultimately But then what happened somewhere in 2010 11 A big started losing the attraction We go sparking seven Spark came and became predominantly useful Spot is also very short and I like a script you can write in fighting and all very easy to write So spark and bigger almost similar So people don't like If I'm learning
3:32:20
something very short and interesting on very fast why should I learn big I can learn by a spark itself right So biggest no longer used by anybody Very literally Biggest gone almost height is not gone because it is secret Secret will be there little the world exist right No questions asked Right No questions Sequel is equal so you can get rid of secret Like how do you get rid of secret What if all the upl developers and you know there's a huge community toe back secretly
3:32:52
Not like big I actually like big more But there is nobody to back big little order people who back secret right And then that is not the only reason Hive is a beautiful data warehouse
3:33:06
It is treated as a data warehouse off her do meaning you can create tables and you can query the data So most of the visualization tools and our you know can use high Even the back and to fetch the crystal sandals So how you became very popular Onda Now what is happening Hive Is there plus spark ISS Using high in spark you have a more your core spark sequence
3:33:31
that uses sequel language Now that guy can process itself It can also talkto So whatever data you have in high spot can or sorry so high is predominantly used these days and I will always be very popular It it may be I think one of the most popular ecosystem Pulis height so far in the industry Because it's been 10 years 12 years and still uh community is very much so The command wise It is very simple because it is sequence on I don't think I would repeat you sequel That's not the
3:34:05
idea behind this session You will already know Sequel but they will just see what is happening behind the scenes If you write a sequel query on some of the advanced concepts in hype like partitioning bucket ing indexing these kind of things on from the industry experience that IHS One major thing There are two major vendors Horton works and Claudia All right Now Horton works uses original Apache How do that is Their propaganda So what high resource Apache predict It's open
3:34:41
source night and whatever major changes other in haIf are predominant important books not included Cloudera is actually not sober That hype We're using a cloud A lab That's bad actually I mean I'm not blaming the lab but clothed I It's actually a bit bad in high site I'll tell you the reason because I teach at Wharton Books That is the reason I'm able to tell in Horton works they have a project called Days Some weird people from Delhi created this project Days is a Hindi word
3:35:16
There is a weird movie by that name or so right Have you heard days They stays speed in Hindi huh So days is up How do you say it's not a replacement There is a project created by Indians Then they gave it up Our chief became open sores There's this faster my produce So what this guy's did normally map It is what happens to read the data right on DUP Then you do mapping and then you store the output in hard disk again shuffle You read the data Do the shuffle stored
3:35:53
output back and hardest they can reduce the will read from hard This or some beard I start Why don't we do all this in in memory We don't want to use hard this Read the data ones do all the map reduce shuffle and then push terraces Serum apparatus becomes 10 times faster That is face So it said Daggett director That's likely graph concept on this was given Apache what Horton Books does If you write a haIf equity they will convert it into what is job not my apparatus job So haIf queries
3:36:25
are very faster on Harden works in a cloud A cluster cloudar doesn't support these because they is this contributed by Horton Will important books They are commentators right So if you're working on a cloud a cluster you will never see days in your life they won't allow So the problem is that if Emma writing a hive query on Claudia it's very slow because reconverted my producer NASA may produce program takes a lot of time Fan I think the same HaIf Korea Important works It's at least
3:36:53
five times faster because it converse with days job by the fall of the space on days This superb You can't compare right So that is one drawback I mean So when you start working in industry so these days mostly people are more important works for majority of reasons But the steak is a cloud Only number off uses are more for Claudia But over a period of time probably people will shift the Horton work so they start button information So then you will be thinking that from
3:37:23
Cloudera there is no answer to this No they have Impala There's a product called Impala A parting partner in pilots Contribution is mostly from Claudia in Pilot esta staying thing work that pays this in memory fast Really Inquiries So I show you Impala We have Impala in the cloud cluster I show your bottom of questions if you want So days itself as the project is not really Uh yeah Not really Uh how do I say important right now since parquet most of these things are going
3:37:57
including tastes because spark is also in memory daisies or same memory So even spark came taste Relevance is not there But the only relevance is that if you're running a hive query It converts it into today's jobs It's faster actually
3:38:12
or Norton Books Platforms So I will show your heart and with cluster probably tomorrow Okay I have an access I'll show you that day's job running You'll see the difference actually So a spot But there are a lot of companies who are in tradition high only high like they don't want to spar with this one He was high for them going to a cloud around maybe a bit challenging
3:38:33
because hard earned bucks body for give you three days right So days this fast or so they is This far tolerant Impala is not I'll discuss public tomorrow probably to a liquor discuss but that sequel engines are too many Actually in this category him there is a fight competition between herself them and it's all marketing gimmick You go to Cloudera they will say that implies the best in the world Never listened Horton What investors say that on hardness recently high
3:39:01
recently announced something called Elope about a high elope is really inquiries
3:39:09
Using high high is normally batch processing because it is converting into map pretty right But recently a passion arms there How you can support realty inquiries That's called elope
3:39:20
Um the real abbreviation I don't know Long leave and process that is not the real one But we used to say Long live in process It is still in beta Not so but the problem lately is available only on her Converse No clothing open source right Because it is a contribution off open source high We just have a level only on Horton but slow direction or get it so they're still with the Impala in violates the only solution they have
3:39:51
So then you talk about higher than a lot of things where you should be aware of the industry What's happening right So these are the reasons why people are shifting towards Horton work Some of the major things I see comfort That doesn't mean cloud rise bad Okay It's not like they are also having great support Their technical support is excellent and even with this Impala there who a lot off to eggs actually does So if you have a problem they leave a solution But a lot
3:40:14
of people are preferring this open source initiating or supporting Listen Orla
3:40:19
open source but the problem is I told you that I was with G They used a little Ap None of the queries work most of the time It is open source right If it doesn't you can't blame Cloud And I will fix it because it is preparatory like I foreign end on drawing Great So open source You get Lord of Features but it will work on normal You cannot guarantee that's one thing but they will blame internally on Apache They say that will give you technical support within the limits so orginally
3:40:48
the product is by Apache So Horton Books They're not fix statically because it's an open source community Most of the things work I mean I'm not saying that everything doesn't work but open source has its own drawbacks sometimes right But I am seeing a lot of people migrating toe Horton books actually these days right So in High Eve what you need to in understand is these things It provides sequel like interface on her do
3:41:16
very important point HaIf does not have storage It uses the data in her group For example you copy a C a C five in which the efforts on what I can do I can go toe high say create a table
3:41:32
on Give this Kema Then I said Lord this year straight to the table What is going to happen to see Yes we will be in her do It is on a steer First it is as blocks and replicas and all but in high you just project a structure so you get a table Once you get a table you can do any sequence most off the sequel right So the idea is to present the structure to the data that is already available in Onda HaIf can create tables from a variety of formats Structured it even log file son
3:42:05
structure Later uh Jason fires different types of files can be read by haIf on put it in our table format So that's the importance off life So it's true that as the hard due date of their host system so the data is initially office on This is just giving a table like structure on top off The bigger I will just maintain This came only because the data is already in How do right So you don't have to load the data when they say Lord the date and the date I will just become in How do I show
3:42:37
you how to load the data I mean how to create a day with Yeah So if you want to run equity you can directly say Right you're sequel says that it counts star from the people But the problem is that will hit high I will read the query on it will convert into a map reduce program jobs created jar file So that will take some time Then when it runs that some operatives job on the funniest part is hard Duke never and distance what is high
3:43:03
for How do put some apparatus program You're getting the point from the hard oops point of view There is nothing called high It understands only my produce So whether you write a my producer high right um apparatus it's the same thing right If you want you can write a map reduce program to query structure later It's very difficult That's why realizing how you it'll translate and create a jar file push it to the cluster
3:43:28
on the plus still will run the map reduce regular program show you the airport you can save it in a table or or anything you can do so you can use all Regulus secret statements You can say in certain to our table from this table So that is still will be If it is matching with the skin it'll insert the same way The only thing is predicting a structure
3:43:46
How do how do publish DFS Only everything is honestly offense It is slow Ah hi Vis Expected to be slow So that is one thing Hi vis Considered to be a batch processing engine so it is not a database Another common problem is that people may not be able to differentiate between a data base on a data warehouse there Different right our data basis Oil pp you called transactional Real time right Very You wonder Melissa can speed a subsequent speed That's a database And you always
3:44:18
have insert update All this kind of operation are date Other houses different our data Other house is a place where all the clean the data comes and lands right from their your reporting Pores can connect and visualize the data Right So high is a data warehouse where you store your final data So let's say you have a lot of tables you ran queries and you go to the final output you stir Store it in high so that your visualize session towards can ah started next year first but we set started and
3:44:50
high but it's in actually ever suddenly meaning If I open high even say selects star I will see your table If I go to the location open the files here Text file Same thing So the next I will be there If I open insists the next are saying text file It'll show me in a nice row Corland for my and another thing is that highest languages actually called excusable
3:45:15
hive Query language It's not a skewered but it is based on sequel 90 Toes impacts More Stop Your queries will work marking but it's technically called its que el hive query language It's up to us to decide I'm in so the idea is to familiar with the family rice with high Then depending on our business use case we can write any sequel queries You can write complex Berries simple queries You need to have some idea what sequel queries like Why does the joint operations What is
3:45:45
a true by quickie You understand what is happening Even a person without knowing sequel can learn height I mean by looking at a query you can understand what is invention off the query Right Hi Vis Notre db Higher query steak minutes even for small day pass it's and can be compared to databases like Oracle HaIf does not provide real time queries So that is what the original high which skein was converting into my produce So it was very very slow Then came tastes So what Harden
3:46:18
But today they started using high plus days which is a bit more faster tonight Cloud I reckon audio stays So Cloudera still runs five into my produce on there and certain that this Impala Impala is that in memory execution engine
3:46:34
but inviolable Back on Lian Claudia
3:46:37
So I will show you Impala and say inquiry will be very very fast with there is no difference Apart from that I mean setbacks and everything saying actually so data warehousing in the sense like usually it is used as a storage engine rather than so what is the data warehouse A data warehouse is very you store all your massive data structure on the intention off five is also seen on all the be a tourist can panic with hype So if you're having something like Tableau for
3:47:06
example Pablo is a visualization toe so I want to visualize there are bits off data from her Do I can connect our blow with her Do on Pablo will be firing sequel queries using high on the day that can be replaced so that's a typical application off a lap system So it's an old lab system online analytical processing It's not an oil BP system so you can't combat it like Oracle because that's like real time But again highest cannot replace
3:47:36
a proper data their house Okay so if you're talking about something like that article for example so they are like how do I say the expensive on fast reliable that ever hopes is understand hives will run up off her due Partly it has all the drawbacks off her do right So monster of the organizations where they do these days is that they categorize their data hold data cold data The whole data will goto something like that are later No there they need to immediately faced the
3:48:07
data run faster queries on the golden gate of its It's OK you can take some 10 minutes to quit He will goto high hive supports Insert statements now It was not there before in such statements are already there But the problem is that eat in search statement will fire a map produced So you do a Bible floor that's better So in high normally when you lord the data you do about Lord you don't inserted on the late destinations off high Support all the credit operations create
3:48:37
read update delete Everything is supported
3:48:41
But again if you want to get the speed you should use a lady That's what I'm saying Real time queries lately and L A p a Is off now Pluralism support So I can say if you are on a cloud that our platform in search and are literally very flu Not like that article that I read general in such a very past But the question is that in our data warehouse usually you don't have to do an insert very rare right because you will go by uploading off the data or your 80 year tours will be dumping
3:49:13
the data there They will collect data from the rd vehemence and then dump it into the so very early You modified the data
3:49:21
So the use cases like that for Yes all the card operations have supported even insert up there delete But it'll be slow on hard Most platform you have acid support enable construction management but it is not as reliable as a database Okay because you can never replace an RGB Um a system with something like Hi but you have our construction management supporter that is always elope Elope is the feature which gives high real time performance But a little API also requires resources It can also it
3:49:55
requires more resources to execute But you get all the acid properties transaction management Everything on this is very important to understand So there is a common land in the face for height It's called how you shell You can open the high shell start typing your equity say create later This could hear they will and then by the query So that is one way to work with height But people may or may not use it because if you are you're in the production and our you might
3:50:28
have some climb right So some what type off clients who use like connected TVs and eight other houses So secrets of or taught or something know something like that You might either You can use that So one thing is that you can use the CIA Lie on high was very vast actually So I'm getting confused in a dog This cli there is Brucie allies the origin of the Allies called haIf and it's simply type high It'll open You do all the blah blah blah The new CLI is called billing for billing
3:51:06
Okay on beeline is a proper jerry basic line So through beeline also you can open the CIA life or you can simply say hi little open the command line for you You can do all the create a word statements and order There is a veggie Why But it is very rarely used high It is district lined It's not jail BBC claimed So then I opened the CIA Lie and say hi daily busy how drivers light toe connect So when you want to connect from an application right your data is you need a driver that's cordage 80 B
3:51:40
c or D B C drivers are that That's one we're talking
3:51:45
So the CLI original haIf CLS for haIf client That's not a jail ABC client You just open it automatically hit haIf and there is It'll directly hit the haIf and you can type or the queries and or the B line is a proper jail BBC Klein which means you can install it in some other machine From there you can connect you can give the server address port number and it'll hit right on It is not written here There is something called high Server This four part is for how you serve this
3:52:19
spot For some reason in the diagram it is not certain But this whole party scored how you set of it Because if you're connecting with their data warehouse there should be a server company We shall accept your queries You know whatever you're typing right So that is this whole part So whether you come through cli and there is a veggie y or jail BBC you all hit the high server right on Higher server is the component which will accept your query So once you write equity
3:52:49
There's a planner Parsa Optimizer Optimize your equity on convert um apparatus program on Push it into the 100 plus in high server itself You have high server one on high server too Okay so the old one is called High Server One which nobody is using Now You don't have to bow that the latest one It's called House over To get that this everybody is using so high server who is responsible for managing all your connection So if you open a connection with high it'll radically hit
3:53:23
this high server toe on from there Whatever query you write this guy has ah compiler optimizer executed If you get the query compile it optimize it push it as an apparatus program My I think I can show you that from here If you go toe Cloutier manager should be here It is very difficult to talk about haIf rather than showing something because talking is fine But you know if I if I show something probably people will understand So if I goto haIf configuration No Uh what is high server HAC How you
3:54:05
said over two So this is called highest ever to the defiant high service house ever to because there was an old high server one which nobody is using now because of performance issues and all so right now whether it is heartened world so low that I would any platform the server component of hi vis called helps over Okay so either you come through a jelly basic lined or the command line you're all hitting How server Okay from there it will come by their query Optimize your
3:54:36
query and push it on There are tons off optimization techniques in life todo you know really equity on better it and our probably we will discuss that tomorrow Some of the optimization techniques Another very important point in this slide ISS this meta store best thing Meaning if I open a ah hi Eva Session and I say create a table If I exit it should not go right there was the paper should persist whatever you're creating So that is where the meta store comes into picture That
3:55:14
is where all the meta data will be stored and in production set up What we do is that we will create on my sequel server on in the my Secret Server Pentastar will be configured
3:55:27
So all the main target off your table like the scheme are access rights Everything will be in the meta store All right Uh on duh So you have tow ensure that the Modesto services running
3:55:40
other ways how you will not start because it needs to talk to make this meta story is very important because even in spark
3:55:48
spark will talk to this man Castro So if I create a hive table I can read it in spark because it can talk to the same McMaster and get the data from the all the meta data will be their name attached Okay My dad's store eyes the Meta data store is a service on that will push the matar data So you have to give her destination in the So here it will be there I'll show you where the metal stories I don't know where they have configured If I go here to hive configuration right Hi Meta store
3:56:27
server trade Ah database type is my sequel The right Uh yes This is very to see on this machine they have configured my sequel and that is where the meta data will be pushed on but you don't have any use off The meta data high will read Invited That's fine Okay And it uses it to persist All the table information and baby you create and everything way You don't have much of a use for that but it is that on in high if you have something called that cost based optimizer CBO
3:57:19
CB ope Yeah I don't know why it is disabled Enable course based optimize a day saver It is disabled for some reason I don't know So this is course CB or Khost based optimizer You are aware off our TV in my sight If you write a very complicated query it will generate multiple query plants How to execute this query This optimizer So what time does if he enable it If you write a very complicated query it'll generate my people plants and based on the cost off each planet will select the best
3:57:54
plan So that is the course based optimizer It is disabled I don't know why it is disable Usually you can enable this on This is how optimization can be done in Haifa So this is one thing CBO we will look into other things Also Uh okay all the configurations are here so start Highview Simply say hi Upset
3:58:24
you simply type higher to start the high ocean may get some warning That's fine on you will easily see a warning saying that high You see Ally iss deprecate er on migration Toby line ISS recommended Er that's okay So what it actually means is that this is the original high see light
3:58:53
on I told you that the B line is the new client So they're saying that used the language Never possible So that's fine I mean to learn how you I think the best ways this sea life they have added some more commands in the B line section actually because the old highest A life supporters getting lower every
3:59:14
so but for learning Corpus center it's all the same revenue used to see a lie or the line I show you how to use the B line or so
3:59:24
um so this is the high shelf What you open by on its very easy to get started with high All you can do is that you can start by creating and data issues they create
3:59:38
they Davies So for us the news that we create something called a database that is the highest level of abstraction So I say create database I don't know something
3:59:54
may minding so I just created our databases So the commanders create
4:00:03
database Onda later these names I'm just calling it as may minding
4:00:10
on If I'd wash shore data basis you can see a lot of babies are there People have creator I see Here these are or other databases that God created inside High on minus May 19
4:00:31
All right you could have created something else Like whatever I mean you remember and you have to say Use the Swiss there baby
4:00:41
You are to say use May 19 These are regular recommends If you're aware of secret there is a way to show that it will display the baby name set
4:00:55
high There is a property where if we can enable it to see the baby I'll search for it You can even print the column headers So now I'm believing you'll say Select star or something It won't show the column headers
4:01:13
The column name You have columns and the names like It won't show but you can make it show there is a command toe Bring that So first we'll make sure everybody's on the same page
4:01:33
Hi friend Current baby said hi You see a life brand current leading So this is how you can print your TV You simply say set high You see a life print current baby Quit true
4:02:17
genital display the baby Can you try this Everybody is able to logon to high even start I know anybody have a problem in by Tony Don't have Right No you have inundation Innovation is very complicated Actually You miss one in the nation nothing will work and fighter I know that a lot
4:02:38
very difficult people were coming from Did you have our Leonard your family with curly braces Here everything is you start a control structure that is an inundation but a terrible take your office I think you're using some medical The that have been to my cast or
4:02:53
only the definition like meta story will have it for is that's what When you normally start hype I said they used a nine day light that there d be It won't show me in which db I'm in So now it is displaying it here Right See this It was showing bracket What is your current The baby eso it will display with several database you are in If you change it will show right on Another important point is that probably towards a little bit worse Adam Inside there is a file called the
4:03:22
higher site Examine Hi Hyphen site dot Examine on that file has around 1000 properties on the whole hive is controlled by the file So these properties are actually from that fight Meaning Either you can do like this You can open a session and say set high blah blah blah crew But if you exit and real organ it will not bring the baby again Your type this or you can ask your admin change in high site example this property so that for everybody who log in the devi will be visible So that
4:03:57
file has lot off properties Actually
4:04:00
try to find that for you
4:04:03
Problem is that I was with Horton works for the past 10 15 days and finding it a bit difficult to read a Florida
4:04:13
but it will be here Gonna figure edition or do one thing Now it's easy what you can do all if you can open one more command prompt Web console
4:04:31
This copy if you go to see you're missing Hi Ive Garnes
4:04:54
Uh yes If you navigate to this location This is common for Cloud of Autumn works everybody If they see how you've corn folder
4:05:04
you will have a file called High Hyphens I don't examine this file Controls end to end off high all the properties right And if I open it you can see all the properties See how you meta story You are right This is the metastasize you are remember So where the connection is there then some other things are there or talking word join How do you searching the editor this one day Then I can fight Right Uh what did we set now How you see a line
4:05:56
No No How you dot It's a spatter Not found It should be there right time And I'm just wondering we're device us baby
4:06:18
Same pattern or home Baby should be there Just as for high
4:06:34
Probably I don't know she or this Um what you say properties are there in this file Probably some of them are So another thing is that you are accessing this file locally I haven't from this machine If I goto Cloudera manager configuration I'll say baby should be there somewhere I think Some of them are actually not visible to us But all this property separate from high site examine all these things will be in high site example actually So that is how you can print the current
4:07:22
debut Warren Well that's not our intention What is our intention So let me show you the data that we have So if you go to this hold it right you will have to files Now
4:07:33
once you start loading that out of life it is very complicated
4:07:39
Even though the queries are very simple people try to get it is like watching a Christopher Nolan movie No you won't understand what happened and what hours he finished the movie or they wouldn't understand what happened So the point is I will write first Then probably we will do it Okay So you will create a table that everybody knows this you will do in have your say create every blah blah blah It'll create a table Fine Then it's next up is that you want to lower the data into the table The
4:08:09
data can be Lord er from local file system or any office
4:08:17
It's a bit tricky so if you create a table and you want to put some data right The data can be loaded from her Dube All from your Lennox I system Right Okay And I will show you what will happen
4:08:32
How do both this on What is going to happen if you do both this What will happen to your data and all these things Like so we'll go step by step Otherwise people get confused a lot in the state It's actually very easy but end of the right people get confused a lot Where is my date of what happened to my data Sometimes you will get local also So what happens These Indian person or sometimes they dump it into some local machine You can lower You can say hi Take it from
4:08:57
my Leanness number Tinto So it is actually coping from limits So how do But there is a way to do it That's what I wanted to show So let's look at the data right We'll also some small analysis Can you open this Fine But I need a better later Actually What bad will do right That file is already available in your data set
4:09:35
Let me download this real quick
4:09:42
Can you see this fight transaction Doc Txt the X and this one It'll be here You will have a court and data folder inside that high inside that I think it's pretty Is a dot txt file right Because for me the extension is not visible Uh just give me one moment I'll just installed this north Purposeless So last class you're watching narcos or watch the answer coming late What happened in the morning So what we will do Oh is that First we will copy the file to the local file system That
4:10:58
is Lennox on Then we'll create a high paper You say lower the data to the hive table on We will see what is happening in that process then So there are cool fires Actually another file I'll cooperate with the office
4:11:11
from there I Lord and other people and show you what is happening It's OK We'll try without this I mean you have not bad anyway right So that's fine I just wanted to show you this Yes So this is the transaction data What we have here on if you look at this data right I just take one line and explain what it is so you can understand Like so if you take this line So the first uh Carla Mr Plants Action I d That's nothing but 18 Then you have the transaction date Then you have the customer
4:11:46
Rieti So 400 toe Double four forest custom Ready Okay on then you have the amount spent $88
4:11:55
Then you have the category off items he bought team sports On what Actually he bought Baseball City or Salt Lake ST off Yuta and credit So it's a typical of sports store transaction What category What item What amount Credit they you know on You have a number off lines like that No it's OK That's okay So this is one data that we have It's a transactional Later on we want to create a high table Lord This data So that's one thing Second data that you have there is a file called a
4:12:30
gust Can you see this on I want to open this with
4:12:36
uh word by Yeah So this is that That's Democrat Later So first colon will be customary First name last name age and profession
4:12:48
Christina Chung aged 55 is a pilot like that night So you have ah store customer data You have our transaction data right And what we want to figure out is that we want to figure out the total amount spent age Weiss by the customers for example I want to know in my store what is the total amount spend by people aged between 20 to 30 on 30 to 40 and 40 to 50 So that is the analysis we want to do So in one data set we have all the transaction like how much among their spending and
4:13:25
all second articles that we have the person's name and age and all So we'll do a join on Then we will argue a case statement and figure out that's what we're going to do So the sequel analysis is not the intention The intention is to show you what is happening behind the sea Onder How high vis dealing this right OK so now what I want you to do is first we will go on Lee the local so we will copy the data to the local fire system From there we will analyze it So I want you to
4:14:01
copy these tools to our local system So how do you do it You have to open the
4:14:07
FDP stuff right So I'll just say f b p Logan on Hopefully it might But for me
4:14:21
up your face So upload these two files Oneness Golda cussed See USB another one Escort the X minus one open Yeah it's like what's here And I need to open one more Okay Luckily for me it is available You have to figure out So use FTP and copy the file stoolie next door to her Do don't upload to her do I'll just wait if you need some time Yeah but uh you don't have the changes Extension Yeah So if it is a text file format it can access any text file on leaving your to mention that the
4:15:40
limited coma or whatever it is so high it can handle text CS VJ sohn Example Where I think parquet So there are multiple file formats it can handle So by before little read all the text But if you're having Jason and all that is so are you guys family with something coarser D Do you have you heard about 70 No Certainly stands for serialized de serializing Okay It's a very common practice So what will happen is that let's say you have a Jason file you know Where this Jason right He and
4:16:13
value huh The key value pair So if I have a Jason file I can ask Have to create a table from that But the problem is that Jason is semi structured data It's not proper coma separated or something On how I will say I don't know what you're talking about I can't find a scheme up So to do that we can use something called a J sincerity Yo Downloaded It's a jar file I don't know whether the latest high has it If not you order download a job file with the Scarlet Jason 30 Okay add it into
4:16:45
haIf on Then you say creator table using the 30 So using this 30 it'll parts that Jason on key will become The column value will become the data like that So surveys are usedto what do they say Read the same structure and structured it I will show you an example off on structure later where you can use something called rejects 30 regular expressions you can use to toe parse the data So high supports of a number Authorities actually read different types off later Jason I will
4:17:19
see if I have a different show So that is again An open source projects more most off them So internally nothing happens They will read the key on the value If it is a Jason 30 on if you do that structure like a table to hide So if high you looks at it high will nor CS key value pair I will say a call I'm heard Aranda value It was just apply the same thing So it is reading the sentence structure later on giving a structure so that it is a parson
4:17:46
So we have fire says right It's a fires that actually So this thing are you able to do are able top floor everybody
4:17:53
on and find this file You have a file core commands Can you see whether it is there
4:18:16
Just comments in the high folder You will have a Fire Corps commence No no no This is not uh cystic commands We have type or copy paste so we don't have uploaded So this Isn't that for me This data is in the folder Cordova Goo That is weather data Other sites Um let me close this Now we can actually create our table so let me just call people This is you can sort onto a create database because the first two steps we have order reading we created a database and then we said Use it Okay on the
4:19:03
first thing you need to do is this Command on Explain it So just copy this Creator table
4:19:10
Just go here and based it on our the semi colon At the end On his tender you have to add a semi colon Otherwise it cannot work on hit Enter Yeah So this is the typical create able Command in sequel It say's creator table andan The payable name is transaction records or whatever name you want you can give And within the brackets you are mentioning the schema off their data so high you're supports a simple and complex data types So those who are aware of data types so you have a simple
4:19:48
our data types like indigenes string and all It also supports complex later life like maps track send All right so they're all supported But here we are just starting with a very simple So I'm just saying even that is a date later Type on The important point here is Nordisk Amos Key My mentioning You even have a data type for date I'm not using it I'm saying string but how you support data handling and all So after that the most important point here is row format
4:20:19
The Limited feels terminated by coma What That means Hive is expecting the data in row by row under the limited risk oma Which means you can Lord only coma
4:20:32
separated files ideally All right I'll tell you what will happen if I do something else that time north using coma What will happen I'll show you But right now they're simply creating a table There is a a coordinator for high So that will Really No we don't have to define So now this table is created right So all of your creator hi can be accessed through Hugh You havoc You write that I show you that There you have auto complete mostly for the hard way If you're working with how there is
4:21:01
not a complete your to tight everything manually Now now is the important thing How do you lower the later Right on There are a lot of things I want to talk overloading the Laker First of all Ireland a fiver is your data right So in my case the data is in this location So just make sure you understand the location off the data
4:21:26
In my case this is the location I just copy this on the data is the Exodus Wonder Txt and I will say lower data local in part into day But so this is the commander
4:22:03
The part I found here I went toe this location I did appear the beauty See this is where I have the data right
4:22:14
It is only next It is only next if and it is very easy If you make any mistake in this a pop cannot be found in NorCal or if there is a correct but is unloading the later so easy to identify whether the command is correct or not In my Casey it says loading the later So that means it is working fine for me Are you also getting the same thing So make sure it's a local part Okay so people get confused if the local park
4:22:43
Yeah because here we have not done the analysis We just loaded the data on here You can buy sequel query we don't have I will be writing secret You don't have to write the java programming nor did or you don't have to know Java jar file It will create a job file Actually if you mention a folder
4:23:02
it will load all the files in the folder So just mentioned exact file He mentioned the four living So there are two files It lorded everything So here since it is a data warehouse it just happens to files But from very trinket that have been a problem right Even I don't know they should be the fight If ice in bites I think are awfully this for and be right So before and with a size off the final actually Now this this part this fine actually I mean you are just loading the
4:23:34
later on you If you want to check you can simply to work Select star from the X and Ricard's You can do a limit by Can you try this those select star from the table Do a limit fight Don't simply on a select star That is around 50,000 records citing So do we limit I want to see that Our Firoz Right So that's like a man this to make sure that data is loaded Now you can upload the data from local as the less How do so I'm just showing how to do this I'll show you from
4:24:12
her do person Both possible runoff selects starving selectively thing from the table I want to show say the full table I limit fireman's top five rows I want to see now If you actually want to see the query write you can write any query Doesn't matter But I can simply say select
4:24:32
going Start some So the Senate counts Star is a query and as you can see the moment I run the queried is firing a map produced job See So the query will fire a map produced job because it is converting it intensive map zero Radio zero
4:24:55
on This is this cluster is really good The what I'm stealing is that we have around 10 notes here in the cluster on only I think I don't 20 people are using it Very less data That's weight is very fast
4:25:08
Actually it is very very slow This is this is unexpected If hi vis fast there is some problem your to troubleshoot So this means that X stars the next time Ensure beautiful table limit five I want to see the top five Rose You're just very find the date eyes Lord No the top Firoz Because they test 50,000 rose If I Same Grillo selects startle display 50,000 I don't want I just want to see No no selects Will North the firearm A producer Because it's a simple read it doesn't need to do
4:25:39
any calculation So the newest aleck star it warned Firearm I produce job simply show you the airport
4:25:48
So just let me know Ari ableto till this your toe Do us that account Start it should FYROM apparatus job and you should see the output One more thing I want you to do in that fine I have given to you right This commands file It might be slightly wrong not wrong Don't copy place That's what I'm saying that I write it here because I use it for a different session also So there if I open this here it let's say lower data in part See the same lord that I in part it is actually Lord at
4:26:19
a local in part right So don't for the same thing and you can correct it Or so whatever you're running here just corrected there So that this reflector right otherwise tomorrow you try and it may not work the way you expect A bit of 60 my producer output I mean it is fighting a map Reduce job right So mappers logs are just being displayed That's what you can suppress it There is a way to suppress it But normally if you submit a job fighting produced if this place the same thing map
4:26:49
reduces That's where finally will see that the circles either the fourth nature actually in height Because even though it is a hive query it is not a haIf could it Some apparatus program that spite shows map on This is good for troubleshooting Sometimes I will show you are errors and warnings and or when it is running I show you how a neighbor called But right now or you're doing a select star right If you want calling names set Hi I'm not cli dot Print door header equals true Okay now if I
4:27:28
go this this is common start right Little card Definitely Because it is on my analysis it has to call for on leader selects Star will not call because it is just reading the paper So are you able to lower the date At least run the basic ready than that's okay He's asking Can you see the Java court High risk right No I am not Found it so far out for that I know that That is in journal toe haIf I mean how it iss converting that into a job Actually we have all this in memories for example
4:28:09
how to write a join in my produces there A select statement doesn't love normal select status Simply showing right doesn't convert matter Wear condition internally There should be some logic again Toe Everything is key value pair and Dr D I Everything is so another drawback is that high is good but you will not know what is happening behind the scenes like end off the day how the court s return You will not know You will see that result but lacks enough If you know sequel You
4:28:39
can just fire the query right Hado into haIf in her due to High van her group Because it isn't hurdle on It is visible in any way so that now the important point is I don't actually want to show you running the query attacks So once you now you have this table and you can write all the sequel query That is not my intention The real question is where is the data First question right So you lorded the data on duh from local You said that means family next So first thing
4:29:10
you have to understand is that data must be in her do So when you did a load data it couldn't goingto Hello How do that's for sure but we're in her do Right So to understand this what you need to do um you have a describe command So let's say describe and the extend records So this is a very common common If you don't describe on the table name it will show you the columns and later dives and all on if you want You can also say describe for matter on the table name So describe for matter
4:29:58
will give you more details about your table that you created on If I hit end it There is a very interesting thing that you can see The thing is it stays stable type ISS managed people So any table that you create in high either it can be a mannish table or something called external people by default everything is managed table I tell you what it's what you mean by managed table OK But you can clearly say see it say's it's a managed table Point number one point number two It displays something
4:30:35
called location Right It saves location
4:30:40
and it says you sir Hi Eve Warehouse So I want you to do one thing Just go to your hue file Broza can you go to this thing If you go toe this Hugh just goto this user folder There's affordable use of Can you go here
4:31:04
All right thing that a lot of users actually a number of yourselves are there Okay So do one thing you say slash Hi
4:31:18
So if you simply type it here You sir slash high You will land on this page So what will happen is that venue installed haIf There is something called warehouse for the hi I will ask You were created on normally in all Hadoop clusters It will be yusor haIf inside that there'll be a folder called Warehouse Okay on If you open this folder
4:31:43
you can see all the BBC's people have creator on You can open your baby My database waas uh May 19 So this is my database May 19 Open that on There is a folder called Be Eccentric Course This is nothing but your table You open this you have the data so try to find out your data by yourself So my point This what is going to happen whenever you're creating a DB in her job This four there is their use Serhiy warehouse inside that will create a folder with your deep meaning
4:32:20
When you create a table afforded with a burning on When you say lower the detritus simply copying the data here are you able to reach this point I mean this place you can also do something interesting I can simply go here on believe the data I related the later from here Right on If I do a select star now Sorry it fired the query because the table is that it looks as though because the data is in order Right Let it run Yeah so are if I do uh select start There is nothing So I did a
4:33:05
select star from the table on There is no data because I deleted it from here Right on You can manually copy the data here if you don't want to type lower data Local in part I can just copy this transaction
4:33:20
here This location Ah that is a scheme of what You don't have a data So now I manually uploaded the data here on If I go back here I should be able to see the people So this is very your data ghosts If you're coping if you're saying lower data local in part the data goes into this You Serhiy warehouse would be before the table for the doctors where it comes Okay So are you able to see this four day
4:33:52
I think Yes And now you are Now I am going toe confuse you Okay so if you are invested this much I'm going to confuse you So now what you did You loaded the data from local file system and I have one more data right So there is one more file called
4:34:10
customer right I want to upload this from her Do so what I'm going to do You can all sort of along with me I'll goto Hugh Okay On DA
4:34:22
in my home for that Raghu
4:34:26
Okay I'll just upload the data here Step number ones So where is the cast file Customer file Copy that Do do in your home for the very very one
4:34:43
Uh this is a bit confusing I mean so remember the location where you have copied that in her do fine Uploaded on Now I want to copy that into customer favor But first we have to create a customer table right You don't have a customer paper so I can go here Andi there is a command here Create table customer See Fine sales based on age group below that there is a common Just copy this Command Come back here on just save based it and it should create a customer table plane So
4:35:45
now you can Lord The data from Lamex you already know how to do that OK But I don't want to know that I want to Lord from how do how do I do it I'll say Lord Data in part I don't say local I say in part se uh Ragu slash into
4:36:13
table customer try This only difference is that you are not using local use a lower data in part on give the Hadoop location
4:36:31
Now I want you to do one thing Where did you upload or their dynasty office discussed file Can you check the data there It will not be there If you can see there is a problem it will move from there
4:36:44
Just the ones saying In my case it wasn't it wasn't Raghu If I refresh here it'll be gone But because the tables were creating a scold managed to people so openly later described it short It is a managed people What is a managed favor A managed table is our table where how you will manage your data You don't have any control So in Hadoop it has already selected a location where it will done board the file So even though you say Lord later from this hard off location will cut it from
4:37:19
there and dump it into that So you can go back to that user high warehouse folder there Your data will be there for sure You don't have any control as toe The other data will be high will control it right Whether you can see the data there Yeah in user high their house the sluggish on ever Is this a prediction Isis appointed That's a sister Prediction The metadata will get updated That's what is happening Because when I say move from this folder to that for lead rather than moving that name nor
4:37:57
will have data meta data It'll point from here to here That's all it is happening It is that I mean other other place we were gone where we have already uploaded the day travel people This will not happen If you're lording from local it will not believe from local Okay so that's a bit confusing but that is how it works I mean people think there is a data What happened Where was my file Right
4:38:24
Because it is managing the data so it will not I love you to keep the data It has to be in the use of high fare Ho stated So that's one thing and that's called a man is stable What we're creating is called a man is stable and now ideally if you have done everything correct like I am doing if you do a short tables you should seize two tables And uh you should always have their time Both their neighbors
4:38:53
anyway define do short tables on If you're just a next are you should have data in both the papers because then only you can go for that Any analysis were true bond rate once you have the data but here we can use equal You are not family with sequel I guess Sequel is the most powerful language in the world but analysis that is nothing which can be secret even though I don't know much else equal and not a sequel guy
4:39:30
So you might have bright complex queries But end of the day that is not the idea You should know what's happening Where is anybody again Like maybe you don't know how even I don't know how to write some queries I'll take hell or somebody will like the query But if you find a quickie what is happening Where it is running that is I'll tell you a very interesting story I have a friend of mine Hey is working in Cave Germany
4:39:52
So I went Oh praying in Cape Germany So he was in my training thing He is my friend He was also sitting in the class I knew that Okay So I was asking the participants how many off you're using Hadoop or high fever And most of them said no there are little guys The point is they were unimportant Worse important words has a tool called Hawk H A W Q Hawk hawkers like high only but with some mortification Hawk has become a little itty no to be fair Previously they used to call it this hawk Okay so
4:40:27
hawk is actually a connection with high only in day after day So these people were using Hawk for past two years but they didn't know that they were using her do Because for a sequel developer it doesn't really matter You like the sequel query on Do we really care where it is hitting for Who is processing your query You don't really care You're your ideas to run the query right Hawk asset is elope So Hawk is an improvement on haIf and now the project is discontinue
4:40:56
It has become that elope Really massive queries on Previously it was known as Hawk So I even like three years back that thing they were running Hope on day didn't know that it was so most of their queries were on her group But that's what I'm saying If you're a sequel developer mostly you don't really care where the queries running You care where the query is actually running or not There's a problem Others a performance issue That is what you're looking that's where something So I
4:41:25
was asking them to actually use Hurdle They said no We have never used her toe They were actually run enquiries for two years Almost Okay So once you have the data what we want to do is uh see I want to do it in a different way You can do it in many ways First thing I'm going to lose that I'm going to create a table so you can get the comments from here School's out one So I'm creating a table called out one If you look at the scheme off this table you will understand what I'm
4:41:57
doing It's a joint Babel meaning I wantedto adjoin on I want o store that itself in this night So usually you don't want to ideally push it into a table But you can do it also So I'm creating this empty table So store my join dresses while you're doing a joint operation This table will hold herself on How do you do a joint query It's very very simple Eso
4:42:26
if you know sequel I think
4:42:28
this will be very easy for you You will do uh insert off Right statement table on It's a typical in our joint right You will say a torch customer number first name age and profession that is from one table Be dot amount and product That's from another people from customer A joint transaction records be on the joining Carla Miss customer number So for those who do not understand this we are merging two tables so this will produce it a set but they want the story somewhere
4:43:04
right So we created an empty table called out one so their little push it So this is a typical syntax off a joint So that a means one from a here is customer table Be here is transaction the course table So from a table I want columns customer number first name age and profession B I want amount and predict And whenever you want to do a joint there should be a common core But then early joined makes sense So the common core Lem here is customer number This one is the common
4:43:35
core customer number and this is called a simple in or join You have outer joints left out right outer for louder Or this everything will work Now it is in certain over light Now the table has no data It was simply dump it If you already have some day title delete everything and lorded You also haven't upended
4:43:55
American simply happened Okay for sequel Guys I have a question Just I thought I'll begin with questions What do you mean by schema on right So a schema All right Do you know what this is If yes what it is
4:44:16
You're our Libya Miss and typical systems are course came our own right meaning If you create a table let's say the table Have six columns If you try to insert seven column great our it'll happen it will say a violator If you cannot insert seven columns to six column paper that is because it is Korsky my own right It will validate the schema while you are writing the later
4:44:41
Hi Eva schema or read Not write meaning I can create a hive table on upload an MP three file It'll properly upload No complaints I can upload a movie properly API ordered no complaints because it will not validate what you are uploading by You were saying Lord the data It'll simply copy that and dump it in the Fordham But when you query the data it'll validate their It'll show another saying that okay and why it is Because why highway scheme Our own greed is for faster
4:45:13
uploads because otherwise if it is violating everything it will take time to Lord the later today I think most of the data warehouses are ski mound read I don't know much about other things but hive is definitely ski mountain by so ski Mount right is typically on your traditional systems So around your joint quickie See what other joint happens It's faster So you But it'll be more frustrating This is like this If hiatus faster there is some problem Actually we heard troubleshoot e Really We do
4:46:13
it So now verify the data expels you lose selects start from
4:46:20
about one limit Five So you should see the joint output here And as you can see you are having these details from the customer data Like Cameron 59 on these details Fundy other than other people might like the amount and the project These things are from the transaction table right You should see the joint Brazil So now you have joined the paper joint O'Davis actually on Now what I will go logically speaking What I will do is that I will use something called a case statement
4:47:11
in sequence So those who are where it's easy for others What is the case statement Very simple I will say that Look at the age column Okay If ages between 20 and 30 categorize the customer into something 30 40 lose something so you because you want to group them right Then only you can some the transactions So in steak well you can easily do that using something called a case statement And that's what I'm going to do So but the store the output probably will create a table I'm just creating in
4:47:43
the immediate tables so create the stable called out to So the stable is called out to And if you look at the schema off this table there's a last corland core level This column does not exist before because that is the place where the case with for services right So that level column will have lower medium based on the age off the customer I'll push the data there Okay
4:48:20
on How do you write a case statement This is one way to do it So I will say that select everything and I will open a case here when age is less than 30 I'll mark the customer Lou ages 30 to 40 ft middle more than 50 old else Others So this is some of the classification will happen on that will act in the last column off the table
4:48:54
No by the four simple case 10 Citicorp Last problem Only if you're writing a simple case it will have the last Carla mounted where the conditions will be added part of a conditioner mentioning So here I'm just saying that only low medium high right 33 categories are mentioning So see this is the case statement right on high This for laughs equal I can't really help it I think there is no other way to write and highlight without your six people I cannot like anything in life
4:49:32
And I so don't worry if you don't invest in sequel but get an idea like this is what will happen if I do it Because high was extensively used
4:49:40
I mean most folk alert All I can say s so that means we are in the last part Now we have created this table called out to And if I were select star from our tooled limit five c Now if you look at the last column every customer is classified all the middle And now we will do something called a group by query We just group them based on lower middle some more the this thing and you will get the final This is to show you how high is running Okay no other Ah on to do that
4:50:38
I will create a table called out three on I want you to tell me the query Let's see how many of you can So this is the third table I created Okay Uh grew by by uh you know this What is that Column level Yeah So I mean I'm just asking So what we're doing here this were simply selecting the data grouping it and counting it on now in the final table If you do a select star you will see the final decision Select star From
4:51:50
how Three So now you can see low middle old on Tortola More spin So who is spending more Not low I think this is bigger right Border Older people are spending more actually I don't know Maybe you're right Uh based on body actually retail store right there Sports items Probably older people are actually sitting buying more these things now Quite interestingly If you don't want to open a shell and I pour this stuff you can go toe shoe There is a query editor and there is
4:52:38
haIf same thing Where you gonna go to this height Onda In here you can select your D V Yeah So it is that the four database go back Um my later basis May 7 deed for May 19 So I'm in mate on here You can do the same thing Ah select So now it is a war to complete another kind of things a little bit right Just simply you know if you just hit play it'll do that So this tool called a hue is very uh you know useful actually because um
4:53:49
uh that is one thing Apart from that let's say I take a quickie So we have a table right out to right senate
4:54:05
Start from out to Lex A limit 15 Something like this I And if I run this the query runs and it gives you options to visualize See you can visualize the data I mean right this is Hughes property Oh in my knowledge no There's a website called get Hugh dot com but they are available on or the cloud A by the four has huge If you buy clothes for you get you order books Doesn't have you go to separately Install if you want Horton books Has this somebody white this vertical somebody
4:54:49
is their heart mentor that has a file manager Same like this You can click and up the order file That's very good actually Humans third party In my knowledge
4:55:01
somebody's from Apache ordinances just using it even on Apache Hadoop Buchanan strolling by for So if you're having a plane Apache how do you want to administrate You can install embody about popularly on Horton books because nobody uses much of Apache How do you are Either Horton works or clouded map are has the worst You wait scored map are controlled Center worst mean I'll show you that your way of my part I had a map of that cluster I lost it Map by control system I
4:55:41
don't know Maybe they have improved
4:55:43
No they have not improved my parts You a score map are controlled system ma'am M C s okay Um yeah This is how it looks like See how the year I mean this is there like Cloudera manager You see this is there You wait Very dreadful foreign sent You can find anything in this I mean the worst you I ever created Probably I mean comparing it to the cloud a manager and all very better the you iPhones everything Will Napa's control system that you wait Uh Anyway I wanted to discuss one
4:56:27
more thing So so far we have done managed tables I want to talk about external table and that is again a bit confusing Okay we'll finish external tables and wind up Probably You have been through a tough session actually So just do one thing I want you to do one small thing
4:56:46
I wanted to create a table
4:56:49
Okay so I'm copying the command
4:56:53
But I'm here Andi I want to change it So just open one more north Back open Sorry
4:57:05
New Right So this commanders crew for creating the transactional table Regular common I just want to change it I will say create external table on either Say transaction records on the score Txt Okay on then I will say location I'll explain what I'm doing Okay First let me try this No control or c my data So I'll just copy this and see whether the command is working Then you can also try Yes it works So uh you want me to increase the phone Let me just increase phone amount double Is there
4:58:14
just only one thing So what is the difference here You are creating something called external people First thing you can know down is that you say create external table So there is a key word called external You're using going number one point number two After all the scheme and all I stay location in location I say use a GL faculty That's my home for that My data What is going to happen if you run this command It'll create a table that's for sure It will also create that folder for you My
4:58:46
data in her do I don't have that Ford it really create that folder for me Let me show you I'll tell you why it is required if I go to my home directory My home it actually Ah Where is it Yeah here it is My data Can you see So this table is this folder is created by my table creation Coming I didn't create it Okay Now what is the idea of an external table When you cleared an external table you have the control as to where is the data So right now there is a folder called My Data Right And all I
4:59:26
need to do is open this folder okay And copy my transaction files here Sorry Just copy this file here I'm just uploading the file on If I say Celik start from the X and that I courts underscore Txt fine I have the data so that the friends here is that what is the difference between man A stable and external table in man A stable You don't have any control as to where is the data You just say Lord the data on high will always keep it in you Serhiy Where House blah blah blah in external table You
5:00:16
say I want the location where I will keep my data So here we mentioned a folder on I can upload All my data here on that will appear in my table more than this So this is one example where we actually use external table Is that customer the sailor They will have some data Onda we will ask Where is your data They will sell my day Tyson s three Amazon is three You can simply say create an external table Boyne Glamour's on its three location or the data will appear in the table The date I will be
5:00:47
there I'm saying you can see that in your cable So any external location you can keep your data on you can access in the table That's what an externality
5:01:01
Dry this a few One Create an external table and a pure the data See whether you can find it Because all the times you may not have control on the data You cannot say that all the time I will copy the data to my place Sometimes you might want to create a table and point the data point You are able to afford them that it's possible from my local system I uploaded to that folder
5:01:28
So once you upload the later to this folder it will appear in the paper My data folder
5:01:36
The data has to be in this forded The only you can see in the table because you're saying the location is there See the business use cases different I'll show you I show you the business use case Okay so what is going to happen is that let's say you are having some babies So here I have in Iraq Oh okay Yet I have some my seek with etcetera Right And what you do is that so you have some data basis right on you Let's say run Scoop You know what This school right School piss your appeal to
5:02:25
you Say Hey scoop do one thing Connect with this Davies every day at 12 pm Well am
5:02:33
midnight so I can show you a job every day Midnight What scope will do School pill connected these two d BCE on It'll run the ideal job It'll pour their data from here and one school gets the data school has to send it somewhere So I have my Hadoop cluster here Right
5:02:53
That is my Haru plaster Right What will do school Build down this data into a folder So there is a folder called I don't Know Who School school Bilkey born pushing the daytime to this for the corner Who now I can create an external table OK
5:03:13
on point that location to the group Right So what will happen when I dress that Exxon automatically that they attacked That school is dumping here will appear in my paper Are you able to understand what I'm saying so typically in an ideal leg What is happening is that every day you will be running it'll jobs to bring data from multiple places on All this data will get done Lead to some folder Now somebody cannot go there and say lower data local in part Rather than doing
5:03:43
that you create an external table point to that folder So the moment of fire lands it is appearing in your table There isn't that is actually use case of external table A school can order scheduling Who's he has to do now if you are asking school can directly them but into hive tables Also it is possible school can take the data from Iraq or dump it into a haIf table That's possible But many jobs are there where you get data from my paper Sources say you are a flume job If you're
5:04:16
running flu imagine it may be collecting log files All the law Faisal number into some folder I can create a simple external table where I can use some certainly to read the Lord on I simply select star All the data will appear in my table Otherwise out to say lower data local and part every time to dump it into the stable Right So the actual yourself for external table is that you know you can simply point to any folder on all the files will appear in your table So how you doesn't
5:04:45
care what is inside You have to make sure that the scheme are matches everything And some people ask this What if I have ah to Indy column data in the folder and you have a 10 column hive table The 1st 10 problems welcome It'll get from there ideally because we just fix wherever it is possible hand Then it'll remove it So all the files will be upended in the table and Sean like fire files wherever you're recorded cannot match or cannot understand this and our values I don't know what this or
5:05:18
later ice mismatch Only five problems will come if they tried toe fit But there is no guarantee night So I really that isnot a use case because in a data warehouse you are not supposed to do If it actually you are not supposed to experiment innovate our house The data should be plead That is why you have this cleaning pre processing data factory operations And all right then Only you take the later toe data warehouse Finally Um another important point about external table is this So
5:05:52
if I do a short tables now I have this table cold Our transaction records and this is a managed table This is my orders Enter table I say drop
5:06:05
Drop able The X in records I dropped the manage stable Can you tell me what will happen to their data Will it retain the data or data will be going That would be going shut
5:06:22
No that everything Data Viselman Okay I'm just asking So I just dropped the man a stable You know what is a managed table I'm asking Table is going for sure The file I manually uploaded Right We will be there Yes No 50 50 We'll check use Ah that is it Hi NATO's
5:06:58
May 19 Do you see a uh Fordham court transaction records The data is gone So Manny stable has this drawback I won't call it as a drawback If so this is very common It happened once in a project that I was working and that is how I came to know I mean normally when you working on learned these things you learn by mistakes something somebody does So we had up a table Very huge stable which was shared among our project
5:07:30
Do three projects are sharing the stable on they had for rights
5:07:35
that pointed by one of the engineer He said Drop the paper
5:07:39
on Dennis Gone So the product So he's had brought the table accidentally He didn't intentionally rude but the problem is that table is gone But they had some recovery mechanism at that point in time In her do in high there is no way to recover Liberec LTD said the live from her do so they had some recovery trash mechanism recycle mechanism Onda Based on that they recovered somehow Okay
5:08:05
That is why when you create a table which you want to share always create an external table drop table I can just use the A paddle right where the table name transaction records on the score E X t Sam dropping the external table table is gone
5:08:26
on I'll goto you sir I can just go to home for the right I'll go here on We had a folder called what is afforded my later the data means because high does not have any control on your external Later you have control You uploaded it will not believe it So this is easy You can recreate the table data will not be lost Right So if you want to share the data share exposed the table always create external tables minus a with very reaction for learning purposely six easy to do manage
5:09:03
table on the operations side they're all the same The queries and everything is same managed with There is no difference But always remember create external tables if you want o share this Okay this be pretty has smaller data I request you to go through that or so Yeah the official website off haIf very very important How you daughter pasture dot org And it has done self information again Like I don't know how much it has Lord of things Actually you can start getting started guide on
5:09:43
high to understand what is height
5:09:51
Yeah I forgot how to connect using beeline Um so I'll show you tomorrow How could run beeline Because I just turned to see by the far higher server Two runs in port number 10,000 serving a starting beeline Your to say Jay DBC connect to 10,004 number and then we connect with the house ever to I will show you how to do that tomorrow Anyway I forgot So please go through this high ive official website It has done center himself information
5:10:46
Um a lot of things have it There is a higher wiki page It's we just so tonight
5:10:55
on And just look at these points we already discussed Ah query execution Wear a patch It stays Apache spark or map Pretty's or probably last bit I can show you face but I forgot my you certainly password It is not here I show you tomorrow taste anyway Query Execution by a days my produce or spot Hi Can you spark as an executive engine
5:11:31
But people don't prefer it because from sparked their fire queries the other way That's more easy You can start spots equal and say fired But how You can actually use park as an execution engine It is possible And this is sub second query retrieval via hive L A P only Horton knocks lately This is what I was talking about elope Elope enables assay transactions faster queries everything that you can think about a normal part of you must within few but that again request resources I mean
5:12:05
even though it stays very fast you will take a lot of resources actually So when you're running taste they is itself is in memory So anything that is in memory requires resources in memory means ram You lose only rhyme So they is itself It's fast because of in memory So you your plaster need to have a lot of time for an Olympian or otherwise it will be very slow
5:12:28
procedure So back this X pl SQL is nothing That is a combination that's called hive query language It's you're lonely You can write your own custom are functions and all inside Hi how you support something or UDF user defined function You can bite So that is They say it doesn't feel sq it as a fancy way of saying that I would feel vice versa in the sentence Yeah see if you're good and Java map produces your choice but map reduces extent extinct right Then you want to spar Spot uses java but
5:13:06
that is Java eight functional java nor Job a seven So a lot of people have trouble there
5:13:13
But even in spar community the most frustrate prefer despite on by a learned scholar then only Java come so that I supported in spot 100% is but developers every list People want neither fighter No scallop because it's easy to write Actually and I I would say some knowledge in sequences or high vis very important Whatever you're doing night I will I will tell you this in Spark in Spark If you write your cord Using fight on and sequel of the sequel is much more optimized because
5:13:50
if I'm writing anything using sequel I have something course came up
5:13:55
right which means I can understand my data What is schema If I have scheme I can invest in my data So if I write up join and filter I can push the filter first and go Join are able to understand
5:14:12
in a school I write a query by the fall it does right by the four Dinescu SQL If you write a grew by join then filter what'll happen feel terrible Come first because why you should do Lord all the data Then finally do a filter first to filter than do it that small optimization that is not possible If you write a language inspired will receive and we like spark chord It is not optimized because spark isno having a very types came up Response sequel has That's what I'm saying Sequel or this
5:14:44
has a preference because off its optimization that is not possible in other languages Like even in apparatus Who right You have no idea How is the date on the line right You're not mentioning any scheme up It has to lower the full later to do analysis
5:14:59
a name apparatus program You right The whole filing the bill ordered You cannot say that selectively This parliament analysts not possible I first told everything has to be loaded So are always the structure languages are having a preference on This job is good I'm not saying jobs back but uh I don't consider it in the further scallop Probably you can relate with Scala Scala is that is similar to java If you know Java Scala you convey these Leader Scott Spark is a good combination Very good
5:15:29
combination What else You have questions
5:15:35
will wind up in five minutes
5:15:38
after your questions So tomorrow we'll be covering something called partitioning But getting indexing and high which are a bit advanced optics actually about very much needed actually understand what is high because they are the highlights off Somebody want to learn how these are the points or if somebody ask you about how high these are what they ask Nobody But nobody will ask How do you do a select star career Hate that Nobody's going to ask They will ask How do you partition your later So at
5:16:09
least if you can understand the logic off it that maybe more than sufficient right And we will look at this NASA our data You have a NASA data See NASA So tomorrow we will analyze Now Star Data You're saying high So I can so show you this Saturday They reject censor the center like him explaining this NASA data So make sure this state assets out there with you Anything else that you know If you look at this case study that we have prepared So this is about the NASA Our data analyzing
5:16:46
the NASA Web server logs with Apache hive And this will be helpful because we will see the same daytime spot in the spark class Or so this data will be the same data So if you understand the data now we can unless it you think spot as well So I guess you are aware off this server logs contain lots off information from Web servers by on this case really will show you how to derive insights from Web server logs So you are going to look at some log files which are generated by Web server on if you
5:17:24
have ever work with Web servers
5:17:28
So let's say your type google dot com on then download something All this request will be hitting the lips over She will get get request put requests than extra dp status code So that is how you are Log files will look like OK I will show you the lock Fine on You can actually download the data for free from this You are We have already downloaded and uploaded the data You don't have to worry but you can It's public data from NASA on defy Scroll down
5:18:00
Look at here before moving to the activity Please go through the extra TP response courts at this link Right So are you aware off something called a street TP response courts Yeah If not you can just go through this You are So for every request there'll be Accord associative Right So if you're browsing and hitting a reps over the server will respond with that court So if it is you 00 it is okay Anything starting it to his sexist You will not want its creator to one or two is accepted Do
5:18:38
nor for is no contend And all 303 direction That website will redirect you to some other place Right on flying There s our 400 on fire hundreds Cervera So sometimes you try to open a website and you will say 501 said we're not found or like that So anything starting with five layers uh in doneness of ah era Right So Gateway time out service unavailable Something something like that You can see So if you want again just to go through this list Right Andi understanding the
5:19:19
data So the data said contain the woman's What off all http request to the NASA Kennedy Space Center's over in Florida So this data is collected from the nozzle tips over on two months Worth all the requests are there And I think that on one million lines are there in the data It's pretty huge data actually Somewhere around one million lines the log files are stored in the Apache common Look for my So that's what a lot of people asked me Uh values that text file It is not better dot Txt
5:19:55
extension Normally you say next fire That is something with dot txt But there are a lot of other later which is called text file It is always not mandatory that you will have a dot txt extension Right So this Apache common look format it is a text file But you cannot open it and not back my You can open it inboard powered or any other thing and it'll show you properly like text a down I'll show you the data Andi
5:20:25
Now let's look at the later So if I go here that IHS uh NASA access lobs and this is the data I say open withs seyward bad Say OK
5:20:43
Eso This is how their data looks like and it's not really pretty I mean in the sense like it's not really nice It's not structured on What are the fields we are interested in In this way So this is a log file on if you look at here the pdf right We have the host iron ditty use our identity time request at this size if I scroll down the first failed is the host making the request Okay so where is the host making the request This is the host this day I am 24 door I net something
5:21:25
So this is the fully qualified their domain name or the Hosni course making the request then what we have next to our the use of identity from the remorse and local mission These are unavailable
5:21:40
So next to fields are not used their dashes That's fine on Then we have the time stamp in They bond our so year This format So here you can see this So this is your time stamped by the time it is making that request time zone So this is this minus 400 is your times on What you see here is that time zone Okay on then There is a request in to the server So here we can see Get It's a get request Okay And somebody is trying to get shuttle mission status new some news something Okay
5:22:18
so that's a get request we have on Then there is an extra vehicular play court Ah assertively play cord is 200 It's a success Okay on then You have this size off the file received So probably in bytes 1839 So this is the data Yeah So you can see that a lot of people are sending request toe the NASA observer And that's how the later looks like now what we want We want to create a hive table to Lord this data on Then we want to write some queries on the science off this vilest 167 megabytes It's
5:22:56
a big file actually Not very small on We will test now how the cluster we behave because they were few fear Start writing the query all of us right The query We'll see how efficient the cluster is anyway Right And so the next question is that how do we actually lowered this daytime toe hype because this data does not have any structure to do that We are using this 30 or the serialized the serialized Now a little bit of information to you I also want you to give you an assignment So
5:23:32
if you simply go toe high on say sir be no you're going to see here This is called necessarily So what is a certainty Certainty is a short for serialized Be serializing Like I said if you're having data which is not really structured like this log file or or J son files or anything you want to read and write you can use this parcel called 30 by and high supports a lot of built in thirties So these are the built in thirties we have on we are going to use This guy rejects You are aware off rejects
5:24:14
but I got that expression So that means you came back that I will of expression high will parses parse it on using this little redid There's an O R C 30 Averil These things will look at later Onda then custom thirties You can write your own thirties if you want That's also possible that I um we will look at this later Let's open this Reject 30 Yeah So it stays if you want toe create a table and use the 30 euros a row for max 30
5:24:52
Remember Previously we used to say raw format de limiter feels Now I say Roe formats 30 on Then I will say this is the 30 Okay on Then you save it Certainly Properties on you will say input Regular expression Whatever regular expression you have you have to put their on That should actually be ableto get the later whatever rejects your writing And then it should give the structure for the data This is the idea right And this is very important The word aspects to fire I
5:25:26
mean I think yesterday didn't discuss it When you create a hive table by before this option is that course stored as a text file which means whatever data you are loading into the hive table it will store it as a text file regular text file be to see a story file or any file So yesterday well ordered some customer data transaction data on They were getting loaded as it is Okay Now you have other options you can say stored as what's the parquet And that is where the improvement comes and I
5:25:59
will talk about it What is the difference in when you say text file or when you say Oh RC file where is the difference But as off Now we'll say this story as text files were not border it is biofire text file huh So that is why yesterday they were not mentioning it It is vital for text file now the date as it is available in this location Can you see here Forward slash gl data on Duh huh One moment Uh what was the name Access right Yeah So this is the file It's called access There is
5:26:41
no extension on it is available in a common folder called GL Data So first let's do one thing Let's start haIf and you really need to start high So start height on Let's create a table So we will use the DB yesterday we created I say use May 19 That was the TV right So you can switch to whichever DVU creator on this is the command to create the table You can copy paste this from the pdf Okay Say copy
5:27:19
and I'll explain what it is
5:27:23
So you will say create a table if not exist NASA log on the fields were interested Our host um identity use their identity
5:27:36
dying request status size and the new Cerro formats 30 We want to use the rejects 30 With 30 properties you can import the regular expression So this is a regular expression we're using on in the output They will just say that So that's part off Saturday You will say that story with equal space is right Andan you say stored aspects child So this is very a saying that story as next five So this is your little expression huh Uh you can go to the pdf And if you cannot copy like this
5:28:13
Right click say select tool There's a selectorial Once you selected you can left click and copy like this So you have to have a some idea about regular expressions if you want to write it from scratch But that's OK Yeah sure Um I show you from console Ah yeah This is a coming I just said haIf Then use the database on then just copy paste or the table creation Come in Right now we have not Lord or the data you don't have The Lord the data that output format string It is
5:28:58
simply saying that there are eight columns light and with space each parliament space It's a bit difficult to explain it Actually Reeb's Onda separates or their data with how many spaces you have and it is looking at digit sand strings on then extracting the interesting feels so difficult Explain that rejects actually
5:29:25
So if you're writing I just from scratch you have to validate it So this was validated Even I don't remember How about it has rotted long back actually Yeah So it's a create table Yeah normal They will Only difference here is that in the table properties You're saying that whatever data that comes to the stable apply the 30 the regular expression Otherwise it cannot recognize the daytime side Right So right now this is the Apache Common Law format right Onda Apache Common look format has a
5:29:59
fixer The scheme are not schema the way the law applies a generator so you can fight it in multiple ways This is one way off writing it so in highway few Look they will simply say that this is a rejects we want here I'll show you
5:30:15
uh so here to say in port Project C the output index is not stored So here you can see the import rejects Are you looking here I mean this was written like if you avoid the output spring also it will work But we are just ensuring that everything looks string I mean just we are manually saying that it is spring that I think it is not updated Highest documentation is not updated No it is not updated Let me see Um
5:30:51
So where is it more about rejects 30 can be found here So they have given to Ling's Actually probably It is Here let me just check So what happens This me Just see if I can find it first se in open source communities What happens is that let's say you are looking at high your spark or anything Somebody will create the perfect so excited created haIf and I will give it everybody Everybody will start working on then Well what will happen Somebody will say that something is broken so I may not
5:31:34
add it In my documentation I'll added in Nigeria July Zuhri shoot lacking right It's a derailing in derailing it clearly say is that there is import rejects an outward form extreme you should use So somebody added Ah yes I will work on adding a 30 how to in some examples in your directory So somebody said that this is not properly for matter So in the jail right is that it Actually it is not an official documentation or is it only when they created Probably property is
5:32:02
not there I'm saying then some particular complain but they have mentioned the link here Right You can see that it is here If you look at here it stays more about can be found here I don't know why they're not updating the documentation but it is there any way So one problem is this If you want to search documentation you may not find everything in the official documentation sometimes So I always look at Jenna because any issues they fix will be injera on Also not everything
5:32:33
Not every time things will not work the way you are expecting Shorter Booger saying that how you plus Jenna not able to read this file Some Some issue You're coming across and there'll be a deer a ticket for that or raise So once we now you have a lower the data So let's continue with that on Where is the location The location is here GL data Right So how do you Lord You say Lord
5:33:02
Wait wait wait wait wait wait
5:33:06
If I do a lower the data in part what will happen
5:33:11
I said lower their time part What is going to happen We can create an external table right That's better I didn't think about that because normally the participants will have the data in other classes and they will upload themselves So where do you want You want to create an external table everybody and point to the later that you do Okay so then you have to modify this So how do you modify this If you weren't an external table he was a create external people Next drop
5:33:48
the original table OK door drop
5:33:50
First door drop Brock table What is the name now Sand Don't So drop it And now we say create external table now Salo these are are fine on The only difference is location slash ngl data Um wait wait wait wait
5:34:19
External table If you're creating it should point to afford it This will load everything from GL data
5:34:28
We don't want that right in GL data You have lot of foreigners like so I will move it I'll see if I can create a folder in jail later Let's see New directory accorded us NASA Yeah and I'll move it inside here Right That should make sense See you have to take care of everything Not this right
5:34:56
This one Right Access action moved to GL Data NASA Right There is no NASA for that Yeah there is Move Fine So I really know if I checked NASA it should be here Now It's your book Otherwise that the Lord a very file from the deal Later for that we're in mourning that right So what should be a location This should be the location Right on Let me see if it works No you can use more letter or capital little
5:35:48
so the table creation will work I know that My question is
5:35:53
toward about the query Select star from works So you have to try this on Duh Let me know it Books But all of us are accessing the same date Okay All of us are in the same data so make sure you can see see properly structured
5:36:24
No I already created a table first Right on I dropped it
5:36:29
My guys Ah how you lord data to the papers We have a shared location So it is better to create an external table Two point So I dropped it and we created the table So here it is Location on Also here is ah location this or social this external you can type small letter or capital folder Name should be case sensitive on do a select star and let me know whether you're able to see the later So there are two approaches not every later can be made structure Also there is one
5:37:06
problem so high Eavis a structure the tool That means somehow you should give structure than little work
5:37:13
There are some type of data like for example there is a former corn avenue a V r o So previously when you get average data But this that's like unstructured You are not able to do anything but now there is an absolutely that they have creator So there are still some type of later where you can or give a structure one point second Boyne How this works in organizations is that like I say there'll be a data ingestion P right So I'm just giving you the example off G in G What
5:37:46
happens C g Ys business is so enormous that nobody has any clue what is happening there He's one of the biggest firms in the world actually write on their their size off That is enormous what they're getting So they have financial applications Okay these financial applications will generate data Okay on how do you get the data They will have a P ice They will pour their daytime Jason for my So it comes in Jason for my on sensor data Dig it sensor data So they have
5:38:21
this locomotive engines train engines in Europe GS created So these train engines whenever they're running will produce sensor data they collected on that data will come through Casca It will come in a Havre format
5:38:36
to look after toe Hadoop ultimately land on her do but their data sensor data is it like events Sensor data becoming events format Like whenever there is an event little generate But there is text file only But the format that they get this called Afro So the point is that data when it lands on her Duke nobody can give any structure to the data nobody on Earth can give So G Hassa data ingestion team Their job is to take the data give some structure and give to the big later team You're
5:39:07
getting my point by because if you directly give it a big day their name they have no clue what over the data So this will be a staging area where all this data lines okay on They had returned some Java code Custom job court What That Java Corba little read avenue Remove Someone wondered feels give some structure send it to next side So So that is how they're injecting the rate I'm saying so Raw data Most of the times you may not be able to directly handle it And second use cases
5:39:38
if you are a big data are developer or like us We don't want daytime the original format But if you're a machine learning guy you want the date and original format Are you getting the difference Like your family data scientist
5:39:53
I don't want anybody toe touch my data I want the original data with all the nonsense that only I can build my machine learning models because they want all the features They don't want to mortify the data But if I am a big data a day like this I am happy with structure I can lose some data I just need the structure for my type of work So these two teams will be there So the machine learning painful the original later they'll be collecting And then they will have their own algorithms
5:40:19
to current state tough guys used to because their data is coming from Europe Streaming data They don't stream life They stream once in a day or something I mean they collect all the sensor data
5:40:33
It's an Amazon The entire set up is an Amazon So there is some sensor which in Amazon you have ah solution right Who is working on Amazon sensor data You have a solution Amazon record
5:40:48
Kindnesses Kindness is right Kindnesses your sensor platform in Amazon so kindnesses will collect all this data and from kindnesses and push Look after
5:40:58
it was somber Castro store right The data there are very less real time decisions they want to do on that data because they restrained data on only time when they're analyzing their data is if the train is not working like it crashed Train engine court crashed Otherwise they don't want to analyze it in real time About 99% of the time the trains are running so this is a place useless for them So when the train engine got crashed or something happened that time they want to
5:41:24
look at the data Last 24 of it What caused the crash So they're not doing because size off the data is in a better place So analyzing this Orson or possibility for them So they electrically all time on batch it and send it once energy or something on it will go through Amazon Italy Landon Kafka Why CAFTA Because from Cathcart to three teams will collect it One team is using the later for something else
5:41:51
distributed on Then one copy will be stored in her do like original data Copy Anybody want they can get it That is how their architecture is right now These is an evolving and our protection They're just moving to Harut now It's not a stable architecture so probably after a couple of years they'll find another way to better this things And or right now they're not doing real time on the this train data They also have flight to data flight This aircraft engines are orgy right Almost
5:42:25
all aircraft all RG 80% days on all these engines and bait Same sensor data they collected But again that's all Not real time It's very rare You get a problem on that family they want to analyze So I hope you are able to reach here right at this point So this is a place where your data is that right If you go to this location I have copy the data there If you look at the pdf what it say's Lord the data selects star Now these are some of the U S Cases I wrote I mean you
5:43:04
can also write your own use cases so find the top end points that received service identity meaning
5:43:14
which are the how many endpoints have received service side edit right I'm just asking like that is 501500 this out ourselves I did it and how many times it has occurred So how do you figure it out There are multiple ways to do it but one of the ways this you can see select status count both requests from the what you say NASA log table grew by Status on you can say having status the register regular expression extract So here I am saying that anything starting with 50 in the status column
5:43:55
because fires Roy's so was either order by status dissenting limit fight That's the equity So I just want to know or the errors starting with 50 on Come them How many are there That's what I'm writing now when you are using having it has to be w operator assignment In having close in high when you're using having close has to be assignment something if you say equal to something equal to something it will become
5:44:27
a photo one It'll become a become once So here I am just saying just comparing that comparison operators really quick Uh no In having only emceeing their clothes and all It'll be normal in having you will say so somewhere There was a documentation on this I'll share it with you Okay Can you try this query and see Now it is taking time 24 seconds Only for the Western Anyway this will take some time Anyways Try the query It will be slow So of a new mentioned the commands It has to be
5:45:10
either for small lettuce or for capital letters He said Capital L location And it's a bug I know this but because other researcher throwing Arab it won't throw in a red on This is a very common thing If you create external tables you mentioned location Even if that location does not exist it may not throw it up and you will try quitting Nothing will come up You will think What is the problem Space is fine but I really this location keyword Sometimes it has a problem So that is what I did
5:45:42
It started working but you're getting Or maybe maybe that itself is not a problem I dropped it and recreated Probably started working Not sure So how much time it is taking for you to run the query 75 seconds Ah that is also one more point Um we say for good Yeah So anyway so somebody was asking about this The reference on regular expression Here you have Ah hi Language man It isn't a pdf on which resource is requested most frequently by the host This is over question Right Because which page is
5:46:47
requested the most on I don't think it's a very difficult query but still will run it just to see So now if you look at here it say's search request go McCown as request come from NASA log grew by request
5:47:11
order by descending limit 30 you are looking at the top 13 requests Actually my you can skip Heather and or but I don't think you can skip the data But I don't think you can skip columns No I don't think you can filter Biggest Ultimately it's on her Do So the data remains on How do Right That's one problem Because if it is initially office uh then there is no way you can feel that Anyway locally I have to see whether he is asking why loading the data Can I filter the data like I don't warned Or
5:47:59
the 10 columns I want only
5:48:01
you You have skip Heather That option is that table properties By the way there is something for table properties Um you just go to Google Say hi Create enable
5:48:16
bvl properties more so in creation off the table there is something called table properties What is it
5:48:39
Uh so table properties Right So here you can mention the you know Skip header and or let me just check no auto compaction mapper memory or a bunch external True You have skip Heather is there that's for sure But I don't think you can skip other columns That is not possible I think Anyway I will just see if I can get back to you on that OK And there are some more queries I have little You can try that Like display the top end host who made maximum requests and all You can try that Find the total
5:49:28
count off different response courts returned by the survey So just try these queries let me know if they're not working I have not actually tried everything It should work Okay So now are talking about Impala a little bit right Probably not in debt But you should know a little bit off Impala because it is heavily Houston industry Impala If it isn't a cloud outside we are included aside If it is hard and work side everything is haIf But in the cloud and outside we have something
5:50:14
called Impala So a small intro Not much but I never show you an Impala query also So what is going to happen if you're having their town or slight So their data there's another one
5:50:32
And this is you can install something called Impala right on If you're using a cloud in our distribution Impala will be by the 14 story you don't have to worry on Impala is its own sequel engine It does not depend on high or anything My So
5:50:53
when you install Impala what will happen There is a demon called in Pilot be in pilot the scoring pilot them and it will start running on every day like something like what I did I mean I'm just saying so this impala they will be running on or the notes festival right on Impala has a shell from where you can fire the query It does not depend on type on If you fire a query what is going to happen is that let's say it hits this machine so the query will be accepted by one off the machine So
5:51:31
let's say this guy right on this guy we'll look at the query now Impala can talked Ohio Smith our store which means whatever tables you are creating in Haifa in public and see so all the tables you create you can quickly using Impala So on whatever tables you create an Impala hive can or soc because they share the same medical store So that is one on one days so on on top off that Another thing is that when you write a hive query it will convert that into my produce So high
5:52:03
this nor bordered about how the query is running because it is the responsibility off map Reduced to running the query Impala will nor convert a map Produce Impala has this Impala DeMann which will run the query for you It is like rtb Emma's they can say not an RTV a mess I'm saying like you feel firing a Oracle query who will run the corridor Akerlund agree Saying like that if I fire the query this Impala bill get the quickie on it Is the responsibility off this
5:52:34
in Pilot Demonstrate Executive Equity So David half the highest Madonna data
5:52:45
Okay they will also have the block Meta data Meaning if you create an Impala table like a hive table and you say Lord the later right now the data is actually in Hadoop on the day Tyson blocks and replicas Now who has the meta data about this name No These guys will copy that to themselves because they should know that is the data in pilot is not going to depend on anything Toe fire The core it'll turn excess Right So that is why the queries are very fast on When the query
5:53:18
hits Impala it runs in memory
5:53:21
So there is a drawback So let's say the data of us here have a block here You have a block here What would happen You'll have a lamb this block that I will come here
5:53:33
on this block Data will come here them On this day I will simply coordinate the curry because the date eyes here and here Right so this guy will coordinate the query on it will run the query in memory The drawback ISS If let's say this mission crashes then back where he will be aborted It is not far are living but haIf good is our fourth Arland because I'm apparatuses for tolerant There is no they Map produced can fade right but in pilot queries are not for hollering So if the one off
5:54:10
the machine because it is a distributor quitting not one machine is running the query Probably 10 missions are running the quickie and if any one of the machine crashes
5:54:19
the query will be aborted But it will not convert a map produce or do anything So your queries a superfast actually that it Clearly it has refers We relate I'm give you the outward whatever you want so it's faster But this reliability issue is there with Impala So what we do if you are having a cloud lackluster
5:54:40
If you're having you deal jobs
5:54:42
we never using pilot because in India jobs behind the scenes it will be running hives equity and probably the query will take three hours to run for our student probably five overs That's fine Even if building that ideal job of machine crashes I am I'm safe because it's ah haIf Kareem I'm apparatus will take care of it But if I run that deal we're using Impala Let's say the query will be faster Who hours probably will take But after one never If one of the machine
5:55:12
crashes my Impala has to restart the whole quitting metro area But again your retail job has to start from beginning You lost one hour right So actually needs highest McMaster for the metadata Impala must use Highsmith Astro So how you should be in start So that is another advantage High will remain because lot off tours requires high even sparks equal sparks equal normally When you configure spark you will tell sparks equal Use Highsmith faster
5:55:47
so that highest metastasize used by everybody So are they able to create All will be in one place might And if the table is there you can either quit using high If you're using Impala that's up to you to decide
5:56:00
But in part also has some drawbacks This now sir table we created You can't quit using impala
5:56:05
It is This rejects is not supported So this table is created using the rejects 30 That is only for high I can see the stable and impala but I won't be ableto very I think if my memory serves me well but most of the regular tables you can query without any problem
5:56:24
Both of them can access at the same time because my past or is a shared place What is my past Or is this your my secret So share place Anybody can access not only high vending by the 100 so far that people exes it need not run You have to install Haven't set a Permata store I'm saying whatever table you create an impala it stores in Highsmith Astro So each data Nord will have an empire argument on each demon will have a copy off Highsmith Store block date up so and what
5:57:00
happens if you fire a query
5:57:02
Any demon can pick it up That's called a coordinated meaning If I write in pilot query broke that are hundreds off data Nords So any data nor running Impala can get the quickie and that Nordisk order coordinator for that query So in this example this is the coordinator because this guy accepted the quickie But this does not have the data That's fine Okay And then once it gets the query does a local look up It's very fast It has the matter it understand Immediately Here is
5:57:30
the data Here is the data It just splits the Korean streams to here and here These guys run the query collect The result should display for you So I really a lot off machines will be there where you're having about the matter Data look up is very very fast There is no frequency or to manually refresh it
5:57:52
It will not automatically copy meaning If I create a hive table it will not appear in Impala I could refresh the metadata There's an option in Impala where you can serve it fresh So if you feel that there is a new table you to refresh only the make America will come Ah block metta later So that will come once your table majorities creator say I'm creating a table Once I created table I lowered the data Then only the meta data off that data will come It will not store the whole Hodoud
5:58:21
Moderator you are how do Cluster has Let's say 1000 that are right Fight Okay Impala the North's or 1000 that are right Metadata
5:58:32
That is useless right You're saying this Hadoop cluster has let's say 1000 files Okay In which you created an Impala table Okay You say in pilot table you loaded file one It will have the matter it off Only this fight
5:58:53
Why should it need all of the meta data the career able hit only this file right Only later to the table I'm saying when you create a table you will say lower the later What data So it never examined reloaded the NASA data So the block information off that now itself a little Remember from hive Or if you're radically lording into it it will remember itself It won't remember the block information off other files
5:59:27
in how to be a whole lot of fires right Not all the fights are in paper Are you getting my point Where him saying it'll remember only Dimitar data off the files associated with the table Otherwise why You should remember or the files Manuel Refreshment is required I mean block It may remember I'm saying the cable properties it will not automatically refresh sorta manually refresh every time So usually we won't do like that Usually even though Impala and High can
5:59:57
communicate we don't do it because high you has its functionality Impala has its functionality meaning whatever tables you have in high If normally they will be used for reliability like ideal job And on that I will hand it impanel you'll you'll create separate tables You will say the same thing Lower data local in part blah blah blah and that will be handled by Impala only Like separate tables They can talk to each other But what I'm saying if you want up to date information you should
6:00:26
say Refresh that only you can get it on There are 23 types off refreshed I don't remember completely There's a complete refresh which will redo and Lord the court element our data from highest caste or lost everything There's a pass Children fish You can go with an issue so these things are available in huge I'm not just saying that you will be wondering So if you go to query editors you can see in pilot right So here is Impala so probably the table you created right
6:00:56
now Shortcoming Impala That's a logic right So if you go back
6:01:01
where is the May 17 May 19 This is my favorite My baby Right See customer now Silo This had an Impala on If I do are select start from NASA Look I don't think you can see it I doubt fire you I doubt it will be able to see it Maybe I don't remember exactly Uh no Failed The Lord Meta data failed Lord Moderator in various stories Description in Pilot does not support this table type Recent 30 library Not supportive
6:01:48
It won't support dexterity That's why you are not able to see that No it won't That's rising in Vilar tables If Empire lies squarely in your data it's not for so what the hell will do it quickly and get the waiter who will Quickly That's the question I find this clearing is far column So here you can see the NASA Hello thing It's not supported But if I look at any other table so that's a select comb Start from Let's say out one right If I do this I really should work
6:02:33
for some reason in by allies of it slowing this highways faster In this case it is not really like is actually very fast
6:02:41
Good There is no operatives with direct wickets and uses the same court If you run you will see Map ridge of starting map produced nothing So we've ill on There is one more thing this refresh manly has to do Normally you do it manually You can write the bats job refresh So I don't know whether the refreshes actually configured here or no for the Impala because this NASA low table is available We just created it right now It is available in Impala so probably and this is the option to refresh Can
6:03:15
you see this Refresh three options will come clear Cash perform incremental metadata update This will sink missing papers in haIf which means if you have a new table in haIf that is not critical to it and then there is invalidate all meta data and rebuild This can both be both resource and Prime Minister When you say invalidate it will relieve the whole Met our data and from scratch It'll bill that'll take a load off So these are the refresh options and impala that you have So
6:03:45
try to create a table see whether you can find it An impala I don't know So we'll create some simple table right We're stable You want to create Maybe I just change this and run it as it is right So I just created And if I go to Impala May 19 what is it No I don't have it So if I say refresh let's say perform in criminal refresh
6:04:29
huh Now it game transaction So ideally if you do this manually
6:04:34
because in pilot cannot automatically identify you created a table in high If you do that will come I needed huh So impanel or so stores a lot off my tar data on Actually Later Reese inquiries are in the cash So if you want to clear it you can see a clear cash
6:04:54
The reason queries You ran It was started in the in the cash little store So sometimes that can actually become big Like you're running a lot off queries So that will affect the performance We can simply say clear the cache Yeah I really should keep it in the cash I have this notice Impala How you son Liam Operatives No I don't think that is possible Probably admit or somebody can do it from her developer side I have never I mean what is the U S case I don't know I mean masking
6:05:22
table masking is there Um I don't know if it is possible I have never tried So table masking you can do so That has to be done by the Edmonds So you can say that master table It will not be visible toe any other process Like Impala or spy record anything if you want but usually we won't do that Usually they restrict access based on you sir So the you said so if I am a gay Okay so all the high tables will be there An impala Let's say But if I am a guy uh who is a developer on my access will
6:06:00
be that I can see only these neighbors in high for these tables sending pilot so behind the scenes everywhere you will have papers where you said level access you can control who can see what tables at the race Everybody will be able to see all the tables I don't think directly there is an option toe Stop them communicating Horton Books her dude doesn't know having by that uh they have this thing Hi Viz There on by default High plus space There's his execution engine Ah Then
6:06:29
elope Elope Is there uh highlight
6:06:32
Actually Elytis much reliable Much faster than Impala to be honest So I'll show you in the living Maybe not Now I'll see a session So this part is clear Right High even high Plus Impala
6:06:47
at least Basics are Hugh has an automatic way off identifying the data But that will work only if you have ah property limited
6:06:55
like coma or space In here You can create a table there It will identify if you have space or coma or something I don't think you're special Letterman's were to the structure So in Parliament this much only you need to know Hands off now I mean I'm just giving it as an option Normally in higher classes you don't teaching pilot or since I was hiding by lacing Beyeler not subject like it has become like so people don't learn it separately
6:07:26
in the projects It because the queries and all our thing the sequel you write everything is saying there is no difference at all on letting is a use case is slightly different No only it has its own It is contributor by clothing In pile is an open source project but mostly through the contributions from Claudia So it is like a preparatory predict So if any shoes comes close that I will support a lot An impala going number one Onda So And the problem is that if you go to other platforms you
6:08:01
won't see in Bala probably that is the reason people don't care much about in pilot Right So if I goto Horton books I will not see Impala Atar If you want security there is something called sentry service century your to set up century I don't think we have century here So in cloud I'm saying Claudia you have something called sentry thin three clouded up So this is called sentry Here you can set up or the you know user level access
6:08:35
So this level access everything Our century again Sentry is open source But so here how users groups
6:08:48
on da access see organization privileges model for higher end impala So
6:08:56
the problem is sentries again Apache But only Florida uses you Goto Horton birds You have knocks and ranger Two tours for this gold ranger on Knox again There Open source but only on Horton works So one problem is a platform Eyes There are some slight differences
6:09:16
It happens so there are slight differences Florida you says this century on in sentry you can configure who should access which stable on all the authorizations The admin has to sit I don't think sentries installing our cluster Which means anybody can see anything even high for high You have one indication you something possible you will create here That is not there I mean as of now they're saying hi Even lender right Hot air But always this by a different I'm
6:09:50
saying right now what they're doing you are going You're logging into one off the machine in the last step It doesn't look like that after getting my point So I think we discussed this in the first class So this is your cloud lab right This is her cloud lab And you haven't engine or so you know going hear from here Your connected the lab on any security Any service you nearly how to enable here and as off now nothing is enable So then you just start how you just hit the cluster
6:10:22
And anybody can see anybody stable or anything that you create Century is not enabled lighting on that Even when you're working you won't have access to sentry and it's totally the arm inside They only can decide who should You can request them You may be not able to see who has access
6:10:45
Yes yes So they will be in the cluster But once you hit their general that will be applied to you like all aboard This is whatever you're having Otherwise you can't restrict The use is like from coming in Horton Books has Ranger Ranger important Waas are injurious Similar I also like the ornaments documentation a lot It is much more precise and clear you real and to support their talking cloud argument is very here Nobody would understand what they mean Look at hardness on a documentations Very
6:11:23
easy to read and understand See comprehensive security for enterprise Heurtaux what ranger does how Ranger works right Everything you have here So Apache arranger offer centralized security for all these things high even space So if you go toe haIf they will show you what I see
6:11:46
There is even a you two presentation Very nice documentation They have the best documentation actually Ordinates cln lippy Live long and processes what I said I think that is another explanation It's not actually live long and process something is Is there Artem bugs They say there's a levelling process I don't think it ISS live long and process
6:12:10
so high Aviators are here on some optimization techniques out here Okay
6:12:16
uh ordinates document is the best if you want to learn or living If you're in a cloud a production fluster Some of the things may not be applicable since it is starkly open source like they have their own base light
6:12:29
So if you are using a cloud a cluster in production you should always refer to order a recommendation because not everything will be seen Some difference will be there right I hope that clarifies Impala So we're talking about in pilot
6:12:48
Um now I have to talk about something called partitioning in Life So how many If you're actually aware off partitioning some off you might be aware of this country considered Yeah For those who are already know this probably started fresher My seed idea is very simple So can you tell me So your manager asked you to create a sale Stable Okay On Duh You are creating a BB called Weeping Under table called sales So we're leaving the location It's a many stable It will be You
6:13:20
said hi Then what Their house
6:13:28
and the baby is what Everything
6:13:32
on the table is What Saiz
6:13:35
This is the location right For the man is stable on Let's imagine a very month you are getting the data London So in January you got the data So it became what John Dorsey is You said lower data local blah blah blah This is where the date I will go You know this already on The data got uploaded here Everybody is happy No problem The problem is as the next month and next month can happens right What is going to happen in your sales folder February What will happen February late I
6:14:14
will get uploaded It's in the same for them By the way on again What will happen Mat's right On April on May And so So the problem is if this is the mortal you're following
6:14:33
after let's a couple of years if you look at it the folder you will have pento and defiles inside the same folder Now the real problem is let's say you're writing a query Something like this Select start from sales right
6:14:53
Bad That's a month in Quito
6:15:02
They say What I think this quickie The problem is haIf has no clue Where is your data
6:15:09
The task to scan a well the files in this folder to give you reticent The percent is here that you know how you doesn't when you write a query it will scan away the folders in the sales folder So if you are having that's 100 fires
6:15:27
Hi It has to scan all the 100 files before producing the Listen the broad back off This is that your queries will be very very slow So in order to avoid this problem if this is the problem you're facing you can do something called partition partitioning off David Okay on in partitioning you have two things That is something called a static partitioning on dynamic partitioning static and dynamic you have what type of partitioning you can do Okay And I will show you that with the help
6:16:07
of an example that will be better But what is the idea in partitioning First you have to identify the column or columns based on that you want to do partitioning So I am assuming that this is my paper structure on most off my queries are based on one equal to something Right So what I can do in high I can say that Okay create a partition people Where the partition corland iss month on what I will do If you provide the data it will automatically identify
6:16:43
on create fold us like this on will place the They don't like this So when you partition based on month okay that my people ways to do this I will show you Technically what is happening behind the scenes is that it will identify the month column Andi as many months you have those many sub folders will be creator on It will move the data Now we know that there are only 12 months in in year So if this is the data model you will first partition based on year then month so
6:17:19
that it'll create afford a record 2016 within that well some folders in 2017 within that But if this is the mortal if you write this query it will only hit here Just people these other sub Ford So your queries will be naturally faster This depends on what is the columns you're having So let's say I'm saying here the column name is Jan So what is the next column You have a week one week toe like that you have I mean this depends on the data Now in this data I don't have date I have a month Korla
6:17:55
my have a day call Um I have a year old now Partitioning is not a must It's not like a very British do partition There are many conditions where you cannot go partitioning So like you said if you're having date you can't partition directly Because if I have a date column on let's say you're looking at last 10 year data How many days will be there 6 3050 If I said partition by data create 6 3050 folders that's useless There is another way to tackle it I'll tell you but you don't
6:18:27
do that So partitioning is applicable if the card in ality is very less some people carnality right How many unique values who have in the corner like in this case or let's say you are getting data from different countries in the world Fine maximum You have 150 or no country vice You can say partition But yeah like you said or another case is that you are having transactions and I want to divide the data They start transaction I d That's not possible because every transaction has
6:18:58
a unique transaction 80 It starts creating those many partitions That's not possible So partitioning should be used on leaving your car tonalities within control like this Right now you're partitioning only on the giancarlo I'm giving him a different use case Let's say you're getting the data from different different countries It also depends on what clearly you remain For example let's say the sister partition data I write a query select star from sales were can critical to India It is the
6:19:27
worst query I can write because my date ice partition on month on that I query based on country it will have no impact
6:19:35
is actually worse So if you know that most off your queries are based on country and month you partition my country and month I mean decelerate It's possible probably first partitioned by month Then convey your country Denman you're to figure out the card in ality So that and also if you are ending up creating Lord off partitions That's not a good idea for Hadoop because how do does not like a lot of sub folders and sub files right So on another drawback I think I discussed
6:20:05
this in the first class This this partitioning if you do in traditional data basis it is sometimes not effective Why Because we had a use case there We heard Oracle plaster like this Four machines are there Okay on Then you are saying Let's say you're looking at the iPhone sales later You're selling iPhone iPhone sales data on you Want a partition The data based on country now number of sales In which country Right So I will say that partition by country But that's a logical partitioning
6:20:45
You're not physically dividing the data The problem is it'll create countries like India China us and all Right So probably US partition will be here This much will be there And probably what this will be What I don't know Give me some beer Country Congo You're getting the problem You're logically partitioning You're saying that Divide the data based on country since in US Lord of people by iPhone What is going to happen to us Partition will be very big This will be your US partition right
6:21:24
I'm talking over the rock Okay And this will be a Congo partition on in Oracle If you fire a query what will happen the machine is responsible for They won't work together So the US partition call This will be very slow because this guy has to turn The Congo partition will be very fast right in her Do poor So what happens You will do the same partitioning Okay you will say that you have US data in their data and all will create self orders But their demand is is that even though
6:21:53
this file will be three TB it'll be on 100 machines blocks
6:21:59
You know actor This will be one machine Are you going Listen what I'm saying So if you're storing the data in Oracle Oracle has no idea Off blocks and replicas So US partition data will be three TB three pp will be one machine It won't divide this If this isn't how do three TB will never be in one machine blocks will get divided right So my query will be faster actually So the partitioning it's much more effective in her do come see right So which means
6:22:31
partitioning in traditional databases and all are not so perfectly It is affecting up to an extent but it also ensure or the partitions are having equal amount of data So by party partitioning is logical in every case even in her dope it is logical in Oracle it is logical in Oracle I'm saying that all the partitions off us would be in one machine There is no way I can further divide the state out You're getting my point right Because it is not hard dope or something Ancestor db So all the U
6:22:59
S Data will be in one machine so that machine will have lot off data later to us But in Hadoop if these are four data on or you will never have three terabyte on one date and he always divide and spread our distributor So my performance is better than partitioning So these are some of the things we understand Then we work right So So That is why some of the companies will say Don't do partitioning Much charting is there Scharping is dividing a table existing table
6:23:28
starting mostly view doing no sequel or actual after 11 J supports starting but it is not extensively useful Actually in no sequel databases you can do something for Table Chardin They were shining It's like physical division your take a table and say Take 10,000 10,000 rose Just divide like three or four and then dump it Okay But that is not highly possible in our baby amiss because in arguments you have normalization It is physically Any partition will physically divide the
6:24:01
data But can you say that I want to divide this table in tow Review to Z two z V That's not possible right
6:24:07
That is logical Physical division means like he's saying Scharping I take a table The table has 2000 rows I can't say divided into five to wonder to wonder 200 Rose keep in five missions That's not possible right In normal rdp amiss that sport physical division You are not giving any condition You're saying Take the full table Split it into 500 100 rules Keep it in flying machines That's not porcelain porcelain RTB Emma's in no sequel It is possible because it
6:24:38
is being normalized No secret Everything is being normalized so I can divide my date and push it and never silverware I want nobody cares It's already distributed in high evil tradition RTV amiss Physical division is not possible You logically what you say Look at this column on if you have 100 values they were in 202 100 Be 1 to 200 You don't have any control You can say this much situation I warned Or this size I want there is no physical control logically or dividing the
6:25:05
later right There is no way for you to understand after march again This affect how you has no clue right It keeps on searching Orilla nor data So creating this Either you can manually create this four days or you can so Normally we don't do like that We run a command and created actually So it is better to do this rather than talking about This is just a basic idea I want you to find out the data set on then start working on it So that s partitioning Can you say this
6:25:37
folder partitioning on There is a word file if you want I mean this has the commands but I have been facing one issue with this word file which is
6:25:52
of any copy paste Sometimes the comments will not work We will see validity work here Okay that that court right particle single coach
6:26:01
it was not identify So we will see whether it works or not So we have nor discuss what is static partitioning or dynamic partitioning And I will talk about this first thing Um let's create a table and then we can start talking about it So do you have the baby Yes right
6:26:23
Okay The baby is here Ah never say create people Just copy this command okay A static partitioning is a bit confusing but we can understand so I'm creating a table Call you sir one as you can see if you look at the schema it has first name last name and I d only three columns on Then I will say partition The by country coma region This means these are my partition columns I want to divide first by country than by region Fine but there is a catch here What is the catch
6:27:06
What is the data you're going to Lord if you look at the data this is the data You starting for one And if you open this data can you tell me how many fails Are there in this data Me three Which our first name last name on Uh I d like if you look at the table I created it is matching here right then I'm saying partition by country coma region like but there is no country or region here I This is static partitioning So static partitioning means you are getting the data you know from the where
6:27:53
the data is coming But the data will not have that columns
6:27:57
Meaning I know that probably this data is coming from so completely quickly US state able to California
6:28:05
And while uploading this date I should mention it The data does not have any columns for country or region or anything If I upload the data you will understand So do one thing Uh copy this to her Do this You're starting forward Can you do it
6:28:22
Uh goto Ashley fs There is uh I just for people who
6:28:33
you're setting for one in fact copy or so that we can save time right I mean these are also required Yeah so then you're to use FTP if you're hoping the local your to use ftp better Poor people here It's faster right Anything And I have a question for you So right now in her Dube will it be already created The country and the region folders for you Partitioning means it is creating supporters like I'm saying create a partition table country and region Will it actually
6:29:20
create a supporter Yeah so no And I think will be created as off now and then What do you need to do this Um where is the word pack
6:29:33
So I will copy this because I don't want to type a lot pretty or to change it Okay Don't type it as cities
6:29:41
because And what is the location that cool for me File name is different I think you certain for one dot Txt Right Is that a txt extension Yeah So now what I'm saying I'm saying that lowered the data And this is my data into the table And while lording the data I'm mentioning the details I'm saying there's a country New Zealand on region ku pic So what will happen It'll create a folder called Continues Land Another folder called Reason to Back Within that this file will go Can you verify that in
6:30:40
the warehouse folder If you have lorded it it should be available The warehouse for did so I got a few Ah I'll go based DFS And where is your data You sir High right You say hi ive their house minus may So many babies are people started May 19 baby Ah here is the user one can you say can prequel to New Zealand There is a folder in her duke or country quarter New Zealand But then that there is region equal Tokyu back within that you have your data right Right yourself and see whether you can see
6:31:39
it This this will be there in your project and assignments So partitioning is very important It's in the world file Right Okay so it's wrong right Sorry Sorry I didn't just showed here So remember this point Static partitioning should be used like we had a use case Okay Where We were getting user data from multiple countries Okay we're not our clients application The use of data was coming from multiple countries but the data we don't have a country column so we used to manually create That's
6:32:25
called static You will say that for the state I in us
6:32:29
now we have to do something called dynamic partitioning Never show you that So make sure you are able to Logan toe haIf on Always do Use your baby some of your using the four day be on when you typed that commanders there already that table exist because Lord of your actually using before db So if you Meanwhile if you look at this data right Hi partitioning Can you tell me what is the difference between this data and the previous one You lorded Exactly It has the city and ah state or whatever
6:33:27
By the way what is Togo and West Bengal doing together whatever Okay so you have some city or region Yeah So this data has the country and city column So the best way to do partitioning here is you create a partition people You said there is a partition by country and state or whatever column on Then you tell high I'll give you the data You decide yourself I'm not going to divide later You look at the column I don't understand how many countries are there or how many states are there Accordingly
6:33:59
create them and put them that is called Dynamo But there's only like 20 line soft data to India today Those many will be created I really on I think the first park it starts with Christmas Island There is no reputation right But there is one thing you have to understand in dynamic partitioning dynamic partitioning is by the for disabled in height Because somebody can come and say that Okay create a dynamic partition table Use transactional record Um then what will
6:34:32
happen Millions off partitions will get created and will screw everything So by default even in production blusters dynamic partition will be safer So first you're to enable it first thing second thing is that you cannot upload the data that I cleared for dynamic partition paper What you need to do First you upload the data toe temporary table or some table OK from that table you insert to partition table
6:35:00
Meaning previously what we used to Ruby created a static partition people Then we said lower data In part it it went We're able to do that because we know it's partition where it is going right now We don't know how you has to decide So what I'll be doing I will create a table first normal table I will upload this ST up No partitioning nothing Then I'll create a partition table on from the original table I will say insert of the other paper So while doing that it will divide Anson that
6:35:29
is only liver toe dynamic partitioning in high OK we'll do that practically so you will understand You can directly load It has to be in a table from their your to insert but in this example it's not really useful because you're having all unique values like normally you should have like fire records for one country or something like that But here Everything is unique Values were having but that's the idea Um let me see if I have another file Okay Because carnations and high just this
6:36:01
land property Okay study one woman Okay No order list I will do one thing Once we finish this particular example on dynamic partitioning we will do something called bracketing or so And once we finish that I will give you this assignment This landed property analysis You can do it by yourself I'm in now It's easy commands that already there on that will help you to reinforce the idea on partitioning This has partitioning also Just try that once I finish Ok
6:36:50
so for the time being for dynamic partitioning what we will do we will create a table called user too You can see what I'm doing from here Same thing I copy paste on the sister partitioning table Right Look at here I'm saying that partition to buy country and region So this table is the one which will have my data partitioned Okay then what I do I create a table call you surgery That's a regular paper First name last name Country
6:37:28
region First name last name our country region Right So do we have only four columns Where is the data You sir In four
6:37:49
three right We have first name last name I D So your scheme I should have five columns right then are late will work I mean the original table So the stable core user three will hold your original later So what you will do first you will load the data here How are you Lord Nor data In part because already it isn't her Do where is in her group with a who slash What's the filing
6:38:21
you say in four Read out the extreme right Is that the name into table use Ah so don't tell this Lord the data toe use of three table Yeah So this use of three table will hold your original data No partitioning Nothing will Just have your data From there I will insert the partition table I cannot directly upload the data partition people in dynamic partitioning So I should copy the data out to one or is in a table from their out inside I am static You are mentioning which country which state
6:39:06
And or manually so you can upload the data say create a folder like like and then from the file at cities here the file has to be divided based on Carla mallets Dynamically maybe How you have to decide So you are lord rate at one table From there you say inserted the partition table on high You will dynamically partition Based on these two columns country and region here I have created right country And yes
6:39:39
first all that increase within country regions So right now we have around 2025 times soft Later Only that much will be there Partition calling shortly Our service came up when you define a partition table
6:39:52
So these other partition columns whether they exist in the data or not exist in the data they have to be outside The less came out your supplication by there you mentioned the scheme so it would not be here It'll be here No it's not required only here because the daughter schema will be including the partition columns So it will understand Okay Now I want you to do one thing If you look at the word file right You see these things Can you see this Right So these are the properties union to
6:40:22
enable for dynamic partitioning So the first one is very easy If you really will understand you are saying that a hive enable dynamic partitioning said hi exit dynamic quotation through this means by default lettuce falls by default Dynamic partitions are not supporter So I'm saying that yes I want dynamic partitions But what about this Let me copy that So even if you enable dynamic partitioning by the fort and high what happens if you create a partition People at least one
6:41:07
static partition should be there So it's a bit complicated to understand So there are a lot of layers of security but everything you can overwrite also So in how you first they say I will not I love you to create dynamic partition at all Only static is allowed So you say said the high except dynamic partition True which means now I can create dynamic partition But then there thinking if you are allowing you to create dynamic partition what if again you try creating 1000 partition 10,000
6:41:36
partition So there is a strict a motor Now we are a non straight What will the strict more tell you Is that even if you are in dynamic partitioning
6:41:45
Okay you have to first create one static partition in the people Rest can be line 11 problem you started You do rest Everything has to be dynamic You can again turn it off So many people have a country huh East partition David So right now our partition is based on what country What is that Country and region right So what they're saying country you manually mention while uploading country partition Jumana dimension reason I will decide in strict more northern Italy Man is internally But it is
6:42:23
saying that manually you mentioned at least one static partitioning to be created It is just a safety measure even in production centres being disable it because who want to do it You are You have three columns on one column we hope to statically mentioned Miss nobody is ableto allow you to do that right But sometimes it is very useful I'll tell you because you're getting the data where there is no country column Okay so you want to add a country partition that will be static You
6:42:52
have some other data Let's say I don't know state or something that already is there so that can be dynamic So in that case you're on how proper knitter Because you will mention country statically There is no other way Let's stop them if you are dynamically creating it So we are turning it off because it's a developer in Mormon It's fine We'll create as many partition as we want Normally this are all true I mean like dynamic partition is force which means you cannot create any
6:43:18
dynamic partition This will be a strict by default So what we do is that yesterday somebody asked me even I forgo to discuss that you can create scripts hive script like sequel script You create sequel script right dot sequel file Staying way I can copy all this commands in a text file Save it has dot SQL I say run It'll run as a script that you weren't type everything right Holder orders that So in the script while doing you will first are these things you know set this does this and just
6:43:49
run it That's how usually we do it It can be a skilled also SQL or SQL little
6:44:01
Technically I have not tried anything apart from that Normally we try either SQL or SQL I don't think if you want any of that it's will it work If you have a secret spirit how am I gonna try Because all this course we have is executed by default people sailors at school only I don't try it It might work but I'm saying so And these are specific to a session So highest languages called excuse hive query language And this is not a script We're not writing scripts This is a shell This
6:44:34
is a quickie script is different What I'm saying all these commands you can right now Text file
6:44:40
not back on You can say Run the north path Little execute one by one You don't have to copy paste Each time it's skewered it is hive query language So So that's what I'm says state yesterday the language of high school excusable which is derived from askew it so 95% it is similar There are some very rare uh commands which only high has sequel doesn't have So I think it is Sequent photos into dialect What It is using Sequel 2002
6:45:17
that I am see for much like for sequel sequel 92 sequel 2000 Toe sequel to those and forth hiatus We Don't sequel doesn't do If I Remember sequel format I'm saying those impacts like Do you find any syntax different from your regular sequence in tanks Are partitioning close A different right That's what I'm saying So if you're doing partitioning and sequel it's not this comment Slight differences there So these two properties I think you're in district What is the enabling
6:45:45
dynamic partitioning on the Morris known strict now again for controlling this is Hi Ive exit maximum dynamic Partition 100 which means you are restricting maximum number off dynamic partition
6:46:01
This 100 then this property is not actually correct The property is correct but you should find their different value For example Ah let's write it as 100 What this means is that so when let's say you're creating dynamic partitioning let's say you have a very large files on Let's save in the column You have 1 50 values countrys Imagine so 1 50 partitions will be created Each partition will be created by one Radusa It's map It is job undocked early so this creation off partitioning table
6:46:43
Vilify Adama produce job And after day so one producer will work to create one partition So if you're 1 50 partitions 1 50 reduces will work here You can control it This dynamic partition for north 100 means maximum number of producers will be 100 used So if you want you can control the number off producers Okay The advantages and disadvantages Different meaning sometimes
6:47:14
what is going to happen You may not really need 150 reduces to do the job Correct So if you have 150 partitions your data is very less you can achieve it You think of produces Maybe So you say limited to 50 Only 50 reduces will be launched So this you're saying so These values are wrong I'm singling in the sense you're saying maximum number of partition is 100 That's fine Okay maximum reduces our 100 So I really hear 100 Reduces will run if you have 100 partition But mappers will depend on
6:47:50
your a input split Like taking so like what data you are getting so that you don't have any control anyway Produces something you have control My point is if you're having a very small file Okay And probably 100 partitions are your only 100 Reduces public can reduce case so can do the job You can control this property
6:48:10
And I didn't make up this properties X have a level in Hioki so people will be thinking that OK from where did you get horses Properties It's properly documented I'll give you the document for this You can copy any of these properties Goto Google just spaced it on how you language man Will will come Uh nor the first one Okay so this is Horton verse I mean the original haIf languish manual Welcome If you open this Uh oh Just search for here You have all the properties Um where is it Hi
6:49:06
You accept dynamic partition before Value Force Okay Whether or not to allow dynamic partition High exit dynamically partition more before value Strict instruct more The user must specify at least one static partition Right High xx maximum dynamic partition before value 1000 high ive extinct Maximum dynamic partition for nor maximum number off to clear in each map producer Nor that means how many producers
6:49:33
will be used total right And you also have other properties for high We will look into that later anyway So if you come here So we have enabled all these features right And now what you should do is that what is the word Fine Yeah You have a simply say insert So this command you will say Insert the data from the table So that partition David this is how you do it And this will fire a map Reduce job you can see on you can see on the screen it'll create partitions Look at my screen or you
6:50:12
can look at your screen Also which you can see the partitions Loading loading loading I'll show you
6:50:18
it was dynamic right Soc country Now regionally right now See Do you see Country Nigeria region for most key work or mind See all these partitions in order dynamically
6:50:34
on Verified that in our group
6:50:37
Verify that this is created in her group My You go here
6:50:42
Um what is the papal table name So if I go to you sir to see how Korea came I think it didn't come It says just Korea North Korea we have right and courts and
6:51:08
yeah I December we're known strict more on any country Your take there'll be a region any region Your take will be a part file because it is our put off a producer Not this way that is Rosa welcomes Reduces I think I don't know how many reduces ran for this That's another thing So this is a little right on if I go to something else So that means those many reduces ran right Was everything Is it'll so otherwise uh the role one toe evidencing I'll just check We can actually see the
6:51:56
statistics here if you go toe
6:52:01
not workflow Job browser There is a job browser on This is the job of Iran Did I show you this No I think I showed first class If I go here there is a mosque
6:52:32
It's not displaying everything It's sewing stage one I'll see anyway So can you are able to run this I said yes I think Yes right Can you see the partitions are no home Introduces the card We'll see that later But are you able to create this religious Locate your rich DFS Onda See whether you can find all the partitions because of the import data right So our data need to be either changed I think because all other date eyes like regular So here If you give like this all you have to
6:53:27
clean the Simon So usually what we do is that I mean if there are only limited records you can you say that big or spark to clean the later you can easily clean it You can say that fine course and then replace So first call a name that you given the partition will be the top level partition But then that will create those minister partitions But the rest of the partitions are loading right There is no problem I think are you start So there is no way to search for ah rejections
6:53:56
from here the auto manually verify because here it says country equal to Korea region equal to know that's hard You didn't lose if it actually it created Korea in the region is north Actually that is incorrect Okay Asper So before lording you have to validate the data that is on living So if you do that there are validators It's candle field evaluation on whether all the fields are like this On exception handling this Your job before you this day does It has an exception That's fine
6:54:26
but usually help the vanity now
6:54:29
So this is usually have you do dynamic partitioning on One more thing is that when you're doing dynamic partitioning there are performance problems sometimes like creating too many dynamic partitions is not advised Even Hadoop will not allow because name nor has to handle all the Macarena for the sub folders on the files like you are having 150 countries on each country has states So if you say divide by country and state thousands off folders will be created so that will
6:55:00
increase the my target on name no So in one rate is good Your queries will be faster but other way so there is no limit as to how many You can create partitions but they say that do it with caution
6:55:15
No only thing is that you are a name nor performance will be affected I like thousands off partitions are not a big problem but so many who is number of partitions are there that will become a problem Actually it's not tables I said Eso Usually when you use partitioning is that if your queries are running slow like on the reason is that inside the same for that you're having my paper files and it has to scan or the files on There are some cases where you cannot go partitioning You may not
6:55:45
be able to find out a logic to do partitioning Probably there is no common column or something Then I can't no partitioning There is another really there It's not like everybody must do partitioning There is no rule like that Now somebody asked the question bite So this will lead us to bracketing But what if I am having
6:56:06
Let's say you're getting data from the country side So you're decided to partition so it becomes country
6:56:14
Your Partition column on the 150
6:56:18
countries Look at Amazon Look at Delhi for example Delhi has presence in 150 countries Okay Andi So customers are buying products off them Andi have some data with this country vice Okay so what They decided they decided to partition based one country Fine No problem Now after this what happened Is that so every time in customer response chasing a predict there is a transaction I D Right on They want to write queries like this Select something Okay Where Country Quito
6:56:58
India Okay Goma Transaction idee equal to 1234 Now the problem is it's already partition based on the country So that part is saved because it will look at only India But within India you have millions of transactions So this transaction fiery will be huge Just to scan all probably there are multiple transaction files Okay on Trans you cannot partition based on transaction I d Because every transaction is unique right So what you're Alexa Kristen and I don't know whether this
6:57:37
is available in our IBMs and or you do something called bracketing bucket You have something called hash partitioning right Can you tell me what this hash partitioning in rd bms I'm asking I didn't missing how Who and also board hash partitioning borders Ash because this pocketing is very similar to hash partitioning in our dbm Aside I have not done hash partitioning So what we do is that see there's a planet's action I d corland right This is your bank transaction
6:58:12
I really column on here One of the values we can have Let's say 1234 Right Like this transaction 80 on let's say one million This is the data you have now You want to divide this data based on transaction 80 and you can order partition You say bucket it when you do bracketing it can be done only on one column You cannot serve two complicating Not possible On I say I want to create Lets it then buckets So you are always mentioned The number of workers see one that is all you
6:58:46
do what is going to happen within each country Partition It will create 10 files or folders It will divide this data based on an internal hashing logic Okay on create 10 buckets 10 files within each country folder So in this query hit it will calculate the hash off this Okay And find out which finalists having it'll hit only there So too easily Understand This is not actually idea but
6:59:17
56789 10 11 Well this other transaction it is now I want to divide them in top 10 files So that is our internal logic Which hive uses I'm not going to tell the same logic But when I say I want 10 buckets 10 files will be created Fine So I say 123
6:59:39
for these other files OK on that Safety Baker Very simple logic You want toe find a mortal or 10 you know more alert Invite what is the mortal or 10 of one one So this will goto this bucket This would also go there I'm able to understand This will go here This will go here Well it is not going like this I'm saying are similar So there's an internal hashing logic on Then when you write this query it'll look at four It'll go the fourth bucket
7:00:15
That's a logic Not this logic It doesn't No more lifting Okay Summer Complicated logic But it will internally divided We don't have the border little internal it Right now many people have a misconception that you should always use partitioning and but getting together No I may have some misconception People who are already working I have seen a lot of people discussing No but in most of the examples when you search you want to learn partitioning people will say that I
7:00:43
will first create partition then inside that marketing That's a practice but you can independently create But getting without partitioning I can simply take a table So I want a bucket it that's all I don't want any partitioning or anything So what will happen in the same folder It'll create those many fires 10 files Yeah physical We can see that marketing we can see Practically we can see if you can do partitioning Don't know about getting because that's the most obvious
7:01:09
thing Very cannot go partitioning You enable bracketing For example India is there within that you don't have any choice on There is no ideal later Decide how many buckets are required So the next question is how many buckets 1000 So ideally you should take care that the file size should be somewhere near to the block size Don't create like one embrace ice bucket
7:01:34
So if you are having let's say that the it I spend GB within a partition Okay you say create that 1000 buckets it will become very very small Don't know that because that will create smaller fights So I really take a block size calculation How many backers soon video to decide some logic your views for that Actually if data becomes big that's not a problem there It should not become small That's the problem Like buckets Ice would normally one nb
7:02:00
Then it'll create too many small files in her to write her job Doesn't like it So if the bucket sizes 500 define it can become born JB it'll still divided into blocks That's fine We are okay with it Nor nor the block size I'm saying bigger than the block size If it is smaller than the blood size how do Boris doesn't prefer that because smaller fires up north very well handled in joint operations These buckets are excellent for joint operations There is something called bucket join
7:02:30
So how you has lot off advanced concepts for the time being you may not be aware off that Okay so what will happen is that I have a table here Okay on this is my corland on this is bucket it
7:02:49
So this means this is divided into let's say fire files 12345
7:02:55
I have another table that also has the same column that is also marketed or civil right And this is also having either saying number of buckets or my different Then you can do a real fast bucket Join There is something going back enjoying because since the data is divided into equal equal partitions the joints will be really fast actually when you pocket So when you write a joint query you will say Use back Better join there is a commander can use I mean there can be
7:03:27
mis matched the annual do joint operations There should be a common core of joint cola on Usually they will have similar increase Right that is idea You're doing joint Otherwise what is the use off a joint So So this is like you are having to tables and you want to do a join operation Normally you can do a joint no doubt But if the joint column Corman column is bucket erred okay then there is something called back and join you can use We should be faster High will internally make it faster
7:03:58
Since it is bucket er the join operation will become faster No hashing so and that is my tipple types of backer joints actually in height so it's very interesting
7:04:08
It's bit advanced cancer but these things are for several in no no that's anyway distant off So it's very rare that you may get a chance on this but people are aware of these things like Bucket joins You can do so we can do But getting practically that will be better right Rather than talking So why don't you look at another data set
7:04:31
Can you open this marketing data
7:04:36
How do you open with what Excel Right See this is a very interesting case study because here you can appreciate more partitioning because we have similar things See you are having street right on the street is having almost unique values some some unique at least city Sacramento zip code You have state you have California bedroom bathroom square feet So this is a love this thing Real estate data and Piper's residential condo A blah blah blah price some price So and how Maney Thanks off data you
7:05:13
have 90 This is like soft data you have Now our intention is to go back getting but little do partitioning or so just to see any way they can do that So what I want you to do is first upload this Bater toe How do you up Lord The file name Israel Underscore ST dot CS week that so finally and I wanted to open this work bag should be here but getting high s So what they're going to do first we will create a table Okay
7:06:26
And this isn't normal table OK
7:06:29
I say create table Ari estate
7:06:33
that's called real estate on this has the schema regular schema No partitioning no but getting terminated by coma But that stuff no changes
7:06:47
Andre will load the data How do you lower the later Lord
7:06:53
They uh in part So I guess you will be able to lower the data So this if you'll create what you can do it slower The data on you have to enable bracketing So again bracketing is disabled So just say
7:07:39
that life and force bucket include
7:07:44
So the command high you don't and four start marketing equipped true with a neighbor marketing night And look at this comment This is probably the most important command Uh huh Create table You are calling the table bucket underscore Table on I have the schema partition by city clustered by street into four buckets What do you mean by this partition by city So city you have common entries right Nothing Sacramento or something So I will say that partition Carla Miss City
7:08:28
Then I say clustered by street
7:08:31
the street column into four buckets
7:08:35
So it will create those many city partitions Each partition will have for this thing before buckets but it is not mandatory Another thing is that if the city has only one line off later only one bucket will be there If it has more data than only the four buckets will make sense Some of the cities will have only one street That's a great only one bucket In that case on now just Lord it and see whether you can see the buckets I want you to try
7:09:11
Be severly how even marketing is disabled now I want although should try us said I'm just doing an insert Once the insert is complete go back to your area Use that high warehouse for that You should be able to find partitions and buckets So I will I will even show you my pockets I mean like we just want to see the buckets You sir Hi Sorry Hi Their house
7:10:06
Uh minus May 19 baby May 19 Baby what is the table We have Book a table If I open bucket table this have the city If I open a city there are four buckets But again bucket ing or so depends on distribution off data See here The last two backers are not having any data But if you're having only one line Oh are just two lines like then only one or two backers will be created I was very in religion like very less later it won't create four buckets here I think you're having more more like some
7:10:48
14 lines I don't know how many you have based on that clear But the last two pockets are empty This example So ideally you should have more data This is happening because we're having very less later But some 1000 lines we have actually But number off mappers will be the right either napper already I said reducer So I'll show you here and that property It will be visible It can be either a mapper or other Do you say now when we created the partition Hey saying I'm not able to see any
7:11:29
reducer jobs Only mapper jobs ran
7:11:32
So see here Maximum number off dynamic partitions allowed to be created in each mapper slash producer No So it can be a mapper job or a producer job If you're having less amount of data that will be a map Only will run but sometimes what happens You will have a huge amount of data on the mappers Will first move the data in the reduces will aggregate in different buckets or partitions Now he's saying he's not able to save Producer's in the job I'm saying it can be either Mapple reduces in the
7:12:06
then firing up partition quickly It depends on resource allocation If the cluster is having resources it'll allocate I got only one mapper Mine is actually only one map in order to sell it Yeah it can be either AMAPOLA producer Doesn't matter Actually to be honest what is going toe happiness when you created that partition table right When you don't insert what is happening map it is job So and what is that map it is job doing That is my question You task to say good again
7:12:37
the date up That is what it is doing right It is not doing any other logic It is just segregating the data based on your partition or bucket you know So like now you're did So Did you take this partitioning with marketing Is that Orson or fighting introducer You're saying that the recent one right with marketing So you are getting radio sir Little map It is that How many one map Even I got one Map it So sometimes what happens if the cluster has enough resources It will fire mappers and producers
7:13:11
so they can complete really fast So mappers will first probably segregate based on partition and reduce It will just calculate the hash and dump it But if the plaster is not having enough resources this job will be very slow Only my personal belongs to They have to complete the whole thing But there is no intelligence in attempting disaster way off the cluster managing your partitioning pluses The logic is certain but you cannot really say that has to be done by reducing or a map right Sometimes
7:13:41
it will be achieved only using a map for sometimes little shuffle and fire radios are also yes In my case it was done There was only one map in my memories In his case I think for reduces their lawns So probably there was some shuffle operation depends on various your date Our So another thing is that then there is a producer there is shuffling off Vaca data show first and then goes to introduce it So I don't know probably in his case some shuffle operation happened
7:14:11
That is where the producer picnic
7:14:13
Anyway it is not really important I'm saying I mean while inserting into a partition table it really doesn't matter whether the map it is doing it or produce that is doing it does not have any impact on the developer side because it is just a finding by column on then moving the later that is always but in a query and all It is very important because if you write a proper query how many map with producers get involved That's very important Hi Can also stole that The Celtics text five So they
7:14:43
so you write a query on the output is normally shown on the screen Like you write acquittal show that you can say save it on actually efforts output whatever Queria the command iss uh in search over right directory There's a command call insert over I directory Then you write your query What will happen is that whatever list our put off the query it will save us a text file on her Do if you want to save it My normally want to see But sometimes you want to say if you want you
7:15:17
can do that also directly It is insert override directory If I remember because we rarely save it as text file Okay Hi Insert
7:15:32
over Right Directory So this is the even insert over Right Directory
7:15:43
Ah but hypocrisy it is there You can try this Command insert overwrite Directory on the new GIF Ah apart on then you say select star So let's try that I will say directory Um what be able to say uh it is suspending Oh by some sort of what Our directory I will say goku slash ABC Go on start Uh give me a table name that we creator just now
7:16:58
Not so long Right There's a NASA logo Right So this means if you create a folder in her job in rgu ABC there that result with the store query results You want to store it somewhere You can do that You can try this out See So when you're creating a hive table it store it as a text file by default Whatever data you're lording into ah hype able it is available in her do And that will be in a text for my Lexie Yes we or any text format on over the period off years there has been
7:17:33
lot off compression techniques to compress the later So while creating a table you can mention how to store the data Now I have some very important works links I will share it with you Um is that a way Okay I can Anyway you're not going to see right I'll just open my made because I just had some things I wanted to show you
7:18:02
I think in your assignment this is there Oh I see Right Oh correct on that is very easy I mean you so storing the data So it is very very easy You just need to understand what is happening Good Yeah very rare Uh customers Are they important right I never quit heart Miley Yes So here does it
7:18:29
so the last important works links which I found very interesting So RC is a format so there's a heart in bookselling but eso RC files in HDP Better compression better performance So I've been creating a hive table Last line you can add store as or C only One thing you need to do justice store SRC on what is going to happen Whatever data you're loading into that table high will compress it or C stands for optimized broke corland format It's a compression technique What is that
7:19:04
going days First our brand is your data is compressed Well that's not a greater hundreds maybe second and to engage in the most important are doing this is that Warsi has indexing There is a technical indexing right now in high If you buy the Ford has indexing but that is very poor If you enable index and you will not get any performance what you are see this It'll compress your data on whatever data is in the compressed file It will create an index inside that So
7:19:34
when you hit queries COLUMNA queries and our your queries will be really fast If the table is having O R C Property Cornum Indexing regular column in Dixon you have right in our TV MSN or what is indexing in orderto faster the queries you will create sort off like Hardaway Say Point
7:19:51
So the orginally WASI will create an index on row and column as well There is raw indexing and column indexing It'll create on everything actually and it is very very efficient Apart from Marcie if you want you can create normal indexing and hype That is something called a bit map index and compact index But they're very inefficient actually because even if you create an index in high if you are not actually dealing with name notoriously office you don't know very Citatah But if if you know
7:20:22
I'll see what is happening is that you are asking how to manage that data compressed data so it will create its own index for the whole data and store it Now I'll share this link But if you scroll down one of the things that they say here is this So these are so if you have a 585
7:20:39
JV file in Haifa If you started as a text file This is the size there is A for Michael R C File It'll become fine or five that is another for Michael Parquet were going toe to anyone If it is all of us it is 1 31 GB That is the level of compression you are getting so far K and R C R That compression formats Uh see it's not very common now because a lot stays optimization or Nazi I see it's roll call in that O R stays off the maestro corner so I see No where is using parquet Some people use
7:21:16
so park is also another compression technique But Parker does not have indexing or anything It's simply compress your data that's all Once he has this our grand age off you know indexing within that right So that's one of to engage
7:21:32
Uh avenue is a serialization formats
7:21:35
How do I say So Let's say you are sending a lot off data You can send the data along with the schema Normally where will you mentioned this Came out in a table now averages a format which can apply scheme Are you into text files without a table or anything by sending the data you can have a headbutt on scheme So it is very common in hard oops systems that you use I've row format OK And if you scroll down
7:22:03
So this is how you are Steven store the data This is one thing you need to understand
7:22:08
So it will create index on Lex a 1st $10,000 You can mention how many rules you need to create Index is a row and column index So here I am saying that $10,000 I want index So what really low 1st 10,000 rows will create an index and store here next $10,000 and create an index and store here So somebody squaring it can understand which you know group off Rose It has to push and it can even skip these $10,000 in one short there So you're writing a filter condition since it knows
7:22:42
what is inside here that it can simply skip this entire 10,000 row and scan here So it's very fast if you do it in our city Other for mentioned I have don't have this ability
7:22:54
Onda There is something called victory ization in high centered one's property victimization will allow you to read 1000 so frozen one short Normally when you're reading the date I treats one draw second or third row if you enable victimization little RC or if they can read 1000 so frozen one short victimization and Orville work on Leona Watson Wars is only for Mac It supports practice ization or so this batch indexing this properties are only available on when you goto Horton
7:23:26
works I told you that you can have a sit properties in haIf you can enable real time queries The condition is that the table should be or see then only acid will work So for all these reasons people prefer urc whenever possible But what can be one drawback These are ordered bandages But what can be one drawback
7:23:48
exactly It has to be compass
7:23:50
meaning your safety or cycles My big require anything which is compressed You have to decompress So compression is okay You're storing it and that's okay But when you're quitting
7:24:03
it cannot It has to encompass silly right So CPU cycles are required but still people are preferring it because they are okay Toe Spend some money on CPUs But still it is very faster on gives you all these properties So just go through this link One link Is this okay Andi try it Is this this is one strike This 10,000 roses called one strike That that one block off roses called once tried actually OK
7:24:38
on this is the spear off queries off one terabyte data So
7:24:44
this is high $1.0 this high 1.1 That's fine On then High bless vector rised Very high placed people E V p is predicated Pushed down You know what It's pretty kid pushed down Like sending your filter first light So if you write a query like this
7:25:07
So they say you are saying that joined the data on then felt the later this bad right
7:25:16
You are saying joined the data Then filter the later First you should filter the data then join So by default in our baby Emma sand Or it will first bring the filter together It will optimize in high You have the man Will you do this That this court ppd predicated push down So if you enable ppd along with the RC the spill Terrible Come first then join Welcome So it will push the predicated Basically that's the idea Oh I see Yeah So and I can show you this I mean I'm not
7:25:47
just blabbering if you look at your uh clouded up this thing
7:25:55
Ah the old man is right So there is the glowed lab
7:26:01
You look at the cloud a manager you can see whether these properties are enabled Probably not all of them are enable That's one thing Let's check I'll go here I'll goto haIf and I will goto configuration Um I don't know if I save ppd Uh no it is not showing Enable victimization
7:26:36
optimization Okay it is an ever so this victimization is the property by which if you are storing the data as a warsi it can read a hose and so frozen one short If your neighbor electorate ization it can breathe thousands off rose in one go and then give it to you So it makes so victimization is anybody
7:26:56
ppd I am not seen I don't know whether it is enabled or something If I says for ppd it is not predicated pushed down I don't see anything predicate No it is not the probably the property itself might not be here It is not available No I ultimately when you are reading it will not get by It'll get row by row There is a thief ordination It is really slow actually In haIf when you're reading the data it's in the raw format right So when you're creating the table you are saying
7:27:29
raw form and the limiter on Let's say your query results requires Let's say our POTUS $10,000 It will not push them toe sandals It'll say Roman drawer to rotary like that it will come It's actually very slow in or is in a life If you enable victimization it can read The minimum 1000 is there in mechanization many $1000 You can mention what everyone those many will come in one shot So it's very fast Actually while getting that isn't on victimization requires RC no other For Michael
7:28:01
Birkett secularization there's a reader for victimization which is available in all what See only as off now I don't think any other format supports secularization like so now the question is that how do you enable at this O R C right optimized bro Column format So
7:28:28
huh So these things are very very important Okay let me see if I can show you from another one So these are some of the things which you can go through So when you create a table all you need to lose that you say store and so RC last line you add stored as a warsi and will become a waxy on it will take the default values You can even disable compression Some people do it So what if you don't have enough CPU power you want to use indexing and or but I don't want to compress the
7:29:04
data You can say compression And if you don't compress the later possible on So these are the properties you can mention in O R C You can mention the comparison The reformed algorithm is called zili X Very efficient You can either say none zili or snappy these other three options So the Libyan snappier compression algorithms if you say none O are see table will be created no compression on Then you have O r c compression size number off bites in each compression chunk So the more
7:29:41
you put the more compression will happen In this case it is to a 56 K V by the boy or C stripes size number off bites in each strike both of basics MB So each I told you write 10,000 rows like that So the size off that is 2 56 MB You can say that in my table Probably your table has one baby raker divided That interval 50 system fits like that on roll index strike 10,000 for every 10,000 role and index will be creator So this you can increase or decrease
7:30:14
compression Size is the number of bytes in each compression junk So that means when it is compressing
7:30:24
it creates something called chunks inside So what should be the size off each minimum size actually so here by it afford it is 2 56 caving the prints on the data as well So we leave these things to default We don't change the compression size and all because the algorithm z LIBOR snap your none You can change on the stripes Size is how it is dividing the data like here the stripes I says to 56 MB both of the six and we will be the size off your one stripe right on in eat stripe It will get
7:31:01
$10,000 and create an index So probably in one stripe it will create like five or six indexes depending on how many rules you are having When you were mentioning the size on Britain that how many rows one index need to be created It's okay So sometimes
7:31:18
I mean this is this other properties just really will understand anything Okay on You don't have to mention this also So normally when you create our city will you just a story that Zorc Because that is better or worse evil taker off the compression itself These are the values you cannot Just if you want to tune them if you send places Store also are so that sort of thing
7:31:38
Yes So if you're say compression none it will not compress the data It will index it It'll definitely index it But no compression will be there Know that the fall is fixed five Because for so long people are using text file I don't know O r c is an evolving standard Okay Andi o r c has different different versions and all with a very high release They will release a new standard for WASI and all X trial is the standard former on you run one the force and everyday toe use A
7:32:07
particular standard like this is better So if you want you can you say That's the only thing you can't really know I'll say fire You can see that you cannot read it because it is like compressed Andi Only haIf can read it I can only create and read it It looked like uh like a zip file only But you won't be ableto open and see what is inside Only haIf or what's the real risk Embry detection like you want to set up a small set up like enough production You want to do it production So off Obviously
7:32:41
union machines Onda You may not be setting up on the hard hope and hype It's nobody People will be having the complete package action when you say how dope and high That's only two doors right So rather than setting only Hadoop and high so it's very easy You have to have machines then you have to go to one of the vendor in the cloud Our Horton books get their distribution Install it If you are using Apache Hadoop you have to first download the origin our source
7:33:10
extracted that are four file score psych exam maliciously oversight Example Yon site Example Mapper set example in each off these files Example Fire sorta manually type the properties Probably take a year to complete almost on you will not sexy Even I feel sexy because very difficult Nobody does that We used to do it I used to set up a partial clusters but very very complicated It is because everything is man with there is no or permission everything And there is no use for that Also Onley
7:33:40
uses that probably you want open source Apache That is what inwards
7:33:44
The Horton Books edition and Apache Additional exactly saying nor differences Yeah even Cloudera important works There is a free edition Okay So since all these companies are selling open source products you can goto Horton books or Clodagh and download their product Install them on as many machines as you want The problem is you will not get technical support in Claudia you have in addition called express addition Okay What is over Cluster Our plaster should be expressive vision Right You can
7:34:16
check it So this is clothing right Um some very could be there I don't think maybe I'll be able to see the addition
7:34:28
because you don't have access to everything right Huh See Clodagh Express
7:34:39
this is free So we're running on eight north or 10 notes Right This cluster subsequently free You don't have to pay a single penny to Claudia But if something happens they won't help you like technical support is not there The commerce elevation is called Florida in their place So when you're installing the cluster it will ask you Do you want express We want in place If you say in the price they will ask Please browse and upload your license Keep that on The installation will proceed
7:35:08
actually So if you want to test or try something any off this there are people who are known Hundreds off north in the city should express addition Now the Horton but sandbox will not run in multiple awards It's only single mission set up that manager you helping start I'm not actually political because the sandboxes descent for one machine only It is no dissent for distributor like it is running only cannot be expanded even if you connect both off them They won't work because they're working as
7:35:36
independent sandboxes a virtual machine But you can download and stronger PC If you start it it will have everything How does high sparked everything just to learn like a single mission Production said that is totally different in Starling and all So that comes in the ATM inside That is why I am not talking much about it You may not be able to understand even if I see because there are a lot of other things apart from just a group or something you have to set up
7:36:05
a lot off repositories than machines Onda middling and racking And then the said that the same you will have its 100 missions One will become master on in every machine You will use the Florida manager or any tool down Lord Cadogan distribute So then you have to first what any true New Yorker Download the StoryCorps Clodagh manager Download it on one machine Then start Clodagh Manager It will ask you how many machines you have you say four machines It last give it 21 Master You said
7:36:36
this mission it will download and store do everything for So Cloudera manager your in store on one machine on then you started It'll immediately ask you how many machines who have you give the people depict the last which furniture My name nor richness a date and origin is so you're select this this this It will download Hadoop distributed install it set it up and give you the cluster That is how this cluster is created Action No no there is no limitation You can help those and
7:37:02
self missions but people don't know what actually mean If it is a production set of they go for enterprise licensing But even the express sedation is pretty stable on the express addition does not have some features off the Enterprise edition like you have something called rolling restarts and rolling upgrades For example let's say you want to restart
7:37:22
a machine in a cluster now usually when you re started it'll have done soft problems because your name nor detect this is down So sometimes in the Hadoop cluster you might want to restart all the machines You applied some patches or something And sometimes you want to upgrade the hurdle person you are running told or six you want to go to road seven good will happen But after that Avery machine should restart So if I do this in a production but it is very difficult So
7:37:49
in the express addition there is no way every mission will go down in the press edition You have something all rolling restart which means one by one it'll really start machines without affecting the cluster end off the day Very machine correctly Starter for your services will not be affected And you also have metering backups advanced backup metering solutions that are all available online in their place This edition will not help All the features will be the Lexx Park or whatever
7:38:14
tools you want Everything will be their commissioning You can commission on the fly You can act machines
7:38:20
Nothing will happen to emissions and get at it immediately They start to become part of a blister
7:38:25
It's called commissioning I don't know if I can show you here
7:38:30
I would not be able to add them but you go to this host minnow on There is something called commission State It is commissioned 11 Now that is a 11 machines running No I can't add because I don't have any permission But there will be an option called Adam Machine here Commissioner So I did not say is commissioned 11 So has often everything is commissioned I don't have any argument writes Otherwise we can just say click Add it lad on the plate And spark requires some
7:39:10
liberal for understanding Before we actually go to the hands on site we can really hands on spot for sure We'll be doing a lot of hands on But if I athletically start with okay write a program you will not understand what you're doing Right So that is why some slight Southern I'm not really a slights person but still some topics we have Horrendous Generally you can go to spark Right Um so first thing you need to understand this What is sparking from where Regard spot Right
7:39:39
So why people are so excited about spot So uh you're your previous session Train of light What is his name Hey was asking me that Can I learn Start I really want to learn spark That's the power of spark I'm not saying you and I want to learn machine learning I'm not saying that way but he won't deliver its part So he was asking me some doubts Like Can I get started with Spark I wanted the machine learning or spark
7:40:06
because everybody's excited Our spot so and all my trainings these days I don't spark most of majority of training sovereign spot eso There is a lot of extra excitement about this story And why is it why you spark is so popular Or is it some magic or something What spark is doing right So it's very easy to understand why it is so popular
7:40:29
somewhere in 2000 and nine Okay we had ah a project called I'm Blab in the University of Berkeley There is a research project called I'm Glad So this sampler projectors even life to the service that's projected Now you enlist your Buckley So there people were trying toe create a new cluster manager Do you remember Young Yeah So yarn is what you are a source manager in huddle on her do version two came in 2012
7:41:10
What you're seeing today So young came into the central like Yang came with her Do we're talking about 1009 So in 2009 there is no young we have ever learning How does one old harder version and there was no good cluster managers So some folks at the University of Berkeley they were trying to create a cluster manager Very similar PR Okay that there is no young okay on They call this project as missiles missiles
7:41:40
Mrs is They're projecting They have given for this project Okay And these guys were doing some r and D and at last they created the custom Manager likes it Very similar peeng my sources there Even now it's an Apache project very similar to yours So why young became so popular because yarn is coming by the fall with her duke If you download her do you will get young If I want to use use my source out install it separately but had hoped will run on top of my source Or so they're
7:42:07
the only difference away that your yarn layer will be handled by missiles Okay so Mrs is a very a famous project Even today some off The folks use it Actually not everybody use it on Missiles was created to those in mine in AbtLab So they created my so stand They said that it's a great cluster manager We're gonna use it blah blah blah right now in order to test the power off missiles Okay First they learned some apparatus programs Of course My produce waas what we
7:42:35
had originally and it was running fine So then that heart why don't we just use a different programming framework And they created something called Spot So spark was created actually toe test missiles Meaning my sources your cluster manager and they created a new programming framework court spark So you can write the program using spot It'll run on missiles on The only difference was that spot was completely in memory Execution like loose most off your lamb So if you have a
7:43:08
cluster manager on if there is a lot of ram getting used you can actually test right whether the cluster manager is good or back So they created this spark programming framework we're in If you write a spark program it will use majority of your ram for faster processing and they tested missiles
7:43:26
and missiles became a sexist Everybody was happy happy People never report much aboard spot they just created as a suburb project Okay But after any year like 2010 or 11 people thought that Hey this park is good because if you write a program in spark first thing people notice is that it is running faster compared with my produce actually why It is running faster So people thought that why don't we develop spark a bit more if people can bite the spark program and if it is
7:43:54
running faster should be great right on The development started on somewhere in 2012 I believe Spark became an Apache project So they contributed to Apache and said that you know we have created a new product It's now make it open source on this is a framework like may produce If you write a spark program instead off a map reduce program it will run faster than my pretties A place I thought Okay let's try that It became an Apache a top level project Okay From there everything
7:44:30
changed from 2012 So actually spark is quite new because it became widely popular somewhere in 2015 only and I started using sparks from photos and 16 only it is me They're the word right So I'm like Good because it began populism Very 15 69 started using spot Okay on Even in 2016 there were very less training since park This people would like excited but nobody knew Artis Park So we started using in 2016 All of us are an end off Like December 2016 It became the most active about your project in
7:45:09
the history And in 2018 17 18 Now this is the biggest contributor Apache project So in a span off 23 years it gained a lot of popularity Right on Now Spark It's like I said the most popular Apache project so far So a pageant has around 300 plus projects in that the number on the spot Guys off now right Also originally there was a spark version zero Then we got a spot Question one Now we are in spot question food or three This is the latest tuition on many companies Run something
7:45:44
Course spot 1.6 very stable The very stable worship That is the reason they're in it Probably I don't know I after 1.6 year $2 0 There is no one north seven After one north sits The next part question is major list to Lord zero Okay And we are on two or three right now If you go to the spot website it will save your aunt or three Currently two or three is not pretty stable
7:46:10
because it's the latest situation So not so stable about one Toto They're very very stable actually on they are learning spark to We're not learning spot one You're not missing anything because inspired We have some improvements on all the features of were not available into So that part is covered on in most of the trainings They will request you Please start It's part to I mean don't start that spark one meaning it's it's it's a start sold but people want to start that spark to So
7:46:41
the major version right now is two for two or three I can say that we are using
7:46:47
and something else happened So what actually made spark more popular is that in 2012 These guys the founder sore spot that is two guys they give a patron said That takes far can go But they were not so happy So what this guy's did badly They found a company called data Bricks
7:47:11
Data bricks so they found a sore spot They gave it to a passer because it become open source obviously but they also want to make money right so it is open source You can't make money So if you give something to Apache it you are actually contributing to the world Probably like for the good scenario If I don't give it up that's a like Microsoft Windows Microsoft Windows is not available with Apache right So it's only money making on So you are not getting any chance
7:47:38
to modify anything or are your contributions or anything If it is not an Apache the only contributions for commercial versions are done by the company so the rope will be limited Very simple The growth will be limited like you will have your own developers who can modify and or but others will not be a level So then it went to Apache Now what happened is that Apache has a spot pushing apart yes part that anybody can download That is 100% It's free Open source again The
7:48:09
spark is available with Cloudera Horton works map are sure that we know So they are using a claw direct lister that has spark installed So that is again same spark or Libby can use it But these guys in 2012 or found this and data breaks Okay company and they're sore purposes to settle spark Nothing else but this company so special because it is created by the founders of spark
7:48:34
the people who actually wrote the source code from this company So if I want only spark I can probably go toe these days because they're the best But people don't go here People go to cloud our Horton books because chances are very rare You want only spark You want other poor source like school people weren't flume You weren't nor data access nothing really school Poor flu The only spot right now one off The participant asked this question I'll get back to that in a moment But
7:49:03
still spark is an independent project which means it does not require her do But mostly you will see it on top of her Do Okay Sparky is actually an independent project which means I can download spot I have sparked running on my windows 10 laptop beautifully Works doesn't got anything like NPF Assize system It will work Okay so about people don't do it because the point is like right now we have a migration happening in g e So I was with the last week before coming here So there we have a
7:49:40
migration happened geese migrating to spark spark and her do so De already has a little cluster So why should this install sparks separately Right so they're already having a hard A cluster so they'll install spark on top off hurdle as an ecosystem told say it will start liberating It's your first for storage and then processing and or so monster off the times you will see spark on top off Hadoop But that does not mean it will run only on top of that you can run it
7:50:08
anywhere Practically I never show you in the slights Very very Condron However you can range so on and so forth Eso data plexus popular in that case on it is available almost everywhere Right So I think we'll run some slides to get you an idea and a disclaimer These slaves out from date of it So some of them will be their marketing slaves The best resource to learn spot OK so a couple of things which are you know I can teach you right A lot of people asked like Give me the
7:50:38
best book to learn spark And it's not just one book Then they will never come back to me So the best book to learn spark is something called spark There definitely a guide Don't buy you two Don't read it Just make a notch Spar there Definitely a guide This is the best book Don't even touch it Why I am saying this Okay you can probably buy our own mind This is written by Zetti Malaria and the Founder of Spark The guy who wrote the Sore scored brought the book and the book
7:51:11
is pretty much like the source court You won't understand anything you and I don't understand half off the book but very good So by it and keep it so people ask and you can show this a spark right Very difficult rendition But it has like accurate information because the guy who wrote spark is writing this book right So if you re leveled it rdd he will go to any level explain quantities So it's a bit difficult to understand Probably don't read it immediately Once you complete
7:51:39
the spark class and those are my them and then you start with this book you get give your very good idea to read There are some other books related to spark available for Gordon me but uh learning spot Right But that is good Learning spark is easy This is very tough actually
7:51:56
this is on This is version two These guys are so order the same spot of definitely guard First part question one which was very easy actually This is a bitch stuff I don't know So this is one place where you can Second is the documentation from
7:52:12
apart Tender Tavaris This is that I learned actually So you open up at your recommendation you open data bricks documentation They will give you a lot off ideas I mean basics at least if you want to learn Right Okay so that's learning spark so book you can keep I think it is not available in India Did anybody try buying it Is there It was in order before Is there
7:52:45
flip card spot The definitive I Amazon or so This is a matter Azahari and Bill Chambers There's other people who wrote spark Okay Um so it's good I mean I'm not against this but try if you want You can get how much it is Inflict car
7:53:08
It's costly right Really If you condone Lord So this is paper got back 2018 actually The night So it was late Esther addition It is right on and basic operations And ah a lot of things that they do But there are two books actually Maybe there are new books Better awareness How do the Definitely a guy then um there is one more book right for her Do oneness Hadoop The definitely a guide This is spot the definitely right You also have four hard doop doop The definitive guy Uh Then there is
7:53:51
one more word book for her Do hard open action I think
7:53:56
that's or sub workbook Ok how do books are a bit more easy to understand in the sense like this part book It's more Lamont oriented on this source Courtside Like how exactly Stuff work Hurdle books up a bit more easy
7:54:13
This is the slide with you to understand If you want to learn spark right on the slightest very important or so If you really want to learn spark why it is important So somewhere in 2400 King that's what The year there on with hard hope You got something called my produce You know where this map produced I don't have to teach right Still 2015 I can say you can see 2007 till 2015 What happened was that we're not The problems with hard was that you have my produce Fine No problem But
7:54:49
what my produce does is batch processing It is slow under this batch processing So people wonder toe try different different type of workloads in her do So some of them wonder toe right sequel queries on top off her do That is where you got haIf then we got impala then we got a presto Now here you have Ah drill on and many tools So around 10 plus different sequel to so there which can explore the data in her do And some other people wondered you know machine learning to be then So
7:55:24
if you in the earlier versions before spot when people want Toto machine learning they use something called Mahut apart from a hook Ah here against him I hold Summit
7:55:36
this issue of machine learning library But what is the problem Machine learning is what OK you guys can teach me right Mission learning is all about What do you honestly think Huh Are fine It is about iteration what you mean by attrition Like you're taking the same data and I trading over and over again right One off The things which my produce cannot do is attrition Because if you're item apparatus program the mapper will run producer will run It will push the later too hard disk
7:56:09
again The second iteration again You're to read it So when Mahut was doing machine learning the problem was that it was very very slow because internally deuces my produce on top of her job So my hope came But this became your a machine learning tool Okay on so interactive recovered for graph processing We got graph flag Kregel So these are the are tools which you use for graph processing on her do tradition had no spark or nothing Then people wanted to do are streaming data analysis
7:56:41
real time data That is where your storm All right so you see the problem right Over the period off eight or 10 years people started developing different different tools to handle different different workloads So if if today I goto ah Hadoop system on if I'm very new to her group and if I ask somebody what will I should learn then he will be confused because all these things you're to learn big high mojo they've sitting to start working with her Do Another problem is if let's
7:57:12
say one fine day you start learning Stone Storm is real time processing your the first learn storm Say FBI storms language then you're didn't started on top of her Do you should learn the integration that only starts working So learning all these different tools actually became a problem Okay on what happened in 2014 when spark came The biggest USB off spark is that it is a unified processing engine What these 50 plus stores on her dope are doing spark alone can do that is the rial
7:57:47
are Grandage of spot Now Many people confuse here because if you go to anybody and ask like wise somebody should learn spot they will say because it is faster
7:57:57
No speed is a byproduct of spot It is faster I know But the real reason why people are migrating to spark is that are these stores whatever they're doing spark alone can do so you don't have to learn like hundreds off tours You learn one tool one language and you can do everything But you need to have some basic idea about what you're going to go For example if you want to go machine learning Spark has something called Emma Lib Machine Learning Library that will allow
7:58:26
you to do machine learning But you should know what is machine learning that at least you can do other ways You cannot explode it but I don't have installed tend towards short when daters sparkling can do it So the primary reason why people are going for spark is this general unified engine
7:58:44
In my experience the second reason is speed Obviously it is faster I will tell you why it is faster how it s spelled faster than normal The third reason is ease off programming like you know fight or no So you can get started with the python in spark The language is fighting on If you want to write the sequel queries or streaming or machine learning for everything you can use fight on in spite That is not the case in the traditional world Big uses a different
7:59:14
language Storm uses a different language So every tool here actually used a different language Even for a graph processing in spar you can use fight them normal python So he's off Programming is another thing because I have struggled a lot because I have a couple off these stores I have worked as part of the project on then The major problem is that you will take some 34 months to learn the whole first Then when you come back that will be a facility and say Go and learn some other tools of its
7:59:44
spot That gap is gone So that is why every company want to migrate to spot No he is asking in spark So there's something or streaming we will come to that streaming data means you want to process time So I gave you an example Right So let's say uh credit card fraud detection I swiped my credit card on that has to be immediately captured by some system And it has to say whether it is throat or not This is a real time right now to implement this one solution I have a spot streaming in spot I can't
8:00:16
do it Okay but Sparv does something called microbe matching meaning it is not able to capture a singer Swipe as it is it will collect one second work data
8:00:27
so I cannot go beyond below one second One second more data But if I'm using something like storm it can even pick one swipe in that microsecond I say that is this way But I'm saying that is not so important I want second worth delay You can suffer at least Still you can manage its but most of the cases But yes spot supports a streaming So in this and this fragility of these guys are going the graph processing is start early gone We have sparked for that Most of the my hope
8:00:54
is gone I told you right days is is there Ah so one will be gone soon or more soon will be gone because stays this middle of my produce and spark What is this It makes your map It is faster That is what is this Basically so spark Mason more faster So it is in the middle So or mostly be going in couple off years Horton boxes The company was promoting days so probably they will keep it because they weren't market trail Right That is the reason Okay let's see what is spark Right So in spark you
8:01:27
have shared doing monitoring and distributing I will show you this practically
8:01:32
on I like this diagram a large because this will show you the spark architecture in a very nice way actually So how does it look to you Like if you look at this diagram where do you think it is Everybody saying it is all a system So the point is are the courthouse park So when you download spot you get something called Spark called Okay on that is the lowest layer off abstraction in spark where you can start programming radically and I will show you what to do there Andi you see
8:02:07
that It's a ram and hard This picture which came up which means spark and use both ram and hard disk And I'll tell you why this is very important later Usually a little systems can also use on These are the four languages currently supported in spark core programming So this is Scala This this iconic Scala this is by Thorn are and Java as off Now in Putri zero only four languages are supported So you should either learn any off this on You already know by turns So it makes sense You can
8:02:43
continue with pytorch I teach spark that Scala that is also a very the source Quarter spark is actually return This kind of that is the reason some off the folks want to learn Scotland with but now so earlier and spark 1.6 version and or if you write a pipe don't court It was nowhere There is one thing you would understand if you are inspired $1.6 And if you write your spark corden pie turn it will be slow on Skylark or will be twice faster It was like that because they didn't do
8:03:16
any optimization job Also slow slower than scholar Fastest one was Scala at that point in time Kind of Got it Still it'll be Ah bit slow because they didn't implement a certain libraries that was related toe java to support the optimization sin java So this is only in 1.6 and one door six is very rare in spark to what they have returned They have returned abstraction layer So that you write according any off this language is your programme will run at the same speed So it
8:03:52
doesn't matter You're writing by torn our style our job everything will be seems as off now we don't have to worry about it Secret is also there But secret is not a language Yes it I mean you can also write queries and all Using sequel Okay go for one more or bitch
8:04:10
Yeah So these are the So you will be thinking that if spark is able to do everything
8:04:16
right how it is able to do it So these are the libraries available in spark meaning When you down Lord spot you get something called coursework Fine on that is like my produce you can write but maybe I'm not a core programmer id only sequel I want Brexit where you use something called Spark Sequel in sparks equal very much like Highland Or you can create tables on there's something called data frame You can create them and query the data on the good news is they integrator
8:04:50
sparks equal with life by default So what will happen in a Hadoop cluster if you install spark Okay And if I open sparks equal I can read the data from my hive table So previously when you're running queries and high of it is very very slow
8:05:06
Now the same tables you can read in sparks equal and then quickly So these days hives just used us The stories Most of the processing will be done by your sparks equal because it is faster than really right Uh another library is Emily This is what I was saying Your machine learning libraries So whatever you are learning you can implement here I don't think deep learning and all are supporters Because for deep learning and all you will be using tensorflow
8:05:34
Do you have deep learning in syllabus Yeah Buy it If I did not come I'm saying your toe down Lord and important separately How Third party but your regression and are like normal stuff Like they're all supported by the fort So what algorithms you learn so far Which one Only regression So no no You buys and decision trees and blah blah blah So they're all coming in family anyway on graphics is the framework Very camera Present your data using a graph Have you ever done graph
8:06:10
processing No Right I have been okay Because why This is important
8:06:17
If you have started engineering computer science you would have learned graph theory right I mean I have learned I don't know was there in your syllabus There's something called graph theory So there we have some stuff like you have ages and relations No go to define like this Right So this will be me This will be you
8:06:41
This will be you one on and then you write relations between them I like you Probably You don't like me So you like someone else like this So I mean you have property graphs Very having are just okay which represents some property And then you have relations between them Okay on Then you can quickly the structure So for some type off the partially worked with a social media company in my novel projects not like Facebook and or they were a startup actually there their entire date
8:07:13
of us like this Even Facebook Right If you're download the data from Facebook it's very difficult to do it Okay you can't directly do it but there is a way to get the data Data means all the publicly available data They will give you the graph Api a performer before Matt and Mr Grabbed a pack comes this graph It doesn't come in j so on or extend millets graph ap moment because Facebook is actually using this graph structure pistol nor this paragraph I think they have
8:07:40
their own some system but the presentation is in the former for graph Actually so many times this is required This graph representation off data and spark can do this graphics library which can represent your later and then streaming later Streaming data eyes where I was saying you have real time data coming in on you want to process it real time on then make a decision based on that eso some off the queries you know if you write like this it will be much more efficient The
8:08:10
multi relational queries say for example
8:08:14
this guy like this guy this So if you want to find our patterns and relations that a presentation using graph is easy in some cases let me give you another example This airport data we had if you're having airports and flight data right So let's say at present airports like this bang you I said I don't know USA somebody And if I represent all the flights which are going in you know I can do this you think sequel or so But the graph Cory's will be much much more faster for me
8:08:45
because this directly structure right a graph structure is able It is easy to drivers in the graph a p I So probably between two airports You have like thousands of flights currently flying so you want to query them and track something's always for airport Date of use graph type of a system rather than a sequel will work but it will be slower if the data is really high even sparks people will be slower So this airport management like how many flights are flying in real time
8:09:14
and so on and support They use this graph FBI a lot
8:09:18
So only for certain use cases where you're having Except we're just and Lord off relations between them social media center that example Night So I want to go find out what you like in Facebook collapsing So I want I have to the Lord of comparison How many friends you have on what your friends actually like Do you actually like that So driver sing these queries and see if it is very difficult in graphic and easily Travers I can just say that these these these notes find
8:09:46
me these type off relations easily
8:09:49
So social media companies do use a lot of good affects actually even to interviews Twitter How many people are you following on How many were followed by you So this relations right these are all learned from a graph API is in Twitter They will not go toe so on And there isn't no sequel This is different Okay There is ah no sequel database called New York for J New York For Jake it's a no sequel Databases They store data in graph format So I mean that is a different use case that
8:10:24
it's for real time inquiries and all but this company actually uses only graph to store the later okay Graphics is different That comes part of spot new for Jason No secret Later Bees I had worked in a small project where they were using your 40 I didn't learn it actually but I have seen that people are actually using it Okay What do you most Some off you might understand for some of human or in distant So we even the one circle for the right So we are in one more orbit Okay
8:10:56
Now let me ask you what do you make out off this
8:11:01
Who is this No Is there anything special about this picture now Fine I mean uh eh So what is this This flamingo Okay What is the Flamingo doing Uh waiting for Hu BB created Right So what is the Fleming or Jewish huh Yeah you are very clearly Exactly This is called Standalone Mortal Spark I didn't create this light It is date every slight So if you have to blame blame them Exactly Flamingos standing on one leg right So to be more creative Okay for So these are different wars in which you
8:11:41
can't run spark basically OK so don't get confused with the name So the top one this single pc That asshole Local local mornings You are running it locally on one machine Okay You cannot expand to more than one machine and you use only for development and testing purpose on that's understood This smart is mess owes him this I cannot mess owes So originally spark was created round up off missiles I told you right so still If you want you can install a my source cluster right
8:12:17
And then it starts park on top off that it will run No problem Right Aunt This is standard on more So what is stand alone more I want in star sparking a cluster But I don't have young I don't have my soul's I don't have anything Then spot give you its own plaster manager that supports standard on
8:12:39
But this is very rare because I really it'll be only young And this is young Spark Orania Yeah So I really In most cases what you see is young this blue bar that is young because most of the organizations will already have a Hadoop cluster and young and it just makes sense to integrate spark On top of that another very important point A lot of people get confused That is I'm writing it Spark has
8:13:11
no storage There is nothing called stories and spark because it is an execution engine Like my operatives this maverick is having any stories No it released a later from her do process It then probably started back in her do night seven Our product spark also has no stories that it's no stories component in spot It is just an execution engine Miss Dreams You should provide the data somewhere So if you're running spark on Hadoop the date eyes are initially office If you're running spark or let's
8:13:48
say your PC Your PC file system has the later standalone norm esos whichever up to you to decide Okay if you're interested some extra knowledge spark an oversold unknown Coburn It is cool then it iss sparkling kubernetes is I think production really know You know what this Corbyn It is so okay For those who don't know probably these things might be useful in future The Resco benefits and rocks actually running somewhere So you have something called doctor Right Let us
8:14:20
something going Dhaka This is a video ordered forward Very old like 45 years I think I don't know why Dokan allows you to Do you know what is a virtual machine Right You know where this a VM So in your laptop if you create a VM
8:14:36
what will happen Let's I install of Indo Sveum The problem is that this Veum will use a lot of resources right Operating systems a reinstall everything So if my laptop has a G Baram I knew to you four GB ram only for the VM What docker does it allows you to create something called the container Okay Meaning
8:14:57
on one laptop I can run like 10 docker containers One the next one windows It will use the libraries it that are already available on my base operating system So it will not install a for real operating system That's what I'm saying So instead of giving four db Rampulla vm I can give one GB to a doctor or fight will imbue to a doctor on If it is only next It is much much good performance So Doctor became a big hit Because if you are a programmer you're not a Java program You want to test it You
8:15:28
want to test it on apple You want protest it only next windows How do you do all of this Right So you can just launched docker containers on your PC itself like 10 off them Run your court See whether it is work very easy Right Gulbuddin it This is the next level of DACA Really have something called container orchestration In a data center you are able to run my people containers like docker containers You can manage all of the music community if I am correct It is a Google project Okay so
8:15:57
it is a date that's in the orchestration So Doctor it's like you are running it on one machine Probably 10 machines Max So here one scene stark or vanity It become our data centre manager actually so you can give like hundreds and thousands of service to community and say that I want these many containers Lord doctor like unfairness it will launch them and manage them for you So cool Bernetti can become one off this instead off yarn or mestizos Or this thing could burn it A
8:16:27
can become your resource manager I can start spark And then I can say that I want to run spot program If I say he turned there it'll go to community It will launch containers for me to run the spot Court
8:16:38
Oh Dr coming this I a cloud cloud Everything is available on flower cloud of living supports So in cloud you have multiple options Either you can directly run Say I go to Amazon I create some machines myself What I can use these services like EMR e m on in our stick my produce I can go to Amazon
8:17:04
and say that Hey Amazon I need you to create a hard sparked Lester for me said 10 machines 100 machines in five minutes It likely it and give it to me And then I can run all my workload But these are like disposable clusters Once you complete your job you could believe them A razor paying money continuously Right So uh so this is on the cloud or so I mean this locally this implies with less clothes So it's available everywhere I don't know I have not extensively going toe over It is
8:17:38
But I think you can search because it is something which is
8:17:43
coming up in a big way
8:17:45
I think Google is the cool kubernetes school burn It tastes whatever you got So this is production grade container orchestration So the problem with your doctor and all is that it is very difficult to manage the containers with Corbyn it you can manage them like it is a cloud only I don't know the exact architecture of community Like whether they launched their own container or docker container but basically so it is all coming to an abstraction now Previously everything was
8:18:19
very clear For example you by yourself on used you can see the hard disk right When you install the operating system Now what Corbyn it essays that give me a data center Your own very
8:18:32
group of servers Give me a group of service like a data center on I will launch as many resources as you want You just tell me how many containers you want I will ensure from where I need to launch how I need to launch So like same like that So like an abstraction It is getting right So but probably so it's a are Coburn It is built upon 15 years of experience off running production Will yours that Google Yeah So originally this came from Google Google was already turning this not in
8:19:01
this name some other means So they now even Hadoop came from Google Right So same like that They're creating the school vanity I think spark Encore Bernetti is available That is why I was speaking about this spark on kubernetes Yeah very much of it With spot gold upward Zero you must have ah running kubernetes cluster blah blah blah Our dog I this doctor on the okay doctor immediacy So internally it is using Docker only Nothing else LA blah blah So it is supported So it is
8:19:42
not in the slide That's what I'm saying in the slide your own sequel Vanity sliders that he told her And if you pay attention to the slide I can show you one small change or to tell me what is a change Okay if you pay attention
8:19:59
Did you see that Is some change in this light What is it James No Somebody is on top of this like that is zookeeper Zookeeper Well you understand Tell you once more See It's going now What You come up You saw that So why did technically means is that all these three modes whether you're running on stand alone or my SOS or yon can be highly available using zookeeper Zookeeper supports your high availability if you like spark if you're running if one machine crashes or or
8:20:35
you know if all these states are maintained by zookeeper so zookeeper is integrated that spot So keep Arisa another Ah ha do because it's from tour Why it is used for communication between the mission's actually to put it very simply like in a very large cluster If one mission goes down then how do you know No it's not like a lord bands How can I explain I'll give you a simple example otherwise so keep her might become another big problem right And so cable isn't admitting
8:21:10
Mostly so you don't have to worry about it too much but still Do you remember I told you you can have ah active name Northern stand by name nor night So in a Hadoop cluster
8:21:24
you have to name North Right
8:21:30
This active this stand by Could it now What is the idea Only and only one should be working Inactive You've acted Clashes Decision Who will tell this exhibition that Sam connecting to the cluster How do I know who is active
8:21:50
I can connect with permission Right So this guy will have an I P address This guy will have an I P address One race elegant Keep on pinging Who is active currently How do I know who is active Who will tell me Right There is nobody to tell So one raise that I can keep on bringing both the machines to see who is alive who is not alive So like that There are many situations in which you're to understand which machine is alive or not alive So Vince will keeper What happens is very simple It is a
8:22:21
service So this active name nor will it disturb in Sochi But okay stand diving Or so the gist of it So keeper Now you just as a bookkeeper who is active it will tell this is active If this guy crashes
8:22:36
okay this guy will informs Okay But that I am the new active You can also keep also capable So it's a coordination mechanism
8:22:44
is okay participants for coordination and high availability What the slide means is that spot support So keep it so for talking between the machines and management and all it can use the keeper if possible If I available So this this is I can also keeper It didn't just come This is actually I can off Zookeeper you are asking I know because normally people are not much aware off zookeeper
8:23:13
No no it is It is for ah all the services Actually
8:23:17
let me show you So zookeeper iss for any service to coordinate So one example I gave us that How do you know Name noticed Ideally this would not automatically come up The zookeeper has been from the state to come up
8:23:33
If you look at the architecture okay If you implement high availability in her do normally what happens if I don't have so caper If this guy goes down after right a command to make this up that is actually of a stage off time right It should automatically come up so these days will be connected with zookeeper and there'll be a key politeness It's and so it's hard to say if this day goes down so people will know that this guy is gone after Let's say a second or two there is a
8:24:01
buy American country It would last this guy come up okay And I've dated a maternity care So basically this day holds met our data Who is doing what in the cluster Otherwise you see there are a lot of services and a hard Oh plus river that is active in Stemberg Not only name nor the source My nature
8:24:18
Your yard resource manager is an active and stand by If your ticket space that is service for Ganassi Space has a much strengthened by So if I goto heard of class very if I started asking everybody who is the master who is the slopes Very difficult So I go to Zoo keeper and this guy will have all this knowledge even a sport for spark You have active and stand by master Okay if you're installing spark independently on that can be coordinated The zoo keeper That is where the slightest thing So so
8:24:46
keeper by the Ford will be available in most off the clusters
8:24:51
Begin Men back now What So this orbit is over Spark can read from almost any fire system on that's right Afforded actually your first I don't know What is this Cube By the way this is as three Amazon local file system that is some other file system may be in any fire system basically So it supports a lot of fire systems within which we are not aware of right local file systems It can read from all these things Any no sequel late ofhis any arguments That is I think a very cool
8:25:30
feature because I will show you if you have a my secret levy spark and directly from the table Or if you have mongo DB or any no sequel databases it can read from the people and process the data outward Probably can store back it in tow That table right It also supports her dupe input formats Spark streaming can work with flu man Kafka
8:25:59
meaning you're doing real time streaming Right So the question is that how do you get the later
8:26:06
So this is always a challenge because what happens is that normally you will have a spark bluster Running so excited is a spark Lester How dope Lester Only very our spark install So let's say you have a sparse blaster learning I want to get to Terra later So actually spark and directly get the twitter later There is no problem It can directly complete luster and you can process it But the problem is if one off the machine who is receiving the this data crashes you will
8:26:37
lose the later for some time
8:26:39
the night in spot So what you can do in this architecture is that for high availability you can either use flew more Kafka So you can ask these guys to get the data for you
8:26:53
Froome or Kafka So flu miss point a point Delivery off data like I can configure a flu major We will see in the Lebanon It's gonna get the date after in picture book here Gaffe guys Something so you can get the later from here So even if your machines are not working data will come here right You're not losing the data So for that reliability you are using a flu man If you have a spark Plus there Right so that's fired cluster and I'm talking only about spot streaming Okay Not like
8:27:25
normal processing When you configure spark streaming one off the machine one off the machine will start working as something called a receiver Okay so what is this machine's job to get the data That's all it cares It is a normal day panel now The drawback of sparks streaming nor the drawback I can say if this machine crashes
8:27:50
If this mission crashes okay your stream will be lost off course It was switched to another machine but it may take some time so that let's say five seconds for 10 seconds You're losing the data right You're not storing it anywhere Got a So if I want to avoid this if this mission goes down then my stream will be gone If I want to avoid this what I can do I can say Kafka Hey get their daytime Kafka Okay From there I'll get to the spot Even if this mission crashes another
8:28:18
mission to come up the data will be available here So sparse streaming If you weren't reliability you have produced Kafka or flu that extreme You may not have the liability Okay So if we are covering spark streaming problem that architectures that I will speak the distributions
8:28:37
we already know And this has some started sticks on DA I think you should be aware off it Developers from 500 plus companies you have 3000 developers and accept likes to try right This is a slight some off you are asking Will some of the tours go extinct Yes your high ever
8:29:00
Now nobody's running a lot off haIf Cory's Everything is sparks equal queries thesis on Mahut for machine learning Everybody has migrated to spark and my live storm most of them itis box streaming Now on this slide just compares the different space and you can learn spot for example Instead off my produced Now most ofus abusing spar the
8:29:25
resource manager can be young or missiles It doesn't matter any off this on the stories level in her dopest STD fs this Paki honest Nor there no backyard wasa stories manager like in 2000 and nine long back when when originally missiles came the stories was handled by a system called a vacuum
8:29:49
Now this is renamed as a loose you I look she or lose you This project is still there with the storage Li like actually a first only distributor storage you can get So one Pirmin problem in spark is that you will try to bring the data from different places Like spark will be running on her do But maybe some of your data is in her do Some of them is in Libya Miss Some of them is in Cassandra So if you use normality if it's it's fine If you're using this document or now it is no less
8:30:20
a loose you It has a cashing live so it can speed up your processing by cashing the data that is on large Brandies
8:30:26
Not extensively used Because setting it up and all are messed actually Okay I have seen it once That time it was called accurately Norton Lucy Oh I think they renamed it If my if my memory serves me well back Uranus I lose you know spending I have to look into Mm Lucy Oh I look she or I look Say what are you going I don't know Ah I looks your formally Thank you Open source Memory speed Burchill Distributor stories Hi Don How many's technologies are solving here that actually this is like
8:31:10
uh So let's say you have earned soft data and you need a story Is layer in between cashing later to make it faster They knew you solution that is only useless on the waste everybody's using We actually offer something on Let's novelist years Ah in assessing SST cashing it looked so this costly or sit is costly but faster but only use cases that if you're processing a huge amount of data like uh we heard like Bankofamerica bankofamerica they had like turn sentence off data to process
8:31:43
like there are bits off data on it keeps on coming So storing them when I see a 1st 1st angry will be very slow end of it will be very slow So they push directly to Asia Europe this thing a Lucio because that's a necessity cashing Lee from there they read it the release faster Actually that is only use case you're seeing for a new CEO and this is another use case off it right You have own pra my stories and cloud stories and you're getting the data It comes to analysis
8:32:11
earlier From there you can start your computing So it is a story is later abstraction You can store data in multiple places Okay On my looks you will get it in one place from their reconstruct processing So it is like bringing the later into a cashing lee and then processing it Okay
8:32:31
Don't worry If you don't know loose yours It's perfectly fine It's not like Ana Mandate every component or something Okay Now can you tell me what is a drawback off my produce If there is any drawback what do you think So Sparky is replacing my produce right So what does that raw back off operators that you So
8:32:51
yes a map It I would put you poor assistant Shuffle output do persist than off course radios are or what you have tow Finally persist Right So that is why my produces very very slow So that is what is in the slide Actually written so on If I have to do and I trade you processing right I read the data and 10 times I have to process it Then I have a shared your the job spending map Pretty jobs have Children because it is very difficult to change them together in the
8:33:17
immediate rear and right or raise will be there right That is we're spark becomes very different because
8:33:27
on the sly just days that you can use who Zito share your all these jobs nothing else
8:33:33
spark There's something called in memory Processing on this is very confusing for many people The first thing you need to understand is that in memory processing means it will use the lamb If available doesn't mean that you always need to give ramp if RAM is provided
8:33:53
So let's say you want to process Benji be file and if you're cluster has been GB Free Ram it will read it in tow The ram on it will do all the calculation Only the final output It will push it into your hard disk Intermediate results are not stored on the hard disk Second point is that it will if what if Ram is not available Then it will start using hard to score So step by step it read it read whatever data that fits into the Ram process it Then again it'll really and process it like that
8:34:25
It has to go There is another way right if you don't have time Still it is faster than that produced okay Because from ground up they have designed the court off spark They are not mortifying my produce court or something Toe create spot on That is why here it stays 10 200 times faster than my produce Typical may produce So in memory processing we will see how it is faster and all Ah that in discus under that I that is Cassandra It just demonstrated you can store the resulting Cassandra from an
8:35:01
American real and store or so
8:35:04
ah these other distributors and applications
8:35:08
which use a spark so off course databases the major distributor and Horton works Clodagh or this guys are having uh you know spot as off Now on these all applications can use spot for processing
8:35:23
So mostly be 80% All right visualization tools So previously if I'm using something like what is it
8:35:32
meant A hobby I Okay toe visualize my later Why did a little fire a query toe My Hadoop cluster high will run the query and visualize it And now sparks equal will run quickly So it's much much faster So all the FBI towers and the ideal tours now used spark for moving the data and processing the data and so on and so forth
8:35:53
The slightest a bit old This is the 100 terabyte sort competition in 2014 So every year there is a sorting competition that will happen Even you can participate if you want So the uh the thing is that they will give you one terabyte off data Okay You have to start it Okay On the data they will already give you the format and or based on how your distorted is it indigenous sorting or cook Sort of What sort They will tell you It already give you the date hours off Sample data on
8:36:25
your right An algorithm to sort can be a Java program Any program recumbent on whoever source a later fastest will be That is the conclusion on in 2014 when this ran ah spark did it in 23 minutes Had open seven map produce and 72 minutes But look at the cluster size spark was running on 206 missions Apparatus was in 2100 missions That is a difference One by $10 machine Still it is faster because Lord of Ramus available everything was in memory So fasting pastor starting that
8:37:01
happened And even for one better by David a sorting and again spark became the winner there Okay
8:37:10
so you are having these notebooks
8:37:13
You know what's in our book right In spite on your site You know what It is likely on how to piece that So I mean the cord is actually returning in order book Okay Ah but we will use the shell Also put Understood because I don't like notebooks personally Okay Do you know where does it driver These are some things you required Otherwise you're gonna understand Spot That's weight
8:37:41
Where does that driver you know nor device driver Okay which drives the program That's actually correct right Right What is the guy So when you're writing a spark program
8:37:52
the program will have something called a driver Okay And this is the master off the program driver Is there mustard off the program on Then you have something called Executive was the slave So let's say I wrote a spark program in the Spar program Definitely will be a driver and then executed Without that the spark program cannot run I mean these are logical concepts I'm saying Okay now when you want to run that program Okay one way you can run is that you can say Run it locally I can say that
8:38:32
Hey I wrote a spark program Okay run the program local When I say local what will happen is that a JV and will be created like a jbm Okay both my driver executor Everything will be inside this So this is the local more of spark Java virtual machine Yeah Still a container will be created similar to this I'm JV amina General EMC Okay so a container will be a locator So this is like a young container Yeah So a container will be a locator in that both your driver and executed will
8:39:14
run So the lock tell more is not very efficient because we will get only one container and everything is running inside that So if you're submitting the spark court actually when you say you have a python code right it will read your logic and convert and learn it in surgery And after everything is insider Jay vehemently without that character even my apparatus programs
8:39:36
we wrote in job right Begin right by third according my pretties
8:39:41
So if I'm writing a pie Thorne I'm apparatus program If I run I will say submit the program Then I have to mention some jar files on those jar files will be in her do They will read your python court and convert that into a format which will run inside a JV um on executor inside a JV m Only because end off their day it is returning job What if it is sparking is in Scala writes Everything has to be inside and gave you so the local more The problem is you get only one container inside that
8:40:16
the driver and executed Everything will run Okay so this is only good for testing purpose right So you just wonder test a spark program or you want to learn spot You will normally say that case parking in the local more I write some court It just burns If you're in a class step This is where things becomes more interesting and this is a bit confusing also Okay so this is my heart Do cluster right
8:40:44
And I have four data nerds Imagine I have 48 and imagine I created a spark program Now the first question is that when you're running the spot program what are you analyzing Okay I'm analyzing a file What is the size off the file Right So let's imagine the file is in her dope Okay Just a simple use case Ah the finalists Here here
8:41:10
and here Just unexamined I'm somebody the portal size off the file
8:41:16
s Let's set 10 GB So I have a ngv filing her do that may be in blocks and or so that's understood on I want to process it using spark now And that is in a cluster Right So when I submit my program to a cluster I have to ask yarn for executors How many executors you want And what is their capacity Meaning I can either say Hey Young give me one executor And in that executor Queen DGB Ram I want no problem What will happen An executable Come here Is that executive second Dana Nothing broken Dana
8:42:00
on this has let's say two in DGB ran Okay I'm some four processor core or something on the new entire spark court will get executed inside this But normally people will not do this It was like one machine is executing everything So possibly what I will do is that I will ask you Hey Young give me Let's say four executors Just an example EMC give me four executors so young will give me four executors
8:42:28
right on each executor I tell Young give me five g b sexy This is five g b This by J B this five g b It's five GB Ram Ram in memory So tortas 20 but five GB Right on So imagine this on then there is a driver right So this is executor Imagine you have five notes in this Hadoop cluster So here my driver is running Okay So what would happen when you submit the program in one off the machine That driver will start running
8:43:02
on in the program You have asked Young give me four execute er's on each executor I want my GP Aramco processor core blah blah blah So young will launch 1234 Here on whatever course you have written this driver will push indoor for machines at the same time on each mission will process your This is how in a cluster spark is processing But this is not so easy as it looks like because you should know how much memory you need for processing right And how many executors
8:43:32
you want Will Wiljan actually give you those I cannot say that Give me 100 executors each with 100 She program the animal say that I don't have capacity I can't give you so normally wherever I have went for consulting and all If you write a spark program you will discuss with your admin Okay I want to write a spark program because inspired If you want to get the full performance it should be in memory
8:43:57
You need maximum ram and that is a challenge in every Hadoop cluster So you will go to our Hado Portman or spark at me and say that I have it on a program I need 100 GB rhyme in the cluster So he will tell you in your program you ask for let's say it and executed 10 GB each something like that right that you will configure as the number of execute er How much Merimee One on the new submit your program so young will launch these things on The driver will be running on a separate machine right
8:44:28
on whatever logic you have written driver will read and struck Pushing to all execute er's on the output normally comes back to the driver Whatever output you have in the driver you can write a logic Either store the output on her do or sorted in Cassandra or wherever you want That it's possible on this driver normally will be your app Master does that monster remember Have master from young
8:44:58
that is your so a lot of people asked What if this machine goes down My driver machine goes down Obviously if your driver mission goes down then the processing will be disturbed A spark program in or crash Okay In young you can configure application master restart time so restart So if this machine goes down it will restart your am On another machine application must earn another machine so that he can continue processing this application Master is in constant past with
8:45:28
your resource manager So there is an I d application i d where it will not tell what you have processed or what execution is going on currently It knows those things So you are a source manageable Launch one more application master that will become your spot Try with get the court It starts running from there So the so the driver is the master part off your program on these executors are where you push your court So depending on your cluster you can ask for the number
8:45:59
of executors on how much memory you want for each executed That is how it actually runs And there is something called application Master We start our timer your corn figure in yarn that we can say 35 settled tryto restarted the application master on the same machine Sometimes what happens This mission will nor crash your application Massively crash The driver process will crash so it'll restarted on the same machine Can we do to many Listen maybe the resource not available so
8:46:27
you can configure that timer And if it doesn't work then a little boat another machine and say Start from there You need to understand something called Are the B
8:46:47
So that is why our d d fundamental cities return So are the lists are the basic building blocks off spot So first thing you need to understand is that in spark your data is represented as something called RBD right Any later that you have If you want to process in spark the first step you'll do is that you create something called in rdd Our dearest times for wrestle aimed distributed data set It is like a variable What a point that you can see right So these slights will give you some idea
8:47:24
about our duty but also practically show what is in our lady
8:47:28
So just look at this picture This picture is very very good in understanding how spark is working Right So I have four blocks off data in her do So that is what is the presenter day Actually if it's right so you know four blocks off data is there Right on I want to process them now that is one file Even though it is divided into four blocks it's one single file I want to process And what is this date of some text data Imagine each block has like this era then a time stamp and a message again
8:48:03
Warning Sometimes stamina Mrs Some log file Imagine right Four blocks off data So we are assuming there on four data nodes and you want to process them now from here onwards it is going to confuse you Okay so I'm saying in our grants you will get confused OK but don't bury So where is the original Later In hard disk Right As blocks initially office So that you're to keep it in your mind Now I have all related on a program Imagine to process the later in the program The first step I
8:48:37
need to do is to create an rdd Okay So I have to say that create an our lady and are really is like a variable You can say or appoint So here the name off My Arditti is called Log lines are ready You can call it RUGU if you want I just say create Hey spot create something called Log lines are ready by reading the data from this file you can give the location Where is the file If I run this what is going to happen on Imagine in this case I have asked for four execute er's
8:49:12
in my spark program I said I want four executors for some reason So what will happen in each data and or 11 execute er will be launched in arterial situation like and I have 11 block on each date on or and I say Hey spark create an RD record Log lines are ready for me And if I hit and there all this data will be corporate into the rand or the main memory because our deary's represent your data in memory
8:49:41
Assuming Ram is available idea can be if I wrote really draw this picture This is actually our data bricks picture like But if I redraw this picture now don't think that always the file is in her room In my typical example I have the filing But there is one small thing here Let's say you have Ah six North and Hadoop cluster 12 three
8:50:10
four 56 That is a market action on block 12 But he for now the big question You want four executors Which missions were launched Executive who is launching executed
8:50:30
Young Young has no data locality of it Miss late Young doesn't know Where is your data Right in map reduce data locality is there because you're blocks are reciting here Let's say you're blocks are here Right So these are the four blocks I write them a produce program What is going to happen in my map Reduce framework It is returned That okay I need to have four mappers One more very important point you to remember in my produce If I'm processing the same data four J
8:51:03
Williams will get launched Got it One here One here One here One here And this will copy Here This will copy Here this will Korpi here This will copy here That is how your map parents Because can you process four blocks using one mapper Not possible You will have number off Mappers equals the number of blocks or number of input splits we say right Ideally you can say number off my bicycle to burn But then what is going to happen This block will process This will
8:51:34
go then Next Broccoli comical process
8:51:38
No no that's not possible Because this size off this map are this container is like one GB or the default er container sizes There This is your young container in my produce So then yarn is daunting Containers for your map Where I really have it launches is that it will launch 11 container for each block right on your block Sizes six or 1 28 and be by So in young Say things You will say that my container sizes one GB I want one GB container Right on What will happen One GB
8:52:08
one TV Bongbong before containers will be used It is very rare that manual you mentioned number off mappers You can't really do that because that will affect your performance
8:52:18
And also if you're launching one container only you can resize it So this is like a one GB container So there is a property called a set Spar daughter dynamic execution enable or disable that is too young So you are telling yarn I want eight executors
8:52:35
but don't kill everybody I'm a use it or may introduce it Settle keep it for you if you uh enable it that is where it will kill You are saying that if it isn't are getting used just kill them I don't want them So if your data is a small if you kill them but coming back to our bodies and our discussion I'm saying that in that example you have four blocks off data So that is this four blocks my heart of clusters Let's say six north Um you want to read this data and process it So
8:53:02
what do you say Yes I want to create an are really the name off the arteries called logline surgery or whatever are really is just like a pointer Okay on when you do that when that court runs what is going to happen is that you asked for four executors or so on Let's assume the executors are launched here so that is an executor here Okay that is one more guy here One more guy here and one more guy here so these blocks will be copied to here So now that is your data Your data is reciting inside
8:53:36
the ram off four machines or four executors This is called an rdd Because otherwise how do you call it Is your data units somewhere toe mention like like a variable So that is holding RPG
8:53:48
So now once the data is available in the ram that's another great eyes There you you have a representation your data So this entire log file is now called Log lines are leading also in spark There is something called partitions
8:54:05
Meaning Right now your data is lying as four blocks So we say that this is a four partition RTD So normally when you're reading from her do you will have blocks Each block will become a partition What if you are Know what reading from her Do your Sam reading from my local PC I can mention how many part patients I need for the later say on my windows laptop Okay I have ah one G b file If I simply read it it will come as a single one GB partition Okay if I want I can say that
8:54:41
Hey from my decision My windows laptop Okay I can say that a spark Read this one GB but create four partitions for me on this What is our grand days Each partition can go to an executed An executor Bill Process off partition not a file So are raise Your data has to be partitioned The more partitions you have the more parallelism you will get No that's just an example Anyway in the personal laptop I cannot launch multiple executors I'm saying you're reading from a person left
8:55:12
up and dumping into a Hadoop cluster Imagine it's not from a Hadoop file system or let's say you're reading from Cassandra Our typical example Cassandra is no Secret Service Cassandra doesn't have blocks or anything
8:55:26
Blocks are on Hadoop So if I get the data from Cassandra I read a table I will get one million rose right One million Rose I cannot give it an executor Right on The processing will be very slow So what I will do I have to have an idea about my data there Seven million date eyes one GV or something I will say that when I'm creating an rdd I can say that I take this data from Cassandra and divided into four partition Each partition will go to an executed An executive who can manage more than one
8:55:57
partition or so minimum burnt partitions should get depending low selects I have an executor with 20 ZB memory So let's say a partitions I says one GB it can manage 20 partitions The locality is not 100% is guaranteed When you launch an executed 100% it is an organic So Is this clear Because he also figure out the number of executors Okay first thing that depends on what is the size of your ego So in how do you normally this is not required because in her do what happens by the fault of
8:56:29
block is a partition Right But again that will make sense Because what is the block size in her job One to India can be 1 28 and B Is that block size Right So I have a file with that is divided into 10 blocks Sort or 10 size off the file is
8:56:54
this MP That is one point
8:56:58
gv I want to process one point G b file Now the question is that how many executors I need That depends on your cluster configuration I can't even process the scene A single executed I can ask Hey Yon give me one execute er g b ram or three JB Rhine Just a safe side I'm saying so Young will launch one executed They say three GB ran Okay on this has 8 10 partitions This file has 10 partitions 10 blocks over 10 partitions will come in that single over the second get process
8:57:32
a copy or isn't a late eyes initially A first anyway So in this case I'm asking for one container So this executor right So the executor you're launching Okay So you can ask how many executors you want He is not understanding what I'm seeing So this is my heart Oh please step
8:57:57
That's what it is not very effective I'm seeing you can do it It is not very effective So the more partitions you have more more processing speed He will get the same example if you take So I have four partitions like four executors I launched on Each is getting the order in tow One partition This will be faster I can also learn just single executor here Copy Or this four here it meant or this so fast and another very important point So these are all related to resource management But you should
8:58:30
also understand this stuff right An executor is a container right A J B M one processor core is required to manage one container
8:58:42
Meaning if this is a dual core machine how many maximum executives
8:58:48
you can launch Don't If you ask for eight executors what will happen doesn't work So an executor is a JV um to manage a jbm jbm Has a ramen CPU ran You can say I want a GB ram 10 GB aram fine but within that to manage ideally for a single JV in one processor core will be allocated minimum one processor core tonight
8:59:15
But again the question If I have 16 g b ram I mean 16 g b round for a container that will have a lot of partitions inside that maybe one CPU core is enough to run all of them The concern What is the processing power off your machine But you have to keep all these things when you're launching a cluster Well from inside I'm saying so to put it short Heidi here you don't help her do these many things When you're launching a spot program you should know the size off your data and how it should be
8:59:43
chunked So the idea is if the number of partitions are more
8:59:50
so let's say I have a file whatever the size maybe X sizes X right If the number of partitions arm or that means the number of splits are more I really So I'm saying that I want 10 partitions so I get 10 partitions Now if each partition can be inside an executor
9:00:07
10 executors can process that later sir Parallelism you get But if it is less that's I want only to partitions Then they're into exit You're pissed Then you know to executives there So I really you're gonna This is less what you say processing So the more partitions you have the more processing power where you can get So if you're lording looks a text file right So if I'm starting a text file I can say that create an rdd number of partition fight So why did it leak Willie split that text file
9:00:37
in tow five on then give it to 11 executed by the pint This unit fire executors to process it So if you go to young Yang will say I don't have enough so I'll give you a little executed So to executive get five partition one guy gets to an 11 3 so on each executives have importance A few court right settle maybe a bit slow So if you have enough resources battle of some works actually so usually would be doing the hard I will I will read all this picture so things will become clear to you I
9:01:08
think I just draw it in one man only So my point And this I just believe this Okay give me one moment No number of partitions So that's what I'm saying If you have a file this is your file right
9:01:22
This is your find on This file is divided into 10 blocks in her room Okay so I have work I want exactly right it But it's a 123 for 56 after 10 blocks you have Okay Now to answer your question right when I leave the later into spark it Lord automatically do something called partitioning on each block will become one partition by the fort and you cannot change it Okay By the fall of this one and most of the cases we keep it But what you can do So also I get This is one block this one
9:02:01
block So all this is red So this is my partition One partition toe partition three Right This is what partition four and help Partition and right are similar to import splits So you have 10 partition now you want to process it That is where the speed and all matters like to process it What you need you need Ram You need CPU pellet Right So there this partition will go This is hard disk blocks are there Is this hard disk Blocks are in your hard disk When I say I create a partition
9:02:41
Partition is Ingraham It won't just sit in the time it has to be inside an executor or a JV m by That is where the data should come Generally the processing should start right So then then you can decide I have n partitions Okay How many executors I need Can I ask for 10 Execute er's Yes you can do You can say I want 10 and execute er's Okay so you will get part Execute her one on here You will get executed then and each partition
9:03:18
will go to an execute Er part patients always go to execute So this p even will go to this Execute er now this executor I have asked for one GB rhyme for this executed I also have one processor court That means this partition can be managed by this executed saying your feet and will go here Okay This executor also have one process on one g b But the point this your young should I love you to give you and these things the same thing I can do in another way I asked for on leave to execute er's
9:04:00
example Okay I asked for two executors This executor has five g b This execute also have five g b This has one processor core This says one processor core on B one B two b three upto be five will come here B six b seven b and will come here now if you write a program Solar Sirota program I said filter the data
9:04:27
So my driver is here right This is what My driver in my driver I said filter the data There is a later in partition So this filter is your logic This logic will be pushed to this Execute er on this Feel terrible Apply here Here here here here Who will apply this process rent time together Right So one processor core I have using that one processor core Can I apparently run this Probably I have my teeth reading and all I can barely process this much amount of greater good at rice one
9:05:00
by ornately process it will take some time Depends on yes your processor core You can increase You can even say I want four course for the convener when you're launching this Ah executor I said one core by default right I can also say Hey Yon give me my executor is executor request forecourt But then the problem is if this is a quad core data on or only one executor can be done because they already took four course If you ask for more you will not get right If this is a quiet
9:05:30
coordinate and or if you lose one executor you cannot launch another executor already or taken Foreclose the same system there is mildly threatening So I think by the father can handle only two partitions Max
9:05:43
my reading means to threats right So in production how we do it is that one street designer replication We look at the data from where the data is coming but it already come partition or you have to partition because if you're reading from her do it already is partition like blocks And or But if you're reading from Cassandra Cassandra doesn't buy the full partition The later so you get like one big chunk So while reading we decided should be partition Yes we partition How maney 10
9:06:13
So 10 mins were to be the size of this much So have been executed Rest unit So then you go to Rodman and say that Can I launch a spark job with this money Bill aggregating he will say yes You will be getting the new launch them now Yes No it isn't No sequel later vis out off this But you can read from there You have connect person spot where you can breathe from Mark Cassandra So are you able to fall over I'm saying I mean some of your like he's stalking Greek and Latin night There is
9:06:44
also a very very you launch the job defied So if you don't want to mention or these stuff you just heard around my job so young will figure out the best way to launch Probably It may be the fastest today but it'll just execute your program
9:06:59
No I don't suppose it So now you're thinking that this guy is trying to fool us because he's saying you can ask poetry I will practically show you three executors running by launching You can actually do this It's not like some theory stuff we know launching us factually You can go to yard say now makes a curator execute her memory driver memory all you can ask and we'll give you those and you can see them Okay I go three execute out for executor or whatever you want right now If you have
9:07:27
understood this much I'm going to again confuse you Because if you ask for an executor right you say I want an executive and how much memory you want Executor you say I want 10 GB
9:07:41
by the far you get only 90% off it Okay because 10% is allocated for system cars it is a container It has to accept the stone calls and respond So you said 10 GBs 10 GBs for nor you'll It will not give you full 10 GB
9:07:58
10% days system military So what do you get Mine GB in this you will not get in this You get only 60% So ultimately some 54% days off the family you will get because it is a JV I'm right There is the rubbish collection J B A management or that request memory
9:08:20
So in reality if you look at a spark cluster if you ask for 10 g B container
9:08:26
you get around 5.8 G before our lady remaining our system will take Please keep it in your mind because this is an interview question So you were going around rgb randomly get there Say no it will not both eso by the file If you ask for 10 GB 10% a system called Celtic can Dana has to communicate with your operating system right and young So for these things it will reserve some memory Right So from 10 Debut became nine g b In this nine g b your jbm has to manage its
9:09:01
garbage collection on communication All this it will take another I think I don't know the exact number 30% of something it'll take It will drill down to total I think 54 off a six person days off this 10 GB only your RTD memory you will get to actually fit the partition Which means in 10 GB you get roughly around six debut to fit the partition so daunting that you get a JV and 10 size you can fit ngp effort partition No it doesn't work like that
9:09:35
So these are actually internal to spot But when we run we will understand I mean in production When we run then your calculation will be wrong right If you don't understand this your calculation will be wrong OK so this part partitions and ah parallelism So now if you look at the picture does it make more sense This picture So you have four blocks you ask for four containers for partitions Ideal case each is loaded and that's called it our duty right now The real question How
9:10:04
do you write a spark program By basically that is what you want Oh apart from RTD or lamb and all once you get the data used to analyze the data So how do I analyze the data in python Did you ah learn something called a higher order function function You know right
9:10:21
So normally when you write a python function you will say function than function Name blah blah blah That's how defender than you write the function and you will reuse the function Why are you creating a function You can garlic anything There is something going on anonymous function or disposable function Excitement to create a function I will use it only ones I don't want it anymore So you don't have really given name for the function or different You can create it on the flight that is called
9:10:49
Anonymous Function Scored Anonymous Function Anonymous
9:10:57
Okay in a spot programming inspired programming What we lose that we have something called higher order functions
9:11:05
It is something called Hyatt Order
9:11:11
Function What is the higher order function Let's say I have a function called ABC I can't pass another function toe This function that's called a higher order functions meaning this ABC is a function on Normally you will pass some parameter or something Some value right you say is this means this That is what you said But I can have a function and I can pass another function toe This function that's called a higher order functions So we will be passing
9:11:43
an anonymous functions here This lambda isn't an honest anonymous function I will show you the court It'll become better but inspired Basically what we do is that once you create and are really now you have the data Really Your data now I want to process the later how do you process the data You have something called transformations
9:12:05
So it is actually there in a spark website It's also a good that that you can look at spark Official website How do you go Spark daughter Patchy dot org Hey White is not showing
9:12:26
Yeah So spot lord Apache dot org is the official website of spark If you go to documentation it'll say latest releases two or three or zero flight on these are the older versions and all If you click on here you can see all the spark versions
9:12:42
You can see that 163 was the last spark One version Now they are on to 30 This is our version I mean latest portion on if you go to the documentation let's say latest release Imagine latest release Okay if you scroll down Okay Here you can see our DVD programming guy See
9:13:03
And if you click on this okay and scroll down a bit
9:13:11
this we will come back You can see a resilient distributed data set or are Really This is what we created like So we just created an rdd at least theoretically So once you create an rdd I will show you how to create it Okay you have are really operations So now my data is available Snr dearly What can I do with the artery
9:13:33
Right So that is where you can start writing your functions Okay Anonymous function on Yeah these are the transformations does This is what you need to understand So these are or transformations You can use them Spark map filter flat map that have many actually So if I want to filter my data I will just call this failure
9:13:58
Okay If I call Filter it will ask me What do you want me to filter within this bracket I will write my expression to filter That is how you figure your data map is another It is like for each Okay map I will call Mapple Ask me like what you want me to do So will write an expression within Mac What map has to perform So these are all higher order functions Map filter flat map These are all higher order functions actually So you do something called transformations in one our dearly
9:14:28
If you apply any off this functions it will create a new rdd That's for transformation that is Have you analyzed your later there So I want to filter moderate I will call the filter transformation
9:14:38
and are the These are immutable very important point Once you create an art dealer you cannot change it You can only create another one by applying some logic you can never edit and are really there immutable right If I go to my ppd Yeah so we have created of a log Lines are very fine builders we have understood on Then what I did probably I'm interested only in error Messages from this are very so you You know you see that a lot of later in four warning error I want only error
9:15:14
messages to filter So what I can do I can Korda fill their transformation Okay so I can call a filter transformation on then I can say that Hey spark
9:15:27
Okay match only error lines and give it to me and I show you how to light the logic This will produce another RBD and I can call it as errors are greedy This is the steps in which you write a spot program First you create Arditti on Now I want to do a filter I will call a filter on it will you know change whatever I mean little filter only ever messages on that I will store it as another are really now somebody was asking me what will happen to the memory right Settled Believe this are very
9:15:59
I mean if there is not enough memory So let's say this are really fit into the memory on Then you call this filter action it will filter whatever this record and this will be gone This is not required now because you have this right Because next processing to start from this step right I will have a show you what you can do okay or is in a little assuming the normal use case you call a function it creates Anna that are really on this side It is gone now Now this is your current data So if you
9:16:27
have any builder in young dynamic allocation then this will be gone
9:16:33
I told you right This is actually a partition And there is a executor running here Now this guy will be ideal because it doesn't know what it cannot predict That there'll be no data right
9:16:43
So that execute So that is one more problem Now your problem is you have four executors The second executor has no data to process because it's filter All the other that is not of this guy will be sitting idle So one data nor will be there when executed will be there That has no data It is simply sit idle so that it becomes a problem right How can I solve that problem It's actually very easy inspire There is a transformation called police Oil is it's a very common transformation And why do
9:17:14
you call coil is you pass a number It will reduce the number off partitions You can resize it because on you have to calculate this or in this example assume you know that See Now if you look at the data second partition is empty The partition has only one line That's also not so So I am assuming that I want a little so I can tell Spar take this rdd apply coil is keep order later but the little partition on O J V I want a little so it will just bring it into two partitions Now for
9:17:48
example just examine if you're having one terabyte off data right on you take a good sample of that data Let's say 10 GB 400 GB on you run it once So then you can understand You know after the filter OK you can call a collect action So let's say after the field that I want to see that I can do it So I know that orders only I'll order Let's say I don't know one GB rate or 10 GB later After the filter When I get the data it's only five That was half is reduced so I
9:18:19
can calculate Okay if I'm loading one terabyte off later I don't have to manage these many jbm So after this step I should call equities I can do it now The most important point surprising to is that let's say you wrote a spark poured in which you wrote these three lines create an rdd filter The rdd you said oil is and you say learn nothing will happen is the surprise Actually you wrote a spark or where you wrote all these three lines OK read from a sheer first filter error message Then
9:18:49
you set oil lease and you submit the job that last of nothing will happen in the cluster because all of this are lazy
9:18:57
There is something or lazy execution Meaning spark will not start execution
9:19:04
unless you call something called an action These are transformations Meaning You're changing the data But you are never saying that Show me the airport right You are saying re partition that filter the data Okay So what Where are you saying Show me the output You're not doing that So unless you call something Gordon Action Where you say that Jimmy The output Nothing will work to do that you can call an action call Collect Collect is a most common action in spark The new call collect Okay
9:19:36
you're telling spar that I want to see the final outward That city So collect will look at this cleaned are dirty So you are asking Give me the output of this cleaned already and spark will understand if I want clean darted I should have other sidedly The 500 hours How did I should read first It'll go to your hard disk Read the data Do all this stuff on Then show this is output I show this on your screen on then the entire pipeline is empty Spot never keeps your data in memory
9:20:05
once the processing is over Okay You got the result on your screen died this year So everything is going No rdd nothing Is there no contain a nonpartisan Nothing will be there Everything is gone
9:20:16
So only for that split second or this things happens Now if I want to know whether I should read Honoree partition What I should do is that I should go back here OK I will do a save off the Sarah you can Or sort of a statement or an hour Really When you say stable to save us a fight our next fight So I will say save as an action on this field that RTD and then I particularly the size like my orders in our latest 10 JB after the filter it is five GB So the process five g b I am launching
9:20:46
this many things so I can be right My court in that so collect will simply display output on the screen That is also an action course save as text file Okay I'm not uh just talking I'm showing step Can you see actions now His question vast But if I don't want to do it So he had a very good question So he's asking So the problem with this is if I run the spark or in a split second everything will happen And then I see the output on my screen No already nothing is there in the memory
9:21:17
because the this is called back or director as I click graph sparkle creator Dad a graph off execution and perform Stefan step poster placed a poor how result And then everything is gone again You run against Lord that's a different thing now So he had a question I will come to that so I will give So collect is an action where you return all the elements in the driver program Meaning you see that you can also say save aspects trial There's an action uh that receive
9:21:47
see save as text file If I say save as text file Foreign are really whatever data is there And rd it'll save us a text file so I can see that in her do poor wherever you're starting it right now Is this the only action available No I have Save s Cassandra Table or many other ways to say the later ideally X files say a video to just to see the later okay in the initial face off reading their data even my producer spark Everything will be saying because it is blocks
9:22:17
you are reading nor difference It makes once the data is available in memory Then their difference comes because here you see view did a filter then words that oil is on Then let's say something else or severely for doing all this The data is still in the member You're not writing anything but there is no intermediate yourself You're pushing too hard this once you call collect you can write 100 functions and then call collect or those manipulation will happen Final
9:22:44
the collective level this place that is a speed I know a very good question Only one action can be called it once Very good question Yeah so there are many actions or let's say collect and say West extra collect is one actions I cannot say Collect on sale at six point No So first is collect will run again If I say us text file again the whole operation has to start So you will be wondering Is that every I can improve it I will show you Ok so probably that is his question So um here I will
9:23:16
show you this So once you execute executor the Dag Dag is nothing but director A cyclist Grass
9:23:24
It is a fancy way of saying spark creates all the steps to be executed in a graph for men And I will practically show you that bag in the wrong spot job You can see that you can see partitions You can see there Dag You can see how many executives it is launching Everything is visible It's not just theory Ethereum thing but unless I speak about it and just show then you will say how this came Know how the partitions actually came You will be confused
9:23:51
and the driver collects the data
9:23:53
So this is the first step if you want Oh look at it once more No if you wonder to think that you're writing a map produces only aggregation so you that you can say some
9:24:05
or multiply or it's an aggregation function But because you get one key on all the values in the producer So what He wondered With all these values you can filter them so you can aggregate them or the water So if I have to write a figure that out right in the mapper side here I can so hear the final status stages the collect So before the collect I can do whatever I want I can filter on here That grand is is it's not just key value pair You can also work with key value person spot so I
9:24:32
can create a key value pair and similar from apparatus I can operate I can reduce and I can toward these things Yeah here he was asking a question I think I will answer that because if you look at the picture I have a very good picture to explain this Just seeing this picture same stuff I'm doing that is law glands Rdd errors are very clean orderly and finally I'm calling an action called Count Countess and other action So if you say count everything will work from
9:24:59
the beginning because it has to count Right So counteraction call And what happened It showed me that A cell test Five winner of fire lines in the artery No surprises Now I also want to call one more action on rt recourse safe to Cassandra The problem is if I call this action by plan is empty Spark has to start from a reading from the block creating the orginal artery then filtering then narrows all Then I can say to Cassandra Okay now one more thing then after I'm doing one more filter I
9:25:32
want only missed a just want from this data See Message one Mr is one message one I just get on living Stage one and then I say collect So I just want to see this But again what will happen It'll start from the up So if you look at this three actions you can see all three actions depend on this Arditti Everything is starting from here by that is where you can gash and rdd
9:25:59
It is possible to cash in our lady So when I say cashing Whatever data you have in that are dearie along with artillery will be cashed on Then there Can you cash You can cash in Graham You can cash in hard disk and you can also say RAM plus hard disk lake where everyone So there are multiple options to cash it on burns your cash and our lady Let's say you say count It will count The new says save to Cassandra It'll start from here because this rdd data is already cashed
9:26:31
on Don't think if you're cashing your cashing for one month once the spark program is over cashing is the leader So this is cashing is valued Only fill your program completes all the action Once all actions are over you exit everything is gone then you're not So I will show you tomorrow and go start a spot spark application If I'm a developer I will create something called spark context Object There's an object I create and that presents my program And I really when my program finishes
9:27:03
I'll kill that object If I kill that everything is going even cashing will be gone What was that cashing His valued Only the spirit of my existing program I'm not cashing it for a programmer right next month That is not possible During that execution time I want to speed it up See I created this circling our little part of it on I say count it will count and delete or this thing from memory right It is not in memory during the execution reason memory after the execution is all where nothing is
9:27:33
in memory Ah Then you should keep in the intermediate data in memory right You should tell a spark I will call an action Show me the output But this rdd whatever data you have don't delete from your pipeline Keep it because I don't really use it for some other action Then anything that's for cash So there is a command called un persist UN processes in action sort of transformation If you say un persistently believe it from market I'll show you how to cash tomorrow It's very
9:28:01
easy Otherwise if you kill your spark context object like your session is gone it'll automatically eso basically So you remember thinking So how do I start programming with spot I I want to write a script spot program So first thing you should learn how to create an RPG If you know how to create an our Lady then you can start with the these simper simper transformations map Now there is also a flip start off all of this The flip side is initially even spar came into
9:28:30
the industry Everybody was mad about our release transformations and I'll write a mad filter People were literally dying on writing these things OK but soon there was a problem with this See the problem is if you write your court using these transformations and actions there is no way spark can optimize your
9:28:54
cold Why am I saying this is that you say physical Okay in the filter you are saying that filter only error messages by unless Parker runs it it doesn't know what you're talking about on this These things do not have a very strict schema are the These are not having very strict schema So then you want to process structured it Alexa like CS Wi Fi Ok I want to really see us We file and I want control I want ski My everything are really is not a good way to process it you can process it but spark
9:29:29
internally will not be able to optimize your court That is where Sparks equal comes into picture for data frames We call it So Spark has a model corresponds sequel
9:29:39
and sparks equal is much smaller powerful on optimizing course park So if any right sparks equal you should know how to create a table and query using secrets and doctor receive quickly so that you see for two program on More than that it is much more optimized and core our duties because you're using all Lambda and Spark has no way to understand what lambda you're using unless it transit
9:30:03
why I wrote the Lambda cord Some beer Court Spark has nowhere to understand what is the meaning off this lambda unless it see the data But if it is a table I can't say filter on This column sparked knows what is that column for Dessert Data archive Where it is right it can avoid that calling by loading the data to optimize I'm saying right so if I write a sequel query it has a filter grew by Then join and something spark can breathe it Apply the scheme and understand what you're talking
9:30:31
about I will talk about in the date out Same plus every possible So any questions on core spot Like so far we have discussed if anything is not clear just asked me Okay The darn thing Ok so these are some of the common transformations Map flat map filter this other name off the transformations on this have some of the actions that we have right Yeah So anyway we will get stocked every the corps hands on tomorrow But we can look at some of the things So when you have a cluster so
9:31:04
I'm just connected to the cluster like now right Spark gives you a shell toe Work like the high shell if you remember So you started high even started typing So very similar toe that spark use you Something called a spark shell on the spark shell is available in Python Scala On our
9:31:27
there is no job Marshal Java assets does not have a shell functionality You're right using I d or something By now if you want your spark shell in scholar
9:31:39
we're going to do python But I wonder Scala Shell I can simply say spark shell Simply type spark iPhone shell Oh oh I think what A special is not working I think I have export apart to get started Spark Shell So let me try this Let's try the by turns shell by spark to the short Berg 99
9:32:15
So this job eight as off now Java it is supported Okay I think in the cluster they have disabled skeletal access Maybe I have exports some configuration to do that That's why I'm not able to open the scholarship for if you want a pie tone shell your type by spark to that's a command wipe I spark to Because we were inspired to this plaster has sparked one and spark to installed You can try this We will just I know some does some basics Not like our grand stuff but I don't know
9:32:51
It's taking this much time Usually the shed It starts very fast
9:32:57
where the shell is not starting Let me just check Yeah So scattered So very simple So this is your fight gunship Okay On what it say's when starting the shell it stays type help and blah blah blah Logging level for spark on and or its A spark version is Stuart toward or zero So on the cloud X lab yearning totaled or zero using by turn version toe 75 That's okay Okay Spark session available as spark I will talk about this later Later What is this Park session But it just stays Spark
9:33:37
session of a liberalized spark Okay And this is your spark shell
9:33:42
So now from here you can start creating are the lease and then writing transformations and everything that you want Ok now I will just see if I can show you this So if you type this comment by spark simply type I spark Okay Bash Dash help
9:34:07
Sorry Sorry by spark too Right
9:34:10
by spot Do hyphen Hyphen Help
9:34:16
Yeah So I want to show you these options Okay Just have a look at this So I'm just saying that I want to start a spark Shelly Okay on I need some help in starting that by So what it say's what are the options you can have So oneness Master you are so this We don't have to worry as off now Okay If I scroll down I can show you Can you see this driver memory So if a new start spark Shelly Okay our driver will be created Okay And then you can ask for how many executors you want right How much memory
9:34:56
you want for the driver by one question on then Here you see What is this executor memory
9:35:05
I told you right You can ask for how many executors you want and how much memory So by default I think it is one GB district in here The fart is one G b Okay but you can ask for execute her memory right on if I scroll down Ah see here Executor core How many processor cores you weren't for An executor Right And see here Driver court How maney processor Corey want for the driver number off executed default toe So if you're starting the spa action like this interactive shell you will
9:35:42
get two executors one driver each execute er has one GB ram
9:35:46
before sitting But using these arguments I can say how many executors I want and what is their memory and so on and so forth Now another important point Let me try this okay Just give me one moment I will just do a pie spark through Okay
9:36:06
Andi let me let me open my mail Because the spark or so has a you white on that you wave us send to me in my mail with us for every cluster It is different
9:36:35
signing Just check this the are more right thing Okay let me just check Okay So one of the you are writing it is not working Probably with worth now I don't know Yeah so can you I see this GL faculty Great learning This is me Okay
9:37:32
On it Say's 55 seconds before I started us Ah Shell There's actually showing you the history server That's what I'm wondering Ok 55 seconds before it's started Let me just go to the application So you have to actually see this You are but it is not showing Okay Maybe I think is giving lower the way Let me exit from here Okay I had a fresh So what you're looking at here is thes history server You are And this is not really useful to us Because once you start a spark shell on
9:38:47
then you accept it It really is play here You can see here by Spark Shell It was launched by me 1.8 minute I exit from here on Then you can see here But this is not really useful to us because this will come early in the history server That means want to exit from spot Okay so there is Ah really you I That port number is not working currently When you're running the spot it'll show there But I can show you something If so this session we I stopped Okay I didn't do anything I stopped the session
9:39:16
And if I goto this application this is the spark ey So here you can see the jobs and there are no jobs on If I expand this can you see executed One added that is executed to add it Because of the fault you are having to execute Er's on here You can see executors Onda There was one driver on to execute er so there was one driver active on two executors Regard as for this right And one more important point is that when you simply say fi spark to if you simply say pie spark to spark will
9:39:59
start in the local morning that means you are not talking to young or anything If you want to talk to young you have to say hyphen Hyphen Master Yeah
9:40:13
this is the option you have to use So whenever you're starting the spark shell otherwise if you start in the local more what will happen Driver Executor Everything will be in a single Jeriome and also that the sources will be located by your operating system nor by young in the local more young has no role in your local more on that is what for learning If you simply want to learn say Spice Part two and then you can just start writing your court Whatever I really
9:40:39
when you're launching the spark shell your to say price Part two on the New York has a hyphen hyphen master young meaning I want to resist it with the yarn on the lawn spot So what this request will go to young and young will allocate containers to you Now let's try to do one thing Let's try to launch it with our own parameters right So what We're there But we're the options we have by Spark do Nash Nash help So let's look at What are the options Let's say one more executed
9:41:13
I warned up Three executed Probably Right So what is option number Execute or hyphen Hyphen Numb executors
9:41:21
I will say three on a hyphen Hyphen executive intercourse Course Let it be there I want memory more So where is memory hyphen Hyphen Executor Memory Right before this one GV I want probably to GP something like that right So what I can do Oh I can simply go to my shell
9:41:44
by a spot Do I can't say master is Yann and I can say number off executors Probably four I don't know And then what is that Executor Memory Why This is okay executor memory I need to G That was easy See Tender Ah by this part does not support any obligation option Why They're saying that probably We're asking for more policy Let's see whether we can get three executors Okay so first we will try that Then we will see whether we can get more memory or something So like I'm saying that
9:42:37
a launch a spark shell The master this young Sorry Um not meme right Memory right here John Command Correct So now I ask for three executors and it started on We can see it here unless we exit So let me exit from here So it will go to the history CEverett Okay
9:43:07
a little fresh here This is what we started right Can you see Execute Er one ordered the weather Three added Can you see So three executors were given to me on You can actually go here Executors Can you see This is one driver These are execute terrorists right You get executors on here You see the memory right So how much memory did you ask Uh I think one of the four beasts selector right One cbb selector actually So for our dearly stories right now you're getting work 3 84.1 uh megabyte So
9:43:57
how this rule actually works Like I told around the roughly 50% off the memory will be taken on remaining it will give it to you will actually get around 600 being this but it display only this much because you are not loading any date as off Now it is simply running executor simply running You don't have any anything inside this light No For the Arditti That's what You don't have any data sets right When we load something we can actually see that memory will occupy How
9:44:34
much amount off are you Do you have a occupied Wait So 3 84 Right now it had a locator but there is no daytime Said that zero by releasing that But when you actually learned it's a 500 MB data show 500 be actually occupied in this because up to a maximum 600 MB Anything you will get because 40% anyway will be gone for your system cars and jbm like so Up to 605 100 something
9:45:04
50 or something You will see here So right now you can see only less memory because we are not actually doing anything with you So that is why it is simply displaying Ah you know this much Onda uh I exited from here So what if we try to change this number of executives for hyphen hyphen executor memory right My memory Doje there is a threshold value I think if it is more than one GB let's try this There is a threshold value or does this Ah it stays on error occurred while
9:45:43
calling on Java spark context required Executor memory is about maximum pra sure for this cluster So the cluster administrator has said that you can ask only maximum this much So what they say 2048 plus 3 84 MBI is about so it is saying that Max two Max You can ask only 2000 for the TV off memory But here what will happen when you launch and execute er is that initially led Locator 3 84 and be ok then we keep our GBs You know what is a access to be a locator So
9:46:25
they're saying that the total memories about the cluster administrator What he has said you cannot launch so we can try 1.5 g or something like that I think 1.5 gigabyte should work Really Let me check if that will work Okay lets it exit and we'll try this tomorrow down Very ok I just wanted to check whether the cluster is running Fine We will try this together tomorrow Down buddy
9:46:57
Okay So how do you mention
9:46:59
so see here Size must be specified as bites kilobytes Megabytes gigabytes
9:47:07
There are bites like so I think you cannot say 1.5 g That will not work Right So how do you mention Ah megabytes em right Did I say G
9:47:22
Yeah So why did I think
9:47:27
I've Gee only right Then why it doesn't work I m sure Work I'm sure Work I really right 1005 100 m It should be Hold number I think you cannot say 1.5 g Like against a one G or Koji Okay so now it launched on every likes it so But now what I'm showing you here is not correct because I'm showing you from the histories over like I'm starting spark shell then exit from there that I'm sure I knew that is the US That is not correct You show you should see Live right Sure If you're running
9:48:10
the job you should see lie that you is not working I was asking for that and I don't know why So these jobs will display anything which is running Why you're not seeing anything here Because we don't have any Arditti or any action Or if you have something you will see here Okay On stage a Sybil talk The stories will show you if you have cash been rdd If your cash and rdd it will come hear the stories section and me Armand is where it will show the job Libraries hand or
9:48:42
executor Press will show on the executors and driver that you have
9:48:46
looks jobs is there If you run a job it'll come like you said This thing don't collect It'll display here Like when you are on an actual job It will display here the diagonal stages It will show you in the job How many states are self there and how those stages are created We will see that tomorrow anyway Okay then you have stories on then You have environment on executors Like so environment will show you the different libraries and all which are part off spark If you go here now you can't
9:49:25
actually mess with me What if I want to see this from the shell Right So I'm just saying uh by spot I'm starting it again Okay on I don't have a GeoEye access now but there is a comment What happened
9:49:45
What it is saying spelling mistake
9:49:52
I am Sorry So there was a comment SC darch underscore Guard
9:50:01
That is a beard command which can show you all this from the shell So now you're seeing with a Jew I What if I don't have is you I can do in a sea dart Underscore corn dark Get All right
9:50:18
Perfect So I remember the comment I'm happy Okay so that is a Viet command Nobody remembers this Come and actually see right on the school corn Okay but the output this weird But you can see something from the output What is that You can see Can you see number of executing somewhere It will be a terrorist driver Memory right This is driver memory And against the master this yarn
9:50:44
somewhere number R c Here executor memory How much It has a locator 1005 100 You can also see number off executed Where do you see It's very difficult to read from this business I can't copy Paste expected please Not even I living me to copy Ha See here Haven't I'm copping I'm getting so this command is like a million dollar command I'm quite sure nobody will know this command Even I came to know very recently How can you see the spark and fix from the CIA Like a c dot Underscore corn
9:51:31
Forget already It's a short cut Actually I assure you that come in This is the camera Please make a note off this because sometimes we will start working in a cluster where they will not give you the way Then how do you know how many executives you launder What happened Your toe Look at this safe this week
9:51:50
So this will give you an idea about the number of execute er's everything Launch live live you're seeing like now you launched now this many of their a c dot underscore corn dog get our
9:52:02
Now The good news is we will try something in the show The bad news is you will actually try most off the things like this Jupiter notebook So how do you create an rdd from by spark import Sc blah blah blah Uh just hard out Clear
9:52:21
Okay so OK really Cup data rotator pattern eyes This brings us to the end of this Big Data Analytics tutorial Now before you guys sign off I'd like to inform that we have launched a completely free platform called us Create Learning Academy But you have access to free courses such as the iCloud And this is marketing Guys thank you very much Fighting this session on have a great learning
Here to help you learn
Sign in to keep a private conversation with your course tutor.
Ask me anything about the material…
Sign in to chat