Managing Data

Summary

Highlights

• Discussions on the challenges and strategies of handling data in cloud environments ☁️.
• Introduction to Google File System and Hadoop Distributed File System for large data handling 📂.
• Exploration of data management from both providers' and users' perspectives 🔄.
• The role of parallel databases in optimized cloud data operations ⏩.
• Illustration of relational database adaptation to cloud needs 📊.

Key Takeaways

• Managing data in the cloud involves heightened security and management challenges due to its remote nature 🌐.
• Conventional database models might not be suitable for large-scale operations in the cloud 🛠️.
• Google File System and Hadoop Distributed File System are pivotal for managing large data files efficiently 📂.
• Parallel processing and shared systems enhance performance in cloud data management ☁️.
• Amazon's Dynamo and BigTable illustrate advanced data storage strategies in distributed environments 🗄️.

Overview

Chapters

• 00:00 - 03:30: Introduction to Cloud Data Management The chapter provides an introduction to cloud data management, focusing on the aspects of managing data within cloud computing environments.
• 03:30 - 07:00: Data Challenges in Cloud Computing In the chapter titled 'Data Challenges in Cloud Computing,' the discussion focuses on the critical role of data within cloud environments. The key issue highlighted is the fact that both data and processing applications reside in external domains, which means they are executed at locations managed by third parties, posing unique challenges.
• 07:00 - 10:30: Database Management Techniques This chapter discusses the complexities and challenges involved in managing databases within cloud environments. The text notes the difficulties in ensuring security when data is hosted on virtual machines in the cloud, as these environments involve elements that are beyond direct control. Additionally, it addresses the concerns from a cloud provider's perspective, highlighting the issues of managing large volumes of data, maintaining data replicas, and ensuring that data is queryable. The discussion underlines both the provider's and user's challenges regarding security and data management in virtual settings.
• 10:30 - 15:00: Parallel Database Architectures The chapter discusses the limitations of conventional relational and object-oriented database models when applied to large-scale applications. It emphasizes that while these models may suffice for small experimental or database applications, they struggle to meet the demands of situations where there is significant data retrieval or high volumes of data. The need for parallel database architectures emerges as a solution to efficiently handle such extensive data processing requirements.
• 15:00 - 21:00: Google File System and Hadoop The chapter discusses the high efficiency of operations in systems like Google File System and Hadoop, which surpass normal operation speeds. It highlights the necessity of rethinking traditional methods due to these enhancements. The focus is on how these systems, originally developed for cloud applications, have emphasized the importance of parallel database access and execution. This includes improvements in read, write, and execute operations that have become increasingly prominent.
• 21:00 - 30:00: Big Data and Distributed Storage The chapter provides an overview of data management in the cloud, focusing on strategies and schemes used by ISPs. It discusses data management from a management perspective rather than a security standpoint.
• 30:00 - 41:00: Dynamo and Key-Value Stores The chapter begins by briefly mentioning relational databases, which are assumed to be familiar to the reader. It then transitions to discussing scalable databases and data services, which include key topics such as Google File System, Big Table, and MapReduce parallel programming paradigm. These technologies are highlighted as important for managing scalable data systems efficiently.
• 41:00 - 50:00: Cloud Data Store Technologies This chapter discusses the importance of scalability in cloud data store technologies. It highlights the need for infrastructures that can scale up and down effortlessly as per the demand, with minimal human or management intervention. This flexibility is crucial in effectively managing both application and data on cloud platforms.

Managing Data Transcription

• 00:00 - 00:30 hello so we will ah continue our discussion on cloud computing today we will discuss about some aspects of managing data in cloud right so as as we understand that in in cloud as
• 00:30 - 01:00 as we have discussed in our earlier lectures that in cloud one of the major aspect is the data because ah at the end of the day your data and even processing applications are in somebody elses ah domain right so they are in the in being executed at some somewhere
• 01:00 - 01:30 else which is beyond your direct control so it is virtually host in some virtual data a virtual machine somewhere in somewhere in the cloud so it becomes tricky ah to on the security point of view that we have discussed not only that if you look at from the other point of view so from the clouds provider point of view managing huge volume of data keeping their replicas and making them quarable and these becomes a again again a major issue
• 01:30 - 02:00 so all over conventional relational or ah object oriented model may not directly fit into the thing right so long you are doing on a on a small instances experimental experimental some database application or ah some small experimentations then its fine but when you have a large scale thing where huge amount of retried going on or the volume of data
• 02:00 - 02:30 is much much higher ah than the than the normal operations then then it is it is we need to look in a different way these are the things which which come in to not only for the cloud it was there a little earlier also like how this parallel database accesses parallel database execution read write execute operations can be done so those things become more more prominent
• 02:30 - 03:00 or a de facto mechanisms when we talk about ah in context of cloud so what we will try to do is more of a overview of how data can be managed in in cloud or what are the difference ah strategies or schemes people or this isps follows and it is not exactly the security point of view it is more of a management data management point of view right ok
• 03:00 - 03:30 so so we will we will talk about a little bit of relational database already known to you then what you known to do that scalable data bases or data services like one of the ah couple of things are important one is google file system big table and there is a map reduce parallel programming paradigm those are the things which comes in back to back when we are doing to the things so what we want to do ah when we were we are managing a anything
• 03:30 - 04:00 on on a cloud platform whether it is application or data we want to make it scalable scalable in the sense the it suites scale as as the requirement goes up so scale up scale down ah in a in a ubiquitous way or minimum interference from the or minimum human or management interference so that type of infrastructure we want to ah come up with right it is true for ah data
• 04:00 - 04:30 also so these are primarily suitable for large volume of massively parallel text processing right that is one of the major thing or it is suitable for environment ah say enterprise analytics right i want to i have a ah if we want to do analytics on a distributed data
• 04:30 - 05:00 stores right it it may be ah a chain of a a shopping or commercial staff or it may be a banking ah organization or financial any financial organization even it is something to do with large volume of other type of data like it metrological data it maybe climatological data something which need to be chant or has a distributed things i need to do some parallel
• 05:00 - 05:30 processing down the line where the actual effect comes into play if you have a simple database with a simple instant then you may not have gone to cloud for that right so it is it is may be a simple system or you buy a very ah a a work on it then the actual effect of cloud things are actual ah advantages of cloud you are not taking out so so we will see that similar to big table models there are google app engines datastore
• 05:30 - 06:00 amazon simple db which are which different provides provide in different flavour but the basic philosophy are same so if we look quickly look at the relational data base which is known to all of you ah or most of you ah users application programs interact with rdbms through sql right so it is the structured query language or sql by which i we interact
• 06:00 - 06:30 with the user programs etcetera so there is a ah relational database management parser which transforms queries into memory and disk label operations and optimize the execution time so in any query we need to optimize the execution time of the query right so if it is a large data base like you whether you do project before select join before or after select that makes a lot of difference though the query may be ah same ah the query
• 06:30 - 07:00 output will be same but the execution time ah may may vary to a great extent right like i have a huge two two data bases like r one say relational databases r one r two and i do some projection or or selection of some of the things right i select select a one a two and then ah do a then do the join whether i do the join before or after makes the things
• 07:00 - 07:30 like suppose ah if i do the select on r one the number of tuples come down from one million to say ah few thousands similarly for r two if i do a select on that right so then joining is much less costlier so whether you do the join first or it said that becomes a thing that is a database optimization problem nothing to do specifically for cloud but relational
• 07:30 - 08:00 database allows you to optimize those things disk space management layer ah this another property that stores data records on pages of contiguous memory block so that the ah disk movement is minimized pages are fetched from the disk into memory as requested state using pre fetching and page replacement policies so this is another aspects of the things like one is looking at that property making it more efficient in the query processing other aspect it make it more efficient in storage terms of things like nearby things ah if the
• 08:00 - 08:30 query requires the some five tables if they are nearby store then the access rate is high so database file system layer so previously we have seen the that rdbm parser then disk space management layer then database file system layer so it is independent of os file system its a separate file system so its in order to have full control on retaining or
• 08:30 - 09:00 realizing the page in the memory files used by the db or database may span multiple disk to handle large storages right so in other sense like if i dependent on the operating system for phase all those things then it is fine when your again database load is less if it is pretty large then the number of hope you take it text it becomes costly so what you need to do we need to do directly interact at the at the much lower level with
• 09:00 - 09:30 the with the hardware or the available resources and that exactly this database file system layer tries to immolate uses parallel io like we have heard about raid disk ah raid one raid two raid five raid six e n type of things arrays or multiple clusters so which keeps a redundant redundancy into the thing so the your ah this failure down time is much less
• 09:30 - 10:00 so that means is it is basically ah full failure proof implementation of the database so usually the databases storage as row oriented that is we had tuples and its a set of row of the same schema optimal for write oriented operation the transaction processing applications relational records stored in contiguous disk pages access through indexes primary key on
• 10:00 - 10:30 specific columns b plus tree is one of the favourite storage mechanisms for this sort of thing column oriented efficient for data warehouse workloads right so those who have gone through data warehouse is so it is a it is a ah high dimensional data huge volume of data and being collected and populated by different things so its not its more of a warehouse rather than a simple database so this is this column oriented storage are
• 10:30 - 11:00 more suitable for data warehouse type of loads aggregate of measures where rather than individual data it is more of the analysis on analytics come into play so it is aggregation of measure columns need to be performed based on the values of the dimension columns so we are not going to the data warehouse so it has a ah it has a different dimension tables and type of things and we need to the operations are more aggregate operations right we want
• 11:00 - 11:30 to do some sort of analysis and type of things so projection of a table is stored on as a stored on a dimension table dimension values in case of a column oriented require multiple indexes if different projection are to be indexed in a sorted order right so it is if it is a different different things because the organization may have different views for different type of data and need to be stored in that fashion so data storage techniques
• 11:30 - 12:00 as we have seen ah it is b plus tree or join indexes ah so if it is ah if it is so one is tow oriented other one is column oriented so this is row oriented data and this is column oriented and we need to have a join index which allows this data to be ah linked to one another so these all these we will get in a any any standard database book or in standard literature primarily as we are following that stuff enterprise cloud computing book
• 12:00 - 12:30 ah for this particular thing so thats why we have mentioned but this is a very standard operation and you can get in any standard books so if we look at the parallel database architectures so it is broadly divided into three aspects one is shared memory one is shared nothing another is shared disk right so i just see the picture fast then come back so this is a typical structure of the shared memory right
• 12:30 - 13:00 so these processors different processors shared the memory here it is a shared disk so different processors shared the ah disk ah here we have shared nothing so individual processor has individual disk so in case of a shared memory suitable for servers with multiple cpus so if there are multiple cpus so if there are multiple cpus memory address space is shared and managed by smp operating systems like ah the memory address this is shared among
• 13:00 - 13:30 these smps and schedule processors in parallel exploiting the processors so it it schedules small things so that means i have a shared memory space and i basically do a execution in a parallel mode so on the extreme other end is shared nothing so cluster independent servers with each of its having own disk space and connected by a network so at the with a back bone high speed network ah if any server shared its own disk space and then do the rest of the execution and if we look at that in between
• 13:30 - 14:00 the thing is the shared disk like its a hybrid architecture so to say independent server cluster storage through high speed network that can be nas or san and clusters are connected to storage data via standard ethernet fiber etcetera what we have shown here so it is it is a shared storage and these different processor access this so based on your application
• 14:00 - 14:30 type of parallelisms you need we can go for any of this structure so here ah we see that it is more ah this more efficient if the if the memory things are more compact where in the other a and we if the processors are individually working on separate data sets and there are machine to say then this could have been a advantage
• 14:30 - 15:00 so if we look at the advantages of parallel db of relational database if you dont want to put that what are the features of relational ah parallel database structures which is which is more advantages for parallel this sort of operations then the relational database efficient execution of sql query by exploiting multiple processors for shared nothing architecture tables partition and distributed across possessing table right so if it is if it is so happened
• 15:00 - 15:30 that i can partition the table and every the the the the data accountant in the table can be executed parallely they can be distributed in the different days and the processor can work that totally depends on your what is your working mechanisms out there so sql optimizer handle this distributed joint so whenever we need to do some join then we need to fall on the distribute ah your sql optimizer so ah distributed two phase commit
• 15:30 - 16:00 ah locking for transaction isolation between the processors ah so these are the some of the features fault tolerant like ah system failures handled by transferring control to standby system so i can have different standby system or some some with some protocol or some policy and then if there is a failure then i can shift that particular execution to a some of the standby system so that is possible in this sight of things and restoring
• 16:00 - 16:30 computation for data though these are the things which are more required for data warehouse type of applications so there are examples of databases capable of handling parallel processing traditional ah transaction processing things are oracle db two sql server data warehouse application are some of the vertica teradata netezza these
• 16:30 - 17:00 are the some of the things which are more of a data warehouse type of database now with these ah background or with these things in our in our store what we say we look at that cloud file system now as we understand it will not go something become totally we cannot through the whole thing out of the thing and start doing something new ah because this database has grown they are fault tolerant they are efficient we have raids and type
• 17:00 - 17:30 of things we need to exploit some of the things and put some more philosophy of which behind the cloud so one of the predominant thing is cloud file google file system was gfs and back to back we have a open source ah stuff called hdfs hadoop distributed file system so which is ah what we say some one to one mechanism set
• 17:30 - 18:00 google file system so google file system design to manage relatively large files ah using a very large distributed clusters of commodity servers connected by a high speed things so it is whether gfs or hdfs they are unable to work on very large data files which are distributed over this community servers typically ah some of the things are linux servers which are which can be which are interconnected through a very high speed line
• 18:00 - 18:30 so they can handle failure even during read write of individual files right during the read write operation if there is a failure it can handled fault tolerant ah it is definitely a necessity so if we have any that is any simple system term that p of system failure probability of system failure is one minus one by one minus one minus probability of
• 18:30 - 19:00 component failure to the power n so for if the n is pretty large then you can say that we can go for a that is the risk of this failure is minimum so supports parallel reads writes appends multiple multiple simultaneous client program so it is parallel read parallel write and update by the client program and we have hdfs that is hadoop distributed file system
• 19:00 - 19:30 which is open source implementation of gfs architecture available on amazon ec two cloud platform from so we have hdfs which is there so if we have a big picture so that how a typical gfs are there so there are some of the components are there is master or the name nodes ah master node in gfs or name node is hdfs and there are client applications
• 19:30 - 20:00 and we have different chunk server in case of gfs and data nodes in the case of hdfs in a typical cloud environment so single master controls the namespace so logically a single master is there which control the namespace so namespace is important because it gives us that how how there are stored how data can be referred its more of a it may modes
• 20:00 - 20:30 of a ah meta data sort of informations which is controlled by the ah master large files are broken into chunks in case of a gfs and block what we called in case of a hdfs stored on ah commodity server typically linux servers called chunk servers in gfs and data nodes in hdfs so replicated three times on different physical rack network segment so this chunk
• 20:30 - 21:00 so what we have we have the gfs or hdfs in the things below that we are having a chunk servers which are ah basically linux servers ah chunk server or data nodes in the things which are the main custodian of the data and they are the every data di is replicated ah on different three times at least three time on different physical rack and network segments
• 21:00 - 21:30 so if you look at the read operation in gfs ah client program sends the full path ah offset of a file to the master right where it wants to read or name node in case of hdfs so we will refer the gfs master node and which is back to back when we it is refer to the name node in hdfs master replies on meta data for one of the replicas of the chunk where these
• 21:30 - 22:00 data is found right client caches the meta data for faster access it reads the data from the designated chunk server so master from the master it gets that and gets the mirror ah this this ah meta data and from there it basically access this chunk server so for read operation any of these chunk server or replicated chunk server will do where write
• 22:00 - 22:30 append operation in gfs is little tricky ah client program sends a full path of file to the master gfs on name node hdfs right the master replies on the meta data for all replicas of the chunks where the data is found the client send data to be appended into the all chunk servers chunk server acknowledges the receipt of the data master designate one of
• 22:30 - 23:00 the chunk server as primary the primary chunks server appends its copy of the data into the chunk by by offset choosing an offset right so that that it do it appending appending can also be done beyond end of file to account for the multiple simultaneous right so this is a pretty interesting thing that even if you can have append end of ah eof ah beyond eof because there are simultaneous writers which are writing and it it basically consolidated
• 23:00 - 23:30 at later stage sends offset to the replica if all replica do not success in writing in the designated offset the client retries right so the repli ah the all offset so idea is that whenever i am looking for a data i need to know that ah for all the three replicas it should be at the same offset ideally so that i the read processed as there is no delay
• 23:30 - 24:00 in that things because once its calculates its directly access the other chunks ah on that offset right so fault tolerant in google file system the master maintains regular communication with the chunk server what we say heart beat messages ah sort of a are you alive type of thing and in case of a failure chunk server meta data is updated to ah reflect failure for failure of primary chunk server the master assigns
• 24:00 - 24:30 a new primary clients ah occasionally we will try to this failed we will try to this failed chunk server update their meta data from the master and retry so in case of a failure the chunk server meta data after a reflect the failure so the chunk server meta data says that there is a failure so the next time you do not allocate or like that and the for failure of the primary server itself the master assigns a new primary so it assigns a new primary
• 24:30 - 25:00 to work on the thing And update the clients occasionally we will try to try to this failed chunk server because it will be ah flagged right now another related stuff is big data or related concept of big data distributed structure storage five system build on gfs right so it is build it is a structure distributed structure storage file system it is build on ah gfs right so data is accessed by row key column key timestamp
• 25:00 - 25:30 so if you look at so it is it is ah it is a ah multiple instances are stored so there is a time key column key and ah of course say row key which says that where the data is there so ah in big table each column can store arbitrary name value pair in the form
• 25:30 - 26:00 of column family and label right so here if you can see that a these are column families and it is labelled and they store a name value ah pair set of possible column family is of a table is fixed ah when it is created so which are the different column families will be there so that is somewhat fix labels within a column family can be created dynamically and at any time so i can recreate the or create the table each big table cell row and column
• 26:00 - 26:30 can store multiple versus of the data in decreasing order of the time stamp so that means it is the chronology is meant it in that fashion so it is multiple persons are stored in a decreasing time stamp so again we see these things so there are different tables there are different tablets which are referred to this table and its a hierarchical structure and we have a master server it is primarily a registry or a meta data repository
• 26:30 - 27:00 so each table in big data is split into rangers called tablets each table is manage by tablet server so its stores each column family for a given row range in a separate distributed file called s s table so this type of management goes into the into play so that the ah my access rate end of the day the axis rate or ah will be pretty high so a single meta data
• 27:00 - 27:30 table is maintained by the maintained by the many ah meta data server the meta data itself can be very large so the meta data while storing this itself can be in that case it is again broken down into split into different tablets a root tablet points to the other meta data tablets so if the meta data are repository a pretty large it is again broken down into different ah tablets and there is a root tablet which
• 27:30 - 28:00 coordinates with the ah your meta data ah hm this tablets and one to real a one to ah emulate or realize that meta data services supports large parallel reads and inserts even simultaneously on the same table insertion done in sorted fashion requires more work can be ah more work ah than the simple append right there is two for a other databases also
• 28:00 - 28:30 because once you insert it is basically you need to push the data aside and create a insertion point where as in case of a append you are putting data at the ah end of the end of the that storage or data ah or the tables so dynamo ah it is developed by amazon that supports large volume or concurrent updates each of which can be small in size ah different
• 28:30 - 29:00 from big table supports bulk read and writes right end is so data model for dynamo it is a simple key value pair well suited for web based ah e commerce type of applications and not dependent underlining distributed file systems right for failure handling conflict resolution etcetera they they do it their self so this is ah typical architecture of
• 29:00 - 29:30 the dynamo where there are several virtual nodes ah and different physical nodes and they are ah logical connectivities are zone so if you look at the dynamo architecture so its a key value pair with arbitrary value ah key value pair with arbitrary arrays of bytes like it uses m d five generates a one twenty eight bit1hash table hash value so it basically try to map that were virtual node will be mapping to by using this has
• 29:30 - 30:00 function range of this has function is mapped as we are discussing that set of virtual nodes arrange in a ring type of thing the object is replicated as a primary virtual node as well as in minus one additional virtual nodes the n is the number of physical ah nodes so that any any the objectives ah replicated into the things each physical nodes are were managed is a number of virtual node at a distributed position on the ring so ah if you look at
• 30:00 - 30:30 that ah this physical node server they are basically linked with this virtual node server dynamo architecture from load balancing for transient failure network partition this can handle write request on object that executed at one of its virtual nodes right forward all the request to all other nodes it is executed one of the virtual node and say in all other all other nodes which have a replicas of the object so that means if
• 30:30 - 31:00 i am a object if it is replicated into another n minus one node so one is updated rest are ah being communicate so there is a quorum protocol that maintains eventual consistency of the replicas when a large number of concurrent reads and writes going on so this quorum tries to find out that which are the ah minimum level of ah replica will be there to handle this large ah rewrite of person so in next we are having this dynamo distributed object
• 31:00 - 31:30 version right creates a new version of the objects in his local time stamp created ah there are algo for ah column consistency so read operation are write operation w so read plus write operation should be ah greater than any of the system is quorum consistent there are overheads which will be coming ah there is a efficient write large number of replicas are to be read and if it is for a b c and read large number of ah large number
• 31:30 - 32:00 of replicas need to be written so these are the two things which are ah they are so it is implemented by different storage engines at node level berkley d b ah used by amazon and can be implemented to using my s q l and etcetera ah another the final concept what we are having is the data store google and amazon of a simple traditional key value pair database stores right google app engines data store in case of amazon what we say simple
• 32:00 - 32:30 d b all ah entities objects in the data store reside on in one big table right ah data store exploit column oriented storage right data store as i mean store data as a column families so unlike our rational traditional thing is a more of a row family or tuple base it is called column family so there are several advantages or several features or characteristics
• 32:30 - 33:00 like multiple index tables are used to support efficient query ah big table horizontally partitioned call sharded and across the disk whereas stored lexicographically in the key values other thing beside lexicographic sorting of the data enables there is a execution of prefix and range queries on key values entities are grouped for transactional purpose because
• 33:00 - 33:30 ah if there is if when we are having transaction so that is a set of entities which are accessed in a more frequent way and index table to support varied varieties of queries so we can have different indexes or different type of queries so it is not we should understand is not a simple a low a database its a large database so in order to do that i cannot churn the whole database so need to slice them appropriately so that based on the ah different variety
• 33:30 - 34:00 different queries it can be executed more efficiently and there are few more properties like automatically it creates indexes ah single property index or there is a kind index supports the efficient lookup queries of form select all type of things ah the configurable in ah indexes and there is a query execution indexes with highest selectivity is ah chosen
• 34:00 - 34:30 right so it is ah when we do the query execution so with this we will stop our discussion here so what we tried to discuss over over see is there different aspects ah we have the notion of our traditional databases which is established fault tolerant efficient and there are different mechanism to do that so we have we have also already this parallel execution things and ah its present so when we deal with a large volume of data in the
• 34:30 - 35:00 cloud which are likely to be there then whats are the different aspects we need to look at so we may not be able to follow the this column oriented or tuple oriented relational database we need to a sorry ah row oriented database we need to four for column oriented data base and there are different ah file system like g f s h d f s and over that this data store dynamo and your simple d v and those things what which are being ah implemented
• 35:00 - 35:30 by various ah inter ah cloud service providers c s ps ah for efficient storage ah access ah i mean read write execution of very very large databases thank you