CS5229 - Lecture 12 - Big Data Quality and Governance (2023)

Summary

Highlights

• Companies often use terms like Big Data and AI as buzzwords to attract attention. 🎯
• Mislabeled data leads to confusion in business meetings, highlighting the need for data governance. 🤯
• The lecture discusses the roles involved in data governance, including the Chief Data Officer (CDO). 🧑‍💼
• Big Data governance involves understanding data's volume, velocity, and variety, known as the three V's. 📏
• Modern governance models emphasize open, collaborative data use across an organization. 📡
• Privacy-related issues are discussed, emphasizing the protection of personal data. 🛡️
• The transition to networked governance allows for faster, more flexible data use. ⚡
• Nisansa de Silva emphasizes the importance of both technical and managerial skills in data governance. 💡

Key Takeaways

• Many companies claim to be data-driven but lack actual Big Data analytics capabilities. 📊
• Big Data quality is crucial for effective decision-making in businesses. 🏢
• Data chaos leads to mistrust and confusion, requiring structured governance to manage. 🔄
• Data quality involves cleaning, standardization, and profiling to ensure consistency and accuracy. ✅
• Data governance includes managing data architecture, storage, and security. 🔐
• The transition from hierarchical to networked data governance improves flexibility and democratization. 🚀
• Privacy preservation is vital, requiring techniques like anonymization to protect sensitive information. 🔒
• Innovative governance models balance control with engagement, improving data utility and trust. ⚖️

Overview

Chapters

• 00:00 - 06:00: Introduction to Big Data Quality and Governance This chapter serves as an introduction to the topic of Big Data Quality and Governance. It is the final lecture in a series on Big Data Analysis Technologies. The lecture, identified as lecture 12, takes a business perspective approach to exploring the subject. The transcript suggests it is a concise lecture but indicates that it contains a great deal of important information. More specific details on what follows in the transcript might include discussions on the significance of quality and governance in managing big data effectively within a business context, though the transcript preview is limited.
• 06:00 - 12:00: Challenges in Big Data Utilization The chapter begins by emphasizing the importance of understanding different concepts related to Big Data. It highlights the various challenges faced when utilizing Big Data. Specific challenges are not detailed in the provided text, but the focus is set on addressing and understanding these challenges.
• 12:00 - 18:00: Data Quality Management The chapter discusses the common claim among companies about being data-driven and utilizing Big Data, while in reality, only a small percentage actually employ Big Data analytics effectively.
• 18:00 - 24:00: Understanding Data Governance The chapter titled 'Understanding Data Governance' addresses the common use of buzzwords like 'big data' and 'machine learning' by companies as a strategy to appeal to employees and clients. However, the chapter suggests these terms are often misrepresentations of reality. It aims to clarify the discrepancies between the promises made by companies and the actual events that occur in the field. The transcript hints at a deeper examination of the industry's practices, inviting readers to explore what truly happens behind the scenes. The chapter exposes the gap between the enticing language used by corporations and the realities of their implementations.
• 24:00 - 30:00: The Role of Data Governance in Business The chapter discusses the challenges faced in business meetings when data is collected manually from various sources such as Finance ID. It highlights the frequent disagreements that arise when presenting analyzed data, where individuals dispute the data's accuracy, leading to conflicts and differing conclusions. The need for robust data governance to ensure consistent, reliable data that aligns stakeholders is emphasized as a solution to these challenges.
• 30:00 - 36:00: Strategies for Data Quality Assurance Data quality assurance requires a unified approach to data transfer, ensuring a common understanding and completeness across the dataset.
• 36:00 - 42:00: Privacy in Data Governance Data chaos can lead to fear, uncertainty, and doubt (FUD) within an organization. Good quality data and a proper structure for managing data are necessary to avoid these issues.
• 42:00 - 48:00: Comparison of Data Governance Systems The chapter explores the topic of data governance systems and begins with a common scenario faced by data scientists and engineers. It highlights the typical concerns expressed by stakeholders such as lack of trust in data numbers, and questions regarding data analytical processes. The chapter sets the stage for discussing how robust data governance can address these challenges by ensuring data accuracy and reliability.
• 48:00 - 50:00: Conclusion and References The chapter 'Conclusion and References' discusses the reactions and queries that may arise from peers and superiors when presented with a report containing complex conclusions and data. It touches on the confusion and curiosity about whether such information can be shared or used, and seeks to address the comprehension gap between technical details and the understanding of peers, bosses, and subordinates.

CS5229 - Lecture 12 - Big Data Quality and Governance (2023) Transcription

• 00:00 - 00:30 welcome to the last lecture of Big Data analysis Technologies this is lecture 12. we will be talking about Big Data quality and governments this is a very business perspective topic it's a rather show to because well but it is a lot of things that are
• 00:30 - 01:00 uh it'll be remembered and I hope that uh you learn some other concept related to Big Data by now and you know the challenges we have with the big data right so let's get on with it so
• 01:00 - 01:30 the first thing is is her so everybody likes to say that they are dead women a lot of companies say that they are data driven we are doing big data um but in reality only a very small um percentage of them actually use Big Data oh analytic sense and so forth right so uh
• 01:30 - 02:00 big data and machine learning these are buzzwords that these companies use tries to use to attract both both employees and plants and most of the time these are lies so let's see how this happens and what actually happens uh typically you spend
• 02:00 - 02:30 hours and hours preparing for a meeting you collect the data manually from Finance ID and so forth you do the analysis you prepare the insights and you go and present but most of the time it would be just result in people fighting over saying no that is not what my data says but that is not what the other person's theater said and that is those are not the conclusions that I draw and so and so forth so what
• 02:30 - 03:00 you need is a unified transfer the data common understanding and complete phrase away beyond the distance that you are taking and transparent data ownership who owns this data is a important question um there is to be answered even between the single organization um to avoid conflicts this is where data governance and data quality comes into
• 03:00 - 03:30 play you have to have good quality data and you need to have proper structure in managing that data yeah so data chaos would result in something we call data fud so fud is just a fancy way of saying fear uncertainty and doubt so basically if you have data it is very
• 03:30 - 04:00 common if you are already working as a data scientist or a data engineer at your business you might have seen you might have heard these comments I don't trust these numbers uh oh that would be one of your bosses would say I don't trust Spa how did you even come to this conclusion oh to your subordinates or your
• 04:00 - 04:30 peers would say why are you showing this showing me this am I even allowed to see this am I allowed to use this now can I use these uh conclusions and again things your bosses who may not even be computer science people would be saying I don't understand this report what are all these complex numbers and then again your peers your subordinates and even your superiors might be interested in where did you get
• 04:30 - 05:00 this data how can I also get this data and also the same people who would say this would be like how can you help me understand this data but you have your own job right so you can't waste time teaching people how to understand data so maybe there is somebody else so all of these all of these is what we call data chaos
• 05:00 - 05:30 now um data instructor infrastructure is about exploding volume velocity and velocity of data we have seen this many many times over the presentations and the solution for that is managing data complex and on the other hand the data consumers the business has a increased Reliance on analytics and
• 05:30 - 06:00 Regulatory reporting they believe what the data says so if they are taking business decisions on these data they do need to have so we do need to have trusted data as a business dependence all right so all together what we need is data calibration understanding Discovery and
• 06:00 - 06:30 first so the two things that we are talking about today are data quality and data governance so data quality works on data cleaning standardization matching profile this is the more technical side of it data governance on the other hand has to do with BP process technology and this
• 06:30 - 07:00 is the more management side of the topic pretty bone through data quality data cleaning standardization and so on so forth in your lectures with uh so we are specifically only looking at things that are targeted at data quality itself um therefore the weight of data quality
• 07:00 - 07:30 would be a little less and Based on data governance would be led more so let's look at data quality first what is data point quality is how much it would fit into the data driven decision making that you are supposed to do and regardless of whether bits or not then you need to look at how
• 07:30 - 08:00 we could meet the expectations of your consumers and how well it represents the objects evens and Concepts that it is intended to represent right uh so on one hand it has to be well fitting so what you are doing in your business and on the other hand it
• 08:00 - 08:30 should be actually useful for the people you are doing the business for and it should actually be relevant to your business so uh this is a amalgamation of three concepts that should come together also um measuring data quality level can help the organization identify data errors
• 08:30 - 09:00 that need to be resolved and and assess whether data in the ID systems are fit to serve it's intended purpose okay so what are data qualitations so you collect data from different sources and when they come together to a single system there might be issues uh uh
• 09:00 - 09:30 this might be oh the the item might be wrong uh there might be indecisiveness in data as in you don't know what this is so you can't even derive it from these data because Samir from Hyderabad because Hyderabad is a city in both of these countries so you can't even derive it here and sometimes you have no idea because there are
• 09:30 - 10:00 londons in there is a London in England and there are many londons in United States if you didn't know now you know so you can't even derive these values so you are at a issue so again uh States so state is the thing in the United States it's in the name United States so California is a state India being a federal country also have
• 10:00 - 10:30 States but if you take something like Sri Lanka or China we would have provinces right Uh Oh Canada we would have provinces and that would be a okay I think yes I might be wrong about Canada but uh China I'm pretty sure actually like I am
• 10:30 - 11:00 difficult oh so how are we going to measure uh just like we mentioned earlier the purpose of data quality is to check whether the data feeds safety business purpose of the uh of your organization but to understand this you need to measure Bitcoin so what we measure is using uh data called Dimensions uh criteria
• 11:00 - 11:30 that we discussed and data profiling so data profiling is to learn it's like learning metadata about something you are trying to learn uh about the data set uh you can extract the effect of analysis and matrices based on the data set that you have and if it is huge you can speak it into small jobs not randomly you can maybe divide by a
• 11:30 - 12:00 certain attribute and analyze maybe geographical information maybe you can divide right from um or temporarily Monday data might be different from this area who knows it depends on your application or maybe Columbus data is different from candidate who knows again depends on your data so you are the person who's handling data you are the person who is familiar with
• 12:00 - 12:30 your business process so you are the person who should decide how you do this Decay classification uh how would you do this clustering into Champs in you trying to analyze this data and the objective as always is to analyze evaluate the data set against the business process
• 12:30 - 13:00 out so what are the data quality dimensions we mentioned earlier we have completeness so data can be complete even if some of the optional data is missing so just because the word says completeness do not think that it means everything should be there okay and
• 13:00 - 13:30 let's say if you have something like uh first name middle name last name and if the full name is not provided why you can definitely deriving from those variables and inside the the question of completeness is not about filling all the slots but it is about whether this data is usable
• 13:30 - 14:00 whether it has all the necessary information necessary attribution so that we can use it in our business use case all right then we have uniqueness real world compared to the number of Records so if you have duplicates
• 14:00 - 14:30 you should be getting rid of a step and you can get the uniqueness as a percentage of things that exits of all the data set all the data points that you have next you have timeliness
• 14:30 - 15:00 is this a reality in this point of time as in if you are going to take certain data into uh let's say it is medical day right so Medical Data from these two years uh
• 15:00 - 15:30 2019 to 2022 so these three or so years would not be so good to predict hopefully the medical situation in the world in let's say 2055 because hopefully there won't be this Corona pandemic by then right and uh also let's say statistics of uh human life it's been expected see
• 15:30 - 16:00 from 1938 to 1945 might not be very accurate for any other time period because that is the time of the World War II and there is a lot of people who were dying because of that so uh the relevancy of timeliness of data is very important on the quality so it is not saying that those data is
• 16:00 - 16:30 stained and it is no longer relevant they are very valid data for that time period but no longer for the time period that you are going to predict oh be appropriate in and then we have are the is the data valid um B data length data types uh the range minimums maximums
• 16:30 - 17:00 just say uh confirmed to the uh the syntax that we are looking for so remember that previous example of uh parts of the name that is there what if our system does need uh surname first name middle name format and cannot work with the full name so in that case
• 17:00 - 17:30 you do need to make sure that your system does contain information about those of the features that are expected all right next we have accuracy the degree to which data is secured to the real world data that you will be seeing so how much uh this data
• 17:30 - 18:00 reflects the so I don't want to use the uh the words training data and test data but you can think about it in that perspective how much would our the data that we have to take the decisions would reflect the examples that we would see in the real world
• 18:00 - 18:30 so let's say uh if you have data collected from United States and your model is going to be running in the United States um you should not uh format your data you you should not format your dates in this data in the way that you would use in the commander right so
• 18:30 - 19:00 they use uh month day year system we use system so we should not do payment here formatting in dates if we are going to uh both collect data and work oh and uh worker system in the United States you should make sure that it works with the assistant because uh it is something
• 19:00 - 19:30 that can cause problems right because as long as the the date is less than 12 it is it can be processed so you can have one six 2022 and in one case it is January 6th and in the other case it is June 1st so a very dangerous scenario uh
• 19:30 - 20:00 similar thing happened with the uh one of the NASA probes that was sent to Mars some people used meters and other group used Pinchers and it crashed all right the last part is consistency there should be the data so you are very familiar with the word consistency because we talked about this uh in
• 20:00 - 20:30 nursing cancer and so forth but here it is a different perspective that we are looking at consistency is the absence of difference when comparing two or more representations of the of a thing against a definition so if they are going to behave in the same manner they should have the same out of the same movement or same
• 20:30 - 21:00 situation in our data all right so here is a whole mess of data but you can see uh how each of these things how each of these uh data quality aspects a question
• 21:00 - 21:30 in this data set so let's look at some completeness this data is missing so every time that there is a range it means completeness is being suffered right confirm this is not confirming to
• 21:30 - 22:00 the knee right when did he say a fraction or this is not confirming to the ranges of the values oh yeah and oh yes our consistency Euro used in United States no Great Britain pounds used in United
• 22:00 - 22:30 States again no you can see another one so here you might be able to construct this from this that is what is shown here so GB maybe you can say GBP here right also you can see issues with Great Britain and United Kingdom over here
• 22:30 - 23:00 and here is a problem of duplication you can see the top here because I mean it's a little large and the issues with the integrity you have this data line and this data line and then issues with accuracy uh here you would not know why it is an issue with technology application but you can see these are things that
• 23:00 - 23:30 can occur uh in these are all data point issues that you would see when you are doing a project right so this is a very quick slide on the life cycle of uh data quality you profile the data you
• 23:30 - 24:00 Define targets and matrices uh you define an increment data quality rules you put it in development you do testing MB View and continuous monitoring because then you go back to here and continue on and on and on so it's a six step process uh the initial uh proofing or ongoing monitoring
• 24:00 - 24:30 optimization so you do the profiling profiling is to discover and access the data structure and identify if there are any anomalies help identifying strengths and weaknesses in uh data and help determine the the project plan that you need to do uh you would you can pinpoint different data errors uh for example
• 24:30 - 25:00 inconsistencies duplicates using data we discussed these early as well then we have the mattresses in matrices uh you can do overall data for overall data or each of the data attributes oh each of the data lines so so basically there is lines and columns you understand that um it should also be based on the data quality Dimensions that we talked about
• 25:00 - 25:30 them earlier so then the data quality rule defining um it is very dependent on the business requirement and it will process specific uh alerts warnings so computes rejects as so these are the feedback that would be fired these are the uh the replies and flags that would be raised so whatever whatever that you need to be triggered
• 25:30 - 26:00 when data quality issues come that is done here development is where you apply the degree rules uh and decide where they should be applied testing is whenever uh the develop should be used for preview how my percentage of the data quality is achieved right
• 26:00 - 26:30 then we come to the different types of data uh used in profile so we have the column structure if you have a new lens uh input data types so these are basically metadata types that would be captured so during data profiling um if you have nulls and blanks if you work with the specific uh pandas you you would see that there
• 26:30 - 27:00 are ways to uh handle both nulls and zeros uh duplication a very common thing to handle uh data lines uh deduplication and so on so forth um and uh patterns can be managed by yes regular expressions and just checking matching patterns frequencies means how open something
• 27:00 - 27:30 happens so uh sometimes uh high frequency data are very useful sometimes low frequency data so let's say if you are doing something like a data mining um of a large amount of dictator then um low frequency might be very useful but on the other hand uh on the same data mining if you are trying to do something like the Google identification uh uh pattern matching uh rule mining then
• 27:30 - 28:00 maybe higher frequency things I could uh okay if you are doing some some quick data OBT natural language data these birds that are repeated always like uh the these stock words might not be very useful but on the other hand birds and uh very rarely use uh would be quite useful then we have the Rangers we have very specific ranges for dates times uh the Springs
• 28:00 - 28:30 uh and numbers so if you have water and so on so forth you would know that there are very uh interesting cases if you are interested um look at how in the game series uh named civilization Mahatma Gandhi the leader is considered the person who's very associated with nuclear weapons if you look uh so as a homework or a side
• 28:30 - 29:00 thing look it up and see how data Range quality checking has not doing that has resulted in one of the most hilarious and uh long-lived bugs in video gaming history and then we have spaces on uh between bird sensors so these are things that if you are having uh strings uh whether whether or not to remove leading
• 29:00 - 29:30 spaces it is not a constant right so if you are um if you're processing essays maybe if you are processing python code don't do that don't remove these faces because then you will lose the uh the nesting information so most of these things that I say in this lecture and most of the other lectures as well will be dependent on the situation so then we have fasting and character sets if you have again
• 29:30 - 30:00 this is for uh spring data if you have upper limits uppercase and slow places uh the numerals do you we uh if you want to keep it only alphanumeric that means ABC A to Z then one two Z one one two nine to zero um do you only need to keep them non-neuty effect that means uh do you do you want to handle Unicode don't you want to handle
• 30:00 - 30:30 it raining or do you want to convert any report to ASCII or do you uh have a need oh Unicode if you are handling things for Signal out I mean definitely you need to keep it as Unicode right 36 um that you derive from the data minimums maximums uh range values itself so you have the domain on one side and then you can get the ranges and you can calculate things like standard deviation and then you can
• 30:30 - 31:00 normalize database extended so if you have a certain column it is possible to take a z value and normalize it some cases that might be important so some cases it might be very important to remove the bias between columns right next we will look at different types of data profiling so here we have
• 31:00 - 31:30 something like single data and uh so single column would be assuming everything is the same um time number of distinct values max value mean value primary keys for in case that sort of thing if you have so that is the cardinality spot um if you have been waiting for the sake of buttons and date types maybe spring
• 31:30 - 32:00 versus number versus speed value distribution you can think of address phone email first name uh on the multicolor uniqueness uh they have to be you can say they had to be unique and distinct counts uh all values in a are also present in b uh if you are looking at uh to uh
• 32:00 - 32:30 when you're looking at multiple columns so these uh things that uh only then on a single column you are only considering one column when you are deciding and here multi color means you are deciding you are you are looking at more than one column at a time uh so inclusion they never uh two records have the same X values do they have the same a value as well so uh let's say if
• 32:30 - 33:00 if the columns have be in the United States and force that they would have dollar right so that is a multi-column operation and exchange rate is uh another example so so you can have the
• 33:00 - 33:30 United US dollars to Great Britain pound and then you have flipped on the other three and they should have a relationship of uh X and one now it's right because otherwise it wouldn't work next we are going to look at the roles in data quality
• 33:30 - 34:00 you have the chief database data stores subject matter expert data quality leader capability and analyst technical specialist process Improvement specialist data quality field so these are just remember I when I started this lecture I said so this is going to be very very varied management type lecture and you can see why right so this is a management structure of uh of the people
• 34:00 - 34:30 that uh in in the data for the pipeline all right um what is uh specific for Big Data uh in Big Data we should care about the bees we have been repeating this all over this lecture Series so they should be explained in
• 34:30 - 35:00 the terms of uh volume velocity velocity variety value and whatever the other means that are added subsequently to the question of Big Data all right now so we are now
• 35:00 - 35:30 moving on to so you saw who is doing data quality roles and uh you see that there are people who have domain no data quality this is why these two topics are interconnected and with that data quality now we are going to look at who is going to control and Trust the data who is going to control the data who is going to trust the data who is going to control the task Trust of the data and that falls under data governance and
• 35:30 - 36:00 Stuarts you can see data columns and stewardship so we we looked at data students under data quality as well so you can see the connection hopefully all right so we have the uh data infrastructure managed by the chief infrastructure officer um information managers data Architects data models uh modelers who will be using Hadoop databases data
• 36:00 - 36:30 integration and everything else that the technologies that we looked at mlib azus Azure and let's say AWS whatever all of it foreign [Music]
• 36:30 - 37:00 these people would be utilizing things like visualization and self service business intelligence so you already have that knowledge assets because regardless of whether you would be using
• 37:00 - 37:30 it or not you will be the one who are building it for weak data so the goal of a chief data officer CDO is data collaboration here so there would be data governance manager data stewards who would be under them and we would be working with data stewards ship platform so you can see
• 37:30 - 38:00 how the two ends of the things that you have been learned so far the very very technical lens to do the analysis storing and working and the Very human part of visualization and so and so forth can be finally bridged together with this lecture with data governance and stewardship so as usual it is the managers who are doing that so thus we are discussing
• 38:00 - 38:30 this managerial ideas all right then let's look at uh 10 elements of Big Data accounts first we have the data architecture we need to care about what is the uh the structure of the data and how we present it how we store it uh how we handle it
• 38:30 - 39:00 then how do we model it for our business members so data can be anything it can be modeled into it's like KO packs um so data governance should care about how the modeling is done next we have to think about where they are stored do we have are we doing rate are we doing duplication uh deruplication do we have sharding
• 39:00 - 39:30 uh what type of data store are we managing and what do we do in a failure who is responsible for uh both the storage itself or the SEC and though they say actually and as we were talking about that we need to care about the data security uh uh we need to make sure that nobody hacks into our big data system and
• 39:30 - 40:00 steals all of our nice QC data uh then we do need to care about the integration and interoperability uh the data and the insights that are derived from our data can be plugged into the overall business process that we are we have
• 40:00 - 40:30 document and content a it goes two ways one way is that we need to have a the documentation on the metadata and what data is and on the other hand we need to know where data is going and where data is being handled and so forth so these are mostly the documentation aspect and also the data descriptions
• 40:30 - 41:00 if you have it so references and mass data were so this might not be relevant to all cases this is relevant when there is a hierarchy of how the data is handled and with some data are dependent on by function no Integrity o value whether
• 41:00 - 41:30 they are dependent on some sort of higher level uh data so for example when we were talking about exchange rates and uh monitoring variables and also uh states it is very dependent on the variables and data on the actual geopolitical information right so um it would be some master data which would contain uh these are the states of
• 41:30 - 42:00 United States so these are the uh monetary units used by various countries Etc so ik very simple and generic example that you can understand but uh so those are like worldwide constants right so uh do not be misguided that that is the only type of masturbator that exists there can be other Master data which is only relevant between your organization
• 42:00 - 42:30 um but the examples that I gave is understandable by everybody in any any case so um but what you call Master data the the level of specific City or the that definition where you draw the line uh it's very dependent on your particular application and especially your particular company right so then we have the policies of data warehousing and business
• 42:30 - 43:00 intelligence so it goes two ways so sometimes we even call the extraction of information from data warehousing data warehousing so it pertains both to the idea of where you put data how you put the big amounts of data because you don't need data warehousing to store 100 lines of code it is only relevant when it is Big Data um how you put it and what level do you put it do you put prior data do you put
• 43:00 - 43:30 process data are you going to keep them and are you going to keep in product files or not and so on so forth and business intelligence is going to be the derivations that I you are going to derive from them so if you do something like Association rule mining as I mentioned earlier how are you going to handle that is it going to be part of it is it going to be open to all of the business processes that are there so because
• 43:30 - 44:00 even that would be sometimes a a derivation of the patterns between the data so then we have the metadata we mentioned uh this earlier even in uh document and content uh what type of metadata do we have uh it plugs into all most of the previous points that we have mentioned and I don't think I need to take too much time explaining what
• 44:00 - 44:30 metadata is to you and finally data quality so as you have seen data quality is an important part of data governance uh the data governs the person who's in charge of data governance should make sure that the data quality is maintained as we discussed earlier all right so now we get to
• 44:30 - 45:00 something that you may have already predicted movies so if you are looking at Big Data dominance there are movies they are always enough views in Big Data so okay let's see we have the volume uh please signal here which I have left as it is but let's look at
• 45:00 - 45:30 um volume you what is can you find the information that you are looking for then we have the value can you find it when you most need it oh this it have value when you find it so it goes both ways right ah telling how a nuclear reactor Works to a Iron Age
• 45:30 - 46:00 person might not be very useful but telling them how to make a certain projectile weapon might be useful so uh even very useful information might not be useful uh depending on the time that you find it is an extreme example that I gave but you get the idea then the veracity are you sure that this is real are you
• 46:00 - 46:30 sure that the information that you're handling is fair so we actually looked at this uh earlier as well when we were talking about other bees uh sometimes veracity is even added to added to the first three these as a very high maribb uh given that that we now live in the age of this information so visualizations can you make sense at the glance if you if you want uh is it
• 46:30 - 47:00 possible we looked at that variety one of the old good old three you see uh [Music] can ask the example says image would be okay then trying to write it down in various various languages right anyhow Willow City again something that we looked at earlier uh information has momentum we have our
• 47:00 - 47:30 social media and so forth which builds up very fast on data uh how is it for today then we have new movies viscosity uh who did it stay with you does it call for Action uh right just this data is this data mundane and it is something
• 47:30 - 48:00 that you expect uh that happens all the time you see falling into the pattern or does it give you an idea and an opportunity or a fit for you to change your business process to change your ideas and how uh your book is organized or even at least sometimes uh how your data is processed then we have reality
• 48:00 - 48:30 so that's it convey a message that we can be set into uh a presentation on Instagram as in the city does it have an um a new Marketplace information that you are getting from this big data all everything these are business problems right unless you are in Academia but even even in Academia you have to be able to sell
• 48:30 - 49:00 your idea to your reviewers who are reviewing your academic paper if it doesn't have a punch if it doesn't give a novelty it is going to get rejected so similarly and and not even similarly more so than that in business world does it have a fund would it catch attention would the consumers care matters okay yeah pause this video
• 49:00 - 49:30 and do a little exercise on listing any issues that are faced in handling data quality and governance related to Big Data in terms of these eight V's
• 49:30 - 50:00 so I'm going to continue hope you pause and did that little task so what are the challenges the diversity of data sources as usual so people are in aspect uh bring various data times and we have context data structures and this increases the
• 50:00 - 50:30 difficulty of data integration so you might your data might be talking about same or similar ideas but you would have a hard time putting them together then we have problems of uh okay volume so remember the first one is variety and the second one is about volume even
• 50:30 - 51:00 though the word is not mentioned explicitly in front right you see how it goes and it is difficult to charge the data quality within a reasonable time then we have but velocity where data changes very fast and the timeliness of data uh is very short Things become irrelevant very fast let's say it is something like stop training it is either you use it and do
• 51:00 - 51:30 your stuff trade or you lose it use it to lose it in most of these business applications and uh requires you to process things very fast all right so in the same aspect we are going to look at the roles for the chief data officer as published by telling the 2014
• 51:30 - 52:00 paper um so there are dimensions of the chief State officer that means today looked at so there is the collaborative Dimension there are some it goes inwards and outwards then the USB data space traditional data and Big Data Dimension there is the value impact of service and strategy
• 52:00 - 52:30 so uh the CDO has the roles that I mentioned in the diagram as a coordinator reporter Arctic Ambassador analyst marketer developer and experimenter so these are all these business business scams and uh here there is the reporting who would they report to um who would the people in this particular
• 52:30 - 53:00 division of a company would report to uh to CDO to CEO uh so CDO is the data officer chief CEO is the operating officer and CFO would be the chief financial so these are the things these are the people that we've been working under a CDO would report to on various uh data quality and data governance issues all right so
• 53:00 - 53:30 let's look at the data quality Dimension um consistency bits uh idea so that is the contextual consistency which is uh how the Big Data the large data sets that we have um are used in the same domain of Interest as in uh
• 53:30 - 54:00 multiple data sets in but in a singular uh uh domain of Interest without I mean the format or the size of the bonus velocity of the plate and then then we have the temporal consistency where it is it's a matter of time and how the data comes in it should be understood in a consistent
• 54:00 - 54:30 time slot uh because Sun as I mentioned earlier it may be giving you very uh extreme examples we discussed how data from one time so it might not be competitive with data from many different types then we have operational consistency where the operational influence of Technology on the uh the production and
• 54:30 - 55:00 we will save the data so you can see uh what are the subdivisions or subdomains of each of these previous that we usually based on uh come into play and these consistencies right so [Music] you can see the can the even the word
• 55:00 - 55:30 consistency is used in contextual because uh they are the the consistency means uh consistency in the regular definition we are data should be consistent right so with the the speed of the data that we are getting so let's say from the velocity column uh they should be substitutions to each
• 55:30 - 56:00 other they should be comparable they should be matchable and again with the variety that you have care about that uh where even though you have multiple sources and micro types they should be working with each other so those are some examples I'm not going to explain each of the cells and each of the columns but you get the idea of how it is structured
• 56:00 - 56:30 so here is the example of a popular tool uh it is PQ at Amazon so this is how internal data quality assurance application the tune of Amazon words so
• 56:30 - 57:00 it is running on Apache spot you know it's starting um they have the data then they calculate the matrices that they have uh uh defined and the the input the data quality constraints and the so it it can come from uh it can come from business decisions or it can come from
• 57:00 - 57:30 as a solutions from uh the data itself so it goes both ways and uh they Define the matrices and the matrices are calculated and they need to be verified and with the calculated uh matrices computed matrices they would get the data quality reports which is used for internal resistance in Amazon
• 57:30 - 58:00 your system does not have to be uh your data board system your your organization's Big Data quality system or the data governance system does not have to follow exactly this but it's a simple and elegant solution so maybe uh it's a good starting point for you as well all right so there are two main ideas so two main
• 58:00 - 58:30 ways of data molecules so the the reason that why this uh title has a ship is because generally they started with this hierarchical data governance which is also called system of reports I'll be using uh either of these words in the either of
• 58:30 - 59:00 these titles in the subsequent slides so this used to be the default way data governance is coming so you have the keypad coordinator keyword oriented it is a very strict hierarchy it is defensive some when something bad happens they rush to fix it and it's also based on scarcity there are a few customers and even fewer producers
• 59:00 - 59:30 there is a compromise because this is the whole system or system which which even predates big data so few we were losing some of the power and the versatility of uh even digitization impact this this has roots in paper and 10 days
• 59:30 - 60:00 of business and it uses digital operators to some extent but not too much it is not uh very scalable for Big Data by we pay a lot of data scientists who are running so this is why you need a shift [Music]
• 60:00 - 60:30 which is also called the system of an engagement see where the CTO is um but CDO now we see experimented by themselves uh it is a mode remember that uh 3D graph that I showed you so they have moved towards outward Direction and The Big Data Direction and the strategy Direction and the offensive is value driven it is
• 60:30 - 61:00 this is why you might have heard these words in business meetings a lot how do we create value it's a very common question in this business questions right and uh now now we are facing abundance there are many people at the same level because it's going to be a great you can see how it is uh arranged in a matrix pattern uh they have there is a
• 61:00 - 61:30 many producers uh the data we say it is democratized that means uh there are each of these uh can be producers that we there are no bread lines bed lines uh these are very uh known for you this case because these are basically cues waiting for a resource so or if you uh thank you
• 61:30 - 62:00 gasket so this is a bread Cube that is what is uh used for this word but basically that is the idea for a resource you need to wait uh until it is available in the previous system but here you don't no longer need to do this so you are customizing the business intelligence and it is and the cheap digital power because you have distributed it among your basically notes right and many many nodes would be servicing
• 62:00 - 62:30 many it is no longer be on this because it's no longer the responsibility of people who are sitting above you to provide you with Services it is uh is anyone any one of your peers who are capable of providing the service can provide and the supports to the customer in in one hand takes if the customer doesn't have a single point of failure they can attach they can be sent to different
• 62:30 - 63:00 nodes many nodes um and this would increase the availability and reduce the time for time between the opening of an issues and closing foreign
• 63:00 - 63:30 [Music] let's say mostly utilized on the access uh support all right so remember I said that I'd be using these words a lot so let's look at a system of records and system of Engagement this is from uh this analysis
• 63:30 - 64:00 is from this paper so we have the hierarchical data governance okay system operator so also called Enterprise content management so these almost interchangeable Network names similarly Network on a system of Engagement and the gate for social business system so hierarchical purpose is the control and regulate because it is the old system where the power and control six with the people who are sitting on the top
• 64:00 - 64:30 and then we have the drop down design okay it is organizing the drop down design and around these three pieces of information that are called records the decomposition happens in Black boxes a PP have no idea what is happening in systems that they have no access to uh
• 64:30 - 65:00 and this can go up to the very top and it assumes it presumes a big picture that means uh PP at bottom level think that the people who are above them would have a better understanding of the picture and that is what they are using to guide them so uh so examples of this uh sapc birthday service now and at last so I think some of you might be using this
• 65:00 - 65:30 next we have the network data Gardens fantastic purpose of innovativeness and it is organized bottom up in the decent uh decent place Bay it incorporates Technologies which encourage via interactions it's leveraged by clouds this is seed money and you have
• 65:30 - 66:00 emergence of this because uh it is going to be a network of people and there would be emergence properties coming from that so examples of this kind of solution is slack or confluence right now let us look at a comparison of the system of Records versus the system of Engagement
• 66:00 - 66:30 here we have on the left system of record and on the right system of Engagement so if you remember system of Records is the old system and system of Engagement is the new system so you can see how uh system of Records record these focused on transaction control and permanent application as their qualities
• 66:30 - 67:00 or by on the other hand system of Engagement is more about collaboration interaction uh and Adobe can open accessibility uh next we now look at how fast can be digitized so in hierarchical data governance so system
• 67:00 - 67:30 record it's established centrally by a session competence Center so just like the control it's going to be the the crust as well is going to be centrally managed um it is also possible to have external pointed trustees with formal roles such as stewards owners and actors and on the other hand for network data
• 67:30 - 68:00 governance so system of Engagement and trust is more complicated because everybody is a peer they are equal then you need to know whether if the data is a fact or an opinion and what is the intention and who below who does this data belong to who is going to watch for it is it today it has it been certified through the standard process and so and so forth so while
• 68:00 - 68:30 it seems from the previous slides that it is a very good then free and uh democratic application to data governance as you can see when it comes to trust and security and that uh sphere of influence you can see that it might be forcing certain problems next we have uh hybrid data commands which is which apps you may have guessed it's a mix of those two uh
• 68:30 - 69:00 so critical assets are controlled top down like hierarchical and the rest of it the things are things that are not so critical uh strongly regulated you can have a peer driven architecture to give more Serendipity or freedom on a shared platform so again uh
• 69:00 - 69:30 the challenges here are digitalization of trust because even though you can certify from the top the critical items you can know you can not certify the ones that are controlled by the peers yes obviously um they bring the the situation or the arrangement that is
• 69:30 - 70:00 coming from each of their sites and in the case of Analytics as for data valuation and Netflix they have to be married to uh the two types of governance as well
• 70:00 - 70:30 and as you can see at the bottom the problem of leadership and role transition uh between uh who is going to control what is also an important Point here so let's look at how the system of record uh and the data sets it is say authoritative source so because it's
• 70:30 - 71:00 going to be a top-down so all activities and information and the surrounding data uh and its meaning and its use works as the foundation and then we have the process the metadata then we need to know where the data comes from so you have data catalog data dictionary then you need to know what the data means so you get the business glossary reference data next you have to know whether the
• 71:00 - 71:30 data is right so you have the policy manager meaning who is going to be the person who's going to be controlling this data oh uh on the other hand so on one hand it is controlling on the other hand it is going to be responsibility so you have find understand and Trust so these three policy three aspects of data
• 71:30 - 72:00 governance being controlled and being utilized uh in this structure next we have to look at the uh the life cycle of the Big Data tech industry so we have new injection of data which goes through Discovery collection indexing query and
• 72:00 - 72:30 integration so integration after the Integrations step there shouldn't be any difference between this new data and the data that existed before I mean there should be a time values and so and so forth but the point is after that step they should be indistinguishable from the data that existed in the Big Data store that you were having
• 72:30 - 73:00 another thing that we have a look at is how the data quality is guaranteed here so we have the data Lake where you have all the data uh at a equal uh balanced scenario and then slowly you move it upwards um you know with the Innovation step with the conceivable step and until uh you arrive
• 73:00 - 73:30 at actionable inside because these are the things that the managers are interested in they are not most of the time they are not interested in numbers and graphs and Trend sensors and so forth you need to give actionable Insight so that is how it is done on value based data governance to move step by step until you can talk in the same language as people who are not
• 73:30 - 74:00 as Tech so yes you are uh on the case of uh valuation cycle we have the consumption value so if we start at that point so you have to get the um policies for data sharing data handling uh and uh data lake so we'll start there
• 74:00 - 74:30 because we always say we will we are data centered and so forth so from there if you start with the little pyramid which we are familiar now uh then we have the prescriptive value where you extract insights and derive use cases and uh manage algorithms that are relevant then we bring it up
• 74:30 - 75:00 into more marketable domain the creating competitive value so so how do we use this insights to um improve the complexity value and from competitive value you get operational value so how these values can be
• 75:00 - 75:30 applied for the business use or handling event driven programs uh and identifying critical data elements which then results in the process and risk value so there you can decide on what is the uh risk management how would
• 75:30 - 76:00 you reduce the risk that you have for the company uh for your business center and so forth how would you optimize your process and so on so forth so these 10 results in in your data store your data Lake and you start the process all over again so this is how uh you go around the data evaluation cycle and add value
• 76:00 - 76:30 to your data all right so post here and try to see whether you can suggest a new dimension new poly Dimension that I did not talk about so far thank you if we thought privacy or privacy depending on however you pronounce it you thought the same as many other
• 76:30 - 77:00 people so why does why is privacy important privacy is going to be an important part in data science forever because data is something that people do not surrender Billy nili and some people are very sensitive about who has access to their data and who
• 77:00 - 77:30 controls their yeah some people are okay some people would post things on Facebook and so on so forth and but some people are very guarded about their data before various things and some uh you can see celebrities have a reason people uh being worried about their private life more so than a random person but even random people have some people are very concerned about uh putting data in public so that means
• 77:30 - 78:00 whoever captures this data should be very careful about uh the privacy of the people who are associated with the data that they collected so I there are some rules laws that govern uh personal data privacy and if you remember very recently not very recently about one and a half years ago you might have gotten a lot of emails
• 78:00 - 78:30 from websites that does uh hosted in Europe because uh across Europe the European Union changed some of the policies that they have for data privacy and that resulted in all of them updating their terms and services about the big data that they collect on behalf of you or about you on their servers so data privacy privacy preservation or
• 78:30 - 79:00 privacy preservation has various levels um you can do that at data collection level you can do that as publishing level or at the data mining level so uh data collection time you can just not collect certain type of data uh things that are de-anonymizing oh at the publishing time you can remove the data columns or features that would be
• 79:00 - 79:30 distinguishable uh de-anonymizable and Publishing and on the data mining time sometimes there are possibilities of reverse engineering these anonymize the time to de-anonymize data so uh you have to be careful at the data mining time as well this goes on to a little bit of
• 79:30 - 80:00 computational Ethics as well since we do not have that as a class I would suggest you look at data I thinks for this particular question all right so what are the personal data attributes um first we have personal information identifiers pii uh we will be talking about this say at several slides so uh
• 80:00 - 80:30 you can say as personal information identifiers oh the other way around personally identifiable information either way it is going to be pii so things like ID name and email definitely going to be a direct culprit of being API so most of the time you are not supposed to release that type of information into uh pure data Lake then there are quasiated files so they
• 80:30 - 81:00 are not directly identified but if you mix and match with other data sources you should be able to de-identify de-anonymize people so there are things like age and depth option Grace so if you have one data set that gives a gender person ways of uh certain people living in a certain area so there is one data set and on the other that's it you have the same information uh agenda protection uh race of the PPE
• 81:00 - 81:30 and let's see uh some kind of a behavior so then you can put these data sets together to match who in what area has this particular Behavior then there are uh the sensitive attributes such as salary relationship start status diseases especially um medical information uh in most developed countries they are very heavily guarded then there are the
• 81:30 - 82:00 nonsensitive attributes which is anything that does not fall into the above cases right so how to manage these the first thing is to suppress by replacing it with the star or something or a null or so and so forth generalization is to replace with generalized values you instead of saying people's uh ages and so and so forth you'll just average uh then there is uh swapping you can swap some of the values with uh either
• 82:00 - 82:30 hard coded values or randomizes but that could be tampering with the data array not much in that particular sense then we have anonymization sorry and optimization there they are we you so remove the you separate the cost identifiers and sensitive attributes into different tables so you cannot match one with the other so you can still give the data out but
• 82:30 - 83:00 uh not which row contains what value uh directly so if you if you break the table apart and not have any quote unquote for in case it would be okay uh you definitely have to uh sort by those uh the shuffle the rows as well otherwise you can just put the tables together so
• 83:00 - 83:30 you have to be included uh obviously then the then the prescription uh which is perturbation which is basically replacing the original value of some sensitive attributes using some fake values again if you do something like that you have to mention that when you release the data set otherwise uh other data scientists are going to run their systems on your data and get uh very bad
• 83:30 - 84:00 results because these are random and fake values so um while we are talking about rear indication and consent um given so in Instagram is not a big deal but uh in European countries and the United States it's a big deal where uh ZIP codes are used very much uh so they have found that using the ZIP code um uh birth date and sex you should they
• 84:00 - 84:30 would be able to identify about 87 percent of uh PP with social security numbers the the social numbers of people so then uh in let's say in United States Social Security number is tied to your tax your bank and so on and so forth and then they can be fraud and steal your money and so forth uh but initially those pills are not considered pii personally added terrible information but
• 84:30 - 85:00 they can be derived and then another study another incident that happened in 2014 New York City release about 173 million taxi trips so it is just details about taxi trips um even the license plates and the taxi identifier uh has been obstigated for anonymization process but it was de-anonymized within hours of being released and um
• 85:00 - 85:30 BP tagged and found where our celebrities were living where they have gone who they are being with and so and so forth uh very within hours of the uh release of data so that was a huge privacy scandal um informed consent uh human subjects if you are you if your data is from humans the human should know about the experiment must say that they agree with the experiment uh and it should be
• 85:30 - 86:00 voluntary and must have the right to be broken set at any time so that is what happened with the Cambridge analytica they are they used the information that you provide to Facebook and the information that is gathered by Facebook to put together and decide who should be influenced for political reasons for political campaigns and so and so forth and it caused a big scandal uh
• 86:00 - 86:30 you need to balance the concrete data and privacy one thing I need to mention as well uh in the uh the taxes study uh how they found celebrities is by cross-referencing with timestamps pictures of celebrities taken from uh Paparazzi and so forth uh of them entering or exiting taxes um to map them together to find personal addresses and how much they think cancer and so forth
• 86:30 - 87:00 then uh here is an example of a data set with the health records so there are names so personally this is a personal identification this is definitely a dangerous field and then we have the Quasi identify verifications which can help in uh finding the identifier and then there is the sensitive attribute which people do not like other people know it so how you
• 87:00 - 87:30 can fix is uh you can generalize this type of data and you can suppress this type of data and again generalize this type of data all right so there are various privacy models from literature there is K anonymity so K anonymity is
• 87:30 - 88:00 well the information for each individual cannot be imminently differentiated from at least K minus one other individuals so if uh when you go to find or be anonymize details if you find at least K similar records then you can say it is anonymity so if
• 88:00 - 88:30 it has 1 million people and when I try to De anonymize your data if your data is equivalent to 10 other people then it has K 10 anonymity then we have ill diversity L diversity is if a regroup of tubers that share the same quasi identifier values have at least L we will be presented sensitive way so that means
• 88:30 - 89:00 if all the Quasi identifiers are similar there has to be at least l number of sensitive values in that section of rows right so they they should be uh well represent that means those should be uh class balanced then we have the T closeness T closeness is the distance between this future not the sensitive attribute in this class and the uh attribute in a whole table should not
• 89:00 - 89:30 be uh should be no more than a threshold P so there are some examples here so here this is a co-anonymous data set why is it the over Anonymous data set because you you cannot uh if the name column was not there
• 89:30 - 90:00 you cannot distinguish any of these lines from each other we're using the policy identifiers to be unique right so all of these are the same so that means this is a full Anonymous data set there is nothing to be distinguishing so you can see that this is in if for a machine learning
• 90:00 - 90:30 perspective for a data scientist if you are looking at trade-to-day scientist perspective you are you must be thinking this is horrible this means we cannot do good prediction on this data yes it is true so you are sacrificing the power of your model the power of your system to save the privacy of the people about whom you have collected the data that is the whole idea you are making weaker and
• 90:30 - 91:00 weaker systems uh and having bad accuracy because you need to uh save previously but that is the that is a sacrifice that you will be asked to do then we have uh so next we have okay diversity so here it is uh it the same so you can see these are the same
• 91:00 - 91:30 right these are the same and they are you have at least three types of labels so again these are same and there are three types of labels these are the same there are three types of Labor so this is a three diverse data set
• 91:30 - 92:00 all right so privacy preservation should be capable of preserving the Privacy as well as data mining utility at the same time that is what I mentioned earlier as well so you are sacrificing the goodness of your data for the sake of privacy but it shouldn't be too much you should not uh the data so much so that it becomes useless so it's a balancing
• 92:00 - 92:30 act that you have to be very careful about so what type of matrices that you can use um you have the previous symmetricians and the utility messages right so you need to balance both of them so on one side you have the previous Matrix so we have the confidence level um where you can calculate the Privacy confidence
• 92:30 - 93:00 then we have the conditional entropy if you are familiar with how entropy is calculated it is uh the uh difference between the idea of how much of information that is contained uh from absolute chaos right so the differential entropy is the difference between the the entropy values if there is uh so hidden failure is the ratio between sensitive patterns hidden within the biopsy uh the ability to So
• 93:00 - 93:30 when you say failure it is on the other hand it is say success of finding out data but for the concern of previously it is a failure on the other hand so the all of those to make sure that privacy is guaranteed and then on the other hand we have the utility matrices uh metrics uh which is trying to guarantee that the data set has not become useless
• 93:30 - 94:00 because of our uh attempts to make it more private so we would do generalization suppression counting how much of uh generalization uh suppression operations were done the loss metrics gives you an idea of the Noma is not stock is each attribution how much of information how much of diversity uh you have lost because of what you did um there is a penalty for each Tuple
• 94:00 - 94:30 based on how much how many other tuples on the data set are indistinguishable profit so remember on on the other hand we are we were trying to get this L diversity and K anonymity to make sure the uh the tuples were same from each other were same to each other and uh indistinguishable so that the Privacy is preserved but here the loss metric on the other hand penalizes that that indistinguishableness
• 94:30 - 95:00 um so what you need to do is to balance both of them and have decent values to both of them so then we have KL Divergence it is not just a method for this scale Divergence is basically to see how the two probability distributions is different from each other so it is not only for this privacy business it's not only for this utility uh checking you can use it in any of your programs any of your applications even if you are it can be used as a nice loss function
• 95:00 - 95:30 if you are doing a machine learning so uh basically if you are predicting a probability distribution and you expect it to be a certain probability distribution instead of checking by label by label what you do is you predict a probability distribution and see how different is the generated probability distribution is from what you expected so in this particular case what they do is you have the
• 95:30 - 96:00 original table as a probability distribution and the sanitized table as a probability distribution and then see how diver these two distributions are so that is basically the idea there so finally uh the few directions on privacy preserving data uh in the case of data governance is privacy can be subjective um it depends on each of the people each of the person and the country and the
• 96:00 - 96:30 company and so forth um most of the time user has the ability to decide what what type of privacy they want and what level of privacy they own sometimes these are overshadowed by the end user agreement that you sign the thing that you scroll down and say Yes usually and on the other hand we have the context area privacy as a future Direction um using iot As again as I mentioned
• 96:30 - 97:00 it's this uh buzzwords uh and identifying privacy requirements based on the context so even if the person does not say something um you should be able to derive it derive the Privacy requirements from uh the context because you have seen similar incidents similar uh comments on countesses these similar contexts before so that is the end of uh
• 97:00 - 97:30 this uh section on talking about how privacy uh has been factored into the case of uh data governance so here we have references on uh oh it's uh presentation post name and this ends
• 97:30 - 98:00 bought this lecture and the lecture series for this course thank you