#64 Batch Norm | Machine Learning for Engineering & Science Applications

Summary

In this enlightening video, we explore batch normalization, a technique to enhance training in deep neural networks, acting as a sequel to data preprocessing. The tutorial delves into the problem of input distribution changes in layers during training and offers batch normalization as a solution. The approach involves using a mini-batch to calculate means and variances for normalization, which greatly enhances training speed and stability. The key mechanism involves adjusting inputs by their mean and variance within mini-batches, followed by scaling and shifting with learnable parameters, gamma and beta. Besides accelerating convergence, batch normalization simulates regularization effects akin to dropout. Challenges like increased computational load and its application in convolutional networks are also highlighted.

Highlights

• Discover how batch normalization serves as a sequel to data preprocessing 🎬
• Learn about the challenges of input distribution shifts in neural networks and how batch normalization tackles this 🌊
• Understand the role of mean and variance calculation in mini-batches for normalization 📏
• Explore the significance of gamma and beta parameters in adjusting network inputs 🛠️
• Realize the dual benefits of increased learning rate and training stability introduced by batch normalization 🚀

Key Takeaways

• Batch normalization helps stabilize and speed up training in deep neural networks 🏃‍♂️
• By ensuring input distributions within layers remain steady, networks can learn more efficiently 📊
• The method uses mini-batches to normalize input data, enhanced by learnable parameters for custom scaling 🔄
• Convergence is faster and less prone to issues like saturation due to large updates in weights 🎯
• Batch normalization mimics dropout, offering similar regularization benefits 🤖

Overview

Batch normalization is an innovative methodology that bolsters the training process in deep neural networks by addressing input distribution shifts that can occur during training between layers. This process involves normalizing the inputs within each layer using mean and variance calculations from mini-batches. The intention is to keep these distributions steady, thereby enhancing the stability and speed of network training.

Through mean and variance calculation, batch normalization effectively controls input activations before they pass through layer non-linearities. The application of gamma and beta parameters provides additional flexibility, allowing custom scaling of inputs beyond mere standardization. This not only accelerates learning by allowing higher learning rates but also prevents issues like saturation during updates, similar to regularization offered by dropout techniques.

Adopting batch normalization introduces minor computational overhead due to the additional layer of calculations. Nonetheless, this is outweighed by the significant improvement in convergence speed and network performance stability. The technique's adaptability extends to convolutional networks, promising widespread applicability across various neural network architectures.

Chapters

• 00:00 - 00:30: Introduction to Batch Normalization In this chapter titled 'Introduction to Batch Normalization,' the focus is on understanding the technique of batch normalization. The video begins with a welcome message, hinting that this chapter builds on previous discussions about data processing. Batch normalization is presented as a technique that aids in the better training of deep neural networks.
• 00:30 - 01:00: Understanding Bias Normalization In the chapter titled 'Understanding Bias Normalization', the transcript addresses the concept of bias normalization in deep neural networks. Initially, it questions what bias normalization is and why it matters when training these networks. A critical issue highlighted is the change in the distribution of each layer's input during training. This occurs because as the weights within the network change, they subsequently alter the input to different layers. This topic will be explored further in subsequent slides.
• 01:00 - 02:00: Training Deep Neural Networks This chapter discusses techniques to ensure stability in training deep neural networks. A key challenge highlighted is the issue of rapidly changing weight distributions between iterations, which can destabilize the learning process. A proposed solution is to maintain relatively stable input distributions to each layer, preventing drastic changes. The chapter suggests exploring functional approaches to address this problem, indicating a focus on practical strategies to ensure smoother network training.
• 02:00 - 03:00: Transformation in Neural Networks The chapter discusses the concept of transformations within neural networks, specifically focusing on the operations that occur in a neural network layer. It explains the process where the input to a layer 'X' undergoes a transformation defined by the equation 'W transpose x + B'. This is a linear transformation that is further processed by passing it through a non-linear function, which is a typical operation in neural networks to introduce non-linearities.
• 03:00 - 04:00: Normalization of Activations The chapter titled 'Normalization of Activations' discusses the concept of transformations in neural networks. It describes how changes in weights (denoted as 'w') during iterations affect the inputs to each layer. The chapter introduces the idea of F1 and FS2 as transformations that occur to inputs at each layer. If there are multiple layers in succession, each characterized by certain parameters (Theta), these transformations will significantly affect the behavior of the network.
• 04:00 - 05:00: Implementing Batch Normalization This chapter discusses the implementation of batch normalization in neural network layers. It explains how changes in weight parameters (Theta) affect the functions and layers within the network. Specifically, it explores the impact of adjusting weights, such as Theta 1 and Theta 2, on the function outputs and subsequent layers. The chapter aims to provide a clear understanding of batch normalization and its role in stabilizing learning by maintaining mean and variance within the neural network layers.
• 05:00 - 06:00: Batch Normalization Algorithm The chapter discusses the challenges faced when layers in a deep neural network, specifically referring to L as an input or another layer, undergo changes. These changes are expected as part of estimating parameters Theta 1 and Theta 2. The text highlights the issues that arise when changes are random and large, impacting the convergence of deep neural networks. Solutions to these convergence problems are suggested but not included in the transcript.
• 06:00 - 07:00: Forward Pass in Neural Networks The chapter 'Forward Pass in Neural Networks' covers the concept of normalizing each activation in a neural network before applying nonlinearity. It emphasizes the importance of using the notation 'x' to represent inputs, clarifying that 'x' generally denotes the training data input. The chapter instructs readers to consider 'x' as the input to every layer in a neural network.
• 07:00 - 08:00: Normalization and Hyperparameters The chapter titled 'Normalization and Hyperparameters' discusses the fundamental structure of neural networks, specifically focusing on the neurons within a layer. Each neuron receives an input, defined as W transpose x + B, which represents the processed output from the previous layer. The chapter details that for a layer with K neurons, there will be K such expressions or inputs for each corresponding neuron, highlighting the systematic processing within neural network layers.
• 08:00 - 09:00: Training and Testing with Batch Normalization The chapter introduces the concept of performing batch normalization on training and testing data. It begins with discussing input values entering a layer, referred to as 'K terms' or 'k inputs.' The process involves using the variable 'X' to denote inputs. Batch normalization involves adjusting these inputs by centering them using Z score normalization. This method calculates each input's deviation from the mean (expectation value) and scales it by the variance. The chapter touches on the calculation of the expectation value of input 'X' and emphasizes understanding the underlying algorithm.
• 09:00 - 10:00: Advantages of Batch Normalization The chapter discusses the benefits of batch normalization, which is a technique used in deep learning models. It begins by explaining the process of calculating the mean of X for each layer in the model. The focus is on a single layer with K neurons, and the algorithm for batch normalization is detailed, highlighting how it contributes to the performance and efficiency of neural networks by stabilizing the learning process and improving convergence rates.
• 10:00 - 11:00: Batch Normalization in Convolutional Networks The chapter discusses the application of batch normalization in convolutional networks, specifically focusing on K neurons. It explains how inputs from the previous layer are processed with a mathematical transformation typically expressed as W transpose X plus b. The explanation avoids visual representation due to complexity and emphasizes the importance of understanding the transformation applied to the neurons.
• 11:00 - 11:30: Conclusion The conclusion emphasizes understanding the internal mechanics of neurons in a neural network layer. It starts by discussing the concept of nonlinearity applied as an activation function to the output of a layer. The focus then narrows to a single neuron, explaining how inputs to it are used. These inputs, through a linear combination represented as W transpose x + p, are processed before the activation function is applied. This reflects a step-by-step breakdown of a neuron's computation within the network.

#64 Batch Norm | Machine Learning for Engineering & Science Applications Transcription

• 00:00 - 00:30 [Music] hello and welcome back so in this video we will look at this um technique called batch normalization which uh helps in training the network a deep Neal Network better um and it's kind of can read of it as a continuation from the data processing PR processing that we saw in
• 00:30 - 01:00 the previous lecture okay so um what first what is we look at what is bias normalization and then we'll but we'll consider um uh you know what is the problem here when you're training a deep neural network so what happens okay so when we train a deep neural networks what happens is that the distribution of each layer's input changes during training we'll see why that is in the next slide but we can see that as we train because the weights keep changing the input to a particular Network
• 01:00 - 01:30 particular layer in the network would be changing dynamically so because if the weights changed drastically between two iterations you have the same issue right and the solution is to somehow um you know make sure that the distributions don't change too much um in a sense that the distribution of the inputs to a layer don't change too much okay so let's see how um what do you mean by that and how we can address that problem okay um so let's consider this uh you know slightly more like a functional form let's say okay so
• 01:30 - 02:00 fub1 and FS2 are some Transformations so what is the transform that that happens in a neural network in a layer so in a layer what happens you have W transpose x + B okay where X is the input to that layer right this is one transform that happens and you you pass it through a nonlinearity so then you can get something like e to minus um
• 02:00 - 02:30 or to minus right can do something like that so that's a transformation that happens right so when w changes with every iteration you can see that you know um the inputs to the particular layer will also dramatically change so you can think of F1 and FS2 as the transformation that happens to your inputs at every layer okay so if you have two layers in succession each layer characterized by Theta by these
• 02:30 - 03:00 parameters Theta which are nothing but the weights and let's say another layer two characterized by this um another set of Weights Theta 2 right so once if as Theta 1 and Theta 2 keep changing you can see that the so if Theta 1 changes then fub1 will change so the input to F2 changes right sorry the input to F2 will change and if Theta 2 changes then FS2 itself will change again L will
• 03:00 - 03:30 change okay so if you have if L is an input on another layer then L will change in the sense the change of course it's expected to happen because we trying to estimate Theta 1 and Theta 2 if you think of FS2 and F1 as as the layers in a deep newal network but since these changes if they are um in the sense um uh if these changes are kind of random and large sometimes then you have problems with convergence in a deep neural network Okay so what we do to
• 03:30 - 04:00 address this problem in a network is to normalize each activation okay so this is before we apply nonlinearity typically that's what is done is we can do we always use x um just to clarify annotation we always use x to denote the input in general this is our training data input that's what we we usually used X for so for the purposes of this video think of X as the input to every layer okay so every layer
• 04:00 - 04:30 has a set of neurons and every neuron has a input coming into it right and what is that input the input coming into every neuron is W transpose x + B this is the output from previous layer okay so if there are K neurons in a layer there will be K such terms right K terms or k inputs
• 04:30 - 05:00 K terms or k inputs so so this is the input that's coming into a layer and what we do is um so i' I've used X pretty much for everything abuse of notation but then what we do is we do x minus the mean or the expectation value of x divided by the variance of X this is what we saw earlier this is your typical um Z score normalization but then how do we calculate expectation of X what is this expectation of X okay so we'll just go through the algorithm and it'll be very
• 05:00 - 05:30 clear as to what how this mean of X is calculated for every layer okay so here's the algorithm okay we are considering um one layer okay considering one layer okay and let's say it has um K neurons okay let k
• 05:30 - 06:00 neurons and we are just trying to see how this batch normalizing transform can be applied to that okay so if you have K neurons right you have this let's say k is four then we have inputs coming in from the previous layer right I'm not going to draw it because it gets too confusing so multiple inputs coming in from the previous layer um and then of course there is this W transpose X not the the aine transform that we do okay W transpose X Plus p
• 06:00 - 06:30 right um and Then followed by nonlinearity that is the output of that particular layer okay that's your activation okay so then once we have that so how do we calculate this so for every neuron if you just take one neuron let's just just take one neuron so we have all these inputs coming to that neuron okay with which we can calculate the linear combination W transpose x + p and then following and then you apply
• 06:30 - 07:00 the nonlinearity to that right that's the output now what uh what do you mean by calculating the mean of X so this is one input okay to that uh particular neon so what we do is we consider a mini batch of M M training samples so there are M training samples in a mini batch right
• 07:00 - 07:30 and we can when in the forward pass we can actually um uh forward pass all the M samples at one what in succession okay and we can compute this W trans for X plus b for each mini batch so if you have M data points then you will have M such um calculated values for each neuron okay so that is for each activation prior to passing it to the
• 07:30 - 08:00 nonlinearity you will have M such values corresponding to each each data point in your mini batch so this mean is calculated over the mini batch so this is for a neuron right so have a neuron and you have M input data points this neuron is let's say the first or second layer and but then when you do the forward path for a neuron using the weights that has already been estimated um or randomly initialized as you do the forward pass through uh um through the
• 08:00 - 08:30 network for every uh every point in that M um points in that Mini U mini batch you will have one linear combination okay so you'll have M such linear combinations which forms with which you will estimate a mean that's your mean and of course once you subtract that mean out you can estimate the variance squared or the standard div square or the mini batch variance and you will normalize every neuron
• 08:30 - 09:00 with that every so for for I equal to 1 to M over the or individual uh input training examples of that mini batch you will calculate this normalized data point okay and once you have done that then we Define two parameters gamma and beta again for every so this if there are four of them here there will be a gamma 1 beta 1 gamma 2 Beta 2 gamma 3
• 09:00 - 09:30 beta 3 and Gamma 4 beta 4 okay so for every neuron they'll have one hyper two two hyper parameters gamma and beta and you will do this transformation so how are gamma and beta um estimated they estimated through backrop because all all this is you can think of this as a linear layer in your network and that's how it is uh typically interpreted so this linear layer is inserted between your your aine transform which is the linear combination of your um neuron activation
• 09:30 - 10:00 from the previous layer and the and the nonlinearity you apply okay so it's in between these two layers you have uh you have the uh batch normalization layer okay so what it does is that so this is a uh inverse this is a transformation that helps to um make sure that your data distribution in the sense that the distribution of your activations of your um that accuration that you compute for
• 10:00 - 10:30 every neuron does not shift too drastically okay they are confined to be within a certain distribution so when this happens then training is automatically faster and it converges faster so you can think of one example where this will work is when one of the W's are too large and you know it it it leads to it might lead to saturation we saw that we talked about it earlier so by doing this normalization you can prevent that from happening also by making sure you can estimate gamma and beta okay okay so you can also see that
• 10:30 - 11:00 this is like a invertible transformation since it canot you can gamma and beta can be estimated so that Yi can be just equal to x i cap okay that's very easy to see I jge you to convince yourself of that okay so if the original calculated value is the one that is actually desirable then gamma and beta can be uh the network would estimate gamma and beta would to be the inverse of this transformation that we did leading to a um leading to Identity okay so this might sound cryptic but you should read the paper we'll post the paper up there
• 11:00 - 11:30 and I urge you to read it okay so um just to recap Once Again so for every neuron in a layer you see that every neuron if this is a fully connected neural network we can think that's one we talking about an MLP for every neuron in a particular layer it gets inputs from the previous layer okay we call those we Den knowe those inputs by X okay so and we are considering only one neuron at a time so for every neuron there is a linear combination of the active ation from the previous layer
• 11:30 - 12:00 that's that's what we call W transpose X plus b now for when you're doing training there's a mini batch of data points M data points so we do the forward pass and we calculate this W transpose X Plus P for every data point in that mini batch and we do a mean and variance for that for that mini batch for that particular neuron okay and then we scale the activation of that neuron for a particular data point input data point by the calculated mean and
• 12:00 - 12:30 standard deviation okay and then of course we multiplied by this gam gamma and add by Beta um to get a transformed variable so this gamma and beta are again estimated by back propagation so remember that if there are K neurons in a hidden layer you will have K such parameters so it's an addition of K parameters once you have done training so this is how it's trained so for every layer you will have a gamma and beta and it will be estimated as you're training by by back propagation
• 12:30 - 13:00 now once you're done training how do you do the uh testing and inference so then you still need remember for testing and inference you would still have to for instance you have to calculate this mu and sigma right you have the gamma and beta but you still have to calculate uh uh the uh mu and sigma so what you do is for that you can compute compute mu and sigma over the entire training set for every layer for every neuron okay that is possible and you already converged on the appropriate values of gamma and beta for every layer and for every neuron
• 13:00 - 13:30 okay uh there are place of computing mu and sigma as running averages again you can do that as well as you know some exponentially weighted averaging schemes are available but this is the way to uh during testing you will calculate mu and sigma for every layer this mu and sigma for the activations for every layer every neuron in every layer calculate by running it through the entire forward pass data set because that will be added computation or or you can just maintain a running average during training so each of these are are fine um so that's one way of doing
• 13:30 - 14:00 it and what are the advantages so apparently this uh the authors of this particular paper comment that um it uh it is it improves it increases learning rate so you can train with a high learning rate leading to faster convergence because sometimes you have high learning rates you have large updates and sometimes leading to saturation that won't happen with this because you are uh doing this um uh you're trying to constrain your uh activation values to life within a range typically that's what you're trying to
• 14:00 - 14:30 do and that helps okay um it can also help you remove Dropout so it acts kind of like a the regularization effects of drop out advantages of drop outs apparently are also carried over by this batch normalization okay uh improve stability during training same thing because if you have uh sometimes your activations can be very large sometimes your weights can become very large uh leading to you know um poor training and that can be um taken care of by this DCH normalization okay the uh extra computation burden is there because you
• 14:30 - 15:00 are adding one more layer between you know extra layer is added before every set of neurons so that's a thing and you need to have significant batch size so if you use a batch size of like one two or three some of the large problems they require know memory constraints might uh if your data sets large memory constraints might uh uh make you choose very small size data sets okay and that case that there'll be no effect the statistical uh you know uh effects are lost by doing that so then uh it's no
• 15:00 - 15:30 point doing that at that for those kind of problems where you have very small bad SES okay if reasonably large bad SES this will work very well okay um so the one question that we have not addressed um is the convolutional network how do you do this in a convolutional l Network okay um it's a very interesting question um actually the paper addresses that the paper talks about how to calculate it um that will be a homework question okay so now I given you the homework already so
• 15:30 - 16:00 for you to read the paper I'll upload the paper soon and uh read the paper and inside the paper they do comment on how this particular batch normalization can be implemented in a convolutional network remember that what is been described in this video is how it's implemented for a fully connected neural network okay so um that is all for batch normalization so um so that's we we wanted to do these two videos together basically um data normalization as well
• 16:00 - 16:30 as a bat normalization because I think it help you understand this better okay thank you [Music]