McCulloch Pitts Neuron and Thresholding Logic

Summary

This lecture, delivered by IIT Madras, explores the McCulloch Pitts Neuron model, a simplified computational representation of biological neurons. Created in 1943 by McCulloch and Pitts, this model processes binary inputs to produce binary outputs, mimicking the way biological neurons work. The session delves into how this model aggregates inputs and defines thresholds to decide neuron firing, covering concepts such as excitatory and inhibitory inputs, thresholding logic, and implementation of Boolean functions. It concludes with an exploration of linear separability in representing Boolean functions using this neuron model.

Highlights

• The McCulloch Pitts Neuron is a simplified model, mimicking complex neural processing with binary inputs and outputs. 🎯
• Invented in 1943, it combines expertise in neuroscience and logic for computational use. 🧠
• Inputs can be excitatory or inhibitory, influencing the neuron's 'fire' decision. 🔥
• Thresholding logic is used to determine when a neuron fires, depending on input aggregation. 🚦
• This model can implement Boolean functions based on its threshold settings. 🔄
• Boolean functions need to be linearly separable to be implemented by this model. 📈
• Geometric interpretations help in visualizing how inputs are divided into decision boundaries. 📐
• The exploration continues into the scope of linear separability for complex Boolean functions. 🔍

Key Takeaways

• Discover the simplistic charm of the McCulloch Pitts Neuron, our humble computational mimic of biological neurons! 🤖
• The model is all about binary magic—yes/no inputs and outputs! 💡
• Understanding thresholding logic is crucial—this is where decisions are made! ✅
• Excitatory inputs encourage the neuron's fire, while inhibitory inputs can spoil the party! 🎉🚫
• The neuron divides our inputs into 'positive' and 'negative' zones for decision making! 🌗
• Explore the power and limitations of neural function in Boolean operations! ⚙️
• Dive into linear separability and watch how these neurons shine—or dim! 🌈💡

Overview

The McCulloch Pitts Neuron model transforms the complexity of biological brains into a computable form, using binary systems to make it easier to process. This 1943 idea by McCulloch and Pitts draws on neuroscience and logic, helping computers start mimicking neural processing.

This fascinating model simplifies the cognitive process by using binary inputs (like simple yes/no questions) and producing binary outputs. Depending on the aggregation of these inputs, a threshold determines if the neuron 'fires'—essentially, making a decision based on pre-set conditions.

As the lecture progresses, viewers are introduced to the crucial concept of linear separability, describing how some Boolean functions can be sorted and executed by the neuron model. The conversation sparks further exploration into cases where Boolean functions might not be easily separable, posing exciting challenges for computational developments.

Chapters

• 00:00 - 00:30: Introduction to McCulloch-Pitts Neuron The chapter 'Introduction to McCulloch-Pitts Neuron' serves as a transition from the discussion of biological neurons to the first computational neuron model known as the McCulloch-Pitts neuron. The introduction provides context by building on the historical overview of neuroscience covered in previous sections.
• 00:30 - 01:00: Background and History In 1943, two gentlemen named McCollum and Pitts, one a neuroscientist and the other a logician, proposed a highly simplified computational model of the neuron. Their interest was in modeling the brain's neurons, which are capable of complex processing. They aimed to create a computational model that mimics neurons by taking certain inputs and producing outputs, potentially evolving to assist in complex decision-making.
• 01:00 - 02:00: Simplified Neuron Model In this chapter, a simplified neuron model is introduced. It is described as having multiple binary inputs, which can represent various yes/no information, such as weather conditions or personal feelings. These inputs help in making decisions, like whether to watch a movie. The neuron model is divided into two parts: a white part labeled as g, which will be explained later, and an f part.
• 02:00 - 03:00: Input and Output Details The chapter 'Input and Output Details' focuses on the concept of Boolean inputs and outputs in decision-making models. It illustrates this with examples such as determining whether it is raining outside, feeling well, or deciding to go outside. The emphasis is on understanding how these Boolean conditions can be used to drive decisions within a model. The chapter likely includes details on how these inputs and outputs are structurally processed and utilized in practical scenarios.
• 03:00 - 05:00: Aggregation and Decision Process This chapter explains the concept of aggregation and decision processes in the context of a biological neuron example. The 'g' function is responsible for aggregating inputs, essentially summing them up. These inputs are binary (0 or 1) and the maximum sum is reached when all inputs are 1. Subsequently, the 'f' function makes a decision based on the aggregated sum.
• 05:00 - 06:00: Excitatory and Inhibitory Inputs The chapter explains how a neuron fires based on excitatory and inhibitory inputs. It uses an analogy where a neuron fires if at least two out of three input neurons (representing 'is the speech funny?', 'is the visual funny?', and 'is the text funny?') are active. This means the neuron will activate if the sum of inputs is greater than two, as each input can be either zero or one. The decision for the neuron to fire is thus based on the aggregation of these inputs.
• 06:00 - 07:00: Function G and Vector X The chapter discusses the inputs for functions g and f, explaining that these inputs can be either excitatory or inhibitory. The focus is initially on removing inhibitory input, which will not be extensively covered further in the course. Inhibitory input is defined as such that if the input is '1', it means it is engaged or active. Further details about this concept will be elaborated, but the primary emphasis will shift away from inhibitory inputs after their introduction.
• 07:00 - 09:00: Function F and Threshold The chapter discusses the function F and the concept of a threshold in decision-making. It emphasizes that certain inputs can override others, acting as a 'party spoiler.' For example, if one input signifies a high temperature, it can dominate the decision, rendering other considerations such as mood, availability, or weather irrelevant. This illustrates the impact of critical inputs in a decision-making process, where one condition can singularly determine the outcome.
• 09:00 - 10:30: Thresholding Logic This chapter discusses the concept of thresholding logic, focusing on how input values determine output decisions. The example given involves deciding whether to watch a movie based on having a high temperature, defined as anything greater than one. It explains the idea of excitatory inputs, which contribute positively to decision-making when they are activated.
• 10:30 - 15:00: Implementing Boolean Functions The chapter titled 'Implementing Boolean Functions' discusses the behavior of excitatory and inhibitory inputs in Boolean functions. It highlights that if an inhibitory input is present, the output becomes 0, turning off the function. The chapter focuses on functions where there is no inhibitory input, emphasizing that the function g operates as a collective representation of all inputs.
• 15:00 - 24:00: Geometric Interpretation In the chapter titled 'Geometric Interpretation', the concept of a vector 'x' comprising inputs x1 to xn is introduced. The function g is described as a function of this vector, essentially being the sum of all n components of x. The inputs to this vector can either be 0 or 1, making the sum equivalent to counting the number of inputs that are 'on' (i.e., equal to 1). Thus, 'g' represents this sum, while the chapter briefly introduces the notion of function 'f' without detailing it in the provided transcript.

McCulloch Pitts Neuron and Thresholding Logic Transcription

• 00:00 - 00:30 [Music] uh hi so let's start with the next module now we are moving on from biological neurons and we'll be talking about the first neuron model the mecolic pits neuron right so again we had covered this briefly during the history
• 00:30 - 01:00 that in 1943 uh these two uh gentlemen mccollum pitts one was a neuroscientist the other logician who proposed a highly simplified computational model of the neuron right and the interest as i said was that our brain has neurons and they are capable of fairly complex processing so can we come up with something which tries to mimic it has a computational model of that which takes certain inputs and produces certain outputs and then maybe slowly evolves to kind of help us take complex decisions right so that's the
• 01:00 - 01:30 idea and they come up with a very simplified model where you have a bunch of inputs these inputs are all binary right so yes no kind of informa inputs it could be is it training outside am i feeling well and so on and based on this i might want to decide whether i want to watch a movie today or not right and this neuron as you can see i've split it into two parts there's a white part and i've labeled it as g now i'll tell you what g is and then there's this f part
• 01:30 - 02:00 and then finally you have the output the output is again a boolean right so right now i'll just focus on the input being boolean and the output is also boolean and i gave you an example of boolean inputs which was is is it raining outside am i feeling well do i feel like going outside and so on and these are the inputs and based on that i want to take a certain decision right now what this model does
• 02:00 - 02:30 is very simple and it just follows what we are trying to show in the neuron example biological neuron example it aggregates all the inputs right so there are these n inputs it aggregates all of them so all that the g function does is a simple aggregation and you could just think of it as a summation it's just summing up all the inputs and these inputs can either be zero or one so at max you could have all the inputs which are one so that would just be a sum of those ones right and then f takes a decision based on
• 02:30 - 03:00 this aggregation right so remember one of the cartoon examples i was showing was that this neuron will fire only if two out of the at least two out of the three input neurons were uh firing which was is the speech funny is the visual funnies the text funny if any two of three these three fire then this will fire which is the same as saying if the sum of the inputs is greater than two because the inputs are either going to be zero or one then the neuron will fire so this f is going to take a decision based on the aggregation right
• 03:00 - 03:30 i've still not completely defined g and f but we will do that so these inputs that you have they can be excitatory or inhibitory right so first i'll just want to get rid of this inhibitory input i'll mention it a bit in detail on now and then we'll not see it again uh mostly in the course so it may be a bit on the next slide but after that we can see it so inhibitory inputs are such that if that input is one that means if that out input is on
• 03:30 - 04:00 then it doesn't matter what the other inputs are the output is always going to be zero right so it's a bit of a uh kind of party spoiler kind of a input and you could think of it so if i'm taking a decision and if the uh input is do i have high temperature right if that input is on then irrespective of whether i'm in the mood to go for a movie whether i'm free today whether there's a good movie outside whether it's raining not raining it doesn't matter if this
• 04:00 - 04:30 input is on my output is going to be zero i'm not feeling i have a high do i have a high temperature when i define higher temperature as anything greater than one not two then i'm not going to go to watch a movie so if i don't have any enable tree input right so none of these inputs are kind of party spoilers they're all uh positive inputs in the sense that helping me take a positive decision so those are the excitatory inputs which if on are good right and so we'll mainly
• 04:30 - 05:00 be talking about excitatory inputs so if there is any uh inhibitory input right so we will have uh y equal to 0 that means the output will be off if the input is inhibitory right now that i have already covered now i want to talk about the case if you don't have any inhibitory input so in that case your function g which is a function of all the inputs right this is important that is a function of all the inputs and i'm going to collectively call this
• 05:00 - 05:30 vector as x n so you have this collection of inputs x 1 to x n and i'm just compactly calling it as x so g is a function of all these inputs or you could think of g is a function of a vector x which has many components n components and it's simply going to be the sum of all these components that's what i was saying so these inputs could either be 0 or 1 so this sum is essentially the number of inputs which are on right the number of inputs which have one that's what the sum is going to be right that's what g is and then what is f so f
• 05:30 - 06:00 takes in g as input right so that's what was written here function f takes a decision based on this aggregation so what is the aggregation the aggregation is essentially g of x that's the aggregation so it's a function of that and it will be one let me just get rid of things on the slide yeah so it will be one if g of x is greater than certain threshold right so again going back to that example the biological neuron would fire if at least two out of the three neurons was on right so what that means is that
• 06:00 - 06:30 if i had n neurons and if my threshold is two that means f will be one if at least two of the n inputs is uh uh on right and you could think of it as this you have multiple inputs based on this which you are taking a decision so every at least two of these inputs are favorable that one input could be it's not raining outside so it's favorable the other input could be that there's a very good movie which has come out in the theater so these two inputs are on then you would go for a movie so at least you want two things to be good and
• 06:30 - 07:00 this could be higher right so if you have if you are a lazy person and if you go out only for very specific things then you might want many conditions to be in place for you to make that decision in which case this threshold would be high right so it doesn't matter what the threshold is what matters is the way in which you are taking decisions which is that you have a certain number of inputs only if certain number of that is positive or is on then only you will take the output as one otherwise your output would be zero right so that's all
• 07:00 - 07:30 that this f does right so this theta is called the thresholding parameter and i said it could depend uh it could be different for different cases it could depend on the data that you have so if you're a lazy person your data might be high if you are like an outgoing person who just looks for any excuse to go out your data would be very low right so this entire thing is called the thresholding logic where you aggregate the inputs then you have a theta and then based on that theta the
• 07:30 - 08:00 threshold you decide whether the output would be one or zero okay so now uh let us implement some uh boolean functions using this mecolic bits neuron right so y boolean functions because uh the model itself only takes boolean inputs and gives you boolean output so that's by definition a boolean function right so you have f uh of g of x right so this f is going to be boolean that means it's the output is going to be zero or
• 08:00 - 08:30 one and all your x's are also zero or one right so you have boolean inputs boolean outputs so this is a boolean function right now what does it mean to implement a boolean function right and this is going to be uh important uh theme in the course what do you mean by you're trying to represent or implement a function all it means is that if i take a standard boolean function right let me take the input function x of a boolean function of two inputs x1 x2 right and let me call it as f of x1
• 08:30 - 09:00 comma x2 okay so i know that if suppose i'm talking about the and function then this is what it looks like right uh that it gives zero for certain inputs and one for certain inputs when i say i want to have a mecolic pits neuron or say any neural network or in general of a computational model which is trying to implement or represent this function what it means is that when i take these
• 09:00 - 09:30 inputs and pass it through that model right the outputs would be the same as what is expected from that boolean function right so this from this boolean function i expect the outputs to be zero in certain cases and one in certain case in fact i mean if the one will appear only if both the inputs are one so if my mechanic pits neuron can actually if i say that it implements this function then it means that when i pass the input 0 0 here the output should be 1. if i pass sorry the output
• 09:30 - 10:00 should be 0 0 1 should be 0 1 0 should be 0 and 1 1 should be one so if it can actually give me the same outputs as the original function then i say that this mechanic pits neuron has implemented that function or is representing that function right so let's let's delve a bit deeper into that so here's a mechanic bits neuron that's the basic representation where you have some inputs here for simplicity i have taken three inputs then you have a threshold and then you have the output zero one so now if i want to
• 10:00 - 10:30 implement the and function so i have three inputs this is not in my control it's just a function of three inputs and the output is going to be boolean so this is fine so i have not i've drawn this as it is and i'm not asking you about what should i do but i'm asking you about what the theta should be right because i have not put in theta so now if this mecolic pits neuron here is going to implement the and function then what should the value of theta be
• 10:30 - 11:00 so the function should give me an output one only if all the three inputs are one right so if all the three inputs are one then what would g of x be it would be 3 right and i want f f of g of x to be equal to 1 only when g of x is greater than equal to theta so now what
• 11:00 - 11:30 should the theta be i want this to be 1 only if g of x is greater than equal to theta so i should set the theta to be 3 right so if i set the theta to be 3 then this function would give me one only when all the three inputs are one if any one input is zero right let's look at the case of one one zero so in that case my g of x would be two which is less than the threshold and my output would be zero which is exactly what i expect from the and function so if i come up with a mecolic pits neuron
• 11:30 - 12:00 where there are three boolean inputs one boolean output and the threshold is set to three then actually i've implemented the ion function because this function or this neuron will give me the exact same outputs as i expect from the and function right similarly now you could see for the or function your function the output should be one if any of the inputs is one right so i've given three inputs if the sum of the inputs is 1 then the output should be 1 or greater than equal to 1 the output should be 1.
• 12:00 - 12:30 that means the theta should be 1 and now you can see if i give the input as 0 0 0 the sum is going to be 0 which is less than the threshold so the output would be 0 in every other case the output would be 1 as is what is expected from the or function right similarly i'll not go into the details of the nor function [Music] and the not function you can look at it independently but here just notice that this inputs i have made them as inhibitory
• 12:30 - 13:00 uh so let me just explain here right so this is an inhibitory input so what that means is that if this input is on okay if it's one then the output is going to be zero and that's exactly what the not function does and when the input is zero i want the output to be one and so remember it's two step right so if if there is an inhibitory input then the output is going to be if the inhibitory input is on then the output is going to be zero that is
• 13:00 - 13:30 satisfied right so whenever there is one the output is going to be 0 but if the inhibitory input is not on that means if it is 0 then what do i want i want the output to be 1. so in that case i will just set the threshold to 0 right and then now since the sum which is trivially zero here is greater than equal to the threshold which is zero so the neuron will fire right so this i had to do because this is the not function where if something is on i want the
• 13:30 - 14:00 output to be off and hence an inhibitory input makes sense here and similarly now you can go back and check why the threshold zero makes sense for the north function right so that's that's how this is so now the question is can any boolean function be represented using a mechanic bits unit right and to answer this question i think we have been just in this arithmetic world right where we are just taking g of x and then applying some threshold and seeing what the output is so to answer this question i think we need to go into a bit of a
• 14:00 - 14:30 geometric interpretation right so let's see what the mcculloch pits neuron is actually doing right so i'll go to this this equation or inequality here right which is x1 plus x2 is greater than equal to 1 right and for a minute i'll ignore the inequality oops i'll ignore the inequality and just replace it by equal to right so now x1
• 14:30 - 15:00 plus x2 equal to 1 is the equation of a line right all of us know this right and this line is already drawn in the figure below as you can see this would be a line which passes through the points 0 1 and 1 0 right because 0 plus 1 equal to 1 minus 0 of course there are many other points on this line but we know that these two points lie on the line right and i only care about since i'm living in a boolean world although this is like
• 15:00 - 15:30 a two-dimensional plane that i've drawn here there are only four points that matter to me zero zero zero one one zero one one because my inputs can only be boolean so the inputs of the form one point one one point two all of them are not possible only boolean inputs are possible so in my boolean world there are only four points and the x1 plus xt equal to one line is the line that i've drawn here right now you can see that what is the mechanic bits neuron doing actually your decision is x one if x one plus x two greater than equal to one
• 15:30 - 16:00 then the output would be one else the output would be zero so essentially the macholic pits neuron is drawing a line such that all points which lie on or above the line right and on or above i'll use this word on or above now but later i'm going to replace it by something else the output is going to be positive right because these are the points for which x1 plus x2 is greater than or equal to 1 and you can see that if i substitute 0 1 1 0 here it will be equal to 1 and when
• 16:00 - 16:30 i substitute 1 1 here the output would be 2 which is greater than 1 right so for these points the mycolic pit's neuron is going to give the output as one because if they are greater than theta theta is equal to one here hence the output would be one and for the point zero zero this is going to be zero which is lesser than the threshold which is one and hence the output would be zero right so all points lying on and above the line the output is going to be one all points lying below the line the output is going to be
• 16:30 - 17:00 zero right a more correct way of saying this is that for all the points which lie in the positive half space of the line positive half space of the line is all the points which satisfy this equation right and negative half space of the line is all the points which satisfy the equation x one plus x two less than one right so the points which line the positive half space of the line which is the shaded portion the output is going to be one and there are only three such points in our boolean world and the output is indeed one for them and for all the points it's like in the
• 17:00 - 17:30 negative half space and there's only one such point zero comma zero the output would indeed be zero for that point right so what the mecolic pits neuron is essentially doing is it's dividing your space of inputs into two halves one which lies in the positive half space and the other which lies in the negative half space and for all the points which lie in the positive half space the output is one and for the other points is zero right so we'll now uh let me just delete this so that people who want to read the slides can read it while i'm talking
• 17:30 - 18:00 so now what we are going to do is that we are going to convince ourselves about this statement that this is actually a line all you are doing is drawing a line and then the points are divided into two halves positive and negative half space let's try to convince ourselves about this if it's not already clear from this example right so let's let's just see so now let's look at the add function right so what's the equation what's the inequality there x 1 plus x 2 greater than equal to 2. so again for a minute i'll forget about the inequality and just say x 1 plus x two equal to two and
• 18:00 - 18:30 that's the dark orange line that has been drawn here right and it makes sense the point one comma one which is one of the four points that i care about actually lies on this line right and that is the only point in my boolean world which will satisfy the equation x 1 plus x 2 greater than equal to 2. if i substitute any other point then it will not satisfy that equation because x 1 plus x 2 would be 0 or 1 in that case which is less than 2. so all these points lie in the negative half
• 18:30 - 19:00 space of the line and the point 1 comma 1 lies in the positive half space of the line so again you have this division of the inputs there are four possible inputs into two half spaces and for one set of points the output would be one and for the other set of points the output would be zero same way you can think of the totology function which would be the threshold there would be zero right because you want it to be always on so even if the input is 0 0 you want the output pd 1 that means if the aggregation is 0 still you want the
• 19:00 - 19:30 output to be 1 which is simply saying that the threshold can be set to 0 so that for 0 0 0 1 1 0 1 1 all four of these the output would be one right and this is the line x one plus x two equal to zero that you see here and again you can see that all the four points in my boolean world they lie in the positive half space or i can lose this above the line here and there's no point which lies below the line so all the points are in this one group and they'll always be one as is expected for the total energy
• 19:30 - 20:00 function right so the main takeaway well before we go to the three point example or three input example not three point three inputs example is that the mecolic pits neuron is essentially drawing a line right and that's a linear decision boundary which is separating the input points into two parts right so that's the main takeaway now what if you have more than two inputs that often in two dimensions it's all makes sense so as you know from the linear algebra course that
• 20:00 - 20:30 uh now instead of a line we'll have a plane right and let's see what that looks like right so for the or function if you take that as an example then 0 comma 0 comma 0 is the only input for which i want the output to be 0 and for all the other 7 inputs i want the output to be one right so now i would before even i show the figure i'm trying to imagine what it should look like it should look like that i have like a plane such that zero comma zero comma zero
• 20:30 - 21:00 lies on one side of the plane and everything else lies on the other side of the plane right that means i would have divided my inputs into two parts such that on one side i have inputs for which the output is zero and there's only one such input and at the other side i have all the inputs which are seven inputs for which the output is one right so let's try to see that so this is again i'm in a boolean world so even though i'm in three dimensional i don't have infinite points i only have eight points zero comma zero zero zero zero all the way up to one one one you know how to construct those eight points
• 21:00 - 21:30 and now i have this plane right which is actually the plane x1 plus x2 plus x3 equal to 1 that's the plane that is there and of course the points 0 0 1 so let me just slide it down oops oh i may not be able to write here okay so the i'll write it later on i let me just be here and you can see that if i have this plane then the point zero comma zero comma
• 21:30 - 22:00 zero is below this plane and all the other points zero zero one all the other seven points i'm not going to call them out they are either on the plane or above the plane right and that makes sense because for the or function as i said the output should be zero only for this one point which is shown in red and it should be one for all the other points right so this triangle that you see is the plane and you can see that there's only one point below the plane and everything else is on or above the plane right and if i want to go back to the
• 22:00 - 22:30 so uh so arithmetically what was it x1 plus x2 plus x3 is equal to 1 right that's the plane that's the blue triangle that you see and you can see that except for the point zero comma zero comma zero any of the seven points will give me an answer which is greater than or equal to one hence the line the positive half space of the line and the point 0 comma 0 comma 0 will give me the sum as 0 which lies in the negative half space of the line right so this
• 22:30 - 23:00 the main point here is that this is not restricted to 2 inputs for n inputs 3 inputs n inputs it will all the analogy would be the same the geometric interpretation would be the same that you'll have in hyperplane and you'll have points lying on one side of the hyperplane and points lying on the other side of the hyperplane right so the story so far is that a single mecopulat pit's neuron can be used to represent boolean functions which are linearly separable so what do i mean by linearly separable i have not defined that yet so linearly separable linear
• 23:00 - 23:30 separability in the case of boolean functions simply means that there exists a plane such that all points for which the output should be zero lie uh on one side of the plane and all points for which the output should be one lie on or the other side of the planet and positive half space and the negative that's all that linear separability means right so that's the story so far that may collect pits neuron can be used to implement such linearly separable boolean functions and later on now the
• 23:30 - 24:00 question is of course what happens if there is a boolean function which is not linearly separable and before that do such boolean functions exist which are not linearly separable so these questions will address as we go along right