Basics of ML AI - Week 2 Challenge

BASICS OF ML & AI - WEEK 2 CHALLENGE

Q1) Calculate all 4 business moments using pen and paper for the below data set.

ANS:

1. First Business Moment - Mean, `mu`

`mu = sum ((X_i)*P(X))`

`mu = (0*0.35)+(1*0.25)+(2*0.15)+(3*0.15)+(4*0.10)`

`mu = 1.4`

2. Second Business Moment - Variance, `sigma^2`

`sigma^2 = sum ((X_i-mu_x)^2*P(X))`

`sigma^2 = (0-1.4)^2*0.35 + (1-1.4)^2*0.25 + (2-1.4)^2*0.15 + (3-1.4)^2*0.15 + (4-1.4)^2*0.10`

`sigma^2 = 0.686 + 0.04 + 0.054 + 0.384 + 0.676`

`sigma^2 = 1.84`

3. Third Business Moment - Skewness, Sk

Sk = `sum (((X_i - mu_x)/sigma)^3*P(X))`

We need to find the standard deviation, `sigma` using the variance value. The standard deviation is the square root of the variance:

Variance = 1.84

Standard Deviation, `sigma = sqrt(1.84)`

`sigma = 1.36`

Sk = `((0-1.4)/1.36)^3*0.35 + ((1-1.4)/1.36)^3*0.25 + ((2-1.4)/1.36)^3*0.15 + ((3-1.4)/1.36)^3*0.15 + ((4-1.4)/1.36)^3*0.10`

Sk = `(-0.38) + (-0.006) + 0.0129 + 0.2442 + 0.6987`

Sk = 0.5698..

Sk = 0.57

4. Fourth Business Moment - Kurtosis, Kt

Kt = `(sum ((X_i - mu_x)/sigma)^4 * P(X)) - 3`

Kt = `(((0-1.4)/1.36)^4 * 0.35 + ((1-1.4)/1.36)^4 * 0.25 + ((2-1.4)/1.36)^4 * 0.15 + ((3-1.4)/1.36)^4 * 0.15 + ((4-1.4)/1.36)^4 * 0.10) - 3`

Kt = `(0.393 + 0.00187 + 0.00568 + 0.287 + 1.3358) - 3`

Kt = `2.02335 - 3`

Kt = -0.97665

Kt = -0.98

Q2) What is the significance of expected value when simple mean (Sum of all observations/number of observations) is already in place?

ANS:

The Expected value of a discrete random variable, X indicates the average value and is denoted as E(X). The expected value for a random variable is its mean. It is the value that you would expect to obtain if you conduct an experiment whose outcomes are represented by the random variable. It is the weighted average of all the possible values where the probabilities of occurrence of the variables are the weights. For a discrete random variable, the probabilities of each random variable are the weights whereas for a continuous random variable, the values of the probability density function are the weights. In Probability theory, the expected value of a random variable is the value you would expect to obtain if the random variable process was repeated for an infinite number of times and the average of those values were taken.

The mean of a data set is exactly the same by definition and concept as the expectation and mainly differs in application. When we have a data set along with its frequencies, then we use the mean formula to calculate the average. By summing up the product of the random variable with its frequency and dividing by the total frequency, we get the average value. If instead, we had a data set with the probabilities of occurrence for each random variable, then we would use the expectation formula to determine the average. Also, if the probabilities for all of the random variables are equal, then we can find the average using the mean directly instead of the expectation. Both are the same conceptually because the frequency of a variable occurring is the same as its probability of occurrence.

For instance, for the above data set we are given the probabilities of the random variable, X. Suppose, we had a sample data of 100 values. Then the frequency of each variable would be the probability multiplied by 100. E.g. for X = 0, the frequency would be 35, X = 1, the frequency would be 25. The probability of each variable was determined by knowing how frequently the variable occurred and divided by the total number of trials. So for a 100 trials, if X = 1 occurred 35 times, its probability would be 0.35 which is how the probability values are obtained in the table.

If we used the frequencies and tried to sum the products of the variable and their respective frequencies, we would get a value of 140. Since the total frequency is 100 (100 trials), then the mean would be 140/100 = 1.4, which is the same as the expectation, E(X).

Q3) Is having skewness in the curve considered to be bad in the analysis?

ANS:

Skewness is defined as the measure of the symmetry (or lack of symmetry) of a data set. Having skewness is neither good nor bad. It simply describes the distribution from a value. Whether skewness is good or bad is dependent on the application of the data.

For a normal distribution, the measures of central tendency (mean, median and mode) are equal to each other whereas for a skewed distribution, the measures are dispersed. For instance, if a distribution is positively skewed, the mean would be found to the right side and conversely to the left side for that of a negatively skewed distribution. This can be undesirable when locating these measures for analysis.

Also, when statistical tests are performed, misleading results can be obtained from data that is highly skewed. In order to overcome this effect, data transformation tools are used to bring the distribution more closely to resemble a normal one. E.g. log transformation is used for positively skewed distributions.

But in general, skewness is neither good nor bad without knowing the application of the data or the purpose for what it was obtained. Different applications have different meanings for skewed or non-skewed data.

A distribution can have a positive skew, no skew or a negative skew or even be undefined. Each instance is shown in the image below with the general shape of the distribution curve.

The above image shows the 3 basic possibilities for the skew in a distribution of data.

A negatively skewed distribution will skewed to the left and most of the values are clustered at the right tail of the distribution. This distribution will have a negative skewness coefficient. A normal distribution will have perfect symmetry and is not skewed either to the left or the right. It will have a skewness coefficient of 0. A positively skewed distribution will be skewed to the right and most of the values are clustered at the left tail of the distribution. This will have a positive skewness coefficient.

The general rule of thumb is that:

If the skewness is between -0.5 and 0.5, the distribution is considered to have no skew (perfect symmetry).

If the skewness is between -1 and -0.5 or 0.5 and 1, the skew of the distribution (whether positive or negative) is moderate.

If the skewness is lower than -1 or higher than 1, then the skew of the distribution (whether positive or negative) is quite high.

Q4)Evaluate the probabilities for continuous normal distribution with given mean = 680 and standard deviation = 31

a)P(X<711)

b)P(X>740)

c)P(600<X<720)

d)P(X=720)

ANS:

First, we need to calculate the z - score. This has the formula:

`z = (x - mu)/sigma`

Where, x is the random variable

Mean - `mu`

Standard deviation - `sigma`

a) P(X < 711):

`z = (711-680)/31`

`z = 1`

Using the Z-score tables, find the corresponding probability for P(Z < 1 )

P(Z < 1) = 0.8413 (this is the area to the left of the z score, since we are finding the probability for x less than 711)

If we use the table of values to the right of the z score, then we would take the corresponding area, 0.3413 and add it to 0.5. The 0.5 represents the 50% of the area that is under the curve up until the mean, 680. So we need to add the area up until the mean and add it to the area up until 711.

This would be:

P(Z < 1) = 0.5 + 0.3413

P(Z < 1) = 0.8413

b) P(X > 740):

P(X > 740) = 1 - P(X < 740)

`z = (740-680)/31`

`z = 1.935`

Using the Z-score tables, find the corresponding probability for P(Z < 1.935)

P(Z < 1.935) = 0.9713 (this is the area to the left of the z score)

We need to find the area to the right of the z - score, because we need the area for when X is greater than 740. So we need to subtract the left score from 1.

P (Z > 1.935) = 1 - P(Z < 1.935)

P (Z > 1.935) = 1 - 0.9713

P (Z > 1.935) = 0.0287

c) P(600 < x < 720):

We need to find the z score for the same range.

When x = 600:

`z = (600-680)/31`

`z = -2.58`

When x = 720:

`z = (720-680)/31`

`z = 1.29`

We need to find the area for the z score less than -2.58 and subtract this from the area of the z score less than 1.29:

P(-2.58 < Z < 1.29) = P(Z < 1.29) - P(Z < -2.58)

P(-2.58 < Z < 1.29) = 0.90320 - 0.00466

P(-2.58 < Z < 1.29) = 0.89854

d) P(X = 720):

`z = (720-680)/31`

`z = 1.29`

We need to find the area at z = 1.29

P(Z = 1.29) = P(1.29 < Z < 1.29)

P(Z = 1.29) = P(Z < 1.29) - P(Z < 1.29)

P(Z = 1.29) = 0.9032 - 0.9032

P(Z = 1.29) = 0.

Q5)Explain the curve on the right side.

ANS:

This curve shows the normal distribution of a random variable. The percentages show the standard deviations from the central value i.e. the mean. The standard deviation is a measure of dispersion and is used to determine how spread apart the values in a distribution/sample are from the mean and from each other. The greater the standard deviation, the more the values deviate from the mean and vice versa. The standard deviation is derived from the variance by finding its square root.

1 standard deviation from the mean is approximately 34.1% to either the left or the right side of the mean. This means that the standard deviation of a normal distribution can lie anywhere within the space of plus or minus 34.1%. So, this means that approximately 68.2% of the values in the distribution fall within 1 standard deviation of the mean.

Similarly, beyond the first standard deviation, 13.6% to the left or to the right of the first standard deviation represent the values that fall within the second standard deviation. This means to say that 95.4% (34.1+34.1+13.6+13.6) of the values in the distribution fall within 2 standard deviations of the mean.

This continues to the third standard deviation and so on with the area under the curve increasing.

For an ideal normal distribution curve, it will have no skewness and so will have perfect symmetry. The mean of this distribution will be found at the centre of the x-axis of this distribution and corresponds to the peak of the distribution. This can be considered as the line of symmetry. For such a distribution, the measures of central tendency i.e. the mean, median and mode are all considered equal. In addition, the total area under this normal distribution curve is 1, because the summation of probabilities of an event is 1.