Lecture 2: Probability and Statistics Part 2

Jason Chen YCHEN148@e.ntu.edu.sg

Copyright Notice

This document is a compilation of original notes by Jason Chen (YCHEN148@e.ntu.edu.sg). Unauthorized reproduction, downloading, or use for commercial purposes is strictly prohibited and may result in legal action. If you have any questions or find any content-related errors, please contact the author at the provided email address. The author will promptly address any necessary corrections.

I. Commonly Used Distributions-Discrete

1. Uniform Distribution $U(a_1,\cdots ,a_m)$$U(a_1, \cdots, a_m)$

This is a discrete uniform distribution where each outcome $a_1,\cdots ,a_m$$a_1, \cdots, a_m$ is equally likely. Let's analyze the components one by one.

1.1 Probability Mass Function (PMF)

The probability mass function (PMF) for the uniform distribution is given by: $\begin{array}{c}p(x)=\{\begin{array}{ll}\frac{1}{m},& x=a_1,\cdots ,a_m\\ 0,& \text{otherwise}\end{array}\end{array}$$p(x) = \begin{cases} \frac{1}{m}, & x = a_1, \cdots, a_m \\ 0, & \text{otherwise} \end{cases}$

Explanation:

• The PMF states that the probability of any specific outcome $x$$x$ (from the set $a_1,\cdots ,a_m$$a_1, \cdots, a_m$) occurring is $\frac{1}{m}$$\frac{1}{m}$.

• If $x$$x$ is not one of the outcomes $a_1,\cdots ,a_m$$a_1, \cdots, a_m$, the probability is $0$$0$.

Proof: The uniform distribution assigns equal probability to each of the $m$$m$ possible outcomes. Since the sum of all probabilities must equal 1: $\sum _{i=1}^{m}p(a_i)=1$$\sum_{i=1}^{m} p(a_i) = 1$; Since each outcome $a_i$$a_i$ has the same probability $\frac{1}{m}$$\frac{1}{m}$: $\sum _{i=1}^m\frac{1}{m}=\frac{m}{m}=1$$\sum_{i=1}^{m} \frac{1}{m} = \frac{m}{m} = 1$. This confirms that $p(x)=\frac{1}{m}$$p(x) = \frac{1}{m}$ is the correct PMF.

1.2 Moment Generating Function (MGF)

The moment generating function (MGF) for the uniform distribution is given by: $\text{mgf}:M(t)=\frac{\sum _{j=1}^m{e}^{a_{j}t}}{m}$$\text{mgf}: M(t) = \frac{\sum_{j=1}^{m} e^{a_j t}}{m}$

Explanation:

The MGF is used to find the moments of the distribution. The first moment (mean) and the second central moment (variance) can be derived from it.

Proof: The MGF is defined as: $M(t)=\mathbb{E}[e^{tx}]=\sum _{j=1}^{m}p(a_j)e^{a_{j}t}=\sum _{j=1}^m\frac{1}{m}{e}^{a_{j}t}=\frac{\sum _{j=1}^m{e}^{a_{j}t}}{m}$$M(t) = \mathbb{E}[e^{tx}] = \sum_{j=1}^{m} p(a_j) e^{a_j t} = \sum_{j=1}^{m} \frac{1}{m} e^{a_j t} = \frac{\sum_{j=1}^{m} e^{a_j t}}{m}$

1.3 Mean (Expected Value)

The mean (or expected value) $\mu$$\mu$ is given by: $\mu =\mathbb{E}[X]=\frac{\sum _{j=1}^m{a}_j}{m}\equiv \overline{a}$$\mu = \mathbb{E}[X] = \frac{\sum_{j=1}^{m} a_j}{m} \equiv \bar{a}$

Explanation:

The mean of a uniform distribution is simply the average of all possible outcomes.

Proof: The expected value is calculated as: $\mu =\mathbb{E}[X]=\sum _{j=1}^{m}p(a_j)a_j=\sum _{j=1}^m\frac{1}{m}{a}_j=\frac{\sum _{j=1}^m{a}_j}{m}$$\mu = \mathbb{E}[X] = \sum_{j=1}^{m} p(a_j) a_j = \sum_{j=1}^{m} \frac{1}{m} a_j = \frac{\sum_{j=1}^{m} a_j}{m}$

1.4 Variance

The variance ${\sigma }^2$$\sigma^2$ is given by: ${\sigma }^2=\frac{\sum _{j=1}^m(a_j-\overline{a}{)}^2}{m}$$\sigma^2 = \frac{\sum_{j=1}^{m} (a_j - \bar{a})^2}{m}$

Explanation:

Variance measures the spread of the outcomes around the mean. It is the average of the squared differences from the mean.

Proof: Variance is calculated as: ${\sigma }^2=\mathbb{E}[(X-\mu {)}^2]=\mathbb{E}[X^2]-{\mu }^2$$\sigma^2 = \mathbb{E}[(X - \mu)^2] = \mathbb{E}[X^2] - \mu^2$

Where: $\mathbb{E}[X^2]=\sum _{j=1}^{m}p(a_j)a_j^2=\frac{\sum _{j=1}^m{a}_j^2}{m}$$\mathbb{E}[X^2] = \sum_{j=1}^{m} p(a_j) a_j^2 = \frac{\sum_{j=1}^{m} a_j^2}{m}$; Therefore, the variance becomes: ${\sigma }^2=\frac{\sum _{j=1}^m{a}_j^2}{m}-{\left(\frac{\sum _{j=1}^m{a}_j}{m}\right)}^2=\frac{\sum _{j=1}^m(a_j-\overline{a}{)}^2}{m}$$\sigma^2 = \frac{\sum_{j=1}^{m} a_j^2}{m} - \left(\frac{\sum_{j=1}^{m} a_j}{m}\right)^2 = \frac{\sum_{j=1}^{m} (a_j - \bar{a})^2}{m}$

1.5 Parameter

This indicates that the parameters $a_i$$a_i$ are real numbers, and there are $m$$m$ such parameters.

1.6 Example

Throw a fair die once: A classic example of a uniform distribution is rolling a fair six-sided die, where each face (1 through 6) has an equal probability of $\frac{1}{6}$$\frac{1}{6}$.

2. Bernoulli Distribution $B(p)$$B(p)$

2.1 Probability Mass Function (PMF)

The probability mass function (PMF) for the Bernoulli distribution is given by:

$\begin{array}{c}p(x)=\{\begin{array}{ll}{p}^x(1-p{)}^{1-x},& \text{if }x=0\text{ or }x=1\\ 0,& \text{otherwise}\end{array}\end{array}$$p(x) = \begin{cases} p^x (1 - p)^{1 - x}, & \text{if } x = 0 \text{ or } x = 1 \\ 0, & \text{otherwise} \end{cases}$​

Explanation:

• When $x=1$$x = 1$, the PMF is $p$$p$, representing the probability of success.

• When $x=0$$x = 0$, the PMF is $1-p$$1 - p$, representing the probability of failure.

Proof: The PMF for a Bernoulli random variable can be written as: $p(x)=p^x(1-p{)}^{1-x}$$p(x) = p^x (1 - p)^{1 - x}$

This expression works for both cases:

• If $x=1$$x = 1$, $p(1)=p^1(1-p{)}^0=p$$p(1) = p^1 (1 - p)^0 = p$.

• If $x=0$$x = 0$, $p(0)=p^0(1-p{)}^1=1-p$$p(0) = p^0 (1 - p)^1 = 1 - p$.

The sum of the probabilities for all possible outcomes must equal 1: $p(1)+p(0)=p+(1-p)=1$$p(1) + p(0) = p + (1 - p) = 1$

2.2 Moment Generating Function (MGF)

The moment generating function (MGF) for the Bernoulli distribution is given by: $\text{mgf}:M(t)=p{e}^t+1-p$$\text{mgf}: M(t) = p e^t + 1 - p$

Explanation:

• The MGF helps us find the moments of the distribution, like the mean and variance.

Proof: The MGF is defined as: $M(t)=\mathbb{E}[e^{tx}]=p{e}^{t\cdot 1}+(1-p)e^{t\cdot 0}=p{e}^t+(1-p)$$M(t) = \mathbb{E}[e^{tx}] = p e^{t \cdot 1} + (1 - p) e^{t \cdot 0} = p e^t + (1 - p)$. This function can be used to derive the mean and variance of the distribution.

2.3 Mean (Expected Value)

The mean (or expected value) $\mu$$\mu$ for the Bernoulli distribution is: $\mu =\mathbb{E}[X]=p$$\mu = \mathbb{E}[X] = p$

Explanation:

The mean represents the probability of success in a Bernoulli trial.

Proof: The expected value is calculated as: $\mu =\mathbb{E}[X]=p\cdot 1+(1-p)\cdot 0=p$$\mu = \mathbb{E}[X] = p \cdot 1 + (1 - p) \cdot 0 = p$

2.4 Variance

The variance ${\sigma }^2$$\sigma^2$ for the Bernoulli distribution is: ${\sigma }^2=p(1-p)$$\sigma^2 = p(1 - p)$

Explanation:

The variance measures the spread of the outcomes around the mean. It reaches its maximum when $p=0.5$$p = 0.5$, indicating the highest uncertainty.

Proof: Variance is calculated using the formula: ${\sigma }^2=\mathbb{E}[X^2]-{\mu }^2$$\sigma^2 = \mathbb{E}[X^2] - \mu^2$

Since $X^2=X$$X^2 = X$ for $X$$X$ being either 0 or 1: $\mathbb{E}[X^2]=p$$\mathbb{E}[X^2] = p$

Thus, the variance becomes: ${\sigma }^2=p-p^2=p(1-p)$$\sigma^2 = p - p^2 = p(1 - p)$

2.5 Parameter

$p\in [0,1]$$p \in [0, 1]$: $p$$p$ is the probability of success in a Bernoulli trial and lies between 0 and 1.

2.6 Example

Toss a coin once, $p$$p$ = probability that head occurs: A typical example is a coin toss, where $p$$p$ represents the probability of getting heads.

If $A$$A$ is an event, then the indicator random variable $I_A$$I_A$ follows the Bernoulli distribution.

An indicator variable $I_A$$I_A$ takes the value 1 if event $A$$A$ occurs and 0 otherwise, which follows a Bernoulli distribution with parameter $p=\mathbb{P}(A)$$p = \mathbb{P}(A)$.

3. Binomial Distribution ( B(n, p) )

The binomial distribution describes the number of successes in $n$$n$ independent Bernoulli trials, where each trial has a probability $p$$p$ of success. The parameters $n$$n$ and $p$$p$ define the distribution.

3.1 Probability Mass Function (PMF)

The probability mass function (pmf) for the binomial distribution is given by:

$\begin{array}{c}p(x)=\{\begin{array}{ll}(\genfrac{}{}{0ex}{}{n}{x})p^x(1-p{)}^{n-x},& x=0,1,\dots ,n\\ 0,& \text{otherwise}\end{array}\end{array}$$p(x) = \begin{cases} \binom{n}{x} p^x (1 - p)^{n - x}, & x = 0, 1, \dots, n \\ 0, & \text{otherwise} \end{cases}$

Explanation:

• $(\genfrac{}{}{0ex}{}{n}{x})$$\binom{n}{x}$ is the binomial coefficient, representing the number of ways to choose $x$$x$ successes out of $n$$n$ trials.

• $p^x$$p^x$ is the probability of $x$$x$ successes.

• $(1-p{)}^{n-x}$$(1 - p)^{n - x}$ is the probability of $n-x$$n - x$ failures.

Proof: The pmf can be derived as the product of the probability of a specific sequence of $x$$x$ successes and $n-x$$n - x$ failures, multiplied by the number of such sequences:

$p(x)=(\genfrac{}{}{0ex}{}{n}{x})p^x(1-p{)}^{n-x}$$p(x) = \binom{n}{x} p^x (1 - p)^{n - x}$ Where $(\genfrac{}{}{0ex}{}{n}{x})=\frac{n!}{x!(n-x)!}$$\binom{n}{x} = \frac{n!}{x!(n-x)!}$ is the binomial coefficient.

3.2 Moment Generating Function (MGF)

The moment generating function (MGF) for the binomial distribution is:

$\text{mgf}:M(t)={(p{e}^t+1-p)}^n$$\text{mgf}: M(t) = \left(p e^t + 1 - p\right)^n$

Explanation:

• The MGF is a powerful tool for finding the moments of the distribution, such as the mean and variance.

Proof: The MGF for a binomial distribution can be derived by recognizing that the binomial distribution is the sum of $n$$n$ independent Bernoulli random variables:

$M(t)=\mathbb{E}\left[e^{tX}\right]=\mathbb{E}\left[\prod _{i=1}^n{e}^{t{X}_i}\right]={\left(\mathbb{E}\left[e^{t{X}_i}\right]\right)}^n={(p{e}^t+1-p)}^n$$M(t) = \mathbb{E}\left[e^{tX}\right] = \mathbb{E}\left[\prod_{i=1}^{n} e^{tX_i}\right] = \left(\mathbb{E}\left[e^{tX_i}\right]\right)^n = \left(p e^t + 1 - p\right)^n$; Here, $X_i$$X_i$ represents the individual Bernoulli trials.

3.3 Mean (Expected Value)

The mean (or expected value) $\mu$$\mu$ for the binomial distribution is:

$\mu =\mathbb{E}[X]=np$$\mu = \mathbb{E}[X] = np$

Explanation:

The mean represents the expected number of successes in $n$$n$ trials.

Proof: The mean can be derived from the sum of the expected values of $n$$n$ Bernoulli trials:

$\mu =\mathbb{E}\left[\sum _{i=1}^n{X}_i\right]=\sum _{i=1}^n\mathbb{E}[X_i]=np$$\mu = \mathbb{E}\left[\sum_{i=1}^{n} X_i\right] = \sum_{i=1}^{n} \mathbb{E}[X_i] = np$

3.4 Variance

The variance ${\sigma }^2$$\sigma^2$ for the binomial distribution is: ${\sigma }^2=np(1-p)$$\sigma^2 = np(1 - p)$

Explanation:

The variance measures the spread of the number of successes around the mean.

Proof: The variance of a sum of independent Bernoulli trials is the sum of the variances of the individual trials:

${\sigma }^2=\mathbb{V}\left[\sum _{i=1}^n{X}_i\right]=\sum _{i=1}^n\mathbb{V}[X_i]=np(1-p)$$\sigma^2 = \mathbb{V}\left[\sum_{i=1}^{n} X_i\right] = \sum_{i=1}^{n} \mathbb{V}[X_i] = np(1 - p)$

Where $\mathbb{V}[X_i]=p(1-p)$$\mathbb{V}[X_i] = p(1 - p)$ is the variance of a single Bernoulli trial.

3.5 Parameter

Parameter $p\in [0,1]$$p \in [0, 1]$ and $n=1,2,\dots$$n = 1, 2, \dots$:

• $p$$p$ is the probability of success in a single trial.

• $n$$n$ is the number of trials.

3.6 Example

Number of heads when tossing a coin $n$$n$ times: If you toss a fair coin $n$$n$ times, the number of heads (successes) follows a binomial distribution with $n$$n$ trials and $p=0.5$$p = 0.5$.

• Binomial Theorem: The formula $(a+b{)}^n=\sum _{x=0}^n(\genfrac{}{}{0ex}{}{n}{x})a^x{b}^{n-x}$$(a + b)^n = \sum_{x=0}^{n} \binom{n}{x} a^x b^{n-x}$ underpins the binomial distribution.

• Application in Finance: The binomial distribution is commonly used in security price modeling, assuming that the price can either rise or fall by a fixed amount during small intervals of time.

Let's break down the problem presented in the slide, which deals with a one-period binomial model for stock pricing. We'll work through the problem step by step to ensure we have the correct solution.

Problem Statement

We are dealing with a stock market model at time $t=0$$t = 0$ where there are two investment opportunities:

Purchase and (short-) selling of stocks:

• Current price: $P_1(0)=p_1>0$$P_1(0) = p_1 > 0$

• Future price $X(T)$$X(T)$ at time $T$$T$: $\begin{array}{c}X(T)=\{\begin{array}{ll}f\cdot e^{rT}+g\cdot p_1\cdot u,& \text{with probability }p\\ f\cdot e^{rT}+g\cdot p_1\cdot d,& \text{with probability }1-p\end{array}\end{array}$$X(T) = \begin{cases} f \cdot e^{rT} + g \cdot p_1 \cdot u, & \text{with probability } p \\ f \cdot e^{rT} + g \cdot p_1 \cdot d, & \text{with probability } 1 - p \end{cases}$​

• $u>d$$u > d$: Here, $u$$u$ and $d$$d$ represent the up and down factors of the stock price.

• $(f,g)$$(f, g)$ is the trading strategy:

  • $f$$f$: The face amount invested.

  • $g$$g$: The number of stocks kept at $t=0$$t = 0$. If $g$$g$ is negative, it indicates a short sale.

Fixed deposit or loan:

• Interest rate: $r\ge 0$$r \geq 0$

• Continuous return over time frame $[0,T]$$[0, T]$: $P_0(0)=1,P_0(T)=e^{rT}$$P_0(0) = 1, \quad P_0(T) = e^{rT}$

• This represents the capital development of a monetary unit.

Step-by-Step Solution

Analyzing the Trading Strategy

The future value $X(T)$$X(T)$ at time $T$$T$ is dependent on the trading strategy $(f,g)$$(f, g)$:

• If the stock price moves up (with probability $p$$p$): $X(T)=f\cdot e^{rT}+g\cdot p_1\cdot u$$X(T) = f \cdot e^{rT} + g \cdot p_1 \cdot u$

• If the stock price moves down (with probability $1-p$$1 - p$): $X(T)=f\cdot e^{rT}+g\cdot p_1\cdot d$$X(T) = f \cdot e^{rT} + g \cdot p_1 \cdot d$

• $f$$f$ represents the amount of money invested in the risk-free asset (e.g., bond).

• $g$$g$ represents the amount of stock held (positive $g$$g$) or shorted (negative $g$$g$).

Risk-Neutral Valuation

In a one-period binomial model, the price of a derivative or a contingent claim can be computed using risk-neutral valuation. The risk-neutral probability $q$$q$ is defined as: $q=\frac{e^{rT}-d}{u-d}$$q = \frac{e^{rT} - d}{u - d}$

The expected value of the future price $X(T)$$X(T)$ under the risk-neutral measure is: ${\mathbb{E}}^{\mathbb{Q}}[X(T)]=q\cdot (f\cdot e^{rT}+g\cdot p_1\cdot u)+(1-q)\cdot (f\cdot e^{rT}+g\cdot p_1\cdot d)$$\mathbb{E}^\mathbb{Q}[X(T)] = q \cdot (f \cdot e^{rT} + g \cdot p_1 \cdot u) + (1 - q) \cdot (f \cdot e^{rT} + g \cdot p_1 \cdot d)$

The present value $X(0)$$X(0)$ at time $t=0$$t = 0$ should equal the discounted expected future value: $X(0)=\frac{1}{e^{rT}}{\mathbb{E}}^{\mathbb{Q}}[X(T)]$$X(0) = \frac{1}{e^{rT}} \mathbb{E}^\mathbb{Q}[X(T)]$

Given $P_0(0)=1$$P_0(0) = 1$ and $P_0(T)=e^{rT}$$P_0(T) = e^{rT}$, the price at time $t=0$$t = 0$ becomes: $X(0)=\frac{q\cdot f\cdot e^{rT}+q\cdot g\cdot p_1\cdot u+(1-q)\cdot f\cdot e^{rT}+(1-q)\cdot g\cdot p_1\cdot d}{e^{rT}}$$X(0) = \frac{q \cdot f \cdot e^{rT} + q \cdot g \cdot p_1 \cdot u + (1 - q) \cdot f \cdot e^{rT} + (1 - q) \cdot g \cdot p_1 \cdot d}{e^{rT}}$

Simplifying this expression gives the initial capital required.

Given that the number of upward movements in the stock price follows a $B(1,p)$$B(1, p)$-distributed random variable, it means:

• With probability $p$$p$, the stock price moves up.

• With probability $1-p$$1 - p$, the stock price moves down.

4. Poisson Distribution $P(\lambda )$$P(\lambda)$ - Limit of Binomial Distribution

Certainly! Let's go through the content in the previous two images step by step, providing detailed explanations and proofs for each component, similar to how we did with the Poisson distribution.

Understanding the Transition from Binomial to Poisson Distribution

Step 1: Start with the Binomial PMF

The binomial distribution for a random variable $X_n\sim B(n,p_n)$$X_n \sim B(n, p_n)$ is given by: $P(X_n=x)=(\genfrac{}{}{0ex}{}{n}{x})p_n^x(1-p_n{)}^{n-x}$$P(X_n = x) = \binom{n}{x} p_n^x (1 - p_n)^{n - x}$

• $n$$n$ is the number of trials

• $p_n$$p_n$ is the probability of success in each trial

• $x$$x$ is the number of successes in $n$$n$ trials

Step 2: Transition to Poisson

Consider the case where $n$$n$ becomes very large ($n\to \mathrm{\infty }$$n \to \infty$) and $p_n$$p_n$ becomes very small ($p_n\to 0$$p_n \to 0$) such that $\lambda =n{p}_n$$\lambda = n p_n$ remains constant. In this scenario, the binomial distribution can be approximated by the Poisson distribution.

We start by rewriting the binomial pmf using this condition: $P(X_n=x)=(\genfrac{}{}{0ex}{}{n}{x}){\left(\frac{\lambda }{n}\right)}^x{(1-\frac{\lambda }{n})}^{n-x}$$P(X_n = x) = \binom{n}{x} \left(\frac{\lambda}{n}\right)^x \left(1 - \frac{\lambda}{n}\right)^{n - x}$

• The binomial coefficient $(\genfrac{}{}{0ex}{}{n}{x})$$\binom{n}{x}$ is given by: $(\genfrac{}{}{0ex}{}{n}{x})=\frac{n(n-1)\dots (n-x+1)}{x!}$$\binom{n}{x} = \frac{n(n-1)\dots(n-x+1)}{x!}$

  As $n$$n$ becomes large, this simplifies to: $\frac{n(n-1)\dots (n-x+1)}{n^x}\approx 1$$\frac{n(n-1)\dots(n-x+1)}{n^x} \approx 1$

• The probability of success term $p_n^x$$p_n^x$ becomes: ${\left(\frac{\lambda }{n}\right)}^x$$\left(\frac{\lambda}{n}\right)^x$

• The term $(1-p_n{)}^{n-x}$$(1 - p_n)^{n-x}$ can be approximated using the exponential function: ${(1-\frac{\lambda }{n})}^{n-x}\approx e^{-\lambda }$$\left(1 - \frac{\lambda}{n}\right)^{n-x} \approx e^{-\lambda}$

Thus, the binomial pmf transitions to the Poisson pmf as: $\underset{n\to \mathrm{\infty }}{lim}P(X_n=x)=\frac{{\lambda }^x{e}^{-\lambda }}{x!}$$\lim_{n \to \infty} P(X_n = x) = \frac{\lambda^x e^{-\lambda}}{x!}$

This is the Poisson distribution with parameter $\lambda =n{p}_n$$\lambda = np_n$​.

The formula shown in the second image is a well-known limit in calculus:

$\underset{n\to \mathrm{\infty }}{lim}{(1+\frac{a_n}{n})}^n=e^a$$\lim_{n \to \infty} \left(1 + \frac{a_n}{n}\right)^n = e^{a}$ This result is a fundamental identity used in the derivation of the exponential function and is central to many proofs in probability and calculus.

Explanation:

• Intuition: This limit expresses the idea that as $n$$n$ becomes very large, the expression ${(1+\frac{a_n}{n})}^n$$\left(1 + \frac{a_n}{n}\right)^n$ approaches $e^a$$e^a$. The parameter $a_n$$a_n$ converges to some constant $a$$a$ as $n$$n$ increases.

• Connection to Exponentials: This result shows how exponential growth can be approximated by a sequence of compounding steps. The expression ${(1+\frac{a_n}{n})}^n$$\left(1 + \frac{a_n}{n}\right)^n$ is a discrete approximation of the continuous exponential function.

• Applications: This limit is often used in proofs related to the convergence of binomial distributions to Poisson distributions, the derivation of compound interest formulas, and the calculation of certain types of sums and integrals.

4.1 Probability Mass Function (PMF):

$p(x)=\frac{{\lambda }^x{e}^{-\lambda }}{x!},x=0,1,2,\dots$$p(x) = \frac{\lambda^x e^{-\lambda}}{x!}, \quad x = 0, 1, 2, \dots$ This is derived as shown above, representing the probability of $x$$x$ events occurring in a fixed interval.

4.2 Moment Generating Function (MGF):

The MGF is defined as: $M_X(t)=\mathbb{E}[e^{tX}]$$M_X(t) = \mathbb{E}[e^{tX}]$. For Poisson distribution, we have: $M_X(t)=\sum _{x=0}^{\mathrm{\infty }}{e}^{tx}\cdot \frac{{\lambda }^x{e}^{-\lambda }}{x!}$$M_X(t) = \sum_{x=0}^{\infty} e^{tx} \cdot \frac{\lambda^x e^{-\lambda}}{x!}$

Simplifying using the series expansion of the exponential function: $M_X(t)=e^{\lambda (e^t-1)}$$M_X(t) = e^{\lambda(e^t - 1)}$

4.3 Mean (Expected Value):

$\mathbb{E}[X]=\lambda$$\mathbb{E}[X] = \lambda$ The mean is simply $\lambda$$\lambda$​, representing the average number of events in the interval.

The expected value (mean) $\mathbb{E}[X]$$\mathbb{E}[X]$ for a Poisson distribution is: $\mathbb{E}[X]=\sum _{x=0}^{\mathrm{\infty }}x\cdot P(X=x)=\sum _{x=0}^{\mathrm{\infty }}x\cdot \frac{{\lambda }^x{e}^{-\lambda }}{x!}$$\mathbb{E}[X] = \sum_{x=0}^{\infty} x \cdot P(X = x) = \sum_{x=0}^{\infty} x \cdot \frac{\lambda^x e^{-\lambda}}{x!}$

By simplifying this using the properties of sums and derivatives: $\mathbb{E}[X]=\lambda$$\mathbb{E}[X] = \lambda$​

4.4 Variance:

$\text{Var}(X)=\lambda$$\text{Var}(X) = \lambda$​ $\mathbb{E}[X^2]=\sum _{x=0}^{\mathrm{\infty }}{x}^2\cdot P(X=x)=\sum _{x=0}^{\mathrm{\infty }}{x}^2\cdot \frac{{\lambda }^x{e}^{-\lambda }}{x!}$$\mathbb{E}[X^2] = \sum_{x=0}^{\infty} x^2 \cdot P(X = x) = \sum_{x=0}^{\infty} x^2 \cdot \frac{\lambda^x e^{-\lambda}}{x!}$

We can rewrite $x^2$$x^2$ as $x(x-1)+x$$x(x-1) + x$: $\mathbb{E}[X^2]=\sum _{x=0}^{\mathrm{\infty }}x(x-1)\cdot \frac{{\lambda }^x{e}^{-\lambda }}{x!}+\sum _{x=0}^{\mathrm{\infty }}x\cdot \frac{{\lambda }^x{e}^{-\lambda }}{x!}$$\mathbb{E}[X^2] = \sum_{x=0}^{\infty} x(x-1) \cdot \frac{\lambda^x e^{-\lambda}}{x!} + \sum_{x=0}^{\infty} x \cdot \frac{\lambda^x e^{-\lambda}}{x!}$

The first term $\sum _{x=0}^{\mathrm{\infty }}x(x-1)\cdot \frac{{\lambda }^x{e}^{-\lambda }}{x!}$$\sum_{x=0}^{\infty} x(x-1) \cdot \frac{\lambda^x e^{-\lambda}}{x!}$ simplifies to: $\sum _{x=2}^{\mathrm{\infty }}\frac{{\lambda }^2\cdot {\lambda }^{x-2}{e}^{-\lambda }}{(x-2)!}={\lambda }^2$$\sum_{x=2}^{\infty} \frac{\lambda^2 \cdot \lambda^{x-2} e^{-\lambda}}{(x-2)!} = \lambda^2$

The second term $\sum _{x=0}^{\mathrm{\infty }}x\cdot \frac{{\lambda }^x{e}^{-\lambda }}{x!}$$\sum_{x=0}^{\infty} x \cdot \frac{\lambda^x e^{-\lambda}}{x!}$ is just the expected value $\mathbb{E}[X]$$\mathbb{E}[X]$, which is $\lambda$$\lambda$.

Thus: $\mathbb{E}[X^2]={\lambda }^2+\lambda$$\mathbb{E}[X^2] = \lambda^2 + \lambda$; Finally, the variance $\text{Var}(X)$$\text{Var}(X)$ is: $\text{Var}(X)=\mathbb{E}[X^2]-(\mathbb{E}[X]{)}^2=({\lambda }^2+\lambda )-{\lambda }^2=\lambda$$\text{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2 = (\lambda^2 + \lambda) - \lambda^2 = \lambda$

4.5 Parameter:

$\lambda >0$$\lambda > 0$: $\lambda$$\lambda$ is the rate parameter, indicating the expected number of occurrences in a given time or space interval.

4.6 Example:

Number of phone calls coming into an exchange during a unit of time:

If $\lambda$$\lambda$ represents the average number of calls per hour, the number of calls in a given hour follows a Poisson distribution with parameter $\lambda$$\lambda$.

Summation of Independent Poisson Random Variables

If $X_i\sim P({\lambda }_i)$$X_i \sim P(\lambda_i)$ are independent, then their sum: $Y=X_1+\cdots +X_k\sim P({\lambda }_1+\cdots +{\lambda }_k)$$Y = X_1 + \dots + X_k \sim P(\lambda_1 + \dots + \lambda_k)$

This means the sum of independent Poisson-distributed variables is also Poisson-distributed, with the rate parameter being the sum of individual rate parameters.

Proof of the Summation of Independent Poisson Random Variables

Statement: If $X_i\sim P({\lambda }_i)$$X_i \sim P(\lambda_i)$ are independent Poisson random variables, then their sum $Y=X_1+\cdots +X_k$$Y = X_1 + \dots + X_k$ is also Poisson-distributed with rate parameter $\lambda ={\lambda }_1+\cdots +{\lambda }_k$$\lambda = \lambda_1 + \dots + \lambda_k$.

Definition of Poisson Distribution: A random variable $X$$X$ is said to follow a Poisson distribution with rate parameter $\lambda$$\lambda$ if its probability mass function (PMF) is given by: $P(X=x)=\frac{{\lambda }^x{e}^{-\lambda }}{x!},x=0,1,2,\dots$$P(X = x) = \frac{\lambda^x e^{-\lambda}}{x!}, \quad x = 0, 1, 2, \dots$

2. Moment Generating Function (MGF): The moment generating function (MGF) of a Poisson random variable $X\sim P(\lambda )$$X \sim P(\lambda)$ is: $M_X(t)=\mathbb{E}[e^{tX}]=\sum _{x=0}^{\mathrm{\infty }}{e}^{tx}\frac{{\lambda }^x{e}^{-\lambda }}{x!}=e^{\lambda (e^t-1)}$$M_X(t) = \mathbb{E}[e^{tX}] = \sum_{x=0}^{\infty} e^{tx} \frac{\lambda^x e^{-\lambda}}{x!} = e^{\lambda(e^t - 1)}$

3. MGF of the Sum: Let $Y=X_1+X_2+\cdots +X_k$$Y = X_1 + X_2 + \dots + X_k$. The MGF of $Y$$Y$ is the product of the MGFs of the independent $X_i$$X_i$: $M_Y(t)=M_{X_1}(t)\cdot M_{X_2}(t)\cdot \cdots \cdot M_{X_k}(t)$$M_Y(t) = M_{X_1}(t) \cdot M_{X_2}(t) \cdot \dots \cdot M_{X_k}(t)$

Substituting the MGF of each $X_i$$X_i$: $M_Y(t)=e^{{\lambda }_1(e^t-1)}\cdot e^{{\lambda }_2(e^t-1)}\cdot \cdots \cdot e^{{\lambda }_k(e^t-1)}$$M_Y(t) = e^{\lambda_1(e^t - 1)} \cdot e^{\lambda_2(e^t - 1)} \cdot \dots \cdot e^{\lambda_k(e^t - 1)}$

Simplifying the expression: $M_Y(t)=e^{({\lambda }_1+{\lambda }_2+\cdots +{\lambda }_k)(e^t-1)}=e^{\lambda (e^t-1)}$$M_Y(t) = e^{(\lambda_1 + \lambda_2 + \dots + \lambda_k)(e^t - 1)} = e^{\lambda(e^t - 1)}$; where $\lambda ={\lambda }_1+{\lambda }_2+\cdots +{\lambda }_k$$\lambda = \lambda_1 + \lambda_2 + \dots + \lambda_k$.

5. Negative Binomial Distribution $NB(r,p)$$NB(r, p)$

The negative binomial distribution describes the number of trials $X$$X$ required to achieve the $r$$r$th success in a sequence of independent Bernoulli trials, where each trial has a probability $p$$p$ of success.

5.1 Probability Mass Function (PMF)

The PMF for the negative binomial distribution is given by:

$\begin{array}{c}p(x)=\{\begin{array}{ll}(\genfrac{}{}{0ex}{}{x-1}{x-r})p^r(1-p{)}^{x-r},& x=r,r+1,\dots \\ 0,& \text{otherwise}\end{array}\end{array}$$p(x) = \begin{cases} \binom{x-1}{x-r} p^r (1 - p)^{x-r}, & x = r, r+1, \dots \\ 0, & \text{otherwise} \end{cases}$

• $x$$x$ is the total number of trials needed to achieve the $r$$r$th success.

• $p$$p$ is the probability of success on each trial.

• $(1-p)$$(1 - p)$ is the probability of failure on each trial.

Derivation of the PMF:

1. Number of Failures: Before the $r$$r$th success, there must be $x-r$$x - r$ failures.

2. Success-Failure Pattern: The probability of having $x-r$$x - r$ failures and $r$$r$ successes (with the last trial being a success) is: $p(x)=(\genfrac{}{}{0ex}{}{x-1}{r-1})p^r(1-p{)}^{x-r}$$p(x) = \binom{x-1}{r-1} p^r (1-p)^{x-r}$

The binomial coefficient $(\genfrac{}{}{0ex}{}{x-1}{r-1})$$\binom{x-1}{r-1}$ represents the number of ways to arrange $x-1$$x-1$ trials into $r-1$$r-1$ successes and $x-r$$x-r$ failures, followed by the final success.

5.2 Moment Generating Function (MGF)

The MGF $M_X(t)$$M_X(t)$ for the negative binomial distribution is given by: $M_X(t)={\left(\frac{p{e}^t}{1-(1-p)e^t}\right)}^r,\text{for }t<-\log (1-p)$$M_X(t) = \left(\frac{p e^t}{1 - (1-p) e^t}\right)^r, \quad \text{for } t < -\log(1-p)$

Derivation of the MGF:

1. The MGF is defined as: $M_X(t)=\mathbb{E}[e^{tX}]$$M_X(t) = \mathbb{E}[e^{tX}]$

2. By summing over all possible values of $X$$X$: $M_X(t)=\sum _{x=r}^{\mathrm{\infty }}{e}^{tx}(\genfrac{}{}{0ex}{}{x-1}{r-1})p^r(1-p{)}^{x-r}$$M_X(t) = \sum_{x=r}^{\infty} e^{tx} \binom{x-1}{r-1} p^r (1-p)^{x-r}$

3. This series is simplified using the sum of a geometric series, leading to the compact form:

$M_X(t)={\left(\frac{p{e}^t}{1-(1-p)e^t}\right)}^r$$M_X(t) = \left(\frac{p e^t}{1 - (1-p) e^t}\right)^r$; This function is valid as long as $t<-\log (1-p)$$t < -\log(1-p)$ to ensure convergence.

5.3 Mean

The mean $\mathbb{E}[X]$$\mathbb{E}[X]$ of the negative binomial distribution is given by: $\mathbb{E}[X]=\frac{r}{p}$$\mathbb{E}[X] = \frac{r}{p}$

Derivation of the Mean:

From the definition of expectation: $\mathbb{E}[X]=\sum _{x=r}^{\mathrm{\infty }}x\cdot (\genfrac{}{}{0ex}{}{x-1}{r-1})p^r(1-p{)}^{x-r}$$\mathbb{E}[X] = \sum_{x=r}^{\infty} x \cdot \binom{x-1}{r-1} p^r (1-p)^{x-r}$ ; This sum can be calculated directly or derived using the MGF by taking the first derivative with respect to $t$$t$ and evaluating at $t=0$$t = 0$: $\mathbb{E}[X]=\frac{d{M}_X(t)}{dt}{|}_{t=0}=\frac{r}{p}$$\mathbb{E}[X] = \frac{dM_X(t)}{dt} \Bigg|_{t=0} = \frac{r}{p}$

5.4 Variance

The variance $\text{Var}(X)$$\text{Var}(X)$ of the negative binomial distribution is given by: $\text{Var}(X)=\frac{r(1-p)}{p^2}$$\text{Var}(X) = \frac{r(1-p)}{p^2}$

Derivation of the Variance: The variance is given by: $\text{Var}(X)=\mathbb{E}[X^2]-(\mathbb{E}[X]{)}^2$$\text{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$

The second moment $\mathbb{E}[X^2]$$\mathbb{E}[X^2]$ can be calculated by taking the second derivative of the MGF: $\text{Var}(X)=\frac{d^2{M}_X(t)}{d{t}^2}{|}_{t=0}-{\left(\frac{r}{p}\right)}^2$$\text{Var}(X) = \frac{d^2M_X(t)}{dt^2} \Bigg|_{t=0} - \left(\frac{r}{p}\right)^2$ $=\text{Var}(X)=\frac{r(1-p)}{p^2}$$= \text{Var}(X) = \frac{r(1-p)}{p^2}$

5.5 Parameter Range:

• $p\in [0,1]$$p \in [0, 1]$: Probability of success in each trial.

• $r=1,2,\dots$$r = 1, 2, \dots$: The number of successes being targeted.

5.6 Example:

Lottery: If a person must purchase tickets until they achieve the $r$$r$th winning ticket, the total number of tickets bought follows a negative binomial distribution.

6. Hypergeometric Distribution $HG(r,n,m)$$HG(r, n, m)$

6.1 Probability Mass Function (PMF)

The PMF of the hypergeometric distribution is given by: $p(x)=\frac{(\genfrac{}{}{0ex}{}{n}{x})(\genfrac{}{}{0ex}{}{m}{r-x})}{(\genfrac{}{}{0ex}{}{n+m}{r})},x=0,1,\dots ,min(r,n),\text{ and }r-x\le m$$p(x) = \frac{\binom{n}{x} \binom{m}{r-x}}{\binom{n+m}{r}}, \quad x = 0, 1, \dots, \min(r, n), \text{ and } r-x \leq m$

• $n$$n$ is the number of black balls.

• $m$$m$ is the number of white balls.

• $r$$r$ is the total number of balls drawn without replacement.

• $x$$x$ is the number of black balls drawn.

Derivation of the PMF:

1. Combination Calculation: The number of ways to draw $x$$x$ black balls from $n$$n$ black balls is given by $(\genfrac{}{}{0ex}{}{n}{x})$$\binom{n}{x}$. Similarly, the number of ways to draw $r-x$$r-x$ white balls from $m$$m$ white balls is $(\genfrac{}{}{0ex}{}{m}{r-x})$$\binom{m}{r-x}$.

2. Total Combinations: The total number of ways to draw $r$$r$ balls from $n+m$$n + m$ balls (black and white) is $(\genfrac{}{}{0ex}{}{n+m}{r})$$\binom{n+m}{r}$.

3. PMF Expression: The probability of drawing exactly $x$$x$ black balls in $r$$r$ draws is then the ratio of the favorable outcomes to the total outcomes: $p(x)=\frac{(\genfrac{}{}{0ex}{}{n}{x})(\genfrac{}{}{0ex}{}{m}{r-x})}{(\genfrac{}{}{0ex}{}{n+m}{r})}$$p(x) = \frac{\binom{n}{x} \binom{m}{r-x}}{\binom{n+m}{r}}$

6.2 Moment Generating Function (MGF)

The hypergeometric distribution does not have an explicit closed-form moment generating function (MGF). This is because the lack of replacement complicates the analysis of the moments in a way that precludes a simple closed-form expression.

6.3 Mean

The mean $\mathbb{E}[X]$$\mathbb{E}[X]$ of the hypergeometric distribution is given by: $\mathbb{E}[X]=\frac{r\cdot n}{n+m}$$\mathbb{E}[X] = \frac{r \cdot n}{n + m}$

Derivation of the Mean:

1. The mean is derived from the linearity of expectation. Since the hypergeometric distribution models the number of successes (black balls) drawn, the expected value is proportional to the fraction of the total balls that are black.

2. $n$$n$ is the number of black balls, and $n+m$$n + m$ is the total number of balls. Thus, the mean number of black balls drawn is: $\mathbb{E}[X]=r\cdot \frac{n}{n+m}$$\mathbb{E}[X] = r \cdot \frac{n}{n+m}$

6.4 Variance

The variance $\text{Var}(X)$$\text{Var}(X)$ of the hypergeometric distribution is given by: $\text{Var}(X)=\frac{r\cdot n\cdot m\cdot (n+m-r)}{(n+m{)}^2\cdot (n+m-1)}$$\text{Var}(X) = \frac{r \cdot n \cdot m \cdot (n+m-r)}{(n+m)^2 \cdot (n+m-1)}$

Derivation of the Variance:

1. Variance accounts for both the proportion of successes and the finite size of the population.

2. The formula takes into consideration the fact that the draws are without replacement, which introduces negative dependence between the draws (i.e., drawing one black ball decreases the probability of drawing another black ball).

3. The variance formula is derived from the second moment and adjusted for the non-independence of the draws: $\text{Var}(X)=r\cdot \frac{n}{n+m}\cdot \frac{m}{n+m}\cdot \frac{n+m-r}{n+m-1}$$\text{Var}(X) = r \cdot \frac{n}{n+m} \cdot \frac{m}{n+m} \cdot \frac{n+m-r}{n+m-1}$

6.5 Parameters

• $r$$r$: The number of draws.

• $n$$n$: The number of black balls.

• $m$$m$: The number of white balls.

• $r\le n+m$$r \leq n + m$: The number of draws cannot exceed the total number of balls.

6.6 Example:

Sampling Industrial Products: If you want to determine the number of defective items (black balls) in a sample drawn from a batch without replacement, the hypergeometric distribution models the probability of finding a specific number of defective items.

Relationship to Binomial Distribution

The hypergeometric distribution can be related to the binomial distribution as the population size becomes large:

$(\genfrac{}{}{0ex}{}{n}{x})(\genfrac{}{}{0ex}{}{m}{r-x})/(\genfrac{}{}{0ex}{}{n+m}{r})\to (\genfrac{}{}{0ex}{}{r}{x})p^x(1-p{)}^{r-x}$$\binom{n}{x} \binom{m}{r-x} / \binom{n+m}{r} \rightarrow \binom{r}{x} p^x (1-p)^{r-x}$

Where: $p=\frac{n}{n+m}$$p = \frac{n}{n+m}$ is the probability of success in each draw.

This approximation holds under the condition that $n,m\to \mathrm{\infty }$$n, m \to \infty$ while maintaining a fixed ratio.

II. Commonly Used Distributions-Continuous

1. Uniform Distribution $U(a,b)$$U(a, b)$

1.1 Probability Density Function (PDF)

The PDF of the uniform distribution $U(a,b)$$U(a, b)$ is given by: $\begin{array}{c}f(x)=\{\begin{array}{ll}\frac{1}{b-a},& \text{if }a\le x\le b\\ 0,& \text{otherwise}\end{array}\end{array}$$f(x) = \begin{cases} \frac{1}{b-a}, & \text{if } a \leq x \leq b \\ 0, & \text{otherwise} \end{cases}$

Derivation of the PDF:

• Uniformity Condition: Since the distribution is uniform, every point within the interval $[a,b]$$[a, b]$ must have the same probability density. This implies that the density $f(x)$$f(x)$ is constant over this interval.

• Total Probability: The total area under the PDF curve must be 1, ${\int }_a^{b}f(x)dx=1$$\int_a^b f(x) \, dx = 1$

• Solving for $f(x)$$f(x)$: $f(x)\cdot (b-a)=1\Rightarrow f(x)=\frac{1}{b-a}$$f(x) \cdot (b - a) = 1 \quad \Rightarrow \quad f(x) = \frac{1}{b-a}$

• Thus, within the interval $[a,b]$$[a, b]$, the PDF is $\frac{1}{b-a}$$\frac{1}{b-a}$, and outside this interval, the PDF is 0.

1.2 Cumulative Distribution Function (CDF)

The CDF $F(x)$$F(x)$ of the uniform distribution is given by: $\begin{array}{c}F(x)=\{\begin{array}{ll}0,& \text{if }x<a\\ \frac{x-a}{b-a},& \text{if }a\le x\le b\\ 1,& \text{if }x>b\end{array}\end{array}$$F(x) = \begin{cases} 0, & \text{if } x < a \\ \frac{x-a}{b-a}, & \text{if } a \leq x \leq b \\ 1, & \text{if } x > b \end{cases}$

Derivation of the CDF:

• Definition: The CDF is the probability that a random variable $X$$X$ takes a value less than or equal to $x$$x$: $F(x)=P(X\le x)$$F(x) = P(X \leq x)$

• For $x<a$$x < a$: The probability is 0 since $x$$x$ is outside the interval: $F(x)=0$$F(x) = 0$

• For $a\le x\le b$$a \leq x \leq b$: The probability is the area under the PDF curve from $a$$a$ to $x$$x$: $F(x)={\int }_a^x\frac{1}{b-a}du=\frac{x-a}{b-a}$$F(x) = \int_a^x \frac{1}{b-a} \, du = \frac{x-a}{b-a}$

• For $x>b$$x > b$: The probability is 1, as $x$$x$ is beyond the upper limit of the interval: $F(x)=1$$F(x) = 1$

1.3 Moment Generating Function (MGF)

The MGF $M_X(t)$$M_X(t)$ of the uniform distribution $U(a,b)$$U(a, b)$ is given by: $M_X(t)=\frac{e^{bt}-e^{at}}{t(b-a)},t\in \mathbb{R}$$M_X(t) = \frac{e^{bt} - e^{at}}{t(b-a)}, \quad t \in \mathbb{R}$

Derivation of the MGF:

• Definition: The MGF is defined as: $M_X(t)=\mathbb{E}[e^{tX}]={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{e}^{tx}f(x)dx$$M_X(t) = \mathbb{E}[e^{tX}] = \int_{-\infty}^{\infty} e^{tx} f(x) \, dx$

2. Applying the PDF: $M_X(t)={\int }_a^b{e}^{tx}\cdot \frac{1}{b-a}dx$$M_X(t) = \int_a^b e^{tx} \cdot \frac{1}{b-a} \, dx$

3. Integration: $M_X(t)=\frac{1}{b-a}{\int }_a^b{e}^{tx}dx=\frac{1}{b-a}\cdot \frac{e^{tx}}{t}{|}_a^b$$M_X(t) = \frac{1}{b-a} \int_a^b e^{tx} \, dx = \frac{1}{b-a} \cdot \frac{e^{tx}}{t} \Big|_a^b$

4. Simplifying: $M_X(t)=\frac{e^{bt}-e^{at}}{t(b-a)}$$M_X(t) = \frac{e^{bt} - e^{at}}{t(b-a)}$

1.4 Mean

The mean $\mathbb{E}[X]$$\mathbb{E}[X]$ of the uniform distribution is given by: $\mathbb{E}[X]=\frac{a+b}{2}$$\mathbb{E}[X] = \frac{a + b}{2}$

Derivation of the Mean:

• Definition: The mean is the expected value of $X$$X$: $\mathbb{E}[X]={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}xf(x)dx$$\mathbb{E}[X] = \int_{-\infty}^{\infty} x f(x) \, dx$

2. Applying the PDF: $\mathbb{E}[X]={\int }_a^{b}x\cdot \frac{1}{b-a}dx$$\mathbb{E}[X] = \int_a^b x \cdot \frac{1}{b-a} \, dx$

3. Integration: $\mathbb{E}[X]=\frac{1}{b-a}\cdot \frac{x^2}{2}{|}_a^b=\frac{1}{b-a}\cdot \frac{b^2-a^2}{2}$$\mathbb{E}[X] = \frac{1}{b-a} \cdot \frac{x^2}{2} \Big|_a^b = \frac{1}{b-a} \cdot \frac{b^2 - a^2}{2}$

4. Simplifying: $\mathbb{E}[X]=\frac{b+a}{2}$$\mathbb{E}[X] = \frac{b+a}{2}$

1.5 Variance

The variance $\text{Var}(X)$$\text{Var}(X)$ of the uniform distribution is given by: $\text{Var}(X)=\frac{(b-a{)}^2}{12}$$\text{Var}(X) = \frac{(b-a)^2}{12}$

Derivation of the Variance:

• Definition: The variance is the second central moment: $\text{Var}(X)=\mathbb{E}[X^2]-(\mathbb{E}[X]{)}^2$$\text{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$

• First, calculate $\mathbb{E}[X^2]$$\mathbb{E}[X^2]$: $\mathbb{E}[X^2]={\int }_a^b{x}^2\cdot \frac{1}{b-a}dx=\frac{1}{b-a}\cdot \frac{x^3}{3}{|}_a^b=\frac{b^3-a^3}{3(b-a)}$$\mathbb{E}[X^2] = \int_a^b x^2 \cdot \frac{1}{b-a} \, dx = \frac{1}{b-a} \cdot \frac{x^3}{3} \Big|_a^b = \frac{b^3 - a^3}{3(b-a)}$

• Simplifying $\mathbb{E}[X^2]$$\mathbb{E}[X^2]$: $\mathbb{E}[X^2]=\frac{b^2+ab+a^2}{3}$$\mathbb{E}[X^2] = \frac{b^2 + ab + a^2}{3}$

• Subtracting the square of the mean: $\text{Var}(X)=\mathbb{E}[X^2]-{\left(\frac{b+a}{2}\right)}^2=\frac{b^2+ab+a^2}{3}-\frac{b^2+2ab+a^2}{4}$$\text{Var}(X) = \mathbb{E}[X^2] - \left(\frac{b+a}{2}\right)^2 = \frac{b^2 + ab + a^2}{3} - \frac{b^2 + 2ab + a^2}{4}$

• Final Simplification: $\text{Var}(X)=\frac{(b-a{)}^2}{12}$$\text{Var}(X) = \frac{(b-a)^2}{12}$​

2. Exponential Distribution $EXP(\lambda )$$EXP(\lambda)$

The exponential distribution is commonly used to model the time between events in a Poisson process, such as the time between arrivals in a queue or the time until a component fails.

2.1 Probability Density Function (PDF)

The PDF of the exponential distribution $EXP(\lambda )$$EXP(\lambda)$ is given by: $\begin{array}{c}f(x)=\{\begin{array}{ll}\lambda e^{-\lambda x},& \text{if }x\ge 0\\ 0,& \text{if }x<0\end{array}\end{array}$$f(x) = \begin{cases} \lambda e^{-\lambda x}, & \text{if } x \geq 0 \\ 0, & \text{if } x < 0 \end{cases}$

Derivation of the PDF:

• Memorylessness Property: The exponential distribution is characterized by the memoryless property, meaning that the probability of an event occurring in the future is independent of the past.

• Poisson Process: If events occur according to a Poisson process with rate $\lambda$$\lambda$, the time $X$$X$ until the first event occurs follows an exponential distribution. The PDF represents the rate at which the events occur.

• Normalization: The PDF must integrate to 1 over the range $[0,\mathrm{\infty })$$[0, \infty)$: ${\int }_0^{\mathrm{\infty }}\lambda e^{-\lambda x}dx=1$$\int_0^{\infty} \lambda e^{-\lambda x} \, dx = 1$​

2.2 Cumulative Distribution Function (CDF)

The CDF $F(x)$$F(x)$ of the exponential distribution is given by: $\begin{array}{c}F(x)=\{\begin{array}{ll}1-e^{-\lambda x},& \text{if }x\ge 0\\ 0,& \text{if }x<0\end{array}\end{array}$$F(x) = \begin{cases} 1 - e^{-\lambda x}, & \text{if } x \geq 0 \\ 0, & \text{if } x < 0 \end{cases}$

Derivation of the CDF:

• Definition: The CDF is the probability that the random variable $X$$X$ takes a value less than or equal to $x$$x$: $F(x)=P(X\le x)={\int }_{-\mathrm{\infty }}^{x}f(u)du$$F(x) = P(X \leq x) = \int_{-\infty}^x f(u) \, du$

• For $x<0$$x < 0$: The probability is 0 since $X$$X$ cannot take negative values in the exponential distribution: $F(x)=0$$F(x) = 0$

• For $x\ge 0$$x \geq 0$: The probability is the integral of the PDF from 0 to $x$$x$: $F(x)={\int }_0^x\lambda e^{-\lambda u}du=1-e^{-\lambda x}$$F(x) = \int_0^x \lambda e^{-\lambda u} \, du = 1 - e^{-\lambda x}$

2.3 Moment Generating Function (MGF)

The MGF $M_X(t)$$M_X(t)$ of the exponential distribution $EXP(\lambda )$$EXP(\lambda)$ is given by: $M_X(t)=\frac{\lambda }{\lambda -t},t<\lambda$$M_X(t) = \frac{\lambda}{\lambda - t}, \quad t < \lambda$

Derivation of the MGF:

• Definition: The MGF is defined as: $M_X(t)=\mathbb{E}[e^{tX}]={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{e}^{tx}f(x)dx$$M_X(t) = \mathbb{E}[e^{tX}] = \int_{-\infty}^{\infty} e^{tx} f(x) \, dx$

2. Applying the PDF: $M_X(t)={\int }_0^{\mathrm{\infty }}{e}^{tx}\lambda e^{-\lambda x}dx=\lambda {\int }_0^{\mathrm{\infty }}{e}^{-(\lambda -t)x}dx$$M_X(t) = \int_0^{\infty} e^{tx} \lambda e^{-\lambda x} \, dx = \lambda \int_0^{\infty} e^{-(\lambda - t)x} \, dx$

3. Integration: $M_X(t)=\lambda \cdot \frac{1}{\lambda -t},\text{for }t<\lambda$$M_X(t) = \lambda \cdot \frac{1}{\lambda - t}, \quad \text{for } t < \lambda$ This condition $t<\lambda$$t < \lambda$ ensures the integral converges.

2.4 Mean

The mean $\mathbb{E}[X]$$\mathbb{E}[X]$ of the exponential distribution is given by: $\mathbb{E}[X]=\frac{1}{\lambda }$$\mathbb{E}[X] = \frac{1}{\lambda}$

Derivation of the Mean:

• Definition: The mean is the expected value of $X$$X$: $\mathbb{E}[X]={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}xf(x)dx$$\mathbb{E}[X] = \int_{-\infty}^{\infty} x f(x) \, dx$

• Applying the PDF: $\mathbb{E}[X]={\int }_0^{\mathrm{\infty }}x\lambda e^{-\lambda x}dx$$\mathbb{E}[X] = \int_0^{\infty} x \lambda e^{-\lambda x} \, dx$

• Integration by Parts: Use integration by parts where $u=x$$u = x$ and $dv=\lambda e^{-\lambda x}dx$$dv = \lambda e^{-\lambda x} dx$: $\mathbb{E}[X]={[-\frac{x{e}^{-\lambda x}}{\lambda }]}_0^{\mathrm{\infty }}+{\int }_0^{\mathrm{\infty }}\frac{1}{\lambda }{e}^{-\lambda x}dx=\frac{1}{\lambda }$$\mathbb{E}[X] = \left[-\frac{x e^{-\lambda x}}{\lambda}\right]_0^\infty + \int_0^\infty \frac{1}{\lambda} e^{-\lambda x} \, dx = \frac{1}{\lambda}$

2.5 Variance

The variance $\text{Var}(X)$$\text{Var}(X)$ of the exponential distribution is given by: $\text{Var}(X)=\frac{1}{{\lambda }^2}$$\text{Var}(X) = \frac{1}{\lambda^2}$

Derivation of the Variance:

• Variance Formula: The variance is given by: $\text{Var}(X)=\mathbb{E}[X^2]-(\mathbb{E}[X]{)}^2$$\text{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$

• Calculate $\mathbb{E}[X^2]$$\mathbb{E}[X^2]$: $\mathbb{E}[X^2]={\int }_0^{\mathrm{\infty }}{x}^2\lambda e^{-\lambda x}dx$$\mathbb{E}[X^2] = \int_0^{\infty} x^2 \lambda e^{-\lambda x} \, dx$

3. Integration by Parts: Apply integration by parts twice, or use the fact that: $\mathbb{E}[X^2]=\frac{2}{{\lambda }^2}$$\mathbb{E}[X^2] = \frac{2}{\lambda^2}$

4. Variance: $\text{Var}(X)=\frac{2}{{\lambda }^2}-{\left(\frac{1}{\lambda }\right)}^2=\frac{1}{{\lambda }^2}$$\text{Var}(X) = \frac{2}{\lambda^2} - \left(\frac{1}{\lambda}\right)^2 = \frac{1}{\lambda^2}$

2.6 Memorylessness Property

Mathematically: $P(X>s+t\mid X>t)=P(X>s)$$P(X > s + t \mid X > t) = P(X > s)$ Where $X$$X$ is a random variable representing the time until the next event, $s$$s$ and $t$$t$ are time intervals.

Explanation:

This property implies that the exponential distribution "forgets" how much time has already passed. For example, if a light bulb's lifetime follows an exponential distribution, the probability that it lasts another hour is the same regardless of how long it has already been on.

2.7 Relationship between Exponential and Poisson Distributions

The exponential distribution is closely related to the Poisson distribution.

Setup:

• Let $T_1,T_2,\dots$$T_1, T_2, \dots$ be independent and identically distributed (i.i.d.) random variables following $EXP(\lambda )$$EXP(\lambda)$.

• Define $S_k=T_1+T_2+\cdots +T_k$$S_k = T_1 + T_2 + \dots + T_k$ as the sum of the first $k$$k$ exponential random variables.

• Let $X_i$$X_i$ be the number of $S_k$$S_k$ values that fall in the interval $[t_{i-1},t_i]$$[t_{i-1}, t_i]$, for $i=1,2,\dots ,n$$i = 1, 2, \dots, n$.

Key Result:

• $X_i$$X_i$ follows a Poisson distribution: $X_i\sim P(\lambda (t_i-t_{i-1}))$$X_i \sim P(\lambda(t_i - t_{i-1}))$.

• This result indicates that the number of events occurring in a given interval $[t_{i-1},t_i]$$[t_{i-1}, t_i]$ is Poisson distributed if the times between events follow an exponential distribution.

Reverse Relationship:

• If the number of events $X_i$$X_i$ in each interval $[t_{i-1},t_i]$$[t_{i-1}, t_i]$ is Poisson distributed, then the time between consecutive events follows an exponential distribution.

Interpretation:

• The Poisson distribution models the number of events in a fixed time interval, while the exponential distribution models the time between those events.

2.8 Interpreting $\lambda$$\lambda$

The parameter $\lambda$$\lambda$ in the exponential and Poisson distributions has an important interpretation.

Exponential Distribution $EXP(\lambda )$$EXP(\lambda)$:

• $\lambda$$\lambda$ represents the rate of events per time unit.

• Mean Time Between Events: The mean or expected time between events is given by $\frac{1}{\lambda }$$\frac{1}{\lambda}$. This means that, on average, events happen every $\frac{1}{\lambda }$$\frac{1}{\lambda}$ units of time.

Poisson Distribution $P(\lambda t)$$P(\lambda t)$:

• In the Poisson distribution, $\lambda t$$\lambda t$ represents the expected number of events in a time interval of length $t$$t$.

• Mean Number of Events: For a time interval of length $t$$t$, the average number of events occurring is $\lambda t$$\lambda t$.

Intuitive Example:

• If $\lambda =5$$\lambda = 5$ events per hour, then on average, 5 events are expected to occur every hour.

• The time between events, on average, is $\frac{1}{\lambda }=\frac{1}{5}$$\frac{1}{\lambda} = \frac{1}{5}$ hours, or 12 minutes.

3. Gamma Distribution $\text{Gamma}(\alpha ,\lambda )$$\text{Gamma}(\alpha, \lambda)$

3.1 Probability Density Function (PDF)

The PDF of the gamma distribution $\text{Gamma}(\alpha ,\lambda )$$\text{Gamma}(\alpha, \lambda)$ is given by: $\begin{array}{c}f(x)=\{\begin{array}{ll}\frac{{\lambda }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{x}^{\alpha -1}{e}^{-\lambda x},& \text{if }x\ge 0\\ 0,& \text{if }x<0\end{array}\end{array}$$f(x) = \begin{cases} \frac{\lambda^\alpha}{\Gamma(\alpha)} x^{\alpha - 1} e^{-\lambda x}, & \text{if } x \geq 0 \\ 0, & \text{if } x < 0 \end{cases}$

• $\alpha >0$$\alpha > 0$ is the shape parameter.

• $\lambda >0$$\lambda > 0$ is the rate (or scale) parameter.

• $\mathrm{\Gamma }(\alpha )$$\Gamma(\alpha)$ is the gamma function.

Gamma Function $\mathrm{\Gamma }(\alpha )$$\Gamma(\alpha)$:

The gamma function $\mathrm{\Gamma }(\alpha )$$\Gamma(\alpha)$ is defined as: $\mathrm{\Gamma }(\alpha )={\int }_0^{\mathrm{\infty }}{y}^{\alpha -1}{e}^{-y}dy$$\Gamma(\alpha) = \int_0^\infty y^{\alpha-1} e^{-y} \, dy$

• For a positive integer $\alpha =n$$\alpha = n$, $\mathrm{\Gamma }(n)=(n-1)!$$\Gamma(n) = (n-1)!$.

• For $\alpha =\frac{1}{2}$$\alpha = \frac{1}{2}$, $\mathrm{\Gamma }\left(\frac{1}{2}\right)=\sqrt{\pi }$$\Gamma\left(\frac{1}{2}\right) = \sqrt{\pi}$.

Derivation of the PDF:

• Generalization of Exponential Distribution: The gamma distribution generalizes the exponential distribution, which is a special case when $\alpha =1$$\alpha = 1$.

• Shape and Scale: The parameter $\alpha$$\alpha$ controls the shape, and $\lambda$$\lambda$ scales the distribution. The PDF decreases exponentially with $x$$x$ but is modified by the $x^{\alpha -1}$$x^{\alpha - 1}$ term, which can increase the probability for small values of $x$$x$ when $\alpha$$\alpha$ is small.

3.2 Cumulative Distribution Function (CDF)

The CDF of the gamma distribution does not have a simple closed form like the exponential distribution. However, it can be expressed in terms of the incomplete gamma function: $F(x)=\frac{\gamma (\alpha ,\lambda x)}{\mathrm{\Gamma }(\alpha )}$$F(x) = \frac{\gamma(\alpha, \lambda x)}{\Gamma(\alpha)}$

Where $\gamma (\alpha ,\lambda x)$$\gamma(\alpha, \lambda x)$ is the lower incomplete gamma function defined by: $\gamma (\alpha ,\lambda x)={\int }_0^{\lambda x}{y}^{\alpha -1}{e}^{-y}dy$$\gamma(\alpha, \lambda x) = \int_0^{\lambda x} y^{\alpha-1} e^{-y} \, dy$

3.3 Moment Generating Function (MGF)

The MGF $M_X(t)$$M_X(t)$ of the gamma distribution is given by: $M_X(t)={\left(\frac{\lambda }{\lambda -t}\right)}^{\alpha },t<\lambda$$M_X(t) = \left(\frac{\lambda}{\lambda - t}\right)^\alpha, \quad t < \lambda$

Derivation of the MGF:

• Definition: The MGF is defined as: $M_X(t)=\mathbb{E}[e^{tX}]={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{e}^{tx}f(x)dx$$M_X(t) = \mathbb{E}[e^{tX}] = \int_{-\infty}^{\infty} e^{tx} f(x) \, dx$

• Substitute the PDF: $M_X(t)={\int }_0^{\mathrm{\infty }}{e}^{tx}\frac{{\lambda }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{x}^{\alpha -1}{e}^{-\lambda x}dx=\frac{{\lambda }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{\int }_0^{\mathrm{\infty }}{x}^{\alpha -1}{e}^{-(\lambda -t)x}dx$$M_X(t) = \int_0^\infty e^{tx} \frac{\lambda^\alpha}{\Gamma(\alpha)} x^{\alpha - 1} e^{-\lambda x} \, dx = \frac{\lambda^\alpha}{\Gamma(\alpha)} \int_0^\infty x^{\alpha - 1} e^{-(\lambda - t)x} \, dx$

• Recognize the Gamma Function Form: $M_X(t)=\frac{{\lambda }^{\alpha }}{(\lambda -t{)}^{\alpha }}\cdot \frac{\mathrm{\Gamma }(\alpha )}{\mathrm{\Gamma }(\alpha )}={\left(\frac{\lambda }{\lambda -t}\right)}^{\alpha }$$M_X(t) = \frac{\lambda^\alpha}{(\lambda - t)^\alpha} \cdot \frac{\Gamma(\alpha)}{\Gamma(\alpha)} = \left(\frac{\lambda}{\lambda - t}\right)^\alpha$