Lecture 1: Probability and Statistics Part 1

Jason Chen YCHEN148@e.ntu.edu.sg

Copyright Notice

This document is a compilation of original notes by Jason Chen (YCHEN148@e.ntu.edu.sg). Unauthorized reproduction, downloading, or use for commercial purposes is strictly prohibited and may result in legal action. If you have any questions or find any content-related errors, please contact the author at the provided email address. The author will promptly address any necessary corrections.

I. Probability Measure

1. Some Definition

Sample space: Ω is the set of all possible outcomes in a random phenomenon.

Example 1.1: Throw a coin 3 times, Ω = {hhh, hht, hth, thh, htt, tht, tth, ttt}

Example 1.2: Number of jobs in a print queue, Ω = {0, 1, 2, ...}

Example 1.3: Length of time between successive earthquakes, Ω = {t | t ≥ 0}

Question: What are the differences between Ω’s in these examples?

Answer:

• In Example 1.1, the sample space Ω is a finite set consisting of all possible sequences of heads (h) and tails (t) when a coin is tossed three times. This is a discrete and finite sample space.

• In Example 1.2, Ω is an infinite countable set representing the number of jobs in a print queue, which can be any non-negative integer. This sample space is also discrete but infinite.

• In Example 1.3, Ω is a continuous set representing the possible lengths of time between successive earthquakes, where t can be any non-negative real number. This sample space is continuous and infinite.

The key differences are that Example 1.1 deals with a finite discrete sample space, Example 1.2 with an infinite discrete sample space, and Example 1.3 with a continuous infinite sample space.

2. Event & Measure

Event: A particular subset of Ω.

Example 1.4: Let A be the event that the total number of heads equals 2 when tossing a coin 3 times. A = {HHT,HTH,THH}

Example 1.5: Let A be the event that fewer than 5 jobs are in the print queue. A = {0,1,2,3,4}

Probability measure: A function P from subsets of Ω to the real numbers that satisfies the following axioms:

1. P(Ω) = 1.

2. If A ⊆ Ω, then P(A) ≥ 0.

3. If A₁ and A₂ are disjoint, then P(A₁ ∪ A₂) = P(A₁) + P(A₂).

   More generally, if A₁, A₂, ... are mutually disjoint, then P(∪ᵢ Aᵢ) = Σ P(Aᵢ).

Example 1.6: Suppose the coin is fair. For every outcome ω ∈ Ω, P(ω) = 1/8.

Ω = {hhh,hht,hth,thh,htt,tht,tth,ttt}

3. Law of Total Probability

Let $A$$A$ and $B_1,B_2,\dots ,B_n$$B_1, B_2, \dots, B_n$ be events where $B_i$$B_i$ are disjoint, $\bigcup _{i=1}^n{B}_i=\mathrm{\Omega }$$\bigcup_{i=1}^n B_i = \Omega$, and $P(B_i)>0$$P(B_i) > 0$ for all $i$$i$. Then

$P(B_j\mid A)=\frac{P(A\mid B_j)P(B_j)}{\sum _{i=1}^{n}P(A\mid B_i)P(B_i)}$$P(B_j \mid A) = \frac{P(A \mid B_j)P(B_j)}{\sum_{i=1}^n P(A \mid B_i)P(B_i)}$

The Law of Total Probability is an important method for calculating the probability of an event by decomposing event $A$$A$ into several mutually exclusive events $Bᵢ$$Bᵢ$, and then summing up the probabilities of $A$$A$ occurring given each $Bᵢ$$Bᵢ$ occurs, weighted by the probabilities of the $Bᵢ$$Bᵢ$​ themselves.

4. Conditional Probability

Definition

If $A$$A$ and $B$$B$ are two events, the conditional probability of $A$$A$ given $B$$B$, denoted $P(A\mid B)$$P(A \mid B)$, is defined as: $P(A\mid B)=\frac{P(A\cap B)}{P(B)}$$P(A \mid B) = \frac{P(A \cap B)}{P(B)}$

provided that $P(B)>0$$P(B) > 0$.

• $P(A\cap B)$$P(A \cap B)$ is the probability that both $A$$A$ and $B$$B$ occur.

• $P(B)$$P(B)$ is the probability that event $B$$B$ occurs.

The formula can be interpreted as "the probability of $A$$A$ happening, given that $B$$B$ has already happened."

Key Properties

• Multiplication Rule: The probability of the intersection of two events can be written as: $P(A\cap B)=P(B)\cdot P(A\mid B)=P(A)\cdot P(B\mid A)$$P(A \cap B) = P(B) \cdot P(A \mid B) = P(A) \cdot P(B \mid A)$ This is useful when calculating joint probabilities when conditional probabilities are known.

• Bayes' Theorem: Bayes' theorem is a powerful result that relates the conditional probability of two events: $P(A\mid B)=\frac{P(B\mid A)\cdot P(A)}{P(B)}$$P(A \mid B) = \frac{P(B \mid A) \cdot P(A)}{P(B)}$ Bayes' theorem is widely used in statistical inference, especially in the context of updating beliefs with new evidence (e.g., Bayesian statistics).

• Law of Total Probability: If $B_1,B_2,\dots ,B_n$$B_1, B_2, \dots, B_n$ are mutually exclusive and exhaustive events (i.e., one of them must occur and no two can occur together), then the probability of an event $A$$A$ can be found as: $P(A)=\sum _{i=1}^{n}P(A\mid B_i)\cdot P(B_i)$$P(A) = \sum_{i=1}^n P(A \mid B_i) \cdot P(B_i)$ This law is useful when $A$$A$ can happen in several different ways, corresponding to the events $B_i$$B_i$.

5. Bayes' Rule

Let $A$$A$ and $B_1$$B_1$, $B_2$$B_2$, ..., $B_n$$B_n$ be events where $B_i$$B_i$ are disjoint, ${\cup }_{i=1}^n{B}_i=\mathrm{\Omega }$$\cup_{i=1}^{n} B_i = \Omega$ and $P(B_i)>0$$P(B_i) > 0$ for all $i$$i$. Then $P(B_j\mid A)=\frac{P(A\mid B_j)P(B_j)}{\sum _{i=1}^{n}P(A\mid B_i)P(B_i)}$$P(B_j \mid A) = \frac{P(A \mid B_j) P(B_j)}{\sum_{i=1}^{n} P(A \mid B_i) P(B_i)}$

Bayes' Rule is a method to calculate the reverse probability, allowing us to update our estimation of the probability of an initial event given the occurrence of another event. This is crucial for revising probability estimates based on new evidence.

Example 1.8: Monty Hall problem

Three prisoners A, B, and C are on death row. The governor decides to pardon one of them and chooses the prisoner to pardon at random. He informs the warden of this choice but requests that the name be kept secret for a few days.

A tries to get the warden to tell him who had been pardoned. The warden refuses. A then asks which of B or C will be executed. The warden thinks for a while and tells A that B is to be executed. A then thinks that his chance of being pardoned has risen to 1/2. Is he right?

Step 1: Initial Probabilities

$P(A)=P(B)=P(C)=\frac{1}{3}$$P(A) = P(B) = P(C) = \frac{1}{3}$

Step 2: Conditional Probability After Warden’s Information

If A is actually released, there is a 1/2 chance that the warden will say B will be released; if B is actually released, there is a 0% chance that the warden will say B will be released; if C is actually released, there is a 100% chance that the warden will say B will be released (you can neither admit the truth about C nor tell A the truth)

Step 3: Apply Bayes' Theorem

Using Bayes' Theorem, we can calculate the probability that $A$$A$ is pardoned given that the warden says $B$$B$ will be executed:

$P(A\mid \text{Warden says B will be executed})=\frac{P(\text{Warden says B will be executed}\mid A\text{ is pardoned})\cdot P(A)}{P(\text{Warden says B will be executed})}$$P(A \mid \text{Warden says B will be executed}) = \frac{P(\text{Warden says B will be executed} \mid A \text{ is pardoned}) \cdot P(A)}{P(\text{Warden says B will be executed})}$

$P(\text{Warden says B will be executed})=(\frac{1}{2}\cdot \frac{1}{3})+(0\cdot \frac{1}{3})+(1\cdot \frac{1}{3})=\frac{1}{6}+\frac{2}{6}=\frac{3}{6}=\frac{1}{2}$$P(\text{Warden says B will be executed}) = \left(\frac{1}{2} \cdot \frac{1}{3}\right) + \left(0 \cdot \frac{1}{3}\right) + \left(1 \cdot \frac{1}{3}\right) = \frac{1}{6} + \frac{2}{6} = \frac{3}{6} = \frac{1}{2}$

Thus: $P(A\mid \text{Warden says B will be executed})=\frac{\frac{1}{2}\cdot \frac{1}{3}}{\frac{1}{2}}=\frac{1}{6}\cdot \frac{2}{1}=\frac{1}{3}$$P(A \mid \text{Warden says B will be executed}) = \frac{\frac{1}{2} \cdot \frac{1}{3}}{\frac{1}{2}} = \frac{1}{6} \cdot \frac{2}{1} = \frac{1}{3}$​

6. Independence

Two events $A$$A$ and $B$$B$ are independent if $P(A\cap B)=P(A)P(B)$$P(A \cap B) = P(A)P(B)$

A group of events $A₁$$A₁$, $A₂$$A₂$, ..., $Aₙ$$Aₙ$ are said to be mutually independent if for any subgroup $Aᵢ₁$$Aᵢ₁$, ..., $Aᵢₘ$$Aᵢₘ$ $P(\cap ᵢ=₁Aᵢⱼ)=\prod P(Aᵢⱼ)$$P(\capᵢ=₁ Aᵢⱼ) = \prod P(Aᵢⱼ)$

Independence is a crucial concept in probability theory, implying that the occurrence of one event does not affect the likelihood of the other.

II. Random Variable and Distribution

1. Random Variable

A random variable is a function from the sample space $\mathrm{\Omega }$$\Omega$ to the real numbers.

Example 1.9:

• $X_1$$X_1$: The total number of heads in a series of coin tosses.

• $X_2$$X_2$: The number of heads on the first toss.

• $X_3$$X_3$: The number of heads minus the number of tails.

Importance of Random Variables

Statisticians need random variables because they provide a way to quantify and analyze outcomes of random phenomena. By mapping outcomes to the real line, we can use mathematical tools and techniques to study their distributions, moments, and other properties.

2. Distribution

A random variable has a sample space on the real line. This brings special ways to describe its probability measure.

2.1 Discrete vs. Continuous Random Variables

• Discrete Random Variable: Can take on only a finite or countably infinite number of values.

• Continuous Random Variable: Can take on a continuum of values.

2.2 Functions for Describing Distributions

• Probability Mass Function (pmf): For discrete random variables, it gives the probability that the random variable takes on a specific value.

• Probability Density Function (pdf): For continuous random variables, it describes the relative likelihood for the random variable to take on a given value.

• Cumulative Distribution Function (cdf): Describes the probability that the random variable takes on a value less than or equal to a specific value.

• Moment Generating Function (mgf): Provides a way to capture all the moments (mean, variance, etc.) of the distribution.

2.3 Summary of Notations and Functions

3. Distribution Functions

3.1 Cumulative Distribution Function (CDF)

A function $F$$F$ is the cumulative distribution function (CDF) of a random variable $X$$X$ if $F(x)=P(X\le x),x\in \mathbb{R}.$$F(x) = P(X \leq x), \, x \in \mathbb{R}.$

3.2 Probability Mass Function (PMF)

A function $p(x)$$p(x)$ is a probability mass function (PMF) if and only if $p(x)\ge 0$$p(x) \geq 0$ for all $x$$x$ and $\sum _{-\mathrm{\infty }<x<\mathrm{\infty }}p(x)=1$$\sum_{-\infty<x<\infty} p(x) = 1$.

For a discrete random variable $X$$X$ with PMF $p(x)$$p(x)$, $P(X=x)=p(x)$$P(X = x) = p(x)$ and $P(X\in A)=\sum _{x\in A}p(x).$$P(X \in A) = \sum_{x \in A} p(x).$

3.3 Probability Density Function (PDF)

A function $f(x)$$f(x)$ is a probability density function (PDF) if and only if $f(x)\ge 0$$f(x) \geq 0$ for all $x$$x$ and ${\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(x)dx=1$$\int_{-\infty}^{\infty} f(x) \, dx = 1$; For a continuous random variable $X$$X$ with PDF $f(x)$$f(x)$, $P(X\in A)={\int }_{A}f(x)dx.$$P(X \in A) = \int_{A} f(x) \, dx.$

3.4 Relationships Between PDF and CDF

• The CDF $F(x)$$F(x)$ can be obtained from the PDF $f(x)$$f(x)$ as follows: $F(x)={\int }_{-\mathrm{\infty }}^{x}f(t)dt.$$F(x) = \int_{-\infty}^{x} f(t) \, dt.$

• The PDF $f(x)$$f(x)$ can be obtained from the CDF $F(x)$$F(x)$ as follows: $f(x)=\frac{d}{dx}F(x).$$f(x) = \frac{d}{dx} F(x).$

3.5 Interpretation of $f(x)$$f(x)$

The PDF $f(x)$$f(x)$ does not give the probability that $X$$X$ takes the value $x$$x$ (which is zero for continuous random variables). Instead, it gives the density of the probability at $x$$x$. To find the probability that $X$$X$ falls within an interval $[a,b]$$[a, b]$, you integrate the PDF over that interval: $P(a\le X\le b)={\int }_a^{b}f(x)dx.$$P(a \leq X \leq b) = \int_{a}^{b} f(x) \, dx.$

3.6 Properties of CDF

If $F(x)$$F(x)$ is a CDF of some random variable $X$$X$, then the following properties hold:

1. $0\le F(x)\le 1$$0 \leq F(x) \leq 1$.

2. $F(x)$$F(x)$ is non-decreasing.

3. For any $x\in \mathbb{R}$$x \in \mathbb{R}$, $F(x)$$F(x)$ is continuous from the right, i.e., $\underset{t\downarrow x}{lim}F(t)=F(x).$$\lim_{t \downarrow x} F(t) = F(x).$

4. $\underset{x\to \mathrm{\infty }}{lim}F(x)=1$$\lim_{x \to \infty} F(x) = 1$ and $\underset{x\to -\mathrm{\infty }}{lim}F(x)=0.$$\lim_{x \to -\infty} F(x) = 0.$

5. $P(X>x)=1-F(x)$$P(X > x) = 1 - F(x)$ and $P(a<X<b)=F(b)-F(a)$$P(a < X < b) = F(b) - F(a)$.

6. For any $x\in \mathbb{R}$$x \in \mathbb{R}$, $F(x)$$F(x)$ has a left limit.

7. There are at most countably many discontinuity points of $F(x)$$F(x)$.

Conversely, if a function $F(x)$$F(x)$ satisfies properties 2, 3, and 4, then $F(x)$$F(x)$ is a CDF.

4. Joint Distribution

4.1 Why We Need Joint Distribution for Multivariate Random Variables

Joint distributions are necessary to describe the behavior of multiple random variables simultaneously. They provide a comprehensive view of how the random variables interact with each other.

Example 1.10 (continued from Example 1.9)

Given: $\mathrm{\Omega }=\{hhh,hht,hth,thh,htt,tht,tth,ttt\}$$\Omega = \{hhh, hht, hth, thh, htt, tht, tth, ttt\}$

• $X_1$$X_1$: Total number of heads

• $X_2$$X_2$: Number of heads on the first toss

We can describe the joint distribution of $X_1$$X_1$ and $X_2$$X_2$ by looking at the sample space and counting occurrences.

4.2 Joint CDF and Marginal CDF

Joint CDF

The joint cumulative distribution function (CDF) of $X_1,\dots ,X_n$$X_1, \ldots, X_n$ is: $F(x_1,\dots ,x_n)=P(X_1\le x_1,\dots ,X_n\le x_n)$$F(x_1, \ldots, x_n) = P(X_1 \leq x_1, \ldots, X_n \leq x_n)$

Marginal CDF

The marginal CDF of $X_1$$X_1$ is: $F_{X_1}(x_1)=P(X_1\le x_1)=\underset{x_2,\dots ,x_n\to \mathrm{\infty }}{lim}F(x_1,x_2,\dots ,x_n)$$F_{X_1}(x_1) = P(X_1 \leq x_1) = \lim_{x_2, \ldots, x_n \to \infty} F(x_1, x_2, \ldots, x_n)$

4.3 Joint and Marginal Distribution Functions

Discrete:

• Joint Probability Mass Function: $p(x_1,\dots ,x_n)=P(X_1=x_1,\dots ,X_n=x_n)$$p(x_1, \ldots, x_n) = P(X_1 = x_1, \ldots, X_n = x_n)$

• Probability for a Set ( A ): $P((X_1,\dots ,X_n)\in A)=\sum _{(x_1,\dots ,x_n)\in A}p(x_1,\dots ,x_n)$$P((X_1, \ldots, X_n) \in A) = \sum_{(x_1, \ldots, x_n) \in A} p(x_1, \ldots, x_n)$

• Joint Cumulative Distribution Function: $F(x_1,\dots ,x_n)=\sum _{t_1\le x_1,\dots ,t_n\le x_n}p(t_1,\dots ,t_n)$$F(x_1, \ldots, x_n) = \sum_{t_1 \leq x_1, \ldots, t_n \leq x_n} p(t_1, \ldots, t_n)$

• Marginal Probability Mass Function: $p_{X_1}(x_1)=P(X_1=x_1)=\sum _{-\mathrm{\infty }<t_2<\mathrm{\infty },\dots ,-\mathrm{\infty }<t_n<\mathrm{\infty }}p(x_1,t_2,\dots ,t_n)$$p_{X_1}(x_1) = P(X_1 = x_1) = \sum_{-\infty < t_2 < \infty, \ldots, -\infty < t_n < \infty} p(x_1, t_2, \ldots, t_n)$

Continuous:

• Joint Probability Density Function: $f(x_1,\dots ,x_n)=\frac{{\partial }^n}{\partial x_1\cdots \partial x_n}F(x_1,\dots ,x_n)$$f(x_1, \ldots, x_n) = \frac{\partial^n}{\partial x_1 \cdots \partial x_n} F(x_1, \ldots, x_n)$

• Probability for a Set ( A ): $P((X_1,\dots ,X_n)\in A)={\int }_A\cdots \int f(x_1,\dots ,x_n)d{x}_1\cdots d{x}_n$$P((X_1, \ldots, X_n) \in A) = \int_A \cdots \int f(x_1, \ldots, x_n) \, dx_1 \cdots dx_n$

• Joint Cumulative Distribution Function: $F(x_1,\dots ,x_n)={\int }_{-\mathrm{\infty }}^{x_1}\cdots {\int }_{-\mathrm{\infty }}^{x_n}f(t_1,\dots ,t_n)d{t}_1\cdots d{t}_n$$F(x_1, \ldots, x_n) = \int_{-\infty}^{x_1} \cdots \int_{-\infty}^{x_n} f(t_1, \ldots, t_n) \, dt_1 \cdots dt_n$

• Marginal Probability Density Function: $f_{X_1}(x_1)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\cdots {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(x_1,t_2,\dots ,t_n)d{t}_2\cdots d{t}_n$$f_{X_1}(x_1) = \int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} f(x_1, t_2, \ldots, t_n) \, dt_2 \cdots dt_n$

Interpreting Joint Distributions

• Joint PMF (Discrete Case): Gives the probability of each possible combination of values of the random variables.

• Joint PDF (Continuous Case): Describes the density of the probability distribution over a continuous range of values.

4.4 Joint PMF and its Relationship with Joint CDF

Joint PMF Definition

For discrete random variables $X$$X$ and $Y$$Y$, the joint probability mass function (joint PMF) $p_{XY}(x,y)$$p_{XY}(x, y)$ is defined as: $p_{XY}(x,y)=P(X=x,Y=y).$$p_{XY}(x, y) = P(X = x, Y = y).$

Joint CDF Definition

The joint cumulative distribution function (joint CDF) $F_{XY}(x,y)$$F_{XY}(x, y)$ is defined as: $F_{XY}(x,y)=P(X\le x,Y\le y).$$F_{XY}(x, y) = P(X \leq x, Y \leq y).$

Relationship between Joint PMF and Joint CDF

The joint CDF can be expressed in terms of the joint PMF as follows: $F_{XY}(x,y)=\sum _{u\le x}\sum _{v\le y}{p}_{XY}(u,v).$$F_{XY}(x, y) = \sum_{u \leq x} \sum_{v \leq y} p_{XY}(u, v).$

Deriving Joint CDF from Joint PMF

Given the joint PMF $p_{XY}(x,y)$$p_{XY}(x, y)$:

1. Definition of Joint CDF: The joint CDF $F_{XY}(x,y)$$F_{XY}(x, y)$ is the probability that $X$$X$ is less than or equal to $x$$x$ and $Y$$Y$ is less than or equal to $y$$y$: $F_{XY}(x,y)=P(X\le x,Y\le y)$$F_{XY}(x, y) = P(X \leq x, Y \leq y)$.

2. Expressing in terms of PMF: This can be expressed as the sum of joint PMFs for all $u\le x$$u \leq x$ and $v\le y$$v \leq y$: $F_{XY}(x,y)=\sum _{u\le x}\sum _{v\le y}{p}_{XY}(u,v)$$F_{XY}(x, y) = \sum_{u \leq x} \sum_{v \leq y} p_{XY}(u, v)$.

Example

Let's assume $X$$X$ and $Y$$Y$ are discrete random variables with the following joint PMF:

Calculating $F_{XY}(1,1)$$F_{XY}(1, 1)$: $F_{XY}(1,1)=\sum _{u\le 1}\sum _{v\le 1}{p}_{XY}(u,v)$$F_{XY}(1, 1) = \sum_{u \leq 1} \sum_{v \leq 1} p_{XY}(u, v)$ $=p_{XY}(0,0)+p_{XY}(0,1)+p_{XY}(1,0)+p_{XY}(1,1)$$= p_{XY}(0, 0) + p_{XY}(0, 1) + p_{XY}(1, 0) + p_{XY}(1, 1)$ $=0.1+0.2+0.1+0.1=0.5$$= 0.1 + 0.2 + 0.1 + 0.1 = 0.5$

Calculating $F_{XY}(2,2)$$F_{XY}(2, 2)$: $F_{XY}(2,2)=\sum _{u\le 2}\sum _{v\le 2}{p}_{XY}(u,v)$$F_{XY}(2, 2) = \sum_{u \leq 2} \sum_{v \leq 2} p_{XY}(u, v)$ $=p_{XY}(0,0)+p_{XY}(0,1)+p_{XY}(0,2)+p_{XY}(1,0)+p_{XY}(1,1)+p_{XY}(1,2)+p_{XY}(2,0)+p_{XY}(2,1)+p_{XY}(2,2)$$= p_{XY}(0, 0) + p_{XY}(0, 1) + p_{XY}(0, 2) + p_{XY}(1, 0) + p_{XY}(1, 1) + p_{XY}(1, 2) + p_{XY}(2, 0) + p_{XY}(2, 1) + p_{XY}(2, 2)$ $=0.1+0.2+0.1+0.1+0.1+0.2+0.1+0.05+0.05=1.0$$= 0.1 + 0.2 + 0.1 + 0.1 + 0.1 + 0.2 + 0.1 + 0.05 + 0.05 = 1.0$

The joint PMF $p_{XY}(x,y)$$p_{XY}(x, y)$ can be found by taking the differences of the joint CDF: $p_{XY}(x,y)=P(X=x,Y=y)=F_{XY}(x,y)-F_{XY}(x-1,y)-F_{XY}(x,y-1)+F_{XY}(x-1,y-1)$$p_{XY}(x, y) = P(X = x, Y = y) = F_{XY}(x, y) - F_{XY}(x-1, y) - F_{XY}(x, y-1) + F_{XY}(x-1, y-1)$.

4.5 Joint PDF and its Relationship with Joint CDF (Continuous Case)

Joint PDF Definition

For continuous random variables $X$$X$ and $Y$$Y$, the joint probability density function (joint PDF) $f_{XY}(x,y)$$f_{XY}(x, y)$ is defined such that: $P(a\le X\le b,c\le Y\le d)={\int }_a^b{\int }_c^d{f}_{XY}(x,y)dydx$$P(a \leq X \leq b, c \leq Y \leq d) = \int_a^b \int_c^d f_{XY}(x, y) \, dy \, dx$.

Joint CDF Definition

The joint cumulative distribution function (joint CDF) $F_{XY}(x,y)$$F_{XY}(x, y)$ is defined as: $F_{XY}(x,y)=P(X\le x,Y\le y)$$F_{XY}(x, y) = P(X \leq x, Y \leq y)$.

Relationship between Joint PDF and Joint CDF

The joint CDF can be expressed in terms of the joint PDF as follows: $F_{XY}(x,y)={\int }_{-\mathrm{\infty }}^x{\int }_{-\mathrm{\infty }}^y{f}_{XY}(u,v)dvdu$$F_{XY}(x, y) = \int_{-\infty}^x \int_{-\infty}^y f_{XY}(u, v) \, dv \, du$.

Detailed Derivation

Given the joint PDF $f_{XY}(x,y)$$f_{XY}(x, y)$:

1. Definition of Joint CDF: The joint CDF $F_{XY}(x,y)$$F_{XY}(x, y)$ is the probability that $X$$X$ is less than or equal to $x$$x$ and $Y$$Y$ is less than or equal to $y$$y$: $F_{XY}(x,y)=P(X\le x,Y\le y)$$F_{XY}(x, y) = P(X \leq x, Y \leq y)$.

2. Expressing in terms of PDF: This can be expressed as the double integral of the joint PDF over the region $(-\mathrm{\infty },x]\times (-\mathrm{\infty },y]$$(-\infty, x] \times (-\infty, y]$: $F_{XY}(x,y)={\int }_{-\mathrm{\infty }}^x{\int }_{-\mathrm{\infty }}^y{f}_{XY}(u,v)dvdu$$F_{XY}(x, y) = \int_{-\infty}^x \int_{-\infty}^y f_{XY}(u, v) \, dv \, du$.

Example

Let's assume $X$$X$ and $Y$$Y$ are continuous random variables with the following joint PDF: $\begin{array}{c}{f}_{XY}(x,y)=\{\begin{array}{ll}2e^{-x-y}& x\ge 0,y\ge 0\\ 0& \text{otherwise}\end{array}\end{array}$$f_{XY}(x, y) = \begin{cases} 2e^{-x-y} & x \geq 0, y \geq 0 \\ 0 & \text{otherwise} \end{cases}$

Calculating $F_{XY}(x,y)$$F_{XY}(x, y)$ for $x\ge 0$$x \geq 0$ and $y\ge 0$$y \geq 0$: $F_{XY}(x,y)={\int }_{-\mathrm{\infty }}^x{\int }_{-\mathrm{\infty }}^{y}2e^{-u-v}dvdu={\int }_0^x{\int }_0^{y}2e^{-u-v}dvdu$$F_{XY}(x, y) = \int_{-\infty}^x \int_{-\infty}^y 2e^{-u-v} \, dv \, du = \int_0^x \int_0^y 2e^{-u-v} \, dv \, du$ $={\int }_0^x{[-2e^{-u-v}]}_0^{y}du={\int }_0^x(-2e^{-u-y}+2e^{-u-0})du$$= \int_0^x \left[ -2e^{-u-v} \right]_0^y \, du = \int_0^x \left( -2e^{-u-y} + 2e^{-u-0} \right) \, du$ $={\int }_0^x(-2e^{-u-y}+2e^{-u})du$$= \int_0^x \left( -2e^{-u-y} + 2e^{-u} \right) \, du$ $={[-\frac{2}{y}{e}^{-u-y}]}_0^x+{[-\frac{2}{1}{e}^{-u}]}_0^x$$= \left[ -\frac{2}{y}e^{-u-y} \right]_0^x + \left[ -\frac{2}{1}e^{-u} \right]_0^x$ $=(-\frac{2}{y}{e}^{-x-y}+\frac{2}{y}{e}^{-y})+(-2e^{-x}+2)$$= \left( -\frac{2}{y}e^{-x-y} + \frac{2}{y}e^{-y} \right) + \left( -2e^{-x} + 2 \right)$ $=(\frac{2}{y}{e}^{-y}-\frac{2}{y}{e}^{-x-y})+(2-2e^{-x})$$= \left( \frac{2}{y}e^{-y} - \frac{2}{y}e^{-x-y} \right) + \left( 2 - 2e^{-x} \right)$

Deriving Joint PDF from Joint CDF

Given the joint CDF $F_{XY}(x,y)$$F_{XY}(x, y)$:

Joint PDF: The joint PDF $f_{XY}(x,y)$$f_{XY}(x, y)$ can be found by taking the partial derivatives of the joint CDF: $f_{XY}(x,y)=\frac{{\partial }^2{F}_{XY}(x,y)}{\partial x\partial y}$$f_{XY}(x, y) = \frac{\partial^2 F_{XY}(x, y)}{\partial x \, \partial y}$.

Example

Assume we have the joint CDF values for certain points as follows:

$\begin{array}{c}{F}_{XY}(x,y)=\{\begin{array}{ll}0& x<0\text{ or }y<0\\ 1-e^{-x}-e^{-y}+e^{-x-y}& x\ge 0,y\ge 0\end{array}\end{array}$$F_{XY}(x, y) = \begin{cases} 0 & x < 0 \text{ or } y < 0 \\ 1 - e^{-x} - e^{-y} + e^{-x-y} & x \geq 0, y \geq 0 \end{cases}$

To find $f_{XY}(x,y)$$f_{XY}(x, y)$: $f_{XY}(x,y)=\frac{{\partial }^2}{\partial x\partial y}(1-e^{-x}-e^{-y}+e^{-x-y})$$f_{XY}(x, y) = \frac{\partial^2}{\partial x \, \partial y} \left( 1 - e^{-x} - e^{-y} + e^{-x-y} \right)$ $=\frac{\partial }{\partial y}\left(\frac{\partial }{\partial x}(1-e^{-x}-e^{-y}+e^{-x-y})\right)$$= \frac{\partial}{\partial y} \left( \frac{\partial}{\partial x} \left( 1 - e^{-x} - e^{-y} + e^{-x-y} \right) \right)$ $=\frac{\partial }{\partial y}(e^{-x}+e^{-x-y})$$= \frac{\partial}{\partial y} \left( e^{-x} + e^{-x-y} \right)$

$=-e^{-x-y}+e^{-x-y}$$= -e^{-x-y} + e^{-x-y}$ $=e^{-x-y}$$= e^{-x-y}$

5. Marginal Distribution

Marginal distributions are derived from the joint distribution by summing (discrete case) or integrating (continuous case) over the other variables.

5.1 Why We Need Joint Distributions

Joint distributions are essential for understanding the relationships between multiple random variables. They allow us to analyze how variables co-vary and the dependencies between them.

5.2 When We Know the Joint CDF, Can We Obtain Every Marginal CDF?

Yes, if we know the joint CDF, we can derive every marginal CDF by taking the appropriate limits.

5.3 Is the Reverse Statement True?

No, knowing the marginal CDFs does not provide enough information to determine the joint CDF. The joint CDF contains information about the dependencies and interactions between the variables that marginal CDFs alone do not capture.

Proof for the Statement: "When We Know the Joint CDF, We Can Derive Every Marginal CDF by Taking the Appropriate Limits"

For Discrete Random Variables:

Given the joint CDF $F_{X_1,X_2,\dots ,X_n}(x_1,x_2,\dots ,x_n)$$F_{X_1, X_2, \ldots, X_n}(x_1, x_2, \ldots, x_n)$: $F_{X_1,X_2,\dots ,X_n}(x_1,x_2,\dots ,x_n)=P(X_1\le x_1,X_2\le x_2,\dots ,X_n\le x_n)$$F_{X_1, X_2, \ldots, X_n}(x_1, x_2, \ldots, x_n) = P(X_1 \leq x_1, X_2 \leq x_2, \ldots, X_n \leq x_n)$The marginal CDF of $X_1$$X_1$ is: $F_{X_1}(x_1)=P(X_1\le x_1)$$F_{X_1}(x_1) = P(X_1 \leq x_1)$; This can be obtained by summing over all possible values of $X_2,X_3,\dots ,X_n$$X_2, X_3, \ldots, X_n$: $F_{X_1}(x_1)=\sum _{x_2=-\mathrm{\infty }}^{\mathrm{\infty }}\sum _{x_3=-\mathrm{\infty }}^{\mathrm{\infty }}\cdots \sum _{x_n=-\mathrm{\infty }}^{\mathrm{\infty }}{F}_{X_1,X_2,\dots ,X_n}(x_1,x_2,\dots ,x_n)$$F_{X_1}(x_1) = \sum_{x_2 = -\infty}^{\infty} \sum_{x_3 = -\infty}^{\infty} \cdots \sum_{x_n = -\infty}^{\infty} F_{X_1, X_2, \ldots, X_n}(x_1, x_2, \ldots, x_n)$

For Continuous Random Variables:

Given the joint CDF $F_{X_1,X_2,\dots ,X_n}(x_1,x_2,\dots ,x_n)$$F_{X_1, X_2, \ldots, X_n}(x_1, x_2, \ldots, x_n)$: $F_{X_1,X_2,\dots ,X_n}(x_1,x_2,\dots ,x_n)=P(X_1\le x_1,X_2\le x_2,\dots ,X_n\le x_n)$$F_{X_1, X_2, \ldots, X_n}(x_1, x_2, \ldots, x_n) = P(X_1 \leq x_1, X_2 \leq x_2, \ldots, X_n \leq x_n)$ The marginal CDF of $X_1$$X_1$ is: $F_{X_1}(x_1)=P(X_1\le x_1)$$F_{X_1}(x_1) = P(X_1 \leq x_1)$; This can be obtained by integrating over all possible values of $X_2,X_3,\dots ,X_n$$X_2, X_3, \ldots, X_n$: $F_{X_1}(x_1)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\cdots {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{F}_{X_1,X_2,\dots ,X_n}(x_1,x_2,\dots ,x_n)d{x}_{2}d{x}_3\cdots d{x}_n$$F_{X_1}(x_1) = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} F_{X_1, X_2, \ldots, X_n}(x_1, x_2, \ldots, x_n) \, dx_2 \, dx_3 \cdots dx_n$

Proof for the Statement: "Knowing the Marginal CDFs Does Not Provide Enough Information to Determine the Joint CDF"

To prove this, let's consider two random variables $X$$X$ and $Y$$Y$.

Example 1: Independent Variables

Suppose $X$$X$ and $Y$$Y$ are independent random variables with known marginal CDFs $F_X(x)$$F_X(x)$ and $F_Y(y)$$F_Y(y)$. The joint CDF of $X$$X$ and $Y$$Y$ is given by: $F_{X,Y}(x,y)=P(X\le x,Y\le y)=F_X(x)F_Y(y)$$F_{X,Y}(x,y) = P(X \leq x, Y \leq y) = F_X(x)F_Y(y)$

Example 2: Dependent Variables

Now consider the case where $X$$X$ and $Y$$Y$ are perfectly correlated, such that $Y=X$$Y = X$. In this case, the marginal CDFs remain the same $F_X(x)$$F_X(x)$ and $F_Y(y)$$F_Y(y)$, but the joint CDF is different: $F_{X,Y}(x,y)=P(X\le x,Y\le y)=P(X\le x,X\le y)=P(X\le min(x,y))=F_X(min(x,y))$$F_{X,Y}(x,y) = P(X \leq x, Y \leq y) = P(X \leq x, X \leq y) = P(X \leq \min(x, y)) = F_X(\min(x, y))$

This demonstrates that knowing the marginal CDFs $F_X(x)$$F_X(x)$ and $F_Y(y)$$F_Y(y)$ alone is not sufficient to determine the joint CDF $F_{X,Y}(x,y)$$F_{X,Y}(x,y)$ because the joint CDF also captures the dependency structure between the variables.

6. Independent Random Variables

6.1 Definition

Random variables $X_1,\dots ,X_n$$X_1, \ldots, X_n$ are said to be independent if their joint CDF factors into the product of their marginal CDFs: $F(x_1,\dots ,x_n)=F_{X_1}(x_1)\cdots F_{X_n}(x_n)$$F(x_1, \ldots, x_n) = F_{X_1}(x_1) \cdots F_{X_n}(x_n)$ for all $x_1,\dots ,x_n$$x_1, \ldots, x_n$.

Discrete Case

For discrete random variables: $F(x_1,\dots ,x_n)=F_{X_1}(x_1)\cdots F_{X_n}(x_n)⟺p(x_1,\dots ,x_n)=p_{X_1}(x_1)\cdots p_{X_n}(x_n)$$F(x_1, \ldots, x_n) = F_{X_1}(x_1) \cdots F_{X_n}(x_n) \iff p(x_1, \ldots, x_n) = p_{X_1}(x_1) \cdots p_{X_n}(x_n)$ where $p(x_1,\dots ,x_n)$$p(x_1, \ldots, x_n)$ is the joint PMF.

Continuous Case

For continuous random variables: $F(x_1,\dots ,x_n)=F_{X_1}(x_1)\cdots F_{X_n}(x_n)⟺f(x_1,\dots ,x_n)=f_{X_1}(x_1)\cdots f_{X_n}(x_n)$$F(x_1, \ldots, x_n) = F_{X_1}(x_1) \cdots F_{X_n}(x_n) \iff f(x_1, \ldots, x_n) = f_{X_1}(x_1) \cdots f_{X_n}(x_n)$ where $f(x_1,\dots ,x_n)$$f(x_1, \ldots, x_n)$ is the joint PDF.

6.2 Property 1: Independence of $X$$X$ and $Y$$Y$

Two random variables $X$$X$ and $Y$$Y$ are independent if: $P(X\in A,Y\in B)=P(X\in A)P(Y\in B)$$P(X \in A, Y \in B) = P(X \in A)P(Y \in B)$ i.e., the events $\{X\in A\}$$\{X \in A\}$ and $\{Y\in B\}$$\{Y \in B\}$ are independent.

6.3 Property 2: Function of Independent Variables

If $X$$X$ and $Y$$Y$ are independent, then $Z=g(X)$$Z = g(X)$ and $W=h(Y)$$W = h(Y)$ are also independent.

Proof: Given two independent random variables $X$$X$ and $Y$$Y$, we need to show that for any measurable functions $g$$g$ and $h$$h$, the random variables $g(X)$$g(X)$ and $h(Y)$$h(Y)$ are also independent. Specifically, we need to prove that: $P(g(X)\le z,h(Y)\le w)=P(g(X)\le z)P(h(Y)\le w).$$P(g(X) \leq z, h(Y) \leq w) = P(g(X) \leq z) P(h(Y) \leq w).$

1. Independence Definition for $X$$X$ and $Y$$Y$

By definition, $X$$X$ and $Y$$Y$ are independent if for all measurable sets $A$$A$ and $B$$B$: $P(X\in A,Y\in B)=P(X\in A)P(Y\in B).$$P(X \in A, Y \in B) = P(X \in A) P(Y \in B).$

2. Consider the Events for Functions of $X$$X$ and $Y$$Y$

Define the events:

• $A=\{X\in g^{-1}((-\mathrm{\infty },z])\}$$A = \{X \in g^{-1}((-\infty, z])\}$

• $B=\{Y\in h^{-1}((-\mathrm{\infty },w])\}$$B = \{Y \in h^{-1}((-\infty, w])\}$

Here, $g^{-1}((-\mathrm{\infty },z])$$g^{-1}((-\infty, z])$ is the preimage of the interval $(-\mathrm{\infty },z]$$(-\infty, z]$ under the function $g$$g$, and similarly for $h$$h$.

3. Relate the Events to $g(X)$$g(X)$ and $h(Y)$$h(Y)$

The events can be written as:

• $\{g(X)\le z\}=\{X\in g^{-1}((-\mathrm{\infty },z])\}$$\{g(X) \leq z\} = \{X \in g^{-1}((-\infty, z])\}$

• $\{h(Y)\le w\}=\{Y\in h^{-1}((-\mathrm{\infty },w])\}$$\{h(Y) \leq w\} = \{Y \in h^{-1}((-\infty, w])\}$

4. Apply the Definition of Independence

Using the independence of $X$$X$ and $Y$$Y$, we know: $P(X\in g^{-1}((-\mathrm{\infty },z]),Y\in h^{-1}((-\mathrm{\infty },w]))=P(X\in g^{-1}((-\mathrm{\infty },z]))P(Y\in h^{-1}((-\mathrm{\infty },w])).$$P(X \in g^{-1}((-\infty, z]), Y \in h^{-1}((-\infty, w])) = P(X \in g^{-1}((-\infty, z])) P(Y \in h^{-1}((-\infty, w])).$

5. Translate Back to the Functions $g(X)$$g(X)$ and $h(Y)$$h(Y)$

This equation translates back to the original functions as: $P(g(X)\le z,h(Y)\le w)=P(g(X)\le z)P(h(Y)\le w).$$P(g(X) \leq z, h(Y) \leq w) = P(g(X) \leq z) P(h(Y) \leq w).$

7. Conditional Distribution

7.1 Discrete Random Variables

For discrete random variables $X$$X$ and $Y$$Y$ with joint PMF $p_{XY}(x,y)$$p_{XY}(x, y)$, the conditional PMF of $Y$$Y$ given $X$$X$ is: $p_{Y|X}(y|x)=P(Y=y\mid X=x)=\frac{P(X=x,Y=y)}{P(X=x)}=\frac{p_{XY}(x,y)}{p_X(x)}$$p_{Y|X}(y|x) = P(Y = y \mid X = x) = \frac{P(X = x, Y = y)}{P(X = x)} = \frac{p_{XY}(x, y)}{p_X(x)}$

If $p_X(x)>0$$p_X(x) > 0$. The probability is defined to be zero if $p_X(x)=0$$p_X(x) = 0$.

7.2 Continuous Random Variables

For continuous random variables $X$$X$ and $Y$$Y$ with joint PDF $f_{XY}(x,y)$$f_{XY}(x, y)$, the conditional PDF of $Y$$Y$ given $X$$X$ is: $f_{Y|X}(y|x)=\frac{f_{XY}(x,y)}{f_X(x)}$$f_{Y|X}(y|x) = \frac{f_{XY}(x, y)}{f_X(x)}$ If $0<f_X(x)<\mathrm{\infty }$$0 < f_X(x) < \infty$, and zero otherwise.

7.3 Remarks

1. For each fixed $x$$x$:

   $p_{Y|X}(y|x)$$p_{Y|X}(y|x)$ is a PMF for $y$$y$ | $f_{Y|X}(y|x)$$f_{Y|X}(y|x)$ is a PDF for $y$$y$.

2. Multiplication Law: $p_{XY}(x,y)=p_{Y|X}(y|x)p_X(x)$$p_{XY}(x, y) = p_{Y|X}(y|x) p_X(x)$ | $f_{XY}(x,y)=f_{Y|X}(y|x)f_X(x)$$f_{XY}(x, y) = f_{Y|X}(y|x) f_X(x)$

3. Law of Total Probability: $p_Y(y)=\sum _x{p}_{Y|X}(y|x)p_X(x)$$p_Y(y) = \sum_x p_{Y|X}(y|x) p_X(x)$ | $f_Y(y)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{f}_{Y|X}(y|x)f_X(x)dx$$f_Y(y) = \int_{-\infty}^{\infty} f_{Y|X}(y|x) f_X(x) \, dx$

4. Independence:

   $X$$X$ and $Y$$Y$ are independent if and only if: $p_{Y|X}(y|x)=p_Y(y)$$p_{Y|X}(y|x) = p_Y(y)$ | $f_{Y|X}(y|x)=f_Y(y)$$f_{Y|X}(y|x) = f_Y(y)$

8. Functions of Random Variables

Question: For given random variables $X_1,\dots ,X_n$$X_1, \ldots, X_n$, how to derive the distributions of their transformations?

8.1 Method 1: Method of Events

Let $\mathbf{X}=(X_1,\dots ,X_n)$$\mathbf{X} = (X_1, \ldots, X_n)$ be random variables and $Y=g(\mathbf{X})$$Y = g(\mathbf{X})$. Then the distribution of $Y$$Y$ is determined as follows: for any event $B$$B$ defined by $Y$$Y$, $P(Y\in B)=P(\mathbf{X}\in A)$$P(Y \in B) = P(\mathbf{X} \in A)$, where $A=g^{-1}(B)$$A = g^{-1}(B)$ is the preimage of $B$$B$ under the function $g$$g$.

Example 1.11 (Univariate Discrete Random Variable)

Let $X$$X$ be a discrete random variable taking values $x_i$$x_i$, $i=1,2,\dots$$i = 1, 2, \ldots$, and $Y=g(X)$$Y = g(X)$. Then $Y$$Y$ is also a discrete random variable taking values $y_j$$y_j$, $j=1,2,\dots$$j = 1, 2, \ldots$. To determine the PMF of $Y$$Y$, by taking $B=\{y_j\}$$B = \{y_j\}$, we have: $A=\{x_i:g(x_i)=y_j\}$$A = \{x_i : g(x_i) = y_j\}$ and hence, $p_Y(y_j)=P(\{y_j\})=P(A)=\sum _{x_i\in A}{p}_X(x_i)$$p_Y(y_j) = P(\{y_j\}) = P(A) = \sum_{x_i \in A} p_X(x_i)$.

Example 1.12 (Sum of Two Discrete Random Variables)

Let $X$$X$ and $Y$$Y$ be random variables with joint PMF $p(x,y)$$p(x, y)$. Find the distribution of $Z=X+Y$$Z = X + Y$.

To find the distribution of $Z$$Z$, we use the convolution of the PMFs of $X$$X$ and $Y$$Y$: $p_Z(z)=P(Z=z)=P(X+Y=z)=\sum _{x}P(X=x,Y=z-x)=\sum _x{p}_{XY}(x,z-x)$$p_Z(z) = P(Z = z) = P(X + Y = z) = \sum_{x} P(X = x, Y = z - x) = \sum_{x} p_{XY}(x, z - x)$.

For the exercise $Z=Y-X$$Z = Y - X$, the distribution can be found similarly: $p_Z(z)=P(Z=z)=P(Y-X=z)=\sum _{x}P(X=x,Y=x+z)=\sum _x{p}_{XY}(x,x+z)$$p_Z(z) = P(Z = z) = P(Y - X = z) = \sum_{x} P(X = x, Y = x + z) = \sum_{x} p_{XY}(x, x + z)$.

8.2 Method 2: Method of CDF

This is a special case of Method 1. Let $Y$$Y$ be a function of random variables $X_1,\dots ,X_n$$X_1, \ldots, X_n$.

1. Find the region $Y\le y$$Y \leq y$ in the $(x_1,\dots ,x_n)$$(x_1, \ldots, x_n)$ space.

2. Find $F_Y(y)=P(Y\le y)$$F_Y(y) = P(Y \leq y)$ by summing the joint PMF or integrating the joint PDF of $X_1,\dots ,X_n$$X_1, \ldots, X_n$ over the region $Y\le y$$Y \leq y$

3. For the continuous case, find the PDF of $Y$$Y$ by differentiating $F_Y(y)$$F_Y(y)$: $f_Y(y)=\frac{d}{dy}{F}_Y(y)$$f_Y(y) = \frac{d}{dy} F_Y(y)$

Example for Continuous Case

Let $X$$X$ and $Y$$Y$ be continuous random variables with joint PDF $f_{XY}(x,y)$$f_{XY}(x, y)$. To find the PDF of $Z=X+Y$$Z = X + Y$:

1. Define the region $Z\le z$$Z \leq z$: $F_Z(z)=P(X+Y\le z)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{-\mathrm{\infty }}^{z-x}{f}_{XY}(x,y)dydx.$$F_Z(z) = P(X + Y \leq z) = \int_{-\infty}^{\infty} \int_{-\infty}^{z - x} f_{XY}(x, y) \, dy \, dx.$

2. Differentiate to find the PDF: $f_Z(z)=\frac{d}{dz}{F}_Z(z)=\frac{d}{dz}({\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{-\mathrm{\infty }}^{z-x}{f}_{XY}(x,y)dydx).$$f_Z(z) = \frac{d}{dz} F_Z(z) = \frac{d}{dz} \left( \int_{-\infty}^{\infty} \int_{-\infty}^{z - x} f_{XY}(x, y) \, dy \, dx \right).$

Using Leibniz's rule for differentiation under the integral sign, we get: $f_Z(z)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{f}_{XY}(x,z-x)dx.$$f_Z(z) = \int_{-\infty}^{\infty} f_{XY}(x, z - x) \, dx.$

8.3 Leibniz's Rule

Example for Continuous Case: Finding the PDF of $Z=X+Y$$Z = X + Y$

Given two continuous random variables $X$$X$ and $Y$$Y$ with a joint PDF $f_{XY}(x,y)$$f_{XY}(x, y)$, we aim to find the PDF of their sum $Z=X+Y$$Z = X + Y$

Step-by-Step Derivation

• Define the Region $Z\le z$$Z \leq z$: The cumulative distribution function (CDF) of $Z$$Z$ is given by: $F_Z(z)=P(Z\le z)=P(X+Y\le z).$$F_Z(z) = P(Z \leq z) = P(X + Y \leq z).$ This can be written as a double integral over the region where the sum of $X$$X$ and $Y$$Y$ is less than or equal to $z$$z$: $F_Z(z)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{-\mathrm{\infty }}^{z-x}{f}_{XY}(x,y)dydx.$$F_Z(z) = \int_{-\infty}^{\infty} \int_{-\infty}^{z-x} f_{XY}(x, y) \, dy \, dx.$

1. Differentiate to Find the PDF: The PDF $f_Z(z)$$f_Z(z)$ is the derivative of the CDF $F_Z(z)$$F_Z(z)$: $f_Z(z)=\frac{d}{dz}{F}_Z(z)=\frac{d}{dz}({\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{-\mathrm{\infty }}^{z-x}{f}_{XY}(x,y)dydx).$$f_Z(z) = \frac{d}{dz} F_Z(z) = \frac{d}{dz} \left( \int_{-\infty}^{\infty} \int_{-\infty}^{z-x} f_{XY}(x, y) \, dy \, dx \right).$ Here, we need to differentiate an integral with respect to $z$$z$. For this purpose, we apply Leibniz's rule for differentiation under the integral sign.

Applying Leibniz's Rule

Leibniz's rule for differentiation under the integral sign states: $\frac{d}{dz}({\int }_{a(z)}^{b(z)}f(x,z)dx)={\int }_{a(z)}^{b(z)}\frac{\partial }{\partial z}f(x,z)dx+f(b(z),z)\cdot \frac{d}{dz}b(z)-f(a(z),z)\cdot \frac{d}{dz}a(z).$$\frac{d}{dz} \left( \int_{a(z)}^{b(z)} f(x, z) \, dx \right) = \int_{a(z)}^{b(z)} \frac{\partial}{\partial z} f(x, z) \, dx + f(b(z), z) \cdot \frac{d}{dz} b(z) - f(a(z), z) \cdot \frac{d}{dz} a(z).$

In our case: $F_Z(z)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{-\mathrm{\infty }}^{z-x}{f}_{XY}(x,y)dydx.$$F_Z(z) = \int_{-\infty}^{\infty} \int_{-\infty}^{z-x} f_{XY}(x, y) \, dy \, dx.$

By applying Leibniz's rule, we get: $f_Z(z)=\frac{d}{dz}({\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{-\mathrm{\infty }}^{z-x}{f}_{XY}(x,y)dydx).$$f_Z(z) = \frac{d}{dz} \left( \int_{-\infty}^{\infty} \int_{-\infty}^{z-x} f_{XY}(x, y) \, dy \, dx \right).$

Since the lower limit of the inner integral is $-\mathrm{\infty }$$-\infty$ (which is constant and its derivative with respect to $z$$z$ is zero) and the upper limit is $z-x$$z - x$, we apply the rule: $f_Z(z)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{f}_{XY}(x,z-x)\cdot \frac{d}{dz}(z-x)dx.$$f_Z(z) = \int_{-\infty}^{\infty} f_{XY}(x, z - x) \cdot \frac{d}{dz} (z - x) \, dx.$

Given that $\frac{d}{dz}(z-x)=1$$\frac{d}{dz} (z - x) = 1$: $f_Z(z)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{f}_{XY}(x,z-x)dx.$$f_Z(z) = \int_{-\infty}^{\infty} f_{XY}(x, z - x) \, dx.$

Leibniz's Rule for Differentiation Under the Integral Sign

Leibniz's rule for differentiation under the integral sign provides a way to differentiate an integral with variable limits and an integrand that depends on the variable of differentiation.

Leibniz's Rule Statement

The rule states: $\frac{d}{dz}({\int }_{a(z)}^{b(z)}f(x,z)dx)={\int }_{a(z)}^{b(z)}\frac{\partial }{\partial z}f(x,z)dx+f(b(z),z)\cdot \frac{d}{dz}b(z)-f(a(z),z)\cdot \frac{d}{dz}a(z).$$\frac{d}{dz} \left( \int_{a(z)}^{b(z)} f(x, z) \, dx \right) = \int_{a(z)}^{b(z)} \frac{\partial}{\partial z} f(x, z) \, dx + f(b(z), z) \cdot \frac{d}{dz} b(z) - f(a(z), z) \cdot \frac{d}{dz} a(z).$

Explanation and Proof

• Consider the integral with variable limits: $I(z)={\int }_{a(z)}^{b(z)}f(x,z)dx.$$I(z) = \int_{a(z)}^{b(z)} f(x, z) \, dx.$

1. Differentiate $I(z)$$I(z)$ with respect to $z$$z$: $\frac{dI}{dz}=\frac{d}{dz}({\int }_{a(z)}^{b(z)}f(x,z)dx).$$\frac{dI}{dz} = \frac{d}{dz} \left( \int_{a(z)}^{b(z)} f(x, z) \, dx \right).$

2. Differentiate inside the integral (where possible): To handle the differentiation, we need to consider three parts:

   • The variation of the integrand $f(x,z)$$f(x, z)$ with respect to $z$$z$.

   • The variation of the upper limit $b(z)$$b(z)$ with respect to $z$$z$.

   • The variation of the lower limit $a(z)$$a(z)$ with respect to $z$$z$.

3. Apply the Fundamental Theorem of Calculus and the Chain Rule:

   By the Fundamental Theorem of Calculus and the Chain Rule, we get: $\frac{d}{dz}({\int }_{a(z)}^{b(z)}f(x,z)dx)={\int }_{a(z)}^{b(z)}\frac{\partial f(x,z)}{\partial z}dx+f(b(z),z)\cdot \frac{db(z)}{dz}-f(a(z),z)\cdot \frac{da(z)}{dz}.$$\frac{d}{dz} \left( \int_{a(z)}^{b(z)} f(x, z) \, dx \right) = \int_{a(z)}^{b(z)} \frac{\partial f(x, z)}{\partial z} \, dx + f(b(z), z) \cdot \frac{d b(z)}{dz} - f(a(z), z) \cdot \frac{d a(z)}{dz}.$

   • Integral with respect to $z$$z$: ${\int }_{a(z)}^{b(z)}\frac{\partial f(x,z)}{\partial z}dx$$\int_{a(z)}^{b(z)} \frac{\partial f(x, z)}{\partial z} \, dx$ considers how the integrand changes with $z$$z$ while keeping the limits constant.

   • Upper limit $b(z)$$b(z)$: $f(b(z),z)\cdot \frac{db(z)}{dz}$$f(b(z), z) \cdot \frac{d b(z)}{dz}$ considers the contribution from the changing upper limit.

   • Lower limit $a(z)$$a(z)$: $-f(a(z),z)\cdot \frac{da(z)}{dz}$$-f(a(z), z) \cdot \frac{d a(z)}{dz}$ considers the contribution from the changing lower limit.

4. Combine the three components: $\frac{d}{dz}({\int }_{a(z)}^{b(z)}f(x,z)dx)={\int }_{a(z)}^{b(z)}\frac{\partial f(x,z)}{\partial z}dx+f(b(z),z)\cdot \frac{db(z)}{dz}-f(a(z),z)\cdot \frac{da(z)}{dz}.$$\frac{d}{dz} \left( \int_{a(z)}^{b(z)} f(x, z) \, dx \right) = \int_{a(z)}^{b(z)} \frac{\partial f(x, z)}{\partial z} \, dx + f(b(z), z) \cdot \frac{d b(z)}{dz} - f(a(z), z) \cdot \frac{d a(z)}{dz}.$