Statistics 101 — What is Sampling With Replacement and Without Replacement ?
Sampling With Replacement
Consider a bag of fruits. For simplicity, let’s suppose there are only 4 fruits in the bag with only one fruit of each kind— Apple, Guava, Mango and Orange.
If you were to pick two fruits with replacement from the bag of fruits, first you pick say Apple. Once you pick Apple and place it in the bag (i.e you’re replacing it in the bag), now you have to pick another fruit. Now there are 4 choices for picking the 2nd fruit. How ?
1st choice, 2nd choice
(Apple, Apple) # Apple can be 2nd choice as it is replaced in the bag.
(Apple, Guava)
(Apple, Mango)
(Apple, Orange)
That is there is a 1/4 probability of choosing any fruit as the second pick. Overall, there are 16 possibilities (4 choices for first * 4 choices for second) of choosing a fruit pair (assuming we distinguish between the first and second). Here are the possibilities:
(Apple, Apple)
(Apple, Guava)
(Apple, Mango)
(Apple, Orange)(Guava, Apple)
(Guava, Guava)
(Guava, Mango)
(Guava, Orange)(Mango, Apple)
(Mango, Guava)
(Mango, Mango)
(Mango, Orange)(Orange, Apple)
(Orange, Guava)
(Orange, Mango)
(Orange, Orange)
The probability of first pick is 1/4 and the probability of the second pick is also 1/4. So the probability of picking any two fruits with replacement is (1/4) * (1/4) = 0.0625
But what if you don’t replace the first fruit before picking the second ? In other words, what happens if you sample without replacement ?
Sampling Without Replacement
Considering the same bag of fruits and you were to pick two fruits but without replacing the first fruit in the bag. Now let’s say you first pick Apple and don’t replace it in the bag. Now there are only 3 possibilities for picking the second fruit. How ?
1st choice, 2nd choice
(Apple, Guava)
(Apple, Mango)
(Apple, Orange)
That is there is a probability of 1/3 for choosing the second fruit. Overall, there are only 12 (4 choices for first * 3 choices for second) possibilities of choosing a fruit pair (assuming we distinguish between the first and second). Here are the possibilities:
(Apple, Guava)
(Apple, Mango)
(Apple, Orange)(Guava, Apple)
(Guava, Mango)
(Guava, Orange)(Mango, Apple)
(Mango, Guava)
(Mango, Orange)(Orange, Apple)
(Orange, Guava)
(Orange, Mango)
The probability of first pick is 1/4 and the probability of the second pick is 1/3. So the probability of picking any two fruits without replacement is (1/4) * (1/3) = 0.083
What’s the difference ?
When you sample with replacement, the two items you pick are independent. Simply, what this means is what we got for the first pick doesn’t affect the second pick.
When you sample without replacement, the two items you pick are dependent. Simply, what this means is what we got for the first pick affects the second pick.