**This research was conducted in order to answer the questions asked for a job interview.**

Facebook recently introduced a new bidding type, **"average bidding”**, as an alternative to its exisiting bidding type, called **maximum bidding”**.

One of our clients, **bombabomba.com**, has decided to test this new feature and wants to conduct an A/B test to understand if average bidding brings more conversions than maximum bidding.

In this A/B test, bombabomba.com randomly splits its audience into two equally sized groups, e.g. the test and the control group.

A Facebook ad campaign with “maximum bidding” is served to “control group” and another campaign with “average bidding” is served to the “test group”.

The A/B test run for 1 month and bombabomba.com now expects you to analyze and present the results of this A/B test.

**You should answer the following questions in your presentation:**

- How would you define the hypothesis of this A/B test?
- Can we conclude statistically significant results?
- Which statistical test did you use, and why ?
- Based on your answer to Question 2, what would be your recommendation to client?

In [1]:

```
%%html
<style>
@import url('https://fonts.googleapis.com/css?family=Orbitron|Roboto');
a {color: #37c9e1; font-family: 'Roboto';}
h1 {color: #C20E69; font-family: 'Poppins'}
h2, h3 {color: #25B89B; font-family: 'Poppins';}
h4 {color: #818286; font-family: 'Roboto';}
</style>
```

In [35]:

```
# Libraries
# Main
#----------------------
import pandas as pd
import numpy as np
from abtest import *
# Data Viz
#----------------------
import seaborn as sns
import matplotlib.pyplot as plt
import pylab
plt.style.use('fivethirtyeight')
# Stats
#----------------------
from scipy import stats
from scipy.stats import skewtest
from scipy.stats import median_test
from scipy.stats import shapiro
from statsmodels.stats.proportion import proportions_ztest
# Settings
#----------------------
pd.set_option("display.max_columns",None);
pd.set_option("display.max_rows",None);
```

In [36]:

```
control = pd.read_excel(r"C:\Users\oe\VBO\datasets\ab_testing_data.xlsx", sheet_name="Control Group")
test = pd.read_excel(r"C:\Users\oe\VBO\datasets\ab_testing_data.xlsx", sheet_name="Test Group")
```

In [37]:

```
def descriptive_statistics(dataframe):
print("Data Descriptive Statistics","\n")
name =[x for x in globals() if globals()[x] is dataframe][0]
print("Dataframe: %s" % name,"\n")
print("Data Shape: ",dataframe.shape,"\n")
print(dataframe.info(),"\n")
print("First 3 rows:","\n",dataframe.head(3),"\n")
if dataframe.isna().any().sum() == 0:
print("There are no missing value.")
else:
print(dataframe.isna().any().sum())
print("_"*60)
```

In [38]:

```
for i in [control,test]:
print()
descriptive_statistics(i)
```

In this research, there are two different groups and will investigate whether there is a statistically significant difference between the averages of these two different groups.

We have 40 observation and 4 feature in this dataset.

Two different groups will be taken from the **"Purchase"** column for comparison.

Control group is standart bidding type: " Maximum Bidding "

Test Group is new bidding type "Average Bidding "

In [39]:

```
control.describe([0.10, 0.25, 0.50, 0.75, 0.90, 0.95, 0.99]).T
```

Out[39]:

In [40]:

```
test.describe([0.10, 0.25, 0.50, 0.75, 0.90, 0.95, 0.99]).T
```

Out[40]:

**Apparently, there are no outliers in these two group data, and the mean and median values are close to each other. However, box-plot graph will be applied to make sure.**

In [41]:

```
AB = pd.DataFrame()
AB["A"] = control["Purchase"]
AB["B"] = test["Purchase"]
AB.describe()
```

Out[41]:

In [42]:

```
AB.describe().T
```

Out[42]:

In [43]:

```
vals, names, xs = [],[],[]
for i, col in enumerate(AB.columns):
vals.append(AB[col].values)
names.append(col)
xs.append(np.random.normal(i + 1, 0.04, AB[col].values.shape[0])) # adds jitter to the data points - can be adjusted
plt.boxplot(vals, labels=names)
palette = ['#25B89B', '#C20E69']
for x, val, c in zip(xs, vals, palette):
plt.scatter(x, val, alpha=0.7, color=c)
plt.show()
```

A is control group -- Maximum Bidding

B is test group -- Avarage Bidding

When we looking at the averages of these two groups, the test group "average bidding", looks better.

However, this is only a mathematical inference and **no business decision shouldn't be made based on this result.**

Whether the result is random or not should be statistically investigated.

The main question is;

**Is there a statistically significant difference between these two average of groups?**

**Assumptions:**

- Normality Assumption
- This situation will be observed first with qqplot, then final decision will be reached with shapiro-wilks test and skewtest

- Homogeneity of variance
- Levene test

In [44]:

```
stats.probplot(AB["A"], dist="norm", plot=pylab)
pylab.title("Control Group - Maximum Bidding")
pylab.show()
```

The distribution seems normal, however the shapiro-wilk test will be applied to be sure.

In [45]:

```
stats.probplot(AB["B"], dist="norm", plot=pylab)
pylab.title("Test Group - Average Bidding")
pylab.show()
```

➠ Ho: The data follows a normal distribution.

➠ H1: The data does not follow a normal distribution.

➠ Ho: There is no difference between the variance of the both groups.

➠ H1: There is a significant difference between the variance of the both groups.

➠ Ho: There is not a statistically significant difference between the means with

➠ H1: There is a statistically significant difference between the means with

- In this case the non parametric test "Mann Whitney - U test" is applied.

- In this case
**"Welch Test"**is applied. - For this option in
**Two Independent Sample T Test Function**`equal_var`

=`True`

argument is applied.

In [46]:

```
test_ab(AB,"A", "B", alpha = 0.05, plot=True)
```

As a result of the statistical tests applied to the data obtained by the two groups in the research, the groups **provided** all the necessary statistical assumptions in the literature.

In this direction, it was **proved** statistically and scientifically that there was **no significant difference** between the **averages** of the two groups with **95% confidence.**

**In line with these results, there is not statistically significant difference between purchase of the control group (Max Bidding) and purchase of the test group (Average Bidding)**

- It is a rate that shows how often users who saw the ad CLICKED the ad.
- Ad Clicks / Ad Shows

In [76]:

```
control_CTR=control["Click"].sum()/control["Impression"].sum()
test_CTR= test["Click"].sum() / test["Impression"].sum()
print(control_CTR)
print(test_CTR)
```

In [70]:

```
df_CTR = pd.DataFrame({"Existing(MAX) CTR":[control_CTR], "New(AVG) CTR": [test_CTR]})
df_CTR.plot.bar(rot=0).legend(loc=3) ;
```

**Hypothesis**

➠Ho: The experiment has no effect on user behavior. (p_cont = p_test)

➠H1: The experiment has an effect on user behavior. (p_cont ≠ p_test)

In [49]:

```
clicks= np.array([control["Click"].sum(),test["Click"].sum()])
shows= np.array([control["Impression"].sum(), test["Impression"].sum()])
proportions_ztest(count=clicks, nobs=shows)
```

Out[49]:

p - value < 0.05 ➠ **Reject Ho hypthesis. ( The experiment has no effect on user behavior. ) **

When the bidding methods are examined, the effect of these methods on user behavior **(clicks)** is **different.**

And this difference is in favor of the current **"max bidding"** method.

Conversion rate is calculated by dividing the **number of conversions** by the **total number of visitors.**

For example, if an ecommerce site gets 200 visitors a month and has 50 sales, the conversion rate would equal **25%**.

In [64]:

```
control_CR = control["Purchase"].sum()/control["Impression"].sum()
test_CR = test["Purchase"].sum() / test["Impression"].sum()
print("Max Bidding CR:", round(control_CR,4))
print("AVG Bidding CR:", round(test_CR,4))
```

In [69]:

```
df_CR=pd.DataFrame({"Existing(max) CR":[control_CR], "New(avg) CR": [test_CR]})
df_CR.plot.bar(rot=0).legend(loc=3) ;
```

**Hypothesis**

➠Ho: The experiment has no effect on user behavior. (p_cont = p_test)

➠H1: The experiment has an effect on user behavior. (p_cont ≠ p_test)

In [71]:

```
purchase= np.array([control["Purchase"].sum(),test["Purchase"].sum()])
impression= np.array([control["Impression"].sum(), test["Impression"].sum()])
proportions_ztest(count=purchase, nobs=impression)
```

Out[71]:

p - value < 0.05 ➠ **Reject Ho hypthesis. ( The experiment has no effect on user behavior. ) **

When the bidding methods are examined, the effect of these methods on user behavior **(Purchasing)** is **different.**

And this difference is in favor of the current **"max bidding"** method.

- Control and test group data were analyzed.
- No NA value was observed in both groups.
- When looked with the boxplot method, it was observed that there was no outlier in both groups.
- Testing was done between these two groups. Control group using the max bidding method as called A Test group using the new method, average bidding as called B
- Two independent samples t test was deemed appropriate for this comparison.
- For two independent samples t test; variance homogeneity and normality assumption were provided.

**Is the new system advantageous?**

As a result of the statistical tests applied to the data obtained by the two groups in the research, the groups **provided** all the necessary statistical assumptions in the literature.

In this direction, it was **proved** statistically and scientifically that there was **no significant difference** between the **averages** of the two groups with **95% confidence.**

In line with these results, **there is not statistically significant difference** between purchase of the control group (Max Bidding) and purchase of the test group (Average Bidding)

**The new method does not benefit your agency in terms of purchasing**

**Did the user who saw the ad click on the ad in the new system?**

There is a difference in **Clicking** between the two methods, but this is in favor of **"Maximum Bidding"**, which is the system your agency **currently use.**

**Did visitors buy the product in the new system?**

There is a difference in **Purchasing** between the two methods, but this is in favor of **"Maximum Bidding"**, which is the system your agency **currently use.**

As a result of the statistical tests carried out, the new system "Average Bidding" has not been observed to benefit your company.

Some advice for action:

**We recommend that you continue with the existing system.**

- A new test can be done by increasing the number of samples.