
1. Re: Why the sample size calculator over estimates the sample real needed?
jrohit Feb 8, 2017 4:22 AM (in response to miguelm62125791)Hi,
The sample calculator gives an estimate on the amount of time and traffic based on the variables provided. Since this is an estimate and not an accurate number the chances of this meeting the actual results for a campaign are less likely. There is a high probability of a campaign reaching the expected lift much earlier or much later depending on the nature of the campaign and components being tested within the campaign.
Hope this helps!
Thanks,
Rohit

2. Re: Why the sample size calculator over estimates the sample real needed?
miguelm62125791 Feb 8, 2017 8:55 AM (in response to jrohit)Thank you for your reply Rohit
The point is that the sample calculator always over estimates the amount of traffic needed to reach conclusive results, this also happens using others size calculators.

3. Re: Why the sample size calculator over estimates the sample real needed?
chintan601 Feb 8, 2017 2:28 PM (in response to miguelm62125791)Hi Rohit, I'm a product manager on Target and let me try to address your concern. First and most importantly, the sample size calculator does not provide an estimate. It stipulates the minimum sample size required in order to guarantee that your falsepositive rate (ie inverse of Confidence) is bounded. Which means that if you desire a 95% confidence (or 5% falsepositive rate), you MUST wait until this sample size has transpired in order to guarantee that only 1 out of 20 times (ie 5%) will a test yield a falsepositive. Only after the test has crossed the sample size, a user should look at the Confidencevalue and ascertain that it is indeed above 95%. If the confidencevalue after the sample side has been acquired is below 95%, this means that at a 95% threshold for significance, your test in inconclusive. If all of this didnt make sense, here is a simple 3step workflow to do ABtesting correctly:
1. Compute the sample size with desired significance (say 95%) and most accurate guesses for "Baseline CR", "Minimum detectable lift". If you have more than 2 experiences, dont forget to apply Bonneferroni correction.
2. Wait until each experience has acquired this sample size.
3. Evaluate only at this point, whether the Confidence value shown in the Reports is above 95%. If its not, your test is inconclusive and you do not have a winner for this test.
I understand this is something you may not have done before, but our years of analysis have shown that if users dont wait until the sample size, their tests are 56% likely to find a falsepositive (ie a 'winner' that actually performs worse than control in reality).
Hope that helps!

4. Re: Why the sample size calculator over estimates the sample real needed?
miguelm62125791 Feb 9, 2017 12:37 AM (in response to chintan601)Hi Chintan,
Thank you for your answer, it helps a lot.
If I understand well, we only should look at the confidence level when the sample reach the minimum sample estimated by the calculator. My question then is, Why target is telling us that there is a 99% confidence level, when the sample isn't enough?
Thank you, best regards.

5. Re: Why the sample size calculator over estimates the sample real needed?
jrohit Feb 9, 2017 1:18 AM (in response to miguelm62125791)Hi,
Target does not take the sample size into account because target reports provide the actual performance of experiences in reports. Thus here the confidence number is a reflection of actual conversion rates captured by the system in the backend and then plotting them to come up with a confidence number.
Chintan, Correct me if my understanding is wrong.
Thanks,
Rohit

6. Re: Why the sample size calculator over estimates the sample real needed?
miguelm62125791 Feb 9, 2017 6:54 AM (in response to jrohit)Hi,
For instance, the following data, represents a test with 97% confidence level:
Variation A: 5000 visitors  250 conversions
Variation B: 5000 visitors 300 conversions (20% lift)
but if we try to estimate which is the sample needed to detect a 20% lift, when the baseline conversion is 5%, the experiment consists in 2 variations, setting a 95% confidence level and 80% statistical power. The calculator says that you will need: 8,155 visitors per sample. (When the truth is that with 5000 visitors you will reach more than 95% statistical confidence)

7. Re: Why the sample size calculator over estimates the sample real needed?
chintan601 Feb 9, 2017 10:51 AM (in response to miguelm62125791)Hi, this is because Target is a tool and it can be used however the user desires. More specifically, Target doesnt know the test's sample size and so it continues to show Confidence even when the value is not valid.

8. Re: Why the sample size calculator over estimates the sample real needed?
chintan601 Feb 9, 2017 10:56 AM (in response to chintan601)Dear miguelm62125791, I'm sorry to say your reasoning is not correct. The science of ABtesting is complex, nuanced and highlyspecific; without going into too much detail all I can tell you is that even if the UI shows 99.9% confidence it doesnt mean absolutely anything (as in, it doesnt mean that you have a 0.01% falsepositive rate) if the minimum sample size has not been achieved.

9. Re: Why the sample size calculator over estimates the sample real needed?
Jason Hickey Feb 9, 2017 12:11 PM (in response to miguelm62125791)One thing that might help to visualize this is a confidence simulator at the URL below. It does a great job of explaining the concept of "reaching confidence" vs "being confident". M
http://destack.home.xs4all.nl/projects/significance/#
Consider this scenario with 100 experiments simulated in an A/A fashion and a sample size of 100k and a pvalue of .05.
pvalue
.05sample size
100kexperiments simulated
100were ever significant
41ended as significant
55 experiments ended as significant (which is what we would expect given a 5% false positive rate) but an astounding 41% in this simulation were significant at some point "during the test".
Target doesn't differentiate between "ended" and "ongoing" so that is why you might see something as significant in the tool, but if the minimum threshhold hasn't been met  that isn't the "end"

10. Re: Why the sample size calculator over estimates the sample real needed?
miguelm62125791 Feb 23, 2017 1:27 AM (in response to Jason Hickey)Hi,
Thank you for your answer Jason.
Something that really worries me, is how to explain the team who works with Adobe Target, that they should ignore the confidence level shown, unless the sample size and number of conversions reach the number suggested by the sample size calculator. Would not be it easier that AT shows an alert when the sample is sufficient?
Thank you, best regards.

Snip20170223_1.png 44.0 K


11. Re: Why the sample size calculator over estimates the sample real needed?
Jason Hickey Feb 23, 2017 6:06 AM (in response to miguelm62125791)Great feedback Miguel, and something definitely worth considering. One benefit of the current approach is that it allows you the flexibility to accept your own range of false positives (either via using the calculator or not) via P Value. The more structure in place in the UI around reporting, the less flexibility you would have. But I definitely hear you!

12. Re: Why the sample size calculator over estimates the sample real needed?
miguelm62125791 Mar 22, 2017 8:20 AM (in response to Jason Hickey)Hi,
I am still thinking about this. The following Sample Size Calculator seems to be using the same mathematical model to estimate the size as the one used to calculate the significance.
For instance, these values represent a significant test result:
Unique visitors expected per variation: 7500
Number of expected conversions: 750
Baseline CR: 10%
Expected Uplift: 10%
However the Adobe Sample Size calculator, estimates a needed sample of almost the double (14748 visitors per variation)
What is the mathematical model behind the Sample Size Calculator? Because it seems obvious that it is not the same used to calculate the significance level.

Snip20170322_5.png 101.0 K

Snip20170322_4.png 196.3 K
