6. Experiment Design, A/B Testing, and Statistical Distributions

Research methods can either be quantitative, qualitative, or a combination of both. Before data can be collected and analyzed, an analyst needs to plan the entire process methodically and thoroughly by considering questions such as the following:

• What is your hypothesis? – In other words, what are you trying to prove?
• What type of data will you need to collect?  -Will you be collecting your own data (primary data) or will you be relying on existing sources?
• How will you collect your data? – Will participants need to be recruited for an experiment? Which part of your website or email should you change to drive up subscription rates?
• How many people do you need to recruit for your experiment? – Determining the right sample size is very important. Too big, and you would have wasted time recruiting more people than you should have.  Too small, and your experiment will lack statistical power.
• How will you analyze your data? – What statistical tests will you utilize to verify if your results happened by chance?

In this chapter, we will clarify important concepts related to experiment design, explain some key statistical tests, and highlight potential pitfalls.