Report an error

5.4 Measuring performance

The digitization of the customer journey makes it far easier than ever before to measure a marketer’s effectiveness at each step along the way. The online footprint left by potential or existing customers enables brands to track the successes and shortcomings of their marketing outreach. Moreover, the emergence of tools such as Google Analytics and Hubspot software have greatly simplified the process of tracking a person’s reaction to different stimuli along the way, providing marketers and communications professionals with valuable insight into popular keyword searches, blog content, email content, and other forms of engagement.

These are just some of the key metrics marketers consider when evaluating their effort at each stage of the customer journey:

Stage: Awareness

Metric #1: Cost per mille (CPM)

One way for marketers to generate awareness is to place banner ads on websites or platforms, and only pay when the ad appears every 1000 times, regardless of whether someone clicks on it. Note that with the ‘impressions’ count, there is no assumption that each impression is generated by a unique viewer. If the same person were to see an ad five times in one day, across different websites and mobile apps, each of those instances would count towards the impressions total.

This model of digital advertising is called ‘Cost per mille’ because ‘mille’ is the Latin word for ‘thousands.’ Brands whose goal is to generate exposure in a campaign may choose this approach. Additionally, it is possible for brands to control their spending through a variety of methods that vary between websites and platforms:

Daily, monthly or quarterly spending limit
Specific dates (ads that run during the holiday season or major sporting events will cost more)
Advertising channels (will the ad appear on a subsidiary website as well?)

Example:

Lobster Land wants to promote its new range of characters and rides. The park’s marketing manager spends $2000 on online display ads which reportedly receive 20,000 impressions in five days. In other words, the ads were shown 20,000 times during that time frame.

The CPM for this effort is $100 because:

The lower the CPM, the more ‘bang for buck’ the business gets.

Metric #2: Cost per click (CPC) or Pay per click (PPC)

If a marketer decides that a campaign’s main purpose is to get people to visit a company’s website by clicking on a display ad, the marketer will evaluate the success of that effort by calculating the CPC. Basing a strategy around CPC means the marketer is only charged each time someone clicks on the ad. Suppose 100 people see a display ad for Lobster Land, but only 50 click on it. Under a CPC agreement, Lobster Land management would only have to pay for 50 clicks, not 100 views.

Major advertising platform providers such as Google typically offer companies a range of payment options for managing CPC costs. Many marketers opt to allow Google to automatically set bids for ads based on the ad’s likelihood to result in a click or conversion. ³

Alternatively, in order to gain more control over their costs, marketers can engage in ‘manual bidding’, a method that enables marketers to place a limit on the amount that they can be charged per click. However, as internet traffic to a website varies, CPC manual bids will need to be evaluated regularly. Marketers should also be aware that lower bids will result in lower campaign traffic, and possibly a lower conversion rate. ⁴

The term “Pay per click”, or PPC, refers to the general strategy of setting up an ad campaign whose costs will be determined by user clicks, whereas CPC specifically refers to the amount of money that the marketer is charged per user click.

Example:

Lobster Land management has created a new sidekick for Larry the Lobster – Sebastian (also a lobster). Management creates a display ad featuring Sebastian prominently, sets a maximum cost per click of $0.50, and lets the ad run for 7 days. Assuming the Sebastian advertisement attracted 30,000 clicks and Lobster Land management were charged the maximum rate per click, the total advertising cost for this period would be $15,000 because:

Stage: Consideration

Metric #1: Email open rates

Email marketing is a form of direct marketing designed to educate potential customers about a brand, and in turn generate sales leads or website visits. This form of outreach costs a business virtually nothing, causing consumers to be subjected to frequent email bombardments. As a result, marketers are confronted with the challenge of persuading consumers to open the email to begin with. Accurate figures about email open rates are hard to come by given that such information are considered confidential to a brand. However, Mailchimp figures published in 2019 show that average open rates vary between 15% to 28%, depending on the industry.

Nonetheless, for a long time, marketers evaluated the success of a campaign by considering the number of emails opened by recipients. Only ‘unique opens’ were used, meaning that even if a person opened an email more than once, that person’s action would only count as one unique open. Undelivered (bounced) emails were typically excluded from the denominator in this calculation.

Example:

Lobster Land’s marketing manager sends an email to the theme park’s 10,000 newsletter subscribers promoting a range of new stuffed animals. Out of those emails that were sent, 96 emails bounced while 2000 emails were opened. 10 emails were opened by the recipient 5 times. What was the email open rate in this marketing campaign?

Answer: 20.19%

Stronger email privacy measures taken by Apple⁵ have weakened email open rate as a reliable measure of success. That’s because Apple preloads emails on its proxy servers, triggering a tracking pixel for every email it processes. Doing so artificially inflates open rates as a marketer could potentially see 100% email open rate among its Apple users. A/B testing on email subject headers have also become unreliable.

Consequently, marketers consider other performance metrics to evaluate their email marketing efforts. They may consider the percentage of emails that were opened in the last five days compared to the email opt-in rate. They may also consider the percentage of people who clicked a link in an email.

Metric #2: Click through rate (CTR) and click rate

A click through rate (CTR) refers to the proportion of people who click on a piece of content after opening your email or seeing your advertisement. The CTR is calculated like so:

Marketers aim for a high click through rate because it suggests that the ad content is relevant to the audience. It is worth noting that some email marketing platforms provide another metric called a ‘click rate’, which should not be confused with a CTR. In a click rate calculation, the denominator consists of all the people to whom the email was delivered, regardless of whether they opened the message to see the ad.

Example:

Lobster Land’s marketing manager notices the following figures in the park’s monthly email newsletter for June:

No. of successful email deliveries: 1000

Total no. of emails opened: 224

Total no. of links clicked: 15

What is the click through rate for June’s email newsletter?

Answer: 6%

What is the click rate for the month’s email newsletter?

Answer: 1.5%

Improving CTRs is very much a trial-and-error process. A/B tests are constantly performed on headlines, calls to action, and ads to increase these rates. At the same time, marketers experiment with elements such as copy length and visual imagery to determine the ‘secret recipe’ needed to pique the reader’s interest. In short, there is no one-size-fits-all approach because audiences respond differently depending on the industry, product, market, and type of advertisement. Hubspot’s figures show the average CTR for search ads varies from 1.35% to 2.65%, while the average CTR for display ads ranges from 0.14% to 0.84%.⁶

Metric #3: Email bounce rate

Bounced emails are emails that are blocked or returned to the sender for various reasons. There are two types of ‘bounces’ for email marketing: soft bounce and hard bounce.

A soft bounce occurs for a several reasons, including the following:

when a recipient’s email inbox is full
if someone is out of office
if the recipient email server is down.

A hard bounce happens when your email just cannot get through due to factors such as:

a spelling mistake in the email address
the recipient email server has blocked delivery
the person no longer works at the company
the email is too large

Example:

Lobster Land’s marketing manager notices the following figures in the park’s monthly email newsletter for June:

No. of hard bounces: 70

No. of soft bounces: 30

Total no. of emails successfully delivered: 1000

Regardless of their nature, bounce rates are ‘red flags’ that should be closely monitored because a brand’s email account could be suspended in accordance with rules placed by internet service providers (ISP)⁷. The ISPs do not publicize figures for ‘acceptable’ bounce rates due to varying incoming email volumes, and to avoid tipping off spammers.

Metric #4: Web page bounce rate

The bounce rate of a webpage refers to the proportion of people who landed on a webpage and then left without visiting another page on the company’s site. A high bounce rate is cause for concern if your website depends on people viewing more than a single page. For instance, if a high percentage of people leave after visiting the main page of your online clothing store, that is troublesome, as it indicates that few visitors are making purchases through your website. On the other hand, if the bounce rate for a piece of blog content is high, this may not be problematic – single page sessions are to be expected, as people tend to read a single article before moving on.⁸

Example:

Lobster Land has just started selling merchandise through its website lobsterland.net. Upon reviewing the website’s traffic, the park’s marketing manager realizes that the bounce rate for visitors coming from the search channel is 80%. Suppose a total of 1000 people visited LobsterLand.com, how many visitors only viewed one page on the site before leaving?

Answer: 800

There could be several underlying reasons for a high bounce rate such as the one described in the example above. Perhaps, for example, users arrived there due to search keywords that are irrelevant to Lobster Land. Alternatively, something on the landing page could be off-putting, or visually unappealing. Either way, it is worth digging deeper to enhance the user experience.

Stage: Purchase

Metric #1: Sales conversion rate

By the time online shoppers reach the ‘purchase’ stage of the funnel, they are ready to “seal the deal,” so to speak. A brand can help these potential customers convert by ‘nudging’ them with the following:

a simple and smooth checkout process
a discount code
promotional offer
customer testimonials
product or service explainer videos
product imagery reflecting high production value
customer service chat option

Making the checkout process as smooth as possible increases the sales conversion rate i.e. the proportion of transactions that occur from the sales leads generated.

Example:

LobsterLand.net attracts 10000 visitors in July, and records 7363 transactions that month. What is its sales conversion rate?

Answer:

Marketers will typically compare conversion rates between different consumer segments e.g. new and existing customers, customers acquired in Q4 and the rest of the year etc while considering these rates together with the average customer lifetime value (CLV) or average spend per order.

Stage: Retention and advocacy

Metric #1: Renewal rates

Customer renewal rates are an important indicator of a company’s future prospects because they measure a business’s ability to generate long term value with a customer. This metric, which is crucial in forecasting future growth, can be measured using one of two methods below:

a) Customer count

b) Revenue

Marketers should plan ahead to engage customers so that healthy renewal rates are maintained.

Example:

2000 out of 3500 Lobster Land passholders from last season renewed their season pass this year. What is the renewal rate by customer count?

Answer: 57.14%

To generate this metric using renewal rate by renewal value, the count values in the numerator and denominator (as shown immediately above) would be multiplied by the dollar value of a Lobster Land season pass. If only a single type of pass is offered, at a constant price, that would yield the same 57.14% outcome. However, if multiple season ticket pricing tiers were offered, a different result would be generated (based on the total value of the renewals, divided by the total potential value of those renewals, if all passholders had chosen the priciest option).

Metric #2: Customer churn

There will always be some customers who do not return (i.e. churn), so no business should expect a perfect retention rate. Nonetheless, a business should pay attention to this metric. Monitoring churn rates across time can help to gauge consumers’ sentiment towards the business. Furthermore, it costs more for a business to attract new customers than to retain existing ones. This is especially true if customer acquisition costs are high, as is the case for “B2B” firms that primarily sell to other businesses.

Customer churn rates are calculated like so:

Example:

1500 out of 3500 Lobster Land passholders from last season did not renew their season pass this year. What is the churn rate?

Answer: 42.85%

Metric #3: Customer lifetime value (CLV)

Customer Lifetime Value (CLV), sometimes abbreviated as CLTV, is one of the most important concepts in marketing. It typically represents the total value that a customer is expected to bring to a company across all the years of the relationship.¹⁰ Business owners who view each new customer in terms of CLV, rather than in terms of the one-time revenue or a profit associated with a single transaction, take a longer-term perspective that often has an ultimately higher payoff value.

CLV is important to bear in mind when assessing marketing campaigns. As companies assess their campaigns, they will often analyze their Cost of Customer Acquisition (CoCA). CoCA is hard to calculate precisely, but as a general rule, we can think of it as the entire cost of the marketing campaign, divided by the total growth in new customers as a result of the campaign. Typically, prospective investors will look to invest in companies for whom the CLV associated with a new customer exceeds the CoCA.

The concept of CLV can also be important when thinking about the way companies make decisions regarding specific customers. When a customer service situation arises, how much energy and effort should the company be willing to exert in order to retain a particular customer’s business? A CLV estimation can help to make that decision.

While there are many ways in which companies could calculate CLV, this is the formula that we will use:

Example:

The annual profit margin for Lobster Land per season passholder is $915. Last year, 64% of Lobster Land’s customers purchased a season pass for the next year. Lobster Land’s discount rate is 6.075%, which means the park has to pay an interest rate of 6.075% if it takes a loan from the bank. (In other words, a discount rate is the company’s borrowing cost). What is the CLV of each Lobster Land customer?

Answer: $1391.80

A CLV can be automatically calculated in Python using a user-defined function like the one shown above. In this example, our function is called clv_calculator. There are three arguments for this function: profit margin, retention rate, discount rate. We then enter the CLV formula after the ‘return’ statement.

When we call the function clv_calculator, the values that we enter inside the parentheses must follow the same order as the arguments in the function that we defined in the previous code cell: profit margin first, retention rate second, and discount rate third.

A multiple linear regression model could also be used to predict a customer’s spend for some particular time period.

Consider the following:

Example:

Lobster Land’s Gold Zone Players’ Club has been around since 1986. The Players’ Club is open to any person, 12 or older. For $250 up front, paid at the start of the season, any member of the Players’ Club receives discounted game tokens throughout the summer season. Build a multiple linear regression model to predict how much a member will spend this season.

Variable	Explanation
gzpc_id	Membership ID
age	A member’s age, measured in years
gender	A member’s gender
consec_years	Number of consecutive years of membership
mar_stat	Marital_status (single or married)
emp_stat	Employment status (salaried, hourly, or unemployed)
day_reg	This is a numeric variable that indicates when the member signed up, relative to the season start date. A member who signed up on Memorial Day would have a ‘0’ in this column. A value of ’10’ here would indicate that the person signed up ten days after the season start date. A value of ‘-10’ here would indicate that the person signed up 10 days before the start of the season.
dist_park	distance from the member’s home to Lobster Land, in miles
spend	total spending for the season, in USD (adjusted to present-day dollars)

Step #1

Explore the dataset. Lookout for outliers because these will skew your model. Be wary of missing values as well – how many are there relative to the entire dataset? If there are many ‘holes’ in your dataset, how will you treat these missing values?

The summary statistics shown below indicate 2 things:

There may be missing values in the ‘spend’ variable.

‘Dist_park’ has a maximum value of 127. Since the mean is 15.68, this value is an outlier. This may not necessarily be problematic for us, but we should be mindful of this as we move through the modeling process.

Step #2

This confirms the presence of missing values in our dataset. There are 7 in the ‘spend’ variable.

Step #3

We will remove ‘gzpc_id’ at this stage because these values have no meaningful input to our model. This is simply a unique identifier, akin to a student ID or a telephone number.¹¹

At the same time, we remove the rows that contain missing values for the ‘spend’ variable. Since ‘spend’ is our outcome variable, we will not consider an imputation here – after all, we do not want to violate the Second Commandment of Imputation from Chapter 1: Thou Shalt Not Impute the Response Variable.

Thankfully, these spend NaNs only account for 0.002% of the dataset. Their removal poses no threat to model validity or statistical significance.

Step #4

It is important to ensure the numerical independent variables are not too strongly related to each other. While collinear input variables do not threaten the overall predictive ability of the model, they can make it difficult to accurately gauge individual variables’ impact on the outcome variable. This is akin to putting both table salt and sea salt into a pot of soup. If the soup is too salty, you would not know whether the table salt or sea salt was the true culprit.

At this point you may wonder: at what point is the correlation value too high? The short answer is, it depends. There is no consensus on a benchmark, as some people believe that anything above 0.7 is too high while others say the rule of thumb is 0.8.¹²

The results below show that none of these variable relationships poses a collinearity risk to our model.

Step #5

Next we begin building a linear regression model by importing Python’s statsmodels package into our environment. Then we convert our categorical data into dummy variables, i.e. 1s and 0s. Why is this necessary? Because linear regression models can only handle data that is presented in a numerical format.

In this example, we need to dummify three variables for our model: ‘gender’, ‘mar_stat’, ‘emp_stat’. At the same time, we enter a command ‘drop_first=True’ to prevent the creation of an unnecessary binary dummy column.

For instance, the ‘gender’ variable in our dataset only contains 2 outcomes: male or female. After dummifying the ‘gender’ variable, we will have one column such as ‘gender_male’. If the people listed in this column have ‘1’ next to their name, we know those members are male; if there is a ‘0’, that member is a female. There is no need for Python to create a standalone column called ‘gender_female.’¹³ Likewise, for a categorical variable with three possible outcomes, we can express all the possibilities with just two binary dummy pairs. Employment status takes three levels: hourly, salaried, and unemployed. If we only use ‘hourly’ and ‘salaried’ for dummy columns, we can know that if each of those takes a ‘0’, the person’s status is ‘unemployed.’ Whenever a categorical variable contains n distinct levels, we can express all the possible outcomes with n-1 binary dummy pairs.

Step #6

We build the model by specifying the independent and dependent variables. The independent variables (the variables that we think have an impact on customer spend) are included below in the ‘X’ group. The dependent variable (the outcome that we wish to predict), is denoted below by ‘y’.

Then we create a y-intercept using sm.add_constant(X). This is the value of ‘y’ when x=0. ¹⁴

Lastly, we fit the model and generate a model summary.

Step #7

A quick scan of the model summary tells us:

The strength of the model (R squared) is 0.785. This means that our independent variables can explain 78.5% of the variability in customer spending. The adjusted r-squared (0.784) is calculated similarly, but includes a slight penalty for the number of predictors in the model.¹⁵

The coefficients for each of the independent variables.
- Listed here in the ‘coef’ column, the coefficients tell us how much a one-unit change in that variable would be expected to impact the person’s Gold Zone spending for the season. The variable ‘age’, for instance, has a coefficient of 44.57. Since ‘age’ here is measured in years, when all else is held constant, each additional year of age is associated with $44.57 of additional expected spending.
- For each of our categorical variables, the first level (the one that’s not seen here in the model output) has become the reference level, and takes a value of 0.¹⁶ We can view the coefficients for the other variables in that category relative to the reference level’s neutral baseline. Looking at marital status, for instance, this output shows us that we would expect a single person to spend $288.58 more than a married person, all else held constant. From the employment status coefficients, we can see that salaried workers are likely to spend much more than hourly workers. Unemployed members of the Gold Zone, however, spend less than those with hourly-wage jobs.

The t-statistic associated with each of the independent variables. The t-statistic tells us how confident we should be of a variable’s significance – the higher the absolute value, the better.

The statistical significance of the independent variables, as denoted by the p-values. The p-values, which can be seen in the P >|t| column, tell us “What is the probability that we could obtain a t-statistic more extreme than this one, through random chance?” The p-value of almost every independent variable is less than 0.05, indicating that they are important factors that influence customer spending. However, ‘dist_park’ has a p-value of 0.484. Its 95% confidence interval straddles zero (-0.951 and 2.009). This means that we cannot even be sure whether a 1 mile increase in a person’s distance from the park is associated with an increase, decrease, or no change in spending. Therefore, we should drop ‘dist_park’ from the next model iteration.

Step #8

Dropping the ‘dist_park’ variable results in no change in the model’s strength. After this variable’s removal, adjusted r-squared remains unchanged at 0.784.

So how do we calculate the estimated customer spend based on the model summary shown above? Recall the multiple linear regression formula:

If a member has the following profile:

30 year old single male with a full time salaried job
a Gold Zone Players Club membership for 5 consecutive years
signed up 10 days after the season started

This person will likely spend this amount at Lobster Land:

Alternatively, we can generate this result with the predict() function from statsmodels, as shown below. Note that in the dataframe used for ‘member’, we include a value of 1 for the constant term, and we include separate values for each of the variables in the model. Had this member been female, or married, we would have simply entered a ‘0’ here for the ‘gender_Male’ and ‘mar_stat_Single‘ entries, respectively.

³ Google Ads (n.d.). ‘About automated bidding’. Google Ads. https://support.google.com/google-ads/answer/2979071

⁴ Google Ads (n.d.) ‘Choose your bid and budget’. Google Ads. https://support.google.com/google-ads/answer/2375454

⁵Kaczanowski, R. (2021, November 9). ‘How Apple’s mail privacy changes affect email open tracking’. Postmark. https://postmarkapp.com/blog/how-apples-mail-privacy-changes-affect-email-open-tracking

⁶Mailchimp (n.d.). ‘Email marketing benchmarks and statistics by industry’. Mailchimp. https://mailchimp.com/resources/email-marketing-benchmarks/

⁷Volovich, K. (n.d.) ‘What’s a good clickthrough rate? New benchmark data for Google AdWords’. Hubspot. https://blog.hubspot.com/agency/google-adwords-benchmark-data

⁸ Mailchimp (n.d.). ‘About bounces’. Mailchimp. https://mailchimp.com/help/about-bounces/

^⁹ Google Analytics (n.d.). ‘Bounce rate’. Google Analytics. https://support.google.com/analytics/answer/1009409?hl=en

^¹⁰ Alexander, L. (n.d.) ‘E-commerce conversion rates across industries’. Hubspot. https://blog.hubspot.com/marketing/ecommerce-conversion-rates-across-industries

¹¹ The term “CLV” can also be used to refer to a customer’s expected value to the business across some particular time period. Even though the word ‘lifetime’ forms part of the CLV acronym, it is not uncommon to hear marketers speak of a customer’s CLV across the next quarter, 6 months, year, etc. In the Gold Zone Players’ Club example, we’ll look at how Lobster Land might predict an individual’s Gold Zone spending across a single season of park operations.

¹² While gzpc_id will not be useful for our predictive model, we may wish to use it afterwards. This unique identifier can function like a primary key in a database table, should we later wish to connect one of the observations in this dataset to a member’s actual identity.

¹³ Variance Inflation Factor, a metric for assessing the multicollinearity risk among variables in an already-built linear model, will be covered in Chapter 16.

¹⁴ Furthermore, if we retained both of the newly-dummified columns for gender, for which the ‘0’ and ‘1’ values of each are mutually exclusive, we would have a linear dependency within our data, which could pose a problem during model fitting.

¹⁵ When building linear models, we will almost always want to include a constant term for the y-intercept. If we do not include one, the y-intercept will be forced through the origin (0.0).

¹⁶ Adjusted r-squared is often, therefore, a better benchmark comparison between different model iterations built from the same dataset. The inclusion of the penalty term based on the number of inputs discourages modelers from including non-significant input terms, which can either increase r-squared or leave it unchanged.