Overview
Whether your team has only started replying to reviews or already has a well-established strategy of processing user feedback, it's important to see what effect this work has on your app's rating.
Users may update their review's rating for better or for worse after communicating with your team. However, app stores don't provide features for tracking these updates, so it's difficult to estimate your impact on the user's opinion.
AppFollow's Reply Effect algorithm helps you evaluate the real effect of your replies. It counts only those review updates that happened as a result of an agent's reply. The system takes into account:
- The chain of agent replies and review updates
- The period of time between the agent's reply and the user's review update
In this article, you'll learn about the Reply Effect algorithm, how it's calculated, and how you can leverage this data across the AppFollow platform.
Reply Effect
The Reply Effect is a vital KPI to create and adjust your reply-to-reviews strategy. The Reply Effect algorithm measures the effect the developer's reply had on the rating of the app.
The algorithm is based on 2 major factors: whether the review was updated after the developer's reply and how much time passed between the reply and the update. Below, you'll find some examples of these factors in action.
Factor 1: The review update took place after an agent's reply.
Example of a review update WITHOUT the Reply Effect:
In this example, the app user did not interact with the app developer. The review was updated independently of any interactions.
Example of a review update WITH the Reply Effect:
In this example, the app user updated their review after an agent replied to the previous version of their review.
A user may update their review's rating or they may update the text of the review without changing the rating.
The review will be considered as a review with the Reply Effect even when only the text was updated.
Factor 2: The review update took place within a specific timeframe – no later than 3 months after the agent's reply.
To optimize your reply strategy, it is important for you to be sure that the review was updated because of an agent's reply and not because of other factors. The Reply Effect has a time threshold that defines the range when an update is considered to have been impacted by a developer's response. If more than 3 months pass between the developer reply and the update, such updates are not considered to have a Reply Effect.
If you need to customize this timeframe, please reach out to our Support Team or directly to your account manager.
Below is an example of a review update WITHOUT the Reply Effect, where the user updated the review more than 3 months after the developer's reply:
This example shows a review update WITH the Reply Effect, since the user updated the review rating the next day after the developer responded:
Where Can I Track the Reply Effect?
Reviews Analysis
The Reviews Analysis page provides the bottom-line stats on the effect your replies had on your reviews. In this view, you'll see the overall effect of your strategy and learn how it impacted the latest versions of the reviews you responded to. Use this section to understand the end result of your customer communications.
This section shows the effect of review updates on the total rating of the app. This takes into account the latest update of the review rating within the selected timeframe. If you responded to a review several times, and the review was updated several times, this view will only show the effect of the latest reply on the latest update. If you want to see the Reply Effect for multiple chains in a conversation, check out the Agent Performance view.
For example, if a user changes the rating of their review from 4 to 3 stars, and then back to 4 stars within the same selected timeframe, we will only count the final review rating update.
The Replies Effect table provides the following data:
1. Total updated reviews - all reviews where the latest update was made within the selected time range.
The list of all updated reviews is available on the Reply to Reviews page by applying the filter by reply effect ("With reply effect" + "Without reply effect").
2. No Reply Effect - all updated reviews that don’t have a developer reply or the review was updated more than 3 months after a developer reply.
A list of all updated reviews without the Reply Effect is available on the Reply to Reviews page by applying the filter by update ("No Reply Effect").
3. Reply Effect - all updated reviews that have a developer reply within 3 months before the update. Such review updates are counted as a result of the developer’s reply. The list of all updated reviews with Reply Effect is available on the Reply to Reviews page by applying the filter by update ("Reply Effect").
The table columns cover:
- Average rating change - the change of the average ratings of the reviews with the changes to the previous period
- Updates to ratings - the share of reviews where the ratings changed to more or fewer stars, or did not have any changes in ratings but had changes to the review text
- Matches - number of reviews with the latest update made within the selected time range
- Share of all reviews - the share of the matches in the total number of reviews received within the selected time range.
Agent Performance
The Agent Performance page provides insights into how your team is performing. In this section, the Reply Effect is counted for all agent-user interactions during the specified timeframe. This allows you to treat each interaction as a separate conversation and understand how an agent is performing in the context of an entire conversation.
For example, if an agent responded to a review 5 times, and a user updated this review after every reply, this review will be counted as 5 separate interactions and we will calculate the Reply Effect for each of them. This shows the actual effort an agent put into the review.
Unlike the Reviews Analysis and Ratings & Reviews Dashboard pages, the Agent Performance section takes multiple chains of reviews and replies into account to calculate the Reply Effect. If you need to evaluate the Reply Effect based only on the final result of the conversation, you can use the Reviews Analysis feature.
Ratings & Reviews Dashboard
The Reply Stats section of the Ratings & Reviews Dashboard helps you get a quick overview of how the Average Ratings changed for reviews with the Reply Effect and without it. You can quickly gauge the effect your responses have on your app's rating. In this view, you’ll find the Average Rating Change for the selected app with and without replies.
On the Ratings & Reviews Dashboard, the Reply Effect is calculated for the latest chain of an agent-user thread. If you responded to a review multiple times, and the review was updated multiple times as a result of those replies, we will show the effect based on the latest review update.
Reply to Reviews
Apply the "Update" and "Rating change" filters to analyze how exactly the review rating changed. Click on "Show history" to see the details on review changes over time.
Replies Chart
The Replies Chart shows how many replies were submitted and how the replies affected the review rating changes.
On this chart, you'll see the total number of replies, replies time, and the Report a Concern stats for the selected timeframe.
On the Replies tab, use the filter by review changes to view how many replies affected the review rating change:
- All reviews - total number of reviews for the period
- Became better - review rating increased after the developer reply
- Became worse - review rating decreased after the developer reply
- Without changes - reviews without rating updates (only review text was updated).
On this page, we calculate the dynamics of every single reply and take the reply history into consideration. We calculate the Reply Effect for every update of the review (there may be several updates per review). This is why Avg. Rating Change on Replies Chart page can differ from the data on the Reviews Analysis page. It specifies how efficient the efforts of your Team are as opposed to eventual app rating dynamics displayed on Reviews Analysis.
Apply the filter by users to view the impact of each team member.
Under the chart, there is the total number of replies sent to the store, reviews received and the average ratings change within the time range.
The Reply Effect will help you understand how the replies are affecting your app rating and get insights on the optimization of your team efforts and strategy.