Data Source Methodology

Inspiration for D2Trends

On February 26th, 2024, an article from The Game Post reported an analysis claiming that The Final Shape (TFS) preorders were "less than a quarter of Lightfall's total figures." This news came only a few months after reports that Bungie had missed revenue targets by 45% and laid off over 100 workers, in part to help avoid a potential takeover from Sony. The resulting delay of TFS from February to June 2024 meant the community was in for another long content drought, right after a historically great year for video games (and a historically awful year for video game developers and layoffs). Player sentiment may have been at its lowest ever, with the appeal of other games to jump to at its highest.

The Destiny community reaction to this analysis was divided, with plenty of concern coming from the content creator community, but also plenty of reasonable doubt from players about the article's numbers and methods. The article itself didn't do enough to explain how it arrived at its numbers, only that it used Charlemagne (a third-party community app) to do so, and provided some details on how Charlemagne works.

However, there were several issues apparent with the analysis presented:

The numbers didn't appear directly on any page from Charlemagne's item analytics, so they couldn't be readily verified.
The numbers for Beyond Light in particular were so much larger than those of Witch Queen and Lightfall that it defied belief, and called into question the entire analysis.
Other community apps, notably bray.tech and emblem.report, appeared to contradict the article's numbers.
The comparison made between TFS and past expansions wasn't a fair or honest one, since TFS preorders will continue to grow up to release, whereas past expansions had already earned all possible preorders.

With all that said, the article did pose an interesting question about how the community could measure player engagement and game financial success by monitoring collection unlocks across accounts through the Destiny 2 API. In my review, Charlemagne's methodology for measuring player accounts appeared to be the best among all potential sources. The key pieces missing were the ability to look at those measures over time, and the appropriate outside context for unique events. If we could compare TFS preorder emblems to those of Witch Queen at comparable points in time (e.g. three months before the expansion arrives), that would be a far more apt comparison for the community to discuss. The problem was none of the data sources provided reliable trending data across all game items for all accounts.

That's where I decided to try to fill the gap. D2Trends aims to capture Charlemagne's item metrics on a daily basis, and present that data in-context to the community for a more informed discussion. It's not possible to perfectly compare to past events without measurement from that time, but setting up this site today may help provide more accurate measurement going forward.

Analysis: Destiny 2 The Final Shape Pre-Orders Appear to be Less Than a Quarter of Lightfall's Total FiguresAnalysis shows Destiny 2 The Final Shape has the lowest number of pre-orders when compared to the pre-orders for the last three expansions.

Data Source Comparison

This project started with a comparison of four potential data sources to determine which was best to use: Charlemagne, bray.tech, light.gg, and emblem.report. Each had their own strengths and weaknesses:

Charlemagne

Strengths:

Scrapes data globally across all accounts ever made (~66M), save for about 0.15% of accounts that have disabled "show my progression" on Bungie's privacy page.
Claims "Global analytics are provided within an hour of real-time."
Has data on nearly all item and collectible types, including weapons and armor, cosmetic items, and even triumphs and titles.
Absolute numbers of accounts that have unlocked each item are provided until that item has been unlocked by about 80,000 accounts.

Weaknesses:

After an item is unlocked by 80,000 accounts, the absolute number is removed from display and only the relative "Global Rarity" percentage remains.
No trending of data.
A few item categories don't appear to be actively measured up-to-date, e.g. emotes.

bray.tech

Strengths:

Has data on all item and collectible types, without exception.
Absolute numbers of accounts that have unlocked each item are provided for all items.
Data is trended in charts when inspecting each item.

Weaknesses:

Their server-based data collector "Voluspa" only collects data from about 1M players, far from global coverage.
No details are provided on how Voluspa decides which player accounts to measure.

light.gg

Strengths:

Has data on all item and collectible types, without exception.

Weaknesses:

No absolute numbers provided for item unlocks.
Total accounts measured is limited to any accounts part of the light.gg community, does not reflect global accounts.

No trending of data.

emblem.report

Strengths:

Absolute numbers of accounts that have unlocked each emblem are provided.

Weaknesses:

Only data on emblems is collected, no other items.
No trending of data.
They only collect data from about 4M players, excluding private profiles - far from global coverage.
No details are provided on how they decide which player accounts to measure.

Charlemagne and D2Trends Methodology

In reviewing all four sites, Charlemagne appeared to be the best source to use here, as its weaknesses could be overcome with the data it provides.

Metrics

Each item is reported with up to three data points: Total Redeemed; Global Rarity; and Adjusted Rarity.

Total Redeemed is sometimes listed as Total Earned depending on which item you're looking at, such as titles. This presents an absolute count of all accounts that have unlocked that item (excluding private profiles). However, it's only present if the total number of account unlocks is less than 80k.
Global Rarity gives the percentage of all accounts that have unlocked that item (excluding private profiles), down to a thousandth of a percent.
Adjusted Rarity is a percentage that changes its meaning depending on the type of item you're looking at.
- For emblems, it's a baseline of "guardians with over 10 hours of playtime."
- For triumphs, it's a baseline of "guardians with Triumph Scores greater than 0."
- For titles, it's a baseline of "guardians with at least 1 title earned."

Given that our goal is to measure total accounts that may have preordered or engaged with content, the Adjusted Rarity metric doesn't provide useful data since it changes baselines, so we can safely ignore that.

The Total Redeemed and Global Rarity metrics, however, are the bedrock of this analysis. Total Redeemed can report absolute numbers of unlocks, whereas Global Rarity can give you an estimate of Total Redeemed so long as you know the size of the global account population.

Estimating Global Population

To estimate the global account population, we can use Charlemagne's data on title analytics. Unlike other items, Charlemagne provides absolute counts of all titles earned, even above 80k. We can take those counts and divide them by their respective Global Rarity to get an Estimated Global Population for each title. Since titles earned by fewer accounts would net a wider margin of error when estimating population this way, we exclude any titles earned by less than 100k accounts to get a more precise estimate. We then average the estimates together to form the Estimated Global Population for each day's measurement.

However, given the averaging we're doing, it is technically possible, though unlikely, for a day's Estimated Global Population to decrease slightly day-to-day, even when logically there can only be a growth of accounts over time. This is possible because the granularity of the Global Rarity metric provides for a margin of error of a few hundred accounts, meaning small increases in absolute numbers with a minimum incremental change in Global Rarity nets a smaller Estimated Global Population. This is a necessary quirk of leveraging the Global Rarity metric on Charlemagne - even if small trends on a day-to-day basis may go down, the broader trends over longer periods of time should always be more accurate.

Applying Global Population to Item Global Rarity

Once you have that Estimated Global Population, you can apply it to every item reported on Charlemagne, and get an estimated number of accounts with that item unlocked, within five ten-thousandths of a percent (0.0005%) of the total population. For example, with a population of 66M players, these estimates should be within +/- 330 of the actual count. That precision is plenty to get a general idea of player engagement and game performance.

Example item with Total Redeemed visible

Example title excluded from global population estimates (<100k)

Example item with no Total Redeemed visible

Example title included in global population estimates (>100k)

Additional Context for Data Interpretation

This data should still be taken with a grain of salt, as these are rough estimates from imperfect sources, and some assumptions are always made when pulling those numbers together.

First, there is specific context that explains the numbers for Beyond Light in The Game Post's article. In the run up to Beyond Light, Destiny 2 made the whole game, including all of its previous expansions and Beyond Light, available on Game Pass for Xbox and PC users. Players who logged in without ever preordering Beyond Light received the preorder emblem for free, unlocking it on their accounts. Since Charlemagne merely checks if the emblem is unlocked, all of these free accounts would have counted as "preorders" according to the article. Destiny was then removed from Game Pass a year later, before the Witch Queen expansion launched, meaning those accounts would not receive the preorder emblem.

However, this doesn't invalidate using Charlemagne to measure current TFS preorders in absolute terms, it only proves that comparing past data when you don't have the necessary historical measurement leads to inaccurate conclusions.

Second, we know that Charlemagne only counts accounts that have unlocked the item in their collections. But players often need to claim an item before it's unlocked, such as when an emblem or ghost shell is claimed from a triumph, or when a player logs in for the first time after preordering TFS and claims the emblem from Rahool.

Third, we know that Charlemagne only counts accounts that are not set to private mode in Bungie's security settings. It's reasonable to assume that the players most likely to search out security settings on Bungie's website and change them are players who are likely more invested in the game than those who just bounce off and don't think about it or play it again.

For both of these reasons, it's fair to assume that Charlemagne's numbers will be a good rough estimate, but are likely slightly less than the true number of accounts that have unlocked any given item.

Page updated

Google Sites

Report abuse