Moderator
Member
Activision has just released a 25 page document which goes in to a deep dive on the topic of SBMM in Call of Duty Multiplayer lobbies. We will highlight some of the important highlights this new document shows as well as provide a link to the document at the end of the post.
Activision first describes what skill is according to them: "For the purposes of core multiplayer matchmaking, we generally define skill as how well a player can be expected to perform against the rest of the population in a given game mode, based on their previously observed performance. At a technical level we are interested in a value with the following properties"
Activision then lists the following properties associated with skill at a technical level which are listed below:
1. It should be constrained between two numbers, otherwise it is difficult to reason about the space of all possible skill values, making analysis of the distribution more difficult.
2. It should be highly predictive: if we base your skill on a specific in-match performance metric (such as “kills per unit time”), it should also be a reliable predictor of your future performance as measured by this metric.
3. It should be summable such that the average skill of multiple players is predictive of their combined skill. This allows for very efficient and predictive team balancing. Team balancing is very important for forming games where the outcome is unpredictable. Blowouts result in players leaving the game which adversely affects the player pool. Team balance itself is covered in more detail later in the document.
4. It should be capable of adapting to a player’s ever-changing performance quickly.
5. It should be resilient: the overall skill distribution should remain accurate in all situations. Simple skill algorithms can shift, inflate, deflate and even collapse when exposed to large population changes such as influxes of fresh players.
Why Even Track Skill?
Activision states that one of the core design principles of Call of Duty is "Player First". Team balance is the first and most important reason to track skill according to Activision. They want to focus on knowing how to expect players performances in matches so that they can provide a balanced in-match experience for players. Without this tracking this can result in blowouts, which most players know are not fun for players on the losing end. Activision actually found that balancing skill against other matchmaking factors quantifiably increases the extent to which most players play and enjoy Call of Duty. When skill is utilized in matchmaking, 80-90% of players experience better end-of-match placement, stick with the game longer and quit matches less frequently.
The Deprioritize Skill Test
In early 2024 Call of Duty loosened the constraints on skill based matchmaking to track player retention. The skill factor that is used in matchmaking is lowered in this test. With deprioritized skill, returning player rate was down significantly for 90% of players. The 10% of highest skilled players came back in increased numbers, but in aggregate, we see meaningfully fewer players coming back to the game.
Call of Duty has also We have also ran experiments to tighten skill beyond their current configuration. This had inverse results, negatively impacting the high skill cohort. This change was not rolled out as a standard approach, as they continue to strive for a balance in their approach to matchmaking.
Activision also commented about Ranked Play:
"Ranked Play Skill is not isolated as a factor in matchmaking for Ranked Play chiefly due to game design. Ranked Play is designed to deliver an expressly competitive environment; accordingly, players must qualify for access to Ranked Play modes. Many players who have qualified for Ranked Play still choose to enter the game in non-ranked playlists. For new players and those who do not participate in Ranked Play, it’s important they can contribute meaningfully to their team and their own personal in-game achievements. The next Matchmaking Series white paper will further detail Ranked Play."
Activision first describes what skill is according to them: "For the purposes of core multiplayer matchmaking, we generally define skill as how well a player can be expected to perform against the rest of the population in a given game mode, based on their previously observed performance. At a technical level we are interested in a value with the following properties"
Activision then lists the following properties associated with skill at a technical level which are listed below:
1. It should be constrained between two numbers, otherwise it is difficult to reason about the space of all possible skill values, making analysis of the distribution more difficult.
2. It should be highly predictive: if we base your skill on a specific in-match performance metric (such as “kills per unit time”), it should also be a reliable predictor of your future performance as measured by this metric.
3. It should be summable such that the average skill of multiple players is predictive of their combined skill. This allows for very efficient and predictive team balancing. Team balancing is very important for forming games where the outcome is unpredictable. Blowouts result in players leaving the game which adversely affects the player pool. Team balance itself is covered in more detail later in the document.
4. It should be capable of adapting to a player’s ever-changing performance quickly.
5. It should be resilient: the overall skill distribution should remain accurate in all situations. Simple skill algorithms can shift, inflate, deflate and even collapse when exposed to large population changes such as influxes of fresh players.
Why Even Track Skill?
Activision states that one of the core design principles of Call of Duty is "Player First". Team balance is the first and most important reason to track skill according to Activision. They want to focus on knowing how to expect players performances in matches so that they can provide a balanced in-match experience for players. Without this tracking this can result in blowouts, which most players know are not fun for players on the losing end. Activision actually found that balancing skill against other matchmaking factors quantifiably increases the extent to which most players play and enjoy Call of Duty. When skill is utilized in matchmaking, 80-90% of players experience better end-of-match placement, stick with the game longer and quit matches less frequently.
The Deprioritize Skill Test
In early 2024 Call of Duty loosened the constraints on skill based matchmaking to track player retention. The skill factor that is used in matchmaking is lowered in this test. With deprioritized skill, returning player rate was down significantly for 90% of players. The 10% of highest skilled players came back in increased numbers, but in aggregate, we see meaningfully fewer players coming back to the game.
Call of Duty has also We have also ran experiments to tighten skill beyond their current configuration. This had inverse results, negatively impacting the high skill cohort. This change was not rolled out as a standard approach, as they continue to strive for a balance in their approach to matchmaking.
Activision also commented about Ranked Play:
"Ranked Play Skill is not isolated as a factor in matchmaking for Ranked Play chiefly due to game design. Ranked Play is designed to deliver an expressly competitive environment; accordingly, players must qualify for access to Ranked Play modes. Many players who have qualified for Ranked Play still choose to enter the game in non-ranked playlists. For new players and those who do not participate in Ranked Play, it’s important they can contribute meaningfully to their team and their own personal in-game achievements. The next Matchmaking Series white paper will further detail Ranked Play."
