Lifecycle Of A FIFA Issue - Part 3

Dec 7, 2020

This series details the phases that a FIFA 21 Issue normally goes through before it can be fixed.

Hey FIFA Fans.

My name is Joel Doonan, and I am the lead Producer for the Player First Operations team, part of the FIFA Live team.

This is the third in a series of Pitch Notes articles focused on a deep dive into the lifecycle of the average issue in FIFA 21.

In this series, I cover the various steps that an issue generally needs to go through in order to be addressed and released in the live version of FIFA 21. 

This 4 part series covers:

• Part 1 - Identification
• Part 2 - Investigation
• Part 3 - Resolution (This article)
• Part 4 - Release

For this article, we are going to look at the Resolution phase of the lifecycle, which, depending on the issue, could have a number of different steps that need to take place before a fix or change could be ready to be released.

One note before we start, I will be using some technical terms at times, but when I am using a term that may not be clear to those that are not familiar with game development, there will be a footnote that explains what that term means. So let’s dive right in and look into how issues get resolved.

Making The Change

If we have reached the Resolution phase of the issue’s lifecycle, that means that we believe we understand the root cause¹ of the issue and that we are ready to start making a change in an effort to resolve it.

Depending on the root cause and nature of the issue, the complexity of making this change can vary drastically.

Making a change to resolve a basic functional issue², where something is clearly and obviously not working as expected, can sometimes be reasonably simple once the actual root cause of the issue has been identified. Well, at least it can be for those with years of coding experience of course.

Identifying the proper change to resolve a more complex issue can be very challenging, sometimes requiring modification or additions to core technologies that the entire game may run on, or potentially changing multiple systems that are owned by different teams.

If the issue, and the intended design of its resolution, is not a clear and obvious one, such as something that requires tuning or balancing changes, this can also be very complex, especially when you need to ensure that whatever change you make doesn’t create new issues in other places.

Given the above, the range of potential capacity needed to make a change is large. It could scale anywhere from a few hours of a single Software Engineer’s (SE)³ time for something relatively simple, up to requiring a strike team⁴ of senior SEs working for weeks or months to identify a safe change for some of the most complex issues.

Regardless of the complexity of the change they are making, during this time that the SEs are working on making the change, they are generally doing so in a local build⁵, a unique version of the game that contains the game plus whatever work is being done for this change. This is to ensure that nothing actually makes it into the mainline build⁶ of the client⁷ or server⁸ code until it has been approved for checking in, which we will cover shortly.

After the change has been made on the local client or server build, the SEs will work with embedded Quality Verification Analysts⁹ to run testing against the change, to both confirm if it resolves the issue that was being seen, and to make sure that it hasn’t caused any obvious new bugs, which we call knock-ons¹⁰.

Depending on the nature of the change, there could be multiple additional steps here before it is finalized and continues to the next steps of the Resolution phase.

As an example of one of those potential additional steps, if the change could result in new server calls¹¹ being made, or in alterations to existing server calls, you may need to run the change through load testing¹², where we have a unique server environment setup to simulate what happens on the servers when players are playing the game. These tests would be run to ensure that the change isn’t going to result in stability issues due to potentially increased load on the servers or cause any issues with the way that the servers process all the various requests from players.

Another example is a step taken by the gameplay team, who have a unique set of tests they use called ‘Gauntlet’, which they run their changes through to ensure that they are achieving the intended result and not causing issues in other gameplay systems.

All these additional steps are intended to gain as much confidence as possible that the change we would be making corrects the issue we are trying to resolve, that it doesn’t introduce any knock-ons, and to prepare for a key step in the lifecycle of an issue, actually checking the change into the game in preparation for eventual release to our live environment¹³.

Code Reviews/Checking It In

Once the SE believes they have a safe change ready, and they have gone through as many of the confidence and verification steps deemed necessary, they are ready for a key step in preparing this change for eventual release, which is checking¹⁴ their change into the mainline build.

Before they are able to do this, a review of the change will take place from a technical lead to ensure that they are aligned with the change being made and the results of the testing. Sometimes this review would have happened much earlier in the process, but it would need to happen before the check-in could take place.

Once the review is completed, the change can now be checked into the mainline build, whether that be the mainline build of the client code, or the mainlines for the various server systems. Once it is checked into the mainline build it is now ‘part of the game’ internally, and it is ready for the final phase in the lifecycle of an issue, the Release phase, which we will cover in the next and final article in this series.

Thanks for taking time to read this article, and I hope that this series will provide some greater insight into what goes into fixing issues that you may have seen while playing a video game.

Joel Doonan and the FIFA Live Team.

Footnotes

¹: The root cause of an issue is the core reason why the issue is actually happening, something identified during the Investigation phase, as detailed in the previous article.

²: When we talk about a functional issue, we are generally talking about an issue that is impacting the ability for something to function in the clear and obvious way it was intended to. For example, if you had an option to choose the color of an arrow, and you selected blue, but the arrow ended up red, that would be an example of the color selector clearly not working as intended. Things like balance/tuning issues or qualitative feedback are not examples of functional issues as these are often matters of opinion or situations where the issue is related to functionality that doesn’t have a clear and obvious way of working at all times. For example, a situation where that previously mentioned color selector had a blue option, but the reported issue is that the blue was not ‘blue’ enough. In this case, how blue, blue should be is not clear and obvious.

³: A software engineer, or SE, is someone who writes the code that makes the game actually work. There are many different kinds of software engineers, as different types of coding work require different skills. For example, doing coding work for server code is vastly different than doing coding work for gameplay.

⁴: A strike team, in this context, is a group of individuals brought together, often for a limited amount of time, in order to tackle a specific problem or project. This frequently involves pulling subject matter experts from different teams into this smaller group in order to combine their knowledge and/or skills. This is generally done in situations where the problem or project is a very challenging or complex one.

⁵: A local build, either client⁷ or server⁸, is an internal version of the game code that is used to work on new changes. It’s a combination of the normal game, plus whatever coding work has been done by an individual or group, often focused on a specific area. This is used to do some initial testing and reviews of a change before it is brought over to the mainline⁶ version of the game.

⁶: The mainline build(s) is where all the various bits of each individual’s coding and development end up once they have been checked-in, and ultimately where the final versions of the client and server code for the game come from. If something is in a mainline build when a release is created, that means it will generally be a part of that release by default.

⁷: The client is essentially the information that is on the disc, or in the case of a digital version of the game, what you download to your console or PC before you are able to play the game.

⁸: When something is server side, what that means is that it is not on the disc, but rather code or information that comes from a server and is delivered through your online connection to that server. There are many different types of server information, and within FIFA 21, many different server systems that you are communicating with depending on which part of the game you are playing.

⁹: An embedded Quality Verification (QV) Analyst is someone from our quality verification organization that sits and works directly with Software Engineers. They test the functionality of specific changes, and try to identify any issues with those changes as part of the development process.

¹⁰: A knock-on is an issue that is created by another change, almost always something that would be unintentional. This can happen in situations where making a change in one area has an impact on the functionality of another area, due to some dependencies or shared code/tech.

¹¹: A server call is when the client makes a request to the server, generally for some new information or to validate existing information. For example, when you log into FUT, there are many server calls that take place to fetch all the information about your FUT Club, such as what items you own, what your status in the various FUT game modes is and what items you have listed on the FUT Transfer Market.

¹²: Load testing is the act of measuring the performance and stability of a server environment by simulating the actions of many people, all at once, as if they were playing the game, performing a bunch of in-game actions and making various calls to the servers. The goal is to ensure that everything will still function correctly when thousands, or millions, of players are playing the game at the same time.

¹³: The live environment, also known as the retail or production environment, is where our players are able to play the released version of the game. We also have a number of usually internal only environments, such as our test environments, where the pre-release versions of the game are tested before they are released.

¹⁴: Checking in a change means that you are committing that change into a build. In the context of this article, we primarily mean that it is being checked into the mainline build(s) for the client and server. Once a change has been checked in, that means it is present in those builds and potentially could be included in a future release.

For more deep dives on FIFA from members of the game team, check out the Pitch Notes Page.

Please Note: This article is describing in general terms what the Dev Teams are working on. We are constantly looking to improve the FIFA experience for everyone, so this article may become outdated as we make adjustments to keep our game fun for everyone.

Stay in the conversation on all things FIFA by liking us on Facebook , following us on our official Twitter and Instagram , our Dev twitter channel @EAFIFADirect, on the EA Sports FIFA Tracker and participating in the official FIFA Forums . Sign-up to receive emails about EA SPORTS FIFA and EA products, news, events, and promotions.