How devs make sure there is no regressions when they submit any changes

Vale-git

Hi. I wonder how the "team" in development's charge face this problem? from github I see a lot of regressions. Even now just switching between two beta I came across regressions.. I know that are betas.. but I don't expect regression bugs rather new bugs.

They have a QC system in place? Idk.. some unit tests? integration tests? Or are so fast placed that they can't use any of them?

Someone know an answer?

Vale.

TwistedTwigleg

For Godot? There is now a test suit that runs automatically when submitting a PR, which helps a bit with spotting regressions or breaking changes, but it does require a test to be written first. I think the Godot team primarily tries to get someone who has experience with the given piece of code to look at it and test it for valid functionality as the primary way to spot regressions or changes in behavior. Ideally the change would be tested on several different projects that all use the system/node/thing that is getting changed, but that can be difficult without having a large volume of projects to sample from. In general, what I try to do when reviewing changes to Godot is test the bug/issue replication project before, after, and try to test at least one other project that uses the same feature to make sure it works the same before and after the change. It doesn't always work, I've accidentally made several regressions myself, but it can help catch problems before they arise sometimes.

I'm not sure how other teams handle it, though I know in enterprise using unit tests are fairly standard. It's not quite as standard for game development, from what I have observed, which I think is mostly due to the complexity and interconnections that are often in game development systems. I think generally game development studios have Quality Assurance (QA) departments that help bug test and spot regressions in most game studios, but I'm not certain.

Calinou

They have a QC system in place? Idk.. some unit tests? integration tests? Or are so fast placed that they can't use any of them?

There are some unit tests in place in the master branch, but this unit testing system wasn't backported to the 3.x branch due to the added maintenance it would require. Moreover, the unit test suite in master currently has relatively low code coverage. It's very difficult to get good code coverage in a game engine as large as Godot, since many parts such as rendering and physics are not deterministic. In comparison, it's much easier to get > 80% test coverage in a small library or framework.

There are also various integration tests present in master, such as the GDScript test suite. Both master and 3.x have crash/memory leak checks on CI (master, 3.x).

Unlike some proprietary game engines out there, we don't have paid QA teams. This means we rely on voluntary testers to test beta releases in their real world projects and report regressions. This is standard procedure in pretty much any piece of open source software out there :)

We're also at the mercy of hardware/driver-specific bugs, especially on macOS, iOS and Android. Most core contributors don't have access to Apple hardware, and generally only have one or two Android devices to perform manual tests with.

Vale-git

As someone who is in iT since pre dos era - and still keen to learn ;) - I understand current godot scenario. Probably to have someone who take care of QA and to write test is right.

But if it do not exist we have t find a solution nevertheless.

My 2cent idea: When people submit an issue devs ask for a "unit" project as test... So need someone who collect all these "unit tests" and use them from now on in a "suite test cases". No need to create anything special.. just a batch where code can be submitted to this batch. It have to be made in two layers. One as single and one as integration.
In this scenario submit a small project must be mandatory because it will be the one used in batch. So people will give it more easy because they will know its necessity.

I know we still have hardware issue as @Calinou mentioned. But when a "batch test" is released people can use it in their own hardware and submit something back (log, email..)