How to diagnose bugs/behaviour that you can't come up even with an idea what might be causing them?

Nanites

Hi,

this is more of a general question, I'm curious how do you go ahead diagnosing behaviour that you can't come up even with any type of explanation and also can't isolate?

In my case, I have a menu located outside my games level, which slides down when you press the menu button. From there, you can then open the level select screen that will make it slide down even more - the level select screen is located above the main menu. The whole thing is a Control node, so if the menu is closed it's rect_position.y = 0, if the main menu is open it is 150, and if the level select is open its 1050.

The menu is opened by an animation player that just has one track moving the rect_position of the control node.

Now, on all my phones and the PCs this behaves normal, with the exception of the Nokia 2.1 (which I use as a minimal target - if my games runs ok on it, I'm fine with the performance. It's a phone from 2018 that was universally critiqued for having bad performance even for its 100$ new price back then). There, I get 60 fps (capped to it) when I start the game. As soon I open the main menu, the fps drop to about 30, and it stays like that, even if I close the menu. Now the weird part: nothing happens if the menu is opened, there is no function that runs or process or physics_process. It's just a container with some buttons...

But the fps stays 30 even if I reload the main scene, I have to restart the game completely to get it back to 60. I tried disabling visibility of the control node holding the menu when it closes, and then the fps will go to 40 after the menu closes and stay this way.

And now even more strange: I added a debug button to set the menu position to 150 = open the main menu. FPS drops to 30 again. I changed the button to move the menu by 5 pixels every time I press it. After 30 presses the menu is at 150, but no fps drop!!!

Also, when the level select screen is open/moved into view, the fps drops to about 10 (!), despite it just beeing a bunch of buttons (again, nothing special about them, no function runs, no animations, nothing in the background). None of my other phones does that, not even the much older Galaxy S4 from 2013, which can't reach 60 fps in normal play, but barely drop fps when the level select is open (losing 5 fps or so) and going back to normal as soon the menu is closed.

I really have no idea how to deal with that... It's so weird. I checked everything that came to my mind, the profiler only shows me that the time lost is in "Idle Time", the memory usage of my game is <100MB. I'm curious how you all would go diagnosing this behaviour... I'm really out of ideas where to even look.

Edit: just tried to start the game with the menu set to 150 and already open. FPS is at 60, but if I close the menu and open it again, it drops back to 30. This makes even less sense :(

cybereality

You can try using the profiler, but it's kind of complex to grok.

Nanites

I tried, but the only value that increases when the fps drops is "Idle Time".

Is there a way to narrow it down more? From what I understand "Idle Time" is pretty much process and all the rendering/drawing? At least I don't have any process functions running slowing it down, all script functions combined are only taking about 0.4ms per frame.

Calinou

See the Optimization tutorials in the documentation, which have some advice on finding and resolving performance issues.

Try reducing the rendering resolution to see if you're bottlenecked by GPU fill rate, or by something else (e.g. draw calls). On desktop, this is done by resizing the window down to a very low size (like 200×200). On mobile, this is done by setting the stretch mode to viewport, using a low base window size in the Project Settings then exporting the project again.

I tried, but the only value that increases when the fps drops is "Idle Time".

Idle Time is indeed the time spent between rendered frames (as opposed to physics frames). It's probably a good idea to rename it to Process Time in 4.0 to avoid confusion. It technically could be renamed in 3.x as well, but it might cause confusion as other places in the engine are still referring to it as "idle time".

cybereality

Transparency can be expensive. Are you using any sort of alpha texture on your menu? That could be the issue. Also, when you move the position far enough, it may go off screen and become automatically culled. Then the problem may be that the menu is rendering all the time (over the game, but invisible).

Nanites

OK I made some progress. I just started deleting items from the menus until the fps drops disappeared. I could narrow it down to one single button. So I started removing changes from it and found it is caused by the custom styles.

This style basically drops the fps by 70% on this phone and causes it to stay low even when it's not visible anymore:

Changing "Corner Details" to the default value of 8 helps a lot, and setting Corner Radius to 0 eliminates the fps drop completely.

It looks like on this particular phone, for whatever reason, rendering rounded corners is extremely performance heavy.

@Calinou do you think this is a bug and I should make a bug report or is this expected behaviour on a very slow phone? I feel like just having a button with rounded corners should not drop the total fps by like half even with default Corner Details...

I'll find a way around it, but it makes me wonder: how many other phones are there where a feature that causes no issue on others, will make my game unplayable...

Nanites

@cybereality said: Transparency can be expensive. Are you using any sort of alpha texture on your menu? That could be the issue. Also, when you move the position far enough, it may go off screen and become automatically culled. Then the problem may be that the menu is rendering all the time (over the game, but invisible).

Not related to this issue, but do you know if Control type nodes that are completely offscreen, but their parent is still visible beeing rendered? Like you have a container of which a few pixels are still visible, but there are other node inside it that are fully off the visible area?

cybereality

I'm not sure, as I mostly work in 3D.

Calinou

I assume using rounded corners disables batching for the stylebox, as it no longer can be a single rect and needs to be drawn as a polygon.

You definitely shouldn't increase corner detail too much. In fact, we should probably cap the actual corner detail so it can never exceed the corner radius used – doing so doesn't actually improve visuals at the default resolution. This means corners may look worse at higher-than-default resolutions when using the 2d stretch mode, but it's probably not noticeable in most scenarios.

Disabling Antialiasing on the StyleBoxFlat may also improve performance at the cost of worse visual appearance.

Also, which phone device models did you test this on? I've tried to reproduce this issue on the master branch (displaying 56 StyleBoxFlats with high corner detail) and FPS remained very high on my desktop PC at 2560×1440 (over 2,500 FPS).

Nanites

@Calinou said: Also, which phone device models did you test this on?

I can only really notice it on the Nokia 2.1, a pretty slow phone from 2018. The Samsung S4, which is much older (2013) but was a fast one when it came out also slow down a bit but not as bad as the Nokia. My more modern/faster phones all run my game with 100s of fps if not capped.

I can't open your project as I don't have Godot 4 installed, I tried to make my own by using a lot of buttons with high corner details but the Nokia would still run it pretty well. I guess it must be a combination of some settings/elements that slow it down that much...

Not sure if it's worth investigating (only way I see is to delete elements from my actual game until the fps drops stop, and reverting project settings one by one?) as it probably would take a lot of time and is a very niche case?

Bimbam

I'm curious how do you go ahead diagnosing behaviour that you can't come up even with any type of explanation and also can't isolate?

Generally, rollback through previous commits to find the point in time the issue arose, and then check the diff to identify the erroneous code. After that it's more general debugging, chucking in break points and such.

Re. your specific problem, I've not deployed anything to mobile, but given this only shows up on one phone, I would initially point my finger at the spec/manufacturer/distro for this kind of thing and start there digging through forums for similar performance problems. I would not be surprised if their hardware drivers are jank, or they implement power saving fps caps when at high resource usage, or the spec is simply not up to the task, and I'd try to find a similar spec phone for comparison.

Comparing to the S4 is not like for like. It's older sure, but this was a flagship device that would have had far more time spent on optimising the software than an entry level phone, and at the time was considered the gaming performance king. It has twice as many cores and twice as much RAM as the Nokia 2.1, so I'm not really surprised it can situationally outperform an entry level phone that's a few years newer. It'd be like comparing a GTX 1080ti to an RTX 3050 (it's more like an RTX 3060/3060 ti in performance still).

The 'Solution' is probably what Calinou et al. have already replied with above and just making a 'Low' graphics mode that doesn't look as nice, but works on potatoes (I'm assuming you've already tried GLES 2renderer instead of 3?), but you'd have to draw the line somewhere re. performance targets. If you're getting a solid 30fps minimum on an S4 I would think you are good for the majority of mobile gamers today, the Google play dashboard/console I think has some stats that'd be useful for determining that.

Nanites

@Bimbam said:

and just making a 'Low' graphics mode that doesn't look as nice, but works on potatoes (I'm assuming you've already tried GLES 2renderer instead of 3?), but you'd have to draw the line somewhere re. performance targets. If you're getting a solid 30fps minimum on an S4 I would think you are good for the majority of mobile gamers today, the Google play dashboard/console I think has some stats that'd be useful for determining that.

I do have a low graphics mode, it came from the realization that there are only a few things that eats most of the performance, but they also give an significant visual effect I don't want to complete sacrifice to support old devices. Low graphics mode turn them down/off, which gives a huge performance boost but also make the game loop more plain. With it, it runs well on both the Nokia and the S4, so I guess thats fine - as you said I have to draw a line somewhere...

This corner issue however was of it's own kind, as with 2 of those buttons I could get the performance on the Nokia to drop from 60 to about 10 (!) fps. I solved it by reducing the Corner Detail setting a bit, it's not noticable on any of the phones (and the button is in view only on the level select screen anyway) but now the Nokia also does not drop fps.

And yes it's GLES 2.