EDIT 2: Got it working already! I used get_viewport().set_size_override when viewport's size_changed signal is emitted. Make sure to turn off "Use Oversampling" in Project Settings -> Rendering -> Quality -> Dynamic Fonts as well since the fonts will be blurry when you switch orientation
So my game should be playable in landscape and portrait orientation. When the game goes from landscape to portrait, the UI elements don't adjust their sizes. I tested on phone but for this post I created a test scene and simply set the window size by code to simulate a phone changing orientation. I have a Camera2D with the default settings. My window size in the project settings is 960x540 and the stretch mode is 2d and aspect is expand.

Here's the look in landscape orientation:
.
.
When the phone changes orientation, I want it to be like this; which happens only if I run the game with 540x960 size in the project settings
.
.
But I get this instead: (game is already running in landscape then switch to 540x960)
.
.
I tried setting the window size in the project setting during runtime but it has no effect. I tried setting the root viewport's size but it also has no effect. It looks like it sticks with the initial window size even if the orientation already changed.
EDIT: It's the same when you have 960x540 size and you entered 540x960 in the test size in the project settings
Out of topic, but anyone know why the game doesn't change orientation if Auto-rotate is off in the phone? The 'Set Orientation' permission is already on in the Android export settings.