Thanks for the help. I already tinkered around some more and found a solution for me yesterday:
I reproduced the problem in a minimal project and was able to find a solution that works for me: Use ConvexPolygonShape instead of ConcavePolygonShape. ConvexPolygonShape is also much easier to setup just by providing all corner vertices.
But I think I also found the reason why Concave did not work for me (untested):
My original Setup was like this:
Area
-> CollisionShape
Then programmatically I added the ConcavePolygonShape with set_shape. This works for some shapes (BoxShape) for example, but did not work for my ConcavePolygonShape.
Instead the solution is to delete the CollisionShape and directly set the shape on Area with
var shape = ConvexPolygonShape.new()
shape.set_points(corners)
var shape_owner = $Area.create_shape_owner($Area)
var $Area.shape_owner_add_shape(shape_owner, shape)
I did not test with ConcavePolygonShape yet. For the moment ConvexPolygonShape fits my needs much better.