This is due to "fireflies" in the generated radiance map. This is typically caused by an HDRI panorama that has an extremely bright sun (usually thousands of times brighter than a LDR white color would be). Most photo-sourced HDRI panoramas aren't suited for games for this reason.
You can alleviate this by using a panorama sky with less extreme dynamic range. You could try to edit the panorama in an image editor to tonemap its dynamic range to something more reasonable.
A more radical solution is to convert your HDR panorama to a LDR format (such as PNG or JPEG), but this will make the radiance map very dark. It might still look OK if you use a DirectionalLight.
In a future Godot version, an import option to tonemap HDR images to a more reasonable range will likely be added.
