Are you experiencing performance issues? If not, don't worry about it. Both CPU and RAM that would be used in either case I can imagine would be minimal.
That said, since you were asking for personal opinions on the matter I generally follow this rule (very loosely):
If the user or system needs to access something immediately without warning, keep it in memory if possible. If a delay is acceptable or usage is predictable then load it on demand.
Honestly, just take a look at your profiler and monitor in the debugger. It tells you how CPU intensive something is as well as how much memory is being used. You can decide what to do from there.