Now how to handle the huge files. The huge files were either textures or meshes. I'll handle textures first.
Most of the textures were enormous, either 4K or even 8K. Loading these gave most of the insane lag spikes.
I set LOD's for almost all textures. The difference in how it runs is immediately noticeable without it impacting the look of the game. The only exception to this is the big rock the lighthouse is on. It looks weird in a lower resolution. I think this is a good exception to have since it is part of a landscape. By the way, when I was running a quick test the FPS was already 60 after this step!
I fixed the enormous meshes by enabling nanite for them. This was especially helpful for the island and the lighthouse.
This step also helped a lot with rendering the scene. Especially the cliff does not render in super high detail anymore.