The first, and arguably most important aspect is what you're starting with. If you're doing a non-vanilla start, chances are that it will either be much more difficult (or much easier) than intended. The mod Starting Choices (Requiem version) makes it much easier. Make sure you also get the Ftweaks patch off of the 3Tweaks page too.
If you're unsure what to start with, then a good recommendation would be a weapon and some armor, vegetable soup and a snack, a tonic of stamina and some health potions, and finally a couple of Knock (Rank I) scrolls. There are plenty of other good combinations too but this is a decent start for just about any character if you're unsure.
Don't forget to use your "Choose Birthsign" and "Choose Blessing" power!
Beware of worshipping Daedra at level 1. Some of them give extreme downsides that can be very difficult to overcome for a level 1 character. Sometimes you are better off choosing another deity and swapping to your preferred at a later point.
Weaker wildlife are easy targets for insights early on. In particular, mudcrabs and slaughterfish are vulnerable targets that can be found in most lakes and rivers.
Avoid forts and towers, as most of them contain a lot of bandits. Look for caves, as those tend to have bandits in smaller numbers that are encountered 1-3 at a time, as opposed to a fort where you might draw every enemy from just one noticing you.
Chests are your most reliable source of insights. They may contain 0-3 full insights. Some chests (usually the ornate "boss" chest at the end of a dungeon or in a bandit camp) are guaranteed to have insights. They also often contain useful loot otherwise.
Wooden chests have breakable locks. You need a combined total of 350/450/550 health and stamina to break these open with an attack. This is your current health and stamina, so you can use things like alcohol or foods to get yourself above the threshold.
Other than lockpicking, you can also use the Knock spells from the Alteration tree, or buy Scrolls of Knock from a merchant (Khajiit caravans or court mages), or buy "Master keys" from Blacksmiths. Scrolls of knock are more expensive, but more versatile as each master key will only unlock the corresponding lock type (I, II, III, and IV for novice, apprentice, adept, and expert respectively).
A list of dungeons in case you aren't sure where to go next.
Falkreath:
Camp north of helgen and west of Riverwood (3 bandits, has map for nearby Treasure chest)
Bannermist Tower (3 bandits and a chest behind an apprentice lock)
Smuggler's Sanctuary (Several bandits but scattered throughout the cave, has boss and boss chest)
Embershard (boss chest and couple normal chests)
Falkreath Watchtower (Necromancer, has normal chest and Woodsman's Friend unique 2H weapon)
Whiterun:
Bandits north of city walls (boss chest behind novice lock)
Poachers by Lorius Farm
Whitewatch Tower bandits
Redoran's Retreat (boss chest and treasure map)
Halted Stream Camp (boss chest, knock I spell tome, several potions and normal chests)
Volunruud Exterior (chests and treasure map NNW uphill)
White River Watch (boss with unique gauntlets and boss chest)
Bandit "ambush" on the road north of Rorikstead
Riften:
Ratway (enchanted stamina regen ring and favor quest)
Zenithar Shrine NE of gate (amulet and bear, no bandits)
Dead-Drop Tower (boss chest)
Broken Helm Hollow (boss chest and favor quest)
Rift Watch Tower
Ruins of Bthalft
Banded-Branch Gorge
Solitude:
Dainty Sload
Wolfskull Cave (pre quest)
Broken Oar Grotto
Fort Hraggstad
Orphans Tear
Ironback Hideout
Windhelm
Uttering hills cave (boss chest)
Wreck of the winter war (boss chest)
Stony creek cave (boss chest)
Cragslane Cavern (boss chest, part of Riften small quest line)
Winterhold
Bleakcoast Lookout
Skytemple Ruins (Skeletons)
The Pride of Tel Vos
Journeyman’s Nook (valtheim treasure map)
Dawnstar
Helvuud
Lord stone
Avanchenzel (No falmer)
Duskglow Crevice (Hammer and Ring of Inebriation)
Shimmermist Cave
Gloomreach
Raldbathar (except last room with enchanted sphere)
Mzinchaleft (centurion boss)
Stillborn Cave
Soljund’s Sinkhole
Skeletons + Draugr: Hamvirs Rest (boss chest)
Single encounter mages scattered around
Skeletons + Necromancer: Lost Tongue Overlook, pre-dragon (random staff)