The first, and arguably most important aspect is what you're starting with. If you're doing a non-vanilla start, chances are that it will either be much more difficult (or much easier) than intended. The mod Starting Choices (Requiem version) makes it much easier. Make sure you also get the Ftweaks patch off of the 3Tweaks page too.
If you're unsure what to start with, then a good recommendation would be a weapon and some armor, vegetable soup and a snack, a tonic of stamina and some health potions, and finally a couple of Knock (Rank I) scrolls. There are plenty of other good combinations too but this is a decent start for just about any character if you're unsure.
Don't forget to use your "Choose Birthsign" and "Choose Blessing" power!
Beware of worshipping Daedra at level 1. Some of them give extreme downsides that can be very difficult to overcome for a level 1 character. Sometimes you are better off choosing another deity and swapping to your preferred at a later point.
Weaker wildlife are easy targets for insights early on. In particular, mudcrabs and slaughterfish are vulnerable targets that can be found in most lakes and rivers.
Avoid forts and towers, as most of them contain a lot of bandits. Look for caves, as those tend to have bandits in smaller numbers that are encountered 1-3 at a time, as opposed to a fort where you might draw every enemy from just one noticing you.
Chests are your most reliable source of insights. They may contain 0-3 full insights. Some chests (usually the ornate "boss" chest at the end of a dungeon or in a bandit camp) are guaranteed to have insights. They also often contain useful loot otherwise.
Wooden chests have breakable locks. You need a combined total of 350/450/550 health and stamina to break these open with an attack. This is your current health and stamina, so you can use things like alcohol or foods to get yourself above the threshold.
Other than lockpicking, you can also use the Knock spells from the Alteration tree, or buy Scrolls of Knock from a merchant (Khajiit caravans or court mages), or buy "Master keys" from Blacksmiths. Scrolls of knock are more expensive, but more versatile as each master key will only unlock the corresponding lock type (I, II, III, and IV for novice, apprentice, adept, and expert respectively).