The 255 mod limit for Skyrim is a limitation as to how many active mod plugins (.esm, .esp) can be in the game. Think of the Skyrim plugin system as a machine with 256 slots (including 0), and each esm or esp takes up one slot approximately. Each slot has a specific index (location), which determines when they are loaded (and which plugin will override which in case of conflicts). So the engine will load plugins from 0 to 255 inclusive.
Well, technically yes, but in practice, no. What actually happens is index FF (255 in hex) is solely reserved for your save game. At slot 255 your game save exists as a sort of "dummy mod"; because of this, no other mod plugin can actually occupy slot 255 (because it's already being used).
This is also the case for slot FE (254); all ESL flagged and .esl plugins use slot FE. This is why ESL plugins have a limit to how many new records they contain; they're tiny plugins that share a single plugin slot with each other.
As a simplification, you'll see modders say that ESLs don't count toward your plugin limit. If we want to get really nitpicky then technically ESLs do take up one slot combined, but this is niche knowledge and there's really no need to flaunt it.
Also ESLs themselves have a limit of 4096 total. In the 1.6.1130 update released on Dec 5th, 2023, Skyrim ESLs ranges were doubled, thus allowing more data to be stored on the ESLs. If you somehow manage to get to the ESL limit, merging is the only option.
To sum it up:
On LE, there was a hard capped plugin limit of 255 .esp/.esm plugins.
On SE (& AE), ESLs were introduced, alongside .esm/.esp, these have a limit of 4096 and all of the esl(s) take up one singular .esp slot.
As a result, you can have a maximum of 4096+254= 4350 total plugins for SE/AE. 254 being of course the amount of .esp/.esm.
In the 1.6.1130 update, ESL record header was doubled, greatly increasing the data an ESL plugin can hold. However this means plugins created in newer creation kit had no support in older versions of the game. BEES fixes this issue.
When you have hit the plugin limit, your game will either not load or will display an error message regarding the plugin limit if you have mods like Engine Fixes.
The full forms are as follow:
* ESL: Elder Scrolls Light plugin
* ESP: Elder Scrolls Plugin
* ESM: Elder Scrolls Master plugin
* ESPFE: ESP flagged ESL. This is a .esp file that loads as an ESL towards the 4096 cap instead of the ESP 254 cap.
Two ways: ESL flagging and merging. ESL flagging is the safer and easier method.
ESL flagging can be done manually via xEdit. More on that here. Alternatively, it can also be automated by an xEdit script such as Ruddy88 ESLifier. However said ESLifier has reportedly some issues functioning on the latest game version. If you encounter any issues with it, there is an alternative, SimpleESLify.
If you are unable to ESL Flag a plugin, you will need to merge it. zEdit is the go-to tool currently. Here is it's complete documentation. here is a video tutorial for merging using zEdit. ESL flagging is preferred over merging.
IMPORTANT: Do not compact FormIDs if you have an ongoing save that uses the mod you are going to compact, as this breaks all references because the actual FormID of items are changed.
Terrible things. Any plugins beyond the limit will overflow and go back to slot 00, overwriting Skyrim.esm. So even if you somehow make it to the menu without crashing you will be presented with a spectacularly unstable and buggy game.