There is no hard triangle limit besides what you can fit into 8MB of N64 RAM (approx. 524000) but your level can run slower if you approach a certain amount depending on if it's geometry (processed on GPU) or collision (processed on CPU).
Geometry
This can be anywhere up to 20000 on emulator before slowing down.
Console recommendation is about 3000
Using Decomp and Rooms in Fast64, this can be arbitrarily high as long as each section of the level is around that count
Collision
This is about 3000 triangles on both emulator and console
The game can slow down if the collision map is too dense at certain points
Make sure to spread out the collision mesh with larger triangles and simpler shapes.
N64 games have several texture pixel formats. The important ones for SM64 are as follows.
RGBA16
Use any color the N64 can output
32 x 64 maximum resolution
Pixels are either fully transparent or fully opaque
CI4
16 color limit using a palette
64 x 64 maximum resolution
Pixels are either fully transparent or fully opaque
IA8
16 grayscale and transparency levels
64 x 64 maximum resolution
I4
16 grayscale or transparency levels (not both)
64 x 128 maximum resolution
Add additional flair and color to your levels.
Colors can be transparent, allowing for seamless texture transitions or geometry that fades out into nothingness.
Binary
Export your level as FBX so that your tool can actually see vertex colors.
Decomp
Set the material preset on a triangle to something with "Vertex Colored" in the name.
Fast64 requires two Face Corner layers in blender, named "Col" and "Alpha", as inputs to these materials.
For custom N64 materials, the vertex color is referred to as Shade Color in the combiner.
A triangle with three different colors applied to each vertex. Note the seamless color interpolation done by the N64.