An important concept here is "Memory Mapped Hardware" all of the peripherals are configured and accessed by setting and getting values from Memory locations. On other platforms there may be dedicated assembly instructions or a completely separate memory space for hardware.
The following diagram is based on a Memory Map document located on github.com
The maximum 32-bit address space would normally be 4 Gigabytes. The N64 uses a very fragmented version of this that really reflects less than 75 megabytes of used space. Most memory addresses are mapped to multiple areas some of which support caching behaviors.
Combining all of these RAM locations with all of the different caching and mirroring options is pretty confusing.
Until we cover the builtin caching features let's just approach it like this.
This may change but it will be explained if/when it does.