The camera system was a first of its kind system, considering Mario 64 was one of the first 3D platforming games. The system makes use of a sometimes dynamic, sometimes hard-coded presets and modes to make the camera work for its given situation.
The camera was supposedly solely written by Takumi Kawagoe, a developer who also helped write the camera for Ocarina of Time. Most of the insight on the camera development comes from interviews with Kawagoe and fellow developer Giles Goddard, as well as the a Nintendo 64 patent that details information on the camera.
The camera consists of several types of modes, beyond the types the player controls. These change depending on the situation, altering angles and positions to try and adapt for the situation. In each of these sections, generally only one specific mode is utilized in a group at a time.
Player Controlled Modes
These modes are modes that the player controls, specifically by usage of changing in the pause menu and using the R button. The idea for these was to help allow players to overcome camera issues and play the game as they want.
This is default camera mode, representing a camera that supposedly is carried by Lakitu. The camera will keep Mario in focus while attempting to give a clear angle of the geometry around Mario.
The default alternative camera mode, this one is selected by usage of the R button. Pressing the button puts the camera into a mode that more closely follows Mario, akin to a zoomed-out first person mode. This can often override other hidden modes, and provides a typically abnormal perspective.
This mode must be selected through the pause menu. While holding R with the option selected, the camera will stop moving and will only continue to angle to follow Mario. This can be abused, as preventing the camera from moving will keep it from travelling to PUs- which crash the game.
Camera Controlled Modes
These modes are activated by many different scenarios, such as the level/area loading, water, actions, floors, and objects. Although these modes are not actively noticed, they greatly control the behavior of the camera. The camera can act almost completely different depending on the mode.
This mode (or rather, lack of mode) is used when on certain screens. The title screen and file select screen use this. This keeps the things being looked at in view and the camera from moving in these set situations.
In Reverse Tower mode, the camera follow either along a vertical axis or stays on a line between the focus and the axis. This gives a perspective as if Mario is climbing around the outside of a structure, with the camera staying in the middle. Areas that use this include the first DDD area, TTC, and the volcano in LLL.
This mode follows decently strictly behind Mario, with the C side buttons giving a 90 degree view from the side while held. This isn't set by default in any levels, but is applied after cannon shots or while swimming underwater.
Similar to the open camera, except this mode follows Mario closer. It is set by default in stages such as BBH and Eyerok's Lair.
As the name implies, this is a sort of first-person mode that activates whenever Mario reads a sign or C Up is pressed. This allows to look at places where the camera normally won't go to.
This camera is activated when fighting certain enemies, specifically King Bob-omb, King Whomp, and Bowser. Specifically, the floor activates for the first two and the latter are loaded in with the level. This mode makes the camera zoom out to allow both the normal focus and a secondary focus (i.e. the boss) on screen. The zooms the camera much farther out.
In this mode, the camera moves along a line and tracks the focus. Outside of cutscenes, the only use of this is in the library in BBH.
The camera wobbles during the animation of running out of health on the ground, and in some cases during the star collecting animation (the other possibility being for it to zoom out).