Each type of mob in Minecraft has a certain AI (Artificial Intelligence) system with different behaviors and mechanics. Passive mobs will flee in random directions after being hurt, while hostile mobs will face and chase/attack the Player as soon as they come close. Neutral mobs will remain neutral until a player or mob attacks it, at which point the neutral mob will turn hostile toward and attack the entity that hit it.
All Mobs
All mobs randomly walk around. When they come to a 1 high wall, they jump. They avoid falling 4 or more blocks (falling 4 or more blocks means taking fall damage), except for hostile mobs in 12w37a. They all float in water except for slimes, magma cubes, and iron golems, which sink.
Passive
Farm Animals
Farming Animals: chickens, cows, pigs, mooshrooms, and sheep, are attracted to light, grass, or wheat (carrots in the pig's case, and seeds in the chicken's case) that is carried by The Player. If two animals of the same species, within five minutes of a feeding, are fed wheat (Carrots for pigs, Seeds for chickens), they will walk towards each other into a "kiss" to spawn a baby species of the fed animal. If an animal is damaged, it will sprint in random directions. All animals wander in a random direction for a short amount of time, then changes its direction. Along with other mobs, animals avoid lava and areas greater than one block downwards, and will jump over blocks in their direction that are no greater than one block. Farming Animals will wander into water if in their path and will always swim upwards to the top of the fluid, even if a block is blocking air.
Pig being ridden by a player
If the player is holding a carrot on a stick, the pig moves in the direction that the player faces. The pig gradually speeds up until it moves at 5 blocks per second. It will stop responding to the player if it touches water.
Villager
When first introduced, Villagers had a similar AI system to other Passive mobs like Farming animals, and remained in their houses unless the wooden door is opened and allowed to walk outside. After a series of twelfth-week snapshots, Villagers have gained a more complex AI system.
As of version 1.2.3, adult villagers will 'socialize'. This is modeled by a pair of villagers moving to face one another at a distance of about a meter, as if they were talking. Sometimes larger groups will form in a similar way. Child villagers 'socialize' in the same way as the adults, and will also sprint around the village, seemingly playing tag. All villagers will attempt to enter houses when it is raining or at night, and will also flee from zombies. Note that identifying a 'house' is a challenge for a computer AI, and this may give rise to unexpected villager behavior.
Villagers will neither attack nor flee if the player attacks them, but if their village is large enough to have spawned an Iron Golem it may arrive to protect them. Villagers will also interact with the golem: when they see one, they stop and stare at it. The golem may in turn offer them a rose as a symbol of friendship. Because the children are shy, when an Iron Golem offers them a rose, they will approach slowly and run away as soon as they have the rose.
House
The bare minimum that a house can be. A door with at least 1 building block at same height within 30 blocks. Villagers will congregate within a few blocks of the door regardless of structure size.
Villagers try to live in 'houses', and they use the number of available houses as a cue for whether or not to breed. To a villager, a house is a door which has at least 1 building block (usually wood) at the same height within 10 blocks of the 'inside' of it. An acceptable 'house' for a villager does not require any form of lighting nor a roof. This means a building can have as high of a roof as possible or no roof at all, and can be completely dark, as long as the opposite wall to the door is within 10 blocks. Also, the floor material of the house is not relevant, and can be 1 block lower or higher than the door upon entering (As long as the door frame accommodates head space for a higher floor).
It is possible to build very many seemingly reasonable house-like structures that the Minecraft AI cannot recognize as houses, and it's also possible to build structures which do count as 'houses' despite being parred down to the very minimum.
Note that because a 'house' needs a door, and zombies in Hard mode can destroy doors, after a zombie attack even the surviving villagers may stop recognizing their former residences as houses and become lost when they wander off into the wilderness. Although, if the complete structure of a 'house' is destroyed except the door, previous residents will still congregate on the 'inside' of the lone door, but newly spawned villagers will not recognize the lone door as a house.
Other important notes: A villager cannot find a house unless the door of the house (regardless of structure) is within 15 blocks. While actively occupying a 'house' (as to escape rain or such), villagers will stay within 3 blocks of the door and will not wander throughout the structure. Villagers will try to squeeze as many of themselves as possible into a single 'house' if it is the nearest to them.
Bat
Bats are the only passive mob in Minecraft that flies, and the only flying mob in the overworld. Bats spawn in "caves", where the light level is 10 or less (higher than for hostile mobs) and hang upside-down during the day. At night or when approached by the Player, bats will take off and fly around randomly. All bats will attempt to enter a cave when daylight approaches, although they are not affected by daylight.
Squid
Squid spawn in water and swim in it. If they're not in water, they don't move. Their tentacles always move, even if they're not in water. If they are not in water they will suffocate and die within a few seconds.
Ocelots
Ocelots sprint away from a player if the player is close. When a player holds raw fish, the ocelot slowly moves closer.
Tamed cats
Tamed cats follow its owner, and if it's near its owner, it will wander within 8 blocks of its owner. Tamed cats will teleport to its owner if more than 16 blocks from its owner if it's standing. Unlike wolves, cats don't attack.
Neutral
Wolves
Wolves are neutral mobs that spawn in almost every biome, except the ocean biome. Wolves live in packs of up to ten wolves that will attack sheep naturally. They will wander around in the pack and ignore you, but if you attack one, all the wolves in the vicinity will turn hostile and attack you. You must then either kill the wolves or let them despawn. They will not attack cows, pigs, chickens, or mooshrooms. They will shake when getting out of water.
Tamed wolves
Tamed wolves follow its owner, and if it's near its owner, it will wander within 8 blocks of its owner. Tamed wolves will teleport to its owner if more than 16 blocks from its owner if it's standing. It will attack any mob or player that its owner attacks. It will also attack any mob or player that attacks its owner. Note: Wolves will not attack creepers.
Endermen
Endermen are neutral mobs that spawn in the End and the Overworld. They will pick up blocks and bring them away naturally, which is bad for you if they want to take down a wall. They are passive at first, but turn hostile if you put the crosshairs in between its eyes or on its torso. Only then, will the enderman become hostile. It will become neutral if it touches water. It teleports when it touches water, sometimes when it's attacked, sometimes when it attacks you, if it's in sunlight, or if it is hit with a projectile. In daylight, Endermen will teleport from tree to tree, trying to avoid daylight. During this state, it will sometimes carry a block, like it wants to save its "precious" dirt block.
Zombie Pigman
Zombie Pigmen are neutral mobs that roam around the Nether in packs of up to 7. Wielding golden swords, they are peaceful at first, but when attacked, all of them in the area attack you, like wolves. They cannot be killed by fire. They are faster than zombies. Some zombie pigmen can pick up items.
Iron Golem
Iron golems are found in villages, and they can be created by the player. Iron golems move slowly. Iron golems attack hostile mobs except for creepers. Iron golems attack players if attacked by the player (except the person who created it), if a player attacks a villager nearby (if in a village), or if the player has a bad reputation in the village that the iron golem is in, being hostile to that player only.
Hostile
Before 1.0, zombies, creepers and skeletons ignored everything except their target, once they had been aggravated by the player or another entity, leaving them vulnerable to environmental dangers such as lava pits and sudden cliffs. The mobs would charge blindly forward instead of attempting to avoid said hazards. This allowed for the creation of simple pitfall traps. As of 12w37a, hostile mobs are now more "fearless" in Hard difficulty.
Zombie
Zombies attack by moving relatively slowly towards a player or a villager, and will damage them on contact. They do not try to avoid being hit, and will continue to pursue the player even when being counter-attacked. If they encounter a door they will bang loudly on it, but they can only actually break through a wooden door on the Hard difficulty setting or in hardcore mode. They cannot break through iron doors. In hard mode, their habit of attacking doors makes them a significant threat to villagers, as after a few days of attacks they can destroy every door in a village, leaving the villagers 'homeless' and highly vulnerable. (See the Villager AI description for further details.) Also, they slow down for a few seconds when hit. They are healed by damage potions and damaged by health potions. Some zombies can pick up items.
Before 1.2
Zombies, Skeletons, and Creepers made no attempt to avoid cliffs or other obstacles, and would simply try to jump over anything in their path.
1.2 AI overhaul
Zombies and Skeletons will attempt to find and run to the cover of nearby trees or caves, or enter bodies of water to protect themselves from burning up during the day. They also try to avoid 4+ block holes and lava where possible. If the player is on the same level as the Mob, with a hole in between, the Mob will go around the hole if possible.
Skeleton
Skeletons will circle the player with a constant radius while attacking in an attempt to evade attacks. If a skeleton is attacked by another mob, it will ignore the player and turn all of its attention to the attacker. This is a good time to fight back with swords, axes and bows. They are healed by damage potions and damaged by health potions.
Before 1.2
The presence of a skeleton is identifiable by the soft rattling sound of its bones. It will walk and jump around aimlessly until it spots the player. At that point, it will start firing arrows in the player's direction with moderate accuracy, aiming at head level when it is at full health and at torso level when injured.[citation needed] When approaching a target, a skeleton will move to its left just before it fires an arrow. It will also run, making itself more difficult to hit. However, these tendencies often get skeletons stuck in small alcoves and lava.
Their arrows travel in a straight line after Beta 1.8, however, prior to Beta 1.8 their arrows used to travel to in a wide arc, which would sometimes cause the arrows to get stuck in the ceiling. If skeletons were in a 2 block high area, they could almost never hit you unless they were very close, or if their health was low, and they were shooting from their torso.
1.2 AI overhaul
Skeletons received a similar AI update to the Zombie AI update. As with zombies, skeletons also attempt to seek out the cover of nearby trees or caves, or enter bodies of water to protect themselves from burning up during the day. Skeletons will also flank the player out of their hiding spot should the player try to hide from the skeleton. For example, a skeleton spots the player and the player attempts to hide around a corner of a wall. The skeleton will seek out the player and attack once they found them. Similar to creepers, skeletons now avoid falls if they can. It is a rare occurrence but you can find skeletons riding spiders. This is of course only if the spider spawned with the skeleton on it. However in the update they no longer circle the player, deeming them more of a target than a threat. Some skeletons can pick up items.
Wither skeleton
Wither skeletons charge at you instead of shooting at you because they hold a sword. If a normal skeleton holds a sword for some reason, it will do this too.
Spider
Spiders run at the same speed as the player, and attack by leaping at you. If encountered in light levels of 9 or higher, they don't try to attack you unless you attack them. However, a spider that spots you in a light level less than 9 and then ventures into a light level 9 or higher will remain hostile. Spiders can see through and climb walls. This will often leave them pushing against the ceiling of an adjacent cavern in a vain attempt to reach the player if spawned nearby underground. Since they're 2 blocks wide, they can't fit through doorways, though this won't stop them from trying.
Cave spider
In addition to what is mentioned in the "spider" section, cave spiders are smaller, and they poison their target, but only in normal mode or harder. Note that being smaller it can fit through smaller spaces as well, such as slabs and 1x1 passageways.
Creeper
Creepers prepare to explode when they are one block away from their target. As long as the target is still in a one block range, creepers will detonate after exactly 1.5 seconds. Before 1.2, they would circle to your left. Starting in snapshot 12w05a, they got the same updates that zombies and skeletons got. Creepers don't burn in sunlight. They will try to pathfind around blocks in their path to get to the player.
It is their path finding that gave creepers their reputation. As they are always trying to get as near the player as possible, they will often appear to hide and seem to ambush the player. They also don't have many weaknesses, making them harder to slay than your typical zombie.
Creepers will flee from ocelots and cats. This is very useful, as tamed cats will follow the player and thus protect them from creepers.
Slime
Unlike other mobs, slimes don't target players, they just move toward the closest one, even if shot by a skeleton or if the player is through a wall. When no player is within 16 blocks, they hop in one direction and run into walls then turn. Because they don't target a particular player, they won't attack you if you attack them in creative. Slimes can't swim. Slimes will not avoid fire, deep falls, or other things that hurt them, making them easy to trap and kill.
Ghast
Ghasts can seek out players from 100 blocks away. Once they come within range and can see you, they shoot fireballs, which have 1/3 the explosive force as a Creeper and light surrounding blocks on fire. Unlike other hostile mobs, Ghasts will not attempt to draw close to the player once aggravated, but will instead hover about 24 blocks from the player, firing repeatedly at them. They do not become aggressive under any circumstances in creative mode, but it has been confirmed that this is a bug. This bug happens because ghasts don't have a search mode.
Spider Jockey
The spider controls how both mobs move. The spider acts like a normal spider, and the skeleton acts like a normal skeleton, except its movement is determined by the spider it rides upon. The skeleton's arrows will sometimes damage the spider jockey itself. This can be encouraged by digging a 1 block wide hole 3 blocks straight down and simply standing in it; the spider will sit over top of the hole and the skeleton will fire straight down in an attempt to hit the player, but will only end up hitting its mount. Occasionally, the skeleton will get off the spider. This will also happen if you log out and back in.
Blaze
Blazes fly if they see you. About every 10 seconds, they will catch fire, then shoot 3 fireballs at you in quick succession. The fireballs will set the ground alight wherever they land. They tend to strafe right (your left if you're facing them) when attacking. Blazes are damaged by water and snowballs, as well as rain/snow.
Bosses
Ender Dragon
The Ender Dragon is currently one of Minecraft's two bosses. Only one spawns in the End at a time, and killing it is one of the two ways to exit the End, other than dying. Killing the Ender Dragon is the only way to finish Minecraft. It can regenerate health by flying near one of the multiple End Crystals in the End. It has 100 hearts, which makes it quite difficult to kill without a diamond sword and bow. The Ender Dragon only takes 1/4 damage from anything that doesn't hit it in the face.
Wither
Withers are an undead type of mob that is spawned when the player chooses, like snow golems and iron golems. When spawned by the player, they will be invulnerable for about 10 seconds, then they will create a huge explosion. They fly 4 blocks above the player. Their three heads (projectiles) will attack the player and all mobs that are not undead. Their heads will cause an explosion like a ghast fireball, but instead of fire, it causes the Wither potion effect. Withers will break blocks between it and the player. When the wither reaches half health, it will stop flying, and it will be immune to arrows.