Each type of mob in Minecraft has a certain AI (Artificial Intelligence) system with different behaviors and mechanics. Passive mobs will flee in random directions after being hurt, while hostile mobs will face and chase/attack the Player as soon as they come close. Neutral mobs will remain neutral until a player or mob attacks it, at which the neutral mob will turn hostile toward and attack the entity that hit it.
Passive
Villager
The Villager has a complex AI system. When they were first added, their AI was very basic and the villagers only stood inside their homes and turned towards the player. (Needs verification)
As of version 1.2.3, Adult Villagers will socialize with each other by stopping in front of each other and facing each other. Also as of said update, juvenile villagers will socialize and sprint around, seemingly playing tag. All villagers will enter houses when it is raining or night, or when they are chased by zombies.
Villagers do not attack or flee when the player attacks them, but an Iron Golem may protect them (the village has to be large enough). Villagers may also interact with the Golem : when they see one, they stop and stare at it. The Golem may in turn give them a rose as a symbol of friendship. Because the children are shy, when an Iron Golem offers them a rose, they will approach slowly and run away as soon as they have the rose.
Hostile
Before 1.0, hostile mobs such as zombies, creepers, and spiders did not pay attention to anything besides the player, leaving them vulnerable to dangers such as lava pits and sudden cliffs. The mob would charge blindly forward instead of attempting to avoid said and other dangers. This made it easier to dispose of mobs, albeit mainly on Creative, as in Survival the player cannot fly and is vulnerable to these as well.
Zombie
Zombies will attack players by slowly moving towards them and damages players by touching them. They do not try to avoid being hit, and continue to pursue the player even when being counter-attacked. They also will break down wooden doors to get to a person or villager. They also target and attack villagers.
Before 1.2
Zombies made no attempt to avoid cliffs or other obstacles, and would simply try to jump over anything in their path.
1.2 AI overhaul
Zombies make some attempt to seek out the cover of nearby trees or caves, or enter bodies of water to protect themselves from burning up during the day. They even will jump into lava to escape daylight[citation needed].
Skeleton
Skeletons will evade the player when hit, and will constantly maintain a certain amount of distance between itself and the mob it is attacking while firing its arrows.
Before 1.2
If a skeleton hits another skeleton with an arrow, they will attack each other until one is killed, then the "survivor" will resume attacking the player if nearby.
The presence of a skeleton is identifiable by the soft rattling sound of its bones. It will walk and jump around aimlessly until it spots the player. At that point, it will start firing arrows in the player's direction with moderate accuracy, aiming at head level when it is at full health and at torso level when injured. When approaching a target, a skeleton will move to its left just before it fires an arrow. It will also run, making itself more difficult to hit. However, these tendencies often get skeletons stuck in small alcoves and lava.
Their arrows travel in a straight line after Beta 1.8, however, prior to Beta 1.8 their arrows used to travel to in a wide arc, which would sometimes causes the arrows to get stuck in the ceiling. If skeletons are in a 2 block high area, they could almost never hit you unless they were very close, or if their health was low, and they were shooting from their torso.
1.2 AI overhaul
Skeletons received a similar AI update to the Zombie AI update. As with zombies, skeletons also attempt to seek out the cover of nearby trees or caves, or enter bodies of water to protect themselves from burning up during the day.
Spider
Spiders run about as fast as you move normally, and they try to jump on you. In light levels of 9 or higher, they don't try to attack you unless you attack them. Spiders can climb walls.
Creeper
Creepers get ready to explode when they are one block away. They explode 1.5 seconds after they start their countdown unless you are far away. Before 1.2, they would circle to your left. Starting in snapshot 12w05a, they got the same updates that zombies and skeletons got. Creepers don't burn in sunlight. They will try to pathfind around blocks in their path to get to the player.
It is their pathfinding that gave creepers their reputation. As they are always trying to get as near the player as possible, they will often hide and seem to ambush the player. They also don't have many weaknessess, making them harder to slay than your typical zombie.
Creepers will flee from ocelots and cats. This is very useful, as tamed cats will follow the player and thus protect them from creepers.
Slime
Unlike other mobs, slimes don't target players, they just move toward the closest one, even if shot by a skeleton or if the player is through a wall. When no player is within 16 blocks, they hop in one direction and run into walls then turn.