Each type of mob in Minecraft has a certain AI system with different behaviors and mechanics. Passive mobs will flee in random directions after being hurt, while hostile mobs will face and chase/attack the Player as soon as they come close. Neutral mobs will remain neutral until a player or mob attacks it, at which the neutral mob will turn hostile toward and attack the entity that hit it.
Passive
Villager
The Villager has a complex AI system. When they were first added, their AI was very basic and the villagers only stood inside their homes and turned towards the player. (Needs verification)
As of version 1.2.3, Adult Villagers will socialize with each other by stopping in front of each other and facing each other. Also as of said update, juvenile villagers will socialize and sprint around, seemingly playing tag.
Hostile
Zombie
Zombies will attack players by slowly moving towards them and damages players by touching them. They do not try to avoid being hit, and continue to pursue the player even when being counter-attacked.
Before 1.2
Zombies made no attempt to avoid cliffs or other obstacles, and would simply try to jump over anything in their path.
1.2 AI overhaul
Zombies make some attempt to seek out the cover of nearby trees or caves, or enter bodies of water to protect themselves from burning up.
Skeleton
Skeletons will evade the player when hit, and will constantly maintain a certain amount of distance between itself and the mob it is attacking while firing its arrows.
Before 1.2
If a skeleton hits another skeleton with an arrow, they will attack each other until one is killed, then the "survivor" will resume attacking the player if nearby.
The presence of a skeleton is identifiable by the soft rattling sound of its bones. It will walk and jump around aimlessly until it spots the player. At that point, it will start firing arrows in the player's direction with moderate accuracy, aiming at head level when it is at full health and at torso level when injured. When approaching a target, a skeleton will move to its left just before it fires an arrow. It will also run, making itself more difficult to hit. However, these tendencies often get skeletons stuck in small alcoves and lava.
Their arrows travel in a straight line after Beta 1.8, however, prior to Beta 1.8 their arrows used to travel to in a wide arc, which would sometimes causes the arrows to get stuck in the ceiling. If skeletons are in a 2 block high area, they could almost never hit you unless they were very close, or if their health was low, and they were shooting from their torso.
1.2 AI overhaul
Skeletons received a similar AI update to the Zombie AI update. As with zombies, skeletons also attempt to seek out the cover of nearby trees or caves, or enter bodies of water to protect themselves from burning up.
Spider
Spiders run about as fast as you move normally, and they try to jump on you. In light levels of 9 or higher, they don't try to attack you unless you attack them. Spiders can climb walls.
Creeper
Creepers get ready to explode when they are one block away. They explode 1.5 seconds after they start their countdown unless you are far away. Before 1.2, they would circle to your left. Starting in snapshot 12w05a, they got the same updates that zombies and skeletons got. Creepers don't burn in sunlight. They, now will try to pathfind around blocks in their path to get to the player.
Slime
Unlike other mobs, slimes don't target players, they just move toward the closest one, even if shot by a skeleton or if the player is through a wall. When no player is within 16 blocks, they hop in one direction.