The answer is pretty simple: the helium leaks out. Helium is a very small atom and latex is pretty porous at the scale of a helium atom (if you ever go looking to buy balloons, you will see that there are "helium-grade" balloons, which try to be thicker and less porous).

The weight balance that keeps a balloon afloat does not leave a lot of room for leakage, so once a little leaks out the balloon falls.

See How Helium Balloons Work for complete details on the weight that a helium balloon can lift!

Here are some interesting links:

• How Hot Air Balloons Work
• Would a balloon filled with vacuum instead of helium float?
• How many regular-sized helium-filled balloons would it take to lift someone?
• About.com: History of Balloons