Never ever use oil based lubricants with condoms. That includes olive oil, baby oil, Crisco, and Vaseline. Also, be very wary of massage oil that may be involved in foreplay (watch the 1st 12 seconds to see why). What if you are in a pinch and really need some lube? Spit lube is your only alternative. Don't believe me? Watch this video:
So how KY Lube is fine but the baby oil breaks the condom in seconds. So it's just not an option and you should be really careful to avoid any kind of oil touching your condoms. Note: the condom lasts under 6 seconds with oil -- not enough time for safe sex.