I've heard before that the costs of infrastructure for a World Cup (basically one mega-huge-wowie-zowie stadium and a couple of smaller stadia, plus ancillary support like security and new hotel construction) is break-even after less than decade (or sooner), since you can re-use the facilities for more soccer matches. Of course, if soccer/football doesn't take off in that county, you're kind of screwed. I think the United States gets over its soccer fixation over with after about a month, and like the Olympiads, has plenty more on its plate for another 2-4 years, so I don't think hosting it would have really pushed the sport into prominence (which rates slightly above F1 racing, yet below NASCAR in in popularity).
Usually, you never get your money back on hosting an Olympics. You're usually stuck with a lot of facilities which serve no other purpose; it's really a greater-good scenario which makes the world think you're no longer a backwater province that eats children and still uses stones as units of currency...which hopefully promotes tourism.