The pedal sinks down but its losing brake pressure when it does. If I didn't let the pedal up and apply them again it would lose all braking power and would not stop.
I don't know anything about how vacuum works, I was assuming there was less vacuum at idle and that was the problem.
Edit: No leak in the brake system either, the level has always stayed full.
The brake booster creates boost from vacuum. Basically, there is a diaphragm inside - one side is exposed to vacuum, the other side is not. This lack of pressure on one side allows you to press the pedal against that side much easier. Because the throttle is closed at idle but the engine is still sucking air, there is a strong vacuum in the intake manifold, and therefore the whole vacuum system.
So let me get this straight. You're sitting still at idle. When you press the brakes, the pedal slowly sinks to the floor. Sounds to me like a fluid bypass inside your master cylinder. The piston inside is supposed to push on fluid, the seals keeping it in front of the piston, and therefore it compresses the fluid. If the seals are bad, fluid seeps behind the piston, and the pedal sinks as the piston moves forward to take the place of the fluid.
When you're moving, not only does the pedal sink, but clamping force also goes away. I don't know, that could be a result of the fluid bypass.
Either way, the sinking pedal is a symptom of fluid bypass, so I would suggest replacing the master cylinder. Unlike my car's brake booster problem, which apparently is very rare, masters go bad when they get old.
Don't even bother with a remanufactured part. Get a new one, pre-assembled with a new reservoir also. It's 90% guaranteed a remanufactured master will leak at the base of the old reservoir which you'll have to hammer into it. I tried two of them and both failed. Brand new one worked perfectly.
EDIT: Like PB said, get in the car with the engine off, press hard on the brake pedal, and see if it sinks. If it does, you've got bypass inside the master.