I bought my 1998 GT with only 76K on the clock for $7000. I overpaid a little, but the car was in outstandingly good shape. It looks like that in Michigan, you can get pretty decent 5th and 6th generation models for $4000 and under easily, although the 6th generation ones seem to be much more rare than its predecessor and its successor.
As far as insurance goes, I've got full coverage with low deductibles and I pay only $125 a month. Being a bit younger, I imagine that it would be a little higher for you, but overall, it isn't that bad. Shop around a bit, you might be able to get better rates in other places.
RE: The 5S-FE
Its not a screamer. Don't let the Toyota badge fool you. You're looking at a 6200 RPM redline, most of the power coming on in the low/mid range. What I love about the car is that I can leave it top gear and pass easily, or when going around corners, I rarely have to go lower than 3rd most times. It helps with the fuel economy, thats for sure.
...And generally, the reliability can't be beat either. Although I just had the problem with the EGR valve, any other time, it runs problem free. We had a '95 Toyota Camry with the same engine and that ran nearly 205,000 miles before the car was in a bad deer accident. All we ever had to do was change the oil, a seal or two, and the timing belt. Otherwise, it was completely trouble-free, and likely would have gone at least another 20K at the previous rate of use.
In my biased opinion, the 6th generation is by far the best-looking of the FWD Celicas, and considering that you can get them for pretty cheap, they aren't a bad idea. Thing is though, I'd recommend shopping around if you can. A Civic Coupe would be a decent alternative, and hell, you could have some fun with a regular-ass Corolla as well.