No. Whilst the Bentley team was effectively Audi in green, you also had Cadillac, Chrysler and MG racing during the earlier of those years, as well as Panoz, although I'm not sure if they count as a factory team. In 2004 and 2005 there were no factory teams in LMP1 at all - the Audis that were run in those years were all privateers.
I meant in LMP1 or whatever it was called back then. 2004 and 2005 was essentially re-runs of the 2003 car without any real competition?
I am of course confused why Toyota is getting so much "hate" when one of the most popular teams essentially did the same for 7 years where TOM K earned 6 victories..