As stated, yes fuel makes a huge difference, but only in races that have fuel usage - Online, A-Spec enduros, B-Spec enduros(probably B-Spec extreme as they have tire wear), I think One-Make races and Arcade have it as an option but not sure. In the others including Seasonals, offline practice, most of B-Spec and 90% of A-Spec it's turned off meaning you always have a medium-low amount of fuel weight.
Not sure how it wouldn't be noticeable in the 4-hour roadster race, it's there for any endurance races I've run. You don't notice it so much if you always run on a full tank as the wearing tires mask the effect somewhat, but if you only take the fuel you need when you pit, your car will be significantly quicker and more responsive. Tires do seem to wear less as well, which they should, but that's tricky to test.
Running Grand Valley 300 in a street car, the difference between full and half fuel on the same tire compound and running very consistent lines was at least 2-3 seconds per lap for me.