Worried? No. You've got bigger things in your life to worry about than a racetrack in a different country.
Tsukuba is in a somewhat rural area and it appears there aren't any residences immediately next to it, but definitely within earshot, and local roads to and from the track certainly pass through these areas. As is often the case in the automotive world, some people have no chill and can't drive loud cars quietly without causing a scene. It is an absolute plague in America especially. There is an oval and dragstrip near my city of Dayton that is notorious for obnoxious drivers entering and exiting the track, especially exiting late at night, and these people are the primary source of complaints and police action rather than actual scheduled track events.
It's understandably difficult for residents to separate the track and the track-goers when it comes to noise, feeling that the track should control them in some way. I also feel this way in my own neighborhood, as there is a customs shop which attracts the wrong kind of Mopar customers, and I often feel that the shop needs to vet or control its customers better. But once people are off the property that's not really their job anymore.
I don't know what Japanese vehicle noise laws are but they would appear to not be very strict as loud exhausts are synonymous with Japanese tuning. I also don't know the culture of track complaints are, which in certain parts of America has indeed caused shutdowns. Japan seems to have the opposite problem of the US - whereas American suburbs are generally expanding and encroaching on previously rural tracks (and airports), Japan's population and actually dropping rapidly, particularly in rural areas, and new development does not seem nearly as robust.
I would assume that the decline of global car culture and the rise of poor behavior is also effecting Japan. Authorities should address bad behavior appropriately. Hopefully the problem is outside the track rather than inside.
