Just my opinion but depends more on how good your TV is. If you have 4k@120hz, you probably should go with X. If 4k@60hz, Serie X isn't going to be a big difference versus S, picture wise that is. It's storage capacity/Disk and other additional options that make difference at that level. One small note I always found interesting is the Series S actually loads faster than the Series X.
I am on a Hisense U6G and a Series S btw. I have also had PS5 and Xbox One S on the the same TV. Afraid to upgrade TV cause then I will have to upgrade to Series X, which the package would cost me at least $1,500. I'd rather buy a large curved monitor and DD for that price.