I usually avoid in-game games too. Did every last scrap of mission content within Witcher 3 with the exception of
anything to do with Gwent - other than what was needed to progress the main story lines.
I've not minded Sabacc, even though the rules still confuse me at times
I'm wondering if it's maybe impossible
not to be in the bad books of at least one of the factions? I've been building up intel 'chips' to sell on to the Pykes to try and raise my status with them to at least 'good', but i can't even get close enough to a trader without being shot at to do so.
Might have to look for brokers with Pyke contracts available and offset any reduction in rep it causes from the other factions with that intel i have saved up.