What are custom tactics? And I am with Newcastle.
Ok, I said I'd get to this tonight, so I'm going to at least get started. Full disclosure though, I'm pretty wiped out from work (13 hour shift today...) and I've had a few to wind down

so this might be a little bit hard to follow
Custom Tactics is an option from the Team Management screen. I brings up the ability to closely modify the way that your team plays, regardless of the formation. Tactics & formation are separate, but obviously you should use tactics that fit your desired formation & playstyle.
Custom tactics are broken down into 3 categories: Build Up Play, Chance Creation, and Defense. Each category has several options within it:
Build Up Play--> Speed
Adjustable from 1-100, this governs how quickly your players will try to look to play the ball upfield. Lower numbers mean slower build up, with high numbers being good for counter-heavy strategies.
Build Up Play--> Passing
Adjustable from 1-100, this governs the distance & frequency of passes your players will look to play through the midfield. Lower numbers mean shorter passes, with higher numbers being more suited to Route 1 football.
Build Up Play--> Positioning
Adjustable between Organized & Free Form, this governs how strictly your players will remain within the zonal restrictions of your formation during build up play. Organized allows for better coverage if you lose the ball, as your players are less likely to be out of position; Free Form is better if you want a lot of interchange and overlap from your players, but carries the risk of being more vulnerable to counter attacks.
Chance Creation--> Passing
Adjustable from 1-100, this governs the distance & frequency of passes your players will look to play in the final third. The adjustment scale is similar to the Build Up passing slider, but is more focused on attacking. This adjustment is very closely tied to the Shooting adjustment; lots of passes with lots of shooting is counterproductive, as is few passes with little shooting.
Chance Creation--> Crossing
Adjustable from 1-100, this governs the frequency with which your players will look to put crosses in. For a team that likes to possess, this options should be fairly low, but you want to make sure it's not a non-existent option. This is easily the most self-explanatory slider here; Crouch=high number, Giovincco=low number.
Chance Creation--> Shooting
Adjustable from 1-100, this governs how the quality & frequency of chances your players will look to take. Low numbers means they wait for the tap-in, high numbers means half a chance from a bad angle is a shot. This slider is very closely related to the Chance Creation- Passing slider, and their values need to have synergy for optimal results.
Chance Creation--> Positioning
Adjustable between Organized & Free Form, this governs how strictly your players will remain within the zonal restrictions of your formation in the final third. Organized is good if your looking for a target man, or want to have a regimented system where the striker(s) anchor(s) the attack. Free Form is better if you like lots of lay-offs and interplay. This is similar to the Build Up- Positioning, but playing Free Form here is a bit less dangerous against a counter, as you're further up the field.
Defense--> Pressure
Adjustable from 1-100, this governs how far up the pitch your team will pressure the ball. Lower numbers have you sitting deep, inviting pressure; good for counter attacking strategies, but not so great if you like to be on the ball. Higher numbers have you hounding the defenders when they're playing it around the back; good for possession-hungry teams, but it wears out players & can pull you out of position if you're not careful.
Defense--> Aggression
Adjustable from 1-100, this governs the team's approach when attacking the ball carrier. Lower numbers focus on containment, while higher numbers pressurize the ball carrier more. This adjustment should be closely linked to your Pressure slider, as they work hand-in-hand.
Defense--> Team Width
Adjustable from 1-100, this governs how much your team will pull over to the side of the pitch that the ball is on. Lower numbers will leave you covering the while width of the pitch all the time, while higher numbers will have your fullbacks at the penalty spot when the ball isn't on their side. Again, this slider should be closely linked to you Pressure slider, and more than that it is dependent on how much you trust your players' marking abilities.
Defense--> Defender Line
Adjustable between Cover and Offside Trap, I really hope this is self-explanatory. It adjusts the shape & strategy of the back line between those 2 overarching ideals.
So that's custom tactics in a nutshell. There's no catch-all for what you should use, because every combination of players in different formations calls for different adjustments to these. If you're still using the Newcastle Default tactics, you should look at modifying them to more suit your current players & desired strategy, because that could be a big part of what's holding you back, so to speak.
If you have any question or whatnot, feel free to ask, I'm always willing to at least try to help out.
Hope this helps!