I confirm, they DO have a strong cast, even the high quality ones. I'm using them too rarely, and this is a fault of mine, but I've used them enough to tell you that they need some work to get a good colour picture (as you said, B&W is easier).
First I tried a medium quality stopper, and was appalled at seeing the strong colour cast - so I thought it made sense to spend more money, and bought a Lee Big Stopper (the smaller, for mirrorless, but same glass). It still has a strong cast, as you'll see by the images I'm going to attach (curiously, I did something with the Big Stopper just the past Saturday).
With Lightroom, it was a big problem: while most of the cast can be corrected with white balance, it's not enough. Lightroom allows colour channel correction, but it's fixed. With Capture One... it's much easier because you can pick the colours that need correction and selectively fix their hue.
Another approach would be to shoot a reference with the ColorChecker Passport and create a specific profile for the shot. This works, but it requires more work, the most annoying stuff being that you need a clamp to keep the color target in a fixed position.
So, my current approach is: shoot without the filter, to get a visual reference for what the final rendering should be; first apply strong white balance correction to the photo with the filter, and then selectively adjust the remaining colours that aren't good yet. For my limited experience, this is the fastest way.
I'm going to attach: a colour reference (not exactly from the same perspective, but it works), the post-processed final photo, and the unprocessed photo straight from the camera.
Sony a6300, Sony a6000, Sony NEX-6, Sony E 10-18mm F4 OSS, Sony Zeiss Vario-Tessar T* E 16-70mm F4 ZA OSS, Sony FE 70-200mm F4 G OSS, Sigma 150-600mm ƒ/5-6.3 DG OS HSM Contemporary, Samyang 12mm ƒ/2, Sigma 30mm F2.8 DN | A, Meyer Gorlitz Trioplan 100mm ƒ/2.8, Samyang 8mm ƒ/3.5 fish-eye II | Zenit Helios 44-2 58mm ƒ/2
Plus some legacy Nikkor lenses.