Wow, I'm shocked reading your story. First, let me congratulate you on your bravery and courage. It's very difficult to talk about these things, even after so much time has passed. I find truth in your words; it's absurd that your grandfather was removed from the ministry because his wife divorced him and not because of the documented cases of CSA. When Jesus tells us that the truth will set us free, he's not only referring to his message but also to these kinds of situations that are so difficult and painful to talk about.
"But whoever causes one of these little ones who believe in me to sin, it would be better for him if a millstone were hung around his neck and he were drowned in the depths of the sea."
Matthew 18:6