His refusal to see the children is another way of punishing you. He probably won't stick to it. I suspect that one of the reasons the marriage came to grief was because of his bullying.

It’s work that never ends, even with my children grown and on their own. During tax-filing season, my two jobs intersect. “Have you filed your taxes? Got any questions?” I ask my daughters.