You may not want to hear this, but I had a similar and painful experience with an African-American male doctor. My mother thought he was terrible too, but he was her PCP. Meanwhile I had seen a doctor before in an out-of-state college town (white and male) who was way less stressful. I told her “It is NOT supposed to be like this” and told her to find a new PCP. I told him flat out that this exam hurt. I never returned.
I had another awkward and frustrating experience with an Asian woman doctor. She was horrendous too and equally cold. And then an Asian male doctor was easy and gentle. Sometimes it really just comes down to the mannerisms of the doctor. I tried others and finally realized why people are so hung up on their doctors. I stuck with the Asian male doctor for years. He (along with my first gyno) proved male gynos can be just as comforting to deal with.
I judge on a case-by-case basis nowadays.