Yeah, I'm worried about that too. My investigator seemed like a cool guy, but asking to change a reference can't be a good look. What my friend told me was along the lines of 'i'm not super comfortable and talking with my wife she's worried about if something ever happens my name will be on your application.' I only wish that he'd just told me 'no' in the first place and I wouldn't be in this mess. I left a message for my IO asking about it, so either way the die is cast.
I guess it depends on how good of a friend he is, but I'd explain it to him that since he already agreed, his name is already on the application and he is now on the hook to explain to the investigating officer why he is not super comfortable. As in, why he doesn't think you should get a license, not why he doesn't want his name associated with your application. Well actually both of those represent a problem. By not being super comfortable, he is implying he doesn't think you are responsible. I mean does he drive a car? Can anyone be super comfortable that they won't get into an accident any time they take their car out? If we go by that, then everyone would just stay locked up in their homes all day. I'd also explain to him again that these are character references, simply stating they know you and based on what they know, they don't think you aren't a threat to yourself or other people. It's not like he's co signing a loan.
The ship has already sailed as far as his name on your application. At this point it's as simple as either he doesn't think you are an upstanding citizen and are a threat to yourself or other people or you are not.
Failing all that, you could say that his wife is anti-gun and was giving him a hard time about it.