Might as well chime in, despite my lack of composed thoughts on the topic at hand:
I am not a pacifist, but my grandparents were. They were Peace Corps members. Grandpa was born in India to missionaries but largely exchanged evangelism for more secular overseas approaches.
My grandparents on the other side of the family allegedly met on the way to their first KKK meeting. According to family lore, they ditched the meeting in favor of a romantic date and ended up devoting their time to each other rather than the organization. I like to believe that's true, but then again I have a right to suspect some bias from the storytellers, my parents.
We contain multitudes. I like to acknowledge the disparity in my own family tree because it reminds me, symbolically, of the chaos we each have in our own hearts and lives -- that is, the breathing, balancing push-and-pull of order and disorder, creative and destructive impulses.
I do not identify as a pacifist, per se, but at the same time I do want to be clear that I value non-violence to the extent that non-violence is a viable solution.
That said, I absolutely believe in the necessity of (violent) self-defense. However, my philosophy of self-defense is that it is ideally 99% preparation, 1% application. Be strong enough that nobody wants to fuck with you; be ready enough so that if they do, you're able to protect yourself and the people and values you care for.
I want to be clear that I don't mean to conflate non-violence with a lack of application of values; I am not endorsing fence-sitting or endgame neutrality.
On the other hand, I prefer to explore non-violent solutions before resorting to self-defense and/or violent retaliation when possible and appropriate. A personal example:
I was a big kid in school; I hit my growth spurts early. When I was about 13 years old, during a school lunch break, I saw a group of five or so bullies my size physically harassing a smaller, younger kid. I stood up to the bullies and told them to lay off the kid. They reacted as you'd probably expect, by turning their attention onto me.
I had achieved my desired result: the bullied kid got his respite and was able to sneak away. Having no further plan of action, I simply turned and walked away (back toward the schoolhouse, of course.) I walked slowly, it was not a retreat. I was followed by the bullies and their heckling, and a crowd of spectators was beginning to form.
Then came the sucker-punches to the back of the head and torso. They were firm, but not hard punches. Exploratory punches. Provocative punches. The punches were questions: What is this nerd doing? Why isn't he fighting back? Would we be able to take him down if he did?
After the first few strikes I turned, made eye contact, and told them firmly to stop. The reply was laughs, jeers, and a few more halfhearted jabs, but by then the wind was out of their sails. The assault wasn't fun anymore. Absorbing their assault and shrugging it off denied it of its intended purpose: to dominate and demonstrate power.
By no means do I intend to imply that calling out violence and walking away is the ideal solution. This is merely a personal anecdote with its own contained context to illustrate the point that non-violent action can have a positive effect and can even help reach satisfactory resolutions.
A few caveats: these positive effects do not always scale well. Standing up to a specific group of bullies is waging a very different type of ideological warfare in contrast to standing up to the idea of Nazism in general. Also, had the blows I received been less exploratory and more earnest, then yes, I suspect I would have done more to defend myself violently. I wasn't there to change minds; I was there to keep a kid from getting bodied and bloodied, and in that context I was successful.
I guess to wit I would say, in the larger context of the potential for living in the shadow of fascist violence: have your (metaphorical?) sock of pennies ready, know how to use it, know when to use it, and do your best to keep yourself, your loved ones, and your ideals healthy and intact.
And if one is comfortable enough on the fence to watch fascists take one's neighbors away, I would venture that one's ideals aren't as healthy as they could be to begin with, and that perhaps one should reexamine and nurture those ideals as a component of their self-defense development.