be brave

Do what you feel in your heart to be right - for you'll be criticized anyway. You'll be damned if you do, and damned if you don't.

Eleanor Roosevelt (1884 - 1962)

Very often, the general public would always have something to say about things that happen around them, including your acts.
The best thing to do is to have your own justification.
For Muslims, we have Al-Quran and As-Sunnah as our guidance. We can have our justifications based on these two.

The story of an old man, a child and a donkey illustrates this point very nicely. Whatever they do, people in the village that they pass by would always have their comments. Not one act is the right one.
It is up to them to decide which is the right one - for them. And, to ignore whatever remarks or comments from others. For they would themselves know that is the best possible thing to do, in their circumstances.