Always - stacked, stoked, and ready to strike! Remember that the 4 rules of gun handling are just that - intended for handling, recognizing that human frailty can result in an unintended discharge due to error. Technically, a holstered gun, even pointed at another, is not being 'handled', and is an inert object until a force is applied to it allowing it to function as intended, IMO. You, and those around you, will be fine with a properly secured handgun.
And if it still bothers you - consider back pocket carry!!!