SARMs are going to be much safer on the prostate as they do not have the androgenic properties. That being said, be careful if you have preexisting conditions. Maybe check with your doctor if you want to be sure.
TRT is a lifelong commitment, not something you quit. If you legit need TRT, quitting would set you back to all the side effects you would have experienced with low T. Isn't that why you went on TRT to begin with?