There are normal dips during times of stress and a decline as you age — just a couple of the many reasons you might not feel up for it as much. Like most things in the sexual realm, it is more complex than you think. What is defined as a low sex drive? The first struggle with what they perceive to be a lower sex drive. What causes low sex drive in men? Your sexual desire is influenced by a variety of physical, psychological, and personal factors, ranging from extra work stress to underlying health conditions.