In general, you'd say the opposite of a Mistress, would be called a "Lover" or "Secret Lover". Technically, speaking it would be "Master" or "Mister"
Depending on context, mistress can have several meanings and therefore several 'opposites' or male equivalents:
Mistress: a woman in position of authority, control, or ownership; male eqivalent = Master
Mistress: a woman skilled in a particular subject or activity; male equivalent = Master
Mistress: chiefly Brit. a female schoolteacher; male equivalent = Master
Mistress: archaic Mrs; opposite = Miss; male equivalent = Mister, Mr