This is not only NOT sexual harassment, but it's not even Harassment.

SEXUAL HARASSMENT:
n. The making of unwanted and offensive sexual advances or of sexually offensive remarks or acts, especially by one in a superior or supervisory position or when acquiescence to such behavior is a condition of continued employment, promotion, or satisfactory evaluation.

HARASS:
1. to disturb persistently; torment, as with troubles or cares; bother continually; pester; persecute.
2. to trouble by repeated attacks, incursions, etc., as in war or hostilities; harry; raid.

No where did you indicate that he made any sexual advances, nor was he even persistent after you said "no".

This is a very serious allegation to make against someone. Despite a lack of evidence, just hearsay could be enough to cost this person his job. Is that what you want, considering the state of the economy?