People throw that term "sexual harassment" around too loosely these days. When you do that, it really weakens others' claims to true sexual harassment. All that appears to have happened here is that he asked for your name, then you asked for his. Then he, in his own way, asked you out. You said no, and that seems to be the end of it. As S.W. pointed out below, it only becomes sexual harassment if he continued beyond that point. No means No, and apparently he knows that, since nothing else happened. It's pretty sad in this day and age that someone can't ask another for a date without being accused of sexual harassment. We, as a society, really need to lighten up when it comes to this kind of stuff.

As for the "blunt" comment, I don't see this as a big issue either. Lots of people smoke weed during their time off. That doesn't mean he was high at the time. After all, he did ask for you to join him "after he got done work". Suppose he offered to buy you a drink? Would that mean he's working his job under the influence?

Just because he apparently smokes weed doesn't mean he should be subjected to drug testing. If you really believe that, then you should probably being calling for everyone who drives a car to be drug tested. After all, there are plenty of motorists who smoke weed.