There's no reason why it wouldn't be 'right' for you two to be friends, unless he's mistreating you or something. He will get to know you better then. If you care about the friendship, stay friends and don't worry about what people say.
He'll love it. If you do, then you're doing yourself a disservice and causing harm toward your mental hygiene. In addition to certain clubs, you may find new friends who are studying the same major as you in high school or college.