how to avoid annoying ppl?

16m here.. this guy wont stop trying to get my attention and wont stop inviting me over and i keep saying im busy.

thing is, he's special needs so i dont want to straight up say "you're annoying leave me alone." although he has other friends so i think he'll be fine even if i stopped engaging with him.

if he sees me, he'll follow me until he gets a response, either physically (pulling my back pack, poking me, etc) or verbally (saying my name over and over and over again) and it really gets on my nerves. even in class, he distracts me by poking me and showing me irrelevant unfunny shit and i genuinely wanna lock in and get good grades.. wtf am i supposed to do.

(btw he's the reason im in a friendgroup so there's no complete way to drop him unless i drop the friendgroup)

TLDR; guy with special needs keeps on nagging me and i want to avoid him since i dont really wanna hurt his feelings