Im sorry youre going through a tough time. I read your story a few times and it seems like your gut is telling you theres something wrong here.
Listen to your gut.
It might hurt for a little while to let him go but you deserve someone who will adore you and not just come to you when its convenient for him.
You should definetly get it off your chest and tell him exactly how you feel and who you are and what you expect from a guy, then walk away.
If nothing else comes out of it at least you can walk away with dignity and respect for yourself.
Just my opinion. Anyway good luck babes