Is this considered a relationship?

I think in order to really call it a relationship, you both have to kinda talk about it and agree. But it does sound like you two are basically officially dating. I'd go ahead and talk to him about it. Just sit down with him one day when you have a bit of time together, and say something like "so about us...".