How do I know for certian if I am being taken advantage of?

I am a 24 year old female with a disability. I was placed in this room and board/isted living thru kaiser ER. I have been here since dec 20 2010. The man who is the operator of the place and I started engaging in an affair at the end of january. (He is married no kids) I feel like I have been coerced into having with him. I know that with a married person is wrong, I don't need judgement just help. I tryed to break it off. I did for a little while, but I fell right back into it. I feel like he's using me. I am just so sad and lonely that I don't want to leave here. I get to see him every day because he is ihss for one of my housemates and he brings food which makes it difficult for me to have seperation. I told a few of the residents here about it, but the operator makes them think I am making up stories. I am an attractive person from what I have been told, I know it my self but I have such low self esteem, I cling to any sort of attention that I have been given. I feel like I have been severly psychologically manipulated. I don't know what to do! Please help me. I can't rely on my parents, and I don't want to burden the friends that I have.