Landlord doesn't allow boyfriend?

I rent one room in two-bedroom apartment. The landlord lives in another room. I had a boyfriend coming over a few times in a week to stay in my room and my landlord went furious saying that he is not allowed to come, only on a very rare occasion and only with his permission. I offered to pay him additional 500$ a month to let my boyfriend come over and stay, but he refused. I do not want constantly ask for his permission cause the landlord is rarely home and it's hard to reach him. I do not understand what of a big deal it is it makes me so mad! The contract does not say anything about visitor policy, but we agreed on words before signing a contract that guests are OK. My bf is very quite, does not bother anyone and stays in my room. But why the landlord acting so nasty! I do not want to leave this place as I really like it and I do not want to stay over my boyfriend's place as it is not safe. What should I do! :(