My husband has feelings for a co-worker?

A few months ago my husband admitted that he has developed feelings for a nurse at the hospital where he works. I was extremely hurt and angry at first, but it made me think about how I was treating him in our marriage. I had to admit that he has always been the affectionate one, and I do not give him the attention he obviously craves. We have a 2 year old daughter, and my husband works nights, he sleeps between 8-10 hours during the day. His admission of feelings was a huge wake up call to me. I have tried to be more attentive and loving, I have also asked that he stop talking to this person, which he said he would. He as admitted to emailing her, and I recently found out he has called her cell phone. I feel like I've driven him to this, he does not want out of the marriage. it makes me crazy that my husband has feelings for someone else, I feel like a failure. He tells me his role in our family is to be the provider and not to expect anything else from him. Should I give up?