Attraction is more than just physical. The physical is the obvious first step, to get the ball rolling. I will flatly acknowledge that my wife is not the best looking girl I ever dated. But she was the only one I would want to be married to. Looks alone get old fast. I dated one girl who was just plain hot, but after the third date, I couldn't take the conversations, or her lack of a decent sense of humor any longer. My friends thought I was an idiot for not sticking with it, but I probably would have ended up strangling her.
I also worked with a girl who was married to a guy much, much better looking than me. But she flirted like crazy. We had a lot of common interests, and she would always tell me that I was "intriguing", whatever that meant. She would call her husband "boring" on nearly a daily basis. I flirted back, a little, but never let it get to a point where I might be inviting a proposition. I wasn't married at the time, but she was. I don't know if she would have followed through, had I played along. Maybe I'm remembering it for more than it actually was, but in retrospect, it seemed like it would have been an easy score. Point is, I think she got bored with her husband, despite the fact that he was a better than average looking dude.
Gojazz79 is right, in that you should marry for love. If you want to keep your options open, don't get married. If you are married, and feel the need to stray, as difficult as it may be, you need to tell your spouse, and open the communication. Trying to salvage a marriage, or dissolve it, is easier (tactically) before someone cheats on the other.
