You can ignore it. There's one quest in the first act that gives you the option of paying someone for something or arm-wrestling, but you should be able to afford to pay. Apart from that, it's a stand-alone minigame.
There's little reward in arm-wrestling, other than money. If it's giving you trouble, I've found that turning down the sensitivity of the mouse (if you can do it easily) while wrestling really helps.