I think the biggest problem stems from games giving you the impression of HAVING to do the minigames. Like you would loose out on XP or rewards that could somehow hinder yourself. The next thing would be the (by me) always dreaded achievements, the proverbial carrot on a stick devs nowadays like to dangle infront of their players bc they need to tick off that thing off a list and get a point somewhere that they then can show off to prove sthg to ..... who ? I do not know, I never "check" the achievements of friends and then laugh at them for NOT doing this and that. I also do not need it for myself, bc I KNOW WHAT I DID.
Roleplaying itself (imo) gets hindered by that. Roleplaying for me is discovering things for myself, giving myself a set of restrictions I think would fit the char I wanna play, not the devs giving me a list of things "i should do" or "could do".
I personally played once through the whole campaing of TW1 drunken. I spent ALL my money on booze and experienced EVERY cutscene in that mind-boggling blur, haze and staggering when you got alcohol intoxication. I played once as the pious witcher, not having any sex, doing any drinking or gambling, sometimes I outright refused to do certain quests. Yes, I lost out on XP but that was not the point. The point was that I created a certain set of rules for myself and sticked to them and that gave me satisfaction.
Which leads back to the original post - playing dice etc. adds flavor to the world. It does not break the story because it is an optional activity that everybody can ignore. In Act3 in TW2 I did not play any dice at Loc Muine and did not do any armwrestling bc I felt that my char would not give shit, he wanted to save Triss/get revenge on Phillipa/whatever. I even skipped some of the sidequests bc I felt MY Geralt would not stop to solve some old Vran mysteries when his loved ones/his country/vergen/a dragon are in danger. I still two-shotted the boss

I played TW2 without any crafting, only using the gear/swords the dev place along the way. I played both parts in power-XP mode, trying to reach the highest level possible and crafting ALL the gear.
But sometimes, you wanna have a break from all the witching and just having the option if you want to - even if it is just a minigame - is good and should be kept in the game. I dont care if it is 2 out of 3 like TW1 or just one round, just the fact that it is there is enough for me. So somebody can have the option of his Geralt being dice-addicted or fistfight-addicted.