I think what the OP reasoning is that because you are picking the card up, and then placing it back down it should lose all buffs, similar to how decoy removes all green strength.
and I disagree with that. How it works now is fine, a card loses all buffs when returned to the hand. Not when moving rows, and that IMO is fine.