The program does not duplicate the transactions. Note @Jim_Harman 's response.
Your issue is not how to transfer shares. You appear to be doing that correctly. Your issue seems to be misunderstanding what Amount Invested is. Amount Invested is NOT cost basis.
Suppose you buy $1,000 of Apple. A year later you sell that holding for $1,200. You had an Amount Invested of $1,000 and got a return of $200. Six month later you buy another $1,000 of Apple. You later sell half that holding for $750. Your remaining shares continue to appreciate to $900. Quicken is telling you that across all time you invested $2,000 in Apple. Your proceeds from the sales are $1,200 and $750 for a total of $1,950. Your current holdings are worth $900. Your $2,000 Amount Invested has produced for you $2,850 in value. Amount Invested is NOT a measure limited to current holdings.
Now the catch is that Quicken does not identify that a transfer is a transfer. It gets recorded as a Remove and an Add. For Amount Invested purposes, that new Add is a new.investment.
That definition/application of Amount Invested has been a major bug-a-boo for a lot of users for many years. I have seen no effort to alter it. What they did add (also many years ago) was Average Annual Return which does not rely on Amount Invested and is a much more accurate and versatile performance measure..