Mutual Funds are typically traded and reported with 3 decimal places.Fractional shares of stocks can go up to 6 decimal places.
The important part about creating transactions involving fractional shares is to always use only- the exact number of shares and - the exact $-amount as reported by the brokerage to avoid rounding errors.
Let Quicken calculate the price per share to as many decimal places as it wants to, even if it differs from the price per share as reported by the brokerage (which usually is rounded or truncated).
For example, enter 0.123 shares for total amount = $12.34 and not $12.34 @ 100.32 per share (This example may not make much of a difference when you crunch the numbers, but try other quantities and amounts and you'll see what I mean.)