How do I mass compress / remove unused securities to decrease file size and quote update time?

boulderwise

Trying to get my new year copy to have a reasonable # of securities, as the current file has several thousand (really) so that the resulting close to empty file just contains a few transactions and all the accounts/registers info. No copy options allow my file size to get less than 100M. Not worried about register performance, but as I do several quote updates a day, having this many securities takes ~5min per update. This is untenable.

You used to be able to make a copy and throw out the securities during the copy process - can't find that option in subscription version.

I'm a multiple decade Quicken user - know most all of the tricks. But this new subscription version has me stumped on how to do the above.
Any and all comments and suggestions welcome.

Be well -

q_lurker

My first suggestion would be to trim down your watch list.  

You can find out exactly how many securities are in Quicken -- Help / Ctrl-Click on About Quicken.

For your one step update summary what does it say for :  _#_ of _#_ quotes and investing headlines updated?

There is now a feature to "archive investment transactions".  (I haven't used it.  no endorsement implied.)  See the help file.

boulderwise

thanks - yeah from help <cnt>about for new year file has
20 a/cs, 102MB, 424 categories, 15265 securities, only 155 transactions.
Can't get the quote window to pause when done - that also used to be an option - so unable to get # of # quotes updated.
As far as unchecking the watchlist box in the securities listing - not up for doing that manually for 15k securities.
Any other suggestions appreciated - Randy

boulderwise

Correction - update summary is available. 14765 of 15223 updated.
Took 5 minutes.
Brand new files w/ 50ish securities takes 15sec.

q_lurker

WOW.  Quicken used to publish file limits that included a soft limit of "2000 tradeable securities"..  But they were never able to offer a definition of tradeable security or identify what happened when you went over that limit.  I am rather surprised time to get  quotes downloaded is your major issue.   

Gotta ask:  Are you trading options such that those constitute a major portion of the 15000?  Some do use 'other' techniques to prevent those from blowing up the security count and confusion.

Watch list -- As I understand it, Quicken downloads quotes for securities based on a three fold check:

1. Has to have a valid ticker, AND
2. Has to be checked in the security list to Download quotes, AND
3. Has to be either owned in an investment account OR in the Watch list 

Choosing to believe you don't currently own 14000 different securities, I am believing it more likely you have 14000 securities in your watch list.  That is supported by my understanding that Quicken automatically adds manually created securities to the list (I'm not sure when new securities are added via brokerage downloads). 

I'll likewise acknowledge the futility of trimming that watch list one at a time for a 15000 list of securities.

That your 'new year' file has only 155 transactions in 20 accounts suggests you found the archive investments tool.  Apparently that did not decrease the number of securities. 

I'm not qualified to suggest ways to create a new file with fewer securities   Perhaps someone else can jump in to help.

@Chris_QPW , @K.O. (Win-Premier), or others -- Any thoughts on boulderwise's predicament?   

Chris_QPW

I'm not aware of any time that you could get rid of securities using copy or any other way other than manually deleting them.  Copy will allow you to delete investment transactions, but that doesn't imply that the security is deleted when no more transactions reference it, especially if that security is on the watch list.

On the subject price quotes.  They definitely can take up a lot of space and processing time.  It is one reason that in my file once I sell the last shares of a security I turn off downloading prices for it.  I have no idea why downloading them takes so much time, but I know that it isn't "fast".

For securities I have:

And the prices for them 11MB:


I can only imagine that the prices for thousands is really large.
You can open a QDF file with 7-Zip as I did (do not try to change things with it, it will mess it up)

You can get rid of prices by using the option in Validate & Repair to rebuild your prices.  The drawback I see on that though is with all those securities it is going to bring back a lot of that price history, but not quite as detailed (no daily after a month).

You can also use the above to extract out the parts and use a trick to get Quicken to rebuild the main QDF file (leaving out the .QPH file).  As you can see the current QDF file is actually a compressed file with all the old separate parts in it.

If you copy each of these parts out and rename them with "data file name" like it use to look like before Quicken 2010 consolidated them:
Current.QDF
Current.IDX
Current.QEL
...
Note that if you have attachments you will have a Attach folder.
For that one you leave its name as Attach, but inside it will be another folder that has to have its name change to the "data file name", Current in this case.

So bottom line as far as getting rid of the securities themselves about the only thing I can think of is to create a new data file.

BTW on the statement "No copy options allow my file size to get less than 100M."  The size of your data file has very little to do with the performance of downloading quotes.  It is the shear number of securities that you are downloading that is taking all the time.

It is a real shame that there isn't multiple select in the security list so that you could turn off getting prices for them in groups instead of individually.

q_lurker

@Chris_QPW -- Any thoughts on creating the new file as a QIF Export / Import?  He has in this 'new year' file only 155 transactions in 20 accounts. 

The main issue that would give me pause is importing 15000 securities via the QIF Export / Import.  But at least in my quick test, I find the securities imported do not get identified as Watch List.  They do all get the Download Quotes checked.  But my experience is that the Download quotes is just one criterion for Quicken actually asking for the quotes.

OR - Don't export the security list.  What I see on the import side is Quicken asking if it should create securities based on transactions being imported.  Only those securities are created.  They are created without ticker and without Download Quotes or Watch list checked.

Chris_QPW

If one is willing to start a new data file, and use a QIF import to get that started, then there is certainly things that can be done.

Note the biggest problems with importing a QIF file are the transfers.  Given only 155 transactions that might be easy to fix, or you can just go with my method here:
http://www.quicknperlwiz.com/changetransfers.html

Being on or off of the watch list hasn't anything to do with if a security is exported in the QIF file.  What definitely matters is if it has a symbol or not.  It doesn't include any that don't have a symbol.

There would be two approaches to trimming out securities using a QIF file.
The first is just don't select "Security list" when export (or maybe just in the import).
Note this selection is for both the securities and the prices.

The second approach is editing the QIF file before importing it.
The securities look like this:

!Type:Security

NMcdonald's Corp

SMCD

TStock

^

And prices look like this:

!Type:Prices

"YHOO",15.925,"10/18'10"

^

BTW importing security prices in the QIF form is EXTEREMLY slow.  If you were to import thousands of securities/prices this way you might be looking at hours if not days for it to complete.

jmaino

I find myself continually coming back and searching the forum, seeing new ppl come up w/ this same question and looking for solutions.
I too have a very large #of securities w/ Max Ref of 14073. The majority of these are custom securities from Options trading (since Ticker doesn't correlate to anything Q knows about).

This thread hits upon a few things I've tried and have also "wished for" such as:
- how can I tell Quicken that it should never add a custom security to a watch list? Semantics don't really make sense since it cannot figure out anything automatically about these anyway - thus there's nothing to watch
- some mechanism to turn off Watch List checkbox in bulk. I'd happily turn off everything in my collection of securities and turn on what I want .... I never use Quicken's watchlist anyway
- how can we possibly get improvements such that large numbers of securities are no longer considered supported - but you'll pay if you try!. I never quite understood why there's any artificial limit unless there's some acknowledgement that there's some order(N^^2) issue going on in the code.

For my datafile, it's gotten to the point where if I add a "REAL" security, it can take 15mins for Q to retrieve the info from the Security, populate my security list, and return a prompt to me. Which is kind of funny because I can add a custom security in seconds and w/o any lapse on Q's part.

I've been thru the debugging tips/tricks/etc and haven't really found anything to help me improve the performance of Q. I have transfers between accts in my Investment accts, so I think this runs me afoul of the idea if using a QIF to create something new and more friendly to Q.

Thanks for guidance and ideas in the past - but I'm still looking to see this improved.

Chris_QPW

jmaino said:

I have transfers between accts in my Investment accts, so I think this runs me afoul of the idea if using a QIF to create something new and more friendly to Q.

That is why I created ChangeTransfers.  As long as they are linked transfers they are never going to import correctly.  But change them to normal categories and then can work fine.  Accounting wise (double book accounting) it works out fine.  One category adds, the second subtracts the total is "neutral".

As for the rest, well Quicken was never designed for "active traders", I don't think that is ever going to change.

Steve Long Time User

jmaino - I have many of those issues as well. Seems like it would be simple to add a preference to prevent placing Options on the Watchlist. After all, it already goes into the account, so there's no need to put it on the watch list. So, for every trade (several a day) I have to delete the options from the watch list.

Many frequently traded options expire within a year (sometimes in a day). I've considered using a separate Quicken file for trading, and just manually updating the main Quicken file as required (i.e. monthly). This can be a calendar year only file, starting a new file each new year. This would speed up both data bases, but you lose some tracking information for trades in the main data base as you would probably only make "balance adjustments" as needed.

If quicken had a way for two files to be referenced by one another, it might make this a viable alternative for high frequency trading. It might even make sense to have each investment account as a separate Quicken file, linked to the main Quicken file as in a hierarchy of Investment accounts. They would need to be dynamically attached/unattached on some periodic time frame, or on demand. This might be accomplished with existing .QIF import/export features.

Quicken actually does a reasonable job with options trading - it's just data entry intensive if your broker doesn't support Quicken. In addition, the data entry for new options is suboptimal, but it works.

So, these ideas I've mentioned appear to be the only options with the current Quicken architecture (forget the simple investing thing - bad idea).

Now, if we can just get better report functionality and the tracking of CASH...we might actually have a reasonable Investment accounting tool.

Jim_Harman

If you would like to see improvements to the Watch List please review and vote on these Ideas
https://community.quicken.com/discussion/7365523/when-adding-a-new-security-make-add-to-watch-list-optional-preference-edited

and this one
https://community.quicken.com/discussion/comment/20101153#Comment_20101153