New Quicken File

Pilotbum

I've created 4 new files using the qif import and am somewhat appalled at what a truly awful job it does with the transfers. One of them using your tool recategorizing the transfers, that one turned out as advertised. The rest of them are all different. It just blows my mind that some of the transfers are just fine and others are a disaster. Some accounts are perfect and others are a disaster. 

Chris_QPW

Pilotbum said:

IF I do get a new file with a qif import to work out for me, is it truly a new file or does the old data still cause problems in a new file the way quicken support has told me? 

It is a new file, and it will leave the old problems behind.  Note I'm not saying that other problems, even with the import can't creep in, but you are certainly not bringing the old problems along.  If the Quicken support person said that they don't understand what is going on.  BTW you can try your entry/validate operation and make sure that isn't happening in the new data file.

When my corruption problem happened, not only did I do this for the period of them where they had the data file, I continued to keep both files up to date for a few months (just in case).  Never saw any problems with the new file.

Note you can reconnect transfers these days, but there might be a few gotchas with it if you try to do it in bulk, at least that is what I remember from trying to do it in the past.  So it might not be really worth it if you think about how much your really jump between the two accounts and of course new transactions will be linked transfers so it doesn't apply to them.  On the other hand you might want to clean them up just so that you don't have filter the reporting of those categories when you don't want these "transfers" showing.

Certainly make a backup before doing any of the build change.

On of the problems you will face is that for every transaction it will prompt you asking if if it is the existing transaction.

The process is straightforward, user find and replace (note that it for all accounts), to find the "transfer category and then replace it with the [Account]/transfer like this:

And get ready hit the y key, as in this case 478 times (you can also just hold the key down).

Chris_QPW

Some updates I notice for Quicken subscription.  Quicken subscription took away the processing of QIF files that was in Quicken 2017 and below.  As such the transactions don't go to the Downloaded Transactions tab/red flags they go directly into the register.  This has advantage that you don't have to go to every account and accept the transactions into the register.  But it also means that if you don't use my program to change transfers into regular categories or the ""Special handling for transfers" option on the QIF import then you will most likely get duplicates (there isn't any "matching of existing transactions processing") of the transfers or at least all the above prompting on every transfer.

Chris_QPW

One more thing you might find useful in creating the new file.  Before doing the import go into the category list and delete them all (not the hidden/built-in ones).  That way you will only have your categories instead of the "new default" ones.

Chris_QPW

OK I found the problem I was thinking of with reconnecting transfers.

The problem is that the "transfer detection" doesn't work for transfers that are in a split transaction.

The end result is that you get the transfer duplicated in each of the accounts.

Pilotbum

Chris_QPW said:

OK I found the problem I was thinking of with reconnecting transfers.

The problem is that the "transfer detection" doesn't work for transfers that are in a split transaction.

The end result is that you get the transfer duplicated in each of the accounts.

You're referring to importing the qif, right? I think that is what I was finding also. 

I've been playing with the qif's, importing, exporting, your program and trying to decide if I want to go through the effort of it all or just give in and start a new file. I just realized that the qif export also does not include attachments (you may have said that). I'm assuming that is why my file size went from ~4.4GB to 49.9KB. I'll have to keep the corrupt file around for the attachments if I go through all this. 

Chris_QPW

Pilotbum said:

Chris_QPW said:

OK I found the problem I was thinking of with reconnecting transfers.

The problem is that the "transfer detection" doesn't work for transfers that are in a split transaction.

The end result is that you get the transfer duplicated in each of the accounts.

You're referring to importing the qif, right? I think that is what I was finding also. 

Not really.  I was talking about changing transfers back to linked transfers if one used my program to change the transfers into regular categories to get a clean import.

Say you have a transfer from "Checking" to "Savings".  After such a changing of the transfers and import you would have a category of: xfrSavings in the transaction "Checking" account and xfrChecking in the transaction in the "Savings" account.

If you renamed the category xfrSavings to [Savings] in the "Checking" account, Quicken will prompt you asking if this is the same transaction as the one in the "Savings" account. If you say yes it will link the two together just like you would want (if you say no you will duplicate the transaction in the "Savings" account.

Now say that one side of that transfer is in a split transaction.  If that is the case Quicken will not prompt you or make any attempt to relink these as a transfer.  So if you do this it will create a duplicate transaction in the other account (it is like you said no to the above prompt).

Pilotbum said:

I just realized that the qif export also does not include attachments (you may have said that). I'm assuming that is why my file size went from ~4.4GB to 49.9KB. I'll have to keep the corrupt file around for the attachments if I go through all this. 

That is one of the limitations of a QIF import.  There isn't any syntax in the QIF format for attachments.  Just like there isn't any syntax for "security lots", so all sells will be treated as FIFO.  That is why there is that link to the discussion back on this thread about all the limitations.

But on the bright side I think you get why the idea of whatever is corrupting your data file isn't going to be transferred to the new data file by QIF.

BTW in may case importing by QIF surfaced a couple of problems.  One was in a long closed employee stock options account the security weren't "sold to zero".  Probably caused by some corrupt sell that didn't get transferred.  Doing a remove shares fixed that.  And the other was a long closed brokerage account that still had cash in it.  A balance adjustment transaction fixed that on.

There are many possible problems.  Anyone that has linked checking accounts with their investment account are most likely going to have problems.  Roth IRA accounts need to be created before hand because the QIF format has no syntax for them so if the account is create by the import you will get a brokerage account, and unlike being able to take a brokerage account and make it tax free and then make it a traditional IRA or even go to a 401K (I think that works).  There isn't any path to a Roth IRA.  The account has to be created that way.  Note that accounts that are already created with the exact same name will just have the transaction imported into them.

Chris_QPW

Sort of a side note on QIF importing.  Last night I was wondering why I didn't ever just "remove the to side of the transfer" like I said Quicken's "special handling of transfers" should have done.  I looked at it this morning and I realized that I had already written that program and had to reject it because Quicken messes even that up, at least in the most recent versions of Quicken.

Even though I deselected "special handling of transfers" in the QIF Quicken still acts like it is on and breaks up the spits and generally messes things up.

What's more I just tried with straight export the QIF and then import into a new file with the "use special handling for transfers" on and it mucks everything up, not even close.  Interesting enough the overall total is about the same, but individual accounts are wildly wrong.

So the only method I have found that works (within the limitations of the QIF format) is the changing of the transfers to regular categories so that this is totally taken out of Quicken's hands.

Pilotbum

Now say that one side of that transfer is in a split transaction.  If that is the case Quicken will not prompt you or make any attempt to relink these as a transfer.  So if you do this it will create a duplicate transaction in the other account (it is like you said no to the above prompt).

I see what you're saying... I've made a couple of test files to start trying things out to see if I want to go through with this... Still on the fence but leaning toward it's not worth trying to recategorize. The split transactions aren't always as easy pick out and in 28 years of data there are going to be thousands of them. For example, I tried doing the Find/Replace with the category "xfrUSAA Classic Checking" and there only appeared to be 1 split among the 2,700+ transactions. So I went ahead did them all except that one split and I was grossly incorrect; there were many (most notably my mortgage payments) that were originally splits FROM the checking account. They weren't visible as split transactions in the Find/Replace because they were split on the checking account side of the transaction to the [mortgage account], [escrow account] & "Mortgage Interest" categories. In the checking account split would have been "xfrMortgage_Account", "xfrEscrow_Acount" & "Mortgage Interest". So the "xfrUSAA Classic Checking" only appeared in the Mortgage and Escrow accounts but not as splits and the "xfrUSAA Classic Checking" categories were on the side without the split. 

[I put a stapler on top of the enter key on the 10-key on the keyboard to keep hitting "yes" to the 2,700 transfers I started with for one account. when it was done it just sat there and blinked at me for another 5-10 minutes while all the repetitive "enter" presses worked themselves out.]

If I do go with a file with the qif imported data I think I'll do like you did and maintain two files through the end of the year, the cleaned and sanitized version and the corrupted version. If the sanitized version works out I'll stick with it, if not, I'll go back to plan A. Plan A being just start a new fresh file. Plan B being a new file with the import of the qif data after using your program but WITHOUT recategorizing the transfers. Plan C being attempting to recategorize the transfers (so far Plan C has been a complete bust).

Refresh my memory, you said your data file has all your old transactions in it with the "xfrAccount_Name" categories in lieu of the actual transfers, right? And that has worked out fine for you?

Pilotbum

Well... another update... I think my decision has been made for me. It'll be Plan A, an entirely new data file with no imported qif data. After doing some trial runs exporting, importing and copying I realized I had not yet attempted to validate the file with the qif imported data. So I started another new file, imported the XfrsAsCategories.qif file, Copied it to another new file DataFileCpy and validated and I got this:

No problem.. I expected some errors. So I validated again and got this...

Ok... still fixing a few things... So I validated again...

To make sure it couldn't fix it I tried one more validation with the exact same results, "6 scheduled transactions corrected"

Then I tried a Super Validate. "6 scheduled transactions corrected"
Then another Validate. "6 scheduled transactions corrected"

6 attempted validations one of which was a Super Validate and I cannot fix the errors. At this point it seems rather clear these problems are not fixable. Would you agree? 

I guess the last thing I may still try is to export qif and import into a new file the data from a backup that supposedly had no errors after a validation and see what that produces. However, when I tried building off that backup a day or two ago it still endd up with errors so my confidence is not that high. 

Pilotbum

Just realized those summary shots are not readable. The first had quite a few errors but none of them were correcting scheduled transactions. 

22 lines of:

Fixed damaged category.  Please check your category list by going to Tools>Category List.

And:

Minor problems with Tags corrected

Tags Corrected

Then validations 2-4, #5 was a Super Validation and #6 was a validation all gave me:

6 scheduled transactions corrected

Chris_QPW

The QIF import has no syntax for scheduled transactions.  Whatever is causing that it isn't the QIF file itself.  Have you checked and seen if there are any reminders?  And if so deleted them?  It might be that Quicken created for some reason.

Pilotbum

By reminders you mean scheduled bills and transactions? There aren't any that I had programmed in but there is a list of the "schedule these?" on the All Bills & Deposits tab of the Bill and Income Reminders window.

*EDIT* I just went in and deleted the "Schedule These?" and validated. Still 6 scheduled transactions corrected. 

Chris_QPW

"scheduled transactions" is the old name for "reminders".

The "schedule these" are just suggestions not actual reminders so I wouldn't think they are the problem.  There is an option in that same Windows for turning them off though.

I'm running my QIF through another round to see if I see the same thing you do on a copy.  On the validate, I also think that the "categories" being corrected is probably "right".  I have no idea where the problem "scheduled transactions"/reminders got introduced, but it doesn't come from your old data file.

Unfortunately Quicken Inc tries to give the impression that "corrupted files" come from users doing the wrong thing, and I'm sure there is some of that.  But from what I have seen there are plenty of bugs in Quicken that cause such problems.

Pilotbum

I just restored the data file that did not show any errors from 10/11, exported the qif, imported it into a brand new data file, copied the data file and ran a validate on the copy. The first validation was EXACTLY the same as the validation of the previous file with 22 fixed categories and the tags. Then I ran 2 more validations and on the second file from the restored backup I kept getting "5 scheduled transactions corrected"

I verified there are NO reminders/scheduled transactions in either file. 

In my original data file that keeps returning "3 scheduled transactions corrected" I once tried to delete all my scheduled transactions and rebuild them only to keep getting the same, "3 scheduled transactions corrected." I then assumed that there was some syntax in the scheduled transactions that had been previously entered that still marked it as a scheduled transaction(?). Now I'm starting to think based on what you're saying this may be a bug in quicken? This all started after the latest release that installed on my computer on the 10/06. 

It may be worth noting I am presently on R29.22 Build 27.1.29.22 (Windows version)

*EDIT* Just for fun, I just opened my original data file with the most current data and I'm still getting the same "3 scheduled transactions corrected" message. But other than that the file works fine (aside from the occasional sluggishness caused by a 4.5GB data file) and there are no other errors.

Chris_QPW

With my test I had the similar fixing of the categories/tags and a couple investment transactions on the first run of validate, but not after that.  And no problems with scheduled transactions/reminders.

One off the wall idea about the problem.  There are "Memorized Payees" in the QIF file.  Maybe the problems come from them.

If you edit the QIF file with a text editor and search for:
!Type:Memorized

And remove that line and all the lines till the next ^ at the beginning of the line, (!Type:Prices, at least in my file, and would image the same in all Quicken exported files)

And then import that file in a new data file that would prove it one way or another.

Pilotbum

Would I get the same result if I just created another qif without memorized payees and imported that? 

Pilotbum

So I didn't go about it the way you suggested rather I created another qif without memorized payees and imported that (there were A LOT of memorized payees in that txt file). Then I created a new file and imported, copied and validated the copy with the same results (6 scheduled transactions corrected). Then I created another new file and imported the same qif but only checked the transactions and accounts to import yet the categories and tags imported as well (I'm assuming since they were in the transactions?) and with all the other options in the qif import unchecked, copied, validated and same results--6 scheduled transactions corrected.

Chris_QPW

Pilotbum said:

Would I get the same result if I just created another qif without memorized payees and imported that? 

Yes, as you have stated in the next comment this would work the same.

Pilotbum said:

Then I created another new file and imported the same qif but only checked the transactions and accounts to import yet the categories and tags imported as well (I'm assuming since they were in the transactions?) 

That is correct, but with the disadvantage that the tax line income/expense, tax line information, and description will not be known for the created categories.

Pilotbum said:

with all the other options in the qif import unchecked, copied, validated and same results--6 scheduled transactions corrected.

Well it was a wild shot.  I have no idea why Quicken is doing that in your data file.

I guess to get a really clean data file you will have to go with Plan A.  Not that I trust it to stay that way or mine or anyone else's for that matter at this point.

Pilotbum

Well... hopefully it's a bug Quicken figures out at some point and fixes. When the beginning of Plan A starts I'll maintain the old file for a while to see if the problem "goes away"

Pilotbum

Well... I've decided to make one more attempt at fixing this file from a restored backup. After I restored the backup I ran a validate--no errors. Then I deleted all scheduled transactions and there were 4 scheduled transactions "repaired." No matter how many validations I do the same errors continue to be in the validation report. Even after a super validation. At that point I'm thinking regardless of what I do I will not be able to get these validation errors to go away even though the file still works fine. 

What is peculiar is depending on the file version I get the errors in I will get a different number of scheduled transactions that were "repaired" even thought the validation will always give me that exact same error with the same number of repaired transactions. 

The only other thing worth noting is I think this all started after the last Quicken update. 

Pilotbum

@Chris_QPW (and whomever else may have followed this thread and be interested...) As I has said before I'd planned on babying my current data file until the end of the year and then starting a new one (as much as I didn't want to). I would frequently continue to run file validations and late November the consistent error of "3 Scheduled transactions corrected" went down to "2 Scheduled transactions corrected" consistently. That gave me hope. Then a couple of days ago it went down to "1 Scheduled transaction corrected" consistently. Today the validation went down to No Errors in any part of the Validation. I ran a validation several times to make sure it was consistent and so far it is. The only thing I noticed was there was an update recently (can't remember what day) but I believe after that was when the errors started disappearing. Given all the steps @Chris_QPW suggested and worked through with me that did not make sense I can only assume this was something that was fixed in the software (even though none of the release notes suggested that). 

Nonetheless, it appears the problem is solved and I can keep my quicken file intact. Thanks again for all your help.

Chris_QPW

@Pilotbum That's great news!

Pilotbum

@Chris_QPW when you were helping me we did a qif export/import to a new file and that eliminated all account & transaction attachments. Is there any other way to delete attachments in a bulk/batch delete without doing the qif export/import?

My quicken file is almost 4.5GB and I've realized that's mostly the attachments. At first I thought the attachment feature was neat but now I find it more cumbersome since my file is so large. I'm hoping that getting rid of the attachments will improve the performance. Sometimes the program seems to "stall" for lack of better words.

Chris_QPW

Yes there is.

File -> File Operations -> Copy..., deselect "Include attachments".

The copy will be stripped of attachments.

Chris_QPW

P.S. Attachments tend not to affect performance, but with that many it might.

Pilotbum

I'll let you know... I've decided to start 2021 with a copy of the file without attachments just to keep the filing systems "clean." Making the copy without attachments reduced the size by 1.5 GB (down to about 3GB from 4.5). I was expecting more as when you were helping me before and I did the qif export/import the file size with all the transactions was less than 1Gb. 

Pilotbum

So... time to re-visit this topic as I have some new information. This pesky error returned but this time I was eventually able to correct it and identify the source of the error... I think... It took me most of the day of trial & error and 10-15 minute validations so this will get long...

Since the last time I finally got this same issue worked out I have been doing far more validations of my data file.

In January I added a new bill reminder/scheduled transaction that was a temporary bill to only be paid for a few months. I added it and made the first payment on 01/25/2021. All was good and all validations worked fine with no errors--and there were many.

This bill was shorter lived than I'd anticipated so the scheduled transaction was deleted. I then ran a validation since the errors I was previously having were with scheduled transactions. The validation report returned the result with "1 scheduled transaction corrected." Like before, no matter how many validations or super validations I'd run the error would not go away.

I then went back and deleted THE ONE SINGLE entry of that scheduled transaction from the register and ran a validation--NO ERRORS!

I then went back and re-entered the transaction into the register and because it was in the memorized payee list it was all entered exactly as before (but it was not entered from a "scheduled transaction"), per se. Then I ran another validation that returned the "1 scheduled transaction corrected" error that would not go away.

So I deleted the manually entered transaction (from the memorized payee list) AND the memorized transaction from list of memorized payees and ran another validation--NO ERRORS.

Now I went in and re-entered the same transaction manually (no scheduled transactions to choose from and no memorized transactions/payees to choose from) but I entered it EXACTLY as it was in the register before--capitalization, punctuation the same, the splits were the same but in a different order and ran another validation--"1 scheduled transaction corrected."

I deleted that transaction and ran another validation--NO ERRORS. Ran another validation and still NO ERRORS.

Now my approach is to rebuild this transaction that is giving me fits one step at a time also making backups after each step that returns the validation results of NO ERRORS.

1. enter the transaction Amount only using today's date rather than the original transaction date of 01/25/2021. The rest of the fields of the transaction were left blank. Validate. NO ERRORS. Create Backup.
   
2. Change the Date ONLY to the correct date of the transaction, all other fields still blank--Validate. NO ERRORS. Create Backup.
3. Itemize the split categories only. Validate. NO ERRORS. Create Backup.
   
4. Add the memo exactly as it was entered before. Validate. NO ERRORS. Create Backup.
   
5. Add Check #. Validate. NO ERRORS. Create Backup.
   
6. Add the Payee exactly as it was entered before. Validate. 1 Scheduled transaction corrected. Validate again for the heck of it. 1 Scheduled transaction corrected.
   
7. Restore previous backup. Validate. NO ERRORS.
   
8. Added Payee but with different punctuation removed parentheses and added a hyphen so it was not exactly the same but words were the same. i.e., 'PayeeName (description)' to 'PayeeName - description'. Validate. 1 Scheduled Transaction corrected. Validate 2 more times with 1 scheduled transaction corrected.
9. Restored previous backup again (from step 5). Validate. NO ERRORS.
   
10. Add the Payee but completely different and random. Validate. 1 scheduled transaction corrected.
11. Deleted Payee. Validate. NO ERRORS. Validate again. NO ERRORS.
    
12. Change the transaction's Status from 'Uncleared' to 'Reconciled' (left payee blank). Validate. NO ERRORS. Create Backup.
    
13. Add the attachments. Validate. NO ERRORS. Create Backup.
    
14. Added another random Payee in the payee field. Validate. 1 Scheduled transaction corrected.
15. Removed random Payee. Validated twice. NO ERRORS either time.
    
16. The final steps I took to test for errors was to enter another (second) transaction for the same amount to validate it. 1 scheduled transaction corrected.
    
17. removed the Payee. All other fields remained. Validate. NO ERRORS.
18. Added a completely different and random transaction from the memorized transaction list. Validate. NO ERRORS.
    
19. Added the transaction causing the problems to a credit card account. Validated. 1 scheduled transaction corrected. When the transaction was deleted and subsequent validation--NO ERRORS.
    

The first 18 steps above were to one account (checking). The 19th was a test transaction to a different account (credit card).

In hindsight I believe it was the same such actions that caused the 'X scheduled transactions corrected' error messages after a validation last year--I'd removed a scheduled transaction. That time it was a paycheck/income reminder that I deleted. This time it was a bill that was the scheduled transaction that was deleted. Once deleted one or more of the past entered transactions seems to be what is causing the error message.

In this recent instance I thought I would be able to narrow it down and I did. But what really puzzled me was,

1. I completely deleted the scheduled entry including it's matching bank downloaded fields, the memorized transactions and re-entered it. I was expecting at that point for the errors to completely go away. That wasn't what happened.
   
2. It is only when I added a payee to the transaction that an error occurs. If the payee is removed there is no error. It does not make a difference what the payee is.
   
3. It is only this transaction and/or combination of transaction attributes that causes a problem. No other transactions cause a problem.
   
4. I added the transaction causing me a problem to a credit card card account (it was originally in a checking account) and validated. 1 scheduled transaction corrected.

If anyone can make sense of this I'm all ears. If this trouble shooting that took me practically all day helps programmers in any way, GREAT! But this is making absolutely no sense to me.

Time to start drinking.

Chris_QPW

Well it doesn’t make any sense to me either.  But in reality when you are dealing with something that corrupts things it isn’t unusual for its behavior to not make sense.

What I’ gathering from this is that they sequence of creating the scheduled reminder, use, and deletion is somehow triggering Quicken to corrupt the account.  I say the account for two reasons, your efforts to get different behaviors by using/not using the reminder/memorize payee/manual entry all resulted in the same thing happening, but once you changed to a different account it didn’t repeat.

I wonder if this would be repeatable in a different account.  If not then it might be a one time corruption in that account and the best action would be to move the transactions to a new account.  But if it is repeatable in another account then I certainly submit that sequence in a problem report (or reference this thread it if is too long) with Help > Report a problem

Pilotbum

It was repeatable in a different account. Bullets 19 & 4 described that in a credit card account. HOWEVER, when it happened in that account it wasn't from a scheduled transaction, it was just from a manually entered transaction (from the memorized list) but that memorized transaction was after I'd deleted the scheduled transaction and the associated memorized transaction. It was a memorized transaction from a manually entered transaction. Did that make sense? If so, are you suggesting I create a new scheduled transaction in a separate account, use it, delete it and see if I get an error? 

And I'd also like to clarify that THIS error was in a different checking account than the last one from 2020.