R28.16 improvement question - What is the risk of a remote Quicken data file?

DaverDee

Hello. I run Quicken Premier for Windows in a Parallels virtual machine (VM) on my MacBook Pro. In updating Quicken to R28.16 and as part of an improved item in this release, I get an expected message that suggests moving my "remote" file to my local machine.

My question is, what is the risk of having *my* Quicken data file remote?

I am confident that the coders assumed remote was defined as a file somewhere needing an internet connection as the message notes locations such as DropBox. However, my "remote" location is still local to my machine as I have the Quicken data files outside the VM and in the Mac's native file system.

I understand how an intermittent internet connection to a Quicken data file may introduce a corruption risk. However, is that the risk or the only risk assumed by Quicken? And for my own selfish bottom line, is there a risk in the configuration I have?

Thanks in advance for the insight.
-DaverDee

p.s. For what it is worth, the reason why I have the Quicken data files outside of the VM is because I use the MacOS' native back up solution called Time Machine (TM). And I've told TM to ignore the extremely large VM file (which contains Quicken) in performing the backup operation. As a result, I get the backup I want without having to backup the extremely large VM file each time TM runs.

GeoffG

The term remote for Quicken means a current data file stored anywhere other than the desktop PC. This can be any drive letter that has a physical drive contained within the case. I have my data files on drive G: and have no issues. You should not be using a Quicken data file on a NAS or any cloud service. Backups are not an issue, but the data file can become corrupted with no obvious indication until months down the road.

Based on the level of push-back from the community, there are a good number of users who have elected to store their files remotely. It's their data file and their choice, but a Quicken data file that has become corrupted with no idea when it became corrupted is not much use.

With all that said, I've seen another user post with a similar issue as yours using Parallels and getting a remote error message. This would seem to be an erroneous error message as a result of the VM ware.

DaverDee

Thanks GeoffG. Based on your comments, I assume then the risk is the connection reliability between the Quicken software and its data file. Therefore, I have no issues as the data file is stored on the same physical drive and the connectivity reliability is a function of the VM.

I've had this config for a number of years with no issues as I am no stranger to running Validate/Repair every so often for preventative measures. The message piqued my interest as not only was it new but an improvement noted in the release notes.

Thanks again for the insight.

-DaverDee

jfclague

I am running on a Parallels VM on my MacBook and the file is also stored on OneDrive but I haven't received this message. I am also on R28.16. Weird.

Rocket J Squirrel

jfclague said:

I am running on a Parallels VM on my MacBook and the file is also stored on OneDrive but I haven't received this message. I am also on R28.16. Weird.

I also could not get the warning to appear - and I tried. I copied my data file to a USB drive and to Dropbox and ran it from those places. No warning appeared in either case.

Chris_QPW

This message is misleading of what test Quicken is using to determine "remotely".


The only test it is doing is if the folder is in "network drive", like shared from another machine or a NAS.

The test doesn't detect "cloud storage" like the message implies.

The confusion comes in because in fact "cloud storage" uses a local drive, not a "network/remote" drive.

What happens with "cloud storage" is that you save you Quicken data file to a "special folder" (or one below it, which is on a local drive) that is being monitored by the "cloud software".  When it detects that the file has changed it then syncs that file to the server (cloud storage).  And of course if a file syncs to the "cloud storage" it then turns around and syncs that to all the machines that are setup to sync to it.

So DropBox, Google Drive, One Drive, and such are all going to be detected as local drives.  And frankly I have no idea why they the USB drive on the list.  Even though they tend to be slow, these days they are very reliable, and even though there are ways to detect this type of drive (which Quicken is clearly not using), there isn't any reason to warn users about putting their data files there.

About the only real way to detect these various cloud storage directories is by looking that the path and seeing if they have the various names in them like Dropbox, One Drive, ...

Chris_QPW

P.S. A network drive doesn't necessarily imply whether the drive is "remote" or not.

If my machine is ChrisPC and I share out c:\Quicken as "Quicken" I can open my data file in two different ways.

c:\Quicken\Current.qdf

\\ChrisPC\Quicken\Current.qdf  (UNC format)

At the most the UNC format will add a network layer to the access, which would only slightly change the performance.  But in truth Windows probably would detect that it is a local drive and just bypass that anyways.

This is why you will get different behaviors with the VMs.  It just depends on how you are making the drive available to the VM.

DaverDee

And do you have any additional insight Into the risk associated with remote disk whether by drive letter or UNC?

Chris_QPW

@DaverDee You have indicated that the VM and the disk where the Quicken file is stored is on the same machine.  If this is the case, then there is no additional risk for accessing that data file whether you access it by drive letter or UNC.  It is "equal" to accessing a "local drive".

The warning is for truly remote drives, ones that are not connected directly to the local machine.

When determining the risk one has to determine the likelihood that an error will be generated (non-corrected) at the level that Quicken would see.  Quicken has no system for correcting disk errors.  It assumes that reads and writes will be error free.  So, any errors could potentially corrupt a data file, but that isn't guaranteed to happen either.  In fact my opinion is that the risk for the Quicken software itself between bugs and such is the greatest risk of corrupting a data file.

Hard drives, SSD, and USB are very reliable and have error correcting built into them.  And as such are the "standard" of "reliability" that Quicken would expect.

Network drives can actually be just as reliable.
That statement might surprise some people looking at the warnings about not using them for Quicken.

But here is why Quicken Inc can't recommend them for use with Quicken data files.
When you look at the Hard drives, SSD, and USB you will notice that they are connected to the machine, and over well-known interfaces.

"Network Drive" on the other hand doesn't tell you much of anything about how it is connected.  Is it wired or wireless?  Is it using SMB or NFS or (there are several different protocols)?  Is the network setup correctly?  What is the chance that the machine/connection will die/drop or have errors while Quicken is reading/writing to the data file?

The very fact that Quicken Inc doesn't know if your network connection is reliable or not means that they can't recommend (they will even discourage) using it for accessing your data file.  Note that all the network protocols will have error correcting built into them, but if you have an unreliable connection there is just so much error correcting can do, especially if the connection is completely lost.

As for the "cloud storage", which is in fact not a "networked drive".  The recommendations of not having your active data file(s) on them comes from a different set of problems.

The first possible problem is syncing the data file when it is in an inconsistent state.
At its core the Quicken data file has a database, and it is pretty old and probably doesn't have any good way to dynamically recover from an inconsistent state (newer databases do).  It does have Validate and Repair, but this is more of a "last resort” kind of thing.  Typical operations might be making changes to several parts like the header information and then the transactions, then the categories, ...  And if the "cloud" program syncs at the wrong time the copy it has might be in and inconsistent/corrupted state.

Quicken locks the data file when it has it open.  And most cloud storage programs will honor that lock and not sync while it is in place. But some cloud storage programs might not honor it. Note that the same is true of backup programs, and in fact backup programs are less likely to honor the lock.

But what about the ones that do honor the lock?
For instance I know that Dropbox, One Drive, and Google Drive do honor the lock.

One problem comes up pretty quickly, and that is Quicken not being able to open the data file.

Let's imagine you have a slow Internet connection and a fairly large data file.
You use Quicken and it locks your data file, and then updates it, and then as you close Quicken it releases that lock.  Well then, the cloud program locks the data file and starts the sync.  And that is going to take a while to complete.  In the meantime, you decide you want to open up that data file again.  Quicken will now be locked out of it and will either say it can't read it or it will bring up the New User dialog.

Another problem comes up from the fact that during various operations Quicken will close the data file and release the lock, which in turn allows the cloud program to step in and lock it.  If Quicken would maintain the lock over such operations there wouldn't be a problem, but that isn't the case.

Operations where it releases the lock, and then tries to almost immediately get it back, (I might not have them all).
Backups
Validate & Repair
Moving investment transactions between accounts (because it does a Validate & Repair after this)
Turning Zillow estimate bar on or off.
Software updates

Rocket J Squirrel

Chris_QPW said:

This message is misleading of what test Quicken is using to determine "remotely".

The only test it is doing is if the folder is in "network drive", like shared from another machine or a NAS.

The test doesn't detect "cloud storage" like the message implies.

The confusion comes in because in fact "cloud storage" uses a local drive, not a "network/remote" drive.

Yes, this is correct. After failing to provoke the warning with a file on USB or Dropbox, I could provoke it with the file on a NAS.

Unfortunately, it makes whoever wrote that warning seem computer illiterate. The wording should be changed.

Chris_QPW said:

Quicken locks the data file when it has it open.  And most cloud storage programs will honor that lock and not sync while it is in place. But some cloud storage programs might not honor it. Note that the same is true of backup programs, and in fact backup programs are less likely to honor the lock.

But what about the ones that do honor the lock?
For instance I know that Dropbox, One Drive, and Google Drive do honor the lock.

I'm not so sure about this. Is there a Windows utility that shows open and locked files and which process has them open?

My impression has always been that Quicken does not hold the file open or lock the file for the duration of the session. Rather, when the user enters data, Quicken does an open-write-close sequence. It is obvious when the file is in a Dropbox folder because on every data change, the little Dropbox icon in the corner of the taskbar shows activity.

I'm willing to be proven wrong, but it would be nice to have a utility that shows open and locked files in real time. There must be one but I can't seem to find one.

Chris_QPW

Quicken holds the lock for the whole session.  Years ago I use to store my data file on a NAS and I very aware of how Quicken locks the file.

One way to find lock files is Sysinternals Process Explorer:
https://www.howtogeek.com/128680/HOW-TO-DELETE-MOVE-OR-RENAME-LOCKED-FILES-IN-WINDOWS/

Here is another program just for seeing if a file is locked and to unlock it.
https://www.iobit.com/en/iobit-unlocker.php#

Chris_QPW

BTW as an old system programmer I can tell you that "open" is a "costly" operation.  No database program worth anything would open and close the database file constantly.  And on top of that the QDF file is a compressed file.  You certainly wouldn't want to be constantly opening and closing that file.

The main reason that open is so costly is because when you open a data file the low level system file system will start caching at least some of start of the data file in prep for future reads.  An open operation can easily take 10 times longer than the average read.

And there isn't any need to close the data file.  Under normal circumstances the data will all be flushed to the disk within a few seconds automatically by the file system.  And for super critical operations the program can use a flush function to speed that up drastically.

Chris_QPW

To add to what I said.  Note that if a process dies all its files are automatically closed.  In rare cases the locks may not be released.  But what is critical is that the data is written to the disk.  "Closing" a file where all the data has already been written to the disk is only about freeing up the system resources.  Not about what will be in the file when it is opened up in the future.  All the warnings that you get about logging out/booting a computer with running programs is because they have no idea if that program(s) have data in memory that they have yet to write out to the disk.

Chris_QPW

Rocket J Squirrel said:

Unfortunately, it makes whoever wrote that warning seem computer illiterate. The wording should be changed.

I would state this differently.

The person isn't "computer illiterate", after all he/she is a programmer.  Instead they certainly "illiterate" in the area what the check they created actually checks and how the "cloud programs" work (and how to check for USB drives).