nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Architectural Goal: Make Each Quicken Dataset Truly Independent

pablomiller

I would like to suggest a long-term architectural direction for Quicken:

Ideally, each Quicken data file/dataset could represent a truly independent and isolated Quicken environment.

This could benefit Quicken as well as its customers: stronger dataset isolation could potentially reduce difficult support cases, make failures easier to diagnose and recover from, improve product reliability, and strengthen long-term customer retention.

A QDF contains our financial data, but a fully functioning Quicken environment also depends upon state maintained elsewhere — Quicken Cloud data, aggregation services, connection methods, authorization mappings, tokens and other online-service information.

There are undoubtedly good technical and security reasons why all of that cannot literally reside inside the QDF. But from the user's perspective, it shouldn't have to.

Perhaps each QDF/dataset could have a unique, immutable identity, with everything Quicken controls or maintains on behalf of that dataset exclusively associated with it.

A more protective architectural principle would be:

Nothing done while operating Data File A should be capable of changing, disconnecting, migrating, corrupting or otherwise affecting the operating state of Data File B.

Ideally, that isolation would hold across different Quicken installations, versions and computers.

Extend the Principle to Backup and Recovery

A complementary goal could be more dataset-complete recovery.

Restoring a known-good backup should come as close as technically possible to meaning:

“Return this Quicken environment to this known-good state.”

Quicken obviously cannot restore something independently revoked or changed by a financial institution. But Quicken-controlled configuration and connection state associated with the dataset could potentially be made recoverable along with it.

Why Invest in This?

Beyond giving customers greater confidence, stronger isolation could make certain classes of failures substantially easier for Quicken Support to diagnose and resolve. Fewer ambiguous interactions among local files, cloud datasets, connection services and installations could mean fewer lengthy troubleshooting and recovery sequences.

And for customers with years or decades of financial history in Quicken, confidence in reliability and recoverability can be an important part of the decision to remain a Quicken customer.

This need not be a single massive development project. It could instead become an architectural destination, with future development progressively moving Quicken toward stronger dataset isolation and recoverability.

The guiding principle would be simple:

The dataset — rather than the installation, computer, Quicken ID, aggregation provider, or some combination of them — should increasingly become the fundamental isolation boundary.

That could result in a more reliable and supportable product for Quicken, and greater confidence in remaining with Quicken for the long term for its customers.

Find more posts tagged with

Reviewed

Status: Reviewed

Comments

Chris_QPW

This is already true:

Perhaps each QDF/dataset could have a unique, immutable identity

If you select Help and then hold down Shift and select About Quicken you will get a dialog that specifies the QCS and Dataset IDs.

If you bring up another data file you will see that the Dataset ID is unique.

Note one might assume it is the Quicken Id that is the unique Id for your subscription, but it isn't. You can freely change it at any time. It is the QCS ID that serves that purpose.

But here is the fundamental problem when we start talking about multiple machines. It is the very fact that Quicken is "machine independent" that the user can mess things up if they aren't careful, or even for a restore from backup.

The data file and the dataset (the one on the server) are a match set that they have to keep in sync.

If a person is on machine A with their data file and dataset all synced up and they have a copy of their data file on another machine that is an older copy of their data file on machine B. If they go to machine B and open it up it is going to connect to the dataset that is more recent which might cause problems. This is why if one is going to share a data file, they need to copy it back and forth and make sure that they are always using the latest version.

To actually prevent someone from "doing the wrong thing" Quicken would have to track the computer and warn someone when they aren't using a match data file and dataset. But it would have to be at most a warning/recovery flow since there would be times where you have to do this.

But problems can extend past this. There is actually no clear way to restore from a backup. If you use an old copy of your data file, you are of course introducing this same out of sync problem. On the other hand, if you use Quicken's restore what it is basically going to have to try to do is blow away the dataset and send what it needs to reconstruct it from the data file. But there is a quirk to this too. The data file and dataset aren't the only players in this; your online services have their own states too.

The bottom line is that it isn't that Quicken Inc didn't think about such problems and put their best effort to try to keep this all in sync. The problem is that they very fact of setting up a disjointed system like this is extremely complicated, and with millions of users, there is bound to be problems.

Chris_QPW

Side note. This complication is why Quicken's Copy now creates a new unique Id and disables all online services. And why one has to never use the old practice of copying a data file using Windows File Explorer and use it as a template for another data file.

Before Quicken 2014 there was no dataset and as such it didn't matter how you copied the data file. But once a data file got connected to a dataset an identical copy used as a template for another data file is going to cause problems. And you can see they knew that also that keeping the online services connected would cause problems too.

This was a bit of a pain because Quicken's copy does something special that was useful. It does a database record by record (not a byte by byte) copy leaving behind deleted data records and possibly some corrupted records. And was used as an extra step besides Validate and Repair to clean up a data file. Now when you use it for that purpose you have to reconnect your online services.

NotACPA

@pablomiller What problem are you having that you think your IDEA addresses?

Because a straight-forward reading of your idea seems to be to create something that already exists, as cited by @Chris_QPW

pablomiller

@Chris_QPW — thanks. Your two comments provide exactly the kind of technical background I hoped this Idea might generate.

Your clarification that the QDF/dataset already has a unique identity is particularly helpful. In that case, I would revise that portion of the Idea: the opportunity isn't to create the identity, but rather to make stronger isolation and recoverability follow from the identity that already exists. If anything, knowing that this foundation is already present makes the architectural direction seem more attainable.

I also don't disagree at all that the QDF, cloud dataset and external online services form a complicated distributed system with multiple states that must somehow remain synchronized. Your examples involving two computers, older copies of a data file, backup restoration, and the changes Quicken has already made to Copy illustrate that complexity very well.

Your observation that there is presently “no clear way to restore from a backup” gets particularly close to one of the problems I'm hoping this Idea might ultimately address. From the user's perspective, restoring a known-good backup should ideally come as close as technically possible to meaning, “Return this Quicken environment to this known-good state.” Obviously Quicken cannot restore an authorization independently revoked by a financial institution, but it seems worthwhile to make everything Quicken does control as isolated and recoverable by dataset as practical.

I'm certainly not suggesting that any of this is simple, or that Quicken hasn't already put considerable effort into managing these interactions. The Idea is intended more as an architectural destination: progressively make the underlying complexity less capable of adversely affecting other datasets, and make recovery of a known-good dataset state more deterministic and understandable to the user.

So I appreciate the additional technical perspective. It helps refine what I'm proposing: not creation of a unique dataset identity, since you've established that already exists, but making increasingly strong isolation and recoverability follow from that identity.

pablomiller

One clarification on the problem this Idea is intended to address:

The problem isn't that Quicken lacks a unique Dataset ID. The problem is that a problem, change, or experiment involving one Quicken environment can have unintended consequences for another, and that restoring a known-good QDF does not necessarily restore the complete Quicken-controlled environment associated with it to that known-good state.

My recent Beta testing experience provided a very real example of the former: activity involving the Beta environment resulted in an otherwise unrelated Production environment no longer functioning as it had before.

I'm deliberately leaving the specifics of that incident in the Beta discussion, because this Idea is intended to address the broader architectural issue rather than one particular Beta problem.

In short, the identity may already be unique; the isolation and recoverability are not yet complete.

That's the gap this Idea is intended to address.

Chris_QPW

I personally have never seen any "Crosstalk" between any data files. But I will also state that I have heard of some people having such problems, but as far as I know it was related to the fact that a given financial institution was actually at the heart of the problem.

I will also state that I have almost entirely given up on testing other user's problems in my data files. Maybe the one off one here or there in my test data file. And the main reason for that is because there is no longer a safe way to recover.

Using an old copy of the data file or using Quicken's Restore both cause problems you have to clean up if the old copy/backup is even a day old

pablomiller

@Chris_QPW — I think your second point gets very close to the heart of what I'm trying to address.

I'm happy to leave the question of actual “crosstalk” open, particularly since the financial institution or another external service can itself be part of the chain.

But your statement that you've almost entirely stopped testing other users' problems because “there is no longer a safe way to recover” is a compelling example of the broader issue.

If restoring a backup that's only a day old can itself create synchronization problems requiring cleanup, then backup recovery isn't really returning the complete Quicken environment to its earlier known-good state.

That is exactly the kind of gap I hope this Idea might eventually help close. And I think your comments are helping sharpen the Idea considerably.

Chris_QPW

Well, my thoughts on this are that whereas it is a good goal I doubt it can be achieved.

I remember companies I have worked for the management would come up with all kinds of catchy phrases. One I remember was "zero defects".

The main problems with this are that first off, it is impossible to achieve and second it doesn't tell you one thing about how you would achieve this.

I'm a practical person I work in the real world, not in a management dream world.

Fundamentally Quicken was never designed for what it is now being asked to do. I think to have a "close to problem free" environment like this you would need to design it from the beginning. It would still be a much more complex problem than the original self-contained data file, but more doable.

What I expect (without being involved) is that the developers involved already looked at all of this and did the best they could given the existing code, the problems they faced, and their skill levels. So, I don't expect that situation to change.

But it certainly a good thing that I'm not running Quicken Inc to my liking or the Quicken users wouldn't have their "99% works" features.

99% sounds good until you have a million users (actually even 100, because that means you have one user that is unhappy).

It is a terrible tug of war even with one user. "Don't add any of those features I don't like and fix the bugs that affect me. And BTW add this feature that I want".

Not to mention "I have this feature that is impossible to add without problems but do it anyways and make sure that there aren't any problems".

This last list is getting so long these days I could probably fill a page with them.

Whereas I can see exactly why users ask for certain features and why Quicken Inc tries to provide them, a lot of their decisions make my stomach turn. And at the top of the list is the lack of third-party standards.

pablomiller

@Chris_QPW — I think we're probably closer on this than it may appear.

I don't view the proposed architectural destination as equivalent to “zero defects,” nor do I expect perfect isolation or recovery ever to be achievable. The value of a destination is that it can influence incremental architectural decisions even when the destination itself may never be reached completely.

Your observation that Quicken “was never designed for what it is now being asked to do” actually strikes me as one of the strongest reasons for establishing such a direction.

My own industrial experience was with electronic controls for engine and powertrain systems. Today's systems routinely do things that the early designs never anticipated and could not have accomplished competitively. Continued evolution became possible only by developing stronger architectures capable of accommodating increasing complexity and capabilities that could not necessarily be foreseen when the architecture was created.

I'm not suggesting that engine controls and Quicken are technically comparable systems. But I do think the architectural principle carries over: a mature product shouldn't forever be constrained by assumptions embedded in an architecture designed for a much simpler world.

As older components are replaced, connection methods change, cloud services evolve, and other parts of Quicken are modernized, each significant change presents an opportunity either to reduce these dependencies or perpetuate them.

I also wouldn't presume that the developers haven't done the best they could within the architecture and constraints they inherited. I'm sure many difficult tradeoffs have been made along the way.

My suggestion is simply that stronger dataset isolation and more deterministic recovery become explicit considerations in those future tradeoffs.

Maybe the destination is never reached completely. If Quicken gets progressively closer, I would still consider that a very worthwhile result.

Jim_Harman

This is certainly an interesting discussion.

One of the things I currently do for testing is to use Quicken's "Create a copy or template" to make a copy of my working file. I move the copy to another machine, rename the file, and use it on the other machine for testing. I never enable Mobile and web sync on my working file or the copy, and I never try to connect accounts in the copy for transaction downloading.

The only problem I have seen with this process is that creating the copy messes up my Tax Planner settings in the working file, as discussed here

https://community.quicken.com/discussion/7976307/tax-planner-settings-still-change-after-making-a-copy-of-data-file-cbt-1132-ctp-17402

I think this is a bug that should be addressed, not related to cloud datasets or anything like that.

@Chris_QPW are you saying that there are other risks with this process?

Chris_QPW

@pablomiller When I say "never designed for" I guess I should explain what I mean. People tend to see the word soft in software and think that means that you can change it in any way you like. That is only truly true before you start. After that you are always constrained by the existing code. Not for everything, but for some things.

There are some things that are "foundational", just like in a building. If you create a foundation for a single-family home and then decide to build a high rise that foundation just won't cut it.

In this case, if you want to have a properly synced system you probably had to have that in the original design to do it right.

I did vote for this though because I think it is worth them looking at it. Since I can't see the code and the problems they are facing I'm not in the best position to judge this, it just my educated guess of what the problems really are.

@Jim_Harman

Like I said before I have never had a problem with any data file to data file kind of problem, and that is even with using Sync to Mobile/Web. So, yes that is as good of a test setup as you can get.

The problems I was referring to have to do with the assumption that you can "just restore to the old state" for whatever problem you were having as long as you aren't talking about something to do with an online service. Those days are gone. They have been gone since they put in the QCS system where they are in between everything. So, I was talking about recovering from problems caused in a single data file. And for use cases where you are trying to isolate a problem. Like you are at a starting state, and you need to try to see if one of several things might have caused the problem and you want to get back to the original state before testing each of the combinations.

pablomiller

@Jim_Harman — thanks. Your Tax Planner example is a very concrete illustration of the isolation principle I'm trying to articulate. Whether it's simply a bug or related to some broader architectural issue, an operation intended to create a separate copy should not unexpectedly alter settings in the original working file.

Your testing practice also illustrates why predictable separation and recovery matter when copies are being used specifically to protect the working environment during testing.

pablomiller

@Chris_QPW — thanks, Chris. I appreciate both the clarification and the vote.

Your latest explanation helps sharpen the Idea into two related objectives: stronger isolation between environments, and more deterministic recoverability within an environment.

Your point that we can no longer assume an old backup will simply return us to the old operating state is particularly important. In that sense, “backup and restore” no longer means quite what users naturally expect it to mean.

I don't expect perfection on either objective — only that both become architectural directions worth moving toward as opportunities arise.