A general framework for fault-safe financial data ingestion and computation
Jul 11, 2026
Epistemology, not just ontology
Reconciliation is applied epistemology. Companion to Reflections on financial advisory and portfolio management.
Reconciliation occupies the basement of high finance. No quant enters the profession dreaming of matching custodial files at seven in the morning; the glamour is presumed to reside several floors above, among the alpha signals, the investment theses, the models, and the backtests with their obliging Sharpe ratios. Financial engineering programs flatter this preference. They teach stochastic calculus, market microstructure, and machine learning as though the inputs to these magnificent engines arrived clean, complete, timely, and true. I’m not your back-office b**tch, a senior quant engineer once protested when I asked him to reconcile an account’s performance. He was mistaken. It is, technically, middle office.
The name financial engineering is more generous than descriptive. There is little engineering in much of financial engineering, and less still in academic finance, with the predictable result that armchair philosophers of markets are abundant while people who can force an idea through the stubborn machinery of commercial reality — vendors, custodians, settlement, taxes, fees, and the client who wires out cash halfway through a rebalance — remain scarce. The discipline lies precisely in that distance. An idea that has never survived contact with a custodial file is not yet an idea about markets; it is an idea about mathematics.
The engineering, moreover, is not identical with the coding. Code can increasingly be delegated, and the writing of it occupies a diminishing share of the craft as machines become more competent at drafting what humans specify. What cannot be delegated so easily are the habits of mind that programming has traditionally imposed: the explicit management of state rather than its tacit assumption; the preference for pure functions and declared side effects; the regularizing instinct that favors a simple mechanism that fails predictably over a clever one that fails inventively; and the presumption, fundamental to fault-tolerant design, that every input will eventually arrive late, duplicated, corrupted, or wrong. The difficulty is that these habits are rarely acquired except through injury. One learns to respect state from the bug that corrupted it, and to respect fault tolerance from the pager that sounded because it was absent. Reflections carried the conclusion in its subtitle: all portfolio managers should be engineers. This essay attempts to explain what that engineering consists of.
Its first business reality is severe but simple: every representation of the world may be wrong, and one must nevertheless act upon it. Fiduciary duty is often a duty of speed. The deposit must be invested, the withdrawal funded, the alpha refreshed, the loss harvested, and the portfolio rebalanced today, at today’s prices, on today’s data. The duty of care does not adjourn while the data is repaired, nor does responsibility ordinarily remain with the defective file when that file produces a trade error. You selected the vendor, designed the ingestion, and decided what to trust, to what degree, and at what moment. The file was wrong may explain the failure, but it does not excuse it, because selecting, supervising, and constraining the file was itself part of the work.
Between the world and its representations lies a gap that finance can neither abolish nor ignore. It prices across that gap, settles across it, and occasionally falls into it. Reconciliation is the discipline of measuring the distance before the distance measures you.
The importance of that discipline becomes clearest when the representation aspires to reproduce not merely an account, but an entire financial life. I recently compared notes with a fintech founder friend and the crew he had assembled at his Hacker House in Atherton. Their project: lifetime financial planning by simulation, in which every consequential choice would be rehearsed in a digital twin before being made in the life itself. What to own and when to sell it, which bill to pay first, how much risk to insure, whether to refinance, which college to attend, when to retire, and when to buy the house would all be run forward, cheaply and repeatedly, through a model intended to contain the whole financial environment.
Take the ambition seriously and the scope is not merely the tax code plus an assortment of optimizers. It is everything: every price of every instrument; every interest rate, exchange rate, yield curve, credit spread, and volatility surface; every statute, regulation, benefit formula, insurance term, debt covenant, fee, penalty, vesting date, tuition schedule, and household cash flow; every relevant fact about the person; and every plausible path along which any of these might change. The replica was to include not simply the financial world as it stood, but the branching futures through which a person and that world might travel together.
Among those recruited to construct it were philosophers working on ontology, an intellectually respectable and, in the age of large language models, distinctly fashionable concern. The instinct was sound. A system that proposes to reason about an entire financial life must first determine what kinds of things inhabit that life, which distinctions matter, and how the entities relate: a person to an account, an account to an asset, an asset to a price, a marriage to a tax status, a mortgage to a house, an obligation to a date. Before the machine can reason about the world, it must possess some account of what the world contains.
Yet the harder problem begins after the world has been named. It is not merely ontological but epistemological. The system must know not only what an account, a price, a marriage, a tax liability, or a future obligation is, but which claims about each are true, how those claims became known, when they were last verified, which authority supports them, how much confidence they deserve, and what should happen when credible sources disagree. A custodian may report one position while an adviser’s ledger reports another; a market-data vendor may revise a price after a trade has already been placed; a tax rule may be enacted, interpreted, challenged, and applied on different dates. Ontology supplies the replica with its nouns and relations. Epistemology determines whether any particular sentence composed from them deserves belief. Reconciliation is the machinery by which that belief is tested, revised, and earned.
The ambition is magnificent because simulation is cheap and life is not: one would prefer to make a thousand mistakes in the copy and none in the original. One imagines the Federal Reserve taking notes. Yet such a replica would also constitute derived state on an imperial scale, and derived state remains subject to the elementary law at issue here: unless it is continually anchored, it drifts. Prices move, statutes change, custodians restate, households surprise themselves, and the world declines the courtesy of remaining still. A twin that is not continuously reconciled against its original will therefore cease to be a twin and become, by degrees too small to notice until they are too large to repair, a fiction with a dashboard.
What follows may thus be regarded as a prolegomenon to any future Laplacean ledger. Before the simulated world can be trusted, someone must build the machinery that keeps it faithful to the world it simulates. That machinery is reconciliation.
In Reflections, I approached the problem through three aphorisms. The first was that information travels at the speed of light, while financial information moves at T+1 — at least in settled form. The second was that concurrent write access to the same custodial account is the original sin, because it introduces race conditions and yields undefined behavior. The third completes the argument: the custodian’s firmware is broken, not necessarily as a feat of engineering but as a social contract, and the only dependable remedies are exclusive access and explicit sequencing.
Together, these propositions describe a distributed system that existed before distributed systems acquired the name: a world of replicas without consensus, messages without exactly-once semantics, clocks without synchronization, and participants who do not share a definition of done. We do not choose this system; we inherit it, fax machines included. A later essay may consider the architecture one would design from first principles — single-writer custody, event-native records, and finality that moves at the speed of information rather than settlement — but for the present we must work within the system history has supplied. Thoughtful practice therefore consists in imposing discipline upon an infrastructure that offers few of the guarantees one would willingly have built into it.
A minimal formalism
The discussion so far has been conducted in prose, and prose eventually cheats. Before the engineering artifacts arrive, it is worth fixing notation precise enough to hold them — not rigor for its own sake, but addressability: every claim in this essay should be a statement about a named object.
Let \(x^{*}(t)\) — the ground truth — denote the state of the world at time \(t\): the account as it is, the price as it printed, the liability as enacted. No one holds \(x^{*}\); it is the thing fiduciary duty is owed toward and the thing no file contains. What one holds are sources. A source \(V\) — a custodian, an accountant, a market-data vendor, one’s own middle office — supplies assertions
\[x^V(s, t), \qquad t \le s,\]what is known at knowledge time \(s\) about the state at observation time \(t\), with the shorthand \(x^V_t := x^V(t, t)\) for the in-time reading. The constraint \(t \le s\) holds for realized observations because sources report the past, and the object is the knowledge triangle of the sections that follow: a row \(x^V(s, \cdot)\) is a vintage, a column \(x^V(\cdot, t)\) a revision history, the diagonal the in-time series, and \(x^V(\text{now}, \cdot)\) the best knowledge. Write \(x^V(\infty, t)\) for the terminal value, settled once the revisions stop. A source’s error, \(e^V(s, t) = x^V(s, t) - x^{*}(t)\), then splits in two:
\[x^V(s, t) - x^{*}(t) \;=\; \underbrace{x^V(s, t) - x^V(\infty, t)}_{\text{pending revision}} \;+\; \underbrace{x^V(\infty, t) - x^{*}(t)}_{\text{bias}}\]The first term, the pending revision, the source will repair by itself, given time; the second, the bias, it will never repair, because it does not know it is wrong. A good source is one whose pending revisions decay quickly in \(s - t\) and whose bias is small — quality is a property of the whole triangle, not of any single print. One clarification prevents a category error: the uncertainty here is epistemic, not ontic. Unlike a quantum state, the past is not undecided; \(x^{*}(t)\) is perfectly definite and merely unobserved. The triangle records ignorance receding, not the world making up its mind — and the only observer effect in finance runs through action, not measurement: reading the file disturbs nothing; trading disturbs everything.
Ground truth never appears in a computable expression. What can be computed are the residuals between sources,
\[\delta^{VW}(s, t) \;=\; x^V(s, t) - x^W(s, t) \;=\; e^V(s, t) - e^W(s, t),\]and this is the first structural fact about reconciliation: it is inference about the unobservable errors \(e^V = x^V - x^{*}\) from the observable disagreements \(\delta^{VW}\). The subtraction cancels whatever the sources share — two witnesses with a common failure mode agree with each other and with nothing else — which is the formal content of the earlier demand for independent witnesses: the value of a second source is not its accuracy but the orthogonality of its bias. Triangulate until no plausible error direction lies in the common blind spot.
Now the dynamics. The state moves under two hands,
\[x^{*}(t+1) \;=\; x^{*}(t) + a_t + w_t, \qquad a_t = \pi\!\left(x^V_t\right),\]where \(a_t\), the action, is the side effect you induce — chosen by a policy \(\pi\) of belief, not of truth, because belief is all \(\pi\) can be given — and \(w_t\), the exogenous flow, is everything the world’s other writers do: fees, dividends, journals, client flows, reorganizations. The expectation \(x^{E}_{t+1}\) follows the intended law of motion,
\[x^{E}_{t+1} \;=\; x^V_t + \pi\!\left(x^V_t\right) + \hat w_t,\]with \(\hat w_t\), the anticipation, the portion of the exogenous term already known — declared dividends, scheduled fees; zero in the simplest case. Tomorrow the source reports \(x^V_{t+1}\), and it will not equal the expectation. Define the innovation and decompose it:
\[\nu_{t+1} \;:=\; x^V_{t+1} - x^{E}_{t+1} \;=\; \underbrace{x^V(t{+}1,\, t) - x^V(t,\, t)}_{\text{revision of the base}} \;+\; \underbrace{\tilde a_t - a_t}_{\text{execution}} \;+\; \underbrace{w_t - \hat w_t}_{\text{exogenous surprise}} \;+\; \underbrace{\eta_{t+1}}_{\text{fresh observation error}}\]where \(\tilde a_t\), the execution, is the action as actually delivered — fills, partials, rejects — and \(\eta_{t+1}\), the print noise, is the new reading’s own error, the revisions it has not yet received. This identity is, in a strict sense, the subject of the essay. Every morning’s surprise is the sum of exactly four things: the base you acted on was revised; the action executed was not the action sent; the world did something unanticipated; and the newest reading is itself provisional. The contract-based reconciliation of the later section is this identity with each term bound to evidence — the execution term to fills and pending settlements, the exogenous term to declared entitlements, the revision term to the source’s own restatements — and a clock on every binding. What survives after each term has claimed its share of \(\nu\) is the break.
The frame this places us in is not exotic; it is filtering. \(x^{*}\) is a latent state, the sources are sensors with lag, noise, and the unusual habit of revising their past readings, the internal book is the posterior, and \(\nu\) is the innovation. Fifty years of state-space discipline transfer intact, including the central habit: a healthy filter is monitored through its innovations, not its states. When the pipeline is sound, \(\nu\) is small, patternless, and fully attributed. A break is a structured residual — and a recurring structured residual is a misspecified model of the world, which the later taxonomy will call a convention error.
The counterfactual is where the money is. With hindsight \(s > t\), the action one should have taken is the counterfactual action \(a^{*}_t = \pi(x^V(s, t))\) — the same policy, evaluated on revised knowledge: a counterfactual over information sets, not over worlds. The action error and its bound:
\[\Delta a_t \;=\; \pi\!\left(x^V(t, t)\right) - \pi\!\left(x^V(t{+}1, t)\right), \qquad \lVert \Delta a_t \rVert \;\le\; L_\pi \,\lVert r_t \rVert,\]where \(r_t = x^V(t{+}1, t) - x^V(t, t)\) is the revision — overnight, at this cadence — and \(L_\pi\) is the sensitivity of the policy to its inputs. Its impact on the world is \(\Delta a_t\) itself, carried forward by the dynamics: a mis-trade does not decay of its own accord; it persists until compensated, at whatever the price has become. The bound names the only two levers that exist. Shrink \(\lVert r \rVert\): better sources, second witnesses, arbitration before use — the ingestion disciplines. Or shrink \(L_\pi\): make the policy less sensitive to precisely the inputs most likely to revise — which is what regularization means operationally, and why a good system is regularized is an engineering theorem rather than a taste. Tolerance bands acquire an exact meaning here: a no-trade band is a region where \(\pi\) is locally constant, \(L_\pi = 0\), so revisions within tolerance induce identically zero action error. One does not ignore small breaks out of laziness; one designs the policy so that small breaks are provably harmless.
Finally, drift. A book \(b_t\) rolled forward on its own beliefs, \(b_{t+1} = b_t + \tilde a_t + \hat w_t\), accumulates drift \(\varepsilon_t = b_t - x^{*}(t)\) like a random walk:
\[\varepsilon_{t+1} \;=\; \varepsilon_t + (\hat w_t - w_t) + \cdots \;\;\Longrightarrow\;\; \operatorname{Var}(\varepsilon_T) = O(T).\]The self-rolled book does not merely err; it wanders. A book re-anchored on the record, \(b_{t+1} = x^C(t{+}1, \cdot) + \text{in-flight}\), carries error bounded by the record’s own lag window: \(O(1)\) against \(O(\sqrt T)\). The anchor rule of the coming sections is that choice, and nothing more.
With the notation fixed, the essay’s artifacts become statements. The stored triangle is \(x^V(s, t)\) made durable; best_knowledge is its bottom row and in_time its diagonal; \(\sigma\) is the knowledge cutoff, and an epoch makes every published number a pure function of a vintage. The custodian and the accountant are the sensors \(x^C\) and \(x^A\); IBOR is the posterior estimate of \(x^{*}(\text{now})\); OBOR is the pinned argument actually fed to \(\pi\), kept so that \(\Delta a\) can later be audited rather than argued. And reconciliation, throughout, is the discipline of the innovation: measure \(\nu\), explain it term by term, put a clock on every explanation, and treat what survives as the truth trying to reach you.
Prior art
None of this formalism is new, and the honest thing is to say so. Economists built the same apparatus decades ago, under the name real-time data analysis — forced to it by statistical agencies that revise, as we are forced to it by custodians that restate — and their results put numbers on the essay’s knobs. Croushore’s survey collects the field. The detour is worth taking slowly, because several of the results are surprising, and the surprises are load-bearing.
The structure of the revision \(r_t\) was the founding question. Mankiw, Runkle, and Shapiro (1984) drew the distinction that still organizes the literature: under the noise model, the in-time reading is truth plus measurement error, so revisions are forecastable from the print itself and the correct response is to shrink before acting; under the news model, the print is already the source’s conditional expectation, so the revision is orthogonal to everything knowable at \(t\) and the print deserves face value. The models have opposite testable signatures — variances fall along a column under noise and rise under news — and the diagnostic is a Mincer–Zarnowitz regression away. The first surprise is how badly the world fits either model. Aruoba (2008) found revisions to major U.S. series biased, persistent, and predictable from information available on announcement day — the initial print is not a rational forecast of the final. The deeper surprise is that the predictability persists decades after being published, which sounds like agencies leaving money on the table until one sees why: a statistical agency does not minimize mean squared error; it optimizes methodological continuity and the appearance of neutrality, and a conservatively biased first print is a policy, not a failure. The engineering lesson generalizes: a source is not a noisy sensor but an agent, its triangle shaped by its loss function rather than yours, and the loss function is discoverable from the columns. Faust, Rogers, and Wright (2005) supply the cross-sectional version — in several G7 countries roughly half the variance of subsequent revisions is predictable in real time. Sources differ in kind, not merely in quality, and jurisdiction is a feature of the data.
The sensitivity \(L_\pi\) has a theorem lineage, and the lineage carries a twist. Muth (1960) and Howrey (1978) give the optimal use of preliminary data — the Kalman gain, shrinking precisely as observation noise grows. Brainard (1967) is the policy version: under uncertainty about how one’s instrument maps to effects, the optimal response coefficient is strictly smaller than the certainty-equivalent one; Orphanides (2003) proved it for real-time data in so many words — react less to what revises more. The twist is that the opposite theorem also holds. Under purely additive noise, certainty equivalence obtains: the optimal policy ignores the uncertainty entirely and acts on the point estimate, because the filter has already done the shrinking. Whether uncertainty should attenuate the action therefore depends on where it enters — noise in the state, act as if certain; uncertainty in the response itself, attenuate. Regularization is not a temperament; it is a placement question, and misplacing it is timidity or recklessness with equal rigor. A good system is regularized — in the right coordinate.
The ragged frontier has a name in that literature — the end-point problem — and its central result is more damning than the name suggests. Orphanides and van Norden (2002) showed that real-time estimates of the output gap revise by roughly as much as the gap itself; the surprise is the decomposition, because the unreliability comes mostly not from data revisions but from the statistic. Trends, gaps, and seasonal factors are two-sided constructions — they need the future — so at the frontier the filter degrades to one-sided and the estimate revises even if the raw data never do. Some quantities are epistemically ill-posed in real time by construction; their revisions are manufactured by the definition, and no improvement in sourcing removes them. The Bank of England draws the conclusion in public: its fan charts shade the past as well as the future, because at short horizons the backcast is nearly as uncertain as the forecast. The recent past is almost as uncertain as the near future. And the stakes are not academic: Orphanides (2001) re-ran the 1970s with real-time vintages and found that the same policy rule that convicts the Fed on revised data comes close to acquitting it on the data the Fed actually saw. A canonical policy blunder partially dissolves into a vintage artifact — the most expensive \(\Delta a_t\) on record, with the verdict depending on which row of the triangle the judge reads. Judging the diagonal from the bottom row does not merely flatter backtests; it rewrites history.
Two further results bear directly on the doors and the arbitration policy, and both are gently humiliating. Koenig, Dolmas, and Piger (2003) showed that estimating a model on latest-vintage history and deploying it on preliminary data is classical errors-in-variables — and that the cure outperforms the disease’s intuition: models estimated on the diagonal and fed the diagonal out-forecast models estimated on revised data, even though the revised data sit closer to the truth. Worse data, used consistently, beats better data, used inconsistently; the in_time door is an econometric consistency requirement, not hygiene. Bates and Granger (1969) give the arbitration policy its closed form — combine sources with weights inverse to their error covariance, so the value of a second witness is literally its covariance term — and the subsequent literature supplies the humiliation known as the forecast combination puzzle: out of sample, the humble equal-weight average routinely beats the estimated optimal weights, because estimation error in the weights swamps their theoretical edge. The arbitration layer, in other words, should itself be regularized — the overbuilt rules engine loses to precedence-and-average not only in maintainability but in accuracy, which is the Three Disciplines’ overbuilding warning restated as an out-of-sample fact. Sargent (1989) supplies the closing caution: the reporting agency may already publish its own filtered estimate rather than raw measurements, in which case shrinking its print is filtering twice — adding lag, not safety. Your vendor may already be a Kalman filter; characterize the triangle before adding another gain.
Every knob in the ingestion layer, in short, has a published estimator, and several of the defaults an engineer would guess are wrong in interesting directions. The disciplines that follow are how a firm earns the data those estimators require.
Three disciplines
Before reconciliation is a process, it is a set of capabilities, and they come in three tiers of ascending difficulty. They are worth specifying before any implementation, because the tiers have an awkward property: they are built from the bottom and justified from the top.
The first is lossless record-keeping: everything received — every file, every message, every acknowledgment — kept verbatim, forever. Storage is cheap, and the vintage you failed to keep is unrecoverable at any price. But the tier is easier to satisfy than to satisfy well. A dump of everything into flat files honors the letter — nothing is lost — and betrays the purpose, because the archive does not exist for its own sake; it exists to feed the tiers above. Governance needs to replay it: re-ingest a corrected feed, reprocess a quarantined batch, prove what arrived when. Attribution needs to address it: retrieve the exact assertion, from the exact source, as of the exact moment. An archive that can be neither replayed nor addressed is not a library but a landfill — the vintage technically present and practically unrecoverable, which is the expensive way of having failed to keep it. Lossless is the floor. Addressable is the requirement.
The second is governance: the ingested record is presumed defective — late, duplicated, restated, occasionally wrong — and the handling of defects is policy rather than heroics at the point of failure. But policy undersells the requirement. The policies must be encoded: a platform whose API makes them first-class — declarable, versioned, testable, changeable — and whose abstractions match the policy patterns that actually occur: quarantine, precedence, quorum, tolerance, escalation. The governing constraint has a name in the software literature — Martin’s Dependency Inversion Principle: high-level modules must not depend on low-level modules; both must depend on abstractions. Concretely, the attribution tier must not depend on how any particular vendor’s defects are handled, and the vendor handlers must not leak upward; both depend on the abstraction between them — a stream of clean assertions carrying provenance, behind which the arbitration rules can change without the tiers above noticing anything but a version number. Invert the dependency and a policy change is a new implementation of a stable interface. Fail to invert it and every policy change makes the higher tiers a little more opaque, until nobody can say which rule produced which number. This is not something most engineers can do. It is something most engineers believe they can do, which is worse, and the signature failure is not underbuilding but overbuilding: a rules engine general enough for policies nobody has, configurable enough that the configuration becomes the new opacity. The right platform encodes the patterns actually observed and nothing speculative. Restraint is the hard part.
The third is attribution: the ability to say, of any derived number, which inputs produced it, under which assumptions, as known when. Performance data is where this tier earns its reputation. GIPS composites want one view, the portfolio manager presents another, the advisor a third, the end investor a fourth — and these are not four formats of one number but four defensible definitions of it. Each must be derivable from the same fact base, attributable to its inputs, revisable coherently when the inputs move, and reconcilable against the custodian’s arithmetic, the accountant’s, and whatever performance software the other side runs. The first tier is an archive; the second, a system of law; the third, an epistemology.
The tiers have a treacherous build order. Logistics runs bottom-up: the archive exists on day one, policies accrete as defects arrive, attribution comes last, if ever. The dependencies run top-down: what the archive must preserve is dictated by what attribution must address; what the policy layer must expose is dictated by what attribution must see through. Build in delivery order while designing in dependency order, or the early tiers harden into shapes the later ones cannot use — the flat-file landfill, the opaque rules engine — and the third tier arrives to find its foundations poured wrong. Built from the bottom; designed from the top. The only defense is to hold the end state in view from the first commit.
None of this is separable from the people. The engineers who build such platforms are rarely the ones who keep them: the builder ships the platform he imagines is wanted and moves on; the keeper inherits the toil the imagination missed. Toil is not noise — it is the ledger on which that misalignment is written, the transaction cost of improvement paid daily by those without the authority to remove its cause. Reading that ledger, and realigning authority with cost, takes a capable engineering lead and an honest engineering culture; no framework substitutes for either. I do not believe the engineering can be separated from the engineers. A first-class data platform cannot be built without a first-class engineering team: the team is the only way to the product, and the product is what drives the team. They compound together, or not at all.
Two tools serve the three tiers, and the next section takes them up in turn. Separation of sources is the simpler instrument: keep the record as received and the truth as derived, side by side, with a policy bridging the two. Bitemporality remembers everything: each assertion is kept with the time it was learned, so any past view can be reconstructed and any derived number attributed — the third tier’s natural substrate. The tools attack the same problem — sources that arrive late, wrong, and revised — and they are substitutes as often as complements. Bitemporality is the more powerful and the more expensive: it confuses analysts, taxes every query, and earns its keep only where revision itself is the subject — performance, backtests, audit. Separation of sources costs almost nothing and covers a surprising share of the ground. A shop can get away with one of the two for a long time. The mistake is implementing neither, and discovering that during a restatement.
Sources of truth
The purpose of financial engineering is to systematically induce side effects in the world. Trade, journal, sweep, elect, withhold: each operation attempts to move an account from one state to another. Whether an action and what action should be taken depends upon the state of the world now.
Here the difficulty begins, because no one has ever seen a portfolio. One sees files about portfolios. The operations desk is an involuntary Yogācārin: it never touches the world itself, only representations of it — vijñapti-mātra, delivered nightly over SFTP. The first engineering question, then, is not simply what is true? It is: what shall we treat as true, for which attribute, in which frame, at what time, and on whose authority?
The phrase source of truth obscures three distinct ideas. The source of record is the party whose books are authoritative. For positions and cash, this is ordinarily the custodian: whatever your database may report, the account is what the custodian’s ledger says it is. The source of truth is a narrower engineering designation — the input your system treats as authoritative for a specified attribute at a specified time. One vendor may govern closing prices, another corporate actions, while the custodian governs settled positions. Derived state is everything computed from those sources: the internal book, the model, the firm’s present belief about the account. The internal book is not the account. It is a belief about the account, and beliefs are precisely the kind of thing that must remain revisable.
Every representation also arrives inside a frame: a cut time, a timezone, an accounting basis, an adjustment policy, a revision policy. A custodial file is not the account in the abstract; it is the account as represented at 21:47 Eastern, under settlement-date accounting, subject to later restatement. Prices are plural — the consolidated close, the exchange’s official close, an evaluated price. Corporate actions are announced, amended, postponed, and sometimes reversed. A number without a frame is not data. It is a rumor with digits.
Separation of sources
At bootstrap, taking a vendor as gospel is often rational. It may be the only practicable way to begin, and the expected cost of error — defect frequency multiplied by book size — may be small enough to absorb.
An enterprise under load faces a different arithmetic. Given enough accounts, instruments, and mornings, a one-in-ten-thousand defect becomes a daily occurrence. Nor do defects distribute themselves politely. They gather in the difficult corners: foreign dividends with withholding, midstream symbol changes, odd-lot corporate actions, reorganizations interpreted with clerical originality. At scale, single-sourcing an attribute is a decision to be wrong coherently.
The remedies are dull because foundations usually are: redundancy, cross-validation, and explicit arbitration. Any fact capable of moving money should, where practicable, have a second independent witness. Sources should be compared before use, with agreement checks and control totals, not merely examined after damage. And disagreement must be resolved by policy — a precedence rule, quorum, or escalation path that determines mechanically which source prevails. At scale, truth is not an input. It is a procedure.
Beneath every ingestion pipeline is one pattern. Ingest the source of record from the vendor, as published. Resolve it — validate, arbitrate, fill — into a source of truth. Act on the truth, never on the raw record. The resolution step exists because the two series have different masters: the record answers to the vendor’s schedule, the truth answers to yours. Act on the record directly and the business breaks every time the vendor is late — and vendors are late. Let the truth float free of the record and it drifts, quietly and compoundingly. The whole design problem is that tension — absorb the record’s delays without inheriting them; track the record without being hostage to it — and the policy that resolves it should be written down, versioned, and boring.
One pipeline can carry the whole apparatus. Consider benchmark weights: a provider publishes them on a lag — sometimes a long one — yet the weights must exist, every morning, for optimization and performance to run. The design persists two series, separately. The provider’s prints, stored as published, are the source of record for what the provider said; late arrivals backfill it, and revision is its job. The operational series is the designated source of truth for everything downstream; it is written once per day, at decision time, and never revised — only superseded. Between the two sits a derivation. Note, too, what the design does not require: no bitemporal machinery — two plain tables and a policy carry the whole weight, which is the point of the separation.
SOURCE OF RECORD — the prints as published; persisted,
append-only; late arrivals backfill; revision is its job
══ p(t-2) ═══════════ p(t-1) missing ═══════ p(t-1) arrived ══▶
│ ┆ │
│ p(t-1) absent at 8:30: p(t) absent too, but
│ present: derive p(t-1) backfilled
│ use the b(t-1)=drift(p(t-2)) overnight: derive
│ print — one hop from the b(t)=drift(p(t-1)),
│ record re-anchored on the
│ │ not drift(b(t-1))
▼ ▼ ▼
══ b(t-2) ═══════════ b(t-1) ══════════════════ b(t) ════════▶
pinned pinned pinned
SOURCE OF TRUTH — the operational series; persisted separately;
never revised, only superseded — pinned at decision time
Each morning the deriver reads the record as it stands and applies the arbitration policy: if the print is there, promote it; if it is not, drift the newest print forward by returns and promote that. The separation is what buys resilience — a delayed record does not block the truth, because the truth is derived, and the business runs. The anchor rule is what keeps resilience from curdling into error: tomorrow’s derivation reads the updated record, never yesterday’s truth. A drifted value is derived state, one hop from a print; it is never itself drifted. A late print therefore costs one day of estimation error, extinguished at the next anchoring — whereas drifting the drift would compound quietly, a random walk of the truth away from the record, discovered months later by whoever reconciles the performance.
The pipeline instantiates the section’s trichotomy exactly: the prints are the source of record, the operational series is the engineered source of truth, and the drift is derived state, promoted only by policy. The record keeps revising its estimate of \(t{-}1\) as better information arrives — delayed updates are its design property, not a defect. The truth is pinned at decision time, never revised, only superseded. Performance measured against the pinned value answers what did we do, given what we knew; performance measured against the revised column answers what would we have done, knowing better. Both are legitimate questions. Woe to the shop that cannot tell them apart.
Bitemporality
The practical consequence is bitemporality. A sound system preserves both the time a fact concerns and the time the system came to believe it: valid time beside transaction time, in the vocabulary the temporal-database literature has carried since Snodgrass. Restatement then becomes an ordinary write rather than a special calamity, and a historical decision can be audited against the information available when the decision was made—not against the cleaner, corrected knowledge available now.
The formal object underneath has been rediscovered by every field that takes revision seriously; only the names change. Arrange the system’s beliefs as a matrix: rows indexed by knowledge date \(i\), columns by observation date \(j\), where \(x(i, j)\) records what was known on date \(i\) about the state of the world on date \(j\) — the \(x^V(s, t)\) of the formalism, taken one source at a time.
For realized observations, as distinct from forecasts or facts announced with future effective dates, the knowledge constraint empties one triangle: \(x(i, j)\) is undefined whenever \(i < j\), because observation date \(j\) had not yet occurred on knowledge date \(i\). What remains is triangular, and each cross-section has a name and a use.
A row is a vintage: the world as it was known on date \(i\). This is the macroeconomists’ term. The real-time datasets of Croushore and Stark at the Philadelphia Fed, and ALFRED at the St. Louis Fed, exist to serve GDP as it stood on a given day, not as it later became.
A column is a revision history: the biography of one observation date—first print, amendment, restatement—the same anatomy actuaries develop, one lag at a time, in run-off triangles.
The diagonal, \(x(t, t)\), is the in-time series: what was known on each date about that same date. The bottom row is the current retrospect: the “final” values, a title held only until the next restatement.
The benchmark pipeline of the previous subsection lives in this picture: the source of record owns a column, revising its estimate of \(t{-}1\) as information arrives; the operational truth owns the diagonal — pinned, one cell per day.
observation date j ─────────▶
1 2 3 4 5
┌────────────────────────────────────────────────┐
knowledge 1 │ x(1,1)◆ │
date 2 │ x(2,1) x(2,2)◆ ∅ (i < j: not yet │
i 3 │ x(3,1) x(3,2) x(3,3)◆ observed) │
│ 4 │ x(4,1) x(4,2) x(4,3) x(4,4)◆ │
▼ 5 │ x(5,1)* x(5,2)* x(5,3)* x(5,4)* x(5,5)◆* │
└────────────────────────────────────────────────┘
row i, read across ─▶ a vintage: the world as known on i
column j, read down ▼ a revision history: the biography of j
diagonal ◆ the in-time series x(t,t)
bottom row * the current retrospect
The diagonal deserves emphasis. An honest backtest walks the triangle row by row. On simulated date \(i\), the strategy may inspect vintage \(i\) and nothing below it. Where observations are available contemporaneously, the freshest frontier lies on the diagonal. Where publication lags intervene, that frontier recedes into the triangle and becomes ragged: the strategy must use the newest observation actually available in row \(i\), not the observation that later history says belonged there.
A strategy simulated against the bottom row instead—today’s cleaned, adjusted, and restated history—trades on information that did not exist at decision time. Vendors sell the cure as point-in-time data; the disease is lookahead bias wearing a respectable face. Decisions are made from the knowledge available in their own row and judged, years later, from the bottom row. A system that stores only the latter has quietly destroyed the evidence of what deciding was actually like.
Naively materialized, the triangle is quadratic in time and almost entirely redundant. Most facts are printed once and never touched again. Each column is therefore piecewise constant, changing only when a revision arrives. The efficient representation is sparse and event-shaped: store the first assertion and each subsequent revision, append-only, then reconstruct \(x(i, j)\) by selecting the latest version of observation \(j\) recorded no later than knowledge date \(i\).
One more column
In Postgres the two dimensions cost exactly two columns: date carries the observation date, updated_at the knowledge time. The second must be assigned by the database, never by the application — a client that supplies its own knowledge time is a witness writing its own timestamps.
create table observation (
series_id bigint not null,
date date not null, -- observation date
updated_at timestamptz not null default now(), -- knowledge time
value numeric, -- null is a tombstone
primary key (series_id, date, updated_at)
);
revoke update, delete on observation from app_rw;
The table is a ledger: insert-only, enforced by grants rather than by good intentions. A revision is a new row for the same (series_id, date) with a later updated_at; a deletion is a tombstone. Nothing is destroyed, so every vintage stays reconstructible. The canonical read rebuilds one — for each key, the newest assertion at or before knowledge time \(\sigma\):
select distinct on (series_id, date) *
from observation
where updated_at <= :sigma
order by series_id, date, updated_at desc;
One column added; every query changed. That is the honest price of bitemporality, and the tax is not the column — it is that every query must now declare which cross-section of the triangle it wants, and SQL will not force the declaration. A naive where date = ... returns every revision, and a sum quietly double-counts them. A join between two bitemporal tables read at different knowledge times manufactures a chimera: a vintage that never existed. The retrospect used where the frontier was owed is lookahead bias, compiled and cached. Each failure returns a number; none returns an error. The bug type-checks.
The remedy is ergonomic before it is technical: make the correct reading the default reading. The ledger lives in its own schema and is never queried casually; every bitemporal table then exposes the same small set of doors, with the same names, so that the suffix does the thinking. Two doors carry nearly all the traffic.
-- the ledger keeps its own schema; api is what analysts see
create schema ledger; create schema api;
-- best knowledge (the bottom row): what we now believe happened.
-- the bare name resolves here, so the naive query is the correct one
create view api.observation as
select distinct on (series_id, date) *
from ledger.observation
order by series_id, date, updated_at desc;
-- in time (the diagonal): what was known by the end of each
-- observation date. Move the cutoff to taste (the 8:30 run, say)
create view api.observation_in_time as
select distinct on (series_id, date) *
from ledger.observation
where updated_at < date + interval '1 day'
order by series_id, date, updated_at desc;
best_knowledge answers questions about the world: research, current reporting, reconciliation against the newest custodial file. in_time answers questions about decisions: backtests, performance as it was published, the audit of what the model saw. The rule of thumb fits on an index card — best knowledge for what happened; in time for what was decided — and the join discipline follows from it: like joins like, best knowledge to best knowledge, in time to in time. The one legitimate mixed join is the revision study — in_time against best_knowledge, per observation date — and it deserves its own named view, because how wrong were the first prints is a question a quant shop should ask on purpose, not produce by accident.
Two more doors serve the specialists. observation_asof(σ) — a function, since views take no parameters — rebuilds an arbitrary vintage; it is the epoch builder’s tool, and when several bitemporal tables meet in one query they must share a \(\sigma\), declared once in a leading CTE and threaded through: the query’s epoch header. observation_revisions, the ledger filtered to one (series_id, date), reads out a column of the triangle — the biography of a number, which is where every break investigation starts: what did it say, when did it change, to what. And the frontier a backtest must walk — the newest observation per series available on each simulated day — is asof evaluated day by day under a lateral join, the closest thing Postgres has to the as-of join kdb+ built a career on.
The doors are boilerplate, so they are manufactured, never handwritten: one template applied to every bitemporal table, so that _in_time means the same thing on prices as on weights as on flows. The suffix becomes the type system SQL doesn’t have. Writes get one door as well — assert_observation(series_id, date, value), which stamps the knowledge time itself; the application never touches updated_at. And when best_knowledge runs hot — it will; it is the default — materialize it as a real table maintained by trigger on ledger insert: the old current-plus-history pattern, where reads become a plain primary-key lookup and correctness survives because the current table is derived. If the two ever disagree, the ledger wins, and the current table is truncated and rebuilt from it — a cache, not a book. The physics cooperate: the ledger’s primary key (series_id, date, updated_at) is exactly the index distinct on wants, an insert-only table never bloats, and updated_at is monotone, so a BRIN index keeps time-slicing cheap as history grows.
The border crossing everyone actually faces is the join: a unitemporal table — flows, trades, an instrument master, anything with a date and no knowledge axis — meeting a bitemporal one. The join is underdetermined, because the unitemporal schema cannot say which cross-section it deserves; the question to settle before writing the on clause is where \(\sigma\) comes from, and there are exactly three answers. If it comes from nowhere — the question is about the world, and the flows just want the best current estimate of the prices — join the bare name on date, and the default door does the rest. If the whole query is an epoch, \(\sigma\) comes from the header: join observation_asof((select t from sigma)) like any other table. The interesting case is when \(\sigma\) comes from each row — the unitemporal table records decisions, and every row carries its own knowledge cutoff. That is a per-row as-of, which in Postgres is a lateral:
select t.trade_id, o.value as price_as_known
from trade t
left join lateral (
select value
from ledger.observation o
where o.series_id = t.series_id
and o.date = t.price_date -- the observation wanted
and o.updated_at <= t.executed_at -- knowledge available then
order by o.updated_at desc
limit 1
) o on true;
When the unitemporal side carries only a date and no timestamp, the row’s true cutoff is unrecoverable and a convention must stand in for it — which is exactly what in_time is: join it on date, and the convention lives in one view definition rather than scattered across queries. And when the observation may not yet exist at the cutoff — publication lag — the decision join becomes a double as-of: the newest observation date at or before the one wanted, and within it the newest assertion known in time.
left join lateral (
select *
from ledger.observation o
where o.series_id = t.series_id
and o.date <= t.date -- newest available observation
and o.updated_at <= t.decided_at -- known at decision time
order by o.date desc, o.updated_at desc
limit 1
) o on true;
The order by does the work — observation date first, knowledge time second — and the result is the ragged frontier of the earlier subsection, rendered in six lines: the join a backtest actually runs. The rule condenses to a border policy. A unitemporal table crossing into a bitemporal one is a tourist, and it must declare something: the bare name if its question is about the world, its own timestamp or in_time if its question is about a decision, the \(\sigma\) header if the query is an epoch.
One honest caveat, in the spirit of the first aphorism. now() is assigned at transaction start, but transactions commit out of order, so a vintage read near the leading edge is not repeatable: a straggler can commit later carrying an earlier updated_at, and the same \(\sigma\) returns different answers before and after. The fix is a watermark — publish only from a \(\sigma\) safely behind the oldest in-flight transaction (pg_current_snapshot() gives the horizon), or simply lag by a grace interval. Even your own database settles on a delay. Information travels at the speed of light; knowledge, it turns out, moves at T+ε.
Restatement
Nothing tests the design like performance. Published returns are the most derivative data a firm produces — sleeve returns feed blended returns, pre-tax feeds after-tax, dailies compound into inception-to-date — and the temptation is to store each series as a fact and maintain it. The trap is the word maintain. One bad price in an equity sleeve, discovered months late, must restate the sleeve’s pre-tax return, its after-tax return, the blended pre-tax and after-tax returns of every portfolio containing it, and every cumulative number downstream since the error. If those series are mutable rows, coherence is a discipline problem across thousands of tables — someone must remember every dependent — and discipline loses. The principle is the opposite: coherence is never maintained; it is derived. Store only observations bitemporally — prices, transactions, flows, lots, corporate actions, custodial positions — and make every return, at every level of aggregation, a pure function of an addressed state of the fact store. Then internal consistency is a theorem, not a chore.
The address needs three coordinates. A published number is \(R = f(\sigma, m, c)\): \(\sigma\) a knowledge-time cutoff over the facts — a vintage; \(m\) a methodology version — the assumption set, itself data; \(c\) a calculation-engine version. Call the triple an epoch. The coherence rule then fits in a sentence: numbers published together come from one epoch. The blend is never computed from stored sleeve returns; it is computed from the same \(\sigma\) the sleeve returns came from, so sleeve, blend, pre-tax, after-tax, and inception-to-date cannot disagree — they are projections of a single snapshot. The bad price stops being a coordination problem: the correction is a new fact — old observation date, new knowledge time — which produces a new vintage, which produces a new value everywhere at once. No one restates the equity sleeve and forgets the blend, because no one restates anything; everything re-derives.
corrected fact: price(x, day d)
(old observation date, new knowledge time)
│
▼
valuation(equity sleeve, d) ✱
│
▼
pre-tax return(equity, d) ✱
│ │
▼ ▼
after-tax return(eq, d) ✱ blend pre-tax(d) ✱
│ │
▼ ▼
blend after-tax(d) ✱ cum blend pre-tax
│ (d..today) ✱
▼
cum after-tax — sleeve and
blend (d..today) ✱
✱ = the dirty cone: recomputed. Outside the cone —
other sleeves, other accounts, all days before d —
content-addressed cache hits.
Re-deriving everything is affordable for two structural reasons. The dependency graph is explicit — price to valuation to sleeve return to after-tax to blend to cumulative — so a correction dirties exactly its downstream cone, not the universe. And returns chain-link: cumulatives are folds over daily atoms, so a fix on day \(d\) recomputes day \(d\) and re-multiplies forward, trivial even months back. Content-address the intermediate artifacts and unchanged subtrees become cache hits — the Nix and Bazel move, applied to returns. The literature is squarely on point: Build Systems à la Carte argues that Excel is a build system; incremental view maintenance and log-derived views are the database renderings of the same idea. A performance database is a build system whose artifacts happen to be returns.
The methodology change is the harder event, and it is harder because it is different in kind. A data correction is a new row in the same triangle; a change in how a transaction type is handled is a different triangle. Hence \(m\): the assumption set is versioned data, never an edit to code. To rebake, build the epoch \((\sigma, m_2)\) in parallel while \((\sigma, m_1)\) keeps serving — blue-green deployment for analytics, so no partially rebaked state is ever visible. A hermetic \(f\) makes the rebake embarrassingly parallel by account, and the graph scopes it: a change to option-assignment handling invalidates only the accounts that ever had an assignment, and the graph knows which. Best of all, the difference between two epochs is a query, not a project — and that diff is the impact study compliance wants before the flip, and the substance of the client letter after it. Reputational exposure is worst when the firm cannot say who is affected, by how much, since when; epochs reduce the question to a select statement.
Two conventions with regulatory teeth complete the design. Error corrections apply retroactively by nature; methodology changes apply prospectively by convention — a seam, itself a recorded and footnoted fact — and the two must never masquerade as each other. And published is not a table but a pointer: which epoch clients see is promoted by policy and moved only as an auditable event — the OBOR move, one level up. What was actually sent to clients is a book the firm itself keeps the record of: the statement archive is immutable, and a restatement is not an update but a superseding event carrying the old number, the new one, the cause, and both epoch identifiers, generated mechanically. Performance, in the end, is not bitemporal but tri-temporal — what happened, what was known, and how it was counted — and the whole apparatus leaves exactly one mutable thing: a pointer.
BOR bores
The industry’s book-of-record alphabet maps onto the trichotomy, though not the way the acronyms suggest. CBOR, the custodial book of record, names the source of record — with one care: the custodian’s ledger is the record, while the CBOR file ingested at T+1 is only a representation of it, cut at their time, on their basis, restatable. One never holds the record; one holds its representations. The record is kept at lot level, for one account, and it admits surprises: fees, journals, transfers, reorgs, and the custodian’s own corrections arrive there first, on nobody’s schedule but theirs.
ABOR, the accounting book, is not a rival truth but a second record-keeper with a different jurisdiction: the accountant is authoritative for NAV, accruals, fees, and income, as the custodian is for settled units and cash. It is also unashamedly revisionist — being right in retrospect is its contract, so accruals true up, expenses adjust, valuations correct. And it doubles as the independent witness for reconciliation, valuable precisely because it shares neither the custodian’s failure modes nor your incentives.
IBOR, the investment book, is derived state, full stop: the firm’s present belief — the newest custodial position, plus everything in flight (today’s fills, pending settlements, estimated corporate actions), projected to now. News crosses this boundary in both directions. Your own actions you observe before the custodian does — you sent the orders, you hold the fills — so IBOR leads CBOR on everything endogenous. The world’s actions — fees, dividends, journals, client flows — surface in CBOR first, so the record leads the belief on everything exogenous. Each book is the other’s news service, with opposite lags. IBOR exists because of the T+1 aphorism: the gap between decision-time and settlement-time truth grew painful enough that the industry gave the patch a four-letter name.
OBOR, the operating book — whatever one actually transacts against on a given day — is derived state promoted to source of truth by policy. It is assembled each morning from three inputs — the newest CBOR, the in-flight ledger that is IBOR, and the corrections surfaced by last night’s ABOR reconciliation — and then frozen, so that every trade can later be audited against what was believed rather than what became true. The discipline that keeps the belief honest is re-derivation: rebuilt each morning from the newest record, never rolled forward from yesterday’s book alone — a self-rolled book must walk quietly away from the custodian’s. The two canonical failures are the two ways to skip a step: transacting off the raw custodial file (stale — selling shares this morning’s fill already sold), and never re-anchoring (the rolling position-keeper whose divergence is discovered by whoever reconciles performance).
Measured against the trichotomy, the four books refuse to line up one-to-one, and the misalignment is itself the lesson. The three labels are not kinds of book but orthogonal properties: record is a fact about authority — who is bound by it; truth is a fact about designation — what your system elects to treat as authoritative; derived is a fact about lineage — what it is computed from. CBOR holds record authority, yet what you designate as truth each morning is a derived representation of it, the ledger itself being unreachable. ABOR is both at once: derived in lineage — the accountant computes NAV from custodial data and prices — yet a record in authority, because the struck NAV is official by agreement, however it was computed. Record status is conferred by social contract, not by computational primitivity. IBOR is the opposite case: never a record, derived through and through, and nonetheless the designated truth for one attribute — position, now — because the present has no record-keeper; past the record’s leading edge, belief is the only candidate for truth there is. And OBOR, derived for market state, is a genuine record for exactly one attribute: what the firm believed when it acted. No party but the firm is authoritative over its own past beliefs — which is why revising the operating book is not correction but falsification.
What separates the four books, in the end, is less content than revision contract. OBOR never revises — that is its entire value. IBOR never revises either: it is superseded, morning by morning, each edition an in-time estimate that stands as issued. CBOR is mostly final — it restates rarely, though when it does, it does so with authority. ABOR revises routinely, by design. This is why the operating book cannot double as the accounting book, however tempting one golden book sounds: no single book can promise both stability at decision time and correctness in retrospect. The contracts are incompatible; the books must be plural.
THE RECORDS — kept by others; authoritative,
each in its own jurisdiction; restatable
┌──────────────────────────┐ ┌──────────────────────────┐
│ CBOR · the custodian │ │ ABOR · the accountant │
│ settled units and cash, │ │ NAV, accruals, income, │
│ lot level, one account — │ │ fees; trade-date basis │
│ sees no sleeves │ │ revised often: right │
│ mostly final; restates │ │ in retrospect is its job │
│ rarely, surprises freely │ │ asks: what is it worth, │
│ asks: what has settled? │ │ officially? │
└────────────┬─────────────┘ └────────────┬─────────────┘
│ T+1 file — a vintage of │
│ the ledger, not the ledger │ independent
│ exogenous news arrives │ witness: its
│ here first │ recon feeds
▼ │ the next
│ derivation
THE BELIEFS — yours; derived state; │
may be partitioned into sleeves │
(the partition must sum to CBOR) │
│
┌────────────────────────────────────────┐ │
│ IBOR · newest record + the in-flight │ │
│ ledger: fills, pending settlements, │ │
│ estimated corporate actions │ │
│ leads CBOR on your own actions; │ │
│ trails it on the world's │ │
│ in time, never revised — rebuilt each │ │
│ morning, never rolled from yesterday │ │
│ asks: what do I hold, right now? │ │
└───────────────────┬────────────────────┘ │
│ decision-time snapshot,│
│ promoted to truth by │
│ policy, then frozen │
▼ │
┌────────────────────────────────────────┐ │
│ OBOR · the operating book of the day │ │
│ pinned: never revised, only superseded │ │
│ asks: what did we believe when we │ │
│ acted? │ │
└───────────────────┬────────────────────┘ │
│ │
▼ ▼
reconciliation — the beliefs against the records;
its findings anchor tomorrow's derivation
Sleeves complicate the picture in an instructive way. The custodian keeps one account, at lot level; a sleeved account partitions it into sub-portfolios by designation — this lot belongs to the equity model, that one to the ladder, the overlay owns the short options. The custodian neither knows nor enforces the partition. Every sleeve-level book — ABOR, IBOR, OBOR alike — is therefore derived state twice over: derived from the records, and derived again through a partition map that exists only in your metadata. The map must also migrate: corporate actions arrive at account level, and a split that multiplies the custodian’s lots multiplies your assignments with it, or silently orphans them.
The partition has one law — it must sum to the account, lot by lot for units and in aggregate for cash — and that law is the sleeve ledger’s only witness. An account-level book can be checked against the custodian and the accountant; a sleeve ledger has no external counterparty at all, which makes it the most dangerous kind of book: derived state that nothing outside the firm can contradict. Reconciliation therefore acquires an internal layer, Σ sleeves = account, and with it a class of break invisible from outside — the account ties perfectly to CBOR while the sleeves misallocate, quietly poisoning per-sleeve performance, attribution, and tax lots without producing a single external symptom.
Cash is where the fiction strains hardest — the aphorism again: sleeving worsens the concurrency problem because cash is shared state. There is one custodial cash balance and \(n\) sleeve cash ledgers. A dividend arrives at the account and must be attributed to the sleeve that held the payer; a fee, a sweep, a client withdrawal arrives at the account and must be allocated by policy. Every allocation rule is a clause of the internal contract, and every exogenous cash event without a rule becomes a break that someone allocates by hand, differently each time. A sleeve’s cash can go negative while the account is flush, and the reverse; only the sum is real. Meanwhile each sleeve’s strategy is a would-be writer to the shared account, so the partition converts one concurrency problem into \(n\) of them — which is why the working answer, in practice, is a single overlay writer that serializes every sleeve’s intentions. But that belongs to the next section.
The atomic transaction framework
An enterprise seldom acts upon the world directly. It instructs a custodian, routes to a broker, delegates to a middle office. Often it should: these functions benefit from specialization, scale, and regulatory moats.
But the moment an instruction crosses a firm boundary, one is executing a distributed transaction without a coordinator, a shared clock, a common log, or a common runtime. There are only messages moving through channels that may delay, duplicate, drop, or reorder them.
Lamport showed that in such a system before and after are constructions, not observations. The Two Generals Problem showed that no finite exchange over an unreliable channel creates common knowledge. Finance answered these theorems in its customary fashion: it lowered the standard from certainty to evidence and attached a deadline.
This answer is not foolish. It is sufficient—but only when engineered deliberately.
Whenever a unit of work is handed to a counterparty, five questions require explicit answers.
1. Do they understand the desired action?
This is the problem of semantics.
Confucius stated the requirement with admirable severity:
名不正，则言不顺；言不顺，则事不成
When names are not right, speech does not accord; when speech does not accord, affairs do not succeed.
The rectification of names is schema design. Code cannot repair undefined meaning. The instruction format must therefore be versioned; its identifiers must be unambiguous—which account, which listing, which tax lot—and its interpretation tested as code is tested, through golden files and conformance cases exchanged before the first live message.
An instruction is a sentence in a shared language. Where the language is not shared, one is not instructing. One is appealing to Providence.
2. Is the action desired at all?
Desirability depends upon a shared understanding both of the current state and of the side effect the instruction seeks to produce.
Sell 100 shares describes a different act depending on whether the account holds 100 unrestricted shares, whether those shares are already committed elsewhere, whether a restriction flag has been set, or whether another writer has issued a competing instruction.
Instructions should therefore carry their assumptions. The proper idiom is compare-and-swap:
Execute this action, provided that these conditions still hold.
Where the counterparty cannot evaluate those preconditions, they must be enforced by sequencing: exclusive write windows, per-account serialization, and temporal separation.
Concurrency across accounts; serialism within one.
Custody favors pessimistic concurrency control because compensation is expensive and path-dependent. Two writers acting upon one account are not conducting a workflow. They are running a race, and the account is the track.
3. How do you know the action has been acknowledged?
Received, accepted, and rejected are distinct states. None means executed.
Every instruction must carry an immutable identifier—an idempotency key—because the channel will eventually deliver a duplicate, and somebody’s retry logic, whether yours or theirs, will eventually fire twice.
Exactly-once delivery is not available. Exactly-once effect is an ordinary engineering achievement: a unique key and a receiver that deduplicates.
Every instruction must also carry a deadline, with conduct on silence defined in advance. Qui tacet consentire videtur may serve a committee. On the wire, silence past the deadline is not consent.
It is an incident.
4. How do you know the work has been completed?
Acceptance is not execution. Execution is not settlement.
Each transaction type therefore requires a defined ladder of finality states. Each rung must be evidenced by a positive artifact—a fill report, a confirmation, a settlement notice—and each artifact must have an expected latency.
Monitoring must consequently be two-sided. A sound system alarms not only on evidence of failure, but on the absence of expected evidence after its allotted time.
The most dangerous state in operations is not failed.
It is presumed done.
5. How do you know it is correct?
Here the framework reaches the limit of self-attestation.
Every artifact received so far—the acknowledgment, fill, confirmation, or settlement notice—has been produced by the counterparty. The actor is grading its own examination. Such evidence is necessary. It is not sufficient.
Correctness requires independent verification. Compute the state the world should occupy, given the prior state and the transactions believed to have occurred. Then compare that expected state with an independently produced representation of the state the world does occupy.
Where possible, triangulate among evidence streams that do not share failure modes: broker confirmation against custodial position against internal book.
That comparison has a name.
It is reconciliation.
The Shape of a Well-Formed Unit of Work
The five questions imply the structure of a sound transaction.
It is atomic at the business level, not necessarily at the message level. A tax-loss harvest consists of a sale and a replacement purchase. Partial execution is not half a success; it is a new risk state—wash-sale exposure on one side, an unintended market-exposure gap on the other.
The operating procedure must define the safety boundary: the set of transactions that can be executed safely, and the permissible disposition of every partial state.
The transaction is idempotent: safe to submit more than once.
It is evidenced: each transition emits artifacts a third party could audit.
And it is compensable, though the word should not seduce us. Markets admit no rollback. One cannot untrade. One can only trade again, at whatever price the world now offers.
Compensation is economic, not logical, and its cost depends on the path taken. The saga pattern in finance is therefore less an architectural choice than a description of reality. The maxim that activity interrupts compounding belongs as much to operations as to investment management.
Every state transition is a new surface on which risk may land.
Contract-based reconciliation
The naive reconciliation is a diff. Pull the custodian’s file, pull the internal book, subtract one from the other, and investigate the nonzero results.
This fails for an instructive reason.
The two snapshots are usually cut at different times, under different accounting conventions, with different flows in flight. Yesterday’s trade settles tomorrow. The dividend has been declared but not paid. The journal has been acknowledged but not posted. The diff reports each discrepancy faithfully, and falsely, as though it were a break.
The desk soon learns that most differences are noise. Once that lesson has been learned, the true break—the one that costs money—disappears into the phantom ones. Alert fatigue arrives with bureaucratic punctuality. Then comes reconciliation theatre: a ritual performed because policy demands it and trusted by no one.
The remedy is to promote reconciliation from a diff to a contract.
For each counterparty relationship, define the invariant that ought to hold—not merely between two files, but between two states given everything known to be in flight:
custodian_position(T)
= internal_position(T)
+ Σ pending_settlements
+ Σ pending_corporate_actions
± convention_adjustments
Cash has an analogous invariant, with accruals, fees in transit, and declared-but-unpaid dividends included on the right-hand side.
The invariant is conditional. It quantifies over the pipeline.
This is the essential point. Whether a difference constitutes a break depends not only on the state of the account, but on the state of the machinery acting upon it. A hundred-share gap is a break when nothing is pending. It is an expectation when yesterday’s fill settles tomorrow.
A reconciliation process that ignores in-flight state asserts a false proposition about the world and then panics at its own mistake.
Under the contractual model, the engine’s task is not merely to discover differences. It is to explain them.
Each observed difference is matched against the in-flight ledger:
This position gap corresponds to that trade, due to settle Thursday.
This cash difference corresponds to that fee, due to post at month-end.
Every explanation carries a clock: a time-to-live derived from the expected latency of the relevant finality state. A timing difference that outlives its explanation ceases to be a timing difference.
Contracts have clocks. Expiry is escalation.
What remains after explanation—the unexplained residual—is the genuine break. Breaks should then be classified by cause, because cause determines both remedy and owner:
- Timing: observe until the time-to-live expires.
- Convention: repair the model rather than the world.
- Data defect: quarantine the source, invoke the arbitration policy, and incorporate the defect into future source selection.
- Operational error: compensate at market.
- Unexplained residual: investigate independently; this is the territory in which fraud hides and auditors earn their keep.
Each class must belong to an owner who possesses the authority to repair causes rather than merely clear symptoms. Pain accumulates wherever agency does not. An unowned break category is a permanent subsidy paid to entropy.
Tolerances, too, are terms of the contract rather than tribal folklore. They should be attribute-specific, versioned, reviewed, and owned. A tolerance without an owner is only a slow leak wearing a signature.
The intellectual lineage of this approach is respectable.
It is Meyer’s design by contract applied at the boundary between firms: preconditions on instructions, postconditions on settlement, and invariants on accounts. Reconciliation becomes the runtime assertion layer of the operation.
It resembles property-based rather than case-based testing. One does not attempt to enumerate every way two files may disagree. One states the properties that must hold and permits the evidence to falsify them.
And it is anti-entropy in the distributed-systems sense: a repair protocol run between replicas that do not share a database—which is precisely what an adviser and a custodian are.
Once phrased this way, a further improvement becomes obvious. Contracts need not be evaluated only at seven in the morning.
Every acknowledgment, fill, confirmation, and settlement notice is new evidence. The invariant can be re-evaluated whenever evidence arrives. Event-driven reconciliation reduces detection latency from a day to minutes. The T+1 file is demoted from ceremony to witness: the slowest witness, and on good days the least surprising.
The Present Tense
A reconciliation program is, in the end, the firm’s empiricism.
Research proposes beliefs about the future. Trading acts upon beliefs about the present. Reconciliation is the apparatus that tests whether the present is what the firm believed it to be—the loop through which belief is revised by evidence rather than by incident.
Only one part of this loop is optional, and it is not the part that establishes the present.
A firm that cannot verify what it holds has no standing to predict what it ought to hold.
This is ultimately a fiduciary claim rather than a technical one. The duty of care is not exhausted by making a prudent decision. It extends to knowing what occurred, because care cannot be exercised toward a state that cannot be observed.
Trust the vendors, custodians, brokers, and middle offices. One must, or nothing moves.
Then verify them—mechanically, contractually, and on a clock.
Trust, but reconcile.
The system one would prefer—single-writer custody, event-native records, finality at the speed of information—belongs to another essay. Until then, we engineer for the world as delivered:
At T+1. In fixed width. Subject to restatement.