I've spent more than 20 years working with data, and some of the most useful techniques I've used would probably make a data-modeling purist cringe.
That doesn't mean I don't believe in good architecture or established best practices. It means I've learned to ask different questions first:
What is the data actually supposed to accomplish? How quickly can we get there, learn something and iterate?
I started thinking this way long ago, including while writing a JavaScript role-playing game called Treason in 2010. It was originally designed to run on an iPhone 3G, with a 400-by-300-position world and as many as 1,000 persistent monsters.
That hardware didn't give me much processing power to waste. But the more interesting lessons turned out to have very little to do with games.
1. Readable Can Be Better Than Elegant
One of the least sophisticated things I did in Treason was store much of the persistent monster and NPC data in text strings.
(I can hear data architects groaning.)
There were more efficient ways to do it, but strings gave me one huge advantage: I could read them. When the player encountered an NPC or monster, I could dump its data string into the game window or a debug box and immediately see its state. As things changed, I could watch the data change in real time.
Years later, I used almost exactly the same idea with several million rows of customer data at a bank.
I was analyzing how customer relationships developed over time: which account customers opened first, what they tended to open next, and which sequences occurred most frequently.
Instead of building an elaborate representation, I concatenated three-character product codes into strings:
Checking. Savings. Consumer credit card.
Would I recommend that as the canonical enterprise data model? Of course not. But for this analysis it was fast, easy to manipulate and immediately readable. I could group millions of customer histories, count pathways, calculate percentages, compare sequences and join the results back to the customer records. I used a similar technique for account-association analysis.
The representation was technically crude. The analysis was not.
Processing efficiency matters, but so do development efficiency, diagnostic efficiency and the ability of a human being to understand the data.
2. Changing State Should Make Intuitive Sense
The monsters in Treason are persistent. If the player wounds one and it escapes, that same wounded creature continues to exist somewhere in the world. Walking away doesn't reset it.
That forced me to think of each monster as an entity with a real state rather than as a temporary encounter.
The same principle applies to customers, accounts, orders and other changing business entities. If state matters, the data should make that state reasonably intuitive rather than forcing someone to reconstruct reality from layers of disconnected transformations.
When something changes over time, its representation should help you understand what it is now and how it got there.
3. Make Important Questions Easy to Answer
Treason represents its world spatially. If I want to know what monster is standing at coordinate 143,88, I don't search through a separate table for an entity whose X and Y values happen to equal 143 and 88.
I look at 143,88.
The structure reflects a question the program needs to ask constantly.
The same idea matters in business data. I've seen production environments where answering a basic business question requires five joins, several transformations and somebody who remembers what CUST_STAT_2 actually means.
If people are going to ask the same question over and over again, the architecture should not make that question unnecessarily difficult to answer.
Sometimes the problem isn't the query.
Sometimes it's the architecture underneath it.
4. Don't Process What You Don't Need
Treason can contain around 1,000 monsters, but I don't process all of them whenever the player takes a step.
Most of them effectively sleep.
Nearby monsters are active. Distant monsters still exist and retain their state, but there is no reason to continually spend processing power on them.
I've used the same principle in professional data transformations. If several million rows enter a process but only 20% need an expensive operation, I often separate that 20%, process it, and merge the untouched records back in afterward.
Passing unchanged rows through every transformation is not necessarily free. Joins, formulas, matching operations and other nodes still consume resources. In AI-driven processes, unnecessary data can also mean unnecessary tokens and unnecessary cost.
This isn't a novel technique, yet I've seen plenty of implementations drag enormous volumes of irrelevant data through processes that have nothing to do with them.
Sometimes optimization simply means knowing what not to process.
5. Order Changes the Answer
Processing order matters in Treason. If several monsters are following the player, moving the farthest one first can cause it to run into the monster ahead of it. Processing the nearby monsters first lets them move out of the way.
Same logic. Different order. Different result.
Consider a customer record with four possible ZIP code fields. One might come from a validated address, another from a salesperson, another from a legacy conversion and another from an old external source.
Taking the first non-null value is easy.
Getting the right value requires understanding which sources deserve priority.
That is why data transformation isn't merely about moving and manipulating values. It also requires understanding what those values mean and when one should take precedence over another.
6. “Best Practice” Is Not Always the Best Solution
One problem in Treason involved line of sight.
I studied established approaches used in earlier role-playing games, including methods associated with Richard Garriott's games. The approaches worked, but both could produce significant inaccuracies in what the player should or should not actually be able to see.
So I developed my own method.
It wasn't simply different because I preferred it. For my game, it produced a more visually accurate result while still performing well on the available hardware.
I later ran into the same problem professionally.
At one company, I needed to fuzzy-match roughly five million short records containing account identifiers. The sophisticated data platform we were using had built-in fuzzy-matching tools, but for this particular problem they were far too slow, and the results still didn't seem complete enough no matter how much I tuned the parameters.
So I wrote my own matching routine in, of all things, JavaScript.
It wasn't designed to be a general-purpose fuzzy-matching engine. It was designed around the specific characteristics of the data in front of me.
It was dramatically faster.
It was also more accurate.
Ultimately, I matched roughly 96% of the account records.
That experience is one reason I'm careful about the phrase “best practice.”
Organizations have good reasons to document best practices. Repeatable processes create consistency, preserve knowledge and keep teams from repeatedly solving the same problem.
But there is a limit to what can be put in a checklist.
You can document how to use Tool X, configure Setting Y and execute Process Z.
It is much harder to document:
Look at the problem carefully. Recognize that the standard solution doesn't fit. Understand why. Then invent something better.
That depends on experience, curiosity, judgment and an understanding of the data itself.
Best practices are valuable.
They should be a starting point, not a substitute for thinking.
7. More Data Is Not Always More Insight
And, because it is 2026, eventually every discussion of data arrives at AI.
AI can find patterns a human might miss and analyze quantities of information that would be impractical for a person to review manually.
But the old lesson about sleeping monsters still applies.
Just because information exists doesn't mean it belongs in the analysis.
There is a tendency to assume that if AI can process more information, giving it more must produce a better result. More columns. More records. More history. Everything.
Sometimes that helps. Sometimes it just gives the system millions of additional things to consider that cannot affect the answer, making the analysis slower, more expensive and potentially less useful.
A person with strong domain knowledge may look at hundreds of attributes and immediately know that most are irrelevant to the question.
There is also the opposite problem: AI can exhaustively analyze columns A and B without realizing that the answer depends on column C sitting in another system that nobody included.
A human who understands the business may know that.
AI makes analytical tools more powerful. It does not make understanding the problem less important.
The Lesson Hasn't Changed Much
When I wrote Treason in 2010, limited computing power made inefficient decisions painfully obvious.
Today we have cloud platforms, distributed processing, sophisticated analytics software and AI systems that would have seemed absurd when I was writing JavaScript for an iPhone 3G.
But I still ask essentially the same questions:
What does the data need to accomplish? How does it change? What needs to be easy to understand? What actually needs to be processed? In what order? Which accepted practices genuinely help? And how much complexity does the problem actually deserve?
After more than 20 years working with data, I still find that some of the best solutions are not the ones with the most sophisticated — or trendy — architecture.
They are the ones that make the problem clearer, get to a reliable answer efficiently and leave enough room to notice something you weren't originally looking for.