Doing the Right Thing — Designing Data-Intensive Applications

Much data is about people, their behavior, interests, identity, and must be treated with humanity and respect; human dignity is paramount. Ethical concepts are not precise or determinate the way computing usually is; they require interpretation and serious, ongoing discourse that the field largely lacks.

Predictive Analytics

Predicting the weather or disease spread is one thing; predicting whether a convict will reoffend, a borrower will default, or an insurance customer will be costly directly affects people's lives. Organizations are cautious, "if in doubt, say no", because the cost of a bad loan or hire exceeds a missed opportunity. But as algorithmic decisions spread, a person labeled risky can be systematically excluded from jobs, travel, insurance, housing, and finance, an "algorithmic prison." Where justice systems presume innocence, automated systems can arbitrarily exclude someone with no proof of guilt and little chance of appeal.

Bias and discrimination

Algorithmic decisions aren't automatically better or worse than human ones, but instead of coding the rules, we let the rules be inferred from data, and if the input carries systematic bias, the system learns and amplifies it. Anti-discrimination law forbids using protected traits (ethnicity, age, gender, sexuality, disability, beliefs), but other features correlate with them. The belief that biased data can yield fair output has been satirized as "machine learning is like money laundering for bias." Predictive systems extrapolate from the past; if the past is discriminatory, they codify it. Data and models should be our tools, not our masters.

Responsibility and accountability

If a human errs, they can be held accountable and the decision appealed; who is accountable when an algorithm errs (a self-driving crash, discriminatory credit scoring)? People shouldn't dodge responsibility by blaming an algorithm, and you should be able to explain how a decision was made. A credit score rests on your own correctable history; predictive analytics reason by stereotype ("people like you") and are opaque. Outputs are statistical, correct on average yet wrong for individuals (an average life expectancy of 80 doesn't predict any one person's death). Blind faith in data is "delusional and positively dangerous"; the same power that could focus aid on those who need it is also used by predatory businesses to find vulnerable people and sell them high-cost loans or worthless degrees.

Feedback loops

Recommendation systems create echo chambers (polarization, misinformation, election impact). Predictive systems create self-reinforcing spirals (credit-score-based hiring → joblessness → poverty → worse score). Algorithmic gas-station pricing in Germany even learned to collude, raising consumer prices. Many such effects can be anticipated with systems thinking, analyzing the whole system, people included: does it amplify existing inequalities or combat injustice?

Privacy and Tracking

When a system stores only what a user deliberately entered, it serves the user. When activity is tracked as a side effect, the service gains interests that may conflict with the user's. Some tracking genuinely helps users (search ranking, recommendations, A/B tests), but ad-funded models push it into detailed, long-retained profiling, better described by a more sinister word: surveillance.

Surveillance

A thought experiment: replace "data" with "surveillance", "we collect real-time surveillance streams and store them in our surveillance warehouse", and the phrases stop sounding good. Digitization has built the greatest mass-surveillance infrastructure ever seen: a microphone in nearly every room (phones, smart TVs, assistants, baby monitors, toys), tracking location, relationships, communications, purchases, and health. A collector may know more about a person than they know about themselves. The difference from past totalitarian dreams is that we voluntarily accept it, and it's corporations (for services) rather than governments (for control), but with car-tracked insurance premiums and fitness-tracker-gated health coverage, and intrusive inference (a smartwatch motion sensor inferring typed passwords), the line blurs.

The "users consented" defense is weak: it's unclear the tracking is necessary (vs ad-funded); users can't understand derived datasets, so consent isn't informed; data about a user reveals things about non-users; extraction is one-way and asymmetric with no negotiation. GDPR demands consent be "freely given, specific, informed, and unambiguous" and withdrawable "without detriment" (consent is the most common but not the only lawful basis, legitimate interest covers e.g. fraud prevention). "Just don't use it" fails when a service is effectively essential for social participation, when network effects impose a social cost, when products use gambling-style engagement tactics, and because opting out is a privilege of those with time, knowledge, and the luxury to miss out.

Privacy and use of data

"Privacy is dead" is false and misunderstands the word. Privacy is not secrecy; it is the decision right, the freedom to choose what to reveal to whom, where to sit on the secrecy-transparency spectrum in each situation. A patient might gladly share medical data with researchers but not if it could block insurance or employment, they must choose. Surveillance doesn't erase privacy rights so much as transfer them to the collector ("trust us with your data"); companies then keep the results secret (revealing them would seem creepy and harm the business), exposing intimate facts only indirectly (ad targeting). Even un-reidentifiable users lose agency over disclosure. Privacy settings that control what other users see are only a start, the service itself retains unfettered internal access. This large-scale transfer of privacy rights from individuals to corporations is historically unprecedented: surveillance was once expensive and manual, and trust relationships (doctor, lawyer) were governed by ethics and law.

Data as Assets and Power

Calling behavioral data "data exhaust" (worthless waste to be recycled) inverts the truth: if ads pay for the service, user activity is a form of labor, and the app is a lure to feed the surveillance machine. Data is a valuable asset (secretive data brokers; startups valued by "eyeballs"), wanted by companies, governments (deals, coercion, compulsion, theft), and acquirers (sold in bankruptcy). Because it's hard to secure and breaches are common, critics call it a "toxic asset," "hazardous material," "the new uranium." When collecting, weigh benefits against the risk of it falling into the wrong hands, including all possible future governments ("it is poor civic hygiene to install technologies that could someday facilitate a police state", Schneier). Knowledge is power, and "to scrutinize others while avoiding scrutiny oneself" is a key form of power that accrues to data-rich companies, largely outside public oversight.

The Industrial Revolution brought growth but also dreadful pollution, child labor, and unsafe conditions; safeguards took a long time but society benefited greatly. Likewise, "data is the pollution problem of the information age, and protecting privacy is the environmental challenge" (Schneier), our grandchildren will judge how we handled it.

Legislation and self-regulation

GDPR requires data minimization, data "collected for specified, explicit and legitimate purposes," "adequate, relevant and limited to what is necessary", which runs directly counter to the big-data philosophy of maximizing collection and exploring for unforeseen uses. So far it's been weakly enforced and hasn't shifted industry culture much. Companies oppose regulation as a burden, sometimes justifiably (over-regulation could block medical breakthroughs), and the benefit/risk balance is genuinely hard. Fundamentally we need a culture shift: stop treating users as metrics, self-regulate, educate users, and above all don't retain data forever, purge it when no longer needed and minimize what's collected, because "data you don't have is data that can't be leaked, stolen, or compelled." Privacy is like a national park or a commons: unprotected, it will be destroyed. Ubiquitous surveillance is not inevitable, we can still stop it.

Real-world examples and analogies

Technology is neutral, use is not, a search engine like a gun. The artifact itself isn't moral or immoral; the harm or benefit lies in how it's used and who it affects. The responsibility can't be offloaded onto the tool.
Algorithmic prison, a no that follows you everywhere. A risk label, accurate or not, can produce a cascade of "no" decisions across jobs, travel, insurance, and housing, confining someone without trial or appeal.
Money laundering for bias, a clean-looking output from dirty input. Feeding discriminatory history through a model and calling the result "data-driven" launders the bias into something that looks objective but isn't.
Credit score vs predictive analytics, "what you did" vs "what people like you did." A credit score judges your own correctable record; predictive analytics judges you by your demographic neighbors, stereotyping, with no recourse if you're in the wrong bucket.
Feedback loop, the poverty-and-credit spiral. A misfortune lowers your score, which costs you a job, which deepens poverty, which lowers your score further, a self-reinforcing trap dressed up as mathematical rigor.
The surveillance thought experiment, swap "data" for "surveillance." "Our surveillance scientists derive insights from our surveillance warehouse" reveals what the cheerful language of "big data" obscures.
Consent that isn't free, a service everyone needs. "Just don't use it" isn't a real choice when the service is essential for social and professional life and carries network effects, opting out is a privilege.
Privacy as a decision right, sharing medical data on your terms. Privacy isn't hiding everything; it's choosing what to reveal to whom. You might share a condition with researchers but not with an insurer, the point is that you decide.
Data as the new uranium, valuable and dangerous to store. Not gold or oil, but uranium: useful, but hazardous, hard to secure, and catastrophic if it leaks or falls into the wrong hands.
Data as pollution, the environmental challenge of the information age. Like factory waste in the early industrial era, data festers and accumulates; how we contain and dispose of it will define how future generations judge us.
Data minimization, the safest data is the data you never collected. Just as you can't spill a chemical you never brought into the building, data you don't retain can't be breached, stolen, or subpoenaed.

Cheat-sheet flashcards

Cover the answer and recall it.

Is a technology good or bad in itself? → No, what matters is how it's used and whom it affects.
What is ethics, per this chapter? → Iterative reflection and dialog with accountability, not a compliance checklist.
Why is predictive analytics ethically fraught? → It makes consequential decisions about individuals' lives.
What is "algorithmic prison"? → Systematic exclusion from society by risk-labeling algorithms, with little appeal.
What do predictive systems do with biased input? → Learn and amplify the bias.
Why don't bans on protected traits suffice? → Other features (postal code, IP) act as proxies.
The satirical phrase for biased ML? → "Machine learning is like money laundering for bias."
What's needed for a fairer future than the past? → Moral imagination, which only humans provide.
Credit score vs predictive analytics framing? → "How did you behave?" vs "How did people like you behave?"
Why are statistical outputs risky for individuals? → They can be right on average yet wrong in individual cases.
What is a feedback loop here? → A self-reinforcing spiral (e.g., credit score → joblessness → poverty).
A non-finance feedback-loop example? → Algorithmic gas-station pricing learning to collude.
What is systems thinking? → Analyzing the whole system including the people, to foresee consequences.
When does tracking become surveillance? → When it serves advertisers via detailed, long-retained profiles, not the user.
The "data → surveillance" thought experiment shows what? → How benign "big data" language hides surveillance.
Corporate vs government surveillance difference? → Corporations collect for services, governments for control.
Four reasons consent isn't meaningful? → Necessity unclear, users don't understand, data about non-users, asymmetric one-way extraction.
GDPR's consent standard? → Freely given, specific, informed, unambiguous; refusable without detriment.
A non-consent lawful basis under GDPR? → Legitimate interest (e.g., fraud prevention).
Why is "just don't use it" not a free choice? → Network effects and effectively-essential services.
What is privacy, properly understood? → A decision right: choosing what to reveal to whom, not total secrecy.
What happens to privacy rights under surveillance? → They transfer from the individual to the data collector.
Do privacy settings solve this? → No, the service itself keeps unfettered internal access.
What's wrong with calling data "data exhaust"? → It's a valuable asset; user activity is more like labor.
Why call data a "toxic asset" / "the new uranium"? → It's valuable but hazardous, hard to secure, dangerous if leaked.
Whose hands must you consider when collecting data? → All possible future governments and bad actors.
The Industrial Revolution analogy? → Growth plus pollution/child labor → regulation; data is today's pollution.
Schneier's framing of data? → "Data is the pollution problem of the information age."
What does data minimization conflict with? → The big-data philosophy of maximizing collection.
The strongest argument for not retaining data? → Data you don't have can't be leaked, stolen, or compelled.
Is ubiquitous surveillance inevitable? → No, we can still stop it.

Discussion and interview questions

Why is it insufficient for engineers to focus only on the technology? (Every system has consequences; ethical responsibility can't be offloaded onto a neutral tool.)
How do predictive analytics differ ethically from forecasting weather or disease? (They make consequential, hard-to-appeal decisions about individuals.)
Why can an algorithm trained on data still produce discriminatory output? (It learns and amplifies bias; proxies leak protected traits.)
Who is accountable when an automated decision is wrong, and what does explainability require? (Humans must remain accountable; decisions must be explainable and appealable.)
What is a self-reinforcing feedback loop, and how can systems thinking help? (Spirals like credit→joblessness; analyze the whole system including people.)
When does behavioral tracking cross into surveillance? (When it serves advertisers via long-retained profiling rather than the user.)
Why is "the user consented" often not meaningful consent? (Necessity unclear, no understanding, non-user data, asymmetric extraction, no real opt-out.)
Explain privacy as a "decision right" rather than secrecy. (Freedom to choose what to reveal to whom; surveillance transfers that right to collectors.)
Why is personal data described as a "toxic asset"? (Valuable but breach-prone, sold in bankruptcy, dangerous under future regimes.)
Compare the information age to the Industrial Revolution. (Growth with pollution/labor abuses → regulation; data is the new pollution.)
What is data minimization, and why does it clash with big data? (Collect only what's necessary vs maximize-and-explore.)
What concrete steps can engineers and companies take to "do the right thing"? (Minimize and purge data, self-regulate, educate users, build for accountability and humanity.)

Key terms glossary

Term	Meaning
Predictive analytics	Using data to predict and decide about individuals
Algorithmic prison	Systematic algorithmic exclusion from society
Bias amplification	A model learning and magnifying bias in its data
Protected traits	Attributes anti-discrimination law shields (race, gender, etc.)
Proxy variable	A feature correlated with a protected trait
Accountability	Being answerable for a decision and its errors
Explainability	Ability to explain how a decision was made
Feedback loop	A self-reinforcing cycle of cause and effect
Echo chamber	A loop showing users only agreeable content
Systems thinking	Analyzing the whole system, people included
Behavioral data	Data generated as a side effect of user activity
Surveillance	Tracking that serves the collector over the user
Consent	Permission that must be freely given and informed
GDPR	The EU's General Data Protection Regulation
Legitimate interest	A non-consent lawful basis for processing data
Network effects	A service's value rising with its user count
Privacy	The decision right over what to reveal to whom
Privacy settings	Controls over what other users can see
Data exhaust	Dismissive term for behavioral byproduct data
Data broker	A firm that buys, aggregates, and resells personal data
Toxic asset / hazardous material	Data framed as valuable but dangerous to hold
Data minimization	Collecting and retaining only what's necessary
Self-regulation	Industry restraint beyond legal requirement
Tragedy of the commons	Shared resource destroyed by unchecked self-interest

Connections across the book (where these themes were touched)

Balancing business needs with user needs; analytical vs operational systems → Chapter 1
Reliability, scalability, maintainability as the engineering goals this chapter scrutinizes → Chapter 2
Derived datasets combining whole-user-base data (hard to consent to) → Chapter 13
GDPR deletion, crypto-shredding, the limits of immutability → Chapter 12
Event-log provenance and auditability as accountability tools → Chapter 13
Anonymization, k-anonymity, and the difficulty of truly deleting data → Chapters 4, 12

This is the final chapter. The closing message: given the impact software and data have on the world, engineers carry a responsibility to work toward a world that treats people with humanity and respect.