Escaping Data Chaos
Also on LinkedIn
This article was first published in the Escaping Data Chaos series. The stories in the series are composites of real projects, anonymised.
Blog / Data In
27 March 2026
One team entered a sea urchin, Paracentrotus lividus, as "spiky thing (purple)". An infrastructure problem, not a people problem.
Data chaos does not announce itself. It builds quietly, one field team at a time, until the day you try to merge three years of observations and find fifteen incompatible puzzles with no picture on the box. Collection and intake is where the monster is born. This post looks at it through ecological data management: what goes wrong when every team invents its own way of recording, and what makes the result consistent without forcing everyone into one rigid system.
A coastal biodiversity programme I know had seven field teams working across four regions. Each team had developed its own way of collecting data over the years, based on what worked locally: Excel sheets, paper datasheets, photos sent over WhatsApp, voice memos transcribed into Word.
For a major EU reporting deadline they gave two junior researchers the job of harmonising everything. Six weeks later the datasets were still not merged. The sea urchin Paracentrotus lividus was in one file, P. lividus in another and "sea urchin" in a third, and on one memorable paper datasheet it was "spiky thing (purple)". Same organism, four formats, thousands of records, and every one needed manual reconciliation. The director told me: "We didn't have a data problem. We had seven data problems that had never been introduced to each other."
Collection formats are fragmented. When each team invents its own system, you get local efficiency and organisational chaos. A team's Excel sheet works perfectly for that team and breaks the moment anyone else needs it. Multiply by seven teams, three years and four regional coordinators who each tidied the files differently, and you have a dataset that resists analysis by design. Nobody is at fault. The system never had standards.
Metadata is missing or inconsistent. Who collected this, and when? With which equipment, which protocol and in what weather? In most datasets I have seen, the answer is somewhere, probably, in an email from 2021. Metadata is the context that turns raw numbers into defensible science, and it is rarely captured because nobody built the step into the workflow. Accurate observations can become scientifically worthless when their provenance cannot be recovered.
Validation comes late. Errors are cheap to fix at entry and very expensive to fix six months later. Most teams check at the analysis stage, or during a stakeholder presentation. By then a misconfigured GPS has put a hundred sampling points in the wrong ocean and a unit error has run through every calculation that touched the variable. The question is no longer how to fix it, but how much has to be thrown away.
There is no intake architecture. Data arrives in fifteen formats because nothing accepts anything and returns one consistent result. Each new source needs a new manual process: a new partner means a new format to negotiate, and a new field season means a new template invented from scratch. Every intake is an improvisation, and knowing how to do it lives in one person's head.
The goal is not to force everyone into one rigid system. Field teams have real constraints, and paper datasheets exist because sometimes you are in a mangrove at low tide without a waterproof tablet. The goal is consistency at the point of entry, not uniformity at the point of collection.
Standard mobile forms with enforced naming and automatic metadata handle most of the problem. Time, GPS position, user and protocol version are captured by the form, and none of them depends on the memory or discipline of the field team. KoBoToolbox, Survey123 or even a well designed Google Form beats fifteen separate Excel templates.
An intake pipeline handles the rest. One pipeline accepts data in any format and converts it to your standard schema on the way in. Paper datasheets are digitised into the same structure as the mobile forms, partner data is harmonised automatically, and old Excel files are migrated once and then retired. The effort goes into building the system, not into reconciling formats forever.
Add checks at entry, not downstream. Set a range on every numeric field, check species names against a reference list and coordinates against the study area, and alert the field team at once, while they can still resample. The misconfigured GPS is caught on day one, not in the presentation six months later.
The programme built a standard mobile collection system with automatic metadata capture, and a simple pipeline that normalised incoming data whatever its source. Its next integration cycle took days instead of weeks. The junior researchers got their weekends back, and the director stopped dreading the reporting season.
Collection chaos feels like a people problem: teams that ignore protocols, individuals who improvise, partners who refuse to standardise. It is an infrastructure problem wearing a people costume. When the easiest thing is to invent your own format, people invent their own format. When the easiest thing is a standard form that handles everything, people use the form. You do not fix this with stricter enforcement. You fix it by making the right thing easier than the wrong thing.
Once intake is systematic, the downstream problems get lighter too: the analysis rebuilds, the reporting marathons and the proposal scrambles, because they were partly symptoms of poor material arriving at the source. Good environmental data collection methods start there. Fix the intake and you start fixing everything.
Escaping Data Chaos
This article was first published in the Escaping Data Chaos series. The stories in the series are composites of real projects, anonymised.
Keep reading

Start here

Data In

Data Work

Data Out

Data Work
From the case studies

Fisheries, EU research

Marine litter, circular economy

Marine litter, citizen science
Case studies
Where to go next