Aa
Contrast
Text size

Blog / Data Out

Your Archive Strategy Is Save It Somewhere

19 June 2026

Six weeks to partially recover five years of biodiversity data from a 2008 Access database.

The project ended three years ago, and the data is somewhere, probably. Long term preservation is where finished projects quietly become inaccessible, file formats expire without warning, and the multi year analysis that would make your next grant application compelling turns out to need data nobody can open. This post is about keeping data ready for environmental data analytics years after the project that produced it.

What it looks like

A biodiversity monitoring programme had fifteen years of species distribution data. It was scientifically irreplaceable, the kind of long time series that makes climate impact analysis possible and makes reviewers lean forward. When a new research lead arrived and proposed a decade spanning analysis for a major EU application, she asked for the historical data.

The first five years were in Microsoft Access databases from 2008. Nobody on the team had Access installed, and the person who had built the databases had retired four years earlier. The IT department found the files on a backup drive, but could not guarantee that the drive was complete, because the backup system of that period had never been documented.

A consultant was hired to extract the data. It took six weeks and the recovery was partial. Half of the metadata existed only in table comments that the extraction tool did not capture, and the field notes that explained anomalies in the second and third years were in a proprietary format that had simply ceased to exist. The fifteen year dataset became a ten year dataset, not because the data was lost but because the means of reading it had quietly expired.

The application went in without the full time series. It scored well on science and lost on demonstrated longitudinal capability to a programme with five fewer years of data that had kept it in open, accessible formats throughout.

Four places where it goes wrong

There is no archive strategy beyond saving files somewhere. Most organisations treat archiving as the last step of a project, which means it never quite happens. The project ends and the team disperses. The data is saved on a shared drive, a hard disk or a cloud folder with a name that made sense at the time, and life moves on. Nobody documents what is there, nobody standardises the formats and nobody decides where the authoritative copy lives. Three years later someone needs the data and the search begins: which drive, which folder, which version was final, who knows the password to that account, and is this the complete dataset or a subset exported for one analysis? The data exists in the way that something exists when nobody can find it.

File formats become obsolete. Proprietary formats from vendors who change direction or get acquired become inaccessible on a timeline you do not control, and even common formats change in ways that break compatibility. Think of the Access database from 2008, the shapefile saved with a projection definition from a retired software version, the Excel file with macros written for Office 2003, or the binary format of a sensor maker that was acquired and renamed twice since the data was collected. None of this felt like a risk at the time, because the software was current and the format was standard. By the time it shows, the person who could have migrated it easily has gone, and what would have taken an afternoon in 2011 takes weeks later.

Storage fails and backups are incomplete. A hard drive fails, a cloud subscription quietly lapses after a pricing change, or a server is retired in an upgrade and someone assumes the data was moved when it was not. Storage failures are not rare. Anything kept in one place will eventually become unreadable, and what matters is whether a second copy exists before it does. Most environmental organisations have a backup system that was set up by someone who has left, documented by nobody, tested by nobody and assumed to be working because nothing has visibly broken.

Multi year analysis is out of reach. Even when old data can technically be opened, it is often unusable for long term analysis, because the means of combining it with current data does not exist. Project phases used different schemas, different variable names, different spatial reference systems and different time resolutions, and each phase assumed the next would sort it out. A programme with twelve years of monitoring may effectively hold four separate three year datasets that need months of harmonisation before any multi year question can be answered. The thing that makes long term monitoring valuable is locked behind an integration problem that nobody solved.

What fixes it

Data collected today should be as accessible and usable in fifteen years as it is now, without anyone having to do anything special.

Archive in open formats. CSV for tables, GeoJSON or GeoPackage for spatial data and plain text for documentation can be read by any software, on any system, in any decade. They need no licence and do not break when a company is acquired. Set a policy that when a project closes, its authoritative archive is exported to open formats with complete metadata. This is in addition to the working formats and not instead of them, and it is the insurance policy.

Make backups something you know works. Keep three copies at least: one on site, one off site and one in the cloud. Verify automatically that each backup finished, and test a restoration regularly to confirm the data can actually be recovered and not just stored. Document the procedure, so the system does not depend on one person knowing how it works. This is not complex infrastructure. It is a few scheduled scripts, cloud storage and a quarterly reminder to restore something, and it takes one setup afternoon.

Run a closure checklist for every finished project. Export to open formats, complete the metadata, check that the backup exists in three places, record where the data lives in a master register and note any known quirks or limits. It takes half a day and turns a future liability into a permanent asset. Treat archiving as the last deliverable of every project, as normal as the final report.

Build links between phases. This overlaps with the integration work described in the post on copy and paste: pipelines, one source of truth and harmonisation rules, applied here to continuity over time. Each new project phase is designed with an explicit connection to the previous one, and when a schema changes, the change is documented and the migration script is written at that moment, not years later when the person who made the change cannot be reached.

The programme rebuilt its archiving after that application. Every active dataset was exported to open formats with complete metadata, historical recovery was done where possible and a project closure checklist became standard. Two years later a new call appeared. The research lead pulled fifteen years of accessible, documented, searchable data in two days. The application included the full longitudinal analysis, and it was funded. The data had always been there. Now it was usable.

The pattern

Preservation failures feel like a resource problem: a dedicated data manager, better IT systems, more careful earlier teams. They are a prioritisation problem disguised as inevitability. Archiving is put off because it produces no immediate value. The project is over, the funder is satisfied and the team has moved on, so half a day on archiving looks like overhead next to active projects.

The cost arrives years later, when the archived data would have made an application competitive and instead takes weeks to partly recover, with the metadata that made it interpretable gone for good. Half a day at project close is worth many times its cost in recovered value, but it does not feel that way at the time. Your historical data is scientifically irreplaceable. Make sure it stays accessible.

Escaping Data Chaos

Also on LinkedIn

This article was first published in the Escaping Data Chaos series. The stories in the series are composites of real projects, anonymised.

Read it on LinkedIn

Recognise this problem?

One conversation to find out if we can help.

Book a free 15 minute call