Data Management

Data management in a business web application is not a single decision made at launch. It is a running concern that affects migration accuracy, compliance obligations, operational reliability and the total cost of ownership. This guide covers the practical elements that business owners, operations managers and buyers of CRM systems, portals and internal tools need to understand when specifying, building or replacing a web application.

How to Plan Data Migration

Moving data from one system to another is where many projects encounter their most visible failures. A migration that appears straightforward on the surface often reveals inconsistencies, missing records and format mismatches once the work begins.

Audit the source data first

Before discussing timelines with a supplier, request a full export from the existing system and inspect a representative sample. Look for duplicate records, incomplete fields, entries that do not match expected formats and data that appears to belong to a different record type. The purpose is not to fix everything at this stage but to understand the scale of the problem.

Map source fields to target fields

Every field in the source system needs a destination in the new application, a clear transformation rule or an explicit decision to discard it. Discarded data should be documented with a reason, particularly if it includes personal data covered by UK GDPR. A field-by-field mapping spreadsheet, reviewed by someone who understands the business meaning of the data, is the minimum practical artefact.

Define transformation rules

Some data will need to be split, combined, reformatted or translated between value sets. For example, a legacy system might store customer status as a free-text field where the new application uses a fixed dropdown. Each transformation rule should be written down, reviewed and tested against real data before the full migration runs.

Plan validation and reconciliation

After a test migration, compare record counts, financial totals and spot-check individual records between source and target. If the source system contains 12,400 customer records and the target shows 12,150, the discrepancy must be explained before proceeding. Agree the validation criteria with your supplier in writing before the migration begins.

Agree a rollback position

If the migration fails or produces corrupted results, you need a clear path back to the previous working state. This means confirming that the source system remains available and unchanged until validation is complete, and that a database backup of the new system exists from before the migration ran.

Where this work commonly fails

• Assuming the data is clean because the old system appeared to work correctly.
• Leaving transformation logic undocumented so it cannot be reviewed or repeated.
• Setting a go-live date before validation is complete.
• Decommissioning the old system before the new data has been reconciled.

How to Create a Data Retention Policy for a Web App

A data retention policy sets out how long different categories of data are kept within an application and what happens when that period expires. It is a operational document with legal implications, and it should be prepared or reviewed by someone with current knowledge of UK GDPR and any sector-specific regulations that apply to your business.

Classify your data categories

Start by listing the types of data the application stores. Typical categories in a business system include customer contact details, transaction records, support correspondence, document uploads, system logs and user account metadata. Each category may have a different retention requirement.

Determine the retention driver for each category

Retention periods are driven by a combination of legal obligations, contractual requirements and legitimate business need. Some financial records must be kept for a minimum period by law. Customer correspondence may be retained for a shorter period based on the purpose for which it was collected. System logs might only be needed for a matter of weeks for operational troubleshooting. The policy should state the driver for each category rather than applying a single blanket period.

Define the deletion or anonymisation method

When a retention period expires, the data must be removed or rendered unidentifiable. Deletion means removing the record from the live database and, where relevant, from backups within a defined timeframe. Anonymisation means stripping identifiers so the data can no longer be linked to an individual. The policy should specify which method applies to each category and the expected timeframe for execution.

Build the policy into the application

A policy that exists only in a document is easy to overlook. The application should have mechanisms to enforce it: automated deletion jobs, reports showing data approaching its retention limit, and flags that prevent retention periods from being silently extended. Discuss with your supplier how these controls will be implemented and tested.

Review the policy periodically

Regulations change, business needs evolve and new data categories are added as the application develops. The policy should have a scheduled review date and a named person responsible for maintaining it.

Data Cleansing Before Migration

Data cleansing is the process of identifying and correcting or removing inaccurate, incomplete or irrelevant records before they are moved into a new system. Migrating dirty data does not save time; it embeds problems in the new application where they become harder and more expensive to fix.

Typical issues to look for

• Duplicates: Multiple records representing the same customer, supplier or transaction, often with slight variations in spelling or formatting.
• Stale records: Accounts that have been inactive for years but were never closed or archived.
• Inconsistent formatting: Postcodes entered with and without spaces, phone numbers with varying prefixes, dates in mixed formats.
• Orphaned records: Line items, notes or attachments that no longer link to a valid parent record.
• Default or placeholder values: Fields filled with "TBD", "N/A" or test data that was never cleaned out.

Decide who cleans what

Some cleansing can be automated: standardising postcodes, trimming whitespace, removing obvious test entries. Other decisions require business judgement: which of two duplicate customer records is the primary one, whether an old account should be merged or closed. Be clear with your supplier about where the boundary between automated cleansing and manual business decisions lies.

Cleanse in the source system where possible

If the source system is still operational, it is often safer to clean data there and then export, rather than exporting first and cleaning in an intermediate spreadsheet. This avoids introducing new errors during the export-clean-reimport cycle and keeps the source system in a defensible state if questions arise later.

Database Options for Web Applications

The database is where your application's data lives, and the choice of database type and hosting model affects cost, control, maintenance burden and the flexibility available to future developers.

Managed cloud databases

A managed database is hosted by a cloud provider who handles server provisioning, patching, backups and basic replication. You configure the database but do not manage the underlying infrastructure. This reduces the operational burden on your team but introduces a recurring cost and a dependency on the provider's service terms.

Self-hosted databases

A self-hosted database runs on servers you control, whether physical hardware or virtual machines you manage within a cloud environment. This gives you full control over configuration, patching schedules and data location, but requires someone with the skills to maintain, secure and back up the database. The total cost of ownership includes not just the server but the people and processes needed to keep it running.

Serverless or managed services tied to a platform

Some application platforms bundle database access into their service, where the database is abstracted behind the platform's API. This can simplify initial setup but may limit your ability to run custom queries, migrate to another provider or access the data directly outside the application.

Questions to put to a supplier

• Where is the data physically stored, and does that meet our compliance requirements?
• Who is responsible for applying security patches to the database software?
• How are backups configured, and how quickly can data be restored?
• What happens to our data if we stop using the service?
• Can we export the full dataset in a standard format at any time?

SQL vs NoSQL: What Business Owners Should Understand

The distinction between SQL and NoSQL databases is a technical one, but it has practical consequences for the kind of data your application can handle reliably and the flexibility available as requirements change.

SQL databases

SQL databases organise data into tables with fixed columns and defined relationships between them. They enforce a schema: every record must conform to the same structure. This rigidity is a strength when data has clear, stable relationships, such as customers, orders and line items. It makes it straightforward to run reports, join data across tables and maintain consistency. Most established CRM, ERP and accounting systems are built on SQL databases.

NoSQL databases

NoSQL databases store data in more flexible structures: documents, key-value pairs or graphs. Records in the same collection do not need to have identical fields. This flexibility is useful when the shape of the data changes frequently, when storing unstructured content like logs or user-generated metadata, or when the application needs to scale writes across many servers. NoSQL is common in real-time analytics, content platforms and systems that ingest large volumes of varied data.

What this means for a buying decision

If your application is a CRM, a customer portal, a document workflow system or an admin panel replacing spreadsheets, a SQL database is the conventional and usually appropriate choice. If a supplier proposes NoSQL for this kind of application, ask them to explain why the flexibility of NoSQL outweighs the consistency and reporting advantages of SQL in your specific case. The decision should be driven by the data model, not by the technology preference of the development team.

It is not irreversible

The database choice matters, but it does not lock you in permanently. Data can be migrated between database types, though the cost and complexity depend on how tightly the application code is coupled to the specific database's features. If future flexibility is a concern, raise it during discovery and ensure the application's data access layer is designed to make a future migration feasible.

Data Import and Export Formats

The formats your application supports for importing and exporting data determine how easily you can move information in and out, integrate with other systems and maintain control of your data over time.

CSV

CSV is the most widely supported format for tabular data. It is simple, human-readable in a text editor and compatible with every spreadsheet application. However, CSV does not natively support data types, character encoding is not always handled consistently, and it struggles with fields that contain commas, line breaks or special characters unless carefully escaped. For straightforward imports and exports of structured records, CSV is usually sufficient.

JSON

JSON represents data as nested key-value structures. It handles hierarchical and variable data well and is the standard format for API payloads. JSON is less convenient for non-technical users because it cannot be opened directly in a spreadsheet without conversion. It is the right choice when importing or exporting through an API or when the data structure does not fit neatly into rows and columns.

XML

XML is older and more verbose than JSON but is still used in some industries and legacy systems, particularly in financial services and government data exchanges. If your application needs to exchange data with systems that require XML, support for it is a practical necessity rather than a preference.

Practical considerations

• Ensure imports include clear error reporting: which row failed, why, and whether the rest of the file was processed.
• Agree on a character encoding standard, typically UTF-8, and document it.
• For large exports, check whether the system supports streaming or chunked downloads rather than attempting to generate the entire file in memory.
• Ask whether exports include all related data or just the top-level records, and understand the difference.

How to Plan Data Archiving

Archiving is the process of moving data that is no longer actively needed out of the primary application database while keeping it accessible for reference, compliance or legal purposes. It is distinct from deletion: archived data is preserved, just not in the main operational store.

When to archive rather than delete

Archive when the data may be needed for future reference but does not need to be readily available in the application's day-to-day operations. Common candidates include completed orders beyond their active return window, closed support tickets, expired user accounts and historical reports. Delete when there is no legitimate reason to keep the data at all, as per your retention policy.

Where archived data lives

Archived data can be stored in a separate database table within the same system, in a dedicated archive database, or exported to flat files in a secure storage location. The choice affects how easily the data can be searched and retrieved. A separate database table keeps the data queryable but still contributes to the overall database size. Flat files are cheaper to store but harder to search without additional tooling.

Access and retrieval

Before archiving, agree how archived data will be retrieved if it is needed. Who has access? What is the expected retrieval time? Is there a self-service mechanism, or does it require a request to the development team? If archived data might be needed for legal proceedings, the retrieval process should be documented and reliable.

Automation

Archiving should be an automated process, not a manual task performed when someone remembers. Define the criteria that trigger archiving, schedule the job to run at a low-traffic time and verify that it completes successfully through your application's operational checks.

Data Governance for Business Applications

Data governance is the framework of roles, rules and processes that ensures data within your application remains accurate, secure, compliant and fit for purpose over time. It is not a one-time setup but an ongoing operational responsibility.

Assign ownership

Every significant data category in the application should have a named owner within the business. This person is responsible for defining what correct data looks like, resolving disputes about data quality and approving changes to data structures. Without clear ownership, data quality degrades gradually and no one has the authority to fix it.

Define data quality rules

Data quality rules specify the standards that data must meet. These might include mandatory fields, valid value ranges, format constraints and cross-field consistency checks. Some of these are enforced by the application at the point of entry; others require periodic review because they depend on business context that the application cannot judge automatically.

Control access appropriately

Not everyone who uses the application needs to see or edit every piece of data. Access controls should be aligned with job roles and the principle of least privilege. Review access permissions periodically, particularly when staff change roles or leave the business.

Document data structures

As the application evolves, fields are added, renamed or repurposed. Without documentation, new team members and future developers must guess what a field means, which leads to errors and inconsistent use. A data dictionary that describes each field, its intended purpose, its valid values and its relationships to other fields is a practical governance tool.

Integrate governance into change processes

When a new feature is added that changes how data is stored or used, the governance implications should be considered as part of the requirements, not discovered afterwards. This includes changes to retention periods, new data categories, changes to access controls and modifications to data export formats.

How to Handle Data Subject Access Requests

Under UK GDPR, individuals have the right to request a copy of the personal data an organisation holds about them. A web application that stores personal data must be capable of fulfilling these requests within the statutory timeframe, which requires both process and system capability.

What the application needs to support

The system should allow an administrator to locate all records associated with a specific individual, including data that might be spread across multiple modules: customer records, order history, support tickets, document uploads, activity logs and any derived data such as notes added by staff. If the application cannot compile a complete picture from a single query, the process becomes slow, error-prone and difficult to repeat.

What to exclude

A subject access request does not require the disclosure of data about other individuals, legally privileged information or information that would prejudice the conduct of a criminal investigation. The application does not need to make these exclusions automatically, but the process for handling them should be defined and the staff involved should understand what must be redacted before the response is sent.

Practical steps

1. Verify the identity of the requester before disclosing any data.
2. Use the application's search and export capabilities to compile all relevant records.
3. Review the compiled data for third-party information that needs to be redacted.
4. Provide the data in a commonly used electronic format, such as CSV or PDF.
5. Document the request, the response and the timeline for your records.

System capability questions for a supplier

• Can the application pull together all data linked to a single individual in one operation?
• Does it export in a format suitable for a SAR response?
• Are activity logs and system-generated data included, or only user-entered fields?
• How long does a typical export take for a customer with a full history?

Referential Integrity: Why It Matters for Business Data

Referential integrity is the rule that every reference from one record to another must point to a valid, existing record. In practice, it means that an order line must always belong to a real order, a document must always be attached to a real customer, and a user assignment must always refer to a real user account.

What happens when it breaks

When referential integrity is not enforced, data becomes unreliable. An order might appear in a customer's history but have no line items, making its total meaningless. A report might count documents that no longer link to any customer, inflating figures. A user might be assigned to a task but their profile has been deleted, leaving no way to identify who is responsible. These problems accumulate silently and are difficult and expensive to clean up after the fact.

How it is enforced

Referential integrity is enforced at the database level through foreign key constraints, which prevent a record from being deleted or changed if other records depend on it. It can also be enforced at the application level through logic that checks dependencies before allowing a deletion. Database-level enforcement is more reliable because it cannot be bypassed by a bug in the application code or a direct database modification.

What to check with a supplier

• Are foreign key constraints used in the database schema?
• What happens when a user tries to delete a record that other records depend on: is the deletion blocked, are dependent records also deleted, or are they orphaned?
• Are there any parts of the system where referential integrity is deliberately not enforced, and if so, why?

Soft deletes and integrity

Many business applications use soft deletes, where a record is marked as inactive rather than being removed from the database. This preserves history and supports audit trails, but it introduces a complication for referential integrity: the application must consistently filter out soft-deleted records in queries, or inactive records will appear in results and reports. Ensure that soft-delete behaviour is defined clearly and applied consistently across all modules.

How to Plan for Data Growth

Data growth affects storage costs, query performance and the complexity of maintenance operations. Planning for it means making informed estimates, choosing a database and hosting model that can accommodate growth, and establishing the operational practices to manage it.

Estimate growth realistically

Start with current data volumes and known growth drivers: new customers per month, transactions per customer, documents uploaded per case, log entries per user action. Apply these to a 12-month and 36-month horizon. These figures are planning estimates, not predictions, and should be revisited as actual data becomes available. Label any worked example as illustrative rather than presenting it as a forecast.

Understand what grows fastest

In many business applications, the fastest-growing data categories are not the core business records but the supporting data: activity logs, audit trails, document versions, notification histories and temporary cache data. A system with 10,000 customer records might have millions of log entries. Understanding this distinction helps you focus storage planning and archiving policies where they matter most.

Choose a hosting model that accommodates growth

Some database hosting models scale storage automatically; others require manual intervention to add capacity. Understand the mechanism, the lead time and the cost curve. A database that is cheap at small volumes but becomes disproportionately expensive at larger ones can create an unpleasant surprise once the application is in production and switching is inconvenient.

Establish baselines and review them

Record the database size at launch and at regular intervals afterwards. Track which tables or collections are growing fastest and compare this against your estimates. If actual growth significantly exceeds the plan, you have time to adjust archiving policies, review storage costs or re-evaluate the hosting arrangement before it becomes a pressing problem.

Archiving as a growth management tool

As covered in the archiving section above, moving inactive data out of the primary database is the primary mechanism for controlling growth without losing access to historical information. An archiving strategy should be part of the initial data plan, not an afterthought introduced when the database becomes unwieldy.