Data Health as Service: Graph-Based Data Quality Monitoring and Observability

The Critical Challenge of Data Quality

In the era of data-driven decision making, organizations face a fundamental question that often goes unanswered: Is our data ready for consumption? Before business users, analysts, or machine learning models can leverage data for insights, they need confidence that the data is accurate, timely, and trustworthy.

Data quality is not a binary state. It exists on a spectrum influenced by multiple factors:

Data accuracy: Does the data reflect reality?
Data timeliness: Is the data current enough for the use case?
Data completeness: Are all required fields populated?
Data lineage: Can we trace the data's origin and transformations?
Data integrity: Are relationships and constraints maintained?

Traditional approaches to data quality monitoring often fall short because they treat these dimensions in isolation. What organizations need is a holistic view that considers how these factors interact and compound to determine overall data health.

Introducing Data Health as Service

Data Health as Service is an innovative approach that leverages graph database technology to provide intelligent data quality monitoring. Rather than relying solely on mathematical calculations, this service uses graph-based inference to deduce data health status from the relationships and dependencies within your data ecosystem.

The core innovation is the Index of Readiness metric, defined as:

Index of Readiness = 1 / (sum(score metrics influencing data quality) + score(data lineage) + score(data integrity))

A score closer to zero indicates healthier data, while higher scores signal potential issues requiring attention.

Why Graph Databases for Data Quality?

Traditional relational databases struggle to model the complex web of dependencies in modern data platforms. Consider a typical enterprise scenario:

Multiple ETL pipelines feed into a data warehouse
Each pipeline has upstream dependencies on source systems
Data products are generated from warehouse tables
Reports and dashboards consume these data products
Each component can fail or degrade independently

This web of relationships is inherently a graph problem. By modeling data health in Neo4j, we gain:

Natural Relationship Modeling

Graph databases excel at representing interconnected data. Dependencies between data sources, transformations, and outputs map naturally to nodes and edges.

Traversal-Based Analysis

Graph queries can efficiently traverse paths to identify:

Missing dependencies
Broken pipeline links
Cascading failure impacts
Data lineage chains

Non-Mathematical Inference

Beyond simple calculations, graph patterns enable qualitative health assessments. A node with missing relationships indicates a health problem, even without explicit error metrics.

Architecture Overview

The Data Health Service follows a clean microservices architecture built on Spring Boot and Neo4j:

+------------------+     +------------------+     +------------------+
|   Data Sources   | --> |  Health Service  | --> |    Dashboards    |
|   (ETL, APIs)    |     |  (Spring Boot)   |     |   (Monitoring)   |
+------------------+     +------------------+     +------------------+
                                  |
                                  v
                         +------------------+
                         |      Neo4j       |
                         |  Graph Database  |
                         +------------------+

Core Domain Model

The service defines three primary entities that form the health monitoring graph:

DataHealth Node

The central enrollment point for data contributors. This node serves as the root of the health assessment graph.

@Node("DataHealth")
public class DataHealth {
    @Id
    private String id;
    private String name;
    private String description;
    private LocalDateTime assessmentTime;
    private Double readinessIndex;

    @Relationship(type = "HAS_CONTRIBUTOR")
    private List<Contributor> contributors;
}

Contributor Node

Represents any component that contributes to data quality, typically ETL pipelines, data ingestion processes, or transformation jobs.

@Node("Contributor")
public class Contributor {
    @Id
    private String id;
    private String name;
    private String type;
    private Double qualityScore;
    private LocalDateTime lastUpdated;
    private String status;

    @Relationship(type = "PRODUCES")
    private List<Report> reports;

    @Relationship(type = "DEPENDS_ON")
    private List<Contributor> dependencies;
}

Report Node

Final output nodes representing consumable data products like analytical cubes, batch reports, or real-time dashboards.

@Node("Report")
public class Report {
    @Id
    private String id;
    private String name;
    private String type;
    private Double integrityScore;
    private LocalDateTime generatedAt;
    private Boolean isValid;
}

Graph Relationships and Health Inference

The power of this architecture lies in the relationships:

// Creating a health assessment graph
CREATE (dh:DataHealth {
    id: 'health-001',
    name: 'Sales Analytics Health',
    assessmentTime: datetime()
})

CREATE (etl:Contributor {
    id: 'etl-sales',
    name: 'Sales ETL Pipeline',
    type: 'ETL',
    qualityScore: 0.95,
    status: 'HEALTHY'
})

CREATE (cube:Report {
    id: 'report-sales-cube',
    name: 'Sales Analytics Cube',
    type: 'OLAP_CUBE',
    integrityScore: 0.98,
    isValid: true
})

CREATE (dh)-[:HAS_CONTRIBUTOR]->(etl)
CREATE (etl)-[:PRODUCES]->(cube)

Detecting Health Issues Through Graph Patterns

The service uses Cypher queries to detect various health conditions:

Missing Dependencies:

// Find contributors with missing upstream dependencies
MATCH (c:Contributor)
WHERE NOT (c)-[:DEPENDS_ON]->()
AND c.type IN ['ETL', 'TRANSFORMATION']
RETURN c.name AS orphanedContributor

Broken Lineage:

// Find reports without valid contributors
MATCH (r:Report)
WHERE NOT ()-[:PRODUCES]->(r)
RETURN r.name AS unreachableReport

Cascading Impact Analysis:

// Find all downstream impacts of a failing contributor
MATCH path = (failed:Contributor {status: 'FAILED'})-[:PRODUCES|DEPENDS_ON*]->(affected)
RETURN affected.name AS impactedComponent,
       length(path) AS distance
ORDER BY distance

Calculating the Index of Readiness

The health service computes the Index of Readiness through a combination of metric aggregation and graph analysis:

@Service
public class HealthCalculationService {

    public Double calculateReadinessIndex(String healthId) {
        // Aggregate quality metrics from contributors
        Double qualityScore = contributorRepository
            .findByHealthId(healthId)
            .stream()
            .mapToDouble(Contributor::getQualityScore)
            .average()
            .orElse(0.0);

        // Calculate lineage score based on graph completeness
        Double lineageScore = calculateLineageScore(healthId);

        // Calculate integrity score from reports
        Double integrityScore = calculateIntegrityScore(healthId);

        // Index of Readiness formula
        Double totalScore = qualityScore + lineageScore + integrityScore;
        return totalScore > 0 ? 1.0 / totalScore : Double.MAX_VALUE;
    }

    private Double calculateLineageScore(String healthId) {
        // Graph traversal to assess lineage completeness
        Long totalNodes = graphClient.countNodes(healthId);
        Long connectedNodes = graphClient.countConnectedNodes(healthId);

        return connectedNodes.doubleValue() / totalNodes.doubleValue();
    }
}

Spring Boot Integration

The service exposes RESTful APIs for health assessment operations:

@RestController
@RequestMapping("/api/v1/health")
public class DataHealthController {

    private final DataHealthService healthService;
    private final HealthCalculationService calculationService;

    @PostMapping("/enroll")
    public ResponseEntity<DataHealth> enrollDataSource(
            @RequestBody DataHealthRequest request) {
        DataHealth health = healthService.enroll(request);
        return ResponseEntity.created(URI.create("/health/" + health.getId()))
            .body(health);
    }

    @GetMapping("/{id}/readiness")
    public ResponseEntity<ReadinessResponse> getReadinessIndex(
            @PathVariable String id) {
        Double index = calculationService.calculateReadinessIndex(id);
        return ResponseEntity.ok(new ReadinessResponse(id, index));
    }

    @GetMapping("/{id}/lineage")
    public ResponseEntity<LineageGraph> getLineageGraph(
            @PathVariable String id) {
        LineageGraph graph = healthService.getLineageGraph(id);
        return ResponseEntity.ok(graph);
    }

    @PostMapping("/{id}/contributors")
    public ResponseEntity<Contributor> addContributor(
            @PathVariable String id,
            @RequestBody ContributorRequest request) {
        Contributor contributor = healthService.addContributor(id, request);
        return ResponseEntity.ok(contributor);
    }
}

Deployment Guide

Prerequisites

Java 14 or higher
Docker for Neo4j deployment
Maven for build management

Starting Neo4j

Deploy Neo4j using Docker:

docker run -d \
    --name neo4j-health \
    -p 7474:7474 \
    -p 7687:7687 \
    -e NEO4J_AUTH=neo4j/password \
    neo4j:latest

Application Configuration

Configure the Spring Boot application in application.yml:

spring:
  data:
    neo4j:
      uri: bolt://localhost:7687
      username: neo4j
      password: password

server:
  port: 8080

health:
  calculation:
    threshold:
      excellent: 0.1
      good: 0.3
      warning: 0.6
      critical: 1.0

Running the Service

# Build the application
mvn clean package

# Run the application
java -jar target/data-health-service.jar

Or using the Spring Boot Maven plugin:

mvn spring-boot:run

Observability and Monitoring

The Data Health Service integrates with standard observability tools:

Health Endpoints

Spring Boot Actuator provides operational insights:

management:
  endpoints:
    web:
      exposure:
        include: health, metrics, info
  health:
    neo4j:
      enabled: true

Custom Metrics

Export data health metrics to Prometheus:

@Component
public class HealthMetrics {

    private final MeterRegistry registry;

    @Scheduled(fixedRate = 60000)
    public void recordHealthMetrics() {
        Map<String, Double> readinessIndexes = healthService.getAllReadinessIndexes();

        readinessIndexes.forEach((id, index) -> {
            registry.gauge("data.health.readiness.index",
                Tags.of("health_id", id),
                index);
        });
    }
}

Alerting Rules

Configure alerts based on readiness thresholds:

# Prometheus alerting rules
groups:
  - name: data-health
    rules:
      - alert: DataHealthDegraded
        expr: data_health_readiness_index > 0.6
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Data health degraded for {{ $labels.health_id }}"

      - alert: DataHealthCritical
        expr: data_health_readiness_index > 1.0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Critical data health issue for {{ $labels.health_id }}"

Real-World Use Cases

ETL Pipeline Monitoring

Track the health of complex ETL pipelines by modeling each stage as a contributor:

CREATE (source:Contributor {name: 'Source System Extract', type: 'EXTRACT'})
CREATE (transform:Contributor {name: 'Data Transformation', type: 'TRANSFORM'})
CREATE (load:Contributor {name: 'Warehouse Load', type: 'LOAD'})
CREATE (cube:Report {name: 'Analytics Cube', type: 'OLAP'})

CREATE (transform)-[:DEPENDS_ON]->(source)
CREATE (load)-[:DEPENDS_ON]->(transform)
CREATE (load)-[:PRODUCES]->(cube)

Data Mesh Health Dashboard

Monitor health across multiple data domains in a Data Mesh architecture:

// Query health across all domains
MATCH (dh:DataHealth)-[:HAS_CONTRIBUTOR]->(c:Contributor)
WITH dh.name AS domain,
     avg(c.qualityScore) AS avgQuality,
     count(c) AS contributorCount
RETURN domain, avgQuality, contributorCount
ORDER BY avgQuality DESC

Regulatory Compliance

Use lineage tracking for compliance with data governance regulations:

// Trace complete lineage for audit
MATCH path = (source:Contributor)-[:DEPENDS_ON|PRODUCES*]->(target:Report)
WHERE target.name = 'Regulatory Report'
RETURN path

Extending the Model

The graph-based approach allows easy extension with new metrics:

Adding Custom Quality Dimensions

@Node("QualityMetric")
public class QualityMetric {
    @Id
    private String id;
    private String dimension; // accuracy, timeliness, completeness
    private Double score;
    private LocalDateTime measuredAt;
}

Temporal Health Tracking

// Create time-series health snapshots
CREATE (snapshot:HealthSnapshot {
    healthId: 'health-001',
    timestamp: datetime(),
    readinessIndex: 0.15,
    qualityScore: 0.92,
    lineageScore: 0.95
})

Conclusion

Data Health as Service represents a paradigm shift in data quality monitoring. By leveraging graph database technology, organizations can:

Visualize dependencies across complex data ecosystems
Detect issues early through relationship-based inference
Trace lineage for compliance and debugging
Calculate holistic health using the Index of Readiness metric
Scale monitoring across data mesh architectures

The combination of Spring Boot's robust service framework and Neo4j's graph capabilities creates a powerful platform for ensuring data reliability. As organizations continue to make critical decisions based on data, having confidence in that data's health becomes not just valuable but essential.

The Index of Readiness provides a single metric that encapsulates the multidimensional nature of data quality. When this index approaches zero, stakeholders can trust that their data is ready for consumption. When it rises, the graph structure immediately reveals where problems lie and what downstream impacts to expect.

This post is based on the data-health-service project, which demonstrates graph-based data quality monitoring using Spring Boot and Neo4j.