Lead Software Engineer

Data Quality Automation Analyst (1) (Databricks-Centric)

Role Overview

Design and automate data validation within Databricks pipelines to ensure accurate migration and ongoing data reliability.

Key Responsibilities

Build automated data quality checks in PySpark

Develop Reconciliation Framework Between Legacy SQL outputs

New Databricks outputs

Implement rule-based validation for healthcare datasets

Automate profiling and anomaly detection

Integrate quality checks into pipeline workflows

Build DQ scorecards and dashboards

Support cutover validation

Required Skills Strong SQL & PySpark

Experience with Databricks validation frameworks

Experience implementing automated DQ checks

Strong reconciliation expertise