Senior Data Engineer

Description

JOB DESCRIPTION

JOB TITLE:

Senior Data Engineer

DEPARTMENT:

500 – IT

REPORTS TO:

Chief of Information Officer

Job Summary

The Senior Data Engineer – Databricks is responsible for building and evolving PharmaLink’s enterprise data, analytics, and AI platform. This hands-on role spans the modern Databricks ecosystem, including data engineering, lakehouse architecture, analytics and dashboards, AI, Databricks Apps, and Lakebase.

The role requires strong engineering fundamentals, independent problem-solving, technical ownership, and the ability to design, build, and operate reliable, scalable production solutions.

A hands-on builder approach is essential, with a focus on developing solutions rather than simply administering platforms. The ideal candidate can work across data engineering, platform architecture, analytics, AI, and data applications; independently investigate complex problems; and take solutions from initial design through production.

Key Responsibilities

• Design, build, and operate enterprise data solutions on Databricks.
• Develop scalable ETL/ELT pipelines using Python, PySpark, Spark SQL, and Delta Lake.
• Design curated data models and reusable data products for analytics, applications, and AI.
• Implement Unity Catalog, security, lineage, access controls, and data-governance capabilities.
• Build and optimize Databricks SQL Warehouses and AI/BI Dashboards.
• Develop data and AI applications using Databricks Apps.
• Design and implement appropriate solutions using Databricks Lakebase.
• Enable AI/ML and Generative AI use cases using Databricks capabilities.
• Build automated data-quality, reconciliation, anomaly-detection, and monitoring processes.
• Integrate data from databases, SaaS platforms, APIs, files, and other enterprise systems.
• Optimize Spark workloads, SQL queries, pipelines, performance, and platform cost.
• Establish engineering standards for Git, CI/CD, testing, monitoring, and production deployment.
• Troubleshoot complex production and data issues and perform root-cause analysis.
• Partner with Data Governance, BI, application, and business teams to deliver trusted data products.

Requirements

QUALIFICATIONS:

• 8+ years of experience in data engineering, data platforms, or related engineering roles.
• 4+ years of hands-on Databricks experience in production environments.
• Advanced SQL, Python, PySpark, and Apache Spark skills.
• Strong experience with Delta Lake and Unity Catalog.
• Strong ETL/ELT, data modeling, and enterprise data architecture experience.
• Experience with Databricks SQL and AI/BI Dashboards.
• Familiarity with Databricks capabilities across AI/ML, Generative AI, Databricks Apps, and Lakebase.
• Strong relational database and data-engineering fundamentals.
• Experience with Git, CI/CD, automated testing, monitoring, and production support.
• Experience working with cloud-based enterprise data platforms.
• Must maintain a positive attitude in a high-performance team environment.
• Must pass background check and random drug screenings.
• Must be adaptable and flexible in dealing with a diverse work force

Preferred Qualifications

• Databricks on Microsoft Azure.
• Azure Data Factory.
• Microsoft SQL Server.
• Fivetran or similar CDC/ELT technologies.
• Power BI.
• Databricks Workflows.
• MLflow and AI/model lifecycle capabilities.
• API and application development experience.
• Azure DevOps or GitHub-based CI/CD.
• Databricks professional certification.
• Experience in healthcare, life sciences, or another regulated industry.

Physical Requirements

• Requires light physical activity performing non-strenuous daily activities of an administrative nature (i.e., sustained phone, headset, and keyboard use) as well as periods of prolonged sitting.