Data Engineer Databricks

Project Description – DQX-Based Data Quality Monitoring We are looking for an experienced senior data engineer / Databricks developer to support the design and implementation of a DQX-based Data Quality Monitoring capability. The objective is to build a scalable solution that enables business users and data owners to monitor, understand, and act on data quality where it matters most. The solution should create a clear link between data input, data usage, and business-critical data quality rules, allowing data owners to define and enforce quality expectations across key data domains. The capability should support business users in self-managing data quality oversight through dashboards, trending, and rule-based monitoring. This includes visibility into data quality performance over time, identification of recurring issues, and transparency on which data points are most critical based on documented business usage. The initial scope will focus on selected Study Management data domain, starting with Study Personnel and Milestones, with the ambition to design the solution so it can scale to additional Clinical data areas over time. The solution is expected to be built natively on Databricks, leveraging relevant data quality frameworks such as DQX, and should support a governed operating model where data quality rules are owned and endorsed by relevant data owners. The expected outcome is a robust and scalable Data Quality Monitoring solution that improves transparency, strengthens data ownership, supports audit readiness, and enables proactive management of data quality issues before they create downstream impact. What we offer: Remote role B2B Contract Rate: 41 euro/h+ VAT Main Responsibilities The consultant is expected to contribute to the following: Design and implementation of a Databricks-native Data Quality Monitoring framework (DQX). Configuration and implementation of DQX-based data quality rules. Development of data pipelines and data models supporting monitoring and reporting. Dashboards and trending views for business users and data owners. Linkage between data quality rules, data usage, and critical data points. Documentation of technical design, rule logic, and operating model. Support for scaling the solution to additional Clinical data domains. Key Requirements Strong hands-on experience with Databricks. Experience with Spark, SQL, Delta Lake, and Python. Experience designing and implementing data pipelines / ETL-ELT. Experience with data quality frameworks, preferably DQX. Understanding of data governance, data ownership, and rule-based data quality monitoring. Ability to translate business requirements into scalable technical solutions. Experience working in complex enterprise environments. Nice to Have Experience with regulated or compliance-heavy environments. Experience with clinical operations, trial operations, or life sciences data. Experience with dashboarding and business-facing data quality reporting. Azure DevOps / CI-CD experience. Experience with integrations, APIs, or downstream system connectivity. Snowflake experience. Understanding of Veeva or related clinical systems.