DevOps Engineer / 1

We are seeking a DevOps Engineer to set up, configure, and operationalize a new Databricks environment with a primary focus on business intelligence (BI), analytics, and data engineering workflows.

Working closely with our ML Ops Engineers, you will ensure Databricks is fully prepared for both traditional BI/data processing use cases and AI workloads - including secure access for data analysts, seamless integration with downstream AI and BI tools, and optimized data pipelines.

Key Responsibilities

Environment Setup & Configuration

• Deploy and configure Databricks workspace(s) for multi-team usage.

• Manage shared clusters, automated job clusters, and interactive clusters.

• Configure role-based permissions aligned with governance policies.

Data Integration Enablement

• Establish secure connections to on-prem and cloud data sources (SQL Server, Data Lake, APIs).

• Build shared ingestion pipelines for BI and analytics teams.

• Automate daily/weekly data refresh schedules.

Connectivity for BI Tools

• Integrate Databricks with BI platforms (e.g., Power BI).

• Optimize query connectors and JDBC/ODBC configurations.

Operational Excellence

• Implement monitoring and logging for jobs and pipelines.

• Define backup and disaster recovery processes.

• Apply cost tracking and optimization practices for cluster usage.

Automation & CI/CD

• Set up CI/CD pipelines for data engineering code.

• Manage deployment workflows for notebooks, SQL queries, and data models.

Collaboration

• Partner with ML Ops Engineers to align infrastructure for ML and BI use cases.

• Work with Data Engineers to maintain central data sources.

• Collaborate with security teams to enforce access controls.

Governance

• Enforce GDPR and internal compliance rules.

• Maintain workspace auditing and logging.

• Document environment setup and operational procedures.