Databricks Engineer with Capital Market/Banking Experience with Azure

Job Title: Databricks Engineer with Capital Market/Banking Experience with Azure
Location: Mississauga, ON (Hybrid, 3 days in a week from office)

Job Summary:

We are seeking an experienced Data Engineer to design, develop, and maintain scalable data solutions. The ideal candidate will have a strong background in Python, PySpark, SQL, Snowflake, Data Factory, Data Lakes, and OpenShift, with a passion for building reliable data pipelines and enabling data-driven decision-making.

Key Responsibilities:

• Develop, construct, test, and maintain scalable data pipelines and ETL/ELT workflows using Python, PySpark, and SQL.
• Manage and optimize data storage solutions in Snowflake and on Data Lakes.
• Integrate data from multiple sources and ensure data quality, integrity, and security.
• Design and implement automated workflows using Azure Data Factory or other orchestration tools.
• Deploy and manage containerized applications and services on OpenShift.
• Collaborate with Data Scientists, Analysts, and Stakeholders to understand data requirements.
• Monitor and troubleshoot data pipelines and resolve performance bottlenecks.
• Maintain documentation of data architecture, workflows, and best practices.
• Ensure compliance with data governance and security policies.

Qualifications & Skills:

• Bachelor's or Master's degree in Computer Science, Data Engineering, or related field.
• Proven experience as a Data Engineer or in similar roles.
• Strong proficiency in Python, with frameworks like PySpark.
• Expertise in SQL and experience with Snowflake Data Warehouse.
• Hands-on experience with data orchestration using Azure Data Factory or similar tools.
• Knowledge of Data Lakes architecture and management.
• Experience deploying and managing containerized applications on OpenShift or Kubernetes.
• Familiarity with cloud platforms (Azure, AWS, GCP).
• Understanding of data security, governance, and compliance best practices.
• Strong problem-solving skills and attention to detail.
• Excellent communication and teamwork abilities.

Preferred Skills:

• Experience with other big data tools (e.g., Kafka, Hadoop).
• Knowledge of CI/CD pipelines and DevOps practices.
• Familiarity with other cloud data services and tools.