Sign up to save this job, get alerts, and apply with an optimized CV.
Azure Data Engineer - ADB, ADF, API integration, Spark, Kafka
From 10000.00
Job description
Comments for Suppliers: Project-Specific Requirements:
Data Engineer will be responsible for creating a pipeline, ETL. Ingesting data from multiple sources like API, SharePoint, SAB.
Communicate with stakeholders to gather and clarify requirements.
Job Description: Data Engineer
We are looking for a Data Engineer who can work as individual contributor within Team, should have good communication skills. Need to involve in requirement discussion and responsible for designing, building and maintaining the infrastructure that enables data collection, storage and processing.
Key Responsibilities:
· Data pipeline development
o Design, build and maintain scalable ETL (Extract, Transform, Load)
o Automate data ingestion from multiple sources (APIs, database, files, etc.).
· Data Storage & Management
o Develop and optimize data storage solutions (e.g. datalakes, warehoures)
o Ensure data integrity, consistency and security across systems.
o Work with Azure cloud storage
· Data Processing & Transformation
o Process and clean raw data for analysis and visualizationreport
o Implement real-time and batch data processing workflows using Spark, Kafka
· Database Management
o Design and optimize relations (SQL) and NoSQL databases.
o Improve database performance indexing, and query optimization
· Data Governance & Security
o Implement Data quality, validation and governance standards.
o Ensure compliance with data privacy
Requirements:
· Strong proficiency in Databricks, Azure Data Factory and API integration.
· Proficiency in programming languages such as Python, SQL and PySpark.
· Familiarity with data modeling, ETL processes and data governance.
· Design, develop and maintain robust and scalable data pipelines using Databricks, Azure Data Factory
· Familiarity with DevOps practices and tools for CI/CD pipelines.
· Experience with Azure cloud platform
· Integrate data from various APIs and external sources to enrich and expand datasets.
· Collaborate with cross-functional teams to understand data requirements and translate them into technical solutions.
· Implement data integration processes to collect and store structured and unstructured data from multiple sources.
· Ensure data quality and integrity throughout the data lifecycle.
· Excellent attention to detail and ability to work independently.
· Strong communication and collaboration skills.
Data Engineer will be responsible for creating a pipeline, ETL. Ingesting data from multiple sources like API, SharePoint, SAB.
Communicate with stakeholders to gather and clarify requirements.
Job Description: Data Engineer
We are looking for a Data Engineer who can work as individual contributor within Team, should have good communication skills. Need to involve in requirement discussion and responsible for designing, building and maintaining the infrastructure that enables data collection, storage and processing.
Key Responsibilities:
· Data pipeline development
o Design, build and maintain scalable ETL (Extract, Transform, Load)
o Automate data ingestion from multiple sources (APIs, database, files, etc.).
· Data Storage & Management
o Develop and optimize data storage solutions (e.g. datalakes, warehoures)
o Ensure data integrity, consistency and security across systems.
o Work with Azure cloud storage
· Data Processing & Transformation
o Process and clean raw data for analysis and visualizationreport
o Implement real-time and batch data processing workflows using Spark, Kafka
· Database Management
o Design and optimize relations (SQL) and NoSQL databases.
o Improve database performance indexing, and query optimization
· Data Governance & Security
o Implement Data quality, validation and governance standards.
o Ensure compliance with data privacy
Requirements:
· Strong proficiency in Databricks, Azure Data Factory and API integration.
· Proficiency in programming languages such as Python, SQL and PySpark.
· Familiarity with data modeling, ETL processes and data governance.
· Design, develop and maintain robust and scalable data pipelines using Databricks, Azure Data Factory
· Familiarity with DevOps practices and tools for CI/CD pipelines.
· Experience with Azure cloud platform
· Integrate data from various APIs and external sources to enrich and expand datasets.
· Collaborate with cross-functional teams to understand data requirements and translate them into technical solutions.
· Implement data integration processes to collect and store structured and unstructured data from multiple sources.
· Ensure data quality and integrity throughout the data lifecycle.
· Excellent attention to detail and ability to work independently.
· Strong communication and collaboration skills.
Required skills
Sign up to apply
Create a free account to apply for this job and get access to:
- AI-powered CV optimization for this specific job
- Save jobs and create custom alerts
- See your CV match score for each job
Company information
- Company
- Astra North Infoteck Inc.
- Location
-
Canada, Alberta, Calgary region, Calgary
Canada - Posted
- 1 year ago
Interested in this position?
Create your free account and tailor your CV to match this job.