Senior Software Developer

We are seeking a highly skilled and motivated Senior Software Developer to join our Data Engineering team. This role is responsible for designing and implementing complex, high-performance data integration solutions using Apollo Spark. The ideal candidate will have deep expertise in Spark internals, PySpark development, and performance tuning, with the ability to lead the creation of custom Spark connectors and optimize core data transformation logic at scale.

You will work with large, heterogeneous datasets, contribute to architecture and ingestion strategies, and ensure that Spark pipelines operate reliably, efficiently, and cost-effectively.

Responsibilities:

• Design, implement, and optimize high-volume data ingestion pipelines using Apollo Spark, integrating internal and external data sources through both standard and custom connectors.

• Lead the development of data integration strategies for moving, transforming, and loading large-scale, diverse datasets using Apollo Spark (PySpark) across cloud environments such as AWS, Azure, or GCP.

• Translate complex business and technical requirements into scalable PySpark notebooks and jobs with clear, maintainable structure.

• Apply advanced performance tuning techniques for Spark applications, including optimizing cluster configuration, partitioning strategies, and resource allocation.

• Port existing ETL/ELT logic and transformation processes from legacy systems into optimized PySpark-based implementations.

• Implement testing, monitoring, and robust error-handling mechanisms within Spark pipelines to ensure data integrity and operational reliability.