Description
We are seeking an experienced Lead Data Engineer with advanced expertise in PySpark and hands‑on experience building ETL pipelines, data lake architectures, and integrating data feeds on AWS.
You will handle both structured and unstructured data, ingesting information from a variety of on‑premises and enterprise sources such as SAP, Intelex, SQL, and OSI PI into AWS. This position provides the chance to work on large‑scale data projects and collaborate with diverse teams in a fast‑paced setting.
Responsibilities
Create, refine, and manage ETL pipelines using PySpark and AWS Glue Jobs to process extensive structured and unstructured datasets
Coordinate data workflows with Apache Airflow, ensuring dependable scheduling, dependency management, and effective error handling
Develop and sustain data feeds from on‑premises and enterprise systems into AWS data lake environments
Integrate with enterprise sources including SAP for ERP and operational data, Intelex for env...
You will handle both structured and unstructured data, ingesting information from a variety of on‑premises and enterprise sources such as SAP, Intelex, SQL, and OSI PI into AWS. This position provides the chance to work on large‑scale data projects and collaborate with diverse teams in a fast‑paced setting.
Responsibilities
Create, refine, and manage ETL pipelines using PySpark and AWS Glue Jobs to process extensive structured and unstructured datasets
Coordinate data workflows with Apache Airflow, ensuring dependable scheduling, dependency management, and effective error handling
Develop and sustain data feeds from on‑premises and enterprise systems into AWS data lake environments
Integrate with enterprise sources including SAP for ERP and operational data, Intelex for env...
Ready to Seal the Deal?
Submit your application today and take the next step in your career with EPAM Systems.
Apply for this Job