About the role
Sr Data Engineer-Noida(Hybrid)
We are looking for a highly skilled Senior Data Engineer to design, build, and maintain scalable data pipelines and integrations within our Databricks-based data platform. The role will focus on expanding our data ecosystem by onboarding new data sources, integrating enterprise applications, and delivering reliable, secure, and analytics-ready datasets.
The ideal candidate will have strong hands-on expertise in Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake, along with experience building production-grade data pipelines and enterprise integrations. Working knowledge of AWS infrastructure and cloud services is important for collaborating with Engineering and other platform teams on connectivity, deployments, and evolving infrastructure requirements.
The candidate will report directly to the Senior Manager - Business Insights & Analytics and collaborate with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders to deliver scalable and reliable data solutions.
Responsibilities
- Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake.
- Design and implement a medallion architecture (Bronze, Silver, and Gold layers) to deliver trusted, well-structured, analytics-ready datasets.
- Build and maintain integrations with enterprise applications, REST APIs, databases, and other source systems
- Developing reusable ingestion frameworks to accelerate onboarding of new data sources.
- Build reliable batch and near-real-time ingestion pipelines with incremental loads, CDC, schema evolution and error recovery.
- Configure and manage Databricks Workflows and implement ingestion and transformation pipelines using Auto Loader and Lakeflow Spark Declarative Pipelines
- Optimize Apache Spark and Delta Lake workloads using execution-plan analysis, efficient join strategies, data-skew handling, partitioning, and SQL query tuning to improve performance and control compute costs.
- Implement Unity Catalog governance, including access controls, lineage, auditing, and sensitive-data protection through masking and row-level security.
- Maintain and support existing data workflows running on Amazon Redshift and Amazon EC2, collaborating with Engineering and IT/Cloud teams on connectivity, IAM permissions, S3 access, deployments, and operational issues.
- Follow engineering best practices, including version control, code reviews, automated testing, CI/CD, and technical documentation.
- Partner with Data Analytics and business stakeholders to translate requirements into trusted, well-documented, analytics-ready datasets.