LEAD DATA ENGINEER - Databricks

Happiest Minds Technologies

Not disclosedBengaluru (On-site)
Posted: 4d agoOpening: 1

Skills Required

Key skills are marked withstar
  • Key skillData Engineering
  • Key skillTechnical Leadership
  • Key skillData Architecture
  • SQL
  • Python
  • Apache Spark
  • Data Pipelines
  • Airflow
  • Cloud Platforms
  • Data Modeling
  • Kafka
  • Data Governance
  • Code Review
  • Mentoring
  • Stakeholder Management

Job Description

Data Engineer Responsibilities: Design, develop, and maintain scalable data pipelines using Databricks and Azure Data Factory. Implement data processing solutions using Spark and Delta Tables to optimize performance and reliability. Collaborate with data scientists and analysts to understand data requirements and deliver high-quality datasets. Manage and optimize data storage solutions in Azure Data Lake and ensure data accessibility through Unity Catalog. Monitor and troubleshoot data pipeline performance and implement improvements as necessary. Document data processes and maintain clear communication with stakeholders regarding project status and challenges. Stay updated with the latest industry trends and technologies to continuously improve data engineering practices. Mandatory Skills: Strong proficiency in SQL for data manipulation and querying. Expertise in Python for data processing and automation. Hands-on experience with Databricks is a deal breaker. Familiarity with Azure Data Factory for orchestrating data workflows. Experience with Spark for big data processing. Knowledge of Delta Tables for efficient data storage and management. Understanding of Unity Catalog for data governance and management. Experience with Azure Data Lake for data storage solutions. Familiarity with Azure Synapse for data integration and analytics. Preferred Skills: Experience with data modeling and ETL processes. Knowledge of data warehousing concepts and best practices. Familiarity with machine learning concepts and tools. Experience with version control systems such as Git. Strong problem-solving skills and attention to detail.

Similar Jobs (35141)

Data Engineer

Accenture

Not disclosedBengaluru

Skills required:

Data Engineering, SQL, Python, ETL, Data Pipelines, Apache Spark, Airflow, Data Warehousing, Data Modeling, Cloud Platforms, Kafka, dbt, Snowflake, AWS, Git

Data Engineer

Accenture

Not disclosedBengaluru

Skills required:

Data Engineering, SQL, Python, ETL, Data Pipelines, Apache Spark, Airflow, Data Warehousing, Data Modeling, Cloud Platforms, Kafka, dbt, Snowflake, AWS, Git

Data Engineer

Accenture

Not disclosedBengaluru

Skills required:

Data Engineering, SQL, Python, ETL, Data Pipelines, Apache Spark, Airflow, Data Warehousing, Data Modeling, Cloud Platforms, Kafka, dbt, Snowflake, AWS, Git

Data Engineer

Accenture

Not disclosedBengaluru

Skills required:

Data Engineering, SQL, Python, ETL, Data Pipelines, Apache Spark, Airflow, Data Warehousing, Data Modeling, Cloud Platforms, Kafka, dbt, Snowflake, AWS, Git

Data Engineer

Accenture

Not disclosedBengaluru

Skills required:

Data Engineering, SQL, Python, ETL, Data Pipelines, Apache Spark, Airflow, Data Warehousing, Data Modeling, Cloud Platforms, Kafka, dbt, Snowflake, AWS, Git

Data Engineer

Accenture

Not disclosedBengaluru

Skills required:

Data Engineering, SQL, Python, ETL, Data Pipelines, Apache Spark, Airflow, Data Warehousing, Data Modeling, Cloud Platforms, Kafka, dbt, Snowflake, AWS, Git

Data Platform Engineer

Accenture

Not disclosedBengaluru

Skills required:

Data Platform Engineering, Python, SQL, Apache Spark, Kafka, Kubernetes, Airflow, Cloud Platforms, Infrastructure as Code, Terraform, Data Lakes, Snowflake, Databricks, Data Governance, CI/CD

Data Platform Engineer

Accenture

Not disclosedBengaluru

Skills required:

Data Platform Engineering, Python, SQL, Apache Spark, Kafka, Kubernetes, Airflow, Cloud Platforms, Infrastructure as Code, Terraform, Data Lakes, Snowflake, Databricks, Data Governance, CI/CD

Custom Software Engineering Lead

Accenture

Not disclosedBengaluru

Skills required:

Team Leadership, Technical Leadership, Software Development, Code Review, System Design, Mentoring, Agile Methodologies, Sprint Planning, Stakeholder Management, Delivery Management, Hiring, Performance Management, Programming, Estimation

Data Modeler

Accenture

Not disclosedBengaluru

Skills required:

Data Modeling, Dimensional Modeling, ER Modeling, SQL, Data Warehousing, Erwin Data Modeler, Normalization, Star Schema, Data Vault, Metadata Management, Data Governance, Snowflake, Requirements Gathering, Stakeholder Management, Documentation

View All 35141 Jobs