As a Data Engineer, your role will encompass:
- Designing and building production data pipelines from ingestion to consumption within a hybrid big data architecture using Scala, Python, Talend etc.
- Gather and address technical and design requirements.
- Refactor existing applications to optimize its performance through setting the appropriate architecture and integrating the best practices and standards.
- Participate in the entire data life-cycle mainly focusing on coding, debugging, and testing.
- Troubleshoot and debug ETL Pipelines.
- Documentation of each process.
Technical Requirements: -
- BSc degree in Computer Science/Computer Engineering. (Masters is a plus.)
- 2+ years of experience as a Data Engineer.
- In-depth understanding of core ETL concepts, Data Modelling, Data Lineage, Data Governance, Data Catalog, etc.
- 2+ years of work experience in Scala, Python, Java.
- Good Knowledge on Big Data Tools such as Spark/HDFS/Hive/Flume, etc.
- Hands on experience on ETL tools like Talend/Informatica is a plus.
- Good knowledge in Kafka and spark streaming is a big plus.
- 2+ years of experience in using Azure cloud and its resources/services (like Azure Data factory, Azure Databricks, SQL Synapse, Azure Devops, Logic Apps, Power Bi, Azure Event Hubs, etc).
- Strong experience in Relational Databases (MySQL, SQL Server)
- Exposure on data visualization tools like Power BI / Qlik sense / MicroStrategy
- 2+ years of experience in developing APIs (REST & SOAP protocols).
- Strong knowledge in Continuous Integration & Continuous Deployment (CI/CD) utilizing Docker containers, Jenkins, etc.
- Strong competencies in algorithms and software architecture.
- Excellent analytical and teamwork skills.
Good to have: -
- Previous on-prem working experience is a plus.
- In-depth understanding of the entire web development process (design, development, and deployment)
- Previous experience in automated testing including unit testing & UI testing.