Data Engineer • Mid-Level

Who you are

You have 5 to 12 years of experience in data engineering, with a strong focus on building and optimizing ETL processes. Your expertise in Python is complemented by a deep understanding of data transformation, error handling, and logging. You are proficient in using Databricks and PySpark to evaluate and perform transformations within data lakes. You possess a solid grasp of data modeling and have experience leading data engineering projects, ensuring that you can handle complex data infrastructures with ease.

Your technical skills extend to Azure services, including Azure SQL Database, Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake. You are comfortable connecting to various data sources, including databases and streaming data, and you have a proven track record of working with large-scale data infrastructures. Your familiarity with cloud warehouses and storage formats enhances your ability to manage data effectively.

You are not only a technical expert but also a strong communicator who can collaborate with cross-functional teams. Your experience in Unix shell scripting and scheduling tools further supports your ability to automate and streamline data processes. You thrive in environments where you can lead projects and mentor junior engineers, sharing your knowledge and expertise to foster a collaborative team culture.

Desirable

Experience with Scala and Apache Spark is a plus, as it allows you to leverage these technologies in your data engineering tasks. Familiarity with additional Azure services or cloud platforms can also enhance your profile, making you a more versatile candidate for this role.

What you'll do

In this role, you will be responsible for designing and implementing robust ETL processes that efficiently read data from various sources, transform it, and load it into managed Delta tables in Databricks. You will utilize your Python skills to handle data transformations, ensuring that all data types are correctly processed and any errors are logged for future reference. Your expertise in Databricks will allow you to evaluate and optimize data workflows, ensuring high performance and reliability.

You will collaborate closely with data scientists and analysts to understand their data needs and provide them with the necessary data infrastructure. This includes building data pipelines that are scalable and efficient, allowing for seamless data access and analysis. You will also be involved in troubleshooting and resolving any data-related issues that arise, ensuring that data integrity is maintained throughout the process.

As a key member of the data engineering team, you will lead projects that focus on enhancing data quality and optimizing ETL processes. Your ability to communicate effectively with stakeholders will be crucial as you gather requirements and provide updates on project progress. You will also have the opportunity to mentor junior engineers, helping them develop their skills and grow within the organization.

What we offer

At Emids, we value innovation and collaboration, providing a supportive environment where you can thrive. You will have access to cutting-edge technologies and the opportunity to work on impactful projects that shape the future of data engineering. We offer competitive compensation and benefits, along with opportunities for professional development and career advancement. Join us in our mission to leverage data for better decision-making and drive business success.