Data Engineer • Mid-Level

Who you are

You have a strong background in data engineering with hands-on experience building and operating large-scale distributed data processing pipelines. Your expertise in lakehouse architectures such as Delta Lake, Iceberg, and Hudi allows you to make informed decisions on when to use them effectively. You are familiar with streaming data systems like Kafka and Flink, and you have experience with Ray, Spark, or ClickHouse at production scale. Your obsessive focus on reliability ensures that the systems you build are robust and do not require late-night interventions.

You thrive in environments where you can design systems that scale to hundreds of petabytes, and you are excited about the opportunity to build data pipelines at a scale that most companies only dream about. Your background in GPU-accelerated data processing, such as RAPIDS or cuDF, is a bonus that enhances your ability to contribute to Exa's ambitious projects.

Desirable

Experience with Lance or other vector-native storage formats is a plus, as is a background in GPU-accelerated data processing. You are eager to tackle complex challenges and are motivated by the impact your work will have on AI applications.

What you'll do

In this role, you will architect and build the data infrastructure that powers Exa's search engine, which serves every AI application. You will design a lakehouse architecture capable of handling over 100 petabytes of web crawl data, ensuring that the data is accessible and reliable for various applications. You will build streaming pipelines that process billions of documents per day for real-time indexing, contributing to the efficiency and effectiveness of the search engine.

You will also be responsible for architecting the data layer for the embedding training infrastructure on Ray, which is crucial for training state-of-the-art embedding models. Your work will involve scaling the ClickHouse deployment to handle analytical queries across petabytes of search logs, ensuring that the system remains performant and responsive.

Collaboration is key in this role, as you will work closely with other engineers to ensure that the data infrastructure meets the needs of the entire team. You will have the autonomy to design systems that not only meet current demands but also anticipate future growth and scalability.

What we offer

Exa provides a dynamic work environment where innovation is encouraged, and your contributions will directly impact the company's success. You will have access to a $5M H200 GPU cluster that regularly lights up tens of thousands of machines, allowing you to work with cutting-edge technology. We are happy to sponsor international candidates, including those on STEM OPT, OPT, H1B, O1, and E3 visas. Join us in building a search engine from scratch and be part of a team that is shaping the future of AI applications.