Open Source Engineer • Senior

Who you are

You have over 10 years of experience building high-performance databases, big data systems, or large-scale data services — your deep understanding of open-source Big Data or AI training systems like Hadoop, Spark, and Flink sets you apart in the field. You are well-versed in the internals of these systems and have a passion for contributing to open-source communities.

Your expertise extends to designing and maintaining efficient distributed dataset operations — you know how to build efficient indices that enable predicate pushdown and accelerate queries in systems like Spark, Ray, or Trino. You thrive in collaborative environments, working alongside other open-source builders to drive integrations and improve operations.

You are proactive in promoting the Lance format in open-source communities and at Big Data conferences — your communication skills allow you to effectively engage with diverse audiences and foster community growth. You are also comfortable operating and improving internal data processing infrastructure, ensuring that it meets the high standards required for cutting-edge AI workloads.

Desirable

Experience with additional data infrastructure systems such as Hive Metastore, Presto, and Trino would be a plus, as would familiarity with Rust programming. You are eager to learn and contribute to the evolution of the LanceDB ecosystem, and you appreciate the value of a generous learning budget and support for open-source contributions.

What you'll do

In this role, you will be responsible for driving open-source community efforts to integrate the Lance format with various data infrastructure systems — your work will directly impact the reach of LanceDB within the broader ecosystem. You will collaborate with a world-class team of open-source builders, contributing to projects across the Apache and AI communities.

You will design and maintain efficient distributed operations for Lance datasets, ensuring that they perform optimally in high-demand environments. Your role will involve building efficient indices that enhance query performance in systems like Spark, Ray, or Trino, which are critical for the success of AI applications.

You will also work on table formats, data encodings, and various aspects of the Lance format in Rust — your contributions will help shape how LanceDB operates and scales in production environments. You will have the opportunity to promote the Lance format in open-source communities and at Big Data conferences, showcasing your work and the innovations you help drive.

What we offer

Joining LanceDB means becoming part of a team that is at the forefront of open-source database technology — you will collaborate on systems that power next-generation AI workloads. The company offers a generous learning budget and support for open-source contributions, encouraging you to grow your skills and make meaningful contributions to the community.

You will work in an environment that values collaboration and innovation, where your ideas and contributions will be recognized and celebrated. The team consists of co-authors of pandas and contributors to HDFS, Arrow, Iceberg, and HBase, providing a rich environment for learning and professional development.

LanceDB is committed to building a diverse and inclusive team, and we encourage you to apply even if your experience doesn't match every requirement. We believe that diverse teams build better products and foster a culture of creativity and innovation.