Software Engineering • Mid-Level

Who you are

You have a strong background in software engineering, particularly in designing and implementing scalable data infrastructure. With experience in Python and familiarity with containerization technologies like Docker and orchestration tools such as Kubernetes, you are well-equipped to handle the complexities of large-scale data systems. You understand the importance of performance and efficiency in data pipelines and have a proactive approach to identifying and resolving bottlenecks.

Your collaborative spirit shines through as you work closely with researchers and other infrastructure teams to ensure seamless integration of datasets into training and inference pipelines. You are detail-oriented, ensuring that dataset interfaces are standardized, discoverable, and easy for other teams to adopt. Your ability to document processes and maintain clear communication is key to fostering a productive team environment.

Desirable

Experience with cloud platforms like AWS is a plus, as is familiarity with data storage solutions such as Elasticsearch. You may also have exposure to machine learning frameworks, which will enhance your contributions to the team.

What you'll do

In this role, you will design and maintain standardized dataset APIs that cater to multimodal data that cannot fit in memory. You will build proactive testing and scale validation pipelines for dataset loading at GPU scale, ensuring that the infrastructure can handle the demands of OpenAI's next-generation models. Collaborating with teammates, you will integrate datasets seamlessly into training and inference pipelines, ensuring a smooth user experience.

Your responsibilities will include documenting and maintaining dataset interfaces, establishing safeguards and validation systems to ensure datasets remain reproducible, and proactively testing for performance bottlenecks. You will play a crucial role in enabling researchers to focus on advancing model capabilities while you handle the scale, efficiency, and reliability required to bring those models to life.

What we offer

At OpenAI, you will be part of a mission-driven team that believes in the potential of artificial intelligence to solve global challenges. We offer a collaborative work environment where your contributions will directly impact the future of technology. Join us in shaping the future of AI and enjoy the opportunity to work with cutting-edge technologies in a supportive and innovative atmosphere.