Software Engineering • Mid-Level

Who you are

You have extensive experience building and scaling web crawlers, or you are excited to ramp up very quickly. Your background includes working with high-performance languages such as Rust or C++, and you are familiar with TypeScript and Playwright. You understand the importance of optimizing systems to an exceptional degree and are passionate about finding high-quality knowledge on the web.

You recognize the challenges of crawling the web at scale, including handling dynamic content, JavaScript rendering, and anti-bot measures. You are comfortable designing systems that can efficiently manage crawl politeness and rate limiting across thousands of domains. Your experience includes creating intelligent crawl scheduling and prioritization algorithms to maximize coverage efficiency.

Desirable

Experience with distributed systems and a strong understanding of web technologies will set you apart. Familiarity with modern web design and the Chrome DevTools Protocol (CDP) is a plus. You are eager to contribute to a team that is building a search engine from scratch and are ready to tackle the challenges that come with it.

What you'll do

As a Web Crawler engineer at Exa, you will be responsible for developing and maintaining a distributed crawler capable of handling over 100 million pages per day. You will optimize the crawling process to ensure politeness and efficiency while navigating complex web structures. Your role will involve designing systems that can detect and manage dynamic content, ensuring that our search engine remains effective and up-to-date.

You will collaborate closely with other engineers to build massive-scale infrastructure that supports our ambitious goals. This includes working with a $5 million H200 GPU cluster and coordinating efforts across a team of talented individuals. You will also have the opportunity to contribute to the development of state-of-the-art embedding models and high-performance vector databases.

What we offer

At Exa, we provide a dynamic work environment where innovation is encouraged. You will have the chance to work on cutting-edge technology and contribute to a project that aims to revolutionize the way AI applications access information. We offer competitive compensation and are happy to sponsor international candidates, ensuring that we attract the best talent from around the world. Join us in building a search engine that serves every AI application and makes a significant impact on the future of information retrieval.