About Exa
Transforming AI queries with precise search infrastructure
Key Highlights
- Raised $112.2 million in Series A funding
- Headquartered in Lower Haight, San Francisco, CA
- 21-100 employees focused on AI search technology
- Unique embeddings-based search engine eliminating AI 'hallucinations'
Exa is revolutionizing AI search infrastructure with its embeddings-based search engine, designed to enhance the accuracy of AI model responses by filtering the internet for precise knowledge. Headquartered in Lower Haight, San Francisco, Exa has raised $112.2 million in funding through several succ...
🎁 Benefits
Exa offers competitive salaries, equity options, and a flexible remote work policy. Employees enjoy generous PTO, parental leave, and a budget for pro...
🌟 Culture
Exa fosters a culture focused on innovation and precision in AI search technology. The team values collaboration and is committed to building a unique...
Overview
Exa is seeking a Software Engineer specializing in Web Crawling to build a search engine from scratch. You'll work with high-performance languages like Rust and C++ to develop massive-scale infrastructure for web crawling. This role requires experience in building and scaling web crawlers.
Job Description
Who you are
You have extensive experience building and scaling web crawlers, or you are excited to ramp up very quickly. You have experience with some high-performance languages such as C++ or Rust, and you are comfortable optimizing a system to an exceptional degree. You care about the problem of finding high-quality knowledge and recognize how important this is for the world. You understand the complexities involved in crawling the web and are eager to tackle challenges like dynamic content and anti-bot measures.
Desirable
Experience with distributed systems and algorithms for intelligent crawl scheduling would be a plus. Familiarity with optimizing crawl politeness and rate limiting across thousands of domains is also desirable. You are someone who enjoys problem-solving and is keen on developing innovative solutions to enhance web crawling efficiency.
What you'll do
As a Web Crawler engineer at Exa, you will be responsible for crawling the entire web, essentially building a Google-scale crawling system. You will design and implement a distributed crawler capable of handling over 100 million pages per day. Your role will involve optimizing crawl politeness and rate limiting across thousands of domains to ensure efficient and respectful crawling practices. You will also work on systems to detect and handle dynamic content, including JavaScript rendering and anti-bot measures. Additionally, you will create intelligent crawl scheduling and prioritization algorithms to maximize coverage efficiency. Collaboration with other engineers and stakeholders will be key as you contribute to building a robust search engine infrastructure.
What we offer
This is an in-person opportunity based in Singapore, where you will be part of a dynamic team dedicated to pushing the boundaries of web crawling technology. You will have access to a $5M H200 GPU cluster and the chance to work on cutting-edge projects that impact AI applications globally. Exa values innovation and encourages you to apply even if your experience doesn't match every requirement. Join us in our mission to revolutionize search technology and make high-quality knowledge accessible to everyone.
Interested in this role?
Apply now or save it for later. Get alerts for similar jobs at Exa.
Similar Jobs You Might Like
Based on your interests and this role
Software Engineering
Woflow is hiring a Software Engineer for their Web Crawling team to develop and optimize web crawlers for high-quality data collection. You'll work with Python and web scraping techniques in a remote environment. This position requires at least 3 years of experience.
Software Engineering
Exa is hiring a Software Engineer specializing in Web Crawling to build a search engine from scratch. You'll work with Rust and TypeScript to develop high-performance systems for web crawling. This position requires extensive experience in building and scaling web crawlers.
Software Engineering
AppLovin is seeking a Software Engineer to help build a next-generation social platform. You'll be involved in architecting robust systems that enhance user engagement. This role requires a strong technical background and a collaborative mindset.
Software Engineering
Software Engineering
Meta is hiring a Software Engineer to work on machine learning projects that connect billions of people globally. You'll tackle complex technical problems and enhance user experiences across various platforms. This role requires a strong background in software engineering.