Learn through an example how to leverage the Elastic Web Crawler to crawl a website, ingest its content & prepare the data for search.

What is the Elastic Web Crawler

Website discovery & web crawler creation

A first crawl

Define crawl rules

Enrich the data extracted by the web crawler

Extract data from custom field

Ingest pipeline to transform data

Enable semantic search with ML Inference pipeline

Test search query

Preparing the web crawler for production deployment

Scheduling

Protected website

Conclusion

The Elastic web crawler is an out-of-the-box tool that enables users to crawl a website and ingest its content into Elasticsearch.

What is the Elastic Web Crawler?

The Elastic web crawler begins each crawl by visiting the entry point URLs. From there, the crawler fetches the content from this webpage and extracts it. The HTML document is transformed into a search document and indexed. Then it follows any links to which that page points to, respecting the rules defined by the user. The new URLs are visited using the same methodology until the crawler has indexed the accessible web pages.

How does the Elastic Web Crawler work?

Web crawler: Add search to a website with the Elastic Web Crawler

Elastic web crawler: Add search to your website

Author

Lionel Palacin

Articles

Elastic web crawler: Add search to your website