Explore OpenAI's function calling capabilities, allowing AI models to interact with external APIs and perform tasks beyond text generation. Learn to implement dynamic function calls, including fetching data from Elasticsearch, enhancing the model's real-time data access and complex operation handling. Discover practical use cases and step-by-step integration in this insightful blog.

Introduction

Prerequisites

Elastic

Open AI

Open-Meteo API

Sample Data

Python notebook

Import packages

Accept Credentials

Function1: fetch_from_elasticsearch()

`def fetch_from_elasticsearch(nl_query):`

Get Index Mapping

Get reference document

Generate Elasticsearch Query DSL based on user query

Execute Query on Elasticsearch

Text to Elasticsearch Query

Query1

Query2

Function2: weather_report()

`def weather_report(latitude, longitude):`

Test function

Function calling

Define functions

Call OpenAI Chat Completion API

Execute selected function

Extend the conversation

Parallel Function Calling

Python Notebook

Conclusion

OpenAI function calling with Elasticsearch

Learn how to ingest data from Snowflake to Elasticsearch using Logstash or a Snowflake Elasticsearch Python Script.

Snowflake credentials<a id="snowflake-credentials"></a>

Elastic® credentials

Methods to ingest data from Snowflake to Elasticsearch

Method 1: Using Logstash

Method 2: Using Snowflake-Elasticsearch Python script

Verify data

Snowflake is a fully managed SaaS that provides a single platform for data warehousing, data lakes, data engineering, data science, data application development, and secure sharing and consumption of real-time/shared data.

What is Snowflake?

Install Logstash.

Go to the Maven Central Repository and download: https://repo1.maven.org/maven2/net/snowflake/snowflake-jdbc.

Click on the directory for the version that you need and download the snowflake-jdbc-#.#.#.jar file.

Create a pipeline by creating file sf-es.conf. Add the snippet and replace all credentials.

Run Logstash.

How to ingest data from Snowflake to Elasticsearch using Logstash

How to ingest data from Snowflake to Elasticsearch

Learn to construct an RAG system with Elasticsearch for semantic search and question-answering on private data. Fetch relevant documents as a Context window and leverage the Gemma model for accurate answers.

Overview

1. Import packages & credentials

Install required packages

Import all dependencies

Get credentials

2. Preparing Documents

Let's download the sample dataset and deserialize the document.

Document chunking

Why is document chunking needed?

Context window limit

Balancing Context Size

Hallucination

How to chunk data?

3. Index Documents

4. Load Gemma model locally using Hugging Face

Why Hugging Face?

Hugging Face login

Initialize the tokenizer with the model (google/gemma-2b-it)

Usage

Create a `text-generation` pipeline and initialize with LLM

5. Create a chain using a prompt template

6. Ask a question

Building a RAG System With Gemma, Hugging Face & Elasticsearch

Author

Ashish Tiwari

Articles

OpenAI function calling with Elasticsearch

How to ingest data from Snowflake to Elasticsearch

Building a RAG System With Gemma, Hugging Face & Elasticsearch