@langchain/google-genai

This package contains the LangChain.js integrations for Gemini through their generative-ai SDK.

Update: Returns JSON

1import { ChatGoogleGenerativeAI } from "@juji/langchain-google-genai";
2import { JsonOutputParser } from "@langchain/core/output_parsers";
3
4const model = new ChatGoogleGenerativeAI({
5  model: "gemini-1.5-flash",
6  json: true
7})
8
9const messages = [
10  new SystemMessage(`
11Translate the following English text into Italian. 
12
13Reply with JSON, using the following JSON schema:
14{"translation": "string"}
15`),
16  new HumanMessage("hi, i am superman"),
17];
18
19const parser = new JsonOutputParser()
20const result = await model.pipe(parser).invoke(messages);
21console.log(result)
22
23// {"translation": "Ciao, sono Superman"}

Installation

1npm install @langchain/google-genai

This package, along with the main LangChain package, depends on @langchain/core. If you are using this package with other LangChain packages, you should make sure that all of the packages depend on the same instance of @langchain/core. You can do so by adding appropriate field to your project's package.json like this:

1{
2  "name": "your-project",
3  "version": "0.0.0",
4  "dependencies": {
5    "@langchain/google-genai": "^0.0.0",
6    "langchain": "0.0.207"
7  },
8  "resolutions": {
9    "@langchain/core": "0.1.5"
10  },
11  "overrides": {
12    "@langchain/core": "0.1.5"
13  },
14  "pnpm": {
15    "overrides": {
16      "@langchain/core": "0.1.5"
17    }
18  }
19}

The field you need depends on the package manager you're using, but we recommend adding a field for the common yarn, npm, and pnpm to maximize compatibility.

Chat Models

This package contains the ChatGoogleGenerativeAI class, which is the recommended way to interface with the Google Gemini series of models.

To use, install the requirements, and configure your environment.

1export GOOGLE_API_KEY=your-api-key

Then initialize

1import { ChatGoogleGenerativeAI } from "@langchain/google-genai";
2
3const model = new ChatGoogleGenerativeAI({
4  modelName: "gemini-pro",
5  maxOutputTokens: 2048,
6});
7const response = await model.invoke(new HumanMessage("Hello world!"));

Multimodal inputs

Gemini vision model supports image inputs when providing a single chat message. Example:

1npm install @langchain/core

1import fs from "fs";
2import { ChatGoogleGenerativeAI } from "@langchain/google-genai";
3import { HumanMessage } from "@langchain/core/messages";
4
5const vision = new ChatGoogleGenerativeAI({
6  modelName: "gemini-pro-vision",
7  maxOutputTokens: 2048,
8});
9const image = fs.readFileSync("./hotdog.jpg").toString("base64");
10const input = [
11  new HumanMessage({
12    content: [
13      {
14        type: "text",
15        text: "Describe the following image.",
16      },
17      {
18        type: "image_url",
19        image_url: `data:image/png;base64,${image}`,
20      },
21    ],
22  }),
23];
24
25const res = await vision.invoke(input);

The value of image_url can be any of the following:

• A public image URL
• An accessible gcs file (e.g., "gcs://path/to/file.png")
• A base64 encoded image (e.g., data:image/png;base64,abcd124)
• A PIL image

Embeddings

This package also adds support for google's embeddings models.

1import { GoogleGenerativeAIEmbeddings } from "@langchain/google-genai";
2import { TaskType } from "@google/generative-ai";
3
4const embeddings = new GoogleGenerativeAIEmbeddings({
5  modelName: "embedding-001", // 768 dimensions
6  taskType: TaskType.RETRIEVAL_DOCUMENT,
7  title: "Document title",
8});
9
10const res = await embeddings.embedQuery("OK Google");

Development

To develop the Google GenAI package, you'll need to follow these instructions:

Install dependencies

1yarn install

Build the package

1yarn build

Or from the repo root:

1yarn build --filter=@langchain/google-genai

Run tests

Test files should live within a tests/ file in the src/ folder. Unit tests should end in .test.ts and integration tests should end in .int.test.ts:

1$ yarn test
2$ yarn test:int

Lint & Format

Run the linter & formatter to ensure your code is up to standard:

1yarn lint && yarn format

Adding new entrypoints

If you add a new file to be exported, either import & re-export from src/index.ts, or add it to the entrypoints field in the config variable located inside langchain.config.js and run yarn build to generate the new entrypoint.