Some of the common use cases of incorporating ChatGPT into your own applications are:

1. Allowing users to query your system in plain English
2. Providing representations of your data e.g., summaries, timelines etc
3. Mapping data from other formats/schemas into the formats/schemas required by your applications

How do you get started and what technical obstacles will you need to overcome?

Prerequisites

Today, OpenAI’s ChatGPT is the best available. Microsoft, Google, Facebook and others are working on competitors that may get better eventually.

You can access OpenAI via API in two ways:

1. Sign up for a paid account at OpenAI.com and use their API
2. Sign up for Microsoft Azure and use their OpenAI API

Both of these run pretty much the same OpenAI engine underneath so your results should be the same.

OpenAI option is easy to get started with but it is questionable whether OpenAI can provide the scale, performance, reliability and security that will be required for most large scale commercial applications.

Azure option is great if you’re already an Azure customer or you have a large scale commercial application. The performance characteristics of large scale commercial use of ChatGPT APIs is not well understood yet. Still we would expect that Microsoft Azure with their experience in cloud services will be able to provide a more scalable, more reliable and more secure solution.

Obstacles

If you try to incorporate ChatGPT API into your own code to build your solution, you’ll run into a few obstacles:

1. ChatGPT prompts are hard to discover.

For example, how do you tell ChatGPT the format/schema of the data you’re giving it to reason about, how do you tell it what format/schema you want to get the output? Yeah you can google around and try various combinations until you get it right.

2. The input and output of ChatGPT is in text. Your application works with structured data like JSON.

How do you get it in a structured format like JSON or XML to incorporate into your code?

3. ChatGPT has a limit on the size of data you can pass into the API.

How do you split your data and/or convert it to vector embeddings?

4. ChatGPT APIs are REST APIs.

You’ll need to write code to call those APIs and handle errors.

LangChain to the rescue

The simpler way is to just use LangChain. It is available for Python and Javascript. LangChain provides prompt templates including predefined operations like specifying data format and schema. It also provides ways to easily have ChatGPT return data in a schema and format you specify.

Javascript Documentation: https://js.langchain.com/docs

Python Documentation: https://python.langchain.com/docs

Example Code (JavaScript)

The following code constructs the prompt for ChatGPT, calls ChatGPT and gets the result in a structured format (JSON).

Add the following package:

npm install langchain zod

Then run the following code:

const {OpenAI} = require('langchain/llms/openai');
const {PromptTemplate} = require('langchain/prompts');
const {LLMChain} = require('langchain/chains');
const {StructuredOutputParser, OutputFixingParser} = require('langchain/output_parsers');
const {z} = require('zod');

const outputParser = StructuredOutputParser.fromZodSchema(
    z.array(
        z.object({
            fields: z.object({
                Name: z.string().describe('The name of the country'),
                Capital: z.string().describe("The country's capital")
            })
        })
    ).describe('An array of Airtable records, each representing a country')
);
const model = new OpenAI({
    // modelName: 'gpt-4', // Or gpt-3.5-turbo
    temperature: 0 // For best results with the output fixing parser
});
const outputFixingParser = OutputFixingParser.fromLLM(
    model,
    outputParser
);
const template = 'Answer the user\'s question as best you can:\n{format_instructions}\n{query}';
const prompt = new PromptTemplate({
    template: template,
    inputVariables: ['query'],
    partialVariables: {
        format_instructions: outputFixingParser.getFormatInstructions()
    }
});
console.log(prompt);
const chain = new LLMChain(
    {
        llm: model, prompt: prompt,
        outputKey: 'records', // For readability - otherwise the chain output will default to a property named "text"
        outputParser: outputFixingParser
    });
const result = await chain.call({ query: 'List 5 countries.' });
console.log(JSON.stringify(result.records, null, 2));

That’s all you need to call ChatGPT with structured data and receive structured data.

Python code is very similar.

For more information you can go to LangChain Documentation.