Testing AI Chatbot Performance - GPT Bot Quality Improvement Methods

General Chatbot vs Expert Chatbot

Ask general ChatGPT:

Me: What's our company's refund policy?
ChatGPT: Sorry, I don't know your company's refund policy.

But if it's a custom chatbot:

Me: What's our company's refund policy?
Custom Bot: The refund policy is as follows:
- Full refund within 14 days of purchase
- Limited to unused products
- Request refund through customer service...

See the difference? Today I'll show you how to create expert chatbots trained on your materials.

When You Need Custom Chatbots

1. Internal Company Documents

• Work manuals
• Regulations/policies
• FAQ

2. Specialized Knowledge

• Research papers
• Technical documents
• Educational materials

3. Personal Content

• Blog posts
• Book contents
• Lecture materials

4. Customer Data

• Past consultation records
• Product information
• Price lists

What is RAG?

RAG = Retrieval Augmented Generation

Simply put:

1. Receive question
2. Find relevant documents
3. Generate answer referring to those documents

[Question] → [Document Search] → [Find Related Content] → [Generate Answer]

General chatbots only do step 3. RAG chatbots do all steps 1-2-3.

Method 1: ChatGPT Custom GPT + File Upload

The easiest method.

Step-by-Step Guide

1. Create Custom GPT

• Access ChatGPT
• Explore > Create a GPT

2. Add Files in Knowledge Section

• Click Configure tab
• Find Knowledge section
• Click Upload files

Supported Files:

• PDF
• TXT
• DOCX
• XLSX
• CSV (Max 20 files, 512MB per file)

3. Set Instructions

You are a customer service chatbot for [Company Name].

Role:
Refer to uploaded documents (refund_policy.pdf, FAQ.docx, product_info.xlsx)
to provide accurate answers.

Rules:
1. Always check uploaded documents first
2. Say "not found in documents" for content not in documents
3. Mention which document and which part when answering

Answer Format:
"According to [document name],
[content]

(Source: [document name] p.X)"

Example:
Question: What's the refund period?
Answer: According to refund_policy.pdf,
you can request a refund within 14 days of receiving the product.

(Source: refund_policy.pdf page 2)

Real Example: Blog Q&A Bot

Prepare Files to Upload:

1. Save blog posts as PDFs

   • Chrome: Ctrl+P → Save as PDF
   • Select 10-20 key posts

2. Create FAQ document

   • Write in Google Docs or Word
   • Organize frequently asked questions

File Structure Example:

blog_post_01.pdf - "Getting Started with AI Tools"
blog_post_02.pdf - "How to Use ChatGPT"
blog_post_03.pdf - "Prompt Engineering"
faq.pdf - Frequently Asked Questions
about.pdf - Author Introduction

Instructions:

You are a specialized assistant for [Blog Name].

Knowledge Base:
Answer by referring to uploaded blog posts and FAQ.

Response Method:
1. Introduce related blog post if available
2. Summarize key points of the post
3. Provide link to read full post

Example:
Question: "Tell me how to write good ChatGPT prompts"

Answer: That's covered in the "How to Use ChatGPT" post!

Key tips:
1. Ask specifically
   Bad: "Write something"
   Good: "Write 500-word marketing blog post"

2. Assign roles
   "You're a professional marketer"

3. Provide examples
   "In this style: [example]"

Read the full post here:
[Blog URL]/chatgpt-tips

Any other questions?

Testing

Me: What's prompt engineering?

Bot: Explained in the "Prompt Engineering" post!

Prompt engineering is the skill of asking AI effectively.

Main techniques:
- Few-shot learning: Provide examples
- Chain-of-thought: Step-by-step thinking
- Role prompting: Assign roles

Check the blog post for detailed examples and practice:
[Link]

(Source: blog_post_03.pdf)

Perfect, right?

Method 2: Claude Projects

Claude can upload even more files!

Features

Capacity:

• Max 100 files
• Total 10MB (Pro: 30MB)

Supported Formats:

• PDF, TXT, CSV
• Code files (py, js, etc.)
• Markdown

Creating

1. Create Project

• Access Claude
• Projects tab
• Create Project

2. Upload Files

• Click Add Content
• Select files (multiple OK)

3. Custom Instructions

You are an internal work manual expert.

Role:
Answer employee work-related questions.
All answers based on uploaded manuals.

Uploaded Documents:
- employee_handbook.pdf (Employee Handbook)
- it_guide.pdf (IT Guide)
- hr_policy.pdf (HR Policy)
- office_rules.pdf (Office Rules)

Answer Rules:
1. Find and answer accurate info from documents
2. Reference all if info in multiple documents
3. If not in documents: "Not in manual. Contact HR team"
4. Recommend latest verification as policies may change

Source Attribution:
Show (Source: [filename]) at end of all answers

Example:
Question: What are the annual leave usage rules?

Answer: Annual leave usage rules are as follows:

• 1st year: 15 days
• 2nd year+: 15 days + (years-1) days
• Max 25 days

Application Method:
1. Intranet > Electronic Approval
2. Fill out leave application
3. Team leader approval

• 3+ days: Apply at least 1 week in advance
• Same-day leave: Emergency only

(Source: hr_policy.pdf pages 12-13)

Any other questions?

Advanced Use: Multiple Projects

Divide projects by purpose:

Project 1: Customer Service
- FAQ
- Product manual
- Refund policy

Project 2: Technical Support
- API documentation
- Troubleshooting guide
- Code examples

Project 3: Sales Support
- Product brochure
- Price list
- Sales scripts

Method 3: Creating with Free Tools

If paid subscription is a burden?

Poe.com + Long Prompts

No file upload, but can include long text in prompt.

Steps:

1. Convert documents to text
2. Include in prompt

Example:

You are a restaurant recommendation expert.

Below is local restaurant information:

===== Restaurant Info Start =====

[Restaurant 1]
Name: Jin Mi Restaurant
Location: 123 Yeoksam-dong, Gangnam-gu
Menu:
- Kimchi Stew: 8,000 won
- Soybean Paste Stew: 8,000 won
- Spicy Pork: 10,000 won
Features: Home-style taste, generous sides
Hours: 11:00-21:00 (Closed Sundays)
Rating: 4.5/5

[Restaurant 2]
Name: Pasta House
Location: 456 Yeoksam-dong, Gangnam-gu
Menu:
- Aglio Olio: 12,000 won
- Carbonara: 13,000 won
- Tomato Pasta: 12,000 won
Features: Handmade pasta, nice atmosphere
Hours: 12:00-22:00
Rating: 4.7/5

[Restaurant 3]
Name: Sushiya
Location: 789 Nonhyeon-dong, Gangnam-gu
Menu:
- Lunch Set: 15,000 won
- Dinner Set: 30,000 won
- Premium Sushi: 50,000 won
Features: Fresh ingredients, reservation required
Hours: 12:00-15:00, 18:00-22:00
Rating: 4.8/5

===== Restaurant Info End =====

Answer based on above information.

Answer Format:
- Restaurant name, location, recommended menu
- Price range
- Features
- Business hours
- "Recommend visiting for more details"

Example:
Question: Recommend lunch place around 10,000 won

Answer: For 10,000 won lunch, I recommend Jin Mi Restaurant!

📍 Location: 123 Yeoksam-dong, Gangnam-gu
🍽️ Recommend: Spicy Pork (10,000 won)
✨ Features: Home-style generous sides
⏰ Hours: 11:00-21:00

The spicy pork is great with rice.
5-6 side dishes all refillable!

Gets crowded at lunch on weekdays,
so visit around 11:30 or 1:30.

Recommend visiting for more details!

Pros:

• Free
• Simple

Cons:

• Text only
• Limited quantity (about 4,000 characters)

Method 4: Google Sheets Integration

Spreadsheets are convenient for structured data.

Preparation

Spreadsheet Example:

Question	Answer	Category
Delivery time?	Takes 2-3 days	Shipping
Shipping fee?	Free over 30,000 won	Shipping
Refund method?	Apply in My Page	Refund
Exchange possible?	Within 7 days of receipt	Exchange

Use in ChatGPT

1. Download spreadsheet as CSV
2. Upload to Custom GPT
3. Set Instructions

You are a customer service chatbot.

The uploaded faq.csv file contains:
- Question (column A)
- Answer (column B)
- Category (column C)

Response Method:
1. Find content similar to question in CSV
2. Provide that answer
3. Suggest other questions from related category

Example:
Question: When will delivery arrive?

Answer: Delivery takes 2-3 days after payment completion.

Related Questions:
• How much is shipping?
• How to track delivery?

Any other questions?

Data Preparation Tips

1. Organize Documents

Good Documents:

• Clear titles
• Structured (titles, subtitles)
• Unnecessary content removed

Bad Documents:

• Too long and verbose
• No structure
• Lots of duplicate content

2. File Names

Bad:
doc1.pdf
final.pdf
final_final_v2.pdf

Good:
refund_policy_2024.pdf
product_manual_smartphone.pdf
FAQ_customer_service.pdf

3. Make Text Extractable

PDFs shouldn't be images. Text must be selectable.

How to Check:

• Open PDF
• Drag text with mouse
• If it copies, OK!

If Not:

• Use OCR tools (Adobe Acrobat, Online OCR)
• Or recreate in Word

4. Add Metadata

Add summary at beginning of document:

===== Document Info =====
Title: Refund Policy
Version: 2024.1
Updated: 2024-05-15
Category: Customer Service
Keywords: Refund, Exchange, Return
====================

[Main Content Start]
...

Improving Quality

1. Force Source Citation

[Add to Instructions]

All answers in this format:

[Answer Content]

---
📚 Source: [filename] [page/section]
📅 Updated: [document date]

Example:
Refunds possible within 14 days.

---
📚 Source: refund_policy.pdf page 2
📅 Updated: 2024-05-15

2. Indicate When Uncertain

Show confidence level:

✅ Certain (stated in document)
⚠️ Estimated (similar content reference)
❌ Uncertain (not in document)

Example:
Question: Do you ship internationally?

Answer: ❌ Uncertain
No international shipping info in uploaded documents.
Please contact customer service (1234-5678) for accurate information.

3. Freshness Warning

[Instructions]

For old policies:
If document is over 6 months old,
add warning at end of answer:

"⚠️ This information is as of [date].
Check with staff for latest policy."

Test Checklist

Must test after creating:

✅ Accuracy
□ Does it find content in documents accurately?
□ Doesn't make up content not there?

✅ Completeness
□ Does it cite sources?
□ Does it synthesize multiple documents?

✅ Usability
□ Are answers easy to understand?
□ Does it suggest additional info when needed?

✅ Exception Handling
□ Handles questions not in documents well?
□ Responds to ambiguous questions?

Update Management

What when information changes?

Version Control

Filename with dates:
refund_policy_2024-05.pdf (May version)
refund_policy_2024-06.pdf (June update)

State in Instructions:
"Prioritize documents with latest dates"

Change Log Document

Add changes.txt:

2024-06-01 Update:
- Refund period changed 7 days → 14 days
- Free shipping threshold 20,000 → 30,000 won

2024-05-15 Update:
- New product line added
- Price list updated

Real Use Examples

Example 1: Personal Blog Assistant

Upload:

• 50 blog posts (PDF)
• Category list (TXT)
• Frequently asked questions (DOCX)

Result:

Me: Recommend AI tools

Bot: There are several AI tool recommendation posts!

1. "Top 5 AI Tools for Beginners" (2024-03)
   - Introduces ChatGPT, Claude, Midjourney etc.
   - Free to start

2. "AI Tools for Work Automation" (2024-04)
   - Zapier, Make.com
   - Includes practical examples

3. "AI for Content Creation" (2024-05)
   - Copy.ai, Jasper
   - Tips for bloggers

Which area are you curious about?

Example 2: Internal Company Support Bot

Upload:

• Employee handbook
• Work manual
• IT guide
• Frequently asked questions

Result:

Employee: VPN won't connect

Bot: Here's how to solve VPN connection issues:

1. Restart VPN program
2. Turn Wi-Fi off and on
3. Reconfirm company account password

If still doesn't work:
- Restart computer
- Reinstall VPN

If problem persists:
Contact IT team (ext. 1234).

(Source: IT_guide.pdf, pages 15-16)

Next Steps

In the next post, we'll cover experiencing customer service chatbots.

We'll explore features and how to create customer service chatbots used by actual companies!

Wrapping Up

Custom chatbots are on a different level from general chatbots. They can contain your specialized knowledge.

Key points:

1. Good document preparation
2. Clear Instructions
3. Continuous updates

Start with 10 documents. Doesn't have to be perfect. Improve as you use it.

Create your own expert chatbot!

---

Related Posts:

• [Previous] How to Embed Chatbots in Blogs
• [Next] Experiencing Customer Service Chatbots

What materials did you use for your chatbot? Share in comments!