All stories

Building an AI agent that answers business phone calls

Phoneline.ai gives small businesses an AI agent that answers calls from their own website and documents, and transfers callers to a person when it should.

Role
Lead engineer, end to end
Timeline
January 2025 to now
Stack
Python, FastAPI, PostgreSQL, OpenAI Realtime API, React
Live product
phoneline.ai

The problem

Small businesses miss calls because the owner, the staff or the tradesperson is busy. Callers hit voicemail or hang up, and the enquiry is lost. A receptionist is expensive, and generic phone bots frustrate callers because they don’t know the business.

The idea

Most of what callers ask, like hours, prices, services and policies, is already on the business’s website. Phoneline turns that content into an agent that answers like someone who knows the business, and hands off the calls that need a person.

Setup takes minutes: sign up, connect the website and documents, and forward calls.

Answers every call

Handles common questions without anyone picking up.

Knows the business

Learns from the website and documents. No training sessions or scripts.

Hands off to people

Transfers callers to the team when needed.

Logs every call

A dashboard shows who called and what they needed.

How it works

  1. Build the knowledge baseThe website and any other sources become a searchable knowledge base.
  2. Talk in real timeThe OpenAI Realtime API handles voice in and voice out, so the conversation flows naturally.
  3. Look up the answerThe agent searches the knowledge base for the relevant passages.
  4. Answer from the business's contentReplies rely on what the business has published, not the model’s general knowledge.
  5. Hand off when neededCallers who need a person are transferred, and every call is logged.

What I built

01

AI pipeline

A RAG system over each customer’s sources, plus prompt design and response evaluation.

02

Voice layer

Real-time phone conversations on the OpenAI Realtime API, connected to RAG, with call-transfer logic.

03

Backend

Python FastAPI services, PostgreSQL, third-party integrations, deployment and infrastructure.

04

Dashboard

A React app for sign-up, agent configuration and call logs.

Architecture

CallerPhone networkForwarded line
CoreVoice servicePython, FastAPI
OpenAI Realtime APIVoice in and out
Knowledge baseRAG lookups
Transfer to teamHand-off
Dashboard + APIReact
PostgreSQLAccounts, agents and call logs, shared with the voice service
Simplified architecture.

Need an AI feature like this?

Send a short description and I'll reply with a concrete first step, usually within a day.