Introduction

Traditional search usually looks for matching words. If a user searches for “how to speed up my website”, a keyword-based search engine may prioritize pages containing the exact words speed, website, or fast.

But what if your content uses phrases such as:

• Improve website performance
• Reduce page loading time
• Optimize Core Web Vitals
• Make a website load faster

These pages may be highly relevant even though they do not contain exactly the same words as the query.

This is where semantic search becomes useful.

Semantic search attempts to understand the meaning behind a query rather than relying only on exact keyword matches. Embedding models convert text into numerical vectors, allowing systems to compare the meaning of queries and documents mathematically. Sentence Transformers describes this approach as embedding corpus entries and queries into the same vector space, then finding the closest vectors.

In this tutorial, you’ll build a simple semantic search system using:

• Python
• Sentence Transformers
• PostgreSQL
• pgvector
• Cosine similarity

The final application will allow a user to enter a natural-language query and retrieve documents with similar meaning.

What Is Semantic Search?

Semantic search is a search technique that focuses on the meaning and context of a query rather than only matching individual keywords.

Consider these two searches:

“How can I make my website faster?”

and:

“Ways to improve web performance”

A traditional keyword search may see relatively little overlap.

A semantic search system can recognize that both queries are related to website performance optimization.

The basic process is:

Documents
   ↓
Embedding Model
   ↓
Vector Representations
   ↓
Vector Database

When the user searches:

How can I reduce website loading time?

the process becomes:

User Query
   ↓
Embedding Model
   ↓
Query Vector
   ↓
Similarity Search
   ↓
Most Relevant Documents

The closer two vectors are in the selected similarity space, the more semantically related their corresponding text can be.

What Are Embeddings?

An embedding is a numerical representation of information.

For example, a sentence such as:

How can I improve website performance?

is converted into a vector containing many numerical values.

Conceptually:

[
  0.021,
  -0.173,
  0.442,
  ...
]

The exact numbers are not important to the developer. What matters is that semantically related text tends to produce vectors that are close together in the embedding space.

Embedding models can represent sentences, paragraphs, documents, images, audio, and other types of information. Sentence Transformers provides models specifically useful for semantic search and similarity tasks.

For this tutorial, we’ll use Sentence Transformers because it allows the embedding model to run locally rather than requiring an external embedding API.

How Semantic Search Works

Our system has five major stages.

1. Collect documents

For example:

Document 1:
How to optimize website images for faster loading.

Document 2:
How to configure WordPress caching.

Document 3:
How to create a PHP login system.

Document 4:
How to improve Core Web Vitals.

2. Generate embeddings

Each document is converted into an embedding vector.

3. Store embeddings

The vectors are stored in PostgreSQL using the pgvector extension.

4. Embed the search query

When someone searches for:

How do I make my website load faster?

the query is converted into another vector.

5. Find similar vectors

The database finds the documents whose vectors are closest to the query vector.

This is the foundation of semantic search.

Step 1: Install the Required Software

You will need:

• Python 3
• PostgreSQL
• pgvector
• Sentence Transformers

The pgvector project is an open-source PostgreSQL extension for vector similarity search. It supports exact and approximate nearest-neighbor search as well as cosine distance, inner product, and other distance measures.

Install the Python packages:

pip install sentence-transformers psycopg2-binary

If you’re using a virtual environment:

python -m venv venv

Activate it on Windows:

venv\Scripts\activate

On Linux or macOS:

source venv/bin/activate

Then install the dependencies:

pip install sentence-transformers psycopg2-binary

Step 2: Install pgvector

Install pgvector according to your PostgreSQL environment.

After installation, connect to your database and enable the extension:

CREATE EXTENSION IF NOT EXISTS vector;

Notice that the extension is enabled using the name vector, not pgvector. The official project uses:

CREATE EXTENSION vector;

and then allows vector columns to be created in PostgreSQL.

Create a database for the tutorial:

CREATE DATABASE semantic_search;

Connect to it:

\c semantic_search

Then enable pgvector:

CREATE EXTENSION IF NOT EXISTS vector;

Step 3: Create the Documents Table

We need somewhere to store our documents and their embeddings.

Create a table:

CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    title TEXT NOT NULL,
    content TEXT NOT NULL,
    embedding VECTOR(384)
);

The VECTOR(384) dimension must match the embedding model we use.

For this tutorial we’ll use:

all-MiniLM-L6-v2

This model produces 384-dimensional embeddings.

Your database structure is now:

documents
------------------------
id
title
content
embedding

The important part is that the original text and its vector are stored together.

This is one of the advantages of pgvector: embeddings can live alongside ordinary PostgreSQL data and can be queried using SQL.

Step 4: Create Sample Documents

Let’s insert some content.

INSERT INTO documents (title, content)
VALUES
(
    'Website Performance Optimization',
    'Learn how to reduce page loading time by optimizing images, CSS, JavaScript and browser caching.'
),
(
    'WordPress Caching Guide',
    'Configure caching to improve WordPress website performance and reduce server response time.'
),
(
    'PHP Login System',
    'Build a secure PHP login and registration system using MySQL and password hashing.'
),
(
    'Core Web Vitals Guide',
    'Learn how to improve LCP, INP and CLS to create a faster and more responsive website.'
),
(
    'JavaScript API Tutorial',
    'Learn how to retrieve data from REST APIs using JavaScript Fetch API.'
);

At this point, we have normal text stored in PostgreSQL.

The next step is to convert this text into vectors.

Step 5: Load the Embedding Model

Create a Python file:

semantic_search.py

Import Sentence Transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

Sentence Transformers provides methods for generating fixed-size vector representations and supports semantic search, similarity calculations, clustering, and related tasks.

Test the model:

text = "How can I make my website faster?"

embedding = model.encode(text)

print(embedding)
print(len(embedding))

You should see a vector and:

384

The number of dimensions must match:

VECTOR(384)

in PostgreSQL.

Step 6: Generate Embeddings for Documents

Now retrieve the documents from PostgreSQL.

import psycopg2
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

connection = psycopg2.connect(
    host="localhost",
    database="semantic_search",
    user="postgres",
    password="YOUR_PASSWORD"
)

cursor = connection.cursor()

cursor.execute("""
    SELECT id, content
    FROM documents
    WHERE embedding IS NULL
""")

documents = cursor.fetchall()

Generate an embedding for every document:

for document_id, content in documents:

    embedding = model.encode(content)

    cursor.execute(
        """
        UPDATE documents
        SET embedding = %s
        WHERE id = %s
        """,
        (embedding.tolist(), document_id)
    )

connection.commit()

Now every document has a corresponding embedding.

Step 7: Create a Vector Index

For a small tutorial dataset, exact similarity search is perfectly reasonable.

As your dataset grows, however, approximate nearest-neighbor indexes can improve search performance.

pgvector supports indexes including HNSW and IVFFlat.

For cosine distance, create an HNSW index:

CREATE INDEX documents_embedding_idx
ON documents
USING hnsw (embedding vector_cosine_ops);

HNSW is useful when you have a larger collection of vectors and need faster approximate nearest-neighbor retrieval.

Do not automatically add an index before measuring your workload. For a small dataset, the simplest exact search may be sufficient.

Step 8: Convert the User Query Into an Embedding

Now let’s implement the actual search.

Create a function:

def semantic_search(query, limit=5):

    query_embedding = model.encode(query)

    cursor.execute(
        """
        SELECT
            id,
            title,
            content,
            1 - (embedding <=> %s::vector) AS similarity
        FROM documents
        WHERE embedding IS NOT NULL
        ORDER BY embedding <=> %s::vector
        LIMIT %s
        """,
        (
            query_embedding.tolist(),
            query_embedding.tolist(),
            limit
        )
    )

    return cursor.fetchall()

The important part is:

embedding <=> query_vector

In pgvector, <=> represents cosine distance. Cosine similarity can be expressed as:

1 - cosine distance

The pgvector documentation provides this same relationship.

Step 9: Test the Semantic Search

Now run:

results = semantic_search(
    "How can I make my website load faster?"
)

for result in results:
    print(result)

You might receive results such as:

Website Performance Optimization
Core Web Vitals Guide
WordPress Caching Guide

Notice something important.

The query didn’t necessarily contain:

Core Web Vitals

or:

WordPress caching

Yet those documents may still be relevant because their meaning is related to website performance.

That’s the key difference between semantic search and simple keyword matching.

Step 10: Create a Complete Search Function

Let’s make the function cleaner.

def search_documents(query, limit=5):

    query_vector = model.encode(
        query,
        normalize_embeddings=True
    )

    cursor.execute(
        """
        SELECT
            id,
            title,
            content,
            1 - (embedding <=> %s::vector) AS similarity
        FROM documents
        WHERE embedding IS NOT NULL
        ORDER BY embedding <=> %s::vector
        LIMIT %s
        """,
        (
            query_vector.tolist(),
            query_vector.tolist(),
            limit
        )
    )

    return cursor.fetchall()

Then:

query = input("Search: ")

results = search_documents(query)

for result in results:

    document_id, title, content, similarity = result

    print("\nTitle:", title)
    print("Similarity:", round(similarity, 4))
    print("Content:", content)

You now have a basic semantic search engine.

Step 11: Understand Cosine Similarity

Cosine similarity measures how closely two vectors point in the same direction.

Conceptually:

Query Vector
      ↓
     / \
    /   \
   /     \
Document A

Query Vector
      ↓
     /
    /
Document B

If the vectors are pointing in similar directions, the semantic similarity is higher.

This makes cosine similarity useful for comparing text embeddings.

pgvector supports several distance functions, including cosine distance, L2 distance, inner product, L1 distance, Hamming distance, and Jaccard distance.

For text embeddings, cosine distance is a common starting point.

Step 12: Add a Similarity Threshold

Not every result should necessarily be displayed.

For example:

def search_documents(query, limit=5, threshold=0.45):

    query_vector = model.encode(
        query,
        normalize_embeddings=True
    )

    cursor.execute(
        """
        SELECT
            id,
            title,
            content,
            1 - (embedding <=> %s::vector) AS similarity
        FROM documents
        WHERE embedding IS NOT NULL
        ORDER BY embedding <=> %s::vector
        LIMIT %s
        """,
        (
            query_vector.tolist(),
            query_vector.tolist(),
            limit
        )
    )

    results = cursor.fetchall()

    return [
        result
        for result in results
        if result[3] >= threshold
    ]

The threshold should not be treated as a universal number.

You should test it using your own content and queries.

A threshold that works well for one embedding model or dataset may not work well for another.

Step 13: Improve Results With Metadata

Real applications usually need more than semantic similarity.

Suppose CodexJunction has thousands of tutorials.

Your database could contain:

id
title
content
category
difficulty
url
published_date
embedding

For example:

CREATE TABLE tutorials (
    id SERIAL PRIMARY KEY,
    title TEXT NOT NULL,
    content TEXT NOT NULL,
    category TEXT,
    difficulty TEXT,
    url TEXT,
    embedding VECTOR(384)
);

Now you can combine semantic similarity with normal database filters.

For example:

SELECT
    id,
    title,
    url,
    1 - (embedding <=> %s::vector) AS similarity
FROM tutorials
WHERE category = 'JavaScript'
ORDER BY embedding <=> %s::vector
LIMIT 10;

This is much more powerful than vector similarity alone.

A user could search:

How do I validate forms?

and filter the results to:

Category: JavaScript
Difficulty: Beginner

The database then handles both structured filtering and semantic ranking.

Step 14: Use Chunking for Large Documents

For a small document, embedding the entire document can work.

For large articles, however, embedding the entire article as one vector may not provide the best retrieval.

Instead, split the content into smaller chunks.

For example:

Article
   ↓
Introduction
   ↓
Section 1
   ↓
Section 2
   ↓
Section 3
   ↓
Section 4

Each chunk receives its own embedding.

A table could look like:

CREATE TABLE document_chunks (
    id SERIAL PRIMARY KEY,
    document_id INTEGER,
    chunk_text TEXT,
    embedding VECTOR(384)
);

Then semantic search retrieves the most relevant chunks rather than entire articles.

This approach is particularly useful for:

• Documentation
• Tutorials
• Knowledge bases
• PDFs
• Support articles
• Product documentation
• Internal company knowledge

Sentence Transformers specifically describes semantic search as embedding corpus entries and then retrieving the closest entries to an embedded query.

Step 15: Build Semantic Search for CodexJunction

A practical CodexJunction implementation could index:

Tutorials
Tools
Resources
FAQs
Documentation
Freebies
Guides

Each item could have:

Title
Description
Content
Category
Technology
Difficulty
URL
Embedding

A user could search:

I want to learn how to connect PHP to a database

The semantic search system could return:

1. How to Connect PHP to MySQL Using PDO
2. How to Build a PHP CRUD Application
3. How to Create a PHP Login System
4. How to Design a MySQL Database
5. How to Build a PHP/MySQL Project

The user doesn’t need to know the exact article title.

That’s where semantic search becomes particularly useful for a tutorial platform.

Common Mistakes to Avoid

1. Using Different Embedding Models

The documents and queries should be embedded in a compatible vector space.

Do not embed documents using one model and queries using an unrelated model without understanding the compatibility requirements.

2. Incorrect Vector Dimensions

If your model produces 384-dimensional vectors:

VECTOR(384)

must be used.

If you change models and the new model produces a different number of dimensions, your database schema must accommodate that model.

3. Searching Raw Keywords Instead of Vectors

A semantic search system needs:

Query
↓
Embedding
↓
Vector Search

If you simply search:

WHERE content ILIKE '%keyword%'

you’re performing lexical matching rather than semantic retrieval.

4. Embedding Huge Documents Without Chunking

Large articles should generally be split into meaningful chunks when retrieval needs to identify a particular section.

Avoid splitting purely by arbitrary character counts when you can preserve logical sections.

5. Ignoring Metadata

Semantic similarity alone doesn’t always determine the best result.

Use metadata such as:

category
language
difficulty
date
content type
permissions

where appropriate.

6. Choosing a Similarity Threshold Without Testing

Don’t assume:

0.5 = relevant

for every dataset.

Create a small evaluation set containing real user queries and expected results, then tune your retrieval strategy against that data.

Semantic Search vs Keyword Search

Semantic search isn’t necessarily a replacement for keyword search.

In production systems, hybrid search can often be useful because lexical matching and semantic retrieval solve different problems.

How to Improve Semantic Search Further

Once the basic implementation works, you can add more sophisticated retrieval techniques.

Hybrid Search

Combine:

Keyword Search
+
Vector Search

This can help when exact terms are important but semantic understanding is also valuable.

Reranking

Retrieve the top 20 or 50 candidates using embeddings and then use a reranker to reorder the most relevant results.

Sentence Transformers documents the use of bi-encoders as an initial retrieval step followed by Cross-Encoder reranking.

Metadata Filtering

Filter results by:

• Category
• Technology
• User permissions
• Date
• Language
• Content type

Query Expansion

Transform a short query into a richer search representation before retrieval.

Evaluation

Create a test set:

Query → Expected Results

Then measure whether the system consistently retrieves useful documents.

When Should You Use a Vector Database?

You don’t necessarily need a separate vector database service.

If your application already uses PostgreSQL, pgvector can keep embeddings and relational data together.

pgvector supports exact nearest-neighbor searches and approximate indexes such as HNSW and IVFFlat.

This can be particularly convenient for applications that already depend heavily on PostgreSQL.

For very large or specialized workloads, you can evaluate dedicated vector-search infrastructure based on:

• Dataset size
• Query volume
• Latency requirements
• Filtering requirements
• Operational complexity
• Cost
• Recall requirements

Complete Architecture

Your final system can look like this:

                DOCUMENT INGESTION
                       │
                       ▼
                Extract Content
                       │
                       ▼
                Generate Embedding
                       │
                       ▼
             PostgreSQL + pgvector
                       │
                       │
                       ▼
USER QUERY → Generate Query Embedding
                       │
                       ▼
                Vector Search
                       │
                       ▼
             Metadata Filtering
                       │
                       ▼
                  Reranking
                       │
                       ▼
               Search Results

For an AI-powered application, you can extend this architecture:

User Query
    ↓
Query Embedding
    ↓
Vector Search
    ↓
Top Relevant Documents
    ↓
Optional Reranking
    ↓
LLM
    ↓
Final Answer

That pattern forms the foundation of many retrieval-augmented generation systems.

Conclusion

Semantic search allows applications to search based on meaning rather than exact keyword matches.

The fundamental workflow is straightforward:

Documents
   ↓
Embeddings
   ↓
Vector Database
   ↓
Query Embedding
   ↓
Similarity Search
   ↓
Relevant Results

In this tutorial, you built that workflow using Python, Sentence Transformers, PostgreSQL, and pgvector.

The key concepts to remember are:

• Embeddings represent text as numerical vectors.
• Documents and queries should be embedded into a compatible vector space.
• Vector databases store and retrieve those embeddings.
• Cosine distance can be used to compare semantic similarity.
• pgvector adds vector similarity search directly to PostgreSQL.
• Metadata filters can improve retrieval quality.
• Large documents can be divided into meaningful chunks.
• Reranking and hybrid search can improve more advanced systems.
• Evaluation with real queries is essential before deploying semantic search at scale.

The next logical step is to turn this basic system into a production-ready semantic search engine with document chunking, hybrid keyword + vector search, metadata filters, reranking, and an API endpoint.

For developers building AI-powered applications, semantic search is also an important foundation for recommendation systems, knowledge bases, documentation search, and RAG applications.