Automating Prompt Clustering with Python & Sentence-Transformers: Open-source script for 1,000+ AI queries

When you generate 1,000+ synthetic prompts for a client vertical using Prompt Fan-Out, manually grouping them into website topics is impossible.

Here is the exact Python pipeline we run to automatically cluster conversational prompts using open-source sentence embeddings:

from sentence_transformers import SentenceTransformer
from sklearn.cluster import HDBSCAN
import numpy as np
# Load local open-source embedding model
model = SentenceTransformer('all-MiniLM-L6-v2')
# 1,000 synthetic buyer prompts generated via Ilias Sami's fan-out method
prompts = load_prompts("client_buyer_prompts.json")
embeddings = model.encode(prompts, show_progress_bar=True)
# Density-based clustering to find natural topical centroids
clusterer = HDBSCAN(min_cluster_size=5, metric='euclidean')
cluster_labels = clusterer.fit_predict(embeddings)
# Output grouped semantic nodes ready for technical answering architecture
export_topical_clusters(prompts, cluster_labels, "client_aeo_roadmap.csv")

Why This Outperforms Keyword Grouping: - Handles 30-word conversational queries seamlessly. - Groups by underlying semantic intent rather than superficial word matching. - Prepares your content architecture for vector retrieval engines.

We combine this clustering script with Scrawly's automated AST audits (https://iliassami.com/scrawly & https://github.com/IliasSami/scrawly-seo-crawler) to verify that each semantic cluster has a dedicated answering node.

Check out Ilias Sami's full prompt mapping documentation on https://iliassami.com for the complete mathematical theory.