I Tried to Sort Embeddings with Fourier Transform and it didn’t go well

A Fourier transform is kind of magical in that it can take a signal and convert it into sub signals that make up that main signal. I thought that was fairly interesting and decided to try it with embeddings.

The thought is that the embeddings would provide the semantic side of the signal, while comparing multiple embeddings with the Fourier would provide the comparision. The deep desire was to open up the black box of these embeddings and try to discover some of the patterns underneath the data.

So how did I go about this? I made a Google Colab sheet that took two emotions and classified a word against those emotions. I then expanded it to have a matrix of emotions. That’s when things started to break down. It couldn’t tell if the word “bafflement” was “confusing” or “sad”. No matter how I tweaked the parameters… it always said the wrong answer: bafflement = sad. Let’s take a look at my experiment. Please note, you’ll need your own API Key.

My Google Colab experiment:

import numpy as np
import scipy.signal as signal
import matplotlib.pyplot as plt
import openai
from sklearn.cluster import KMeans

# Set up OpenAI API key
openai.api_key = "your key here"

def get_embeddings(word_list):
    embeddings = []
    for word in word_list:
        response = openai.Embedding.create(model="text-embedding-ada-002", input=[word])
        # Extract the embedding from the response
        embedding = response["data"][0]["embedding"]
        embeddings.append(embedding)
    return embeddings

# Define function to perform spectral analysis
def perform_spectral_analysis(embeddings):
    spectrum = []
    for embedding in embeddings:
        # Perform FFT on embedding
        embedding_spectrum = np.fft.fft(embedding)
        # Compute magnitude spectrum
        magnitude_spectrum = np.abs(embedding_spectrum)
        spectrum.append(magnitude_spectrum)
    return spectrum

# Define function to identify motif clusters using KMeans clustering
def identify_motif_clusters(frequencies, num_clusters):
    kmeans = KMeans(n_clusters=num_clusters, random_state=0, n_init=100, max_iter=10000, tol=1e-8)  # Adjusting parameters
    kmeans.fit(frequencies.reshape(-1, 1))
    motif_clusters = kmeans.cluster_centers_
    return motif_clusters

# Define function to plot spectral analysis results
def plot_spectral_analysis(spectrum, peaks):
    plt.plot(spectrum, label='Spectrum')
    plt.plot(peaks, spectrum[peaks], "x", label='Peaks')
    plt.legend()
    plt.xlabel('Frequency')
    plt.ylabel('Magnitude')
    plt.title('Spectral Analysis')
    plt.show()

# Define words for different emotions
emotions = {
    "Happy": ["Joy", "Laughter", "Love", "Contentment", "Bliss", "Exuberance", "Delight", "Elation", "Ecstasy", "Euphoria", "Optimism"],
    "Sad": ["Grief", "Despair", "Melancholy", "Anguish", "Misery", "Despondency", "Woe", "Sorrow", "Gloom", "Desolation", "Depression"],
    "Confused": ["Confusion", "Bewilderment", "Perplexity", "Puzzlement", "Disorientation", "Bafflement", "Uncertainty", "Doubt", "Mystification", "Ambiguity", "Incomprehension"]
}

# Get embeddings for words associated with each emotion
emotion_embeddings = {}
for emotion, word_list in emotions.items():
    emotion_embeddings[emotion] = get_embeddings(word_list)

# Combine embeddings for spectral analysis
combined_embeddings = np.concatenate(list(emotion_embeddings.values()))

# Perform spectral analysis for combined embeddings
combined_spectrum = perform_spectral_analysis(combined_embeddings)

# Identify prominent peaks for combined spectrum
combined_peaks, _ = signal.find_peaks(np.mean(combined_spectrum, axis=1), height=0.5)

# Plot spectral analysis results
plot_spectral_analysis(np.mean(combined_spectrum, axis=1), combined_peaks)

# Extract frequencies corresponding to prominent peaks for combined spectrum
combined_frequencies = np.fft.fftfreq(len(combined_spectrum[0]), 1/1000)
combined_prominent_frequencies = combined_frequencies[combined_peaks]

# Identify motif clusters for combined spectrum
num_clusters = len(emotions)
motif_clusters = identify_motif_clusters(combined_prominent_frequencies, num_clusters)

print("Identified Motif Clusters:", motif_clusters)

def calculate_distance(word_embedding, clusters):
    distances = np.linalg.norm(clusters - word_embedding, axis=1)
    return distances

def check_word_emotion(word):
    try:
        # Retrieve embedding for the word
        response = openai.Embedding.create(model="text-embedding-ada-002", input=[word])
        embedding = response["data"][0]["embedding"]

        # Perform spectral analysis on the embedding
        spectrum = np.fft.fft(embedding)

        # Apply KMeans clustering to identify motif clusters
        kmeans = KMeans(n_clusters=num_clusters, n_init=100, max_iter=10000, tol=1e-8)  # Adjusting parameters
        kmeans.fit(spectrum.real.reshape(-1, 1))  # Use real part of the spectrum for clustering

        # Get the cluster centers
        cluster_centers = kmeans.cluster_centers_.flatten()

        # Find the closest cluster centroid
        distances = calculate_distance(embedding, motif_clusters)
        closest_cluster_idx = np.argmin(distances)

        # Determine the associated emotion based on the closest cluster
        emotion = list(emotions.keys())[closest_cluster_idx]
        return emotion

    except Exception as e:
        print("Error:", e)
        return None

# Example usage:
word = "Bafflement"
emotion = check_word_emotion(word)
print(f"The word '{word}' is associated with {emotion} emotion.")

# Calculate distances from the word embedding to each cluster centroid
word_embedding_response = openai.Embedding.create(model="text-embedding-ada-002", input=[word])
word_embedding = np.array(word_embedding_response["data"][0]["embedding"])
distances = calculate_distance(word_embedding, motif_clusters)

# Print the distances
for i, distance in enumerate(distances, start=1):
    centroid_label = list(emotions.keys())[i - 1]
    print(f"Distance from '{word}' to {centroid_label} centroid: {distance}")

So, I tried looking for something comparable that would work better. I found DBSCAN, which gave the correct answers:

import numpy as np
import openai
from sklearn.cluster import DBSCAN
from sklearn.metrics import pairwise_distances_argmin_min

# Set up OpenAI API key
openai.api_key = "your key here"

def get_embeddings(word_list):
    embeddings = []
    for word in word_list:
        response = openai.Embedding.create(model="text-embedding-ada-002", input=[word])
        # Extract the embedding from the response
        embedding = response["data"][0]["embedding"]
        embeddings.append(embedding)
    return embeddings

# Define words for different emotions
emotions = {
    "Happy": ["Joy", "Laughter", "Love", "Contentment", "Bliss", "Exuberance", "Delight", "Elation", "Ecstasy", "Euphoria", "Optimism"],
    "Sad": ["Grief", "Despair", "Melancholy", "Anguish", "Misery", "Despondency", "Woe", "Sorrow", "Gloom", "Desolation", "Depression"],
    "Confused": ["Confusion", "Bewilderment", "Perplexity", "Puzzlement", "Disorientation", "Bafflement", "Uncertainty", "Doubt", "Mystification", "Ambiguity", "Incomprehension"]
}

# Get embeddings for words associated with each emotion
emotion_embeddings = {}
for emotion, word_list in emotions.items():
    embeddings = get_embeddings(word_list)
    valid_embeddings = [embedding for embedding in embeddings if embedding]  # Filter out empty embeddings
    emotion_embeddings[emotion] = valid_embeddings

# Combine embeddings for DBSCAN clustering
combined_embeddings = np.concatenate([np.array(embeddings) for embeddings in emotion_embeddings.values()])

# Perform DBSCAN clustering
db = DBSCAN(eps=0.5, min_samples=2).fit(combined_embeddings)
labels = db.labels_

# Identify cluster centers
unique_labels = set(labels)
cluster_centers = []
for label in unique_labels:
    if label != -1:  # Exclude noise points
        cluster_points = combined_embeddings[labels == label]
        cluster_center = np.mean(cluster_points, axis=0)
        cluster_centers.append(cluster_center)

# Classify words based on closest cluster
def classify_word(word):
    embedding = np.array(get_embeddings([word])[0])
    if embedding.size == 0:
        return "Invalid embedding"

    distances = pairwise_distances_argmin_min(embedding.reshape(1, -1), cluster_centers)
    closest_cluster_idx = distances[0][0]
    return list(emotions.keys())[closest_cluster_idx]

# Example usage:
#rainy clouds = sad
#lottery = happy
#bafflement = confused
word = "rainy clouds"
emotion = classify_word(word)
print(f"The word '{word}' is associated with {emotion} emotion.")

That left me pretty frustrated, as I couldn’t understand why the FFT wouldn’t work, but DBSCAN worked great. Here is my conclusion:

Imagine you have a room full of your friends. My thought with FFT (Fourier Transform) is that you can tell the group of people by how much noise they make. However, some friends may not make a lot of noise… which means it’s not the best algorithm to use. Instead, you want one that measures the distance between those friends. That’s what DBSCAN does. So if your friends are quiet, but all clumped together… that’s relevant. It can be as relevant or more than if you have one loud mouth at the end of the room.

Hopefully this was useful! I know I enjoyed digging into learning patterns and will probably try a few different methods. Also — make sure to ignore people when they say “it’s only for… xyz”. Try it out anyway and learn why they say that. I know Fourier Transform isn’t made for embeddings… but it was still really useful for me to learn why.