I used two different separate large language models (LLM) and two separate customization approaches in this project. Each method has different strengths, and so I felt it was important to explore both and display their outputs together on this site. Both models were already pretrained on a large dataset when I augmented them with my own writing, every scrap of it I could get my hands on. This included journals, letters, emails, blog posts, fictional stories, poems, and all sorts of schoolwork, academic, and professional writing–if I wrote it, retained the rights to it, and held onto it in either a digital or physical format until summer 2024, it made its way into the dataset.

Method 1: I used a relatively small model (GPT-Neo-125m) and trained it directly on my writing. Causal language models generate text from a prompt by predicting the next likely word or token in a sequence. It takes a lot of data to train a model to produce coherent sentences from scratch, much more writing than I have or will produce. GPT-Neo had already been trained on massive amounts of data and could respond to prompts legibly; my additional training (known as fine-tuning) tweaked its parameters so that it would better mimic my specific language patterns.

Method 2: Instead of actually training models further, you can also customize them by adding information into their prompt, a method known as Retrieval-Augmented Generation or RAG. In order to find the most relevant context automatically, I broke all of my writing up into chunks and embedded them into a vector knowledge base–a high-dimensional space which groups chunks of similar meaning close together. Any time I entered a question, the question was also embedded to find the 7 chunks closest to it in the embedding space. I then fed those chunks to a question-and-answer model (Qwen2-7B-Instruct), instructing it to act as a digital clone of Sarah Immel, who had written these documents, and answer a question in her voice based on the chunks of context.

For the detailed breakdown of training parameters, prompts, and generation settings, as well as the specific strengths and weaknesses of each approach, you can read my dissertation, which I’ll add a link to here once my project and assessment have been finalized.