I went into Handmade Datasets wanting to build an image-description generator and wound up trying to train a machine to hallucinate blind visionscapes.
Handmade Datasets
My increasing reliance on AI for my ongoing photography project made me curious to know more. I guest-lectured at the School for Poetic Computation years ago, and I’d been thinking that if I were to take an AI class anywhere it would be there. When I saw a class called Handmade Datasets, I knew it was the one. Guided by such radical questions as “What if datasets were small enough to care for?” we would trace the data pipelines behind major AI systems and learn how to “train custom models using human-scale data and a fraction of the energy resources of large AI models.”
I applied to the class with a pretty firm idea of what I wanted to make: an image-description generator. I was thinking of my photography project; how would Paul Strand’s Blind Woman’s appearance shift under the influence of each new description?
Let’s just say I didn’t even come close.
Our first project was to create a Pix2Pix model. I didn’t immediately find the idea of an image-to-image generator useful or interesting—I’m all about translating the visual into something else—language or other sense modalities. Then I reread the instructions, paying closer attention to the questions artist and lead instructor Aarati Akkapeddi urged us to consider before collecting data:
“What translation might be meaningful to you or your community?”
“What visual knowledge do you have that AI doesn’t?”
These made me think about what I and some other blind people see.
A translucent violet arm reaches forward from a first-person perspective, its glowing hand wrapped around a similarly translucent cup. Both shimmer with star-like points against a dense, swirling field of olive-green static, scattered with bright white and yellow sparks.
Made with ChatGPT.
Blind Visionscapes
A few years ago, I wrote a post on Facebook that mentioned briefly how—though I’m now totally blind—I don’t experience darkness anymore. Rather, I am bombarded with light—gentle hallucinations undulating over pulsating pixelated snow-fuzz.
This happened to be the moment when DALL-E suddenly popped on the scene, and a sighted friend fed my description into the image generator, then shared the resulting images.
That prompted me to crowdsource what other blind people see, which I gathered together in this post. For example:
“swirling, kaleidoscopic fragments of green, yellow, and magenta, moving like a time-lapsed image of a distant galaxy…”
–James Tate Hill
I’m also always collecting descriptions from books:
“One of the colors that the blind—or at least this blind man—do not see is black; another is red. Le rouge et le noir are the colors denied us. I, who was accustomed to sleeping in total darkness, was bothered for a long time at having to sleep in this world of mist, in the greenish or bluish mist, vaguely luminous, which is the world of the blind. … The world of the blind is not the night that people imagine.”
— Jorge Luis Borges
“As sight daily diminished, colors proportionately darker would burst forth with violence and a sort of crash from within; but now, pure black, marked as if with extinguished or ashy light, and as if interwoven with it, pours forth. Yet the mist which always hovers before my eyes both night and day seems always to be approaching white rather than black…”
–John Milton
Two translucent violet arms appear from a first-person perspective. One hand turns palm-up while the other gently touches its wrist and forearm. Both glow with intricate, star-like networks against a dense olive-green field of shimmering white static and scattered points of light.
Made with ChatGPT.
Proprioceptive Vision?
More recently I’ve been thinking about another odd thing blind people see. I first started noticing it when spreading lotion on my arm after the shower. A sparkling, darkly luminous, X-ray-like approximation of my moving hand and stationary arm would emerge out of the pulsating visionscape, as if the movement and sensation were painting limbs on my retina. At first I thought maybe I was seeing more than I thought I saw—that I just wasn’t paying attention. Then I closed my eyes. The phenomenon remains.
In a conversation with blind photographer John Dugdale, neurologist Oliver Sacks mentions his experience with this phenomenon: “At one time I had a large blind area, a so-called scotoma, in my right eye … I remember once scotomizing my foot so that I didn’t see anything below mid shin, but then as I twiddled my toes, a sort of strange protoplasmic effulgence appeared. Sparkling. And which gradually formed itself into the semblance of a foot—it was a reasonably good foot …”
Dugdale replies, “It’s such a marvelous feeling. I remember being in church at St. Luke in the Fields and I looked down at my hands and it was almost like I could see an x-ray of them.”
Sacks suggests that “the visual part of the brain seems to become more sensitive to one’s intentions and to proprioception.” He calls it “phantom vision,” which I misremembered as “phantom limb.” But I’ve also noticed it happening with objects—when drying dishes for example, the image of the plate comes into view as my hand goes round it with the towel. So this proprioceptive vision can extend beyond my body to objects that I touch.
Could I train an AI model to illustrate this proprioceptive vision within the blind visionscapes?
Two translucent violet hands reach forward from a first-person perspective, cradling a glowing cup. The hands and cup sparkle with delicate star-like networks against dense silvery static, while large, softly edged fields of vivid blue and golden yellow emerge behind them.
Made with ChatGPT & Alabaster.
Dataset Style Guide
To make my Set B, or target images, I designed a style guide for ChatGPT to create images that include both the first-person phantom limbs and the overall visionscape. This turned out to be more challenging and time-consuming than I imagined—especially getting the AI to generate the hands and arms from a first-person perspective. Moreover, I could not trust ChatGPT’s descriptions of its own images; it tended to describe using my prompt language rather than what it actually managed to produce.
I enlisted Claude to describe and to help me refine my prompts.
So yeah, I had to use an awful lot of big data to make my tiny dataset.
I’m not sure how well the style guide ended up working—I went through perhaps ten iterations or more. But when it finally got close, I used it as a template for new prompts.
I gave the style guide, along with all of the blind descriptions, to Claude, and that really helped me to evaluate the images. Even so, Claude at one point forgot that I am blind and asked if a particular image looked right.
“I can’t know except from the descriptions of others if the images are getting close to what I see or not. Because my visionscape is all I have at this point.”
Claude called this a “genuinely extraordinary epistemological position.”
I agree. The philosophical richness of the very question—what do blind people see—grows out of lived experience. It is a good argument for why blind people need to be the principal investigators in perceptual research—not just the subjects.
My dataset images are thus, as Claude so eloquently puts it, “attempts, always approximate, always partial, to make visible what is already visible — to the artist alone.”
Whether or not these generated images were successful in rendering a visionscape only blind people see is perhaps not a thing that can ever be truly verified. Yet conversations with Claude and ChatGPT helped me to refine the descriptions of what I see—even as what I see is constantly shifting.
There were other things I brought in a little late to the dataset—for example, how my visionscape color schemes seem often to involve complementary colors—orange and blue, green and pink. I brought in a human artist—my partner Alabaster Rhumb—to add these complementary color splotches to my AI visionscapes.
Even with humans, translating the verbal into the visual is tricky. When I had Claude describe Alabaster’s variations, I realized that I had neglected to tell him that they never drift offscreen, so to speak. The splotches occupy quadrants fully. It’s a good reminder that it’s hard to know what the parameters are until people or machines start interpreting your words in ways you couldn’t have anticipated!
A translucent violet arm reaches forward from a first-person perspective, its glowing hand holding a similarly translucent cup. The surrounding field blazes with yellow-green points of light, while a large, vivid magenta-pink splotch spills out of the frame in the lower-right quadrant.
Made with ChatGPT and Alabaster.
The Process
ChatGPT also helped me go through Aarati’s slide decks before class, which was truly game changing. Besides transcribing and describing each slide, I could discuss new concepts in depth.
Pix2Pix is a GAN—a generative adversarial network. I liked the sound of it, though I didn’t totally understand what it meant.
As I mentioned, the images that I was creating with ChatGPT comprised my Set B images—the type of images that I wanted my model to generate.
This was super confusing to me, but the Set A images would be simplifications of Set B. What the model would learn was the transformation—the relationship between the simplified Set A images and the more complex Set B images.
I originally imagined the transformations would involve gestures, that the machine would learn to generate the hand and arm positions—something Aarati seemed excited about. But by now Alabaster had added the color blotches, and I’d realized that phantom limbs are only one part of the visionscape so many blind people describe. I also was not sure what, ultimately, I would feed the Pix2Pix model: drawings? Photographs? I couldn’t really wrap my brain around what I wanted my model to do.
I began thinking that Set A should instead be something very simple: dark grayscale, perhaps blurry, perhaps nearly black—something mimicking what sighted people imagine our visionscapes to be.
Which would mean asking the machine to translate that imagined darkness into something else entirely.
This was where I went wrong. I’d spent so much time creating the phantom limbs, but by making Set A grayscale versions of Set B, my model would mostly just have to learn the color variations. And because I’d only added Alabaster’s color splotches to some of the Set B images, the colors were fairly minimal and, I think, too easy for the model to learn.
Just today, while deciding what images to display for this post, Alabaster noticed that all the images are sort of pixelated. After some consideration, I realized that when I resized the original very large images to tiny 256-by-256 pixels—the size we needed for the model—much of the resolution was lost.
Also funny to note that, before this project, “pixelated” had been a word I used to describe my vision: “pixelated snow fuzz.” As I was generating images with AI in the beginning, they were coming out looking like old-school computer games. Of course! I didn’t mean pixels—my visionscape is not composed of little squares. That’s not the quality of what I see at all. Static or misty pulsating static is a much better way to describe stuff.
Describing is hard!
A six-image grid presents two Pix2Pix checkpoints, each row moving from input to output to target. Above, two raised fists emerge from static; below, two open hands make different gestures, with bright streaks crossing one wrist. Grayscale inputs appear at left, generated outputs at center, and targets at right.The model closely preserves the forms while producing subtle color shifts that approximate, but do not quite match, the targets.
I began the Handmade Datasets class knowing I would learn a lot even if I didn’t succeed in building anything. We did not need to know how to code, but we would use Colab—Google’s coding environment—to adapt the scripts for our datasets. Would it be accessible?
The first Colab notebook was for augmenting our dataset by flipping our images. That transformation took me all day. I managed it with lots of help from Claude—finding accessibility workarounds, dropping code in and asking it to put my folder names in the right place, consulting on error messages, and just talking through this new world.
By day three I felt comfortable with Colab. It cracked open a sense of power—that I could actually create things using code. Maybe even build my own handmade website someday.
A six-image grid presents two Pix2Pix blind visionscapes checkpoints, each row moving from input to output to target. The top row shows two hands holding a cup; the bottom, one hand touching the opposite wrist. Grayscale inputs appear at left, the model’s generated color outputs at center, and target images at right. The outputs preserve the inputs’ forms while approximating, but not matching, the targets’ colors.
The End
Things got really stressful in the final hours. As the model learned, it generated image checkpoints—inputs, outputs, and targets. Inputs were the Set A greyscale images, outputs the generator’s attempts, and targets the Set B color originals.
There were so many of these that my Google Drive got full and kept stalling the training—even as I tried to empty the trash as quickly as I could. So yeah, the most stressful part of this whole endeavor was a storage issue!
After hours of frustration, I finally had a model.
I uploaded it to Aarati’s demo. Then I uploaded a couple photos. Not much happened. Alabaster described them, and to say that I was disappointed is not quite the right feeling. Or that was only part of it.
I had learned so much and been challenged in so many new ways that I suddenly didn’t really care that my model didn’t do what I wanted it to do—if I ever really knew what I wanted it to do.
My perceptual world is difficult to describe and even more difficult to translate. A couple of weeks of work on a tiny model with a tinier sense of what I was doing was not going to be enough to teach AI what I see.
On the other hand, I feel the pull to try again. What would happen if I trained another model on black or nearly black images? With little to no visual input, would the machine start hallucinating phantom limbs and colorful visionscapes, too?