The command prompt: A black, empty screen and a blinking line, waiting for my input. Could you imagine that this prompt is the access to art or AI art? The command line shows a visual result of your terms, not ASCII art or related items. Nowadays, those prompts have different shapes, such as a form field or the use of voice interfaces.
Your imagination of thought, translated to words in your natural language (mostly in English), gives you access to new creativity. In the words of advertising, you need to express a good core idea, you want to get visualized. Similar to creatives being used to Image search engines, the need to find the correct words gets more and more important.
To explore my thoughts I look at the evolution of the way we tell machines, what to do and how CLIP and other neural models transform the way we could create what we are looking for. Whether it be AI art or maybe any picture we want to see, in general or what I would call AI graphics.
Computer language
A computer language is a language we use to talk to machines. It has been in use for about 70 years.In terms of human history, a very short time.
The first, more popular programming languages found their roots between 1960 and 1970. The first “high-level programming language” was Plankalkül from around 1945. Even earlier, the roots of programming go back to Ada Lovelace, yes, a woman, in the year 1883.
Nowadays, we even have low code or no-code ways to speak with machines. Mostly those use visual environments to drag together code components or nodes and are called visual programming languages.
So, in the past, we needed to write code in a language, following grammar. Now, we could also drag together many nodes and hope for an effect, while playing with parameters. At some point of customization those visual environments also allow to add code manually. But it's not essentially necessary to write the code yourself anymore.
DALL·E and other neural networks allow access to Artificial intelligence as a source of processing your thoughts via a text prompt. If you want, you could say that you tell the machine, what it should program with your natural language. Your chosen words define the quality of the output you are aiming for.
Natural language and natural interfaces
The natural language is the way we as humans speak with each other. To some extent even our voice-interfaces around us are now able to communicate with us in a way that feels almost natural. In this case our command prompt mostly starts with “Alexa”, “Siri” or “Hey Google” to initialize the dialogue. Natural language processing takes the task of converting speech into words in the background to take the right commands.
In 2013, there was a trend to use natural language forms in interfaces to make form usage feel more natural. Forms were integrated into sentences rather than presented as a separate option. Stats are more usable in natural language, but confusing numbers make it difficult to understand the impact of improvements.
The tendency to make interfaces more natural or more human, so we could use our natural senses to interact, is present. But the evolution of the interfaces also depends on technological developments.
There are many interesting developments, especially if you investigate the MIT Media lab and the tangible media group. They strive for seamlessly coupling the worlds of bits and atoms by giving dynamic physical form to digital information and computation. Research on venous materials and human-flora interaction provides insights into our future.
One of the easiest ways to interact now is our language, expressed by voice interfaces. When using Neuralink or similar chips inside our bodies, it's likely that we'll need to use interfaces with a well-structured language initially.
If you are looking for more scientific terms on how the language is being used for generating visual results, look for text-to-image synthesis.
GANs produce visual effects resembling the WebGL example on alteredqualia.com that utilizes DeepWarp, a method commonly employed for gaze movement.
CLIP, GANs or AI driven Art and approaches of deep learning.
Natural language processing is also an important tool when it comes to playing with so called CLIPs. CLIP is “Contrast Language-Image Pre-Training”, a neural network trained on a variety of text-image pairs. CLIP got introduced around January 2021.
Before getting into CLIP, a short excursion on learning. Deep learning is part of machine learning and helps machines to learn. The machine is supposed to increase its own skills without the help of humans. On contrast to machine learning, where humans interfere in the learning process, in deep learning humans take care of providing information for learning. To do so, we could also use our input prompt. The process of analysis is up to the machine itself.
An example for machine learning would be connecting words and images by labelling the images with the appropriate word. A typical combination for this cause would be ImageNet and WordNet.
CLIP is a neural network model that is already trained on 400,000,000 (image, text) pairs. So, this means that this is the data that is used in training the CLIP Model. You can input an image to the model and get the most probable caption or summary of it. Alternatively, you can use natural language as an input. The true magic comes when you see examples. Travis Hoppe shares some interesting applications in his Twitter account.
To get a full “beginners” understanding of the CLIP model, I recommend this article: “A beginner’s guide to the CLIP Model”. You will find many interesting articles on kdnuggets.com around artificial intelligence, computer vision, data science, machine learning, natural language processing or programming (Python).
To try CLIP yourself, you could use VQGAN developed by the University of Heidelberg. It’s a GAN (a generative adversarial network), those are special types of neural networks consisting of “Generator” and “discriminator” networks. There is a GAN that generates images. The generator is like an artist and the discriminator is like a critic during training. When the generator can produce an image that the discriminator can't distinguish from a human-made one, we can use it for creating art.
There are huge libraries of available GANs for very different purposes. Some of the lists I could recommend:
To play with the technology to learn, use artbreeder and generate new images; Artbreeder is also based on GAN based image generation.
Exploring Applications of CLIP
How to get started? I would always recommend playing with the technology. Most technology is too complex for an early deep dive, so start by playing. If you found your motivation to dig deeper, do it with all your might as you need a lot of patience to get a good understanding.
So, let’s look at VQGAN + CLIP you could use directly yourself online. This is a collab notebook, a small development environment running Python programming language. Your task is to input some values... To get started, I would recommend to only change the “natural language prompt” of what you want to see. In my case I tested some results of different apps with the sentence “Children playing in the forest” (because my kids are attending a kindergarten in the wood while writing these lines).
These are my first results using the CLIP linked above (to try yourself, open the link, basically go through the page and always execute the play button to the left and wait till there’s a check icon next to it, to signal it’s loaded), only for chapter “Parameters” enter a text. In the end you could also create a video. The results here are early iterations. The program would continue till the “free” volume is used up.
To compare my result, I checked for other applications out there and their result. So there’s for example an application that is also available as an app; In wombo.art you could test directly. The result from wombo art had similar patterns as my test with YQGAN and looked like the picture on the at the top left.
Then I used another popular web-service to compare: Nightcafe you can see on the right.
You see, when using a webservice, you end up getting one result. Using a google colab, you get many iterations and it's not as fast as the webservice or app. (Google Colab or Colaboratory is a way to execute Pyhton code in the browser).
There are also other CLIPS on google colab you could use. The following picture is the (an early iterative) result of the combination VQGAN+CLIP (z+quantize method):
You could continue playing with many more version out there. Here you will find an overview of newer applications: https://softologyblog.wordpress.com/
You notice that the basic style follows one pattern and it’s obvious nightcafe also uses some way of VQGAN combination. To create magic, or what I would call magic based on a text prompt, there’s still some way to go.
The magic I aim for: I tell the machine what I want to see, and they give me iterations of the thing I type in in. It mustn't be art or AI art.
To create this magic, we look at DALL·E. OpenAI released DALL·E 2 in April 2022. They show the real magic of the future. By the way, I wouldn't call the results art. Mostly we generate graphics.
A short introduction to DALL·E, DALL·E 2 and DALL·E mini
“DALL·E 2 is an artificial intelligence model that takes a text prompt and/or existing image as an input and generates a new image as an output. DALL·E 2 was developed by researchers at OpenAI to understand the capabilities and broader implications of multimodal generative models. To help us and others better understand how image generation models can be used and misused.”
(From their own description)
To make it shot, look at their website and get a feeling of the beauty that could be generated with DALL·E 2 or watch the short video on youtube:
Unfortunately, DALL·E 2 is not easily available to test yet. But you could sign up on their waiting list.
According to OpenAI DALL·E 2 isn't trained on faces, to avoid the risk of deep fakes. The danger is already there. Finally it the time to doubt any images being shown to us!
On twitter try a search for #dalle or #dalle2 and you find many inspiring pictures. Going through the magical results, it feels as if image searches could get a relic of the past soon or will only be needed for special causes. Image stock market could be AI generated in my point of view. Here are some randomly selected shots from some twitter members (all linked):
Yes, it’s a pity, we all can’t directly play with this fascinating technology!
Luckily I found DALL·E mini. And that's your friend to play! By the way, I also created my post image with Dall-e mini. I asked for "AI Art blue and yellow". One of the results could be found on the overview site of my writings.
The description of DALL·E mini:
“OpenAI had the first impressive model for generating images through DALL·E. DALL·E mini is an attempt at reproducing those results with an open-source model.”
They also have DALL·E mega, a large version of the mini, that is being trained. To see the fascinating world of training, look into the training space.
And finally, here’s your playground to play with the DALL-E mini: https://huggingface.co/spaces/dalle-mini/dalle-mini
That’s my result on "playing children in the forest"; quite an improvement in a very short time. If you play more with DALL·E mini you will notice, that faces still are really not trained much. But I already love it!
To feel the new power of words and to discover some magic along using words, go out and play for yourself!
By the way, there's another tool on the web, giving you similar results. It's neuralblender. Here's my kids results from them:
If you want to get inspired by other AI Artists, look at the work of Refik Anadol, his process is the application of a lot of machine learning to a real world of art. As I said, in my point of view, the term of AI Art is mostly wrong, looking at the output of DALL·E or other CLIP applications, as the outputs are not expressive. In my point of view, we speak about AI graphics.
Maybe you could guess, what my prompt in this case was (created with DALL·E Mini):
Have fun playing!
And guess what... My challenge is to build websites via a text prompt. Currently I am looking for available sources to have a good segmentation (Structural attributes, Style attributes). Let's see when I will share my results or findings.
Some more tipps on playing with DALL·E mini... use objects. To avoid deep fakes, faces won't work. Play around with the text prompt. You could try with the attribute, in the style of... or try the DrawBench prompts being used for Google Imagen first.
Google Imagen
While publishing this article on Mai 24th, Google came up with a competitor to OpenAI DALL·E 2. The market is on fire! And Google-Imagen impresses the same way, DALL·E 2 did.
Be excited to check out the first tweets on the topic #imagen.
And we will all be happy to play with the first, real demos!
Recommended further readings
- Infinite Images and the latent camera
- Hierarchical Text-Conditional Image Generation with CLIP Latents
- DALL·E 2
- Imagen by Google
- StyleGan 3
- Explore my newer articles around the topics:
How AI Assistants Can Help You with Text and Art
Generative Art – how to get started with AI Art
Midjourney and DALL-E 2 – first hands on impressions
Stable Diffusion celebrates new forms of creativity
This Article was mentioned on manfred-feiger.com