What a good artist understands is that nothing comes from nowhere. All creative work builds on what came before. Nothing is completely original.
from Steal Like an Artist: 10 Things Nobody Told You About Being Creative.

Stable Diffusion – what is it?
Stable Diffusion is a model that can create digital images from text or use text prompts to guide image-to-image translations. It will be released on August 22, 2022.Looking at the research paper you will notice that some names you might have come across in former publications on GANs will show up again, so all people being mentioned in the research paper of VQGAN are mentioned: Patrick Esser, Dominik Lorenz and Robin Rombach.
Understanding the heritage can help comprehend the rapid development of VQGAN in just one year. The progress has been extraordinary.
Currently, all main competitors (DALL·E 2, midjourney and Stable diffusion) rely on the principle of diffusion to create their images, so-called latent diffusion models. If you want to get a deeper understanding, I recommend a good written article from Louis Bouchard or watch his video:
Screenshot from "What is Stable Diffusion?" by Louis Bouchard from YouTube.
Looking at results with VQGAN and Clip, see my older article and results from one year later, give you an idea about the speed of development. We face many competitors and an impressive research speed. I assume the next challenge is render speed while generating the images and optimizing the diffusion models. In the meantime, the world is allowed to unleash its full potential by connecting more libraries together to Stable Diffusion.
I really recommend trying some of the text-prompts yourself. Generating images on your computer can be difficult. The process involves using language prompts and the speed of image generation tells you how complex it is. On my MacBook (with m1) it’s running ok, but I am not sure on which PC builds you could run it.
Playing with the software, you'll observe cases the current implementations struggle with - one of the harder ones is stigmatizing the world around us. No wonder if the models are trained by common images of the web. The most common pictures on the web are pictures more beautiful than reality.
Watch "Stable Diffusion - What, Why, How?" by Edan Meyer, a one-hour instructional video on creating diffusion videos. It's presented greatly and a great resource to get started with Stable Diffusion.
Screenshot from Edan Meyer "Stabe-Diffusion – What, Why, How?" from YouTube
If you want a shorter version of Stable Diffusion, look at the 2-minute paper.
Stable Diffusion – Quickstart via dreamstudio, collab, huggingface
If you had a look at midjourney or DALL·E 2 before, you experienced simple text-to-image prompts to create your artworks. Being Open Source opens a variety of inputs for Stable Diffusion. I try to give you an overview (with a raise in complexity and flexibility).
Dreamstudio.ai
Dreamstudio is the easy access to Stable Diffusion and the answer to popular midjourney or DALL·E 2. To get started, you need to login. Also, the prompt guide is only available when logged in.
The main difference to midjourney (using discord) and DALL·E 2 (basically relying on the text prompt) is the interface you get provided. Besides a text prompt you also get some rulers to modify the settings. Though the wording being used is still geeky, it is progress to the simple text inputs from the other main rivals. By the way, there also used to be an official discord bot, but they shut it down.
Dreamstudio.ai GUI screenshot – taken from my computer
The payment model for dreamstudio feels more complicated than for midjourney or DALL·E 2. You also get free credits to get started but the amount of cost varies with your settings; here’s their explanation:
At default settings, it’s one credit per image. Depending on the image resolution & step count you choose, this can go as low as 0.2 credits per image or as high as 28.2 credits per image. Here is a detailed chart for explanation:
Steps512x512512x768512x1024768x768768x10241024x1024100.20.50.80.91.31.9250.51.21.92.33.34.7501.02.43.84.66.69.4751.53.65.76.99.914.11002.04.87.69.213.218.81503.07.211.413.819.828.2Pricing table from dreamstudio.ai FAQs (as of September 2022)
If you need some extra guidance on the photo editor, watch the video from their FAQs: Getting Started With DreamStudio Website Beta, Part Five: Inpainting/Outpainting.
In the FAQs you will find a lot of other helpful sources to get started.
All in all, it’s cool to have a maintained/managed tool showing the potential of Stable Diffusion and easily experiment. If you like what you see, you could continue to pay or switch to Open Source.
As you have seen, the tool also has some limitations at the moment: the size of the images is limited to 1024x1024 (midjourney allows 2048x2048). The size is directly connected to the render speed.
But no worries about the size: one of the biggest sins in photo editing is also a relic of the past. Upscaling pictures is now possible with the help of other AI image upscaling tools. There are so many tools nowadays, maybe you find some inspiration in my other article “How AI Assistants Can Help You with Text and Art”.
You can also look for GIT repositories to size the images. One I could recommend is this txt2imghd. There's also a version on replicate.com.
Stable Diffusion on Huggingface.co
You must have come across hugging face if you got into text to image before. I shared some stuff from hugging face before, such as Dalle-mini, a free Dalle alternative from earlier this year. I really recommend anyone being interested in the topic of AI and machine learning, to visit hugging face every now and then. Under “spaces” you will find apps you could directly test as web-apps. And you notice (sorted by likes) that there’s also an app for Stable Diffusion.
Hugging Face Spaces – Apps to test online
In fact, there are more apps using Stable Diffusion to transform images. The one on the left is the simple text to image prompt and the right one could be used for modifying pictures as well. Feel free to try one of the two here:

Hugging Face Stable Diffusion Demo Shot
Hugging Face Stable Diffusion – Diffuse the rest – Shot

Here are some of the outputs being generated in the tool:

And the fun part: It’s for free! There’s no limitation in terms of credits and it’s a perfect playground to get used to how text-to-image prompts work in general. As mentioned before when explaining midjourney or Dalle-2, I recommend reading through some text prompts… or as the magic of open source already unfolds, use sites like lexica.art and look at the eye candy and copy prompts.
Screenshot from Lexica-art
As an alternative to lexica.art, you could also try arthub.ai.
Now I almost forgot to quickly explain what diffuse the rest could do for you… it’s kind of the beautifier of your paintings. Draw like in MS Paint and add a text prompt, about what your line drawing might be. In my case presented with an owl (owl got very popular in all this process of text-to-image generation) – the transformation of kid paintings is also quite fun 😉
Example for diffuse the rest: Simple line drawing on left gets transformed to a result written in the text prompt.
Stable Diffusion on Google Colab
Google Colab is another resource you need to know when it comes to working with Python in the context of machine learning. Google Colab makes data science, deep learning, neural networks, and machine learning accessible to individual researchers who cannot afford expensive computing infrastructure.
Screenshot from Stable Diffusion notebook on Google Colab
Edan Meyer explains how to use google colab with his own modified colab notebook in his great video “Stable Diffusion – What, Why, How?”. He is explaining the whole process to create images, similar images or even videos. A deeper understanding of how using the code works help to unleash the full creative potential.
In very simple words, you open a colab notebook and follow from top to bottom. On the left you find some little play icons to execute. After executing one after another, there might be some text prompts or variable to modify to lead you to your result.
Local Apps and other Web GUIs for Stable Diffusion
Since we have the first open-source release, it feels the market is exploding. I saw downloadable apps to come and go within 2 weeks, so it’s hard to give you a list of all the options there are. Also, there's no guarantee they are still working. Many people are fascinated by these tools. Some want to make money quickly, others want to share, and everyone wants to keep improving the technology.
So, I start with another Web GUI for Stable Diffusion called peacasso:
Screenshot Pecasse, Web GUI to execute Stable Diffusion prompts
And here’s a desktop app to run Stable Diffusion for the mac (m1,m2) called DiffusionBee:
Screenshot DiffusionBee – Desktop app for Mac M1
Feel free to share more cool apps, sources in the comments. As of Mid September 2022 there were so many tools, it is hard to keep an overview.
Replicate is another interesting cloud API I wanted to give a try. If you use the tools you will notice, rendering takes time. Using a cloud API for rendering can improve energy efficiency and reduce your carbon footprint, while also decreasing rendering times.
Explore the offerings of recplicate.com, there’s a clean Stable Diffusion install as well as other interesting tools to help you to optimize your images or get a desired effect.
Screenshot from replicate.com – Cloud API based machine learning tools

Stable Diffusion spreadsheet UI on rowy.io
Stable Diffusion – local installation on MAC M1
Finally, some thoughts and links on installing Stable Diffusion locally on your computer. I personally only installed it on an M1 Macbook. With the advantage of the RAM being able to be shared.
There's a tutorial on options to work with Stable Diffusion locally I discovered while finishing the article, but it seems to be the official one (I didn't test it though as I didn't want to break my running version); the tutorial is from AssemblyAI itself: "How to run stable diffusion locally to generate images".
When I tested my local install, I basically followed along with another introduction from replicate.com: "Run Stable Diffusion on your M1 Mac's GPU".
In addition to the instructions, some extra tips, as I also had some trouble while installing. First, check for the correct Python version. You need 3.10, which could be downloaded and installed from python.org.
If you have trouble to switch the python version, there could be many reasons (I couldn't probably name). I used a version control, as options there were ASDF and pyenv. I decided on pyenv and followed the instructions and had no further issues to switch the version.
Another hint I could share with you... watch your folders. Always execute the prompts from the correct folder and be prepared to wait a little for your prompts to be executed.
Here are some renderings on sculptures from my local tests:

Stable Diffusion – local installation on PC
It took me a little while to test the PC installation, but I guess this guide here is quite good and straightforward:
AUTOMATIC1111's SD-WebUI (Windows)
Following the guide you have similar steps as on mac, but it is simpler in the execution as there are more installers to run. Finally you have a webGUI to get going.
Please note, that you need a GPU with at least 6 GB of VRAM. 10 GB or more on a NVIDIA GPU with CUDA is highly recommended to allow larger image sizes. A business card, such as the RTX A series is preferred as the energy consumption compared to the value is much better. Maybe the upcoming NVIDIA generation will bring even more value.
On the tutorial site you also get some other guides (I didn't test):

Stable Diffusion Installation guides
Stable Diffusion – other installs

Stable Diffusion in TensorFlow / Keras

Creativity unleashed with Stable Diffusion
To get an idea about the full potential I collected some random examples from the past month. There are so many exciting opportunities. People create comics, movies or just play around.
Kali Yuga models
Kali Yuga provides some cool custom-trained models for Stable Diffusion. Here are some shots taken from Twitter:
KaliYuga on github

from Orobic Oak / twitter
from Digital Jawns / twitter

You might have guessed it, there are also notebooks on google colab around to work with them. An alternative would be jupyter notebooks, but to keep it simpler, we stay with google colab.

Notebook for #textilediffusion.
Notebook for #Pulp-Sci-Fi-Diffusion
Notebook for #lithography-diffusion
Notebook for #watercolor-diffusion
Notebook for #pixel-art-diffusion

Those notebooks are quite easy to handle. I used the #textilediffusion, to generate this sample piece here:

Stable Diffusion and #textilediffusion example from the author
Click the images to jump to the examples. Due to data policy, I don't embed external sources.
Stable Diffusion and Music:

from TomLikes Robots / Twitter
Stable Diffusion and 3D

from CoffeeVectors / twitter
from Infinite Vibes / twitter

from merlino_games / twitter
Here's the GitHub Source for the Stable Diffusion blender texture generator.
Another link, that is connected to Unreal and midjourney is the generation of textures.
Stable Diffusion and Image Tools
Stable Diffusion Photoshop Plugin

Screenshot from Website from Christian Cantrell
There are also options for text-to-image in Canva and GIMP already and Nightcafé and artbreeder seem to have changed their models towards stable diffusion as well (you might remember that earlier this year they both used some GAN+CLIP solution, as mentioned in my article).

Stable Diffusion to generate color palettes from text descriptions

from Matt DesLauriers / Twitter
Other sources to mix with Stable Diffusion or to watch in general:

Get3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
OpenAI Whisper
Google DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (GitHub implementation for Stable Diffusion and Google DreamBooth and another Stable Diffusion Implementation with DreamBooth)

Stable Diffusion and DreamBooth – great video to watch

Final thoughts on Stable Diffusion AI – image jockeys on random turntables.
I love it. Going for the open-source path is so great to unleash new instruments of inspiration and with lucky punches, even to create art. Art is expressive and the artworks being created might get expressive. For me it's an enhancement of our daily practices being creative. I also don't think anyone must be afraid of losing jobs. For precise tasks it is still tricky to work with the tools.
The models are only as good as the inputs; handling the inputs is not the easiest practice and as generating the images takes some time… in terms of client work, the way using any generator might not be the fastest option to choose. As always, I assume clients want to negotiate cost as they think everything is going easier now. But currently the image generation at a professional level is still not precise enough. But the tools are already there to help. Google Dreambooth for example could help a lot to bring consistency into image series.
Looking at the development speed I assume the topic about precision will be a matter of the next years, as well as the topic about speed and custom training sets. I am also worried a bit about the footprint being generated from all the image generations (including all the learning upfront). In hugging face there's a note on the Stable Diffusion model card itself: 11250 kg CO2 eq for the learning (calculated with the machine learning emissions calculator).
The development in text-to-image is far from its boarders yet. But it's so impressive to see the speed of the community to adopt these technologies.
Currently the tools help to discover creativity to finally create a mix of outputs we have seen before. In the future I am sure we can use the techniques to do whole mood boards for clients and in some cases even final graphics or artworks for them.

The only art I’ll ever study is stuff that I can steal from.
David Bowie

Like the evolution of web bandwidth, I think we need some more speed in terms of image generation to open new options and create beautiful human-centered-artworks as one day maybe images could be generated in real time based on likings of visitors. I also assume, at the moment we reached a certain grade of automated artworks, more and more people will enjoy discovering real artworks with their hands again and value of the real original artworks will raise.
In general we shouldn't trust media without a critical perspective, especially when we see images. History showed us that people will abuse technology. So I hope for people to find value in real in-person events after Covid pandemic again. Digital could and will be a lot of fake.
The use of some kind of style transfers on Smartphones is very likely by next year. Dynamic Video generation is also on its way (see make-a-video from meta ai), so we have text-to-video generation and music, VR worlds and games are also around the corner, as Midjourney founder David Holz said. Due to open-source development, I won't say it takes 10 years. And also Google Colab and other services to run the software will change their business models to profit from this boom.
Advertising or Branding will also be affected. It will be an exciting journey in the next two or three years. Osmosis pretends to have some of the features that could be interesting for an early stage. Just imagine adding a target group to the prompt, an industry or a segment. For Learning there are many galleries out there, such as dribble, FWA and many more 🙂 it will be fun to see where this journey goes! (Saying this not being in an international network agency anymore).
As mentioned many times, the development speed is incredible. The current hot tool is google dreambooth (connected to Stable diffusion). Every month there might be another unexpected cool tool to pop up. Maybe even photoshop integrated the tools for their next release.
For me, design turned more into managing design over the years but still design is functional, and art is expressive. Currently it feels easier to create automated expressiveness in fantasy worlds, but nonetheless I also use AI tools during the creative process. The results of machine learning give you a condensed view of common thoughts. So, I often used the prompts to check for color ranges of certain clients, compared it, what google says and easily created something one could call the "wisdom of the net" result.
Our traces on the net and our databases of artists created the look we could mix now. We are the image jockeys of generations that are already there. Unfortunately there's still a lot of randomness in the result to utilize it more professionally.

My "wisdom of the net" results from huggingface
My "wisdom of the net" results from huggingface

So long, enjoy and go out and play to discover the fun playing with those great tools!

Recommended readings

Understand latent diffusion models
Get a full introduction to Stable Diffusion

Midjourney and DALL-E 2 – first hands on impressionsMaybe you remember my article speaking computer language – welcome AI art, speaking about the new role of t
Generative Art – how to get started with AI ArtThe world of art is a diverse and beautiful place. The more we explore new forms of art, the more we are
How AI Assistants Can Help You with Text and ArtHave you ever wished you had a personal assistant to help you with your writing or with your artworks?
AI
Stable DiffusionStable Diffusion is a machine learning, text-to-image model to generate digital images from natural language descriptions. The underlying approach was

Parti Website Screenshot
In this article I want to share my first hands on experience with Midjourney and DALL-E 2 with you. With midjourney moving to open beta and starting their v3 image generation algorithm on the 25th of July, another player pushed to the market… and I guess one or two days later, finally my invite for DALL-E 2 arrived.
At that day I was happy at first, but when I noticed another day later, that another E-Mail I used for signing for DALL-E 2 only a few months ago, despite the first one, being used for application far more than half a year ago, I was a bit disappointed. I thought it’s all connected to their new pricing model and the beta start of midjourney and I got disappointed a bit. The exclusiveness was gone and “mass-market” arrived.
In general the market on AI Art bots seems to push a lot. So for example DALL-E mini, the free tool I mentioned in my other article, is now called craiyon. There are also other tools Worth being mentioned in the context of AI art tools, such as Latitude Voyage (of course in beta) or Apples GAUDI, a prompt tool for indoor 3D scenes.
Luckily I heard the U.S. Copyright office said A.I. Art can't be copyrighted, so I hope no war will heat up this exiting market in the near future. So far I only know that there are patents on algorithms being used. Hopefully the market will show even more diversity in the future.

Midjourney Screenshot Website
Midjourney beta – first impressions
Let us first look at our new player. Midjourney. The first impression of midjourney is being geekier and less serious than DALL-E 2. It’s a more playful entrance to a world everyone could test right away. After joining the beta, you will be redirected to Discord, no waiting list, no exclusive clubbing.
Before we investigate discord, a quick explanation of who Midjourney is. Midjourney is an independent research lab exploring new mediums of thought and expanding the imaginative powers of the human species.

IN 10 YEARS, YOU’LL BE ABLE TO BUY AN XBOX WITH A GIANT AI PROCESSOR, AND ALL THE GAMES ARE DREAMS.
David Holz in an Interview with the verge

David Holz, the founder of midjourney, did some other geeky, funny things in the past. He used to be co-founder of leap motion, a great exploration of the power of gestures. Midjourney is a small company right now with around 10 people working there.
Exploring the background of midjourney and all their way, how they speak to their audience, in other words their tonality feels great to me. It has the explorative spirit and playfulness I always try to teach students to get started in digital.
Signing up for a free account, you get 25 credits, these can be used to generate images within the discord channel. For anyone being interested, it is even possible to run midjourney on ones own discord server.
After that, you’ll have to pay — either $10 or $30 a month. The pricing is connected to the number of images you want and the privacy you expect. And if you are a huge company with more than 1 billion revenue a year, you get the corporate membership for 600 US dollar a year.
So how does it work?

Screenshot Discord Channel midjourney
If you never used discord before, see it as a chat tool, kind of a slack for the masses, with a non-business focus.
So, in the channel you will find all the information there are in general, on announcements, updates and so on. If feels like you as beta tester are part of the team, and even the boss is there, and you could influence the development process. After all, that’s what a beta program is about.
In terms of my professional perspective being more experienced with product and user testing, this is a great way for that kind of product, as the product itself is being used in the same environment.
So here are the steps to create images:

Look for a newbies Channel and enter it (in the picture you see newbies-29, newbies-59 and newbies-89)
If you have a picture in mind, about what you want to see, start typing “/imagine”, followed by whatever you want to see.
Within 60 seconds you should see 4 images.
With the 4 images being generated, you could upscale the image or create variations. Therefore, there are U1 to U4 (für upscaling) and V1 to V4 (for version) below the images being generated. Upscaling means that the images scale up to around 1664x1664 Pixels
Save the result, click on the generated image (to enlarge) and with the right mouse button being pressed, select save image to your local computer.
Finally, you could also ask the bot to send the results to your Direct messages, by reacting with the envelope emoji.

Prompt dialogue in discord (taken from quick start guide)
Example from discord stream with 4 generated images

Prompt dialogue to use envelope (taken from quick start guide)

This is enough to create visual beauty.
If you need inspiration, look in midjourney ressource list, there are great links:

The artist visual style encyclopedia (as of Mai 2023: link not working anymore)
The artist visual style encyclopedia – if the style is part of your text prompt it will take effect – such as cyberpunk city, by Franz Marc:

My result on cyberpunk city, by Franz Marc on midjourney
My result on cyberpunk city, by Franz Marc on midjourney

There’s also a prompt builder to guide your inputs and see which variables are available:

Prompt guide
And a collection on “prompt styles and weights”.
All in all, the learning curve, if you are used to discord and used a text prompt before is very flat. Only if you are new to discord and text prompts, you need more orientation.
After all I could say, I loved it. I love the playfulness of the whole app, the results and all the “social” aspects, seeing what all others are building, getting notifications about own results and the whole way it is being setup. It is great fun to try and the documentation is very helpful.

DALL-E 2 Intro screen
DALL-E 2 beta – first impressions
With the magic of the new and little frustration about the almost open invites, I waited to test my DALL-E 2 invites after I tested midjourney.
As mentioned, entering DALL-E app is a completely different experience. It is you and the text prompt, not the hectic news stream and the dynamics you experience in midjourney. Based on the interface/overall app decision, one could say, that DALL-E 2 wants you to reflect about what you type, while midjourney is a competing social feed, about who creates the coolest results.
The cost system at DALL-E 2 is based on credits. You get 50 credits for the first month for free and 15 credits every month after. With only 15$ you could buy additional credit packs worth 115 credits, which equals around 460 images. OpenAI allows commercial use of DALL-E images. Compared to the image stock market, the costs of the images will be fairly low, but one must be award that you have to get used to the text prompts and the corresponding structure of the input.
That being said let’s try to get a result for our cyberpunk city, by Franz Marc.
In the case of DALL-E, I couldn’t find a quick hands on, official documentation; so, I tried the sentence as following: A cyberpunk city in the style of Franz Marc, as the example being shown by DALL-E on their homepage also used the "in the style of" syntax.
My result on that one was quite… disappointing. I don’t know why DALL-E missed the style of Franz Marc that much:

DALL-E 2 result on a cyberpunk city in the style of Franz Marc
To investigate further, I needed a prompt guide for DALL-E 2. Googling for some help, you find

The DALL-E 2 prompt book – my recommended choice
OpenAI DALL-E 2 prompt guide

Going through the prompt options, you notice, that there are more parameters than on midjourney. Lightning, Camera position/angles, genres, usage contexts and many more.
So, I gave it another try, changing the prompt a bit leading to a much better result in terms of the style:

DALL-E 2 result on a painting from Franz Marc showing a cyberpunk city
Unfortunately, my tries with real artists didn’t work out so good with DALL-E 2 at the start, so I had another try…
My result from midjourney:

Midjourney result
And the same prompt from DALL-E 2:

First result on DALL-E 2
Getting used to the prompt dialogue on DALL-E increases the results a lot:

Adjusted text prompt
Using the right prompts is the essential part to get the best results. If your prompt is done right, it seems DALL-E is more precise, though midjourney is so much fun to use.
For DALL-E there's even a paid service being launched called promptbase to help you with using the right prompts:
Screenshot from promptbase.
If you want to discover more, I recommend searching the web yourself. On twitter I found some comparisons, having a similar impression I do:

From twitter
Resume
Is it possible to do a final resume on the tools? I guess no. On the one hand I am not experienced enough with both tools, on the other hand I think both tools are great and give us a glimpse into the future. The documentation on midjourney and the entry barrier is lower, so to get started, I would recommend midjourney in any case. The more you are aiming for precise results (as an alternative to stock images), the more it feels DALL-E 2 is the way to go.
Will be exiting what the next two month offer in terms of AI generation tools. 2022 is very exiting so far in terms of AI art generators and it will be exiting to see what's next.
What's next? Update mid August – dreamstudio arrived
As mentioned, the speed new, similar offerings are popping up on the market is impressive. It seems, AI in combination with images is going mainstream. Another tool to check out is dreamstudio. Access is quite easy as well and the results are also impressive. Also the editor looks promising, see my screenshot below. After my family holiday I might try a daily use check of all those great tools.
Screenshot dreamstudio and results on "A dream of a distant galaxy, by Hiroshige Utagawa, matte painting trending on artstation HQ."
Recommended readings

The AI that creates any picture you want, explained – The process behind DALL-E 2 (and others) explained
What AI means for human artists – great collection about views from the artists/designer perspective
Making AI Art with Midjourney – great article with examples and explanations
The Creativity of Text-based Generative Art – paper on text-based generative art
The trouble with DALL-E – Five thoughts on the use of artificial intelligence to accelerate the digital art market
My follow up article on Stable Diffusion: Stable Diffusion celebrates new forms of creativity

Generative Art – how to get started with AI ArtThe world of art is a diverse and beautiful place. The more we explore new forms of art, the more we are
How AI Assistants Can Help You with Text and ArtHave you ever wished you had a personal assistant to help you with your writing or with your artworks?
AI
Speaking computer language – welcome AI ArtThe command prompt: A black, empty screen and a blinking line, waiting for my input. Could you imagine

The command prompt: A black, empty screen and a blinking line, waiting for my input. Could you imagine that this prompt is the access to art or AI art? The command line shows a visual result of your terms, not ASCII art or related items. Nowadays, those prompts have different shapes, such as a form field or the use of voice interfaces.
Your imagination of thought, translated to words in your natural language (mostly in English), gives you access to new creativity. In the words of advertising, you need to express a good core idea, you want to get visualized. Similar to creatives being used to Image search engines, the need to find the correct words gets more and more important.
To explore my thoughts I look at the evolution of the way we tell machines, what to do and how CLIP and other neural models transform the way we could create what we are looking for. Whether it be AI art or maybe any picture we want to see, in general or what I would call AI graphics.
Computer language
A computer language is a language we use to talk to machines. It has been in use for about 70 years.In terms of human history, a very short time.
The first, more popular programming languages found their roots between 1960 and 1970. The first “high-level programming language” was Plankalkül from around 1945. Even earlier, the roots of programming go back to Ada Lovelace, yes, a woman, in the year 1883.
Nowadays, we even have low code or no-code ways to speak with machines. Mostly those use visual environments to drag together code components or nodes and are called visual programming languages.
So, in the past, we needed to write code in a language, following grammar. Now, we could also drag together many nodes and hope for an effect, while playing with parameters. At some point of customization those visual environments also allow to add code manually. But it's not essentially necessary to write the code yourself anymore.
DALL·E and other neural networks allow access to Artificial intelligence as a source of processing your thoughts via a text prompt. If you want, you could say that you tell the machine, what it should program with your natural language. Your chosen words define the quality of the output you are aiming for.
Natural language and natural interfaces
The natural language is the way we as humans speak with each other. To some extent even our voice-interfaces around us are now able to communicate with us in a way that feels almost natural. In this case our command prompt mostly starts with “Alexa”, “Siri” or “Hey Google” to initialize the dialogue. Natural language processing takes the task of converting speech into words in the background to take the right commands.
In 2013, there was a trend to use natural language forms in interfaces to make form usage feel more natural. Forms were integrated into sentences rather than presented as a separate option. Stats are more usable in natural language, but confusing numbers make it difficult to understand the impact of improvements.
The tendency to make interfaces more natural or more human, so we could use our natural senses to interact, is present. But the evolution of the interfaces also depends on technological developments.
Venous materials from MIT lab – Watch example on Vimeo
There are many interesting developments, especially if you investigate the MIT Media lab and the tangible media group. They strive for seamlessly coupling the worlds of bits and atoms by giving dynamic physical form to digital information and computation. Research on venous materials and human-flora interaction provides insights into our future.
One of the easiest ways to interact now is our language, expressed by voice interfaces. When using Neuralink or similar chips inside our bodies, it's likely that we'll need to use interfaces with a well-structured language initially.
If you are looking for more scientific terms on how the language is being used for generating visual results, look for text-to-image synthesis.
GANs produce visual effects resembling the WebGL example on alteredqualia.com that utilizes DeepWarp, a method commonly employed for gaze movement.

CLIP, GANs or AI driven Art and approaches of deep learning.
Natural language processing is also an important tool when it comes to playing with so called CLIPs. CLIP is “Contrast Language-Image Pre-Training”, a neural network trained on a variety of text-image pairs. CLIP got introduced around January 2021.
Before getting into CLIP, a short excursion on learning. Deep learning is part of machine learning and helps machines to learn. The machine is supposed to increase its own skills without the help of humans. On contrast to machine learning, where humans interfere in the learning process, in deep learning humans take care of providing information for learning. To do so, we could also use our input prompt. The process of analysis is up to the machine itself.
An example for machine learning would be connecting words and images by labelling the images with the appropriate word. A typical combination for this cause would be ImageNet and WordNet.
CLIP is a neural network model that is already trained on 400,000,000 (image, text) pairs. So, this means that this is the data that is used in training the CLIP Model. You can input an image to the model and get the most probable caption or summary of it. Alternatively, you can use natural language as an input. The true magic comes when you see examples. Travis Hoppe shares some interesting applications in his Twitter account.
To get a full “beginners” understanding of the CLIP model, I recommend this article: “A beginner’s guide to the CLIP Model”. You will find many interesting articles on kdnuggets.com around artificial intelligence, computer vision, data science, machine learning, natural language processing or programming (Python).
To try CLIP yourself, you could use VQGAN developed by the University of Heidelberg. It’s a GAN (a generative adversarial network), those are special types of neural networks consisting of “Generator” and “discriminator” networks. There is a GAN that generates images. The generator is like an artist and the discriminator is like a critic during training. When the generator can produce an image that the discriminator can't distinguish from a human-made one, we can use it for creating art.
There are huge libraries of available GANs for very different purposes. Some of the lists I could recommend:
The GAN Zoo
GANS awesome applications
To play with the technology to learn, use artbreeder and generate new images; Artbreeder is also based on GAN based image generation.

Artbreeder, Image generation based on GAN
Exploring Applications of CLIP
How to get started? I would always recommend playing with the technology. Most technology is too complex for an early deep dive, so start by playing. If you found your motivation to dig deeper, do it with all your might as you need a lot of patience to get a good understanding.
So, let’s look at VQGAN + CLIP you could use directly yourself online. This is a collab notebook, a small development environment running Python programming language. Your task is to input some values... To get started, I would recommend to only change the “natural language prompt” of what you want to see. In my case I tested some results of different apps with the sentence “Children playing in the forest” (because my kids are attending a kindergarten in the wood while writing these lines).
These are my first results using the CLIP linked above (to try yourself, open the link, basically go through the page and always execute the play button to the left and wait till there’s a check icon next to it, to signal it’s loaded), only for chapter “Parameters” enter a text. In the end you could also create a video. The results here are early iterations. The program would continue till the “free” volume is used up.

First iterations with VQGAN + CLIP
Number 100
Number 150

from wombo.art
from nightcafe

To compare my result, I checked for other applications out there and their result. So there’s for example an application that is also available as an app; In wombo.art you could test directly. The result from wombo art had similar patterns as my test with YQGAN and looked like the picture on the at the top left.
Then I used another popular web-service to compare: Nightcafe you can see on the right.
You see, when using a webservice, you end up getting one result. Using a google colab, you get many iterations and it's not as fast as the webservice or app. (Google Colab or Colaboratory is a way to execute Pyhton code in the browser).
There are also other CLIPS on google colab you could use. The following picture is the (an early iterative) result of the combination VQGAN+CLIP (z+quantize method):

You could continue playing with many more version out there. Here you will find an overview of newer applications: https://softologyblog.wordpress.com/
You notice that the basic style follows one pattern and it’s obvious nightcafe also uses some way of VQGAN combination. To create magic, or what I would call magic based on a text prompt, there’s still some way to go.
The magic I aim for: I tell the machine what I want to see, and they give me iterations of the thing I type in in. It mustn't be art or AI art.
To create this magic, we look at DALL·E. OpenAI released DALL·E 2 in April 2022. They show the real magic of the future. By the way, I wouldn't call the results art. Mostly we generate graphics.
DALL·E 2 screenshot, see more on their website
A short introduction to DALL·E, DALL·E 2 and DALL·E mini
“DALL·E 2 is an artificial intelligence model that takes a text prompt and/or existing image as an input and generates a new image as an output. DALL·E 2 was developed by researchers at OpenAI to understand the capabilities and broader implications of multimodal generative models. To help us and others better understand how image generation models can be used and misused.” (From their own description)
To make it shot, look at their website and get a feeling of the beauty that could be generated with DALL·E 2 or watch the short video on youtube:

Unfortunately, DALL·E 2 is not easily available to test yet. But you could sign up on their waiting list.
According to OpenAI DALL·E 2 isn't trained on faces, to avoid the risk of deep fakes. The danger is already there. Finally it the time to doubt any images being shown to us!
On twitter try a search for #dalle or #dalle2 and you find many inspiring pictures. Going through the magical results, it feels as if image searches could get a relic of the past soon or will only be needed for special causes. Image stock market could be AI generated in my point of view. Here are some randomly selected shots from some twitter members (all linked):



Yes, it’s a pity, we all can’t directly play with this fascinating technology!
Luckily I found DALL·E mini. And that's your friend to play! By the way, I also created my post image with Dall-e mini. I asked for "AI Art blue and yellow". One of the results could be found on the overview site of my writings.
The description of DALL·E mini: “OpenAI had the first impressive model for generating images through DALL·E. DALL·E mini is an attempt at reproducing those results with an open-source model.”
They also have DALL·E mega, a large version of the mini, that is being trained. To see the fascinating world of training, look into the training space.
And finally, here’s your playground to play with the DALL-E mini: https://huggingface.co/spaces/dalle-mini/dalle-mini
That’s my result on "playing children in the forest"; quite an improvement in a very short time. If you play more with DALL·E mini you will notice, that faces still are really not trained much. But I already love it!

To feel the new power of words and to discover some magic along using words, go out and play for yourself!
By the way, there's another tool on the web, giving you similar results. It's neuralblender. Here's my kids results from them:

If you want to get inspired by other AI Artists, look at the work of Refik Anadol, his process is the application of a lot of machine learning to a real world of art. As I said, in my point of view, the term of AI Art is mostly wrong, looking at the output of DALL·E or other CLIP applications, as the outputs are not expressive. In my point of view, we speak about AI graphics.
Maybe you could guess, what my prompt in this case was (created with DALL·E Mini):

Have fun playing!
And guess what... My challenge is to build websites via a text prompt. Currently I am looking for available sources to have a good segmentation (Structural attributes, Style attributes). Let's see when I will share my results or findings.

Some more tipps on playing with DALL·E mini... use objects. To avoid deep fakes, faces won't work. Play around with the text prompt. You could try with the attribute, in the style of... or try the DrawBench prompts being used for Google Imagen first.
Dalle Mini: a rocket in the style of Matsuo Basho

Google Imagen
While publishing this article on Mai 24th, Google came up with a competitor to OpenAI DALL·E 2. The market is on fire! And Google-Imagen impresses the same way, DALL·E 2 did.
Be excited to check out the first tweets on the topic #imagen.
And we will all be happy to play with the first, real demos!
Screenshot from Imagen website

Recommended further readings

Infinite Images and the latent camera
Hierarchical Text-Conditional Image Generation with CLIP Latents
DALL·E 2
Imagen by Google
StyleGan 3
Explore my newer articles around the topics:How AI Assistants Can Help You with Text and ArtGenerative Art – how to get started with AI ArtMidjourney and DALL-E 2 – first hands on impressionsStable Diffusion celebrates new forms of creativity

FAQs
What are AI GraphicsVisual images or designs on some surface, such as a wall, canvas, screen, paper, or stone, to inform, illustrate, or entertain being generated with the help of Artificial Intelligence.
Deconstructing AI Art masterpieces: Create Art like Refik AnadolWhat If you were asked to create art like Refik Anadol? Or to produce some "AI art"?
In case you
Midjourney and DALL-E 2 – first hands on impressionsMaybe you remember my article speaking computer language – welcome AI art, speaking about the new role of t
Generative Art – how to get started with AI ArtThe world of art is a diverse and beautiful place. The more we explore new forms of art, the more we are
How AI Assistants Can Help You with Text and ArtHave you ever wished you had a personal assistant to help you with your writing or with your artworks?
AI