What a good artist understands is that nothing comes from nowhere. All creative work builds on what came before. Nothing is completely original.
from Steal Like an Artist: 10 Things Nobody Told You About Being Creative.

Stable Diffusion – what is it?
Stable Diffusion is a model that can create digital images from text or use text prompts to guide image-to-image translations. It will be released on August 22, 2022.Looking at the research paper you will notice that some names you might have come across in former publications on GANs will show up again, so all people being mentioned in the research paper of VQGAN are mentioned: Patrick Esser, Dominik Lorenz and Robin Rombach.
Understanding the heritage can help comprehend the rapid development of VQGAN in just one year. The progress has been extraordinary.
Currently, all main competitors (DALL·E 2, midjourney and Stable diffusion) rely on the principle of diffusion to create their images, so-called latent diffusion models. If you want to get a deeper understanding, I recommend a good written article from Louis Bouchard or watch his video:
Screenshot from "What is Stable Diffusion?" by Louis Bouchard from YouTube.
Looking at results with VQGAN and Clip, see my older article and results from one year later, give you an idea about the speed of development. We face many competitors and an impressive research speed. I assume the next challenge is render speed while generating the images and optimizing the diffusion models. In the meantime, the world is allowed to unleash its full potential by connecting more libraries together to Stable Diffusion.
I really recommend trying some of the text-prompts yourself. Generating images on your computer can be difficult. The process involves using language prompts and the speed of image generation tells you how complex it is. On my MacBook (with m1) it’s running ok, but I am not sure on which PC builds you could run it.
Playing with the software, you'll observe cases the current implementations struggle with - one of the harder ones is stigmatizing the world around us. No wonder if the models are trained by common images of the web. The most common pictures on the web are pictures more beautiful than reality.
Watch "Stable Diffusion - What, Why, How?" by Edan Meyer, a one-hour instructional video on creating diffusion videos. It's presented greatly and a great resource to get started with Stable Diffusion.
Screenshot from Edan Meyer "Stabe-Diffusion – What, Why, How?" from YouTube
If you want a shorter version of Stable Diffusion, look at the 2-minute paper.
Stable Diffusion – Quickstart via dreamstudio, collab, huggingface
If you had a look at midjourney or DALL·E 2 before, you experienced simple text-to-image prompts to create your artworks. Being Open Source opens a variety of inputs for Stable Diffusion. I try to give you an overview (with a raise in complexity and flexibility).
Dreamstudio.ai
Dreamstudio is the easy access to Stable Diffusion and the answer to popular midjourney or DALL·E 2. To get started, you need to login. Also, the prompt guide is only available when logged in.
The main difference to midjourney (using discord) and DALL·E 2 (basically relying on the text prompt) is the interface you get provided. Besides a text prompt you also get some rulers to modify the settings. Though the wording being used is still geeky, it is progress to the simple text inputs from the other main rivals. By the way, there also used to be an official discord bot, but they shut it down.
Dreamstudio.ai GUI screenshot – taken from my computer
The payment model for dreamstudio feels more complicated than for midjourney or DALL·E 2. You also get free credits to get started but the amount of cost varies with your settings; here’s their explanation:
At default settings, it’s one credit per image. Depending on the image resolution & step count you choose, this can go as low as 0.2 credits per image or as high as 28.2 credits per image. Here is a detailed chart for explanation:
Steps512x512512x768512x1024768x768768x10241024x1024100.20.50.80.91.31.9250.51.21.92.33.34.7501.02.43.84.66.69.4751.53.65.76.99.914.11002.04.87.69.213.218.81503.07.211.413.819.828.2Pricing table from dreamstudio.ai FAQs (as of September 2022)
If you need some extra guidance on the photo editor, watch the video from their FAQs: Getting Started With DreamStudio Website Beta, Part Five: Inpainting/Outpainting.
In the FAQs you will find a lot of other helpful sources to get started.
All in all, it’s cool to have a maintained/managed tool showing the potential of Stable Diffusion and easily experiment. If you like what you see, you could continue to pay or switch to Open Source.
As you have seen, the tool also has some limitations at the moment: the size of the images is limited to 1024x1024 (midjourney allows 2048x2048). The size is directly connected to the render speed.
But no worries about the size: one of the biggest sins in photo editing is also a relic of the past. Upscaling pictures is now possible with the help of other AI image upscaling tools. There are so many tools nowadays, maybe you find some inspiration in my other article “How AI Assistants Can Help You with Text and Art”.
You can also look for GIT repositories to size the images. One I could recommend is this txt2imghd. There's also a version on replicate.com.
Stable Diffusion on Huggingface.co
You must have come across hugging face if you got into text to image before. I shared some stuff from hugging face before, such as Dalle-mini, a free Dalle alternative from earlier this year. I really recommend anyone being interested in the topic of AI and machine learning, to visit hugging face every now and then. Under “spaces” you will find apps you could directly test as web-apps. And you notice (sorted by likes) that there’s also an app for Stable Diffusion.
Hugging Face Spaces – Apps to test online
In fact, there are more apps using Stable Diffusion to transform images. The one on the left is the simple text to image prompt and the right one could be used for modifying pictures as well. Feel free to try one of the two here:

Hugging Face Stable Diffusion Demo Shot
Hugging Face Stable Diffusion – Diffuse the rest – Shot

Here are some of the outputs being generated in the tool:

And the fun part: It’s for free! There’s no limitation in terms of credits and it’s a perfect playground to get used to how text-to-image prompts work in general. As mentioned before when explaining midjourney or Dalle-2, I recommend reading through some text prompts… or as the magic of open source already unfolds, use sites like lexica.art and look at the eye candy and copy prompts.
Screenshot from Lexica-art
As an alternative to lexica.art, you could also try arthub.ai.
Now I almost forgot to quickly explain what diffuse the rest could do for you… it’s kind of the beautifier of your paintings. Draw like in MS Paint and add a text prompt, about what your line drawing might be. In my case presented with an owl (owl got very popular in all this process of text-to-image generation) – the transformation of kid paintings is also quite fun 😉
Example for diffuse the rest: Simple line drawing on left gets transformed to a result written in the text prompt.
Stable Diffusion on Google Colab
Google Colab is another resource you need to know when it comes to working with Python in the context of machine learning. Google Colab makes data science, deep learning, neural networks, and machine learning accessible to individual researchers who cannot afford expensive computing infrastructure.
Screenshot from Stable Diffusion notebook on Google Colab
Edan Meyer explains how to use google colab with his own modified colab notebook in his great video “Stable Diffusion – What, Why, How?”. He is explaining the whole process to create images, similar images or even videos. A deeper understanding of how using the code works help to unleash the full creative potential.
In very simple words, you open a colab notebook and follow from top to bottom. On the left you find some little play icons to execute. After executing one after another, there might be some text prompts or variable to modify to lead you to your result.
Local Apps and other Web GUIs for Stable Diffusion
Since we have the first open-source release, it feels the market is exploding. I saw downloadable apps to come and go within 2 weeks, so it’s hard to give you a list of all the options there are. Also, there's no guarantee they are still working. Many people are fascinated by these tools. Some want to make money quickly, others want to share, and everyone wants to keep improving the technology.
So, I start with another Web GUI for Stable Diffusion called peacasso:
Screenshot Pecasse, Web GUI to execute Stable Diffusion prompts
And here’s a desktop app to run Stable Diffusion for the mac (m1,m2) called DiffusionBee:
Screenshot DiffusionBee – Desktop app for Mac M1
Feel free to share more cool apps, sources in the comments. As of Mid September 2022 there were so many tools, it is hard to keep an overview.
Replicate is another interesting cloud API I wanted to give a try. If you use the tools you will notice, rendering takes time. Using a cloud API for rendering can improve energy efficiency and reduce your carbon footprint, while also decreasing rendering times.
Explore the offerings of recplicate.com, there’s a clean Stable Diffusion install as well as other interesting tools to help you to optimize your images or get a desired effect.
Screenshot from replicate.com – Cloud API based machine learning tools

Stable Diffusion spreadsheet UI on rowy.io
Stable Diffusion – local installation on MAC M1
Finally, some thoughts and links on installing Stable Diffusion locally on your computer. I personally only installed it on an M1 Macbook. With the advantage of the RAM being able to be shared.
There's a tutorial on options to work with Stable Diffusion locally I discovered while finishing the article, but it seems to be the official one (I didn't test it though as I didn't want to break my running version); the tutorial is from AssemblyAI itself: "How to run stable diffusion locally to generate images".
When I tested my local install, I basically followed along with another introduction from replicate.com: "Run Stable Diffusion on your M1 Mac's GPU".
In addition to the instructions, some extra tips, as I also had some trouble while installing. First, check for the correct Python version. You need 3.10, which could be downloaded and installed from python.org.
If you have trouble to switch the python version, there could be many reasons (I couldn't probably name). I used a version control, as options there were ASDF and pyenv. I decided on pyenv and followed the instructions and had no further issues to switch the version.
Another hint I could share with you... watch your folders. Always execute the prompts from the correct folder and be prepared to wait a little for your prompts to be executed.
Here are some renderings on sculptures from my local tests:

Stable Diffusion – local installation on PC
It took me a little while to test the PC installation, but I guess this guide here is quite good and straightforward:
AUTOMATIC1111's SD-WebUI (Windows)
Following the guide you have similar steps as on mac, but it is simpler in the execution as there are more installers to run. Finally you have a webGUI to get going.
Please note, that you need a GPU with at least 6 GB of VRAM. 10 GB or more on a NVIDIA GPU with CUDA is highly recommended to allow larger image sizes. A business card, such as the RTX A series is preferred as the energy consumption compared to the value is much better. Maybe the upcoming NVIDIA generation will bring even more value.
On the tutorial site you also get some other guides (I didn't test):

Stable Diffusion Installation guides
Stable Diffusion – other installs

Stable Diffusion in TensorFlow / Keras

Creativity unleashed with Stable Diffusion
To get an idea about the full potential I collected some random examples from the past month. There are so many exciting opportunities. People create comics, movies or just play around.
Kali Yuga models
Kali Yuga provides some cool custom-trained models for Stable Diffusion. Here are some shots taken from Twitter:
KaliYuga on github

from Orobic Oak / twitter
from Digital Jawns / twitter

You might have guessed it, there are also notebooks on google colab around to work with them. An alternative would be jupyter notebooks, but to keep it simpler, we stay with google colab.

Notebook for #textilediffusion.
Notebook for #Pulp-Sci-Fi-Diffusion
Notebook for #lithography-diffusion
Notebook for #watercolor-diffusion
Notebook for #pixel-art-diffusion

Those notebooks are quite easy to handle. I used the #textilediffusion, to generate this sample piece here:

Stable Diffusion and #textilediffusion example from the author
Click the images to jump to the examples. Due to data policy, I don't embed external sources.
Stable Diffusion and Music:

from TomLikes Robots / Twitter
Stable Diffusion and 3D

from CoffeeVectors / twitter
from Infinite Vibes / twitter

from merlino_games / twitter
Here's the GitHub Source for the Stable Diffusion blender texture generator.
Another link, that is connected to Unreal and midjourney is the generation of textures.
Stable Diffusion and Image Tools
Stable Diffusion Photoshop Plugin

Screenshot from Website from Christian Cantrell
There are also options for text-to-image in Canva and GIMP already and Nightcafé and artbreeder seem to have changed their models towards stable diffusion as well (you might remember that earlier this year they both used some GAN+CLIP solution, as mentioned in my article).

Stable Diffusion to generate color palettes from text descriptions

from Matt DesLauriers / Twitter
Other sources to mix with Stable Diffusion or to watch in general:

Get3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
OpenAI Whisper
Google DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (GitHub implementation for Stable Diffusion and Google DreamBooth and another Stable Diffusion Implementation with DreamBooth)

Stable Diffusion and DreamBooth – great video to watch

Final thoughts on Stable Diffusion AI – image jockeys on random turntables.
I love it. Going for the open-source path is so great to unleash new instruments of inspiration and with lucky punches, even to create art. Art is expressive and the artworks being created might get expressive. For me it's an enhancement of our daily practices being creative. I also don't think anyone must be afraid of losing jobs. For precise tasks it is still tricky to work with the tools.
The models are only as good as the inputs; handling the inputs is not the easiest practice and as generating the images takes some time… in terms of client work, the way using any generator might not be the fastest option to choose. As always, I assume clients want to negotiate cost as they think everything is going easier now. But currently the image generation at a professional level is still not precise enough. But the tools are already there to help. Google Dreambooth for example could help a lot to bring consistency into image series.
Looking at the development speed I assume the topic about precision will be a matter of the next years, as well as the topic about speed and custom training sets. I am also worried a bit about the footprint being generated from all the image generations (including all the learning upfront). In hugging face there's a note on the Stable Diffusion model card itself: 11250 kg CO2 eq for the learning (calculated with the machine learning emissions calculator).
The development in text-to-image is far from its boarders yet. But it's so impressive to see the speed of the community to adopt these technologies.
Currently the tools help to discover creativity to finally create a mix of outputs we have seen before. In the future I am sure we can use the techniques to do whole mood boards for clients and in some cases even final graphics or artworks for them.

The only art I’ll ever study is stuff that I can steal from.
David Bowie

Like the evolution of web bandwidth, I think we need some more speed in terms of image generation to open new options and create beautiful human-centered-artworks as one day maybe images could be generated in real time based on likings of visitors. I also assume, at the moment we reached a certain grade of automated artworks, more and more people will enjoy discovering real artworks with their hands again and value of the real original artworks will raise.
In general we shouldn't trust media without a critical perspective, especially when we see images. History showed us that people will abuse technology. So I hope for people to find value in real in-person events after Covid pandemic again. Digital could and will be a lot of fake.
The use of some kind of style transfers on Smartphones is very likely by next year. Dynamic Video generation is also on its way (see make-a-video from meta ai), so we have text-to-video generation and music, VR worlds and games are also around the corner, as Midjourney founder David Holz said. Due to open-source development, I won't say it takes 10 years. And also Google Colab and other services to run the software will change their business models to profit from this boom.
Advertising or Branding will also be affected. It will be an exciting journey in the next two or three years. Osmosis pretends to have some of the features that could be interesting for an early stage. Just imagine adding a target group to the prompt, an industry or a segment. For Learning there are many galleries out there, such as dribble, FWA and many more 🙂 it will be fun to see where this journey goes! (Saying this not being in an international network agency anymore).
As mentioned many times, the development speed is incredible. The current hot tool is google dreambooth (connected to Stable diffusion). Every month there might be another unexpected cool tool to pop up. Maybe even photoshop integrated the tools for their next release.
For me, design turned more into managing design over the years but still design is functional, and art is expressive. Currently it feels easier to create automated expressiveness in fantasy worlds, but nonetheless I also use AI tools during the creative process. The results of machine learning give you a condensed view of common thoughts. So, I often used the prompts to check for color ranges of certain clients, compared it, what google says and easily created something one could call the "wisdom of the net" result.
Our traces on the net and our databases of artists created the look we could mix now. We are the image jockeys of generations that are already there. Unfortunately there's still a lot of randomness in the result to utilize it more professionally.

My "wisdom of the net" results from huggingface
My "wisdom of the net" results from huggingface

So long, enjoy and go out and play to discover the fun playing with those great tools!

Recommended readings

Understand latent diffusion models
Get a full introduction to Stable Diffusion

Midjourney and DALL-E 2 – first hands on impressionsMaybe you remember my article speaking computer language – welcome AI art, speaking about the new role of t
Generative Art – how to get started with AI ArtThe world of art is a diverse and beautiful place. The more we explore new forms of art, the more we are
How AI Assistants Can Help You with Text and ArtHave you ever wished you had a personal assistant to help you with your writing or with your artworks?
AI
Stable DiffusionStable Diffusion is a machine learning, text-to-image model to generate digital images from natural language descriptions. The underlying approach was

Contentforge dialogues
Ryte dialogues

Just like any tools, it has its limits. But if we use it in the right way, and with common sense, AI writers could be great writing assistants that help us to get over the writer’s block or come up with ideas at scale.
To give you an idea, about my used tools in this article: I used contentforge as the holder of my article, then I used rytr to write the blog section, mentioned above. In contentforge I used the "generate content" button from time to time, to see what happens.
Some of the generated text is great, and I can see myself using this type of tool in the future for smaller tasks or ideas generation. For longer articles or books, I absolutely believe that a human writer is needed to create a compelling story.
(so, this paragraph before was added "generatively" as an add on to the article I just wrote 🙂 quite funny)

But yes, these tools could be a great support. I am more worried about all the people using them for just pushing their SEO stuff instead of writing contents reflecting their believes, experiences or research.
When it comes to research and real articles, they don't help you to create the real content. But they could be handy to assist some SEO parts, such as a summary, some headlines in between and so on. To do a deep review, I miss the experience.
It is likely that all these tools make use of deep learning and GPT-3, a generative pre-trained transformer. So in my special case, not being a native speaker, tools (also grammarly, jasper or others) could help to improve my writing in terms of grammar or the words being used in general. The other bigger help I see in the area of SEO, if you want your content to be fully optimized for SEO. So it's kind of a polisher for your contents and an ideation source as well.
If you wonder, why I don't speak about chatbots in the context of text,... cause I think they are an own category, there are also many tools around, most act in a conversational way and are trained by machine learning principles. Here's a good entry/overview.
AI assistants in graphic design
I already mentioned some tools to create so called "AI art" in my article "Speaking computer language – welcome AI Art". I guess I also mentioned some logo generators out there already, maybe I also shared a link to Nikolay Ironov, a designer and neural network.
To wrap it up, I clustered some of the tools in some categories here:
AI generated images
For generating images at a general scale, we could include all the tools being mentioned in my other article on AI art before. DALL-E (1,2, mini) or Imagen and all the Ganbreeders such as artbreeder, nightcafe, wombo.art are some of the AI assistants for graphic design, to name a few.
But there are also tools to create people that operate with AI, like the incredible Metahuman builder from the new unreal 5 engine, there are AI driven ways to create people images. Generated photos is such as platform. There’s also a face generator available to create an image. Of course, you could get quick results, that look like fake quickly, but there are also images, that look quite promising;

Image from generated photos
Similar to generated photos, you could create AI actors with notaperson.ai. In my point of view the characters sill fell a bit static, but the powers of creating random people is there. If you don’t believe, watch the random results on thispersondoesnotexist.com. With the help of StyleGAN2 a lot of images are generated in a GAN way. The world of fakes is real, and besides images, also speech could get modified easily.
If you don’t like real pictures, look at thiswaifudoesnotexist.net. (A waifu is a fictive female Japanese manga/anime character).

Generated Waifu Image
AI generated logos
There are also a bunch of logo generators out there. Nikolay Ironov offers one and on logoai.com, you could quickly generate your own logos with some guided dialogue. A similar builder could be experienced here as a service from shopify or at logomaster.ai.
You might think, how scarry this is? No, it’s not. Honestly, I included AI tools in my exploration on the branding of a client already. Of course, branding is more than a logo, as it’s about the identity of the brand. Each personality has different characteristics that need application to the branding guidelines.
But now let’s assume you type in a DALLE Mini interface the attributes of a brand or a short description and look at the result; now compare it to some other text-input based generator. Won’t you think you get some kind of color scheme on that?
At least for my last branding project, I included AI Tools in my research. I got some straightforward color schemes I could extract and that were helpful. I didn’t use one of the mentioned logo generators though, but they could also be helpful in the exploration process as they help to narrow down the chosen path we want to follow.
In my case the brand itself was connected to AI and Edge Computing, so there was a brand fit with the tools being used. At that moment it felt right to ask the question, how an AI might see the company, reminding me of a nice piece of art by Christian Mio Loclair called Narciss, a a robot, built to analyze its own physical embodiment. Looking at itself and reflecting, what it sees; so why not use such tools, if they match to the brand?
AI drawing
I am not too experienced with AI drawing tools and so I only share the one I discovered some years ago. Autodraw. A Tool to help you create icons. It gives you proposals based on the shape you were drawing. So, in my case it recognized my fish shape and gives me proposals about other fish related shapes at the top.
Form recognition from autodraw
Another fun application in a similar direction is https://quickdraw.withgoogle.com/. Challenge your Mouse drawing skills with form recognition.
Other AI tools for images
Finally, another tool you could give a test is an deep learning based image upscaler.
Screenshot from imageupscaler.com
Resume
Is there a final wrap-up on my mentioned AI tools at all? Yes, there is... we are getting more and more tools that could distract and being applied wrong lead to bad results. Whether it be a world full of fakes (in terms of visual, text, voice) leading to less trust in digital environments or a tendency we often see when tools streamline processes. A lot of trash gets produced.
The danger of expecting magic with the tools is also there. These are tools that could assist, but they don’t define the plan, what you are aiming for. So, define the strategy upfront and evaluate afterwards. A sentence that sounds so true since 20 years and more being in digital for almost anything, whether it be creating websites, apps, designing things or any business objectives. Tools can’t replace goals and a strategy. AI assistants or tools are in this case not different from any other form of tools.
The perfect AI tool must integrate to your own process. In my case I see most potential in the exploration stage. As most creative processes are similar and alike, I refer to an easy system, such as human-centered design. We speak about inspiration, ideation, and Implementation. During these stages of the creative process,
AI could help you exploring your thoughts, flipping the problem and many more things that help you to get going. All the mentioned “methods” would be placed within the stage of inspiration. Also, in terms of a reframing of the topic, AI could help, as it is another perspective on the problem – often from a “wisdom of the crowds” perspective, as the trained algorithms are trained by the masses.
During the Implementation I see AI as an add-on to evaluate own results, in terms of writing to check the grammar, in terms of design to check for uniqueness or testing.
So finally, I could say, I believe in a lot of the automations being possible with AI, but like UX-Design, that got streamlined over the years with bet applied patterns (and corresponding dark patterns), AI-tools will lead to the same effect, some rebound effects: Imagine a lot of sites being written by AI and google ranks being strongly influenced by AI. The overall quality of the content will drop, and a search won’t be user friendly anymore. Hopefully AI results will get recognized and filtered.
Or on the graphic side, customers think of using AI builders for logos and more website design from finished templates. This will lead to less identity, uniqueness on the web and meaningless branding as all look the same. Uniqueness will be the struggle of the future as it is the struggle for all streamlined processes.
I recommend anyone being interested, to have a look at the tools yourself. There are enough tools with a free version to play around. Have fun and use the tools for good!

Stable Diffusion celebrates new forms of creativity2022 is the year of text to image generators so far. With a crazy hunt for new tools starting in summer.
Speaking computer language – welcome AI ArtThe command prompt: A black, empty screen and a blinking line, waiting for my input. Could you imagine
Value, objectivity and the risk of being trapped by searchDear Google, could you please eliminate the stupid goldfish attention span myth?
I know the myth is a
Artificial intelligenceintelligence (AI) is intelligence demonstrated by machines, as opposed to the natural intelligence displayed by animals including humans. AI research has
Midjourney and DALL-E 2 – first hands on impressionsMaybe you remember my article speaking computer language – welcome AI art, speaking about the new role of t
Generative Art – how to get started with AI ArtThe world of art is a diverse and beautiful place. The more we explore new forms of art, the more we are

The command prompt: A black, empty screen and a blinking line, waiting for my input. Could you imagine that this prompt is the access to art or AI art? The command line shows a visual result of your terms, not ASCII art or related items. Nowadays, those prompts have different shapes, such as a form field or the use of voice interfaces.
Your imagination of thought, translated to words in your natural language (mostly in English), gives you access to new creativity. In the words of advertising, you need to express a good core idea, you want to get visualized. Similar to creatives being used to Image search engines, the need to find the correct words gets more and more important.
To explore my thoughts I look at the evolution of the way we tell machines, what to do and how CLIP and other neural models transform the way we could create what we are looking for. Whether it be AI art or maybe any picture we want to see, in general or what I would call AI graphics.
Computer language
A computer language is a language we use to talk to machines. It has been in use for about 70 years.In terms of human history, a very short time.
The first, more popular programming languages found their roots between 1960 and 1970. The first “high-level programming language” was Plankalkül from around 1945. Even earlier, the roots of programming go back to Ada Lovelace, yes, a woman, in the year 1883.
Nowadays, we even have low code or no-code ways to speak with machines. Mostly those use visual environments to drag together code components or nodes and are called visual programming languages.
So, in the past, we needed to write code in a language, following grammar. Now, we could also drag together many nodes and hope for an effect, while playing with parameters. At some point of customization those visual environments also allow to add code manually. But it's not essentially necessary to write the code yourself anymore.
DALL·E and other neural networks allow access to Artificial intelligence as a source of processing your thoughts via a text prompt. If you want, you could say that you tell the machine, what it should program with your natural language. Your chosen words define the quality of the output you are aiming for.
Natural language and natural interfaces
The natural language is the way we as humans speak with each other. To some extent even our voice-interfaces around us are now able to communicate with us in a way that feels almost natural. In this case our command prompt mostly starts with “Alexa”, “Siri” or “Hey Google” to initialize the dialogue. Natural language processing takes the task of converting speech into words in the background to take the right commands.
In 2013, there was a trend to use natural language forms in interfaces to make form usage feel more natural. Forms were integrated into sentences rather than presented as a separate option. Stats are more usable in natural language, but confusing numbers make it difficult to understand the impact of improvements.
The tendency to make interfaces more natural or more human, so we could use our natural senses to interact, is present. But the evolution of the interfaces also depends on technological developments.
Venous materials from MIT lab – Watch example on Vimeo
There are many interesting developments, especially if you investigate the MIT Media lab and the tangible media group. They strive for seamlessly coupling the worlds of bits and atoms by giving dynamic physical form to digital information and computation. Research on venous materials and human-flora interaction provides insights into our future.
One of the easiest ways to interact now is our language, expressed by voice interfaces. When using Neuralink or similar chips inside our bodies, it's likely that we'll need to use interfaces with a well-structured language initially.
If you are looking for more scientific terms on how the language is being used for generating visual results, look for text-to-image synthesis.
GANs produce visual effects resembling the WebGL example on alteredqualia.com that utilizes DeepWarp, a method commonly employed for gaze movement.

CLIP, GANs or AI driven Art and approaches of deep learning.
Natural language processing is also an important tool when it comes to playing with so called CLIPs. CLIP is “Contrast Language-Image Pre-Training”, a neural network trained on a variety of text-image pairs. CLIP got introduced around January 2021.
Before getting into CLIP, a short excursion on learning. Deep learning is part of machine learning and helps machines to learn. The machine is supposed to increase its own skills without the help of humans. On contrast to machine learning, where humans interfere in the learning process, in deep learning humans take care of providing information for learning. To do so, we could also use our input prompt. The process of analysis is up to the machine itself.
An example for machine learning would be connecting words and images by labelling the images with the appropriate word. A typical combination for this cause would be ImageNet and WordNet.
CLIP is a neural network model that is already trained on 400,000,000 (image, text) pairs. So, this means that this is the data that is used in training the CLIP Model. You can input an image to the model and get the most probable caption or summary of it. Alternatively, you can use natural language as an input. The true magic comes when you see examples. Travis Hoppe shares some interesting applications in his Twitter account.
To get a full “beginners” understanding of the CLIP model, I recommend this article: “A beginner’s guide to the CLIP Model”. You will find many interesting articles on kdnuggets.com around artificial intelligence, computer vision, data science, machine learning, natural language processing or programming (Python).
To try CLIP yourself, you could use VQGAN developed by the University of Heidelberg. It’s a GAN (a generative adversarial network), those are special types of neural networks consisting of “Generator” and “discriminator” networks. There is a GAN that generates images. The generator is like an artist and the discriminator is like a critic during training. When the generator can produce an image that the discriminator can't distinguish from a human-made one, we can use it for creating art.
There are huge libraries of available GANs for very different purposes. Some of the lists I could recommend:
The GAN Zoo
GANS awesome applications
To play with the technology to learn, use artbreeder and generate new images; Artbreeder is also based on GAN based image generation.

Artbreeder, Image generation based on GAN
Exploring Applications of CLIP
How to get started? I would always recommend playing with the technology. Most technology is too complex for an early deep dive, so start by playing. If you found your motivation to dig deeper, do it with all your might as you need a lot of patience to get a good understanding.
So, let’s look at VQGAN + CLIP you could use directly yourself online. This is a collab notebook, a small development environment running Python programming language. Your task is to input some values... To get started, I would recommend to only change the “natural language prompt” of what you want to see. In my case I tested some results of different apps with the sentence “Children playing in the forest” (because my kids are attending a kindergarten in the wood while writing these lines).
These are my first results using the CLIP linked above (to try yourself, open the link, basically go through the page and always execute the play button to the left and wait till there’s a check icon next to it, to signal it’s loaded), only for chapter “Parameters” enter a text. In the end you could also create a video. The results here are early iterations. The program would continue till the “free” volume is used up.

First iterations with VQGAN + CLIP
Number 100
Number 150

from wombo.art
from nightcafe

To compare my result, I checked for other applications out there and their result. So there’s for example an application that is also available as an app; In wombo.art you could test directly. The result from wombo art had similar patterns as my test with YQGAN and looked like the picture on the at the top left.
Then I used another popular web-service to compare: Nightcafe you can see on the right.
You see, when using a webservice, you end up getting one result. Using a google colab, you get many iterations and it's not as fast as the webservice or app. (Google Colab or Colaboratory is a way to execute Pyhton code in the browser).
There are also other CLIPS on google colab you could use. The following picture is the (an early iterative) result of the combination VQGAN+CLIP (z+quantize method):

You could continue playing with many more version out there. Here you will find an overview of newer applications: https://softologyblog.wordpress.com/
You notice that the basic style follows one pattern and it’s obvious nightcafe also uses some way of VQGAN combination. To create magic, or what I would call magic based on a text prompt, there’s still some way to go.
The magic I aim for: I tell the machine what I want to see, and they give me iterations of the thing I type in in. It mustn't be art or AI art.
To create this magic, we look at DALL·E. OpenAI released DALL·E 2 in April 2022. They show the real magic of the future. By the way, I wouldn't call the results art. Mostly we generate graphics.
DALL·E 2 screenshot, see more on their website
A short introduction to DALL·E, DALL·E 2 and DALL·E mini
“DALL·E 2 is an artificial intelligence model that takes a text prompt and/or existing image as an input and generates a new image as an output. DALL·E 2 was developed by researchers at OpenAI to understand the capabilities and broader implications of multimodal generative models. To help us and others better understand how image generation models can be used and misused.” (From their own description)
To make it shot, look at their website and get a feeling of the beauty that could be generated with DALL·E 2 or watch the short video on youtube:

Unfortunately, DALL·E 2 is not easily available to test yet. But you could sign up on their waiting list.
According to OpenAI DALL·E 2 isn't trained on faces, to avoid the risk of deep fakes. The danger is already there. Finally it the time to doubt any images being shown to us!
On twitter try a search for #dalle or #dalle2 and you find many inspiring pictures. Going through the magical results, it feels as if image searches could get a relic of the past soon or will only be needed for special causes. Image stock market could be AI generated in my point of view. Here are some randomly selected shots from some twitter members (all linked):



Yes, it’s a pity, we all can’t directly play with this fascinating technology!
Luckily I found DALL·E mini. And that's your friend to play! By the way, I also created my post image with Dall-e mini. I asked for "AI Art blue and yellow". One of the results could be found on the overview site of my writings.
The description of DALL·E mini: “OpenAI had the first impressive model for generating images through DALL·E. DALL·E mini is an attempt at reproducing those results with an open-source model.”
They also have DALL·E mega, a large version of the mini, that is being trained. To see the fascinating world of training, look into the training space.
And finally, here’s your playground to play with the DALL-E mini: https://huggingface.co/spaces/dalle-mini/dalle-mini
That’s my result on "playing children in the forest"; quite an improvement in a very short time. If you play more with DALL·E mini you will notice, that faces still are really not trained much. But I already love it!

To feel the new power of words and to discover some magic along using words, go out and play for yourself!
By the way, there's another tool on the web, giving you similar results. It's neuralblender. Here's my kids results from them:

If you want to get inspired by other AI Artists, look at the work of Refik Anadol, his process is the application of a lot of machine learning to a real world of art. As I said, in my point of view, the term of AI Art is mostly wrong, looking at the output of DALL·E or other CLIP applications, as the outputs are not expressive. In my point of view, we speak about AI graphics.
Maybe you could guess, what my prompt in this case was (created with DALL·E Mini):

Have fun playing!
And guess what... My challenge is to build websites via a text prompt. Currently I am looking for available sources to have a good segmentation (Structural attributes, Style attributes). Let's see when I will share my results or findings.

Some more tipps on playing with DALL·E mini... use objects. To avoid deep fakes, faces won't work. Play around with the text prompt. You could try with the attribute, in the style of... or try the DrawBench prompts being used for Google Imagen first.
Dalle Mini: a rocket in the style of Matsuo Basho

Google Imagen
While publishing this article on Mai 24th, Google came up with a competitor to OpenAI DALL·E 2. The market is on fire! And Google-Imagen impresses the same way, DALL·E 2 did.
Be excited to check out the first tweets on the topic #imagen.
And we will all be happy to play with the first, real demos!
Screenshot from Imagen website

Recommended further readings

Infinite Images and the latent camera
Hierarchical Text-Conditional Image Generation with CLIP Latents
DALL·E 2
Imagen by Google
StyleGan 3
Explore my newer articles around the topics:How AI Assistants Can Help You with Text and ArtGenerative Art – how to get started with AI ArtMidjourney and DALL-E 2 – first hands on impressionsStable Diffusion celebrates new forms of creativity

FAQs
What are AI GraphicsVisual images or designs on some surface, such as a wall, canvas, screen, paper, or stone, to inform, illustrate, or entertain being generated with the help of Artificial Intelligence.
Deconstructing AI Art masterpieces: Create Art like Refik AnadolWhat If you were asked to create art like Refik Anadol? Or to produce some "AI art"?
In case you
Midjourney and DALL-E 2 – first hands on impressionsMaybe you remember my article speaking computer language – welcome AI art, speaking about the new role of t
Generative Art – how to get started with AI ArtThe world of art is a diverse and beautiful place. The more we explore new forms of art, the more we are
How AI Assistants Can Help You with Text and ArtHave you ever wished you had a personal assistant to help you with your writing or with your artworks?
AI