In the past year, the impact of generative AI and other AI assistants on the realm of applied arts has been truly remarkable. From generating unique and captivating visual designs to assisting in the creation of innovative videos, AI has proven to be an invaluable tool in pushing the boundaries of artistic expression.

One of the most notable ways generative AI has influenced applied arts is in the realm of digital artworks. With its ability to generate infinite variations of designs and patterns, AI has empowered creators to explore new aesthetic realms and create mesmerizing, one-of-a-kind digital artworks. Creators are now able to harness the power of AI to bring their visions to life, unleashing a whole new level of creativity and imagination.

What does an AI test assistant tell you about the role of AI in applied arts?

If you ask an AI about the impact of AI assistants in the last year, it would sound like (generated with Linguix, a writing assistant I use while writing posts):

AI assistants have streamlined the creative process, saving artists and designers alike significant amounts of time and effort. These assistants are capable of performing tedious tasks, such as image sorting and research, allowing artists to focus more on the actual creation of their art. By taking care of the mundane and repetitive aspects, AI assistants have allowed artists to devote their energy to the aspects that truly matter - crafting innovative and thought-provoking artworks.

The impact of generative AI and other AI assistants on applied arts is not limited to digital mediums alone. In the field of sculpture, for example, AI has played a significant role in helping artists design and produce intricate and complex pieces. By utilizing algorithms and computational methods, AI has offered a fresh perspective on sculptural creation, pushing artists to explore novel forms and experiment with new techniques.

In conclusion, the influence of generative AI and other AI assistants on the realm of applied arts in the past year cannot be overstated. From enabling artists to delve into uncharted aesthetic territories in digital art to revolutionizing the creative process, these technologies have undeniably reshaped the landscape of artistic expression. As we move forward, we can only anticipate further advancements and a continued partnership between artists and AI, opening up even more possibilities for the world of applied arts.

Reading through the answer sounds very compelling, and truly I think this influences the direction of my post; there are many aspects and directions, I didn’t take into consideration before. It also felt a bit like… wow, did I miss so many things?

Let’s ask google’s bard, to provide me some examples to some of these mentioned improvements last year (ChatGPT’s memory ends in January 2022, so it wouldn’t help).

My first question for Bard:

Could you name some artists using AI to design sculptural creations?

As you might expect, I ended up having a smile on my face. At first, Refik Anadol was the top-ranked artist, followed by other artists who are, based on bard connected to the usage of AI.

The true reason for my smile was the double test and the same results; so I tested the question “Could you name artists using ChatGPT or text-to-image generators for their artwork?”.

The result was the same list, with the same errors; Bard didn’t distinguish between artists in general having some relation to computer art or playing with computer algorithms, such as Josef Albers, Maurizio Cattelan mainly known for sculptures and installations, and other artists using algorithms in general.

The most entertaining outcome from the list is the incredible hallucination of Markus Rehm, or at least I hope it is, as I must admit I am not familiar with this artist.

AI has been used in New Media art for quite some time.

Honestly, I don’t know which renowned artists really came up with new pieces of art within our last year that made use of ChatGPT or Text-to-image generators to build the foundation of the artwork itself.

In my understanding, the text being generated by AI shows a misconception about art. Especially new media art is and has always been about the edges of our technology surrounding us.

The popularity of new media art has been consistently rising since around 2014/2015, particularly in the areas of image generators and algorithmic experimentation. Google DeepDream was released in 2015 and many artists played with it. I recommend the read of Memo Akten “#DeepDream is blowing my mind” from 2015.

I know everybody hates #Deepdream, but I really love it. However, it’s not the aesthetics that I love, but the poetry of what’s happening inside the algorithm.

Memo Akten on DeepDream

There’s also a related video to his DeepDream experiment from 2015 on Vimeo.

Besides Akten, also Mario Klingemann started with some DeepDream experiments back in 2015, before he made more use of the emerging Generative adversarial networks (GAN) and created I guess his most famous piece of art in 2018, Memories of Passersby I. By the way, Mario is also one of my heroes as he accompanies my life for a long time; so he was a master in Adobe Flash and so many things that also played a role in my life. Get inspired by his journey and watch this video on YouTube.

GANs and DeepDream played a role for many artists. In 2016, we had great projects, such as “the nightmare Machine” by the MIT Media Lab, and also the beautiful project “The Next Rembrandt” got introduced in 2016. With Edmond de Belamy, a Generative Adversarial Network portrait painting from 2018, being sold for $432,500 at an auction, more and more artists realized that there is a lot of potential in that field.

For a long time GANs were the core to generate images; so before we had DALL-E (in version 2), Imagen, followed by Midjourney, Stable Diffusion and others, we relied on GAN-related image generators to generate pictures.

Back in 2022 I wrote a post on “Speaking computer language – welcome AI Art” and it gave a glimpse into the current status quo (before mass market tools dominated and changed the development). With the introduction of CLIP (Contrast Language-Image Pre-Training) and other similar technologies, we witnessed the emergence of image production tools. However, it is the arrival of the latest text-to-image generators that revolutionized the market, turning it into a mass market phenomenon.

This might sound like I’m joking, but I am not: I believe that one of the hardest problems in artificial intelligence, which still has not been solved satisfactorily, is humor. Once machines actually know how humor works and are not just involuntarily funny sometimes like they are now, they will have achieved true intelligence. And as a free bonus, with that understanding will automatically come the ability to know when to be serious.

Mario Klingemann on the question “What are the hardest problems at the intersection of technology and society that deserve more attention?” in MIT Technology Review

Regarding jokes and humor, at least Sam Altmann had a great reaction after the introduction of Elon Musks Grok.

It’s all about the data – New Media artists don’t care so much about ChatGPT and Text-To-Anything generators.

I already explained the role of data for the art being created by Refik Anadol in my post “Deconstructing AI Art masterpieces: Create Art like Refik Anadol”, in general AI is at its core a set of data technologies to analyze and transform an input.

In the past, artists had to classify the input images for AI models themselves. Now, there is a modern form of slavery where people are paid very low wages, around $2 per hour, to classify data. The problem is that the content they have to classify can be traumatic, see the article “OpenAI Used Kenyan Workers Making $2 an Hour to Filter Traumatic Content from ChatGPT” published in vice magazine.

Getting back to the initial listing, AI proposed artists making use of ChatGPT or text-to-image-generators and the top result of the list, Refik Anadol.

Most of his current artworks still rely on the GANs, so there are different ones, such as DCGAN, PGAN, and StyleGAN algorithms being trained on massive datasets.

Unsupervised — Machine Hallucinations — MoMA, recently being displayed in the MoMA is one of the works of Refik Anadol. The thought of playing with “hallucinations” beautiful reflects machine hallucinations in an expressive style.

Art that is influenced by Text-To-Image generators? Yes, dear Bots, here are some examples.

AI tools are great, no question, no doubt. Creating text, editing text, writing code, editing code, finding inspiration, testing ideas, … there are so many great ways to make use of AI. So let’s finally look at some examples that really relate the usage of mass market tools being available since around a year.

AI fashion examples

AI fashion in general has great examples of how to use text-to-image generators. Though many use the looks to get quick money again (this time in the shape of NFT), I wouldn’t blame them for doing so. Better than generating fake content on the web to gain traffic and generate money with it; see my main criticism in part 2 of the series “who owns the truth – the web”.

“Contemporary” AI Art

I don’t even know if there is something like a contemporary AI Art section, but if so, I would put some work from Violet Forest into this category. Her series “Just an AI girl” is just fun and is playing with contemporary clichés and one of the main issues mainstream text-to-image generators have had the last year; displaying strange hands.

Violet Forest (Violeta Lopéz)

Danielle King

Artificial Childhood Memories is an invitation to reflect on our own memories, the stories we choose to reshape, and the ever-evolving digital canvas upon which we sculpt and re-work the narratives that define us.

Description from Joyn.xyz on Danielle Kings’ work “Artificial Childhood Memories”

In "Artificial Childhood Memories," I delve into the delicate interplay between technology and nostalgia. This exhibition explores the ways in which AI can be harnessed to recreate and reimagine our fading memories, shedding light on the malleability of the stories we tell ourselves.

Danielle King describing her work “Artificial Childhood Memories” on her website.

Art is expressive and should also be related to our society and what moves our zeitgeist. From my understanding of the provided Answer, so far it seems that language models don’t understand differences within applied arts and they also see all "creators" as artists. The field of applied arts is far wider than only arts. It’s the world of different design disciplines and architecture as well.

Looking at my entry point, I had no chance to speak about applied arts in any design discipline so far… only because I was surprised how wrong the Chat output was.

The role of AI tools, such as ChatGPT or Text-to-image, video or anything generators, has on the design practice.

Simple answer: it’s a massive support for finding ideas and especially directly testing ideas. Being able to use language is now a competitive advantage and there’s no need to only rely on pure sketching skills.

But still, I don’t see finishing skills in AI. New Media Artists have proven over the last 10 years, that the manual selection process, or I would call it the classical role of an Art Director, is needed to point AI into the right direction.

Our dear AI buddies help to be far quicker – But still, they are supporting tools

For me being a solopreneur sometimes, sometimes part of a team or as a design-consultant on a project, I had different usages of AI. The best is sketching and selling ideas in a rush. So I had this little superhero being created for a client with some variations within half a day till I got the client's approval. The work to finally get the model created with 3D and some post-production/optimization took much longer.

The image was prepared with leonardo.ai and quickly modified in Photoshop to present to the client.

A main benefit for me is the brainstorming and ideation part. So every time I am stuck with something such as preparing a script for lectures, I could use AI as my muse to get distracted and refocus my thoughts.

The article “AI Reveals the Most Human Parts of Writing” from Wired Magazine illustrates this role of an AI-supported Writing buddy quite well. For some writers, the use of AI writing tools is like a new ghostwriter, but also the effect, that AI is giving you the direction of the text, might emerge. Same in my case for this post. The generated text of AI was a bubble and I wanted to elaborate on it to find out, that also the AI Chatbots had no clue about what they were talking in the end, AI dominated the direction of this text.

In terms of writing, AI assistants are a great aid, also in terms of usability. I assume it is much more pleasing to read an optimized text than my own, plain thoughts.

Text-to-video: Runway ML, Kaiber, Pika Labs and others

For me, there’s no use case yet to say, AI itself is the main tool yet to go somewhere. They massively help to get a feeling of potential looks; so as of November 2023, text-to-video made some impressive advancements, and it’s not Runway ML alone being the only interesting pick. But still it is one of the most impressive.

To get an idea of what Runway ML is capable of, it’s best to look for results on Twitter. @maxescu work is worth a look. He recently produced a video “the outworld” that could also be seen on YouTube, that is 100% AI created (and it is impressive). I would also recommend to filter Twitter results and look for #runwaymlgen2.

Ammaar Reshi is another, highly talented AI Movie Director with great demos.

Other contenders are also there, but all lack the quality of Runway ML at the moment. Maybe Kaiber has the most potential at least for some kind of looks. Other shots are from Pika Labs and from Genmo.

The beauty of these tools is that even when artists or people who try this tool explore it, it helps all creatives broaden their horizons. Creative work is heavily influenced by what we have seen.

UX Design and AI

To give a status quo about UX design and AI, I first start with a personal perspective based on my working process and experience during the last year.

On the visual part of the process, I tried text-to-image generators and their website output; but currently I wouldn’t suggest the use to text-to-image generators for more than a creative direction of a mockup and I wouldn’t always recommend to show it to clients.

The thing is, you easily get an interesting visual appearance. But in terms of structure, user guidance, or in short in terms of usability, my feeling was that the generated screens show one of the biggest pain points that brought me business in 2023.

The best recommendations I could give so far would be Galileo AI, Visily, WireGen for Figma, in terms of tools and 3M VAS to check, to buddy your design.

I had to conduct UX workshops in two agencies that have their strength in classical design or editorial design. Interestingly, both complained about their lack of usability when they incorporated it into their online designs or magazines. Based on my experience, this is a true point I see now in AI-generated websites. They look good based on an arrangement perspective, but are not usable at all.

Jakob Nielsen, part of the world renowned Nielsen Norman Group and maybe the godfather of usability, wrote an article on his own legendary blog about the “Unreliability of AI in Evaluating UX Screenshots”. For a small study, 12 e-commerce screenshots were given ChatGPT for evaluation.

The suggestions that were given were inconsistent and untrustworthy. Nielsen compares the results with an early intern and sees perspective in a Version ChatGPT6 maybe in some years to provide some valuable help in evaluation.

Ok, this article is from one of the strictest masters in usability, but Nielsen isn’t AI in general; he has a lot of great use cases, where AI assistants are helpful, such as ideation.

As another article on “AI-Powered Tools for UX Research: Issues and Limitations” from Norman Nielsen Group said, one of the core problems is that most available tools can’t process visual input yet. This is where ChatGPT's ability to vision comes into play and will show us where the future leads. Similar to the artists, I assume if Jakob Nielsen would “teach” his Bot to classify, it would be able to classify more Usability downsides easier.

ChatGPT Vision – pushing boundaries

I didn’t expect my post to get so long, so I try to get the curve slowly. For me the next major step now is to add more senses to AI. ChatGPT being able to hear, see and speak is massive.

Let’s look at some early visions:

Mckay Wrigley has some great early examples of what ChatGPT Vision is already capable of. Also Pietro Schirano has some interesting examples on the application of ChatGPT Vision.

Summary on the use of AI for applied arts

Yes, you are right. I didn’t look at all developments in applied arts; for me, it is mainly about a glimpse and my personal attitude towards the topic.

When it comes to applied arts, I could write endlessly and share more examples, in music for example, there’s the recent creation of Beatles song and a great short film explaining the process. Or we have “The Project Gutenberg Open Audiobook Collection”, a collection of free audiobooks created with neural text-to-speech technology.

The major issue with all this development is the speed of development and the wide range of tools. It is insane. On the business side, I stopped two clients not to push forward with an AI-supported website chat earlier that year; I told them, it’s not yet the time. 5 months later you found easy tools to train the chat with the website content easily on every corner… but still, I tell them to stay calm; wait for the European Act on the usage of A, and then we see which providers there are with a suitable solution at fair price (of course one could also use an own hosted solution now, but that’s not the point).

Staying calm in times of noise is my personal approach to current AI developments. All investments are pumped into AI, being a front-runner on a business side needs money and willingness to adapt quick; there’s no simple finish yet.

As you might have noticed on my blog, I wrote a lot more about text-to-image generation before we reached the state of mass market. And honestly, it would have been hard for me to write about it, as every few months some new exciting developments would have changed the direction of my thoughts. I always sing up for some early access stuff; so currently I am mostly waiting for access for everArt.ai.

I love AI for applied arts and I love the progress in technology we face. The speed is insane, and I assume the fear of missing out on some of the stuff is big. Especially if you are thinking about business ideas and applications for “quick” money. I also tried some stuff, trained my own models, built some Next.js apps and so on… but finally realized, that’s not my turf.

My role in this game is observing, applying and consulting. And honestly, this is quite relaxing. Why chase for quick money, if you could serve clients better and quicker? Providing happiness is a good sake and more fun to aim for.

As my counter program to the speed in AI developments I recently bought an iDraw H A3 to have some fun “drawing” things. Maybe I also share some post on that one some day.

My true recommendation on AI tools is, integrate them to your workflow, step by step and do yourself a favor to not try each and every tool. Relax, find your influencers and stick to the stuff that works best for you till you have a use case, some elaboration of another tool might be needed. AI Technology changes so rapidly, so don’t waste your time on trying tools.

OpenAI CEO Sam Altman said on social media that a new Moore's Law may appear soon, that is, the number of intelligence in the universe doubles every 18 months. If you imagine this number, you might understand how important it is to stay calm.

The only hope is that downsides of AI, such as bad datasets that infringe intellectual property law or unethical training data, get fixed. Also, the compensation of humans whose work has become training data needs improvement.

In arts and craft there's no truth; there is only exploration.

In the following article I share my experiences from teaching and give you some recommendations on how to dive into the world of Extended Reality (XR). In my case, with a focus on AR and MR experiences. I start with a brief theory introduction, to at least cover some essential basics.

AR, MR, VR, XR or spatial computing?

So first, let's quickly explain some acronyms we love in our tech world. XR refers to extended realities, which is a broad term that encompasses Virtual Reality (VR), a completely virtual world, and Augmented Reality (AR), the real world with virtual objects.

Mixed Reality (MR) is a blend between Augmented Reality (AR) and Virtual Reality (VR). In MR experiences, virtual elements are added to the real world, allowing us to interact with them. This means we can influence both the real and virtual worlds. In contrast, with VR, we solely manipulate the virtual world.

Reality has always been too small for the human imagination. We’re always trying to transcend.

Brenda Laurel - author of "Computers as theater" (1980)

Extended realities are exciting as the only limitation to our experience is our imagination. We love diving into fiction and discovering new realities or escaping our physical realm around us. But it's also diving into the unknown, which might cause fear or makes us cautious, especially in the process of making.

The term Spatial Computing was defined by Simon Greenwold in 2003. He described Spatial Computing as the human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces. 

Here are some examples to get an understanding about each discipline:

Anandale (VR)

Anandale is a virtual reality art experience by Kevin Mack. Explore an abstract three-dimensional labyrinth flying through evolving shapes and colors populated with intelligent artificial life called Blorts.

Magic Bench by Disney research

An older example on a mixture of applied AR and MR to interact with virtual characters in a fun way. Please watch the video and explore the disney research site for further details.

Pokemon Go

I guess the most famous AR application out there. No big explanation needed. Maybe it's time to re-look at the Pokemon go app to see where it developed over the years.
Also a simple way to learn simple AR interactions and to get a feeling for the spatial in general.

Others

Microsoft Hololens and it's applications

Start designing in AR or MR – sketch ideas.

Most undergraduate students might feel intimidated by the examples because they only have a basic understanding of 3D and interaction. Their main skill is arranging objects, which they learned in graphic design. The next important question that arises is which program to use. However, instead of finding excuses, we should embrace what sets designers apart and simply sketch our ideas.

Before discussing concepts like presence, agency, the story, or different types of input interactions, always take the time to think about and outline your idea. For all types of designing for new media, there's a big hurdle to learn or dive into some technology one never used before.

Designing for AR – exploring solutions to refine idea

In AR there are many applications to test an idea. Once you have an idea or sketch, you can explore different solutions and understand how they were achieved. This will help you refine your idea and simplify it to get started. Most likely you end up looking at many tools, such as lens studio, AR Core, Spark AR, AR Kit or maybe unity3D, InstaVR or roundme. There are many apps and most of them are for free to get started. Unfortunately the entry barrier for many of the mentioned tools isn't the lowest.

The key here is simplifying. New programs and software can be overwhelming if you're not experienced. Unfortunately, when it comes to new media practices, you have to explore a program or technology to see if your idea is feasible. Instead of being overwhelmed, it's best to simplify an idea to get into the stage of testing and making. So to give you a feeling of easier entry points, I show you three simpler ways to get started:

Artivive

There's Artivive, for example, a platform to create simple AR art by linking classical paintings or posters with a digital layer. Artivive is working as long as no further interaction is needed, and the development process is very simple. Create a video that gets played when a user hovers his capture device (mostly your phone) over the painting or artwork. The object itself could be transformed to a QR code, so there's no additional activation needed.

Similar to artivive there's the magic photos app, that is even simpler. To put it simply, imagine having a still photograph that comes to life as a short video with just a hover of your camera.

To get started with Artivive, here are some tutorials to follow.

Adobe Aero

I was unsure if I should include Adobe Aero on the list, but it's the next logical step after Artivive. It allows you to not only place videos, but also other objects, and easily interact with them in an AR environment.

Adobe Aero is great for exploring and experiencing interaction in virtual spaces. It's the best way to understand how ideas can be brought to life and to learn about spatial environments and how to interact with them.

In terms of the limitations of the software I have to admit that Adobe Aero is a beta software (for years now). Adobe is definitely prioritizing the incorporation of an increasing number of AI features into their software suites in order to stay ahead of the curve in the AI landscape, alongside midjourney and stability AI. Hopefully Apples Visio Pro will push the development of Adobe Aero next as designers need to learn how to design for XR.

Also, for one of my last years classes, most students used Adobe Aero to create their interactive AR posters. By building upon their existing strength in graphic design, students can take their skills to new heights in the virtual realm. Some examples of the students work will follow.

Here you find some good starting tutorials showing a good range of the capabilities of Adobe Aero. You will notice, that Adobe Aero is mainly the app to put together the assets you create in other apps:

Example Projects from Students

As being said, we used Adobe AERO for a course around new media in general, besides looking at other contemporary technology, such as generative design with the help of p5.js or the use of AI image generators.

For their submission, the students were asked to select a topic that suits under the overall theme of "change". A fictional exhibition at a museum. Their task was to create an Augmented poster to promote the exhibition.

Here you find two examples of what the students came up with. One is about the topic dementia and the changing state of mind (left side). The other one is about the topic of speed, the change in speed and the double meaning (right side). What you see are only simple videos. In case of the right example, the Augmented experience is much more interesting, as it is interactive and you are able to "take the pill".
The left example is working better as a simple video. Both great works of the students.

Other resources (more advanced and more contemporary):

Get started with Geospatial Creator in Adobe Aero

Create world-scale augmented reality experiences in minutes with Google’s Geospatial Creator

Adobe After Effects

To bring across an idea, there's no need to build AR in "real". After effects is a great tool to simulate a fictive world. Its tracking abilities allow users to simulate real-world objects and envision a fictional future world, all without the need for actual VR or MR glasses. Undergraduate students often use After Effects to learn motion design and interface design. This knowledge can be used to imagine AR or MR worlds and the best part is, there's no need to really build the app, but it's an advanced instrument to sketch an idea even further.

After Effects is perfect for prototyping UI concepts. The only missing component is the lack of a real interaction experience. But the resources required to bring the vision to life must be generated. This process primarily involves utilizing gestures and movement to gain a comprehensive understanding of the interactions involved and therefore also deepens the interaction knowledge.

Check out Volodymyr Kurbatov's YouTube videos for awesome examples of how After Effects can be used. They are a great source of inspiration! Here are some shots from his channel:

Here's a tutorial on how to build the card effect from Volodymyr Kurbatov with after effects:

Iteration for refinement of AR and MR ideas.

When you start learning, it can be challenging. To understand how your design works in a new medium, try different approaches and think about how you've implemented it. Remember the rules you've learned from interaction design. Usability in space behaves quite differently.

There's no "simple" touch, as it's hard to identify if one interacted with an object by purpose or by accident. An early prototype with the help of After Effects or Aero helps to refine the rough concept.

For implementation other, more complex programs are needed, but the true question would be, is this still the space of the designer? Depends on your role. To better understand and explore new media, I recommend exploring new spaces and programs. This will enhance your contextual understanding.

My understanding of generative design, new media design in general is that playing with the medium itself, whether it is the data, the visuality or the code, leads to a better understanding of the medium and therefore a better result.

The trickiest part and most annoying parts are the intersections with other programs or limitations of software. Especially when it comes to importing and exporting data formats from one source to another.

One of the most challenging aspects of teaching is explaining the importance of learning to overcome obstacles, to feel frustrated with a project or software, and ultimately, to redirect and improve the approach towards finding a solution.

Working with AR or MR designs in the current working environment provides a unique opportunity to gain a deeper understanding of the inherent limitations of new media technologies as they emerge. Designing for Extended Realities (XR) is more challenging compared to using AI generators, which have significantly improved in just one year and are now almost ready for the mass consumer market. AR, MR, and VR have been around for a long time, but they haven't gained popularity among the general public. This is because there are several challenges such as hardware and software limitations, concerns about data privacy (like the case of Google Glass in 2012), and changes in how we interact with the technology, which make it quite complex.

Edge cases of developing solutions in Adobe Aero or After Effects

Adobe Aero is a prototyping tool that falls somewhere in the middle fidelity range. It allows for some interactions and also provides a simulated visual experience. The problem with Adobe Aero is not just the beta issues, which cause strange errors like problems with file naming or importing/exporting. There are also strange issues with the file import functionalities, especially when it comes to 3D assets.

So one major issue is placing a real interactive 3D Object on the stage that is animated. For Adobe business subscribers, there are a lot of FBX files on Mixamo to be downloaded for free. For educational subscribers, the option isn't there.

So if you want to have a look at this part, there are easy tutorials, such as this one. To make your own FBX in Blender or Cinema4D, you should carefully adjust the export settings. It would be helpful to follow specific tutorials like the one for Blender or the one for Cinema4D to ensure it meets your requirements. Also a better tutorial on the intersection of Cinema4D and Adobe Aero is this one from schoolofmotion.com.

Working with 3D in various fields like web design, movie production, and game development often involves dealing with file export problems. And I assume we all hate exporting issues from the one source to the other... hopefully our magic AI developments also help to overcome this barrier.

Adobe After Effects is mainly considered a tool for low-fidelity prototyping. It lacks real object interactions, so we simulate interactions to create additional assets. After Effects is better than Adobe Aero in terms of conceptualizing because it allows you to focus on your creative goals without having to deal with technical problems during idea development.

Further Resources

• Road to VR – news site on VR topics
• UploadVR – VR and AR news
• VRScout – News site
• AR/VR journey – News site
• Virtual Reality Pop – Articles around XR
• Publications from Stanford on VR
• Free book on virtual reality by Steven M. LaValle

If anyone asked me at the beginning of the year about AI, I would have said, it’s quite useful for text processing and predictive tasks. In the realm of artificial items' creation in graphics, art, or music, it felt like a playground mainly related to the arts.

GPT-3 as an accelerator – GPT-4, entry to the mass market?

The power of GPT-3 unfolded its power, starting in June 2020 and throughout the last years. We saw many upcoming apps using these skills to help you write, structure or code.

Due to their familiarity with the topic, most know or have tried some software for writing text, including jasper, rytr or others. As mentioned in my other article How AI Assistants can help you with text and art, they are useful, but help you more on the surface and for mass-tasks. Maybe GPT-4 will bring massive change and acceleration to the market. GPT-4 is expected to be released in 2023 and will push the market forward again.

Besides known applications, we also find novel approaches to things we are accustomed to, such as "googling", for example. Metaphor is another way to use a search engine, maybe not leading to issues google (and others) are already facing. See my criticism in the article "Value, objectivity and the risk of being trapped by search".

Getting back to GPT-4, most of us don’t know that Microsoft has exclusive rights on GPT-3. When GPT-4 comes out, it may give them an edge. They could add new features to Microsoft Word to make it a writing assistant.

The whole AI market helps them filter the best ideas from current apps and incorporate them into their own product. Having started my career in the first dot com bubble, I currently see similar dangers, as startups pop out from nowhere. Prototypes get released and make incredible money within a few days. From self-portrait apps such as Avatar AI and profilepicture.ai to logo makers and Figma plugins like Magician, there are plenty of great tools to help you create amazing designs.

Applying GPT or Stable Diffusion is now more accessible due to decreased entry barriers. On the other hand, we also see giants firing employees (and maybe shifting their business focus towards more integrated AI applications).

Do we face a paradigm change in applied creativity?

During the days of the Renaissance, humanity itself moved into the center of discovery and human creativity. Do we face a paradigm shift towards machines?

I am sure many people think like that, but not only for creative purposes. We ask for more automation in many aspects of our lives and replace repetitive tasks with automation. In my opinion, this is fine.

It's unsettling how rapidly technology is evolving. Everyone could start generating their own comics, storylines, always supported by AI.

On the one hand we have the text-to-image generators, on the other we see tools writing for oneself and adding other dimensions of ideas. For me, it feels nearly impossible to keep track of all the developments.

With the release of ChatGPT on the 30th of November, another fascinating tool arrived on the scene. If you ask ChatGPT what it is, it answers something like this: ChatGPT is an AI-based chatbot that allows users to interact and ask questions with a virtual assistant in real time. This chatbot is driven by an advanced machine learning algorithm that adapts to the user's conversational style and interests to provide a customized and enjoyable chat experience. This chatbot can be used to write poems, to do some programming for you or write songs. Explore the creativity of users on Twitter and you will see that there’s nearly no limit. The hype cycle of innovation and the peak of inflated expectations is already here.

Various business ideas around ChatGPT see the light of day each day. With the release of GPT4 we might reach another crazy dimension of text to anything tools. Bloomberg titles “ChatGPT Could Be AI’s iPhone Moment". I guess not yet, but we are close. My guess would be GPT4 would be this moment as ChatGPT is a kind of version GPT3.5 and already shows how impressive it is.

The development speed is incredible. Stable Diffusion is currently available in version 2.1, making progress on some details, and accelerating other tools such as the hyped lensa app (available for iOS as well).

In June I thought, Dall-E mini was wow, but since summer we have seen more advanced wows with midjourney, dalle-2 and Stable Diffusion.

As I mentioned in my article Speaking computer language – welcome AI Art, the text prompt gets more important. Expression of thoughts and ideas as a tool to facilitate creativity. In a current course at university called “Interaction in space”, we touch diverse forms of interactions and now face generative programming. During the course, I told the students that generative programming helps with your creative process. The use of generative programming could generate inspiration and provide you with results and directions you might not have considered. The same applies to text-to-anything converters. It’s a supporting tool in the exploration of ideas.

Looking at text-to-image tools, it is crazy how fast one could test visual ideas and modify them into another direction. With ChatGPT you could generate initial code and modify it with the ChatGPT assistant. Maybe learning programming could get easier for students.

The downside of open data

One of the biggest issues I see currently is the problem with scraping data from the web and using this data for training AI models. There's no control and being a creative I could collect great pieces from my favorite designers and feed my own models. You could scrape data from collections like FWA or CSSDesignAwards to train a Website Design model. Maybe such a solution is already available in the market.

My point is... is it fair? Using data someone else collected and categorized for your own models? As in the hyped times of bitcoin, ethical standards are not the focus of business ideas. It's only about making money. Many people who once mined with cryptocurrencies now lend their GPU power to machine learning. Local learning (I tried with a GeForce 3060) takes ages and the footprint of generating all those models is also huge.

I am concerned about the fact that anyone could work with someone else's data without asking for permission. A new Image format including scraper protection might help since everyone should be able to decide if someone else can access the data. Any blog or written word falls into the same category.

noai and noimageai as a solution against bots – as long as they respect them

After some research, I discovered that there's already a solution for the scraping part of it. Deviant Art introduced the noai and noimageai for all their users by default back in November 2022: UPDATE All Deviations Are Opted Out of AI Datasets.

The solution could also be implemented fairly easily on your website – so this is only a bot protection if the bot respects the rules:

Don't allow AI at all:
<meta name="robots" content="noai">

Don't allow AI to use images:
<meta name="robots" content="noimageai">

Make sure none is used
<meta name="robots" content="noai, noimageai">

Still the problem is the ethical agreement itself. The idea of a Hippocratic Oath for developers, ML applicants or anyone using AI is great. Scrapers/bots need to respect rules and those rules must be implemented directly. Similar to early stages of privacy policies, there's no opt-in by default. It's always an opt-out by default before anyone uses another's data.

As artists are mostly affected, it is great to support them by using the protected good of major corporations, see the discussion and examples in this post. Or the way ArtStation reacted, by removing its own creations.

For all creatives I could finally say: trust in ideas and learn to use generative AI.

Block ChatGPT from your website

In addition to the former options to prevent the usage of your images, since August 2023 there are ways to also indicate that ChatGPT should respect your content and not index it. To do so, put the following code into the robots.txt file. Opting out seems the new paradigm.

User-agent: GPTBot
Disallow: /

Living in Europe, I am quite sure, we will have new regulations in the near future; that on the one hand do good at saying no to auto-opt in, but on the other hand add too much overhead, which stops progress.

In the end, it will be interesting what happens if many AI-generated contents flood the internet and AI crawlers work with this mostly generative, bad content. In German there's an expression "the cat bites its tail", which means it's kind of a circulating problem not coming to an end (similar to the expression catch-22).

Related Contents

Microsoft and OpenAI Working on ChatGPT-Powered Bing in Challenge to Google

Updates from January 2023:

2023: The year of the conductor – great article on the current status quo on AI in terms of mainstream technology

A helpful explanation about some parts ChatGPT is struggling with and about how WolframAlpha might be helpful: Wolfram|Alpha as the Way to Bring Computational Knowledge Superpowers to ChatGPT

Great article on the consent issue from the creators of haveibeentrained.com: AI Art and the Problem of Consent.

Have I been trained website

The Great Generative AI Debate: To Use or Abuse?

All online Contents should be crawlable for AI

Updates from October 2023:

Data Poisoning generative Image Models

More Basics from the web

2022 is the year of text to image generators so far. With a crazy hunt for new tools starting in summer. Following some GANs of 2021, such as VQGAN and others, I mentioned in my article “Speaking computer language – welcome AI Art", the hunt for the next generation of text-to-image generation started in April when OpenAI introduced the magic of DALL·E 2.

During the year I shared articles around the topic of AI tools for image generation and if you had asked me in January, I wouldn't have expected such a development. I tested midjourney and DALL·E 2 and shared my thoughts on different ways to utilize AI tools. There's also Google launching more and more new tools on the market, one of the latest is DreamBooth, something that could be added to the workflows of the broadly available text-to-image tools, but that's another story.

While finishing my review on midjourney and DALL·E 2, I noticed the official launch of Stable Diffusion. Then I struggled, as Stable Diffusion is different from all the other tools before: it’s Open Source. Being Open Source, it is a creativity machine on its own, as it is possible to connect it to other tools as well. My core question was… where to start to share my thoughts? Explaining prompt guides as for midjourney or DALL·E 2?

In my following article I wanted to give a broad overview of the current status quo as of September 2022. Ways to use Stable Diffusion and share possibilities of open source software happening currently.

What a good artist understands is that nothing comes from nowhere. All creative work builds on what came before. Nothing is completely original.

from Steal Like an Artist: 10 Things Nobody Told You About Being Creative.

Stable Diffusion – what is it?

Stable Diffusion is a model that can create digital images from text or use text prompts to guide image-to-image translations. It will be released on August 22, 2022.Looking at the research paper you will notice that some names you might have come across in former publications on GANs will show up again, so all people being mentioned in the research paper of VQGAN are mentioned: Patrick Esser, Dominik Lorenz and Robin Rombach.

Understanding the heritage can help comprehend the rapid development of VQGAN in just one year. The progress has been extraordinary.

Currently, all main competitors (DALL·E 2, midjourney and Stable diffusion) rely on the principle of diffusion to create their images, so-called latent diffusion models. If you want to get a deeper understanding, I recommend a good written article from Louis Bouchard or watch his video:

Looking at results with VQGAN and Clip, see my older article and results from one year later, give you an idea about the speed of development. We face many competitors and an impressive research speed. I assume the next challenge is render speed while generating the images and optimizing the diffusion models. In the meantime, the world is allowed to unleash its full potential by connecting more libraries together to Stable Diffusion.

I really recommend trying some of the text-prompts yourself. Generating images on your computer can be difficult. The process involves using language prompts and the speed of image generation tells you how complex it is. On my MacBook (with m1) it’s running ok, but I am not sure on which PC builds you could run it.

Playing with the software, you'll observe cases the current implementations struggle with - one of the harder ones is stigmatizing the world around us. No wonder if the models are trained by common images of the web. The most common pictures on the web are pictures more beautiful than reality.

Watch "Stable Diffusion - What, Why, How?" by Edan Meyer, a one-hour instructional video on creating diffusion videos. It's presented greatly and a great resource to get started with Stable Diffusion.

If you want a shorter version of Stable Diffusion, look at the 2-minute paper.

Stable Diffusion – Quickstart via dreamstudio, collab, huggingface

If you had a look at midjourney or DALL·E 2 before, you experienced simple text-to-image prompts to create your artworks. Being Open Source opens a variety of inputs for Stable Diffusion. I try to give you an overview (with a raise in complexity and flexibility).

Dreamstudio.ai

Dreamstudio is the easy access to Stable Diffusion and the answer to popular midjourney or DALL·E 2. To get started, you need to login. Also, the prompt guide is only available when logged in.

The main difference to midjourney (using discord) and DALL·E 2 (basically relying on the text prompt) is the interface you get provided. Besides a text prompt you also get some rulers to modify the settings. Though the wording being used is still geeky, it is progress to the simple text inputs from the other main rivals. By the way, there also used to be an official discord bot, but they shut it down.

The payment model for dreamstudio feels more complicated than for midjourney or DALL·E 2. You also get free credits to get started but the amount of cost varies with your settings; here’s their explanation:

At default settings, it’s one credit per image. Depending on the image resolution & step count you choose, this can go as low as 0.2 credits per image or as high as 28.2 credits per image. Here is a detailed chart for explanation:

If you need some extra guidance on the photo editor, watch the video from their FAQs: Getting Started With DreamStudio Website Beta, Part Five: Inpainting/Outpainting.

In the FAQs you will find a lot of other helpful sources to get started.

All in all, it’s cool to have a maintained/managed tool showing the potential of Stable Diffusion and easily experiment. If you like what you see, you could continue to pay or switch to Open Source.

As you have seen, the tool also has some limitations at the moment: the size of the images is limited to 1024x1024 (midjourney allows 2048x2048). The size is directly connected to the render speed.

But no worries about the size: one of the biggest sins in photo editing is also a relic of the past. Upscaling pictures is now possible with the help of other AI image upscaling tools. There are so many tools nowadays, maybe you find some inspiration in my other article “How AI Assistants Can Help You with Text and Art”.

You can also look for GIT repositories to size the images. One I could recommend is this txt2imghd. There's also a version on replicate.com.

Stable Diffusion on Huggingface.co

You must have come across hugging face if you got into text to image before. I shared some stuff from hugging face before, such as Dalle-mini, a free Dalle alternative from earlier this year. I really recommend anyone being interested in the topic of AI and machine learning, to visit hugging face every now and then. Under “spaces” you will find apps you could directly test as web-apps. And you notice (sorted by likes) that there’s also an app for Stable Diffusion.

In fact, there are more apps using Stable Diffusion to transform images. The one on the left is the simple text to image prompt and the right one could be used for modifying pictures as well. Feel free to try one of the two here:

Here are some of the outputs being generated in the tool:

And the fun part: It’s for free! There’s no limitation in terms of credits and it’s a perfect playground to get used to how text-to-image prompts work in general. As mentioned before when explaining midjourney or Dalle-2, I recommend reading through some text prompts… or as the magic of open source already unfolds, use sites like lexica.art and look at the eye candy and copy prompts.

As an alternative to lexica.art, you could also try arthub.ai.

Now I almost forgot to quickly explain what diffuse the rest could do for you… it’s kind of the beautifier of your paintings. Draw like in MS Paint and add a text prompt, about what your line drawing might be. In my case presented with an owl (owl got very popular in all this process of text-to-image generation) – the transformation of kid paintings is also quite fun 😉

Stable Diffusion on Google Colab

Google Colab is another resource you need to know when it comes to working with Python in the context of machine learning. Google Colab makes data science, deep learning, neural networks, and machine learning accessible to individual researchers who cannot afford expensive computing infrastructure.

Edan Meyer explains how to use google colab with his own modified colab notebook in his great video “Stable Diffusion – What, Why, How?”. He is explaining the whole process to create images, similar images or even videos. A deeper understanding of how using the code works help to unleash the full creative potential.

In very simple words, you open a colab notebook and follow from top to bottom. On the left you find some little play icons to execute. After executing one after another, there might be some text prompts or variable to modify to lead you to your result.

Local Apps and other Web GUIs for Stable Diffusion

Since we have the first open-source release, it feels the market is exploding. I saw downloadable apps to come and go within 2 weeks, so it’s hard to give you a list of all the options there are. Also, there's no guarantee they are still working. Many people are fascinated by these tools. Some want to make money quickly, others want to share, and everyone wants to keep improving the technology.

So, I start with another Web GUI for Stable Diffusion called peacasso:

And here’s a desktop app to run Stable Diffusion for the mac (m1,m2) called DiffusionBee:

Feel free to share more cool apps, sources in the comments. As of Mid September 2022 there were so many tools, it is hard to keep an overview.

Replicate is another interesting cloud API I wanted to give a try. If you use the tools you will notice, rendering takes time. Using a cloud API for rendering can improve energy efficiency and reduce your carbon footprint, while also decreasing rendering times.

Explore the offerings of recplicate.com, there’s a clean Stable Diffusion install as well as other interesting tools to help you to optimize your images or get a desired effect.

Stable Diffusion – local installation on MAC M1

Finally, some thoughts and links on installing Stable Diffusion locally on your computer. I personally only installed it on an M1 Macbook. With the advantage of the RAM being able to be shared.

There's a tutorial on options to work with Stable Diffusion locally I discovered while finishing the article, but it seems to be the official one (I didn't test it though as I didn't want to break my running version); the tutorial is from AssemblyAI itself: "How to run stable diffusion locally to generate images".

When I tested my local install, I basically followed along with another introduction from replicate.com: "Run Stable Diffusion on your M1 Mac's GPU".

In addition to the instructions, some extra tips, as I also had some trouble while installing. First, check for the correct Python version. You need 3.10, which could be downloaded and installed from python.org.

If you have trouble to switch the python version, there could be many reasons (I couldn't probably name). I used a version control, as options there were ASDF and pyenv. I decided on pyenv and followed the instructions and had no further issues to switch the version.

Another hint I could share with you... watch your folders. Always execute the prompts from the correct folder and be prepared to wait a little for your prompts to be executed.

Here are some renderings on sculptures from my local tests:

Stable Diffusion – local installation on PC

It took me a little while to test the PC installation, but I guess this guide here is quite good and straightforward:

AUTOMATIC1111's SD-WebUI (Windows)

Following the guide you have similar steps as on mac, but it is simpler in the execution as there are more installers to run. Finally you have a webGUI to get going.

Please note, that you need a GPU with at least 6 GB of VRAM. 10 GB or more on a NVIDIA GPU with CUDA is highly recommended to allow larger image sizes. A business card, such as the RTX A series is preferred as the energy consumption compared to the value is much better. Maybe the upcoming NVIDIA generation will bring even more value.

On the tutorial site you also get some other guides (I didn't test):

Stable Diffusion – other installs

• Stable Diffusion in TensorFlow / Keras

Creativity unleashed with Stable Diffusion

To get an idea about the full potential I collected some random examples from the past month. There are so many exciting opportunities. People create comics, movies or just play around.

Kali Yuga models

Kali Yuga provides some cool custom-trained models for Stable Diffusion. Here are some shots taken from Twitter:

You might have guessed it, there are also notebooks on google colab around to work with them. An alternative would be jupyter notebooks, but to keep it simpler, we stay with google colab.

• Notebook for #textilediffusion.
• Notebook for #Pulp-Sci-Fi-Diffusion
• Notebook for #lithography-diffusion
• Notebook for #watercolor-diffusion
• Notebook for #pixel-art-diffusion

Those notebooks are quite easy to handle. I used the #textilediffusion, to generate this sample piece here:

Click the images to jump to the examples. Due to data policy, I don't embed external sources.

Stable Diffusion and Music:

Stable Diffusion and 3D

Here's the GitHub Source for the Stable Diffusion blender texture generator.

Another link, that is connected to Unreal and midjourney is the generation of textures.

Stable Diffusion and Image Tools

Stable Diffusion Photoshop Plugin

There are also options for text-to-image in Canva and GIMP already and Nightcafé and artbreeder seem to have changed their models towards stable diffusion as well (you might remember that earlier this year they both used some GAN+CLIP solution, as mentioned in my article).

Stable Diffusion to generate color palettes from text descriptions

Other sources to mix with Stable Diffusion or to watch in general:

• Get3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
• OpenAI Whisper
• Google DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation (GitHub implementation for Stable Diffusion and Google DreamBooth and another Stable Diffusion Implementation with DreamBooth)

Final thoughts on Stable Diffusion AI – image jockeys on random turntables.

I love it. Going for the open-source path is so great to unleash new instruments of inspiration and with lucky punches, even to create art. Art is expressive and the artworks being created might get expressive. For me it's an enhancement of our daily practices being creative. I also don't think anyone must be afraid of losing jobs. For precise tasks it is still tricky to work with the tools.

The models are only as good as the inputs; handling the inputs is not the easiest practice and as generating the images takes some time… in terms of client work, the way using any generator might not be the fastest option to choose. As always, I assume clients want to negotiate cost as they think everything is going easier now. But currently the image generation at a professional level is still not precise enough. But the tools are already there to help. Google Dreambooth for example could help a lot to bring consistency into image series.

Looking at the development speed I assume the topic about precision will be a matter of the next years, as well as the topic about speed and custom training sets. I am also worried a bit about the footprint being generated from all the image generations (including all the learning upfront). In hugging face there's a note on the Stable Diffusion model card itself: 11250 kg CO2 eq for the learning (calculated with the machine learning emissions calculator).

The development in text-to-image is far from its boarders yet. But it's so impressive to see the speed of the community to adopt these technologies.

Currently the tools help to discover creativity to finally create a mix of outputs we have seen before. In the future I am sure we can use the techniques to do whole mood boards for clients and in some cases even final graphics or artworks for them.

The only art I’ll ever study is stuff that I can steal from.

David Bowie

Like the evolution of web bandwidth, I think we need some more speed in terms of image generation to open new options and create beautiful human-centered-artworks as one day maybe images could be generated in real time based on likings of visitors. I also assume, at the moment we reached a certain grade of automated artworks, more and more people will enjoy discovering real artworks with their hands again and value of the real original artworks will raise.

In general we shouldn't trust media without a critical perspective, especially when we see images. History showed us that people will abuse technology. So I hope for people to find value in real in-person events after Covid pandemic again. Digital could and will be a lot of fake.

The use of some kind of style transfers on Smartphones is very likely by next year. Dynamic Video generation is also on its way (see make-a-video from meta ai), so we have text-to-video generation and music, VR worlds and games are also around the corner, as Midjourney founder David Holz said. Due to open-source development, I won't say it takes 10 years. And also Google Colab and other services to run the software will change their business models to profit from this boom.

Advertising or Branding will also be affected. It will be an exciting journey in the next two or three years. Osmosis pretends to have some of the features that could be interesting for an early stage. Just imagine adding a target group to the prompt, an industry or a segment. For Learning there are many galleries out there, such as dribble, FWA and many more 🙂 it will be fun to see where this journey goes! (Saying this not being in an international network agency anymore).

As mentioned many times, the development speed is incredible. The current hot tool is google dreambooth (connected to Stable diffusion). Every month there might be another unexpected cool tool to pop up. Maybe even photoshop integrated the tools for their next release.

For me, design turned more into managing design over the years but still design is functional, and art is expressive. Currently it feels easier to create automated expressiveness in fantasy worlds, but nonetheless I also use AI tools during the creative process. The results of machine learning give you a condensed view of common thoughts. So, I often used the prompts to check for color ranges of certain clients, compared it, what google says and easily created something one could call the "wisdom of the net" result.

Our traces on the net and our databases of artists created the look we could mix now. We are the image jockeys of generations that are already there. Unfortunately there's still a lot of randomness in the result to utilize it more professionally.

So long, enjoy and go out and play to discover the fun playing with those great tools!

Recommended readings

1. Understand latent diffusion models
2. Get a full introduction to Stable Diffusion

Maybe you remember my article speaking computer language – welcome AI art, speaking about the new role of text prompts and different image AI assistants. I focused on a general overview on GANs, DALL-E and DALL-E Mini, as an access to DALL-E 2 wasn’t available yet for me. Also, Google Imagen being mentioned in the article has no free option yet… but with only one and a half month passing since my last article, Google also introduced another tool, called Parti, a tool being said to produce the best results of any AI art generators so far and Stable Diffusion is also there to launch its service soon.

In this article I want to share my first hands on experience with Midjourney and DALL-E 2 with you. With midjourney moving to open beta and starting their v3 image generation algorithm on the 25th of July, another player pushed to the market… and I guess one or two days later, finally my invite for DALL-E 2 arrived.

At that day I was happy at first, but when I noticed another day later, that another E-Mail I used for signing for DALL-E 2 only a few months ago, despite the first one, being used for application far more than half a year ago, I was a bit disappointed. I thought it’s all connected to their new pricing model and the beta start of midjourney and I got disappointed a bit. The exclusiveness was gone and “mass-market” arrived.

In general the market on AI Art bots seems to push a lot. So for example DALL-E mini, the free tool I mentioned in my other article, is now called craiyon. There are also other tools Worth being mentioned in the context of AI art tools, such as Latitude Voyage (of course in beta) or Apples GAUDI, a prompt tool for indoor 3D scenes.

Luckily I heard the U.S. Copyright office said A.I. Art can't be copyrighted, so I hope no war will heat up this exiting market in the near future. So far I only know that there are patents on algorithms being used. Hopefully the market will show even more diversity in the future.

Midjourney beta – first impressions

Let us first look at our new player. Midjourney. The first impression of midjourney is being geekier and less serious than DALL-E 2. It’s a more playful entrance to a world everyone could test right away. After joining the beta, you will be redirected to Discord, no waiting list, no exclusive clubbing.

Before we investigate discord, a quick explanation of who Midjourney is. Midjourney is an independent research lab exploring new mediums of thought and expanding the imaginative powers of the human species.

IN 10 YEARS, YOU’LL BE ABLE TO BUY AN XBOX WITH A GIANT AI PROCESSOR, AND ALL THE GAMES ARE DREAMS.

David Holz in an Interview with the verge

David Holz, the founder of midjourney, did some other geeky, funny things in the past. He used to be co-founder of leap motion, a great exploration of the power of gestures. Midjourney is a small company right now with around 10 people working there.

Exploring the background of midjourney and all their way, how they speak to their audience, in other words their tonality feels great to me. It has the explorative spirit and playfulness I always try to teach students to get started in digital.

Signing up for a free account, you get 25 credits, these can be used to generate images within the discord channel. For anyone being interested, it is even possible to run midjourney on ones own discord server.

After that, you’ll have to pay — either $10 or $30 a month. The pricing is connected to the number of images you want and the privacy you expect. And if you are a huge company with more than 1 billion revenue a year, you get the corporate membership for 600 US dollar a year.

So how does it work?

If you never used discord before, see it as a chat tool, kind of a slack for the masses, with a non-business focus.

So, in the channel you will find all the information there are in general, on announcements, updates and so on. If feels like you as beta tester are part of the team, and even the boss is there, and you could influence the development process. After all, that’s what a beta program is about.

In terms of my professional perspective being more experienced with product and user testing, this is a great way for that kind of product, as the product itself is being used in the same environment.

So here are the steps to create images:

1. Look for a newbies Channel and enter it (in the picture you see newbies-29, newbies-59 and newbies-89)
2. If you have a picture in mind, about what you want to see, start typing “/imagine”, followed by whatever you want to see.
3. Within 60 seconds you should see 4 images.
4. With the 4 images being generated, you could upscale the image or create variations. Therefore, there are U1 to U4 (für upscaling) and V1 to V4 (for version) below the images being generated. Upscaling means that the images scale up to around 1664x1664 Pixels
5. Save the result, click on the generated image (to enlarge) and with the right mouse button being pressed, select save image to your local computer.
6. Finally, you could also ask the bot to send the results to your Direct messages, by reacting with the envelope emoji.

This is enough to create visual beauty.

If you need inspiration, look in midjourney ressource list, there are great links:

The artist visual style encyclopedia – if the style is part of your text prompt it will take effect – such as cyberpunk city, by Franz Marc:

There’s also a prompt builder to guide your inputs and see which variables are available:

And a collection on  “prompt styles and weights”.

All in all, the learning curve, if you are used to discord and used a text prompt before is very flat. Only if you are new to discord and text prompts, you need more orientation.

After all I could say, I loved it. I love the playfulness of the whole app, the results and all the “social” aspects, seeing what all others are building, getting notifications about own results and the whole way it is being setup. It is great fun to try and the documentation is very helpful.

DALL-E 2 beta – first impressions

With the magic of the new and little frustration about the almost open invites, I waited to test my DALL-E 2 invites after I tested midjourney.

As mentioned, entering DALL-E app is a completely different experience. It is you and the text prompt, not the hectic news stream and the dynamics you experience in midjourney. Based on the interface/overall app decision, one could say, that DALL-E 2 wants you to reflect about what you type, while midjourney is a competing social feed, about who creates the coolest results.

The cost system at DALL-E 2 is based on credits. You get 50 credits for the first month for free and 15 credits every month after. With only 15$ you could buy additional credit packs worth 115 credits, which equals around 460 images. OpenAI allows commercial use of DALL-E images. Compared to the image stock market, the costs of the images will be fairly low, but one must be award that you have to get used to the text prompts and the corresponding structure of the input.

That being said let’s try to get a result for our cyberpunk city, by Franz Marc.

In the case of DALL-E, I couldn’t find a quick hands on, official documentation; so, I tried the sentence as following: A cyberpunk city in the style of Franz Marc, as the example being shown by DALL-E on their homepage also used the "in the style of" syntax.

My result on that one was quite… disappointing. I don’t know why DALL-E missed the style of Franz Marc that much:

To investigate further, I needed a prompt guide for DALL-E 2. Googling for some help, you find

1. The DALL-E 2 prompt book – my recommended choice
2. OpenAI DALL-E 2 prompt guide

Going through the prompt options, you notice, that there are more parameters than on midjourney. Lightning, Camera position/angles, genres, usage contexts and many more.

So, I gave it another try, changing the prompt a bit leading to a much better result in terms of the style:

Unfortunately, my tries with real artists didn’t work out so good with DALL-E 2 at the start, so I had another try…

My result from midjourney:

And the same prompt from DALL-E 2:

Getting used to the prompt dialogue on DALL-E increases the results a lot:

Using the right prompts is the essential part to get the best results. If your prompt is done right, it seems DALL-E is more precise, though midjourney is so much fun to use.

For DALL-E there's even a paid service being launched called promptbase to help you with using the right prompts:

If you want to discover more, I recommend searching the web yourself. On twitter I found some comparisons, having a similar impression I do:

Resume

Is it possible to do a final resume on the tools? I guess no. On the one hand I am not experienced enough with both tools, on the other hand I think both tools are great and give us a glimpse into the future. The documentation on midjourney and the entry barrier is lower, so to get started, I would recommend midjourney in any case. The more you are aiming for precise results (as an alternative to stock images), the more it feels DALL-E 2 is the way to go.

Will be exiting what the next two month offer in terms of AI generation tools. 2022 is very exiting so far in terms of AI art generators and it will be exiting to see what's next.

What's next? Update mid August – dreamstudio arrived

As mentioned, the speed new, similar offerings are popping up on the market is impressive. It seems, AI in combination with images is going mainstream. Another tool to check out is dreamstudio. Access is quite easy as well and the results are also impressive. Also the editor looks promising, see my screenshot below. After my family holiday I might try a daily use check of all those great tools.

Recommended readings

1. The AI that creates any picture you want, explained – The process behind DALL-E 2 (and others) explained
2. What AI means for human artists – great collection about views from the artists/designer perspective
3. Making AI Art with Midjourney – great article with examples and explanations
4. The Creativity of Text-based Generative Art – paper on text-based generative art
5. The trouble with DALL-E – Five thoughts on the use of artificial intelligence to accelerate the digital art market
6. My follow up article on Stable Diffusion: Stable Diffusion celebrates new forms of creativity

The world of art is a diverse and beautiful place. The more we explore new forms of art, the more we are introduced to a new way of looking at the world. Art is one of the oldest forms of human expression and creativity and one of the newest forms of art is generative art or AI art or NFT art we could see at a large, popular scale in the shape of NFT galleries. AI art is an art form that uses artificial intelligence and robotics to create art. This form of art is exciting because it is a new and fresh way of seeing the world.

As I am fascinated by AI artists such as Refik Anadol, Mario Klingemann, Sougwen Chung, Memo Akten and many others, I wanted to share some ways to get started with Art in the context of AI. As Art using artificial intelligence is more complex and involves GANs, Machine Learning and other more advanced concepts, my focus on this article is the entry to AI Arts in general. In my point of view generative Art or Design is the best way to start in this area.

I also shared some more articles on the topic.

• If you want to know more about GANs, DALL-E, CLIP and other approaches of deep learning, follow this link.
• If you want to know more about my fascination for Refik Anadol, or if you don't know his work, follow this link.

I love generative worlds surrounding us. While teaching, I like to share some basic concepts about generative design, in terms of randomness or the magic of patterns. It’s so great to influence the idea generation process with the help of the computers. Something I also recommended in the use of AI tools in general. Besides giving a recommendation for the beautiful designed book Generative Coding on the web by Benedikt Gross et all, I want to share some ways and tools to get started in your own creative process.

For this article, I won’t discuss differences in AI art or generative Art as being generative is an underlying pattern for both. If we speak about generative, we speak about autonomous processes that are not under direct control of humans. To get started with AI in art or generative art, you need a basic understanding of accepting your role in the process.

You are not the worker to paint all the time, you are a conductor in paving the way for a desired outcome. So finally, everyone is a director to choose from directions computers generate for them. If you finally reached some degree of mastery, the process of creation will be more straightforward, but in the beginning, it is very explorative.

I also don’t separate between generative design and generative art. Art is expressive, as John Maeda once explained in his Bermuda quadrilateral, whether designs task is to communicate. (There are newer models on this one, too, therefore I recommend looking at Neri Oxmans Krebs Cycle of Creativity).

Generative art saw its first influences at the end of the 19^th century. Playing with geometry or vantage points and the rise of technology and machines paved the way for generative art. Besides visual art, there are many other ways to explore the generative power as well.

Ai art and Literature

Philip M. Parker is maybe one of the most extreme examples on generative literature or AI in literature. He “wrote” or generated more than 1 million books already. He developed some algorithms capable of automatically generating textbooks, poems, crossword puzzles or other books, mostly targeted to the so called longtail, you might have heard in the context of SEO before. So he is targeting specific, niche markets. He’s maybe my proof point of my fear of AI text writing tools, as a lot of crap could be produced and Search engines fail to display the best quality results, see my issue with the myth of the goldfish attention span in my other article.

Ai art and Music

The historic roots in generative music are even older as for generative visual arts. At the end of the 18th century, the “Musikalisches Würfelspiel“ (musical dice game) became a popular way of passing the time. Known names, such as Wolfgang Amadeus Mozart played the game being introduced by Johann Philipp Kirnberger. Back in our time, the most known figure on generative music is Brian Eno. And there are a lot of beautiful tutorials on the web on how to get started with generative music (I have those two still on my list to review):

And here are some generative music tools to explore:

• Alda - a text-based programming language for music composition.
• SuperCollider - a plattform for audio synthesis and algorithmic composition.
• Csound - A sound and music computing system.
• Sonic Pi - The live coding music synth for everyone.

So far, I am not experienced with any of the mentioned, so I can’t recommend any of the mentioned tools, but based on my research all of them a worth having a look at.

Architecture

In the case of architecture, I would prefer to speak about generative design in architecture. In terms of the production process, you could imagine that 3D printing in every scale will lead to new results. Like 3D printing these bots could also be used to brick or other construction related items. To get an idea about the status of technology, visit Autodesk.

For the design process in architecture, we see different degrees of generativeness, we have parametricism and generative design, as described earlier in this article. The most famous designer to apply generative principles is Zaha Hadid Architects.

To find out more, here’s an article on how architecture firms are using generative design.

Look at Eric Mesplé using ferrofluid to create art… so doesn’t it also feel generated? Maybe there’s a generative part to it, but not from computers, but from nature. And sometimes it's good not to fit in any category.

Programming languages – Javascript, Processing

The “classical” way nowadays to get started with generative Art is learning Processing. A programming language specifically designed for creatives. If you investigate the history of the programming language, you will find know names, such as Casey Reas and Ben Fry. They wanted to create a language that makes it easier for creatives to explore programming as a tool for making art.

Processing is available since 2001 and helped many creatives in the world to unfold their creativity by getting easier access to a programming language. Under the processing foundation, there’s also another library available, that got introduced in 2013: p5.js.

P5.js is a library for the programming language Javascript. In P5.js and processing we call a program a sketch. Like our sketching skills on paper, we draw programs with simplified lines of code. I used p5.js to create most of my post pictures and on the homepage and on my writing overview, I have scripts to generate the visuals. Especially on the homepage, each refresh of the page will give you a unique result.

A few lines of javscript code in p5.js could generate the generative look of the two images – the original visual study was done in 1974 by French media artist Vera Molnar.

function setup() 
{
createCanvas(500, 500);
noFill();
rectMode(CENTER);
noLoop();
}
function draw() {
background(220);
for (var y = 50; y < 500; y = y + 100) 
{
for (var x = 50; x < 500; x = x + 100) 
{
var r = 2;
for (var w = 100; w > 0; w = w - r) {
rect(x, y, w, w);
r = random(2, 20);
}
}
}
}

Is any of the two mentioned language difficult to learn? If you read some articles from me, you know I am not a person about absolute feedbacks, and again it is a “depends”. I have some experience teaching generative programming skills for undergraduate designers – the ones being more playful in their way of approaching new items succeed better than others who take it too serious at the start. While the ones being structured, wanting to dig to the core, are the ones who will stay with it on the long run (if they also discover the fun, playfulness part of it).

Even fewer lines of code are needed to generate visual results like the following:

function setup() {
createCanvas(450, 400)
background(255)
}

function draw() {
stroke(0)
line(200, 200, mouseX, mouseY)
ellipse(200,200,mouseY)
}

Programming in general is learning a language, a syntax, some grammar… and I wouldn’t recommend learning programming like a language. Learn it as a process of discovery, of fun playing around. If you got your first confidence, aim for a low-fidelity project you want to realize, don’t aim for big targets or you will get lost directly.

I think p5.js or processing are both great ways to get going. The easy part about both is that there’s an online editor you can use and open sketches from others, right from the start. So looking into my sketch library, you will also find many “copies” of others or modified version from other people.

To get started, I recommend the book Generative Design: Visualize, Program, and Create with JavaScript in p5.js or all the code samples of the book to get started from their website.

3D engines

Mostly I grab students’ attention, when I tell them that everywhere who uses randomness produces generative work. As per definition, as soon as some of our human control gets handed over to the computer, it is generative. And this generativeness is present in almost all the software we have today. You could even use randomness in photoshop.

Speaking about 3D and real-time engines, we speak about the big players out there, Cinema4D, 3DS Max, Maya, Lightwave and Blender, the main known 3D graphics Engines on the market. All of those enable you to be generative and you will find a lot of tutorials there. Those are the main programs being used to create the visual 3D look of designed elements around us.

Also, for specific applications, such as architecture, you will find applications of generative design, so for example the combination of rhino 3D and grasshopper also allows generativeness in architecture.

Overall, there are far more resources on blender specific tutorials. Such as two great articles on medium I could recommend:

• Creative Coding in Blender 2.92: A Primer
• Blender 2.8 Grease Pencil Scripting and Generative Art

Besides the mentioned 3D graphic programs, there are also Real-time engines. Unity and Unreal are the ones, that are quite popular on a wider scale.

There are also tutorials for all of those producing real-time images; here’s one on unity – how Carl Emil Carlsen creates generative art with Unity

As I already mentioned in the article Deconstructing AI Art masterpieces: Create Art like Refik Anadol, in the art scene the tools vvvv, openframeworks, cinder or touchdesigner are quite popular and both could be used to strive for looks and installations we see from contemporary AI Artists.

Besides the mentioned tools, there are many more out there, worth having a look. In the realm of programs, there’s for example Houdini, and if you explore the world of javascript programming language, there are great libraries available as well: three.js has a great user base and awesome tutorials to get started.

Recommendations to get started with AI Art, Generative Art, Generative Design or Creative Coding

If you looked at some of the links, you might have noticed, that mostly code is involved. To be perfectly honest, I don’t believe in sticking with generative design or art, if you won’t touch code. To get started, I recommend starting with p5.js, as it is related to javascript syntax and script syntax is quite useful as an entry point. If you never looked into programming but learned some languages before, we have similarities, also in programming we have different kind of languages that have a similar grammar and syntax. So script languages is a language family cluster, in the case of javascript, it allows you to extend your skillset with a lot of other contemporary libraries and adapt to other script languages easier.

But be aware, learning programming is a non-linear process and more a marathon than a sprint. The threshold concept is a good description for this way up some stairs, discovering new perspectives with every climb.

Everyone in this country should learn how to program because it teaches you how to think.

Steve Jobs

The other thing I would strongly recommend is looking at touchdesigner. There’s a free option for the program and there are so many awesome tutorials out there. Another aspect, why I would recommend it is, it is a node-based system, which means you place modules on a surface and connect them and modify values of the modules.

Getting used to node-based systems is highly recommended as more and more programs introduce node-based options and it allows some playfulness in the early exploration quite easily. Even blender has some node options now.

Another great thing about touchdesigner is the underlying programming language there. It's python. On the road to mastering Art with the help of artificial intelligence, Pyhton is one of the broadest used programming languages in the context of machine learning, so you could connect touchdesigner in a better way to work with underlying data sets or scrape data from the web.

As I spoke about node-based interfaces, I would also recommend cables.gl. To some extent, as it is on my personal list for a more in-depth exploration, and it is also quite accessible for non-coders. I also like that they are still in beta but have a quite good base of users with high quality examples. Another one you might have heard of is nodes.io – your thinking space for exploring ideas with code, also node-based 🙂

And as a final statement, though I really learned to hate that program before… which, in my case means I have a biased opinion since I studied almost 20 years ago and started my experience with 3D with 3DS Max and didn’t like my first-time switch… go for blender if you want to explore the world of 3D. But if you never used 3D software before, set clear goals of what you want to achieve.

There are quite some concepts one must get used to using 3D software. I was never so happy about designing 3D stuff at all. I guess I had most fun modelling some game assets for some student work, a long time ago… but rendering for days to see it was crab, wouldn’t relate to my impatient nature at that time. Now having two kids, I learned to be more patient and maybe I restart in 3D, in blender again.

Also, unity is quite accessible, but more suitable for games, apps and effects. I gave it some tries and quite liked it. Both blender and unity provide a great userbase and a lot of plugins, tutorials, and examples to learn from.

If it’s more about the visual generative part, go with touchdesigner. I was quite impressed by the result my recent course could generate by learning the program themselves. I will add some recommended tutorial links on my recommendations within the next weeks. Enjoy!

Easy entries to programming

1. The coding train – great examples on learning p5.js, machine learning and fun entertainment watching Daniel Shiffman, kind of a Bod Ross in more fun on programming.

Getting started with touchdesigner

1. Introduction to TouchDesigner – by Ben Voigt – a general overview of what the tool could do for you
2. TouchDesigner Beginner Course – by bileam tschepe (elekktronaut) – great tutorials for starters from a self-learner
3. alltd.org – the biggest resource and collection on touchdesigner tutorials
4. Derivative tutorial list
5. Tutorial list from The Interactive & Immersive HQ on Youtube
6. Noto the talking ball – interesting, more advanced tutorial collection
7. Exsstas – interesting, more advanced tutorial collection
8. paketa12 – interesting, more advanced tutorial collection
9. Akenbank – interesting, more advanced tutorial collection
10. The interactive & Immersive HQ – interesting, more advanced tutorial collection
11. plyzitron – some visual inspiring stuff with the option to buy sources via gumroad.
12. Matthew Ragan – great ressource collection on teaching touchdesigner
13. musichackspace – paid courses

Getting started with cables.gl

1. Cables documentation
2. Cables YouTube Channel
3. Introduction to cables.gl – build a data-driven gradient by Kirell Benzi

Books

1. Form+Code in Design , Art, and Architecture (2010) –  by Casey Reas
2. Generative Design: Visualize, Program, and Create with JavaScript in p5.js (2018) – Benedit Gross

Old school resources

1. Visual complexity – old school collection of examples for visual complexity

Have you ever wished you had a personal assistant to help you with your writing or with your artworks?

AI Assistants are becoming more popular and advanced each day. Back in my article on speaking computer language, I gave a short introduction to GANs, DALL·E or Google IMAGEN, text prompts to tell AI what you want to see. In fact, I never thought about AI writer before that article.

Recently I was on Appsumo and realized how many offerings there are on AI writers. There's contentforge, neuron writer, wordhero, rytr, whitecream, creaitor.ai and so many more. There are far more offerings for doing the copywriting part, than to create images. In fact, in terms of images, we mostly have "art" generators, AI graphics are not available on broader scale.

So, for writing this article, I used some of the mentioned tools, to generate parts of my content for me. In general, I must admit, that I optimize my contents for SEO, but I don't do it to drive only traffic. I write my articles because these topics relate with me. I was a bit shocked when I used one generator (whitecream) to write me a complete blog article.

The quality was good. It was one of those articles, that are great for Search engines. It used emotional words, had a good text structure in terms of SEO and all in all, it was a good job from a SEO perspective. From the Content perspective, it was too much on the surface. So I realized the immediate downside of this tools... search is having an even bigger problem, I already mentioned in my article on being trapped by search, that the quality of the content is not checked and why wrong “myths” circle in top results on google and others.

The created contents are good, but the quality of content on the web in general is endangered a lot, if everyone only "writes" to get Search traffic. Therefore, I support the Indieweb and in my point of view, it is more important than ever before to change search algorithms towards an “authority” to people behind the article and their field of competence.

The role of AI writers in the future of copywriting

With AI assistants, copywriters can put their skills to use in a more efficient way. They can make sure that they are not wasting time on skillsets that they don't have and instead focus on what they are best at - creativity, emotions, and research.

We should not think of these AI writers as a replacement for human copywriters. They just aid the content writers by getting rid of writer’s block and generating content ideas at scale.

AI assistants for writing are increasingly getting popular in the workplace. Some companies use them when they need to generate content for a specific topic or niche. While digital agencies use them to generate all kinds of content for their clients.

Believe it, or not, these three paragraphs were also written from an AI writer. You notice the text tends to advertise for AI writers. But this tone of voice, the writer got, was good, even for me, reflecting my thoughts on the AI writers in general. Similar to what I said about Art tools, Copywriting tools could be great addition to streamline your workflow.

Just like any tools, it has its limits. But if we use it in the right way, and with common sense, AI writers could be great writing assistants that help us to get over the writer’s block or come up with ideas at scale.

To give you an idea, about my used tools in this article: I used contentforge as the holder of my article, then I used rytr to write the blog section, mentioned above. In contentforge I used the "generate content" button from time to time, to see what happens.

Some of the generated text is great, and I can see myself using this type of tool in the future for smaller tasks or ideas generation. For longer articles or books, I absolutely believe that a human writer is needed to create a compelling story.

(so, this paragraph before was added "generatively" as an add on to the article I just wrote 🙂 quite funny)

But yes, these tools could be a great support. I am more worried about all the people using them for just pushing their SEO stuff instead of writing contents reflecting their believes, experiences or research.

When it comes to research and real articles, they don't help you to create the real content. But they could be handy to assist some SEO parts, such as a summary, some headlines in between and so on. To do a deep review, I miss the experience.

It is likely that all these tools make use of deep learning and GPT-3, a generative pre-trained transformer. So in my special case, not being a native speaker, tools (also grammarly, jasper or others) could help to improve my writing in terms of grammar or the words being used in general. The other bigger help I see in the area of SEO, if you want your content to be fully optimized for SEO. So it's kind of a polisher for your contents and an ideation source as well.

If you wonder, why I don't speak about chatbots in the context of text,... cause I think they are an own category, there are also many tools around, most act in a conversational way and are trained by machine learning principles. Here's a good entry/overview.

AI assistants in graphic design

I already mentioned some tools to create so called "AI art" in my article "Speaking computer language – welcome AI Art". I guess I also mentioned some logo generators out there already, maybe I also shared a link to Nikolay Ironov, a designer and neural network.

To wrap it up, I clustered some of the tools in some categories here:

AI generated images

For generating images at a general scale, we could include all the tools being mentioned in my other article on AI art before. DALL-E (1,2, mini) or Imagen and all the Ganbreeders such as artbreeder, nightcafe, wombo.art are some of the AI assistants for graphic design, to name a few.

But there are also tools to create people that operate with AI, like the incredible Metahuman builder from the new unreal 5 engine, there are AI driven ways to create people images. Generated photos is such as platform. There’s also a face generator available to create an image. Of course, you could get quick results, that look like fake quickly, but there are also images, that look quite promising;

Similar to generated photos, you could create AI actors with notaperson.ai. In my point of view the characters sill fell a bit static, but the powers of creating random people is there. If you don’t believe, watch the random results on thispersondoesnotexist.com. With the help of StyleGAN2 a lot of images are generated in a GAN way. The world of fakes is real, and besides images, also speech could get modified easily.

If you don’t like real pictures, look at thiswaifudoesnotexist.net. (A waifu is a fictive female Japanese manga/anime character).

AI generated logos

There are also a bunch of logo generators out there. Nikolay Ironov offers one and on logoai.com, you could quickly generate your own logos with some guided dialogue. A similar builder could be experienced here as a service from shopify or at logomaster.ai.

You might think, how scarry this is? No, it’s not. Honestly, I included AI tools in my exploration on the branding of a client already. Of course, branding is more than a logo, as it’s about the identity of the brand. Each personality has different characteristics that need application to the branding guidelines.

But now let’s assume you type in a DALLE Mini interface the attributes of a brand or a short description and look at the result; now compare it to some other text-input based generator. Won’t you think you get some kind of color scheme on that?

At least for my last branding project, I included AI Tools in my research. I got some straightforward color schemes I could extract and that were helpful. I didn’t use one of the mentioned logo generators though, but they could also be helpful in the exploration process as they help to narrow down the chosen path we want to follow.

In my case the brand itself was connected to AI and Edge Computing, so there was a brand fit with the tools being used. At that moment it felt right to ask the question, how an AI might see the company, reminding me of a nice piece of art by Christian Mio Loclair called Narciss, a a robot, built to analyze its own physical embodiment. Looking at itself and reflecting, what it sees; so why not use such tools, if they match to the brand?

AI drawing

I am not too experienced with AI drawing tools and so I only share the one I discovered some years ago. Autodraw. A Tool to help you create icons. It gives you proposals based on the shape you were drawing. So, in my case it recognized my fish shape and gives me proposals about other fish related shapes at the top.

Another fun application in a similar direction is https://quickdraw.withgoogle.com/. Challenge your Mouse drawing skills with form recognition.

Other AI tools for images

Finally, another tool you could give a test is an deep learning based image upscaler.

Resume

Is there a final wrap-up on my mentioned AI tools at all? Yes, there is... we are getting more and more tools that could distract and being applied wrong lead to bad results. Whether it be a world full of fakes (in terms of visual, text, voice) leading to less trust in digital environments or a tendency we often see when tools streamline processes. A lot of trash gets produced.

The danger of expecting magic with the tools is also there. These are tools that could assist, but they don’t define the plan, what you are aiming for. So, define the strategy upfront and evaluate afterwards. A sentence that sounds so true since 20 years and more being in digital for almost anything, whether it be creating websites, apps, designing things or any business objectives. Tools can’t replace goals and a strategy. AI assistants or tools are in this case not different from any other form of tools.

The perfect AI tool must integrate to your own process. In my case I see most potential in the exploration stage. As most creative processes are similar and alike, I refer to an easy system, such as human-centered design. We speak about inspiration, ideation, and Implementation. During these stages of the creative process,

AI could help you exploring your thoughts, flipping the problem and many more things that help you to get going. All the mentioned “methods” would be placed within the stage of inspiration. Also, in terms of a reframing of the topic, AI could help, as it is another perspective on the problem – often from a “wisdom of the crowds” perspective, as the trained algorithms are trained by the masses.

During the Implementation I see AI as an add-on to evaluate own results, in terms of writing to check the grammar, in terms of design to check for uniqueness or testing.

So finally, I could say, I believe in a lot of the automations being possible with AI, but like UX-Design, that got streamlined over the years with bet applied patterns (and corresponding dark patterns), AI-tools will lead to the same effect, some rebound effects: Imagine a lot of sites being written by AI and google ranks being strongly influenced by AI. The overall quality of the content will drop, and a search won’t be user friendly anymore. Hopefully AI results will get recognized and filtered.

Or on the graphic side, customers think of using AI builders for logos and more website design from finished templates. This will lead to less identity, uniqueness on the web and meaningless branding as all look the same. Uniqueness will be the struggle of the future as it is the struggle for all streamlined processes.

I recommend anyone being interested, to have a look at the tools yourself. There are enough tools with a free version to play around. Have fun and use the tools for good!

The command prompt: A black, empty screen and a blinking line, waiting for my input. Could you imagine that this prompt is the access to art or AI art? The command line shows a visual result of your terms, not ASCII art or related items. Nowadays, those prompts have different shapes, such as a form field or the use of voice interfaces.

Your imagination of thought, translated to words in your natural language (mostly in English), gives you access to new creativity. In the words of advertising, you need to express a good core idea, you want to get visualized. Similar to creatives being used to Image search engines, the need to find the correct words gets more and more important.

To explore my thoughts I look at the evolution of the way we tell machines, what to do and how CLIP and other neural models transform the way we could create what we are looking for. Whether it be AI art or maybe any picture we want to see, in general or what I would call AI graphics.

Computer language

A computer language is a language we use to talk to machines. It has been in use for about 70 years.In terms of human history, a very short time.

The first, more popular programming languages found their roots between 1960 and 1970. The first “high-level programming language” was Plankalkül from around 1945. Even earlier, the roots of programming go back to Ada Lovelace, yes, a woman, in the year 1883.

Nowadays, we even have low code or no-code ways to speak with machines. Mostly those use visual environments to drag together code components or nodes and are called visual programming languages.

So, in the past, we needed to write code in a language, following grammar. Now, we could also drag together many nodes and hope for an effect, while playing with parameters. At some point of customization those visual environments also allow to add code manually. But it's not essentially necessary to write the code yourself anymore.

DALL·E and other neural networks allow access to Artificial intelligence as a source of processing your thoughts via a text prompt. If you want, you could say that you tell the machine, what it should program with your natural language. Your chosen words define the quality of the output you are aiming for.

Natural language and natural interfaces

The natural language is the way we as humans speak with each other. To some extent even our voice-interfaces around us are now able to communicate with us in a way that feels almost natural. In this case our command prompt mostly starts with “Alexa”, “Siri” or “Hey Google” to initialize the dialogue. Natural language processing takes the task of converting speech into words in the background to take the right commands.

In 2013, there was a trend to use natural language forms in interfaces to make form usage feel more natural. Forms were integrated into sentences rather than presented as a separate option. Stats are more usable in natural language, but confusing numbers make it difficult to understand the impact of improvements.

The tendency to make interfaces more natural or more human, so we could use our natural senses to interact, is present. But the evolution of the interfaces also depends on technological developments.

There are many interesting developments, especially if you investigate the MIT Media lab and the tangible media group. They strive for seamlessly coupling the worlds of bits and atoms by giving dynamic physical form to digital information and computation. Research on venous materials and human-flora interaction provides insights into our future.

One of the easiest ways to interact now is our language, expressed by voice interfaces. When using Neuralink or similar chips inside our bodies, it's likely that we'll need to use interfaces with a well-structured language initially.

If you are looking for more scientific terms on how the language is being used for generating visual results, look for text-to-image synthesis.

GANs produce visual effects resembling the WebGL example on alteredqualia.com that utilizes DeepWarp, a method commonly employed for gaze movement.

CLIP, GANs or AI driven Art and approaches of deep learning.

Natural language processing is also an important tool when it comes to playing with so called CLIPs. CLIP is “Contrast Language-Image Pre-Training”, a neural network trained on a variety of text-image pairs. CLIP got introduced around January 2021.

Before getting into CLIP, a short excursion on learning. Deep learning is part of machine learning and helps machines to learn. The machine is supposed to increase its own skills without the help of humans. On contrast to machine learning, where humans interfere in the learning process, in deep learning humans take care of providing information for learning. To do so, we could also use our input prompt. The process of analysis is up to the machine itself.

An example for machine learning would be connecting words and images by labelling the images with the appropriate word. A typical combination for this cause would be ImageNet and WordNet.

CLIP is a neural network model that is already trained on 400,000,000 (image, text) pairs. So, this means that this is the data that is used in training the CLIP Model. You can input an image to the model and get the most probable caption or summary of it. Alternatively, you can use natural language as an input. The true magic comes when you see examples. Travis Hoppe shares some interesting applications in his Twitter account.

To get a full “beginners” understanding of the CLIP model, I recommend this article: “A beginner’s guide to the CLIP Model”. You will find many interesting articles on kdnuggets.com around artificial intelligence, computer vision, data science, machine learning, natural language processing or programming (Python).

To try CLIP yourself, you could use VQGAN developed by the University of Heidelberg. It’s a GAN (a generative adversarial network), those are special types of neural networks consisting of “Generator” and “discriminator” networks. There is a GAN that generates images. The generator is like an artist and the discriminator is like a critic during training. When the generator can produce an image that the discriminator can't distinguish from a human-made one, we can use it for creating art.

There are huge libraries of available GANs for very different purposes. Some of the lists I could recommend:

The GAN Zoo

GANS awesome applications

To play with the technology to learn, use artbreeder and generate new images; Artbreeder is also based on GAN based image generation.

Exploring Applications of CLIP

How to get started? I would always recommend playing with the technology. Most technology is too complex for an early deep dive, so start by playing. If you found your motivation to dig deeper, do it with all your might as you need a lot of patience to get a good understanding.

So, let’s look at VQGAN + CLIP you could use directly yourself online. This is a collab notebook, a small development environment running Python programming language. Your task is to input some values... To get started, I would recommend to only change the “natural language prompt” of what you want to see. In my case I tested some results of different apps with the sentence “Children playing in the forest” (because my kids are attending a kindergarten in the wood while writing these lines).

These are my first results using the CLIP linked above (to try yourself, open the link, basically go through the page and always execute the play button to the left and wait till there’s a check icon next to it, to signal it’s loaded), only for chapter “Parameters” enter a text. In the end you could also create a video. The results here are early iterations. The program would continue till the “free” volume is used up.

To compare my result, I checked for other applications out there and their result. So there’s for example an application that is also available as an app; In wombo.art you could test directly. The result from wombo art had similar patterns as my test with YQGAN and looked like the picture on the at the top left.

Then I used another popular web-service to compare: Nightcafe you can see on the right.

You see, when using a webservice, you end up getting one result. Using a google colab, you get many iterations and it's not as fast as the webservice or app. (Google Colab or Colaboratory is a way to execute Pyhton code in the browser).

There are also other CLIPS on google colab you could use. The following picture is the (an early iterative) result of the combination VQGAN+CLIP (z+quantize method):

You could continue playing with many more version out there. Here you will find an overview of newer applications: https://softologyblog.wordpress.com/

You notice that the basic style follows one pattern and it’s obvious nightcafe also uses some way of VQGAN combination. To create magic, or what I would call magic based on a text prompt, there’s still some way to go.

The magic I aim for: I tell the machine what I want to see, and they give me iterations of the thing I type in in. It mustn't be art or AI art.

To create this magic, we look at DALL·E. OpenAI released DALL·E 2 in April 2022. They show the real magic of the future. By the way, I wouldn't call the results art. Mostly we generate graphics.

A short introduction to DALL·E, DALL·E 2 and DALL·E mini

“DALL·E 2 is an artificial intelligence model that takes a text prompt and/or existing image as an input and generates a new image as an output. DALL·E 2 was developed by researchers at OpenAI to understand the capabilities and broader implications of multimodal generative models. To help us and others better understand how image generation models can be used and misused.”
(From their own description)

To make it shot, look at their website and get a feeling of the beauty that could be generated with DALL·E 2 or watch the short video on youtube:

Unfortunately, DALL·E 2 is not easily available to test yet. But you could sign up on their waiting list.

According to OpenAI DALL·E 2 isn't trained on faces, to avoid the risk of deep fakes. The danger is already there. Finally it the time to doubt any images being shown to us!

On twitter try a search for #dalle or #dalle2 and you find many inspiring pictures. Going through the magical results, it feels as if image searches could get a relic of the past soon or will only be needed for special causes. Image stock market could be AI generated in my point of view. Here are some randomly selected shots from some twitter members (all linked):

Yes, it’s a pity, we all can’t directly play with this fascinating technology!

Luckily I found DALL·E mini. And that's your friend to play! By the way, I also created my post image with Dall-e mini. I asked for "AI Art blue and yellow". One of the results could be found on the overview site of my writings.

The description of DALL·E mini:
“OpenAI had the first impressive model for generating images through DALL·E. DALL·E mini is an attempt at reproducing those results with an open-source model.”

They also have DALL·E mega, a large version of the mini, that is being trained. To see the fascinating world of training, look into the training space.

And finally, here’s your playground to play with the DALL-E mini: https://huggingface.co/spaces/dalle-mini/dalle-mini

That’s my result on "playing children in the forest"; quite an improvement in a very short time. If you play more with DALL·E mini you will notice, that faces still are really not trained much. But I already love it!

To feel the new power of words and to discover some magic along using words, go out and play for yourself!

By the way, there's another tool on the web, giving you similar results. It's neuralblender. Here's my kids results from them:

If you want to get inspired by other AI Artists, look at the work of Refik Anadol, his process is the application of a lot of machine learning to a real world of art. As I said, in my point of view, the term of AI Art is mostly wrong, looking at the output of DALL·E or other CLIP applications, as the outputs are not expressive. In my point of view, we speak about AI graphics.

Maybe you could guess, what my prompt in this case was (created with DALL·E Mini):

Have fun playing!

And guess what... My challenge is to build websites via a text prompt. Currently I am looking for available sources to have a good segmentation (Structural attributes, Style attributes). Let's see when I will share my results or findings.

Some more tipps on playing with DALL·E mini... use objects. To avoid deep fakes, faces won't work. Play around with the text prompt. You could try with the attribute, in the style of... or try the DrawBench prompts being used for Google Imagen first.

Google Imagen

While publishing this article on Mai 24th, Google came up with a competitor to OpenAI DALL·E 2. The market is on fire! And Google-Imagen impresses the same way, DALL·E 2 did.

Be excited to check out the first tweets on the topic #imagen.

And we will all be happy to play with the first, real demos!

Recommended further readings

1. Infinite Images and the latent camera
2. Hierarchical Text-Conditional Image Generation with CLIP Latents
3. DALL·E 2
4. Imagen by Google
5. StyleGan 3
6. Explore my newer articles around the topics:
   How AI Assistants Can Help You with Text and Art
   Generative Art – how to get started with AI Art
   Midjourney and DALL-E 2 – first hands on impressions
   Stable Diffusion celebrates new forms of creativity

FAQs

What if you were asked to create art like Refik Anadol? Or to produce some "AI art"?

In case you haven't heard of him, go directly to the vimeo channel of Refik Anadol and enjoy the moment of discovery!

When I first saw his pieces (only virtually so far), I was flashed. I really love that look of organic movement and abstract shapes. And one question that came to my mind was… how on earth was it done? As I did some research about him a little while ago, but recently rediscovered my love for his work, I wanted to share my findings about him and what could be called AI art.

For me, art reflects humanity’s capacity for imagination. And if I push my compass to the edge of imagination, I find myself well connected with the machines, with the archives, with knowledge, and the collective memories of humanity.

Refik Anadol in an interview with MoMA in 2021

To get an idea who Refik Anadol is and what AI in the context of art is, I recommend watching the ted talk with him.

Painting with data

Melting Memories or Machine hallucinations are examples of his stunning artworks collection. Many of his artworks take the perspective of the computer: what does it look like if a computer is dreaming?

In a world surrounded by data, Refik Anadol uses this data and presents it in a visually stunning representation. These data sets could include many terabytes.

To work with the data he uses machine learning (ML) to produce interactive AI projects. If you are up for Quantum AI by google, there’s also a project you could check out: Quantum memories. And here’s some insight from the data perspective on youtube.

In the past five years, we’ve trained more than 100 AI models, and used close to five petabytes of raw data.

Refik Anadol in an interview with MoMA from 2021

As I mentioned Refik Anadol works a lot with data. For the piece “Melting Memories”, he used EEG data to paint. In the description of the artwork you could find more details on how the masterpiece was realized. Refik describes the use of EEG data and some challenges converting it to vvvv. A live programming environment, everyone, who ever looked at interactive environments might have heard of. IIf you are searching for software that is native to mac, you could give touchdesigner a try. If you are more of a programmer, you could also look at openframeworks.

Besides data, it seems Refik Anadol likes giant canvasses for his art. So he also likes facades to transform architectural spaces. The illumination of Frank Gehry’s Walt Disney Concert Hall is an iconic example. Titled WDCH Dreams, he makes use of projection mapping to illuminate the building.
Google Arts & Culture has an interesting page about Refik Anadol, explaining more facets of the piece.

To get the projection right, you could see that he had a model of the building in his office.

In my practice, I’m not just creating a bunch of shiny pixels. I’m doing my very best to bring depth to surface by understanding exactly why technology may be purposeful and impactful for my imagination. Not because it’s the hype, or because it’s new, or because it is popular.

Refik Anadol in an interview in Harpers Bazaar Arabia

What are the takeaways from AI Art?

It is truly fascinating to see how Refiks Anadols' work circles around our current digital, AI-driven and real landscape. Of course, the work does not address every aspect that is relevant at the moment. This includes environmental aspects, as I mentioned in my article Improving the world as a designer. However, his art makes future aspects of AI tangible in a visually appealing way. Everything I expect from art.

There’s an interesting discussion in MoMA magazine on “Modern Dream: How Refik Anadol Is Using Machine Learning and NFTs to Interpret MoMA’s Collection”. The article describes Refik's Journey to get into his machine hallucination series. It is an excellent source of inspiration, as well as an overview of the status quo of AI-art and its beautiful perspectives. In terms of the ever emerging NFT landscape maybe even an option to get into AI-art yourself. If your alarm signals are ringing, when i mention NFT, be assured, there are eco-friendly NFTs, too.

By the way, if you want to dive deeper into equipment, I guess one piece I forgot about is the projector he is using. And if you want to get a comprehensive overview of his most famous pieces and the money he earned, you can find more on artnsketch. Unfortunately I didn't find details on how he creates the depth in his works. However, I am pretty sure it's the combination of architecture and the illusion of the depth, that creates the magic you might know from playing with generative art.

If you want to dive into the world of generative art, I would recommend getting started with p5JS or processing, but also touchdesigner offers awesome tutorials to get started. Another way to learn is following Daniel Shiffman, aka the coding train on youtube. No matter if generative programming or getting started with Machine Learning (ML). Watching him is already a classic. There's also an impressive collection of AI artists, tools and resources at aiartists.org.

Those principles being applied within AI-artworks are, in my opinion, relevant to design disciplines as well. Interacting in 3D space and mapping visually engaging worlds in a metaverse or other VR experience is challenging and requires more thought, as the world of AI is already showing. Maybe Refik offers such a world for us in one of his latest projects: dataland.

For now my final summary is, thank you, Refik Anadol, for inspiring creative visual discoveries and an inspiring combination of contemporary technologies to provide beauty.

❤️ @refikanadol

Other articles and sources:

1. AI Artists
2. Some resources I collected in my article "Generative Art – how to get started with AI Art"
3. My other writings often focus on how AI is being used in Design, Art, and society in general and I often share some thoughts.
4. AI Art: How artists are using and confronting machine learning | HOW TO SEE LIKE A MACHINE – great video from MoMA with some more description of Refik's work
5. Ted Talk from Refik Anadol: "How AI art could enhance humanity's collective memory"

Recommended tutorials to get started:

1. Refik Anadol Particles in TouchDesigner, Part One - Tutorial
2. Refik Anadol Particles in TouchDesigner, Part Two - Tutorial
3. Some more links to tutorials and how to get started with AI Art you will find in my article Generative Art – how to get started with AI Art