Generating AI Art from Text with Google Colab

With Dream, using AI tools to generate art has officially gone mainstream in a big way, powered by the viral popularity it gained on TikTok. Amazingly, the technique that powers this app was introduced less than a year before the app itself was released, when OpenAI announced CLIP and DALL-E – a model to score whether some text describes the contents of an image and another model to generate images from text, respectively.

The period between the announcement of DALL-E and the release of Dream is in itself an interesting one, during which many AI researchers and hackers got hooked to playing around with these techniques and posting their weird creations on Twitter. As DALL-E was not open-sourced but CLIP was, these same researchers and hackers found ways to cobble together their own approximations of DALL-E by combining the image-generating powers of VQ-GAN with CLIP, as covered well in the article Alien Dreams: An Emerging Art Scene some six months before Dream came to be.

Adverb @advadnoun

The Big Sleep Here's the notebook for generating images by using CLIP to guide BigGAN. It's very much unstable and a prototype, but it's also a fair place to start. I'll likely update it as time goes on. colab.research.google.com/drive/1NCceX2m…

The speed with which AI researchers and hackers started playing with and refining these techniques was in large part powered by one tool: Google Colab. A descendant of the IPython and Jupyter notebook interfaces already commonly used within the AI community, Colab is basically a Google Doc in which you can run code. Importantly, the compute backing running this code is free and moreover comes with a GPU, making it very appealing for AI applications. This enabled people to not only share images from their own CLIP-like implementations, but also to directly share the code necessary to use and build upon these implementations – with no annoying setup steps necessary. Starting with The Big Sleep, new Colab notebooks were regularly developed and released to the community, continually accelerating the process of innovation for such applications. Here’s just a sample of such notebooks that have come out in the past year:

Aran Komatsuzaki @arankomatsuzaki

When you generate images with VQGAN + CLIP, the image quality dramatically improves if you add "unreal engine" to your prompt. People are now calling this "unreal engine trick" lol e.g. "the angel of air. unreal engine"

Colab was not the only thing accelerating progress. Perhaps as important was the sheer number of people playing around with these algorithms, and in the process discovering fun tricks for what could be included in the text inputs to yield different results. Surprisingly, just telling the models to generate something “high resolution” or “rendered by Unity” could often lead to much nicer results, not to mention qualitatively different. These tricks were shared around Twitter, but also on other community spaces such as EleutherAI’s Discord.

Rivers Have Wings @RiversHaveWings

By a user in EleutherAI Discord, using the VQGAN+CLIP bot there: "recursive recursion of the recursive imagination of a landscape by james gurney" and "landscape of recursion by james gurney". CLIP understands that "recursion" means you put the thing inside itself etc?

So, Google Colab is special in that it is a breeding ground for innovation that enables the whole AI community to play around with new ideas and release their findings into the world – which turned out to be especially true for text-to-image AI art creation. Moreover, Colab allows anyone to play around with cutting edge AI, with the only requirements being a Google Drive account and the time to figure out how a given notebook works. This is especially true for greetings AI images from text, with there being handy tutorials and newer Colab notebooks with user-friendly interfaces that make it easier than ever. In fact, there is a nice compilation of notebooks for all sorts of applications of AI, not just generating images. So, if you haven’t done so already, give it a try!

An image of ‘friendly terminator’ I just made in Google Colab

Last Week in AI

Generating AI Art from Text with Google Colab