The city of Prague has become my second home, and it’s also where I connected with a group of artists/designers and researchers of the Uroboros collective. The events they organise offer quite unusual perspectives on technology and society, guided by a more-than-human worldview. One of their recurring themes, especially researched by Lenka Hamosova, is “embodiment”, an aspect so often neglected in common tech discourses, and one I instinctively connect to and know will become crucial in the future.

So when this community announced that they are organising a summer school with the curious name of “PermaPrompting Feral AI Agents”, I had to be a part of it!

An intriguing crossover

The title of this summer school alone was packed with fascinatingly complex concepts:

• AI Agents – the emergent (and hyped) paradigm where generative AI systems get enhanced with more decision-making capabilities and access to tools to interact with other systems. It’s also criticised by people such as Signal founder Meredith Whittaker.
• Permaculture – which I knew very vaguely but was eager to learn more about! And what was PermaPrompting about?!
• Feral – a word that I had encountered before at Uroboros events, especially through Markéta Dolejšová’s research work on multi-species research and more-than-human ecologies.

As someone with degrees in both AI and Fine Arts who worked at the intersection for more than a decade, these kinds of interdisciplinary conferences and surprising cross-overs are exactly where I live. I didn’t know how permaculture would fit into that yet, though :-)

And then there was that provocative line in the description of the summer school: “Machine learning use, transform or refuse”? This hit right at the heart of the question I’ve been struggling with since the hype of generative AI: Whether there can be emancipatory and responsible uses of a technology which might be fundamentally unethical at its core. I also observed during the last months that there is a more mainstream movement emerging to “refuse AI” altogether.

The Programme

From July 21–24, we gathered in a building of the Academy of Fine Arts (AVU) in Prague’s Letná district. The participants were impressively international: they came from Turkey, India, Bosnia, the Netherlands, Poland, Italy, Mexico, the USA, Ireland, Germany (me) and of course the Czech Republic. Their professions ranged from philosophy, microbiology, architecture, automotive design, photography, and more.

The goal was to build our own “agents” — however they chose to interpret that term (more on that later). We were supposed to bring our own datasets to work with.

Along this way, we were treated with inspiring talks, technical workshops and artistic lecture performances.

Day 1: Foundations and Philosophy

The first day opened with welcomes from organisers Enrique Encinas and Markéta Dolejšová. After participant introductions, philosopher Ramón Alvarado from the University of Oregon gave a talk on “What is an Agent” — exploring the concept of agency across different philosophical traditions.

Day 2: Python and Commandline

On the second day, we got hands-on experience with Smol Agents, a Python library by HuggingFace for building AI agents with just a few lines of code. After this, philosopher and designer Denisa Kera presented her research on conversing with datasets (such as satellite data) with the help of LLMs. She also talked briefly about her research on ergative languages and LLMs. The rest of the day was dedicated to independent work on our projects.

Day 3: Feralities and Intuition

The third day started with a presentation by Markéta Dolejšová on “Practising Feralities” and a workshop by Lenka Hamosova exploring the role of intuition and the body when interacting with generative AI technology.

Day 4: Presentations

The final day focused on wrapping up projects and presenting our work to the group. What projects were developed during the summer school? A web-based game mimicking a pollination garden for bees. A synthetic movie speculating about invasive plants in Prague. A “popstar generator” based on a dataset of natural volcanic behaviour. And more!

I got lucky to collaborate with Dutch designer Giliam Ganzevles during the summer school on an AI/web project. It all started with the data that Giliam brought to the summer school: “Circular soil chromatographs”. It was the first time I saw these kinds of images and got immediately hooked on their beauty and expressiveness (they are analogue datavisualisations!)

Together we built a web application that is half speculative art/AI project and half scientific image analysis – but that deserves its own blogpost!

The highlights

These were 4 extremely intense days, both intellectually, creatively and socially! Long days that started in the morning and ended in bars late at night. The interdisciplinary mix of people was so nice and something I would love to have more often in my life. The kind of conversations that emerged there is something that I wouldn’t find in pure technical spaces (like an “AI agent hackathon”) nor pure art contexts.

The tensions

That said, my dual background — technical AI knowledge and art practice — created some interesting tensions throughout the event. This is a common theme in interdisciplinary work, and I think these kinds of frictions are what make such gatherings valuable. Three specific points kept me thinking:

💥 Agent (Terminology)

Throughout the event, I watched the word “agent” get applied to everything with remarkable generosity. Simple chatbots? Agents. Basic LLMs with system prompts? Agents. A python programme that outputs something? Agents.

This drove me a bit nuts. I was aware that the term Agent has several meaning in different contexts (Philosophy, social science, psychology). But I thought when we speak about concrete technology, we should stick to the technical terminology?

💥 Including AI technology in the more-than-human discourse

It makes me deeply uncomfortable to see how the more-than-human discourse is extended to AI systems. Terms like “feral AI” seem to suggest that these systems possessed a genuine wildness or autonomy.

I’ve spend years arguing against AI anthropomorphisation. One of the myths we tackled on aimyths.org was literally “AI has agency”. The more-than-human discourse makes sense to me for actual non-human entities: animals, plants, and ecosystems are genuine actors. But extending this to AI seems to undo years of demystification work: treating prediction machines as autonomous beings and hiding the agency of the ones that make them such as companies like OpenAI.

I am probably missing a lot of nuances in the more-than-human (or New Materialism or Object Oriented Ontology) discourse, but applying it to AI and seems to me to play right into the hands of the big AI monopoly companies and their AGI and longtermists cult members.

💥 Blind Spots in “Critical AI”

I was expecting a critical art bubble discourse about AI, but was surprised about the lack of critical awareness when it comes to actually using the tools. Here we were, discussing AI resistance while casually burning through energy-intensive models. Video generation or computation-heavy reasoning processses in “agentic AI” — the ecological implications flew under the radar. Similar to the privacy implications, when uploading pictures of peoples faces or conversational notes to some “cloud”.

What I’m Taking Away

This was a precious event! Despite of these tensions described above — or rather, because of them I found this summer school to be incredibly inspiring and productive.

Uroboros collective again managed to create a safe space that encourages discovery, experimentation, bringing people from disciplines to the table. The overall atmosphere was so generous and creative, the participants were incredible, and Enrique’s warm moderation kept everything flowing beautifully.

Practicing interdisciplinary communication is always quite exhausting I guess. Reading everything that is being said twice with regard to different interpretations is quite energy-demanding. It seems almost like my body enters another state of metabolism due to it!

I definitely want to learn more about the more-than-human discourse and philosophy such as new materialism and object oriented ontology. And see how critical AI literacy would fit in there without sacrificing the crucial points.

And it was wonderful that I could bring my dog Lillet along, who was warmly welcomed despite her sometimes weird and socially awkward behaviour. This openness is just a nice side effect of the organizers’ genuine commitment to multi-species research :-)

The summer school “PermaPrompting Feral AI agents” (21-24.07.2025) is part of the “Ars Biologica” series of summer schools. The next event will take place in Budweis, CZ.

Links

• Open Call and Programme of the summer school: https://collective.uroboros.design/open-call-permaprompting/
• Uroboros collective (organizers of the summer school): https://collective.uroboros.design/
• Lenka Hamosova (founding member of Uroboros collective) on Medium: https://medium.com/@lenkahamosova

Email is the lifeline of my professional existence. As a freelance software developer, it is the primary way I connect with clients, secure new gigs, manage projects, and, of course, send invoices. Missing an email could lead to lost opportunities and unnecessary confusion. Simply put, email is how I feed myself and my dog!

Given its importance, switching my email account seemed like a daunting task. I had several concerns: How long would the transition take? Would there be any downtime? Could emails get lost in the process? Would my emails suddenly be marked as spam?

I wrote this short article to answer these questions and share some best practices. Hopefully, it will also ease some of your insecurities.
TL;DR: It’s fairly easy, the risks are low, and you can do it too!

This short article is for you if you have a custom email address and domain (e.g., yourname@yoursite.com) and want to move it to a new email provider.

This article is NOT for you if you have a Gmail address or similar (yourname@gmail.com) and want to switch email provider.

Choosing ProtonMail as my new email provider

Recently, I decided to switch my email provider. My requirements were: enhanced privacy, enhanced security for me and my clients, reasonable pricing, good user experience (admin interface, mail client). And most importantly, I needed to keep my already existing custom domain: alexasteinbruck.com

After some market research and comparison, I chose Proton Mail for these main reasons:
1. they put privacy at first place
2. their pricing is reasonable
3. their mail client has a great UI
4. their code is open source

Proton Mail prioritizes privacy above all else, as stated in their commitment:

“We provide easy-to-use alternatives to Big Tech services and their surveillance business models. With Proton, your data is protected, not exploited.”

Their privacy features include end-to-end encryption ensuring that not even Proton itself can read your emails. Moreover, Proton Mail detects and disables tracking technologies embedded in incoming emails. Beyond these core offerings, Proton Mail includes several other interesting features, like disposable email addresses, a free VPN service and Proton Scribe, an integrated privacy-friendly version of ChatGPT.

Two other impressive facts about Proton:

• Proton was founded by a group of scientists who met at CERN. They were advised by Sir Tim Berners-Lee who is often called the inventor of the world wide web.
• Proton operates two data centers to ensure reliability and security. One is located in Lausanne, while the other serves as a backup in Attinghausen, housed in the former K7 military bunker under 1,000 meters of granite rock. Here’s a Quartz reportage about it.

How to do the migration (outline)

Disclaimer: Please check the actual website of Proton for more detailed and up-to-date instructions, this is just an overview!

Step 1 — Sign up with Proton Mail

Sign up with Proton Mail. Then go to the “Domain names” settings page and click Add domain or Review. This will launch a convenient assistant that will walk you through the necessary steps!

Step 2 — Update DNS records

Open a new tab and login at your domain provider. That’s the company where you once registered your custom domain (it might also be where you bought your web space/hosting). It is important to understand that hosting and domains are different things that should always be decoupled: You could cancel your hosting plan, but keep your domain.

Navigate to the DNS settings of your domain provider. Here you will see a table of your DNS records.

1. Add a DNS record of type TXT (this is to prove to Proton that you are indeed the owner of the domain) to your list of DNS records. Copy the values from the Proton assistant website.
2. Add 2 MX records — these are the actual records for email routing: mail.protonmail.ch and mailsec.protonmail.ch
3. Add a few more DNS records for mail origin verification, security and spam prevention: SPF (that’s a record of type TXT), DKIM (record of type CNAME) and DMARC (record of type TXT)

What needs to happen with your old DNS records?

That depends on the type of records.
Old MX records: You need to delete/overwrite them. Alternatively keep them and give them a priority that has a higher number (meaning lower priority) than the Proton entries.
Old SPF/DKIM/DMARC records: A single domain can have only 1 TXT record for SPF! But the value of the SPF record can reference multiple servers in the text content.
Check in Protons migration assistant for more recommendations!

Step 3 — Wait ⏳

How long will the transition take?

This is hard to tell. DNS is an unpredictable beast. It can take up to 72 hours for the new DNS records to update through the system. In my case it took only about 3 hours for the whole procedure to be completed. In your Proton admin interface you can see the status of the migration.

FAQs/Questions

Is there any “downtime”?

In case that the new MX records have not yet been picked up by the DNS servers worldwide, they will still refer to the old MX records pointing to your old email provider. This means you could still receive emails on your old email server, so don’t shut it down too quickly.

Can emails get lost in the process?

It is unlikely for incoming emails to get lost, if you have any MX records present in your DNS settings. In the rare case that an email could not be sent to one mail server or the other, they will “bounce” and the sender would be notified by a message: “Undelivered Mail Returned to Sender”.

Can both email servers be “active”

If you keep both MX entries yes. The record with the lowest priority is used first, then the higher ones until one server responds. If they have similar priorities then the emails would be randomly split between the two servers (this is how load balancing is usually done). But one single email will never be delivered to more than one server!

More tips to smooth the process

1. Notify Your Key Contacts: Inform your important contacts about the upcoming migration. This reduces the risk of missed communications.
2. Choose the Right Timing: Plan the migration for a weekend or a period when your email activity is lower.
3. Adjust TTL Before Migrating: Prior to starting the migration, set your MX records to a low TTL (Time to Live). This ensures that DNS servers will update the records more quickly once you switch to the new email server, speeding up the transition and reducing the likelihood of email delivery issues.

Conclusion

The transition to a new email provider turned out to be fairly easy and not as risky as I initially thought. This was largely due to Proton Mail’s user-friendly transition assistant, which streamlined the process and minimized potential issues. Overall, the migration went smoothly, confirming that with the right tools and preparation, changing email providers can be a straightforward task.

One Last Thought

If you prioritize security, it’s crucial to ensure that your domain provider also offers robust security features. Relying solely on a secure email provider isn’t enough. If someone gains access to your domain provider account, they could alter DNS settings and redirect your MX records to a different mail server. This would allow them to impersonate you and intercept emails intended for you. Therefore, securing both your email provider and domain account is essential for comprehensive protection.

Appendix / DNS Terminology

DNS = Domain Name System
TTL = Time to live, refers to how long a DNS server can serve a cached DNS record.

Relevant DNS record types

MX — a mail exchange record
TXT — a text record
CNAME — canonical name record

How to refer to the columns in the DNS table:

the record type, e.g. TXT
the host name, e.g. @
the value, e.g. protonmail-verification=xxx.

Special records of type TXT

- SPF (Sender Policy Framework) — to authenticate the sender and ensure that only authorized servers can send emails from your domain
- DMARC (Domain-based Message Authentication, Reporting and Conformance) — protect from email spoofing

Special records of type CNAME

- DKIM (DomainKeys Identified Mail) — to prevent hackers to mess with your email, to verify that messages haven’t been tampered with in transit
- SPF, DKIM and DMARC are used for email security and authentication.

Keywords: headless CMS, headless wordpress, gallery, custom post type, Gatsby, ACF, Pods, Custom meta fields, repeatable fields, gatsby-plugin-image, gatsby-source-wordpress, GraphQL

The usecase

I am building a website that contains multiple image galleries. I am using Wordpress as the CMS (in a so called “headless CMS” mode) and build the real site with Gatsby.

This article is about how to customize the Wordpress CMS/Admin interface to include a “Gallery” post type and how to query the resulting data model in Gatsby/Graphql.

This article is NOT about building or designing a gallery component with React/Gatsby.

My requirements/requests/feature requests

The gallery should be both a) user friendly on the Wordpress side and b) its data model should integrate harmoniously with the Gatsby/GraphQL universe. This means:

- it should be free! I don’t want to spend any money on Wordpress plugins (looking at you, ACF Pro)
- the gallery should support any number of images
- the order of images should be defined by the user, ideally by means of a drag’n’drop UI
- it should be compatible with gatsby-source-wordpress (GraphQL based)
- it should be compatible with gatsby-plugin-image

It sounds like a simple use case, but turned out to be harder to realise (if you don’t want to spend money)! I researched 3 solutions, but only the first fulfilled all my requirements: Pods. At the end of the article I share two more alternative solutions.

Solution: Pods! (with a little hack)

The Pods framework is the open source alternative to Advanced Custom Fields (ACF). This tutorial uses Pods version 3.2.1.

First create a pod with name “Gallery” and a field group named “Gallery fields”, then add a field of type “File/Image/Video” named “Images” and set the “upload limit” to “multiple files”

How do I query this in Gatsby?

First you need to enable GraphQL visibility for your pod:

Then query “wpGallery” (the name of your pod prefixed with “wp”) and the field “images” (the name of your custom field):

query MyQuery {
  wpGallery(id: {eq: "cG9zdDoyOTA="}) {
    id
    images {
      nodes {
        gatsbyImage(width: 900)
      }
    }
  }
}

{
  "data": {
    "wpGallery": {
      "id": "cG9zdDoyOTA=",
      "images": {
        "nodes": [
          {
            "gatsbyImage": {
              "images": {
                "sources": [
                  {
                    "srcSet": "/_gatsby/image/358421b5b0035ac93a42cc6d46b7e7ad/2601eded68646ab34949d76d53ad7f2e/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.avif?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fwlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg&a=w%3D225%26h%3D150%26fm%3Davif%26q%3D75&cd=e7373efa71b459fa1ff1b97721c3da36 225w,/_gatsby/image/358421b5b0035ac93a42cc6d46b7e7ad/7bb857502b00873ff560c6e47a556b47/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.avif?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fwlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg&a=w%3D450%26h%3D300%26fm%3Davif%26q%3D75&cd=e7373efa71b459fa1ff1b97721c3da36 450w,/_gatsby/image/358421b5b0035ac93a42cc6d46b7e7ad/7071a6526579ae261f81b8bc4463365f/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.avif?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fwlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg&a=w%3D900%26h%3D600%26fm%3Davif%26q%3D75&cd=e7373efa71b459fa1ff1b97721c3da36 900w,/_gatsby/image/358421b5b0035ac93a42cc6d46b7e7ad/15ac18a8e8615988613cad19537f1c4e/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.avif?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fwlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg&a=w%3D1800%26h%3D1200%26fm%3Davif%26q%3D75&cd=e7373efa71b459fa1ff1b97721c3da36 1800w",
                    "type": "image/avif",
                    "sizes": "(min-width: 900px) 900px, 100vw"
                  },
                  {
                    "srcSet": "/_gatsby/image/358421b5b0035ac93a42cc6d46b7e7ad/ab11f2182071e384443dffa045bdc85a/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.webp?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fwlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg&a=w%3D225%26h%3D150%26fm%3Dwebp%26q%3D75&cd=e7373efa71b459fa1ff1b97721c3da36 225w,/_gatsby/image/358421b5b0035ac93a42cc6d46b7e7ad/bcfaf29f3ee3fb1b3ebbc12a78cfd3a1/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.webp?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fwlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg&a=w%3D450%26h%3D300%26fm%3Dwebp%26q%3D75&cd=e7373efa71b459fa1ff1b97721c3da36 450w,/_gatsby/image/358421b5b0035ac93a42cc6d46b7e7ad/e4c8d7aecd9714785ba817d01c221dbc/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.webp?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fwlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg&a=w%3D900%26h%3D600%26fm%3Dwebp%26q%3D75&cd=e7373efa71b459fa1ff1b97721c3da36 900w,/_gatsby/image/358421b5b0035ac93a42cc6d46b7e7ad/d3a4d104262dc3a57513c01c59ce88ed/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.webp?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fwlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg&a=w%3D1800%26h%3D1200%26fm%3Dwebp%26q%3D75&cd=e7373efa71b459fa1ff1b97721c3da36 1800w",
                    "type": "image/webp",
                    "sizes": "(min-width: 900px) 900px, 100vw"
                  }
                ],
                "fallback": {
                  "src": "/_gatsby/image/358421b5b0035ac93a42cc6d46b7e7ad/e9e96822177d1cdf77ea1240c2a11d68/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fwlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg&a=w%3D225%26h%3D150%26fm%3Djpg%26q%3D75&cd=e7373efa71b459fa1ff1b97721c3da36",
                  "srcSet": "/_gatsby/image/358421b5b0035ac93a42cc6d46b7e7ad/e9e96822177d1cdf77ea1240c2a11d68/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fwlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg&a=w%3D225%26h%3D150%26fm%3Djpg%26q%3D75&cd=e7373efa71b459fa1ff1b97721c3da36 225w,/_gatsby/image/358421b5b0035ac93a42cc6d46b7e7ad/accbf945a4094cd86d6f5636e23417dd/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fwlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg&a=w%3D450%26h%3D300%26fm%3Djpg%26q%3D75&cd=e7373efa71b459fa1ff1b97721c3da36 450w,/_gatsby/image/358421b5b0035ac93a42cc6d46b7e7ad/27448af9126a8394bb236d421215d85d/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fwlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg&a=w%3D900%26h%3D600%26fm%3Djpg%26q%3D75&cd=e7373efa71b459fa1ff1b97721c3da36 900w,/_gatsby/image/358421b5b0035ac93a42cc6d46b7e7ad/c059c0edbdd574ab1b7bf2f752b4c647/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fwlodzimierz-jaworski-2BbwrlmIaX8-unsplash-scaled.jpg&a=w%3D1800%26h%3D1200%26fm%3Djpg%26q%3D75&cd=e7373efa71b459fa1ff1b97721c3da36 1800w",
                  "sizes": "(min-width: 900px) 900px, 100vw"
                }
              },
              "layout": "constrained",
              "width": 900,
              "height": 600,
              "backgroundColor": "rgb(184,136,88)"
            }
          },
          {
            "gatsbyImage": {
              "images": {
                "sources": [
                  {
                    "srcSet": "/_gatsby/image/36980c936cf4dd8450a45772432b0482/3708ce2a4f25ec91a919c6d9555185f8/image-1.avif?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage-1.jpeg&a=w%3D225%26h%3D180%26fm%3Davif%26q%3D75&cd=97ba0befaf041547ee8d103d2944682b 225w,/_gatsby/image/36980c936cf4dd8450a45772432b0482/802098db786cef50e197ef74f50cb348/image-1.avif?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage-1.jpeg&a=w%3D450%26h%3D360%26fm%3Davif%26q%3D75&cd=97ba0befaf041547ee8d103d2944682b 450w,/_gatsby/image/36980c936cf4dd8450a45772432b0482/fb118e5b30f1e15bea9cc774b966092a/image-1.avif?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage-1.jpeg&a=w%3D900%26h%3D720%26fm%3Davif%26q%3D75&cd=97ba0befaf041547ee8d103d2944682b 900w,/_gatsby/image/36980c936cf4dd8450a45772432b0482/d1de0b1a8153f7c6d4e2633c7f5a7a79/image-1.avif?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage-1.jpeg&a=w%3D1800%26h%3D1440%26fm%3Davif%26q%3D75&cd=97ba0befaf041547ee8d103d2944682b 1800w",
                    "type": "image/avif",
                    "sizes": "(min-width: 900px) 900px, 100vw"
                  },
                  {
                    "srcSet": "/_gatsby/image/36980c936cf4dd8450a45772432b0482/b8f4e899a486e60196419eb35f37bc5a/image-1.webp?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage-1.jpeg&a=w%3D225%26h%3D180%26fm%3Dwebp%26q%3D75&cd=97ba0befaf041547ee8d103d2944682b 225w,/_gatsby/image/36980c936cf4dd8450a45772432b0482/26f0ddc1a5bd4576bce36b80b070c2ad/image-1.webp?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage-1.jpeg&a=w%3D450%26h%3D360%26fm%3Dwebp%26q%3D75&cd=97ba0befaf041547ee8d103d2944682b 450w,/_gatsby/image/36980c936cf4dd8450a45772432b0482/dc03710d3a93c7667d53754ef3f1253b/image-1.webp?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage-1.jpeg&a=w%3D900%26h%3D720%26fm%3Dwebp%26q%3D75&cd=97ba0befaf041547ee8d103d2944682b 900w,/_gatsby/image/36980c936cf4dd8450a45772432b0482/bec98bd98ea6a7bb83efe73b5ec43d9b/image-1.webp?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage-1.jpeg&a=w%3D1800%26h%3D1440%26fm%3Dwebp%26q%3D75&cd=97ba0befaf041547ee8d103d2944682b 1800w",
                    "type": "image/webp",
                    "sizes": "(min-width: 900px) 900px, 100vw"
                  }
                ],
                "fallback": {
                  "src": "/_gatsby/image/36980c936cf4dd8450a45772432b0482/6636af1d59a2d8aeba7e72991e51ceaf/image-1.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage-1.jpeg&a=w%3D225%26h%3D180%26fm%3Djpg%26q%3D75&cd=97ba0befaf041547ee8d103d2944682b",
                  "srcSet": "/_gatsby/image/36980c936cf4dd8450a45772432b0482/6636af1d59a2d8aeba7e72991e51ceaf/image-1.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage-1.jpeg&a=w%3D225%26h%3D180%26fm%3Djpg%26q%3D75&cd=97ba0befaf041547ee8d103d2944682b 225w,/_gatsby/image/36980c936cf4dd8450a45772432b0482/8655e58e51785ab73611a2608e69b245/image-1.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage-1.jpeg&a=w%3D450%26h%3D360%26fm%3Djpg%26q%3D75&cd=97ba0befaf041547ee8d103d2944682b 450w,/_gatsby/image/36980c936cf4dd8450a45772432b0482/d033b7c618abb2f1f62fe01fe409178e/image-1.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage-1.jpeg&a=w%3D900%26h%3D720%26fm%3Djpg%26q%3D75&cd=97ba0befaf041547ee8d103d2944682b 900w,/_gatsby/image/36980c936cf4dd8450a45772432b0482/582efc2f619e31a1a8a81576ff1f6514/image-1.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage-1.jpeg&a=w%3D1800%26h%3D1440%26fm%3Djpg%26q%3D75&cd=97ba0befaf041547ee8d103d2944682b 1800w",
                  "sizes": "(min-width: 900px) 900px, 100vw"
                }
              },
              "layout": "constrained",
              "width": 900,
              "height": 720,
              "backgroundColor": "rgb(88,104,56)"
            }
          },
          {
            "gatsbyImage": {
              "images": {
                "sources": [
                  {
                    "srcSet": "/_gatsby/image/4fb3b94908b64bb6f750236008c445e6/2601eded68646ab34949d76d53ad7f2e/image.avif?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage.jpeg&a=w%3D225%26h%3D150%26fm%3Davif%26q%3D75&cd=dc54c0b0da4be912cb958b8bdd61c97a 225w,/_gatsby/image/4fb3b94908b64bb6f750236008c445e6/7bb857502b00873ff560c6e47a556b47/image.avif?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage.jpeg&a=w%3D450%26h%3D300%26fm%3Davif%26q%3D75&cd=dc54c0b0da4be912cb958b8bdd61c97a 450w,/_gatsby/image/4fb3b94908b64bb6f750236008c445e6/7071a6526579ae261f81b8bc4463365f/image.avif?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage.jpeg&a=w%3D900%26h%3D600%26fm%3Davif%26q%3D75&cd=dc54c0b0da4be912cb958b8bdd61c97a 900w",
                    "type": "image/avif",
                    "sizes": "(min-width: 900px) 900px, 100vw"
                  },
                  {
                    "srcSet": "/_gatsby/image/4fb3b94908b64bb6f750236008c445e6/ab11f2182071e384443dffa045bdc85a/image.webp?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage.jpeg&a=w%3D225%26h%3D150%26fm%3Dwebp%26q%3D75&cd=dc54c0b0da4be912cb958b8bdd61c97a 225w,/_gatsby/image/4fb3b94908b64bb6f750236008c445e6/bcfaf29f3ee3fb1b3ebbc12a78cfd3a1/image.webp?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage.jpeg&a=w%3D450%26h%3D300%26fm%3Dwebp%26q%3D75&cd=dc54c0b0da4be912cb958b8bdd61c97a 450w,/_gatsby/image/4fb3b94908b64bb6f750236008c445e6/e4c8d7aecd9714785ba817d01c221dbc/image.webp?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage.jpeg&a=w%3D900%26h%3D600%26fm%3Dwebp%26q%3D75&cd=dc54c0b0da4be912cb958b8bdd61c97a 900w",
                    "type": "image/webp",
                    "sizes": "(min-width: 900px) 900px, 100vw"
                  }
                ],
                "fallback": {
                  "src": "/_gatsby/image/4fb3b94908b64bb6f750236008c445e6/e9e96822177d1cdf77ea1240c2a11d68/image.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage.jpeg&a=w%3D225%26h%3D150%26fm%3Djpg%26q%3D75&cd=dc54c0b0da4be912cb958b8bdd61c97a",
                  "srcSet": "/_gatsby/image/4fb3b94908b64bb6f750236008c445e6/e9e96822177d1cdf77ea1240c2a11d68/image.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage.jpeg&a=w%3D225%26h%3D150%26fm%3Djpg%26q%3D75&cd=dc54c0b0da4be912cb958b8bdd61c97a 225w,/_gatsby/image/4fb3b94908b64bb6f750236008c445e6/accbf945a4094cd86d6f5636e23417dd/image.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage.jpeg&a=w%3D450%26h%3D300%26fm%3Djpg%26q%3D75&cd=dc54c0b0da4be912cb958b8bdd61c97a 450w,/_gatsby/image/4fb3b94908b64bb6f750236008c445e6/27448af9126a8394bb236d421215d85d/image.jpg?u=http%3A%2F%2Fcms.mywordpresswebsite.de%2Fwp-content%2Fuploads%2F2024%2F05%2Fimage.jpeg&a=w%3D900%26h%3D600%26fm%3Djpg%26q%3D75&cd=dc54c0b0da4be912cb958b8bdd61c97a 900w",
                  "sizes": "(min-width: 900px) 900px, 100vw"
                }
              },
              "layout": "constrained",
              "width": 900,
              "height": 600,
              "backgroundColor": "rgb(88,168,72)"
            }
          }
        ]
      }
    }
  }
}

Now there is one problem: The order in which the images appear in the output of the GraphQL query is not the order in which the images appear in the CMS!

This appears to be a bug in the Pods GraphQL integration.

Workaround: Querying WP RestAPI to get correct image order

I found this workaround:
Wordpress also offers a RestAPI that holds the same data as the GraphQL endpoint. Luckily, in the RestAPI the order of the images is correctly encoded. So we can do an additionally request to this RestAPI just to get the order of the images and then apply this information to the result of the GraphQL query.

First, enable the RestAPI for the pod itself, and then also for the custom fields (!).

Then you can do the following request, e.g. server side in your gatsby-node.js:

const response = await fetch(
    "http://cms.mywordpresswebsite.de/wp-json/wp/v2/gallery/293"
  )

• /wp-json/wp/v2 is the default endpoint of the Wordpress RestAPI
• /gallery is the name of our pod
• /293 is the id of a specific gallery instance

The JSON output contains an image field which is an array which contains the images in the order in which they were defined in the CMS.

In the JSON output, every image has an ID field that is identical to the databaseId in the GraphQL output. This way both can be matched.

{
  "id": 293,
  "slug": "gallery-1",
  "type": "gallery",
  "link": "http://cms.mywordpresswebsite.de/gallery/gallery-1/",
  "title": {
    "rendered": "Gallery 1"
  },
  "content": {
    "rendered": "",
    "protected": false
  },
  "template": "",
  "acf": [],
  "images": [
    {
      "ID": "291",
      "guid": "http://cms.mywordpresswebsite.de/wp-content/uploads/2024/05/wlodzimierz-jaworski-2BbwrlmIaX8-unsplash.jpg",
      "post_type": "attachment",
      "post_mime_type": "image/jpeg",
      "pod_item_id": "291"
    },
    {
      "ID": "281",
      "guid": "http://cms.mywordpresswebsite.de/wp-content/uploads/2024/05/image-1.jpeg",
      "post_type": "attachment",
      "pod_item_id": "281"
    },
    {
      "ID": "279",
      "guid": "http://cms.mywordpresswebsite.de/wp-content/uploads/2024/05/image.jpeg",
      "post_type": "attachment",
      "post_mime_type": "image/jpeg",
      "pod_item_id": "279"
    }
  ],
}

This is the best solution I could find to build a free gallery admin interface in Wordpress for Gatsby. In the last section I would like to share more attempts to solve this usecase and their pro and cons.

Appendix (Two other solutions)

Solution 2: Advanced Custom Fields (ACF)

When it comes to customizing Wordpress, the “Advanced Custom Fields” (ACF) Wordpress plugin is a big one. ACF has a free version and a paid version (ACF Pro). I am going with the free version. This tutorial uses ACF version 6.2.9.

After creating a custom post type “Gallery” I am looking for any field types that might be a good fit for a gallery, such as “Image” or “Gallery”. Bummer: “Image” does not have a “repeater” option, and “Gallery” is a paid feature of ACF Pro :-(

A workaround could be to set up a fixed number of image fields. This is a bit ugly and does not offer the user much flexibility. But it might be okay in some situations.

How do I query this in Gatsby?
First, make sure that you have enabled the “Show in GraphQL” option in the ACF settings in Wordpress!

Then in the GraphQL query I then query “galleryfields” (or whatever custom name I gave to my fieldgroup, and then query every single image field that I created:

query MyQuery {
  allWpGallery {
    nodes {
      id
      title
      galleryfields {
        image1 {
          node {
            gatsbyImage(width: 900)
          }
        }
        image2 {
          node {
            gatsbyImage(width: 900)
          }
        }
        image3 {
          node {
            gatsbyImage(width: 900)
          }
        }
      }
    }
  }
}

Tip: It can be useful to use GraphQL fragments to avoid this repetition.

Pros
- Admin interface is constrained to only image fields
- In Gatsby I can access every single image seperately and build whatever components with it

Cons
- predefined number of image fields
- User cant change order easily

Solution 3: A simple Wordpress Page

This solution is pragmatic. The thinking goes: A regular Wordpress page contains a free editor field which can hold almost anything, including images. So it can be used as a gallery: The user inserts the images in the Gutenberg editor as “blocks” in any order. No custom post types or custom meta fields needed.

How do I query this in Gatsby?
In my GraphQL query I then query the field “content” which returns the whole HTML blob which contains the images inline.
Great: The plugin gatsby-source-wordpress converts even the inline images into Gatsby images! (Detailed documentation is available on Github)
Notice how in the response the image source links don’t point to the Wordpress instance anymore, but to the Gatsby site instead (relative path: /_gatsby/file/…).

Pros
- Unlimited, arbitrary number of images
- Good UI for inserting images and changing the image order

Cons
- It’s all one big HTML blob, but I might want to access the images individually to build a completely different markup from them with React
- Too little control over layout, Wordpress includes its own markup and classes
- The standard Wordpress block editor offers users too much freedom

The new campaign by the “Center of Humane Technology” is dripping with sensationalist X-risk AGI hype and pseudo-science. That’s not how we should educate the public about the problems associated with AI.

In their one-hour talk which has been viewed by more than 2.6M people on Youtube, Aza Raskin and Tristan Harris request the audience to stop for a moment and take a deep breath. Raskin closes his eyes, and you hear the intimate sound of his breath through the microphone. Then with the serenity of an esoteric mentor, he instructs the people in the audience to practice kindness towards themselves:

“It’s going to feel almost like the world is gaslighting you. People will say at cocktail parties, you’re crazy, look at all this good stuff [AI] does (…) show me the harm, point me at the harm, and it’s very hard to point at the concrete harm. So really take some self-compassion.”

“AI Dilemma” is the new campaign by the “Center of Humane Technology” (CHT), a non-profit founded by Aza Raskin and Tristan Harris in 2018 to educate the public and advise legislators about the harmful impact that technology can have on individuals, institutions, and society. In their early years, their primary focus was on social media and the so-called “attention economy” which led to their major involvement in the Netflix documentary “The social dilemma” in 2020.

The new campaign focuses on the risks associated with Artificial Intelligence — or as they call it “Gollems”— a tongue-in-cheek acronym for a group of AI technologies they summarize as “Generative Large Language Multi-Modal Models”. “Gollems” include systems like ChatGPT and like.

According to Raskin and Harris, there have been two contact points of humanity with AI so far:

The first contact was due to the proliferation of social media and recommendation algorithms (“curation AI”). The results of this first contact had been disastrous to the psyche of the individual and the functioning of society: Information overload, addiction, doom scrolling, polarization, fake news, etc.

The second contact of humanity happens now in the year 2023 with “creation AI” — or as they call it “Gollems”. And this is where the Center for Humane Technology steps in: “We should not make the same mistakes as with social media”. Raskin and Harris want to warn us about the dangers of these “Gollem”-type AIs before they get integrated everywhere and become “entangled” with society. “We can still choose the future we want”.

Raskin and Harris have consulted many experts in the area of AI safety about what the actual problem is with this “Golem AIs”. “We’re talking about how a race dynamic between a handful of companies of these new Golem class AI’s are being pushed into the world as fast as possible” they say. “The reason we are in front of you is that the people who work in this space feel that this is not being done in a safe way”.

So they’re all about slowing down and democratic dialogue. That sounds good for now. Even if it is still a little unclear where they actually locate the problems of these “Gollem” systems.

“50% of AI researchers believe …” wrong numbers, no context.

There is a sentence that they show 3 times throughout their talk — it seems to be the backbone of their alarming argumentation:

“50% of AI researchers believe there’s a 10% or greater chance that humans go extinct from our inability to control AI”.

This jaw-dropping number appears to result from a 2022 survey conducted by an institution called “AI Impacts” (hardly readable in the small print on the slide). Raskin and Harris don’t give the audience any contextual information about these numbers.

It is utterly irresponsible to speak of “50% of AI researchers” given that it is literally just 80 people from a biased survey.

But it is even worse: These numbers are plain wrong, and the survey methodology is deeply flawed as has been explained here and here. How the survey was conducted: The organization approached 4271 AI researchers who published in a specific year at NeurIPS conference. Only 738 people agreed to participate in the survey (you can see already that it is a biased dataset). And only 162 of those actually answered the very specific question that the quote is referring to. It is utterly irresponsible to speak of “50% of AI researchers” given that it is literally just 80 people from a biased survey.

Raskin and Harris also don’t further elaborate what “uncontrolled” means or in what way the extinction of humanity would actually happen.

We’re really NOT talking about the AGI apocalypse. Really, really, really not!

One of the preposterous things about their talk is that they really, really, really want us to know that they are not talking about the “AGI apocalypse” (13:08, 42:20), also known as the “AI takeoff” scenario. They describe this scenario as follows: “AI becomes smarter than humans in a broad spectrum of things, it begins the ability to self-improve, then we ask it to do something — you know the old standard story of ‘be careful what you wish for because it will become true in an unexpected way’ — you wish to be the richest person so the AI kills everyone else”

But remember, this is not what they are here to talk to us about!

This is very confusing, because the “AI apocalypse” or “AI takeoff” scenario is exactly what motivated the flawed survey they love so much that they show it 3 times.

The organization behind the survey has the discrete name “AI Impacts” and belongs to the “Machine Intelligence Research Institute” (MIRI), formerly called “Singularity Institute for AI” located in Berkeley, California.

“As part of the broader Effective Altruism community, we prioritize inquiry into high impact areas like existential risk” can be read on their website, as well as a listing of their sponsors, among them the Future of Life Institute (FLI) and the Future of Humanity Institute (FHI). The FHI was founded by Nick Bostrom, the author of the famous book “Superintelligence” and inventor of the term “existential risk”.

So what are the actual problems with AI?

Here’s a list of things they consider problems due to AI:

Reality collapse, Fake everything, Trust collapse, Collapse of law contracts, Automated fake religions, Exponential blackmail, Automated cyberweapons, Automated exploitation of code, Automated lobbying, Biology automation, Exponential scams, A-Z testing of everything, Synthetic relationships, AlphaPersuade

While some of these points represent actual and even short-term dangers to Western society and will make the internet a more chaotic and hostile place, their listing seems quite arbitrary and, most importantly, it’s a very privileged and very white view on the effects of AI on humanity.

The fact that they call social media the “first contact of humanity with AI” is plainly ignorant: People, especially marginalized groups have much longer been affected by AI algorithms in negative ways. The automation of inequality is a reality: Biased algorithms in areas such as policing, social welfare, finance and recruiting have had and are still having huge impacts on the realities of real human lives.

They also completely ignore the production conditions behind AI and present it as something that comes out of research and then just needs to be deployed by companies. This view ignores the environmental impacts of training these huge models, as well as issues with data privacy and IP.

“The doomsday scenario is not a manufacturing A.I. transforming the entire planet into paper clips, as one famous thought experiment has imagined. It’s A.I.-supercharged corporations destroying the environment and the working class in their pursuit of shareholder value.” as Ted Chiang puts it in the New Yorker.

Last but not least, their argumentation does not even make sense. They don’t explain how the types of problems that they care about (synthetic media, manipulation) would lead to the extinction of humanity.

Background: X-Risk doomerism — a well-funded brand of “AI safety”

There are 2 camps for how academics, politicians and tech people think about the risks and harms posed by AI. And there’s quite a rift between them.

On the one hand, there is the group commonly referred to as AI ethicists who are concerned with the risks and impact of AI systems in the here and now. Take for example AI researcher Timnit Gebru, a former ethicist at Google, and her paper that focused on the problems with large language models, such as their environmental and financial costs as well as biased and discriminatory outcomes and potentials for deception.

On the other hand, there is a growing group of people whose main concern is that superintelligent AI might terminate our human civilization. This group often refers to itself as “AI safety” people. A big stakeholder of this ideology is the “Effective Altruism” community, a predominantly white male group of people who have increasing influence in politics.

A more extreme version of “Effective Altruism” is “Longtermism” a dangerous ideology which prioritizes the long-term future of humanity and de-prioritizes short-term problems. Their goal for humanity is to become “technologically enhanced digital posthumans inside computer simulations spread throughout our future lightcone” (Aeon).

Climate change and the increasing gap between the rich and the poor are seen as negligible problems. Nick Bostrom, the author of the book “Superintelligence”, called alleviating global poverty or reducing animal suffering “feel-good projects of suboptimal efficacy”.

Both Effective Altruism and Longtermist movements are backed by big money: Among many tech billionaires, Peter Thiel, Elon Musk and Sam Bankman-Fried have pumped money into Effective Altruism organisations. In 2021 the Effective Altruism movement was backed by $46 billion in funding.

Effective Altruism has an increasing impact on AI research: OpenAI was funded by Elon Musk and Peter Thiel. And last year, Sam Bankman-Fried proposed 100.000$ at the AI conference NeurIPS for papers on the topic of “AI safety”. Timnit Gebru summarizes:

“Research priorities follow the funding, and given the large sums of money being pushed into AI in support of an ideology with billionaire adherents, it is not surprising that the field has been moving in a direction promising an “unimaginably great future” around the corner while proliferating products harming marginalized groups in the now.” (Wired)

More “AGI apocalypse” rhetoric and suggestions

Intended or not, the “AI dilemma” campaign is dripping with fear-introducing suggestions of existential risk (X-Risk) and the AGI apocalypse (something they are not talking about, remember?)

1. They compare the danger of AI with the danger of nuclear weapons multiple times.
2. The narrative of “emergent capabilities” portrays the AI models as ticking bombs: “Suddenly”, GPT learned to speak Persian — “And no one knows why” Raskin breathes in the voice of a scary storyteller.
3. The phrase „Silently taught themselves research grade chemistry” suggests that these models have the autonomy to teach themselves and reinforces the myth that AI algorithms have agency.
4. The phrase “They make themselves stronger” reinforces the “AI takeover” myth. They say this makes them more dangerous than “nukes”.
5. When explaining RLHF (Reinforcement Learning from Human Feedback) they say it’s “about “How do you make AIs behave”. And they compare it to clicker training for dogs. This metaphor is already problematic because they compare an ML model to an intelligent animal, but they go further: When you go out of the room, the dog will do what it wants. This suggests that if you leave AI systems alone (“uncontrolled”), they will forget what you tell them and go rogue: “As soon as you leave the room they’re gonna not do what you ask them to do” (33:23). This is such a wrong and deeply problematic narrative, suggesting that these models have a free will on their own and need to be “tamed”.
6. Lastly, they chose the cheeky acronym “Gollem” to describe AI technology — in Jewish folklore, Golem refers to a human-like being created from inanimate matter. That’s another reference to AGI.

Conclusion

The “AI Dilemma” campaign is amalgaming X-Risk style alarmist rhetoric with a quite one-sided (social-media rooted) perspective on AI risks, especially regarding generative models. They mention many here and now risks, such as the proliferation of fake content and the speed in which these models are released to the public without enough assessment of the safety of the results. But Raskin and Harris are ignoring the long history and presence of negative effects of AI technology on marginalized groups.

Raskin and Harris are “tech designers turned media-savvy communicators” (Wired). They are masters in storytelling and persuasion. Tristan Harris started his career as a magician and then studied “persuasive technology” at Stanford. It is quite surprising though that the critique of persuasion and manipulation through social media has been a core theme in their work, however they apply persuasion mechanisms happily themselves.

It’s important to note that Raskin and Harris are no AI specialists. They fall victim to the same hype and misleading AI narratives as the general public does. Especially when these narratives are backed by big money. As mentioned earlier, the lobby behind X-Risk AGI doomerism is strong.

They say that AI is an abstract topic, and we are lacking metaphors to help us think about AI. This is something I can 100% agree with. They say that they want to provide the audience with more metaphors that are grounded in real life to give “a more visceral way of experiencing the exponential curves we are heading into”. If visceral means scaring people and then asking them to do breathing exercises to turn down their blood pressure, this is bad.

The “Social Dilemma” has shown that there is an audience for their flavour of one-sided populist technology criticism. But this is no education. It is highly questionable if tech criticism needs to be “entertaining”, and we should question who benefits from this framing.

Some ideas on how to stay informed on problems and risks associated with AI

If you want to broaden your knowledge about the real harms of AI, here’s a list of things you can do:

• Subscribe to the newsletter of AlgorithmWatch, a non-profit research and advocacy organization from Germany: https://algorithmwatch.org
• Follow these AI ethics researchers on Twitter: Timnit Gebru, Margaret Mitchell, Emily Bender, Ian Brown on AI regulation/policy
• Watch “Coded Bias” — it’s on Netflix ;-)
• Interactive explanations and technical background for topics such as bias, fairness and privacy: https://pair.withgoogle.com/explorables/
• Read the book “Atlas of AI” by Kate Crawford: https://www.katecrawford.net/
• Have a look at the AIAAIC database, that holds 1,000+ incidents and controversies driven by and relating to AI, algorithms, and automation: https://www.aiaaic.org/aiaaic-repository

In spring 2023 I had the opportunity to curate a conference at the Hochschule für Gestaltung Schwäbisch Gmünd as part of my researcher position at AI+D Lab and KITeGG.

From May 10–12, 2023, the third KITeGG summer school took place there. Under the title “reshape — forum for AI in Art and Design” we invited numerous international experts to get an overview of the many ways in which AI is relevant for designers.

The following text is a speech I gave at the opening of the conference on May 10 in the auditorium of the HfG!

reshape (1)

reshape is the name of a function in the Python programming language, or more precisely in the NumPy library, which is used in virtually all AI programs.

What you can do with NumPy is number crunching: it contains functions for working with vectors, i.e. lists of numbers and matrices. In machine learning, the whole world is mapped into numbers, words as well as images, sounds and movements. And these vectors are what goes into a neural network. The reshape function can change the shape of these vectors, for example, turn a 1-dimensional vector into a 2-dim vector, or a 3-dim into a 1-dim vector.

The reshape function “changes the shape without changing the content” says Numpy’s documentation.

reshape (2)

The conference you are at right now is also called reshape. Our slogan “reshape the landscape of art and design” — fits easily into the rhetoric we are surrounded by more often lately when it comes to AI: “disruption”, “revolutionize”, “blowing up”, “turn upside down”, “massive news” — We are experiencing a new AI hype right now.

It’s a bit surprising and feels like Déjà vu, because the last wave of AI hype/AI summer was not that long ago, circa 2014 with the breakthrough of the Deep Learning technique.

At the center of this new AI summer is “Generative AI” — systems like Stable Diffusion or ChatGPT that are capable of generating realistic artefacts like images or text.

Generative AI is nothing new

“Generative AI” has also been around for a long time, and artists and designers have always worked with it (albeit with changing technical foundations):

• In the 1980s — Harold Cohen & his program “AARON”. The first time that AI technologies were introduced into the world of computer art. The program could understand and generate colors and shapes. In this picture we see Cohen coloring the generated shapes by hand.
• In the 1990s: Karl Sims “Evolved Virtual Creatures”. Sims used evolutionary/genetic algorithms
• From 2015: Deep Learning early adopters: Addie Wagenknecht, Alex Champandard, Alex Mordvintsev, Alexander Reben, Allison Parrish, Anna Ridler, Gene Kogan, Georgia Ward Dyer. Golan Levin, Hannah Davis, Helena Sarin, Jake Elwes, Jenna Sutela, Jennifer Walshe, Joel Simon, JT Nimoy, Kyle Mcdonald, Lauren McCarthy, Luba Elliott, Mario Klingemann, Mike Tyka, Mimi Onuoha, Parag Mital, Pindar Van Arman, Refik Anadol, Robbie Barrat, Ross Goodwin, Sam Lavigne, Samim Winiger, Scott Eaton, Sofia Crespo, Sougwen Chung, Stephanie Dinkins, Tega Brain, Terence Broad and Tom White.

The artist & researcher Memo Akten was one of those “early adopters”. His video work “Learning to see” (2017) is a good example: Here you can see how Akten feeds his video input into a neural network in real-time, which then interprets this image data. A kind of semantic style transfer.

The work of artists has also always been an experiment with the shortcomings, the gaps and the glitches of these technologies, which often came directly from academic AI research and were “misappropriated” by them.

They often had to have a deep technical understanding of these systems to be able to bend and twist them in such a way. Even if algorithms were available as open source, designers had to really dig into it deeply.

A revolution of accessibility

What the current revolution is all about is a revolution in accessibility and availability. Technologies have opened up to a breadth of people.

They have been given a new interface that is usable by everyone, and that interface is called natural language.

This AI hype now even feels almost justified! Concrete consequences are already noticeable in various areas. It is an enormous push of innovations with an unbelievable speed.

A few innovations from the last 2 months (March — April 2023):

• VQGAN-CLIP: Here is a comparison of the quality of generated images 1 year ago — and the state of the art in April 2023 called “stable diffusion XL”.
• NVIDIA Video generation: https://www.youtube.com/watch?v=3A3OuTdsPEk Here a resolution of 2000x1000 pixel is achievable.
• GPT-4 has been released. This language model is multimodal, i.e. it can “understand” images (for example: sketch of website → working website code, or even derive complete recipes based on photos of meals).
• The Llama language model from Meta (comparable to GPT-3) runs on laptop CPU, smartphone and even on a Raspberry Pi (Link)

At the last KITeGG summer school (November 2022, HfG Offenbach), Stable Diffusion (released in August 2022) was the technology and the revolution everyone was talking about: Finally, everyone could generate arbitrary images simply by a text description.

Three months later, on November 22nd, ChatGPT came out and has since turned everything upside down. With ChatGPT, we have experienced the “Stable Diffusion Moment”, only for Large Language Models (LLMs).

Large Language Models (LLMs)

Large Language Models are neural networks with several billion parameters that have been trained using large amounts of text. LLMs find patterns in these large amounts of text and learn the statistical probability with which a word follows the next word.

This technique sounds as mundane as the autocomplete function on our cell phones, but it adds an astonishing level of complexity:

LLMs can summarize texts, translate, generate essays, write scientific papers, write working code, and generate ideas. You yourself will probably be able to tell your very own story of how you used ChatGPT and were surprised!

Some even go so far as to say that these systems are capable of “reasoning” — one of the holy grails of AI research.

These surprising capabilities have led to a number of people now claiming we have achieved or are at least close to achieving “AGI” (Artificial General Intelligence). And from this point in the discourse, it is hard to distinguish from science fiction. People talk about AI as if it were a being with its own will, and its own agenda.

You may have heard about the Open letter “Pause Giant AI Experiments” signed by prominent people. It calls for large AI labs to pause their research so that society and regulators can keep up.

This letter has drawn a lot of criticism; Emily Bender, a well-known AI researcher in the field of Natural Language Processing, wrote on Twitter that it was dripping with AI hype and myths. It also stems from an ideology called “Longtermism”, who have a very specific agenda for the “future of humanity”.

If we suspect a real counterpart, a thinking being, in these AI systems of automated pattern recognition, then perhaps it is like animals looking into a mirror. The journalist James Vincent calls this the “AI mirror test”.

Do we pass the “AI mirror test”?

The mirror test is used in behavioral psychology to find out whether a creature/animal has ego consciousness. There are a few variations of this test but the core question is: Does the creature recognize itself in the mirror or does it think it is another creature?

We as humanity are collectively facing a mirror test right now, and the mirror is called Large Language Models.

“The reflection is humanity’s wealth of language and writing, which has been strained into these models and is now reflected back to us. We’re convinced these tools might be the superintelligent machines from our stories because, in part, they’re trained on those same tales. Knowing this, we should be able to recognize ourselves in our new machine mirrors, but instead, it seems like more than a few people are convinced they’ve spotted another form of life.”

Source: “Introducing the AI mirror test, which very smart people keep failing”, James Vincent, The Verge 02/2023

If you compare this kind of discourse about AI with the (technical) reshape concept from the beginning, there are worlds in between! And between these 2 extreme poles are now also designers and artists.

I think it’s a pretty tumultuous time for creatives right now.

How does this all feel for designers?

On the one hand, we would like to see AI as a tool: Image generators are handy tools to visualize ideas or create renderings. Language models can help interaction designers code so they can build prototypes faster, etc. And new tools/integrations/improvements are popping up every day. The speed of innovation is immense.

On the other hand, we are repeatedly confronted in the media with the narrative of a general intelligence, a precursor to superintelligence, that solves complex design tasks with more effectiveness and creativity than one could ever do oneself. Being “replaced by AI” has recently become a concern of creative professionals like designers and software developers.

It really is a paradox: on the one hand, technology promises “superpowers for creatives”, on the other hand, those same creatives fear for their relevance and future.

What does the Reshape Symposium want?

With this conference we want to take a more differentiated look at the various points of contact between design and AI technology, away from the newly strengthened AI myths and black-and-white views of AI replacement.

We would like to take a differentiated look at AI systems and their technical properties instead of speaking in general terms of “an AI”.

We ask ourselves: Where does the responsibility of designers lie, what is their role, and how can they influence the course of AI development?

The conference will look at these issues from 3 axes:

• Designing for AI (Designing AI systems)
• AI for Design (Creative AI)
• Teaching AI (to creatives)

Designing for AI (Designing AI systems)

This involves the design of AI-based interfaces and products, for example systems that work via gesture recognition or voice input, or the design of generative interfaces themselves and integrations.

What are the challenges and opportunities here? What is the broader social context of these technologies? What do designers need to know in order to design these systems responsibly?

We have prepared a series of talks to address these questions:

• Nadia Piet — First thing in the morning, Nadia Piet talks about practices to design the user experience for AI-based systems and interfaces
• Catherine Breslin — Next up is a talk by Catherine Breslin on conversational design, how machines and humans are conversing, and how LLMs will change the future of voice assistants
• Ploipailin Flynn — Then Ploipailin Flynn talks about the dark side of pattern recognition and how social patterns like racism are notoriously reproduced by AI-based systems, and design strategies to deal with it
• Emily Saltz talks about synthetic media, AI-generated artefacts that will become more and more a part of our everyday lives, and what that means for product design

AI for Design (Creative AI)

How can AI technologies be integrated as tools in the creative toolbox of artists and designers? How can AI serve idea generation and foster human creativity instead of narrowing and flattening it? How do these tools “fit in the palm of your hand”?

Here we look forward to a talk on Thursday afternoon by design studio oio, who work out of a chic tiny house in the middle of London. In their workflows, they focus on “post-human collaboration” and develop products and tools for a “less boring future”.

Also, Tom White — one of those early adopters of AI technologies for creative use, will tell us about his experiments with Machine Vision and his latest projects.

Teaching AI (to creatives)

That is the main question of the KITeGG project: How can the topic of AI be communicated to creative people, especially to design and art students? What knowledge, and skills are important?

On which level of complexity do we move? From technical basics (we remember the Python function reshape) to high-level concepts and ethical issues like bias, privacy or IP rights: What level of depth is realistic to achieve?

How to develop an intuition for AI usecases? How to convey the ability to consciously assess the benefits and risks and decide when AI-based technologies should not be used?

To tackle these topics, we have prepared 2 panels: “AI Industry” and “KITeGG — Learnings from 1 year of AI education at design schools”.

And not to forget the workshops from earlier this week, the results of which will be presented on Friday.

I am looking forward to the upcoming 2.5 days of the conference with you!

• Reshape Konferenz Website: https://reshapeforum.hfg-gmuend.de/
• More about KITeGG, a German research project about the integration of AI into Art + Design Education: https://gestaltung.ai

Eröffnungsrede von “Reshape forum for Artificial Intelligence in Art and Design” (Mai 2023)

Im Frühjahr 2023 hatte ich die Gelegenheit eine Konferenz an der Hochschule für Gestaltung Schwäbisch Gmünd zu kuratieren.

Vom 10.-12. Mai 2023 fand dort die dritte KITeGG summer school statt. Unter dem Titel “reshape — forum for AI in Art and Design” luden wir zahlreiche internationale Expert*innen ein um einen Überblick über die zahlreichen Arten zu erhalten, in denen KI für Designer*innen relevant ist.

Der folgende Text ist eine Rede, die ich anlässlich der Eröffnung der Konferenz am 10. Mai in der Aula der HfG hielt!

reshape (1)

reshape ist der Name einer Funktion in der Programmiersprache Python, genauer gesagt in der Programmierbibliothek numpy, die bei so gut wie allen KI-Programmen Einsatz findet.

Was man mit numpy machen kann, ist Zahlenakrobatik: Es enthält Funktionen für das Arbeiten mit Vektoren, also Listen von Zahlen und Matritzen. Im Machine Learning wird alles in Zahlen abgebildet, Wörter genauso wie Bilder, Klänge und Bewegungen. Und diese Vektoren sind das, was in ein neuronales Netz hineingeht. Die Funktion reshape kann die Form dieser Vektoren verändern, z.B. aus einem 1-dim Vector, einen 2-dim Vector machen oder einen 3-dim Vector.

Die Funktion reshape “ändert die Form, ohne den Inhalt zu ändern” heißt es in der Dokumentation von numpy.

reshape (2)

Die Konferenz auf der Sie gerade sind heißt auch reshape — Unser Slogan “reshape the landscape of art and design” — reiht sich umstandslos ein in die Rhetorik von der wir in letzter Zeit häufiger umgeben sind wenn es um KI geht: “disruption”, “revolutionize”, “blowing up”, “turn upside down”, “massive news” — Wir erleben gerade einen neuen KI-Hype.

Es ist ein wenig verwunderlich und fühlt sich an wie ein Deja Vu, denn die letzte Welle des KI-Hypes/KI-Sommers ist noch gar nicht so lang her, ca. 2014 mit dem Durchbruch der Technik des Deep Learnings.

Im Zentrum dieses neuen KI-Sommers steht “Generative AI”, also Systeme wie Stable Diffusion oder ChatGPT, die in der Lage sind realistische Artefakte wie Bilder oder Text, zu generieren.

Generative AI is nothing new

Aber auch “Generative AI” gibt es schon länger und Künstler*innen und Designer*innen haben von jeher damit gearbeitet (wenn auch mit wechselnden technischen Grundlagen):

• 1980-er Jahre: Harold Cohen + sein Programm “AARON”, das erste mal dass KI-Technologien in die Welt der Computerkunst eingeführt wurden. Das Programm konnte Farben und Formen verstehen und generieren. Hier sieht man Cohen beim “ausmalen”
• 1990er-Jahre: Karl Sims — Evolved Virtual Creatures (benutzt Evolutionäre Algorithmen)
• 2015 Deep Learning, early adopters: Addie Wagenknecht, Alex Champandard, Alex Mordvintsev, Alexander Reben, Allison Parrish, Anna Ridler, Gene Kogan, Georgia Ward Dyer. Golan Levin, Hannah Davis, Helena Sarin, Jake Elwes, Jenna Sutela, Jennifer Walshe, Joel Simon, JT Nimoy, Kyle Mcdonald, Lauren McCarthy, Luba Elliott, Mario Klingemann, Mike Tyka, Mimi Onuoha, Parag Mital, Pindar Van Arman, Refik Anadol, Robbie Barrat, Ross Goodwin, Sam Lavigne, Samim Winiger, Scott Eaton, Sofia Crespo, Sougwen Chung, Stephanie Dinkins, Tega Brain, Terence Broad and Tom White.

Der Künstler+Researcher Memo Akten war einer dieser “early adopters”. Seine Videoarbeit “Learning to see” (2017) ist ein gutes Beispiel: Hier sieht man wie Akten in Echtzeit sein Videoinput in ein Neuronales Netz füttert, was diese Bilddaten daraufhin interpretiert. Eine Art semantischer Style-Transfer.

Die Arbeit von Künstler*innen war auch immer ein Experiment mit den Unzulänglichkeiten, den Gaps und den Glitches dieser Technologien, die oft direkt aus der akademischen KI-Forschung kamen und von ihnen “zweckentfremdet” wurden.

Oft mussten sie über ein technisches Tiefenverständnis dieser Systeme verfügen um sie derartig benden und twisten zu können. Auch wenn Algorithmen open source verfügbar waren, so musste man als Designer*in sich doch relativ tief “hinein-nerden”.

Eine Revolution der Zugänglichkeit

Was die jetzige Revolution ausmacht, ist eine Revolution der Zugänglichkeit und Verfügbarkeit. Die Technologien haben sich der Breite der Menschen geöffnet.

Sie haben ein neues Interface bekommen, was für jede*n benutzbar ist, und dieses Interface heißt menschliche Sprache (Natural Language).

Dieser KI-Hype fühlt sich nun sogar fast berechtigt an! Konkrete Folgen sind in verschiedenen Bereichen bereits spürbar. Es ist ein enormer Schub an Innovationen mit einer unglaublichen Geschwindigkeit .

Ein paar Innovationen aus den letzten 2 Monaten (März — April 2023):

• VQGAN-CLIP: Hier ist ein Vergleich der Qualität generierter Bilder vor 1 Jahr — und dem state of the art im April 2023 namens “stable diffusion XL”
• NVIDIA Video generation: https://www.youtube.com/watch?v=3A3OuTdsPEk Hier ist eine Auflösung von 2000x1000 pixel erreichbar
• GPT-4 wurde veröffentlicht. Dieses Sprachmodell ist multimodal, d.h. es kann Bilder “verstehen” (Beispiel Skizze von Website → funktionierender Website Code, oder auch komplette Rezepte auf der Basis von Fotos von Mahlzeiten ableiten)
• Das Llama Sprachmodell von Meta (mit GPT-3 vergleichbar) läuft auf Laptop CPU, Smartphone und sogar auf einem Raspberry Pi (Link)

Bei der letzten KITeGG summerschool (November 2022, HfG Offenbach), war Stable Diffusion (veröffentlicht im August 2022) die Technologie und die Revolution über die alle sprachen: Endlich konnte jedermensch einfach durch eine Textbeschreibung beliebige Bilder generieren.

3 Monate später, im November 22 kam ChatGPT raus und hat seither alles auf den Kopf gestellt. Mit ChatGPT haben wir quasi den “Stable Diffusion Moment” erlebt, nur für Large Language Models (LLMs).

Large Language Models (LLMs)

Large Language Models sind neuronale Netze mit mehreren Milliarden Parametern die mithilfe von großen Mengen Text trainiert wurden. LLMs finden Muster in diesen großen Mengen Text und lernen die statistische Wahrscheinlichkeit, mit der ein Wort auf das nächste Wort folgt.

Diese Technik klingt ersteinmal so banal wie die Autocomplete-Funktion auf unseren Handys, und doch ergibt sich eine erstaunliche Komplexität daraus:

LLMs können Texte zusammenfassen, übersetzen, Essays generieren, wissenschaftliche Papers verfassen, funktionierenden Code schreiben, Ideen generieren, und Sie selbst werden wahrscheinlich ihre ganz eigenen Geschichte erzählen können, wie sie ChatGPT genutzt haben und überrascht waren!

Man geht teilweise sogar so weit davon zu sprechen, dass diese Systeme “Reasoning” beherrschen (also logische Schlussfolgerungen ziehen können) — ein heiliger Gral der KI Forschung.

Diese überraschenden Fähigkeiten haben dazu geführt, dass es nun eine Anzahl von Leuten gibt die behaupten wir hätten “AGI” (Artificial General Intelligence) erreicht oder würden zumindest kurz davor stehen. Und ab diesem Punkt im Diskurs ist es schwer von Science Fiction zu unterscheiden. Man spricht von KI, als hätte wäre es ein Wesen mit einem eigenen Willen, und einer eigenen Agenda.

Vielleicht haben Sie von dem Open letter gehört “Pause Giant AI Experiments”, der von prominenten Personen unterschrieben wurde. Darin wird gefordert dass große KI-Labore eine Pause in ihrer Forschung einlegen, so dass die Gesellschaft und Regulatoren Schritt halten können.

Dieser Brief hat viel Kritik geerntet, Emily Bender, eine bekannte KI-Forscherin im Bereich Natural Language Processing, schrieb auf Twitter, er würde nur so von KI-Hype und Mythen triefen. Außerdem rührt er von einer Ideologie her, die sich “Longtermism” nennen, die eine ganz spezielle Agenda für die Zukunft der Menschheit haben.

Wenn wir in diesen KI-Systemen der automatisierten Mustererkennung ein echtes Gegenüber, ein denkendes Wesen vermuten, dann ist es vielleicht so wie wenn Tiere in einen Spiegel schauen. Der Journalist James Vincent nennt das den “AI mirror test”.

Bestehen wir den “KI Spiegel Test”?

Der Spiegeltest wird in der Verhaltenspsychologie dazu benutzt herauszufinden, ob ein Lebewesen/Tier über ein Ich-Bewusstsein verfügt. Es gibt ein paar Variationen dieses Tests aber im Kern ist die Frage: Erkennt sich das Lebewesen im Spiegel selbst, oder denkt es, es wäre ein anderes Lebewesen?

Wir sind als Menschheit gerade kollektiv mit einem Spiegeltest konfrontiert, und der Spiegel nennt sich Large Language Models.

“The reflection is humanity’s wealth of language and writing, which has been strained into these models and is now reflected back to us. We’re convinced these tools might be the superintelligent machines from our stories because, in part, they’re trained on those same tales. Knowing this, we should be able to recognize ourselves in our new machine mirrors, but instead, it seems like more than a few people are convinced they’ve spotted another form of life.”

Vergleicht man diese Art des Diskurses über KI mit dem (technischen) reshape-Begriff vom Anfang, so liegen da doch Welten dazwischen! Und zwischen diesen 2 Extrem-Polen stehen nun auch Designer*innen und Künstler*innen.

Ich denke, es ist gerade eine ziemlich aufwühlende Zeit für Kreative.

Wie fühlt sich das alles für Designer*innen an?

Auf der einen Seite möchten wir gern KI als ein Tool sehen: Bildgeneratoren sind praktische Tools um Ideen zu visualisieren oder renderings zu erzeugen. Sprachmodelle können Interaction-Designer*innen beim Coden helfen, damit sie schneller Prototypen bauen können, etc. Und jeden Tag schießen neue Tools/Integrationen/Verbesserungen aus dem Boden. Die Innovationsgeschwindigkeit ist immens.

Auf der anderen Seite sehen wir uns medial immer wieder konfrontiert mit dem Narrativ einer allgemeinen Intelligenz, einer Vorstufe zur Superintelligenz, die komplexe Designaufgaben mit einer Effektivität und Kreativität löst als man es selber je tun könnte. Von “KI ersetzt zu werden” ist seit kurzem ein Sorge von kreativen Berufen geworden wie Designer*innen und Software Entwickler*innen.

Es ist wirklich paradox: Einerseits verspricht die Technologie “Superkräfte für Kreative”, auf der anderen Seite fürchten eben jene Kreative um ihre Relevanz und Zukunft.

Was will das Reshape Symposium?

Wir wollen mit dieser Konferenz einen differenzierteren Blick einnehmen auf die verschiedenen Berührungspunkte von Design und KI-Technologie, abseits von den neu erstarkten KI-Mythen und schwarz-weiß Sichten von KI-Replacement.

Wir möchten KI-Systeme und ihre technischen Eigenschaften differenziert betrachten, statt generalisierend von “einer KI” zu sprechen.

Wir fragen uns: Wo liegt die Verantwortung von Designer*innen, was ist ihre Rolle, und wie können sie den Verlauf der KI-Entwicklung mitbestimmen?

Die Konferenz wird diese Themen von 3 Achsen her beleuchten:

1. Designing for AI (Designing AI systems)
2. AI for Design (Creative AI)
3. Teaching AI (to creatives)

Designing for AI (Designing AI systems)

Hier geht es das Design von KI-basierten Interfaces und Produkten, beispielsweise Systeme, die über Gestenerkennung oder Spracheingabe funktionieren, oder das Design von generativen Interfaces selbst und Integrationen.

Was sind hier die Herausforderungen und Möglichkeiten? Was ist der breitere gesellschaftliche Kontext dieser Technologien? Was müssen Designer*innen wissen, um diese Systeme verantwortungsvoll zu gestalten?

Für diese Fragen haben wir eine Reihe von Talks vorbereitet:

• Nadia Piet — Gleich morgen früh spricht Nadia Piet über Praktiken, um die User Experience für KI-basierte Systeme und Interfaces zu gestalten
• Catherine Breslin — Anschließend kommt ein Talk von Catherine Breslin über Conversational Design, darüber wie Maschinen und Menschen konversieren, und wie LLMs die Zukunft von Sprachassistenten verändern werden
• Ploipailin Flynn — Anschließend geht es bei Ploipailin Flynn um die dunklen Seiten von Mustererkennung und wie gesellschaftliche Muster wie Rassismus notorisch von KI-basierten Systemen reproduziert werden, und um Design-Strategien um damit umzugehen
• Emily Saltz spricht über synthetic media, also KI-generierte Artefakte, die in Zukunft immer mehr Teil unseres Alltags werden und was das für das Produktdesign bedeutet

AI for Design (Creative AI)

Wie lassen sich KI-Technologien als Werkzeuge in den kreativen Werkzeugkasten von Künstler*innen und Designer*innen integrieren? Wie kann KI der Ideenfindung dienen, und menschliche Kreativität fördern, anstatt sie zu einzuengen und zu verflachen? Wie “liegen diese Werkzeuge in der Hand”?

Hier freuen wir uns auf einen Talk des Design-Studios oio am Donnerstag nachmittag, die aus einem schicken tinyhouse mitten in London heraus arbeiten. In ihren Workflows setzen sie auf “post-human collaboration” und entwickeln Produkte und Tools für eine “less boring future”.

Außerdem wird uns Tom White — einer dieser “early adopters” von KI-Technolgien für kreative Nutzung, wird uns von seinen Experimenten mit Machine Vision erzählen und seinen neuesten Projekten.

Teaching AI (to creatives)

Das ist ja die Leitfrage von dem KITeGG Projekt: Wie kann man das Thema KI an Kreative, insbesondere an Design und Kunst studierende vermitteln? Welches Wissen, welche Skills sind wichtig?

Auf welcher Komplexitätsebene bewegt man sich? Von den technischen Basics (Wir erinnern uns an die Python-Funktion reshape) hin zu high-level Konzepten und ethischen Fragen wie Bias, Privatsphäre oder IP Rechten: Welcher Grad von Tiefe ist realistisch zu erreichen?

Wie kann man ein Gespür für KI-Usecases entwickeln? Wie vermittelt man die Fähigkeit die Vorteile und Risiken bewusst einzuschätzen und zu entscheiden, wann KI-basierte Technologien nicht genutzt werden sollten?

Dafür haben wir 2 Panels vorbereitet: “AI Industry” und “KITeGG — Learnings from 1 year of AI education at design schools”. Nicht zu vergessen sind auch die Workshops von Anfang der Woche, deren Ergebnisse am Freitag vorgestellt werden!

Ich freue mich auf die kommenden 2,5 Tage Konferenz mit Ihnen! Auf tolle Gespräche und Austausch zwischen verschiedenen Disziplinen!

Reshape Konferenz Website: https://reshapeforum.hfg-gmuend.de/

Mehr zu KITeGG: https://gestaltung.ai

1. 🎸 It’s a wild time for Generative AI
2. 🌄 A new era of (creative) AI has begun
3. 🗣️ Language input has democratized AI
4. 🔲 Foundation Models are here to stay
5. 🖼️ “AI art” is now mainstream
6. 😎 AI myths/bullshit & how to stay cool
7. 🧑‍🏫 What to teach? Expertise/Skillset/Literacy
8. 🧠 Building up ML Intuition
9. 🦮 Help for self-help
10. 🤓 Technical soft skills are equally important
11. ⚔️ Teaching a critical attitude towards AI
12. 🕹️ The evolution of no-code tools
13. 🥨 AI education needs constant and diverse integration into teaching
14. 👩‍🎓 There will always be these 3 types of students
15. 📚 The problem with knowledge bases
16. 🌎 The institutional landscape for Creative AI
17. Wrap up

In spring 2020 the University of Art and Design Burg Giebichenstein launched the XLab — the first dedicated laboratory for Artificial Intelligence & Robotics at an Art University in Germany. My focus was on (creative) AI/ML, while my colleague Simon Maris was responsible for everything related to (creative) robotics.

What is a “lab”? When one thinks of a lab, one usually thinks of research. This report, however, will be about teaching, namely, what it means to teach Artificial Intelligence at a university that trains future artists and designers.

Before we start — Is there a need to explain why AI should be taught at an art school in the first place? The question has many different answers to it, check out my article here!

At XLab we developed different formats, from a podcast to student lab “residencies”. In the summer of 2022, we exhibited some of our research results at Futurium in Berlin.

UPDATE 11/2022: I am working at the AI+D Lab at Hochschule für Gestaltung Schwäbisch-Gmünd, as part of KITeGG, a joint project of five German universities on the integration of AI in design teaching.

🎸1. It’s a wild time for Generative AI

The (research) field of Generative machine learning has seen a crazy boost in the last 2,5 years. The synthetic creation of text, sound and images is getting more realistic and accessible at an unprecedented speed.

• You can now create photo-realistic images based on your wildest imagination — right on your laptop. (Stable Diffusion)
• You can tell your editor to write code for you instead of typing it yourself, e.g. to build a website (CoPilot)
• You can even create realistic video sequences without ever using a camera lens
• Have you been to a Prompt Battle yet?

What does this mean to fields like art and design mostly concerned with creating (generating) things?

🌄 2. A new era of (creative) AI has begun

During the last 2 years, some things have changed fundamentally. It’s a development that started with GPT-3 and culminated with DALLE-2 and Stable Diffusion.

1. Language input has democratized AI
2. Foundational models
3. “AI art” is now mainstream

🗣 3. ️ Language input has democratized AI

The text-to-x paradigm enables a new democratization of AI tooling — the capability of writing in a natural language (mostly English) is everything that’s needed (and of course access to a computer and the internet)

🔲 4. Foundational Models are here to stay

The shift towards foundation models is a game changer. They are big models based on gigantic datasets that no single individual would be able to train on their own. These foundational models will not go away, they will only become more powerful.

What does this mean for us as creators? In the long term, it might shift our agency towards:

• Finetuning these models
• Combining different models
• Building integrations, interfaces and applications on top of these models

🖼️ 5. “AI art” is now mainstream

The conception that a computer can generate art has become commonplace. 2022 was the year that AI Art * became mainstream — mainly through the breakthrough of text2image tools like DALLE and Stable Diffusion.

Is this a bad thing? No. but it might require a redefinition of the identity of a lab that calls itself an “AI lab”. Or does it?

Many people don’t know that AI art existed long before 2022! There is a fascinating history of AI art, and there are entire research fields such as Computational Creativity that have existed for decades! One of my goals is to raise awareness of this history and the foundational work that has been done before the hype!

(*) At least the generative side of AI art. There are of course many creative applications that leverage other ML capabilities, like classification, e.g. with sensor data

😎 6. AI myths/bullshit & how to stay cool

AI Myths are still prevalent — even among the smart and critical species of art students! While the demystification of AI should be a core part of AI education, it’s also important to be patient.

At the opening presentation of the XLab in 2020, one student in the audience raised his hand and asked: “And where can I now borrow the AI”? Implying that there was an embodied entity sitting in some corner of our lab.

I admit — It used to put me in rage every time I heard the phrase “an AI” or “the AI” because the wording implied that there really was something like “an intelligence” — a sentient being that acts according to its own goals and intentions.

This isn’t just a grammar quirk, but a categorical error: It’s confounding narrow AI (what we have) for AGI or strong AI (sci-fi dream). In consequence, this misconception influences what we expect from AI technology and how we interact with it — also creatively.

But it’s also important to find the right moments for doing demystification. Hands-on interaction with AI technology is probably the best way of demystifying:

“Oh really? GPT can not even count properly??!”

🧑‍🏫 7. What to teach? Expertise/Skillset/Literacy

It can't be the goal to replicate a computer science curriculum at an art university. That’s simply not feasible and also not desirable! So what are the skills and knowledge that need to be taught? What should an art design student know about AI?

Fact is: Machine Learning is orders of magnitude more complex than, let’s say, Frontend-Development. Is that even a fair/just comparison? I think yes because both are sets of technologies instead of just one technology.

Doing machine learning “from scratch” does not only require some proficiency in the basic software-dev stack (Python, command line, git), but also machine learning skills and some math and statistics (which is what all ML is, essentially).

But what does “from scratch” even mean? It certainly means a different thing today than 5 years ago. There are always abstractions in various degrees.

In the same way, as there are drag-drop no-code tools to build websites, there are drag-drop no-code tools to do machine learning. But abstractions come with the drawback of limited freedom/customization.

In my teaching experience, I put an emphasis on explaining general concepts like supervised learning (lots of labelled data, correlations, optimization), instead of going deep into implementation details (gradient descent, etc.)

🧠 8. Building up ML Intuition

It's easy to teach the use of AI tools, but it's quite hard to teach and train intuition for the general capabilities of ML, when it is useful and when not.

I always cite Rebecca Fiebrink at this point. She asks:

“When and why is it creatively useful to find patterns, make predictions and generate new data?”

(Rebecca Fiebrink at her 2018 Eyeo Talk, Video)

Often students came to our lab who simply wanted to somehow integrate AI into their projects. Like a vitamin kick. Or even worse: With the hope that “the AI” will automatically spit out a whole project, like the design for a website, in an end-to-end manner.

🦮 9. Help for self-help

Often it is helpful to just mention and explain certain terminology, then the students can do an internet search themselves:

Sentiment Analysis, Facial Landmarks, Latent Space Walk

Terminology is key to helping students to navigate the field independently. And being able to navigate independently is key because teachers and a lab have limited capacities for individual support.

🤓 10. Technical soft skills are equally important

Abstracting your usecase, use right terminology in order to search for tutorials and code on a similar problem and adapting the code to your usecase.

Debugging skills, strategic googling, frustration tolerance. Desensitization towards the red colour of error messages.

These are “soft” skills (or should we call them “essential” skills?) that are equally important.

⚔️ 11. Teaching a critical attitude towards AI

My goal was and still is to train students to participate in the social discourse about AI as critical thinkers.

This includes raising awareness about the real problems with AI: Worker rights instead of robot rights. Bias issues and the automation of inequality.

For artists and designers this is equally important!

🕹️ 12. The evolution of no-code tools

In early 2020, RunwayML was our go-to starting point. It was the no-code AI tool that was easiest to use. Besides being like an “app store” for pre-trained ML models, RunwayML also offered the possibility to train custom models with your own dataset. That turned out to be a good instrument and platform for hands-on teaching. However, students required more flexibility.

Google Colab was a platform that we also used in teaching. Students could run open-source repositories they found on Github, without the need for an (often complicated) local setup. Computing power (GPU) could be rented on demand. Drawback: Google Technology.

There is a tradeoff between high-level GUI tools (RunwayML, Teachable Machine) and low-level toolchains (individual Python scripts, Colab Notebooks, etc.) in terms of flexibility

Now, text2x tools (like GPT and DALLE) are the newest generation of no-code AI tools!

🥨 13. AI education needs constant and diverse integration into teaching

It isn’t ideal to teach AI as an isolated subject. Instead, it needs constant integration in teaching, it needs consolidation. It should be taught as “salt” to different applications, “rather than its own food group” (as mimi onuoha and mother cyborg put it in their “People’s guide to AI”). For example:

• ML for film/video: video editing, semantic footage management, etc.
• ML for drawing: assisted drawing, etc.
• ML for interface design: making sense of sensory data, etc.
• and many more…

👩‍🎓 14. There will always be these 3 types of students

1. Those with no knowledge of programming and no ambition to learn it → use nocode tools
2. Those with some experience in programming but are missing ML foundations, can work on code level up to a certain degree
3. A few who are willing to tenaciously learn what is needed to build custom things

There will always be individual students vehemently committed to nerding out on something, and who go very far in that respect at certain points.

📚 15. The problem with knowledge bases

Knowledge bases can be a useful building block of AI education. However, knowledge bases require permanent maintenance because of the speed of development in the field. An up-to-date Google search usually brings more useful results than outdated knowledge. Persisted knowledge becomes stale.

🌎 16. The institutional landscape for Creative AI (Institutions and Ecosystem in Germany and Europe)

When we started the lab in 2020, there was only a small amount of German institutions that had an interest in creative AI, mostly as part of a broader critical AI discourse:

• KIM (Karlsruhe University of Arts and Design)
• Schaufler Lab (Technical University Dresden)

In the UK there were

• the Creative AI lab (a collab of Serpentine R&D & King’s College London)
• UAL Creative Computing Institute (with Creative AI education pioneer Rebecca Fiebrink)

In the US, there were many pioneering institutions and communities that very early acknowledged the potential of ML for art and design: The school of Poetic computation, Frank Ratchye Studio for Creative Inquiry, the Processing community, …

Not to forget, the online community AIxDesign was already striving, and still is! Check them out :-)

In December 2021 the German research project KITeGG (“Making AI Tangible and Comprehensible: Connecting Technology and Society Through Design”) was launched as a collaboration of 5 German art universities. One of its outstanding goals is to develop its own computing infrastructure that frees users from being reliant on Google’s cloud infrastructure.

17. Wrap up

I am far from having all the answers to the question: How do you teach AI at a university of the arts? And the answers have to be found anew in parallel to the constant developments.

We need to talk about what is considered basic or essential AI skills in art and design education:

How important are technical implementation skills really?

• Coding skills and proficiency in coding ecosystems
• Deep learning knowledge, including math and statistics

How much should we invest in “discourse skills”?

• AI ethics (bias, intellectual property, power structures, worker rights, environmental issues)
• Some immunity against AI bullshit in the media
• An intuition for the limits and drawbacks of AI solutions, when to say ‘no’

Fortunately, there is KITeGG, where implicit and explicit research is being done on these questions! And I am excited to be a part of that!!

Appendix

XLab

The XLab was part of the EFRE-funded project “BurgLabs” together with its two sister labs for sustainability (SustainLab) and biotechnology (BioLab) at the University of Art and Design Burg Giebichenstein in Halle, Germany. It started in May 2020.

AI+D Lab

Lab for AI and Design at the University of Design Schwäbisch-Gmünd, within the framework of the KITeGG research project.

KITeGG

A four-year BMBF-funded research project on design and artificial intelligence that started in December 2021. Behind KITeGG are 5 German (art) universities. Their research goal is among others to develop teaching concepts related to the topics of “Creative Machine Learning”.

This article explains VQGAN+CLIP, a specific text-to-image architecture.

You can find a general high-level introduction to VQGAN+CLIP in my previous blog post “VQGAN+CLIP — How does it work?”

Here I am looking at the specific VQGAN+CLIP implementation written by artist/programmer Katherine Crowson (aka @RiversHaveWings) which went viral in the summer of 2021.

📍 To be exact, I am looking at this Google Colab notebook. (Be aware that there might be newer versions of this notebook with more cool optimizations.)

↕️ Tip: I suggest turning on line numbering in Google Colab: Tools → Settings → Editor → Show line numbers

🤓 Extra: There is a little dictionary of Machine Learning terms at the bottom of this article: general terms and more specific terms from this implementation. Because I love dictionaries.

General facts about this notebook

• It uses PyTorch, a popular machine learning framework written in Python
• It connects two existing (open-source, pretrained) models: CLIP (OpenAI) and VQGAN (Esser et al. from Heidelberg University)
• It is structured in the following sections/cells:
  — Setup, Installing libraries
   — Selection of models to download
   — Loading libraries and definitions
   — Implementation tools
   — Execution

A high-level overview of the algorithm

From my older blog post “VQGAN+CLIP: How does it work?”: “CLIP guides VQGAN towards an image that is the best match to a given text.”

Because CLIP is able to represent both text and images in the same feature space, we can easily calculate the distance between these two.

Here’s a simple visualization of the algorithm. The cycle represents one optimization iteration.

A core concept: Inference-by-optimization

In Machine Learning there is this core distinction between training and inference:

• Training is the optimization process of finding the right weights of your model in order to minimize a loss function.
• Inference is the process of using a pre-trained model to make predictions

Training is most often the resource-intensive part requiring a GPU for effective computation. Inference is, for most models, a rather light operation, it could run on a CPU and sometimes even on edge devices (such as a mobile phone or a Raspberry Pi).

The VQGAN-CLIP architecture kind of blurs the distinction of training-vs-inference, because when we “run” VQGAN-CLIP we’re kind of doing inference, but we’re also optimizing.

This special case of inference has been called “inference-by-optimization”. That’s why we need a GPU to run VQGAN-CLIP.

Variable naming choices and what they refer to

• Perceptor → CLIP model
• Model(also sometimes named the “Generator”) → VQGAN model
• Prompt → the model we’re going to train when we run the notebook
• z → A vector as input for VQGAN for synthesizing an image
• iii → A batch of CLIP-encoded image cutouts

The notebook step by step

STEP 0. Downloading the pre-trained models (CLIP & VQGAN)

First CLIP and VQGAN repositories are git-cloned (Cell “Setup, Instaling Libraries” lines 6 and 9).

Then we also download a pre-trained VQGAN model (Cell “Selection of models to download”): For every model there is a .yaml file containing basic model parameters and a .ckpt file that contains the weights of the pre-trained model (called a checkpoint).

The pretrained CLIP model download is a bit harder to spot: It happens in the clip.load() function (cell “Excecution”, line 16), which is documented in CLIP’s Github repository as follows: “Returns the model (…). It will download the model as necessary. The name argument can also be a path to a local checkpoint.”

STEP 1. Generating the initial z vector (Cell “Excecution”, line 29–36)

We generate an intitial VQGAN-encoded image vector. In case we’ve input a starting image this will be the VQGAN embeddings ( model.encode ) for this image. In case the user hasn’t provided a starting image it will be a tensor filled with random integers (torch.randint ), aka a random noise vector. This VQGAN-encoded image vector is referred to as z.

STEP 2. Initializing the optimizer with z (Cell “Execution”, line 39)

opt = optim.Adam([z], lr=args.step_size)
The first argument in the constructor contains the parameters that you wish to optimize, in our case it’s z, aka the image vector with which we start the optimization process.

STEP 3. Instantiating the Prompt models for every text prompt (Cell “Execution”, line 46–49)

For every text prompt provided by the user:
 — we encode it with CLIP
 — and with this encoding, we create our own Prompt model. These models are what we’re going to train when we run the notebook
 — we add this model to an array named pms (I guess this stands for “prompt models”)

STEP 4. The actual optimization Loop (Cell “Execution” line 134–144)

This simple loop does nothing more than call the train() function as many times as defined in max_iterations(as set by you the user in the cell named “Parameters”). Note: if it’s set to -1 (the default) the loop will go on forever (until you stop the cell manually or an error occurs).

The actual optimization procedure

Forward pass: We start with z, a VQGAN-encoded image vector, pass it to VQGAN to synthesize/decode an actual image out of it, then we cut it into pieces, then we encode these pieces with CLIP, calculate the distance to the text prompt and get out some loss(es).

Backward pass: We backpropagate through CLIP and VQGAN all the way back to the latent vector z and then use gradient ascent to update z.

ascend_txt()
Synthesizes an image with VQGAN, cut the image into pieces, encode the image pieces with CLIP, then pass them to the Prompt model(s) to calculate the loss(es), and finally saving the image to disk. (see detailed explanation of ascend_txt below)

loss.sum()
Summing the losses. Remember, loss is an array of tensors. This works in PyTorch: If you have multiple losses you can sum them up and then only call backward() once.

loss.backward()
Computing the gradients for all losses (it does not yet update the weights)

opt.step()
Calling the optimizer (which we initialized in step 2) to update z.

⛰️ What happens in ascend_txt ?

This function formulates the loss terms for optimization. It returns an array of losses per prompt.

Why is it named ascend_txt? It refers to gradient ascent, which works the same manner as gradient descent (just with a different goal: maximization instead of minimization of some function).

Here’s what’s happening in ascend_txt:

out = synth(z)
We’re synthesizing an image with VQGAN ( model.decode ): based on z, the vector that is being optimized in every training step

iii = perceptor.encode_image(normalize(make_cutouts(out)))float()
We create a batch of cutouts from this image and encode them with CLIP (see detailed explanation about MakeCutouts below)

for prompt in pMs:
result.append(prompt(iii))
We go over each of our “Prompt” models (instances of the Prompt class) and pass the cutout-batches through it in order to calculate the loss per prompt (see detailed explanation about the Prompt model below)

imageio.imwrite(filename, np.array(img))
add_stegano_data(filename)
We save the image and add metadata to it via steganography (see detailed explanation below about Steganography)

return result
We return an array of losses (loss per prompt).

🔥 Is it CLIP? Is it VQGAN? What exactly is being trained or optimized in this notebook?

We’re not training a VQGAN model and we’re also not training a CLIP model. Both models are already pretrained and their weights are frozen during the run of the notebook.

What’s being optimised (or “trained”) is z , the latent image vector that is being passed as an input to VQGAN.

🔥 The Prompt class

A model called “Prompt” is the place where we calculate how similar image and text are (the loss). There might be more than one of these models in case that a) there was more than one prompt in the user input, or b) a destination image was defined by the user.

The class Prompt subclasses the PyTorch NN module base class. Note: When calling an instance (let’s say we named itprompt ) of the Prompt class withprompt()we are actually calling the forward() method of the Prompt class.

The forward() function of the Prompt model is the core of the algorithm, it calculates how similar image and text are.

It’s important to understand what is referred to by input , self.embed and dist here!input is the (CLIP encoded) image. Or more accurately: a batch of image cutouts.self.embed is the (CLIP encoded) text prompt that the model was instantiated with.dists stands for “distance” and refers to the mathematical distance between the embeddings of input and text.

The return value (the actual loss) of the forward function refers to this distance. It is actually a tensor representing the loss that looks like this:

tensor(1.0011, device=’cuda:0', grad_fn=<MulBackward0>)

(btw, the grad_fn property means that a previous function (MulBackward0) resulted in having the gradients calculated. History is always maintained in these PyTorch tensors, unless you specify otherwise)

✂️ MakeCutouts

CLIP can only deal with low-resolution images as input. However, VQGAN is capable of creating high-resolution images. In order to compromise between the two, we cut the image into pieces and pass them to CLIP as a batch.

On top of cutting the images into pieces MakeCutouts does a couple of image transformations: Distortions, horizontal flip, add blur, and more. It uses kornia for this, a computer vision library that provides functions for image augmentations.

Why these transformations? In his wonderful blogpost Ryan Moulton explains:

“Much like how digital artists flip their canvas to double check their proportions, and artists in traditional media will rotate around their canvas to view it from different angles as they’re working, giving CLIP randomly rotated, skewed, slightly blurred images produces much better results.”

Other interesting aspects of this notebook (Steganography, etc.)

🕵️‍♀️ Steganography

The notebook uses steganography to add metadata to each generated image saved on disk, such as: text prompt, the type of model, the random seed, the iteration number and more.

This is handy in case you want to keep track of your experiments, reproduce some of your own results or see how other people created the images they published on the internet.

Steganography is the practice of concealing messages in a file or a physical object that can not be noticed by the naked eye. This notebook uses the “LSB” steganography technique, which stands for “least significant bit”. The idea is to store the hidden message by overwriting the least significant bit of each pixel of the image. Here’s a quick explainer of LSB.

Want to quickly check which information is hidden in an image? You can use an online Stegano Decoder like this one.

Be aware that once you edit the image (e.g. change the filesize or do some Photoshop edits) this hidden information will be gone.

A little dictionary

Checkpoint

A capture of the models internal state (weights and other parameters) at a certain time in training. Necessary for inference or for resuming training.

Embedding

A low-dimensional, learned vector representation into which you can translate high-dimensional vectors. Generally, embeddings make ML models more efficient and easier to work with.

Loss

A measure of how different a model’s predictions are compared to the actual label of the data. Basically a measure of how bad the model is. To determine the loss, a model must define a loss function.

Loss function

A function for calculating the loss.

One-hot encoding

A technique for representing categorical data. The encoding has the shape of a matrix with binary data.

Seed

The number used to initialize the state of a (pseudo-) random number generator. If you use the same seed the generator will produce the same output. This is useful for reproducibility.

Steganography

Steganography is the practice of concealing a messages in a file or a physical object that can not be noticed by the naked eye.

Tensor

A type of data structure or mathematical object that is similar to a vector or a matrix. Mathematically, tensors are a superset of vectors. In PyTorch it’s the core data structure: all the inputs and outputs of a model, as well as the model’s parameters and learning weights, are expressed as tensors.

• Video: Mathematical explanation
• Video: Tensors in PyTorch

Vector Quantization

A technique for easing computation (which also minimizes carbon footprint). It replaces floating points with integers inside the network.

Z-Vector

A vector containing random values from a Gaussian (normal) distribution. It is usually passed as the input into a pretrained GAN generator which results in generating a real-looking fake image.

Cool Resources

• Blogpost by Ryan Moulton on how he tweaked the code of VQGAN+CLIP to generate interesting panorama images: https://moultano.wordpress.com/2021/08/23/doorways/
• An explainer on the CLIP+BigGAN implementation by Hao Hao Tan: https://wandb.ai/gudgud96/big-sleep-test/reports/Image-Generation-Based-on-Abstract-Concepts-Using-CLIP-BigGAN--Vmlldzo1MjA2MTE
• “VQGAN+CLIP how does it work” by Alexa Steinbrück https://alexasteinbruck.medium.com/vqgan-clip-how-does-it-work-210a5dca5e52
• “Alien Dreams: An Emerging Art Scene” by Charlie Snell: https://ml.berkeley.edu/blog/posts/clip-art/
• Interview (video) with Ryan Murdock (aka @advadnoun) by Derrick Schultz (Artificial Images): https://www.youtube.com/watch?v=OYWGSDeQYlc
• Twitter thread by Tanishq Mathew Abraham: https://twitter.com/iScienceLuvr/status/1468115406858514433
• “The illustrated VQGAN” by Lj Miranda, especially the Appendix with a diagram of the “family tree” of VQGAN+CLIP and how it came to be! https://ljvmiranda921.github.io/notebook/2021/08/08/clip-vqgan/