The AI revolution is now open source

Aincrad is reborn from The Seed.

One of my favourite anime series of all time is the first season of Sword Art Online. It’s a story about your typical massive multiplayer online role-playing game (MMORPG), an earlier Metaverse to all of you young ones, this is an immersive VR space where users log in using a device called NerveGear, while their consciousness experiences the virtual world, their body is asleep. The creator of the game changes the settings so that all players are trapped inside, their bodies in a coma, while their minds inhabit the virtual space. After a lot of adventures, most of our heroes manage to escape and to wake up. I won’t spoil the rest of the series, but it’s great entertainment for gamers like me who have spent hours trapped in MMORPGs of our own volition, but I digress. After everyone wakes up, there’s a regulatory review over the game, which is eventually shut down. What is interesting for this blog post is that the evil developer that created the game arranges for it to be released to the public as an open source programme called “The Seed”. This allows other developers around the world to replicate their own versions of the game. The ethics of this deployment of a potentially dangerous technology are not really discussed, it just happens. The interesting aside is that regulatory efforts are impossible because everyone in the world has access to The Seed, and can launch their own version of the game. Decentralisation wins!

I have been reminded of Sword Art Online because of the recent explosion in artificial intelligence art tools. While I have discussed mostly DALL·E in previous blog posts, one of the most fascinating tools deployed recently has to be Stable Diffusion by Stability.ai. In a field that has been dominated by Google and OpenAI, Sability.ai practically emerged out of nowhere. The company got started at the end of 2020 by Emad Mostaque, and it has 35 employees listed on LinkedIn. But don’t be deceived by the relative youth and size of the company, this may sound like hyperbole, but I believe that this little startup is probably set to change the AI landscape as we know it.

Almost everyone who is using Stable Diffusion seems to agree that it is superior to other tools such as MidJourney and DALL·E. The reason for this seems to be a combination of working with an extremely large dataset consisting of over 5 billion image and text pairs, as well as using diffusion models developed by its researchers, specifically working with latent diffusion models. Not only is Stable Diffusion considered to be better, but they have made waves by releasing the code to the public under as open source (code in Github here). This has turned what could be a niche tool into potentially ground-breaking development.

Why is this release so important? Open source has become a standard in software development, so much so that even large companies such as Microsoft now rely on it. What I think makes Stable Diffusion different is that it is rare for such a startup to release its code so early in the development cycle. AI art has exploded since last year with the widespread adoption of diffusion models across the nascent industry, and releasing such a tool to the public could accelerate developments in this area. This is already taking place, the code was released in August 22, and there are already dozens of applications that are adding to the release, a well as creating accompanying tools. Moreover, the code itself can even run in desktop computers, permitting users with some technical knowledge to try the software by themselves.

img2img in action with Doom.

A few developments that I have seen so far that I like:

• image2image, which allows people to draw a basic image and have the system improve on it.
• A Photoshop plugin that allows you to use SD to improve your own images.
• There’s also an upcoming plugin with design app Figma to use SD in designing products.

There are so many different possible applications, some of which we probably haven’t even thought about, it seems like every day brings about a new improvement.

So what about the licensing? There are two interesting aspects here that I wanted to highlight, mostly because Stable Diffusion is one of the most free AI projects (free as in freedom, not as in beer) that I have encountered in this space. Unlike OpenAI, which claims ownership over images created with DALL-E, and MidJourney, which claims ownership for some types of non-paying accounts, StableDiffusion releases all images created with its online tool (called DreamStudio) and with their Discord app with the Creative Commons CC0 licence. In their terms of use, Stability.ai says:

“All users, by use of DreamStudio Beta and the Stable Diffusion beta Discord service hereby acknowledge having read and accepted the full CC0 1.0 Universal Public Domain Dedication (available at https://creativecommons.org/publicdomain/zero/1.0/), which includes, but is not limited to, the foregoing waiver of intellectual property rights concerning any Content. User, by use of DreamStudio Beta and the Stable Diffusion beta Discord service, acknowledges understanding that such waiver also includes a waiver of any such user’s expectation and/or claim to any absolute, unconditional right to reproduce, copy, prepare derivate works, distribute, sell, perform, and/or display, as applicable, and further that any such user acknowledges no authority or right to deny permission to others to do the same concerning the Content.”

I think that this is completely unprecedented, users of these two online tools acknowledge that all of their works will be, for all practical effects, in the public domain. This has the interesting effect of entirely bypassing all copyright authorship and ownership debate regarding AI outputs produced with Stable Diffusion’s own DreamStudio tool. It doesn’t matter if the works have copyright or not, by all practical effects they do not. CC0 acts as a dedication to the public domain wherever that is possible, and as a full licence of all of the exclusive rights of the author wherever it’s not possible to do so (relinquishing copyright is complicated).

Edit: It’s important to point out that the ToS only cover two tools, if you download Stable Diffusion and run it on your own, or if you run any other tool based on SD, these terms do not apply, and the images may have copyright (following the discussion here).

The second interesting licensing issue is the open source release itself. While most open source software is released with the GPL, or with an academic licence such as MIT, Stable Diffusion is being released using a new set of licences called OpenRAIL. These are licences that are written specifically with machine learning in mind, much in the same way that we have specific open licences for data and content. There are two guiding principles with OpenRAIL licences:

• Open: these licenses allow royalty free access and flexible downstream use and re-distribution of the licensed material, and distribution of any derivatives of it.
• Responsible: OpenRAIL licenses embed a specific set of restrictions for the use of the licensed AI artifact in identified critical scenarios. Use-based restrictions are informed by an evidence-based approach to ML development and use limitations which forces to draw a line between promoting wide access and use of ML against potential social costs stemming from harmful uses of the openly licensed AI artifact. Therefore, while benefiting from an open access to the ML model, the user will not be able to use the model for the specified restricted scenarios.

Stable Diffusion uses the Creative ML OpenRAIL-M licence. On the face of it, this is a standard open source licence that covers the model, the code, and derivatives (not to be mistaken with the image outputs, these are independent of the code licensing). The licence grants the licensee “a perpetual,worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare, publicly display, publicly perform, sublicense, and distribute the Complementary Material, the Model, and Derivatives of the Model.” It also has a grant of patent, pretty standard as well.

The innovative part is in the copyleft section, that is, the restrictions imposed on licensees. This section includes a list of ethical precepts that must be kept in any re-use of the licence. Traditionally, copyleft clauses have been used to keep the software free down a distribution chain. Increasingly there are calls to include ethical clauses in software licences. The restrictions included in this OpenRAIL licence are:

• In any way that violates any applicable national, federal, state, local or international law or regulation;
• For the purpose of exploiting, harming or attempting to exploit or harm minors in any way;
• To generate or disseminate verifiably false information and/or content with the purpose of harming others;
• To generate or disseminate personal identifiable information that can be used to harm an individual;
• To defame, disparage or otherwise harass others;
• For fully automated decision making that adversely impacts an individual’s legal rights or otherwise creates or modifies a binding, enforceable obligation;
• For any use intended to or which has the effect of discriminating against or harming individuals or groups based on online or offline social behavior or known or predicted personal or personality characteristics;
• To exploit any of the vulnerabilities of a specific group of persons based on their age, social, physical or mental characteristics, in order to materially distort the behavior of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;
• For any use intended to or which has the effect of discriminating against individuals or groups based on legally protected characteristics or categories;To provide medical advice and medical results interpretation;
• To generate or disseminate information for the purpose to be used for administration of justice, law enforcement, immigration or asylum processes, such as predicting an individual will commit fraud/crime commitment (e.g. by text profiling, drawing causal relationships between assertions made in documents, indiscriminate and arbitrarily-targeted use).

I’m not averse to ethical licences at all, but I fear that a list such as this is practically impossible to enforce, and it turns Stable.ai and its licensees into an ethical police, trying to look at the actions of downstream developers. Having said that, I think that ethical open licences are an interesting development following up from the original ideas of copyleft licences, which were created precisely to code ethics into software development, namely the sharing ethos of free software.

Concluding

I think that opening the source code is a bold move by Stability.ai. The effect is that if the company were to disappear tomorrow, the source code will remain, replicating everywhere. I’m not sure about the length of the ethical precepts contained in the copyleft section of the OpenRAIL licence, but I’m willing to be proven wrong that this is workable. I leave you with a picture generated in MidJourney with the prompt “utopian city ruled over by open source licences”.