We just raised a $30 million Series ARead more
View more posts
2021-04-0213 minute read

Workflows are coming to Supabase

Francesco CecconEngineering

This week we launched Supabase Storage, which leaves one other huge piece of the stack that everyone is asking for: Functions.

TLDR

We're not releasing Functions today. Trust us, we know you want them. They are coming, just not today.

But we are building something that we think you're going to like: Workflows. We haven't finished building it yet, but Workflows are a "piece" of the Function story and arguably an even more exciting feature.

Everyone wants functions

Firebase Functions

Firebase functions are relatively simple. If you use Serverless, AWS Lambda, Cloudflare Workers, Next.js API routes, or Netlify Functions, then you know how they work. A Firebase function executes some code which you provide, without you managing a server.

Specifically for Firebase, they have another key feature - they can be triggered by database events. For example, you can trigger a function whenever a Firestore Document is updated.

This is great, but it is still limited for a few real-world use cases. For example, what if you want to send an email to a user one day after a user signs up. Or one year? There is no queuing functionality in Firebase. You'd have to manage a process like this yourself, probably using a cron-job.

A better solution?

We searched for some open source tools which we think are solving this problem well. We looked at NodeRed, n8n, Airflow, and about 10 other tools. They are amazing tools on their own, but for the Supabase Stack they ultimately had the same shortcomings that we found with Storage providers - most of them lacked deep Postgres integration.

We went back to the drawing board and asked, "if we could wave a wand and get the perfect solution, what would it look like?". The tool that came very close is AWS Step Functions. The only problem: it's not open source. Luckily, their state language is.

Using this states language, we are building an open source orchestration engine for managing very complex Workflows with queueing, etc. It will be built with Elixir.

This engine won't execute code. Instead, it will coordinate and execute existing functions wherever they live: AWS, GCP, Azure, OpenFaas, and of course Postgres.

We plan to add "modules" which work natively: email services, notification services, and other platforms.

The engine is deeply integrated with Postgres. Jobs, queues, and logs will be stored and accessible by SQL. We'd like to give a shoutout to the Oban team here, their robust job processing was a big missing piece for the engine. And most importantly, it's backed by Postgres!

Once ready, we will make this available in the Supabase Dashboard with a Zapier-like interface.

What are Workflows

Workflows orchestrate and execute functions in response to a database event (insert, update, delete) or a HTTP call (direct invocation).

You can use them to rapidly develop microservices (once we have functions) without worrying about servers.

Workflows are stateless - the output of a state becomes the input of the next state.

Workflows are defined using Amazon States Languages, so you can import your workflows from AWS (although we are still building handlers for most AWS resources).

Workflows can be persistent (the default). This means they are tolerant to server restarts, but it also means they need to use the database to store their state.

Workers can be transient. These are fully in-memory if you don't want to store the execution state (for example, IoT applications that trigger workflows very often). Transient workflows are not restarted if the server crashes or is restarted.

Example

A typical use-case for workflows is sending emails. For example, you might want to send a user an email one day after they sign up. In database terms we can say: "trigger an email workflow whenever there is an insert on the users table."

Let's break this down into steps, then tie it all together at the end:

Sending an email

1SendEmail:
2  Type: Task
3  Next: Complete
4  Resource: my-email-service
5  Parameters:
6    api_key: my-api-key
7    template_id: welcome-email
8    payload:
9      name.$: '$.record.name'
10      email.$: '$.record.email'
11

Here we have a "Task" which triggers a call to an email service (like Mailgun or Postmark). Specifically, it's telling the service to send the welcome-email template, and it's providing it a name and an email as parameters.

Waiting a day

Since we don't want to send the email immediately, we need to tell Workflows to wait one day

1WaitOneDay:
2  Type: Wait
3  Next: SendEmail
4  Seconds: 86400
5

Here "one day" is specified in seconds.

Trigger on insert

We mentioned that you could trigger a workflow whenever there is an "insert" on the users table. But what if you insert multiple users at once? Not a problem - we can loop through all the inserts with a Map:

1EmailUsers:
2  Type: Map
3  End: true
4  InputPath: '$.changes'
5  Iterator:
6    StartAt: CheckInsert
7    States:
8      CheckInsert:
9        Type: Choice
10        Default: Complete
11        Choices:
12          - Variable: '$.type'
13            StringEquals: INSERT
14            Next: WaitOneDay
15

In this part, we have a task "EmailUsers", which iterates through all the database events ($.changes) and checks if they are INSERTs.

Tying it all together

Let's see how it looks all together:

1---
2Comment: Email users after one day
3StartAt: EmailUsers
4States:
5  EmailUsers:
6    Type: Map
7    End: true
8    InputPath: '$.changes'
9    Iterator:
10      StartAt: CheckInsert
11      States:
12        CheckInsert:
13          Type: Choice
14          Default: Complete
15          Choices:
16            - Variable: '$.type'
17              StringEquals: INSERT
18              Next: WaitOneDay
19        WaitOneDay:
20          Type: Wait
21          Next: SendEmail
22          Seconds: 86400
23        SendEmail:
24          Type: Task
25          Next: Complete
26          Resource: send-templated-email
27          Parameters:
28            api_key: my-api-key
29            template_id: welcome-email
30            payload:
31              name.$: '$.record.name'
32              email.$: '$.record.email'
33        Complete:
34          Type: Succeed
35

The workflow receives the following JSON data from Supabase Realtime:

1{
2  "changes": [
3    {
4      "columns": [
5        {
6          "flags": ["key"],
7          "name": "id",
8          "type": "int8",
9          "type_modifier": 4294967295
10        },
11        {
12          "flags": [],
13          "name": "name",
14          "type": "text",
15          "type_modifier": 4294967295
16        },
17        {
18          "flags": [],
19          "name": "email",
20          "type": "text",
21          "type_modifier": 4294967295
22        }
23      ],
24      "commit_timestamp": "2021-03-17T14:00:26Z",
25      "record": {
26        "id": "101492",
27        "name": "Alfred",
28        "email": "alfred@example.org"
29      },
30      "schema": "public",
31      "table": "users",
32      "type": "INSERT"
33    }
34  ],
35  "commit_timestamp": "2021-03-17T14:00:26Z"
36}
37

Next Steps

We've already open sourced the Workflow interpreter here. It's built with Elixir, so you can find it on Hex here.

After we've ironed out a few bugs we will integrate it into the Supabase Stack. As with all Supabase features, we'll add a nice UI to make prototyping extremely rapid. We'll integrate the UI with the code (via Git) to make sure everything is version controlled.

Related articles
Updates for Supabase Functions
Workflows are coming to Supabase
View all posts
Last post

Supabase Beta March 2021

April 6, 2021
release-notes
Next post

PgBouncer is now available in Supabase

April 2, 2021
databaseengineering

Build in a weekend, scale to millions