diff --git a/PHASE_1_VERTICAL_SLICE.md b/PHASE_1_VERTICAL_SLICE.md index e0caebe92..aa1ab270c 100644 --- a/PHASE_1_VERTICAL_SLICE.md +++ b/PHASE_1_VERTICAL_SLICE.md @@ -2,19 +2,18 @@ **Branch:** `feat-demo-1-vertical-slice` -**Goal:** Implement minimal end-to-end flow execution - from UI button click through Edge Function to real-time status updates. Validates entire integration stack with **client-side auth only**. +**Goal:** Implement minimal end-to-end flow execution - from UI button click through Edge Function to real-time status updates. Validates entire integration stack with **no authentication required** - just public anon key access. **Success Criteria:** - ✅ Supabase initialized in demo app -- ✅ Client-side anonymous auth working - ✅ Edge Function with 1-step test flow executes - ✅ pgflow packages vendored correctly -- ✅ pgflow client connects from UI +- ✅ pgflow client connects from UI (anon key only) - ✅ Button click starts flow - ✅ Status updates in real-time - ✅ No console errors -**Philosophy:** Build the thinnest possible slice through the entire stack. UI will be ugly - that's fine. Goal is to prove integration works. **No server-side auth needed - demo is public!** +**Philosophy:** Build the thinnest possible slice through the entire stack. UI will be ugly - that's fine. Goal is to prove integration works. **No authentication - just public demo with anon key!** --- @@ -22,31 +21,66 @@ ### 1. Add pgflow Client Dependency -Edit `apps/demo/package.json` - add `"@pgflow/client": "workspace:*"` to dependencies: - ```bash -pnpm install +cd apps/demo +pnpm add @pgflow/client +cd ../.. ``` +This will add `"@pgflow/client": "workspace:*"` to dependencies automatically. + ### 2. Initialize Supabase ```bash -cd apps/demo && supabase init && cd ../.. +cd apps/demo && npx -y supabase@latest init && cd ../.. ``` --- -### 3. Configure Supabase for pgflow +### 3. Install pgflow -Edit `apps/demo/supabase/config.toml` - add `"pgflow"` to `[api]` schemas: +Run the pgflow installer: -```toml -schemas = ["public", "pgflow"] +```bash +npx pgflow@latest install +``` + +This will: +- Update `supabase/config.toml` (adds pgflow schema, connection pooling) +- Copy pgflow migrations to `supabase/migrations/` + +### 4. Create Anon Permissions Migration + +Create `apps/demo/supabase/migrations/_demo_anon_permissions.sql`: + +```sql +-- Grant anon role access to start flows +GRANT USAGE ON SCHEMA pgflow TO anon; +GRANT EXECUTE ON FUNCTION pgflow.start_flow TO anon; + +-- Grant anon role read access to pgflow tables for real-time updates +GRANT SELECT ON pgflow.flows TO anon; +GRANT SELECT ON pgflow.runs TO anon; +GRANT SELECT ON pgflow.steps TO anon; +GRANT SELECT ON pgflow.step_states TO anon; +GRANT SELECT ON pgflow.deps TO anon; + +-- Enable real-time for anon role +ALTER PUBLICATION supabase_realtime ADD TABLE pgflow.runs; +ALTER PUBLICATION supabase_realtime ADD TABLE pgflow.step_states; +``` + +### 5. Restart Supabase and Apply Migrations + +```bash +npx -y supabase@latest stop +npx -y supabase@latest start +npx -y supabase@latest migrations up ``` --- -### 4. Copy Vendoring Script +### 6. Copy Vendoring Script ```bash mkdir -p apps/demo/scripts @@ -54,7 +88,7 @@ cp examples/playground/scripts/sync-edge-deps.sh apps/demo/scripts/ chmod +x apps/demo/scripts/sync-edge-deps.sh ``` -### 5. Update Vendoring Script Paths +### 7. Update Vendoring Script Paths Edit `apps/demo/scripts/sync-edge-deps.sh` - replace `PLAYGROUND_DIR` with `DEMO_DIR`: @@ -65,7 +99,7 @@ VENDOR_DIR="$DEMO_DIR/supabase/functions/_vendor" --- -### 6. Add Nx Target for Vendoring +### 8. Add Nx Target for Vendoring Edit `apps/demo/project.json` - add `sync-edge-deps` target: @@ -77,7 +111,7 @@ Edit `apps/demo/project.json` - add `sync-edge-deps` target: } ``` -### 7. Build Dependencies and Vendor +### 9. Build Dependencies and Vendor ```bash pnpm nx build core dsl @@ -86,93 +120,233 @@ pnpm nx sync-edge-deps demo # Verify: ls apps/demo/supabase/functions/_vendor/@ --- -### 8. Create Test Flow Definition +### 10. Create Test Flow Worker Directory + +Create worker directory with flow definition inside: + +```bash +mkdir -p apps/demo/supabase/functions/test_flow_worker +``` -Create `apps/demo/supabase/functions/_flows/test-flow.ts`: -- Import Flow from `@pgflow/dsl` -- Create simple 1-step flow with slug 'test-flow' -- Handler returns: `Hello, ${input.run.message}!` -- Note: Access run input via `input.run.*` pattern +### 11. Create Test Flow Definition -### 9. Create Edge Function Worker +Create `apps/demo/supabase/functions/test_flow_worker/test_flow.ts`: + +```typescript +import { Flow } from '@pgflow/dsl'; + +export default new Flow<{ message: string }>({ slug: 'test_flow' }).step( + { slug: 'greet' }, + (input) => `Hello, ${input.run.message}!` +); +``` + +**Note:** Flow slug is `test_flow` (with underscore), matching the worker directory name. + +### 12. Create Edge Function Worker + +Create `apps/demo/supabase/functions/test_flow_worker/index.ts`: -Create `apps/demo/supabase/functions/demo-worker/index.ts`: ```typescript import { EdgeWorker } from '@pgflow/edge-worker'; -import TestFlow from '../_flows/test-flow.ts'; +import TestFlow from './test_flow.ts'; EdgeWorker.start(TestFlow); ``` + **This 3-line pattern is critical - it's how all pgflow workers are set up!** -### 10. Test Edge Function Locally +### 13. Create Deno Import Map + +Create `apps/demo/supabase/functions/test_flow_worker/deno.json`: + +```json +{ + "imports": { + "@pgflow/core": "../_vendor/@pgflow/core/index.ts", + "@pgflow/core/": "../_vendor/@pgflow/core/", + "@pgflow/dsl": "../_vendor/@pgflow/dsl/index.ts", + "@pgflow/dsl/": "../_vendor/@pgflow/dsl/", + "@pgflow/dsl/supabase": "../_vendor/@pgflow/dsl/src/supabase.ts", + "@pgflow/edge-worker": "../_vendor/@pgflow/edge-worker/index.ts", + "@pgflow/edge-worker/": "../_vendor/@pgflow/edge-worker/", + "@pgflow/edge-worker/_internal": "../_vendor/@pgflow/edge-worker/_internal.ts", + "postgres": "npm:postgres@3.4.5", + "@henrygd/queue": "jsr:@henrygd/queue@^1.0.7", + "@supabase/supabase-js": "jsr:@supabase/supabase-js@^2.49.4" + } +} +``` + +**Critical:** This maps all `@pgflow/*` imports to the vendored packages (one level up), including subpaths and required dependencies! + +--- + +### 14. Configure Edge Function in config.toml + +Edit `apps/demo/supabase/config.toml`, add at the end: + +```toml +[functions.test_flow_worker] +enabled = true +verify_jwt = false +import_map = "./functions/test_flow_worker/deno.json" +entrypoint = "./functions/test_flow_worker/index.ts" +``` + +**Critical:** `verify_jwt = false` allows public demo access without authentication. + +--- + +### 15. Build Client Package + +```bash +pnpm nx build client +``` + +### 16. Test Edge Function Locally ```bash cd apps/demo -supabase start # Then in another terminal: -supabase functions serve demo-worker +npx -y supabase@latest start # Then in another terminal: +npx -y supabase@latest functions serve test_flow_worker ``` -### 11. Create Client-Side Supabase Configuration +### 17. Create Client-Side Supabase Configuration Create `apps/demo/src/lib/supabase.ts`: -- Create Supabase client with URL and anon key (use env vars or local defaults) -- Create PgflowClient wrapping the Supabase client -- Export both for use in components -- **Key point:** Pure client-side - no server hooks, no cookies! ---- +```typescript +import { createClient } from '@supabase/supabase-js'; +import { PgflowClient } from '@pgflow/client'; -### 12. Create Minimal Test UI +// Hardcoded local Supabase defaults (Phase 1 - production config in Phase 6) +const SUPABASE_URL = 'http://127.0.0.1:54321'; +const SUPABASE_ANON_KEY = 'sb_publishable_ACJWlzQHlZjBrEguHvfOxg_3BJgxAaH'; -Replace `apps/demo/src/routes/+page.svelte` with basic test interface. +export const supabase = createClient(SUPABASE_URL, SUPABASE_ANON_KEY); +export const pgflow = new PgflowClient(supabase); +``` -**Key patterns to implement:** -- Anonymous auth: `await supabase.auth.signInAnonymously()` in onMount -- Start flow: `pgflow.startFlow('test-flow', { message: 'World' })` -- Listen to events: `run.on('*', (event) => { ... })` -- Svelte 5 state: `let status = $state('idle')` -- Display status updates and output +**Note:** Get the anon key from `npx -y supabase@latest status`. Environment variables for production will be added in Phase 6 (Deploy). + +--- -**Remember:** Use `onclick={handler}` not `on:click` (Svelte 5 syntax) +### 18. Create Minimal Test UI + +Replace `apps/demo/src/routes/+page.svelte` with basic test interface: + +```svelte + + +
+

pgflow Demo - Phase 1 Vertical Slice

+ +
+ +
+ +
+

Status

+

{status}

+
+ + {#if output} +
+

Output

+
{output}
+
+ {/if} + + {#if events.length > 0} +
+

Events

+ {#each events as event} +
{event}
+ {/each} +
+ {/if} +
+ + +``` + +**Key patterns:** +- Use `onclick={handler}` not `on:click` (Svelte 5 syntax) +- Svelte 5 state: `let status = $state('idle')` +- Start flow: `pgflow.startFlow('test_flow', { message: 'World' })` +- Listen to events: `run.on('*', (event) => { ... })` --- -### 13. Start Dev Server +### 19. Start Dev Server ```bash -pnpm nx dev demo # Ensure supabase start running +pnpm nx dev demo # Ensure npx -y supabase@latest start running ``` -### 14. Test End-to-End +### 20. Test End-to-End Open http://localhost:5173/, click button, verify: - Status: `idle` → `starting...` → `running` → `completed` - Output shows `"Hello, World!"` -- Console shows event stream +- Events section shows real-time event stream --- ## Validation Checklist - [ ] Supabase initialized, pgflow packages vendored -- [ ] Test flow + worker created -- [ ] Anonymous auth working (check Network tab for auth.signInAnonymously) -- [ ] `supabase start` and `functions serve demo-worker` running +- [ ] Test flow + worker created in `test_flow_worker/` directory +- [ ] Worker configured in `config.toml` with `verify_jwt = false` +- [ ] Deno import map created with all dependencies +- [ ] `npx -y supabase@latest start` and `functions serve test_flow_worker` running - [ ] Dev server starts, button click starts flow -- [ ] Status updates real-time, output appears, no console errors +- [ ] Status updates real-time, output appears, events stream visible +- [ ] No authentication needed - works immediately on page load +- [ ] Flow slug is `test_flow` (with underscore) +- [ ] No console errors --- ## Troubleshooting - **Vendoring fails:** Check `ls pkgs/core/dist`, rebuild with `pnpm nx build core dsl` -- **Edge Function won't start:** Check `supabase status`, verify vendored files exist -- **Anonymous auth fails:** Check browser console, ensure Supabase anon key is valid +- **Edge Function won't start:** Check `npx -y supabase@latest status`, verify vendored files exist - **Flow doesn't start:** Check browser console - Supabase connection, pgflow schema in config.toml, flow slug matches - **No real-time updates:** Check Realtime enabled, Edge Function logs, Svelte `$state` reactivity - **TypeScript errors:** Verify Svelte 5 syntax (`$state`, `onclick`) -- **Auth issues:** Remember - this is all client-side! No server hooks needed +- **Anon key issues:** Get correct key from `npx -y supabase@latest status`, ensure hardcoded in `lib/supabase.ts` **Rollback:** Mock pgflow client to debug Edge Function separately diff --git a/PHASE_2_ARTICLE_FLOW.md b/PHASE_2_ARTICLE_FLOW.md index 4e86e7006..d312a0a1c 100644 --- a/PHASE_2_ARTICLE_FLOW.md +++ b/PHASE_2_ARTICLE_FLOW.md @@ -18,77 +18,202 @@ ### 1. Install Required Dependencies -Add to `apps/demo/package.json`: -- `"@xyflow/svelte": "^0.1.18"` (DAG visualization) -- `"shiki": "^1.0.0"` (syntax highlighting) +```bash +cd apps/demo +pnpm add @xyflow/svelte shiki +cd ../.. +``` + +This installs: +- `@xyflow/svelte` - DAG visualization +- `shiki` - Syntax highlighting + +--- + +### 2. Create Article Flow Worker + +Create new Edge Function for the article flow: ```bash -pnpm install +cd apps/demo +npx -y supabase@latest functions new article_flow_worker ``` +This creates `apps/demo/supabase/functions/article_flow_worker/` directory. + --- -### 2. Create Article Processing Flow +### 3. Create Article Processing Flow -Create `apps/demo/supabase/functions/_flows/article-flow.ts` with 4 steps: +Create `apps/demo/supabase/functions/article_flow_worker/article_flow.ts` with 4 steps: -1. **fetchArticle** - Calls r.jina.ai API, returns `{ content, title }` -2. **summarize** - Depends on fetchArticle, simulates failure on first attempt (`attemptNumber === 1`) -3. **extractKeywords** - Depends on fetchArticle (runs parallel with summarize) -4. **publish** - Depends on both summarize and extractKeywords +```typescript +import { Flow } from '@pgflow/dsl'; + +export default new Flow<{ url: string }>({ + slug: 'article_flow', + maxAttempts: 3, + baseDelay: 1, + timeout: 60 +}) + .step({ slug: 'fetch_article' }, async (input) => { + // Call r.jina.ai API + const response = await fetch(`https://r.jina.ai/${input.run.url}`); + const content = await response.text(); + return { content, title: 'Article Title' }; + }) + .step({ slug: 'summarize' }, (input) => { + // Simulate failure on first attempt + if (input.attemptNumber === 1) { + throw new Error('Simulated failure for retry demo'); + } + return `Summary of: ${input.steps.fetch_article.title}`; + }) + .step({ slug: 'extract_keywords' }, (input) => { + // Runs parallel with summarize + return ['keyword1', 'keyword2', 'keyword3']; + }) + .step({ slug: 'publish' }, (input) => { + // Depends on both summarize and extract_keywords + return { + articleId: 'article_123', + summary: input.steps.summarize, + keywords: input.steps.extract_keywords + }; + }); +``` **Key patterns:** -- Parallel execution: summarize and extractKeywords run simultaneously +- Flow slug is `article_flow` (with underscore) +- Parallel execution: summarize and extractKeywords run simultaneously (both depend only on fetch_article) - Retry simulation: Use `attemptNumber` param to fail first attempt - Flow config: `maxAttempts: 3, baseDelay: 1, timeout: 60` --- -### 3. Update Edge Function Worker +### 4. Create Edge Function Worker -Edit `apps/demo/supabase/functions/demo-worker/index.ts` - use ArticleFlow: +Create `apps/demo/supabase/functions/article_flow_worker/index.ts`: ```typescript import { EdgeWorker } from '@pgflow/edge-worker'; -import ArticleFlow from '../_flows/article-flow.ts'; +import ArticleFlow from './article_flow.ts'; EdgeWorker.start(ArticleFlow); ``` -### 4. Set Environment Variables +--- + +### 5. Create Deno Import Map + +Create `apps/demo/supabase/functions/article_flow_worker/deno.json`: + +```json +{ + "imports": { + "@pgflow/core": "../_vendor/@pgflow/core/index.ts", + "@pgflow/core/": "../_vendor/@pgflow/core/", + "@pgflow/dsl": "../_vendor/@pgflow/dsl/index.ts", + "@pgflow/dsl/": "../_vendor/@pgflow/dsl/", + "@pgflow/dsl/supabase": "../_vendor/@pgflow/dsl/src/supabase.ts", + "@pgflow/edge-worker": "../_vendor/@pgflow/edge-worker/index.ts", + "@pgflow/edge-worker/": "../_vendor/@pgflow/edge-worker/", + "@pgflow/edge-worker/_internal": "../_vendor/@pgflow/edge-worker/_internal.ts", + "postgres": "npm:postgres@3.4.5", + "@henrygd/queue": "jsr:@henrygd/queue@^1.0.7", + "@supabase/supabase-js": "jsr:@supabase/supabase-js@^2.49.4" + } +} +``` + +--- + +### 6. Configure Edge Function in config.toml + +Edit `apps/demo/supabase/config.toml`, add at the end: + +```toml +[functions.article_flow_worker] +enabled = true +verify_jwt = false +import_map = "./functions/article_flow_worker/deno.json" +entrypoint = "./functions/article_flow_worker/index.ts" +``` + +**Critical:** `verify_jwt = false` allows public demo access without authentication. + +--- -Create `apps/demo/supabase/.env.local` with `JINA_API_KEY` (optional for now) +### 7. Set Environment Variables -### 5. Rebuild and Re-vendor +Create `apps/demo/supabase/.env.local` with `JINA_API_KEY` (optional for now): ```bash -pnpm nx build core dsl +JINA_API_KEY=your_jina_api_key_here +``` + +--- + +### 8. Rebuild and Re-vendor + +```bash +pnpm nx build core dsl client pnpm nx sync-edge-deps demo -cd apps/demo && supabase functions serve demo-worker ``` --- -### 6. Create pgflow State Store +### 9. Test Edge Function Locally + +```bash +cd apps/demo +npx -y supabase@latest start +# In another terminal: +npx -y supabase@latest functions serve article_flow_worker +``` + +--- + +### 10. Create pgflow State Store Create `apps/demo/src/lib/stores/pgflow-state.svelte.ts`: -Create a class-based store using Svelte 5 runes: -- `run = $state(null)` - Current flow run -- `activeStep = $state(null)` - Currently executing step -- `steps = $derived(...)` - Map of step states derived from run -- Export singleton instance +```typescript +import type { FlowRun } from '@pgflow/client'; + +class PgflowState { + run = $state(null); + activeStep = $state(null); + + steps = $derived(() => { + // Derive step states from run + if (!this.run) return new Map(); + // Implementation: map step states to slugs + return new Map(); + }); +} + +export const pgflowState = new PgflowState(); +``` **Purpose:** Central state management for flow execution, used by all UI components +**Key patterns:** +- Use Svelte 5 runes: `$state` and `$derived` +- Export singleton instance +- Will be updated in Phase 3 when building UI + --- ## Validation Checklist +- [ ] Article flow worker created (`article_flow_worker/`) - [ ] 4-step flow created with simulated retry -- [ ] Edge worker updated to use ArticleFlow +- [ ] Worker configured in `config.toml` with `verify_jwt = false` +- [ ] Deno import map created with all dependencies - [ ] pgflow State Store created and exported -- [ ] All dependencies installed +- [ ] All dependencies installed (`@xyflow/svelte`, `shiki`) +- [ ] Edge Function serves successfully - [ ] Build succeeds --- diff --git a/PHASE_3_DAG_DEBUG.md b/PHASE_3_DAG_DEBUG.md index 1c46dc236..504fc96f4 100644 --- a/PHASE_3_DAG_DEBUG.md +++ b/PHASE_3_DAG_DEBUG.md @@ -77,17 +77,22 @@ Import styles in `apps/demo/src/routes/+layout.svelte` ### 11. Test Complete Flow ```bash -cd apps/demo && supabase start -supabase functions serve demo-worker # Another terminal -pnpm nx dev demo # From monorepo root +cd apps/demo +npx -y supabase@latest start +# In another terminal: +npx -y supabase@latest functions serve article_flow_worker +# In another terminal (from monorepo root): +pnpm nx dev demo ``` Open http://localhost:5173/, click "Process Article", verify: - DAG nodes light up -- Parallel execution visible (summarize + extractKeywords) +- Parallel execution visible (summarize + extract_keywords) - Retry on summarize step (fails → succeeds) - Debug panel updates +**Note:** Worker name is `article_flow_worker` (with underscore) matching the flow slug `article_flow`. + --- ## Validation Checklist diff --git a/PHASE_4_CODE_EXPLANATION.md b/PHASE_4_CODE_EXPLANATION.md index 72dab5e62..1436a815f 100644 --- a/PHASE_4_CODE_EXPLANATION.md +++ b/PHASE_4_CODE_EXPLANATION.md @@ -18,23 +18,22 @@ ## Tasks -### 1. Add Shiki Dependency +### 1. Verify Shiki Dependency -Already added in Phase 2, verify: +Already installed in Phase 2, verify: ```bash # Check installation -ls node_modules/shiki +ls apps/demo/node_modules/shiki ``` **If missing:** ```bash cd apps/demo pnpm add shiki +cd ../.. ``` -**Validate:** Shiki installed - --- ### 2. Create Code Panel Component diff --git a/PHASE_5_RESULTS_MODALS.md b/PHASE_5_RESULTS_MODALS.md index 7eb30cf0d..b909af1c5 100644 --- a/PHASE_5_RESULTS_MODALS.md +++ b/PHASE_5_RESULTS_MODALS.md @@ -91,9 +91,10 @@ Update layout structure: ```bash # Restart everything cd apps/demo -supabase functions serve demo-worker - -# In another terminal (from root) +npx -y supabase@latest start +# In another terminal: +npx -y supabase@latest functions serve article_flow_worker +# In another terminal (from root): pnpm nx dev demo ``` diff --git a/PHASE_6_POLISH_DEPLOY.md b/PHASE_6_POLISH_DEPLOY.md index 4be25edd3..0d84b83bf 100644 --- a/PHASE_6_POLISH_DEPLOY.md +++ b/PHASE_6_POLISH_DEPLOY.md @@ -249,12 +249,17 @@ Test checklist: ### 12. Deploy Edge Functions to Supabase +Deploy both workers to production: + ```bash cd apps/demo -supabase functions deploy demo-worker -supabase secrets set JINA_API_KEY=your_actual_key +npx -y supabase@latest functions deploy test_flow_worker +npx -y supabase@latest functions deploy article_flow_worker +npx -y supabase@latest secrets set JINA_API_KEY=your_actual_key ``` +**Note:** Both `test_flow_worker` and `article_flow_worker` need to be deployed for the demo to work. + --- ## Validation Checklist diff --git a/START_HERE.md b/START_HERE.md index 97905b84d..f1654909c 100644 --- a/START_HERE.md +++ b/START_HERE.md @@ -47,7 +47,7 @@ The implementation uses these branches (create as needed): | Phase | Branch | What Gets Built | Done | | ------- | ------------------------------ | ----------------------------------- | ---- | | Phase 0 | `feat-demo-0-foundation` | Fresh SvelteKit app + Nx setup | [x] | -| Phase 1 | `feat-demo-1-vertical-slice` | Client-side auth + end-to-end proof | [ ] | +| Phase 1 | `feat-demo-1-vertical-slice` | Client-side auth + end-to-end proof | [x] | | Phase 2 | `feat-demo-2-article-flow` | 4-step flow + stores | [ ] | | Phase 3 | `feat-demo-3-dag-debug` | DAG viz + Debug panel | [ ] | | Phase 4 | `feat-demo-4-code-explanation` | Code panel + clicks | [ ] | @@ -67,13 +67,36 @@ The implementation uses these branches (create as needed): **Then expand incrementally:** -- Phase 2: 4-step flow + DAG/Debug visualizations -- Phase 3: Interactive elements -- Phase 4: Polish + deploy +- Phase 2: 4-step flow + state management +- Phase 3: DAG visualization + Debug panel +- Phase 4: Code panel + interactions +- Phase 5: Results + modals +- Phase 6: Polish + deploy **Why?** Catches integration issues early. Everything after Phase 1 is UI. -**Auth simplicity:** Client-side only! No server hooks, no session management, just `supabase.auth.signInAnonymously()` - perfect for public demos. +**Auth simplicity:** Client-side only! No server hooks, no session management, just public anon key - perfect for public demos. + +--- + +## 📝 Implementation Patterns + +**Every phase follows this pattern for Edge Functions:** + +1. **Create worker:** `npx -y supabase@latest functions new _worker` +2. **Flow lives with worker:** `functions/_worker/.ts` +3. **Worker imports flow:** `import Flow from './.ts'` +4. **Deno import map:** `functions/_worker/deno.json` with relative paths to `../_vendor/` +5. **Configure in config.toml:** + ```toml + [functions._worker] + enabled = true + verify_jwt = false + import_map = "./functions/_worker/deno.json" + entrypoint = "./functions/_worker/index.ts" + ``` + +**Critical:** `verify_jwt = false` allows public demo access without authentication. --- diff --git a/apps/demo/.gitignore b/apps/demo/.gitignore index 3b462cb0c..34feae50d 100644 --- a/apps/demo/.gitignore +++ b/apps/demo/.gitignore @@ -21,3 +21,4 @@ Thumbs.db # Vite vite.config.js.timestamp-* vite.config.ts.timestamp-* +supabase/functions/_vendor/ diff --git a/apps/demo/package.json b/apps/demo/package.json index ba5d4080f..242096e3f 100644 --- a/apps/demo/package.json +++ b/apps/demo/package.json @@ -37,6 +37,7 @@ "vite": "^7.1.10" }, "dependencies": { + "@pgflow/client": "workspace:*", "@supabase/supabase-js": "^2.78.0" } } diff --git a/apps/demo/project.json b/apps/demo/project.json index 5e0d63684..0bd9b38d8 100644 --- a/apps/demo/project.json +++ b/apps/demo/project.json @@ -26,6 +26,14 @@ "cwd": "apps/demo" }, "dependsOn": ["build"] + }, + "sync-edge-deps": { + "executor": "nx:run-commands", + "dependsOn": ["core:build", "dsl:build"], + "options": { + "command": "./scripts/sync-edge-deps.sh", + "cwd": "apps/demo" + } } } } diff --git a/apps/demo/scripts/sync-edge-deps.sh b/apps/demo/scripts/sync-edge-deps.sh new file mode 100755 index 000000000..d9457ac60 --- /dev/null +++ b/apps/demo/scripts/sync-edge-deps.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +DEMO_DIR="$(dirname "$SCRIPT_DIR")" +MONOREPO_ROOT="$(cd "$DEMO_DIR/../.." && pwd)" +VENDOR_DIR="$DEMO_DIR/supabase/functions/_vendor" + +echo "🔄 Syncing edge function dependencies for local development..." + +# Clean and create vendor directory +rm -rf "$VENDOR_DIR" +mkdir -p "$VENDOR_DIR/@pgflow" + +# Verify source directories exist +if [ ! -d "$MONOREPO_ROOT/pkgs/core/src" ]; then + echo "❌ Error: core package src directory not found" + exit 1 +fi + +if [ ! -d "$MONOREPO_ROOT/pkgs/dsl/src" ]; then + echo "❌ Error: dsl package src directory not found" + exit 1 +fi + +# Copy core package TypeScript source +echo "📋 Copying @pgflow/core (TypeScript source)..." +mkdir -p "$VENDOR_DIR/@pgflow/core" +cp -r "$MONOREPO_ROOT/pkgs/core/src" "$VENDOR_DIR/@pgflow/core/" +cp "$MONOREPO_ROOT/pkgs/core/package.json" "$VENDOR_DIR/@pgflow/core/" + +# Fix .js extensions in core imports +find "$VENDOR_DIR/@pgflow/core" -name "*.ts" -type f -exec sed -i 's/\.js"/\.ts"/g' {} + +find "$VENDOR_DIR/@pgflow/core" -name "*.ts" -type f -exec sed -i "s/\.js'/\.ts'/g" {} + + +# Create index.ts redirect for core +cat > "$VENDOR_DIR/@pgflow/core/index.ts" << 'EOF' +export * from './src/index.ts'; +EOF + +# Copy dsl package TypeScript source +echo "📋 Copying @pgflow/dsl (TypeScript source)..." +mkdir -p "$VENDOR_DIR/@pgflow/dsl" +cp -r "$MONOREPO_ROOT/pkgs/dsl/src" "$VENDOR_DIR/@pgflow/dsl/" +cp "$MONOREPO_ROOT/pkgs/dsl/package.json" "$VENDOR_DIR/@pgflow/dsl/" + +# Fix .js extensions in dsl imports +find "$VENDOR_DIR/@pgflow/dsl" -name "*.ts" -type f -exec sed -i 's/\.js"/\.ts"/g' {} + +find "$VENDOR_DIR/@pgflow/dsl" -name "*.ts" -type f -exec sed -i "s/\.js'/\.ts'/g" {} + + +# Create index.ts redirect for dsl +cat > "$VENDOR_DIR/@pgflow/dsl/index.ts" << 'EOF' +export * from './src/index.ts'; +EOF + +# Copy edge-worker source (not built) - preserving directory structure +echo "📋 Copying @pgflow/edge-worker..." +mkdir -p "$VENDOR_DIR/@pgflow/edge-worker" +# Copy the entire src directory to maintain relative imports +cp -r "$MONOREPO_ROOT/pkgs/edge-worker/src" "$VENDOR_DIR/@pgflow/edge-worker/" + +# Simple fix: replace .js with .ts in imports +find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.ts" -type f -exec sed -i 's/\.js"/\.ts"/g' {} + +find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.ts" -type f -exec sed -i "s/\.js'/\.ts'/g" {} + + +# Create a redirect index.ts at the root that points to src/index.ts +cat > "$VENDOR_DIR/@pgflow/edge-worker/index.ts" << 'EOF' +// Re-export from the src directory to maintain compatibility +export * from './src/index.ts'; +EOF + +# Create _internal.ts redirect as well since edge-worker exports this path +cat > "$VENDOR_DIR/@pgflow/edge-worker/_internal.ts" << 'EOF' +// Re-export from the src directory to maintain compatibility +export * from './src/_internal.ts'; +EOF + +# Verify key files exist +if [ ! -f "$VENDOR_DIR/@pgflow/core/index.ts" ]; then + echo "⚠️ Warning: @pgflow/core/index.ts not found after copy" +fi + +if [ ! -f "$VENDOR_DIR/@pgflow/dsl/index.ts" ]; then + echo "⚠️ Warning: @pgflow/dsl/index.ts not found after copy" +fi + +if [ ! -f "$VENDOR_DIR/@pgflow/edge-worker/src/index.ts" ]; then + echo "⚠️ Warning: @pgflow/edge-worker/src/index.ts not found after copy" +fi + +echo "✅ Dependencies synced to $VENDOR_DIR (TypeScript source)" \ No newline at end of file diff --git a/apps/demo/src/lib/supabase.ts b/apps/demo/src/lib/supabase.ts new file mode 100644 index 000000000..0008335a3 --- /dev/null +++ b/apps/demo/src/lib/supabase.ts @@ -0,0 +1,9 @@ +import { createClient } from '@supabase/supabase-js'; +import { PgflowClient } from '@pgflow/client'; + +// Hardcoded local Supabase defaults (Phase 1 - production config in Phase 6) +const SUPABASE_URL = 'http://127.0.0.1:54321'; +const SUPABASE_ANON_KEY = 'sb_publishable_ACJWlzQHlZjBrEguHvfOxg_3BJgxAaH'; + +export const supabase = createClient(SUPABASE_URL, SUPABASE_ANON_KEY); +export const pgflow = new PgflowClient(supabase); diff --git a/apps/demo/src/routes/+page.svelte b/apps/demo/src/routes/+page.svelte index cc88df0ea..dcc87ea8d 100644 --- a/apps/demo/src/routes/+page.svelte +++ b/apps/demo/src/routes/+page.svelte @@ -1,2 +1,107 @@ -

Welcome to SvelteKit

-

Visit svelte.dev/docs/kit to read the documentation

+ + +
+

pgflow Demo - Phase 1 Vertical Slice

+ +
+ +
+ +
+

Status

+

{status}

+
+ + {#if output} +
+

Output

+
{output}
+
+ {/if} + + {#if events.length > 0} +
+

Events

+ {#each events as event} +
{event}
+ {/each} +
+ {/if} +
+ + diff --git a/apps/demo/supabase/.gitignore b/apps/demo/supabase/.gitignore new file mode 100644 index 000000000..ad9264f0b --- /dev/null +++ b/apps/demo/supabase/.gitignore @@ -0,0 +1,8 @@ +# Supabase +.branches +.temp + +# dotenvx +.env.keys +.env.local +.env.*.local diff --git a/apps/demo/supabase/config.toml b/apps/demo/supabase/config.toml new file mode 100644 index 000000000..89a1eaf0d --- /dev/null +++ b/apps/demo/supabase/config.toml @@ -0,0 +1,360 @@ +# For detailed configuration reference documentation, visit: +# https://supabase.com/docs/guides/local-development/cli/config +# A string used to distinguish different Supabase projects on the same host. Defaults to the +# working directory name when running `supabase init`. +project_id = "demo" + +[api] +enabled = true +# Port to use for the API URL. +port = 54321 +# Schemas to expose in your API. Tables, views and stored procedures in this schema will get API +# endpoints. `public` and `graphql_public` schemas are included by default. +schemas = ["public", "graphql_public", "pgflow"] +# Extra schemas to add to the search_path of every request. +extra_search_path = ["public", "extensions"] +# The maximum number of rows returns from a view, table, or stored procedure. Limits payload size +# for accidental or malicious requests. +max_rows = 1000 + +[api.tls] +# Enable HTTPS endpoints locally using a self-signed certificate. +enabled = false +# Paths to self-signed certificate pair. +# cert_path = "../certs/my-cert.pem" +# key_path = "../certs/my-key.pem" + +[db] +# Port to use for the local database URL. +port = 54322 +# Port used by db diff command to initialize the shadow database. +shadow_port = 54320 +# The database major version to use. This has to be the same as your remote database's. Run `SHOW +# server_version;` on the remote database to check. +major_version = 17 + +[db.pooler] +enabled = true +# Port to use for the local connection pooler. +port = 54329 +# Specifies when a server connection can be reused by other clients. +# Configure one of the supported pooler modes: `transaction`, `session`. +pool_mode = "transaction" +# How many server connections to allow per user/database pair. +default_pool_size = 20 +# Maximum number of client connections allowed. +max_client_conn = 100 + +# [db.vault] +# secret_key = "env(SECRET_VALUE)" + +[db.migrations] +# If disabled, migrations will be skipped during a db push or reset. +enabled = true +# Specifies an ordered list of schema files that describe your database. +# Supports glob patterns relative to supabase directory: "./schemas/*.sql" +schema_paths = [] + +[db.seed] +# If enabled, seeds the database after migrations during a db reset. +enabled = true +# Specifies an ordered list of seed files to load during db reset. +# Supports glob patterns relative to supabase directory: "./seeds/*.sql" +sql_paths = ["./seed.sql"] + +[db.network_restrictions] +# Enable management of network restrictions. +enabled = false +# List of IPv4 CIDR blocks allowed to connect to the database. +# Defaults to allow all IPv4 connections. Set empty array to block all IPs. +allowed_cidrs = ["0.0.0.0/0"] +# List of IPv6 CIDR blocks allowed to connect to the database. +# Defaults to allow all IPv6 connections. Set empty array to block all IPs. +allowed_cidrs_v6 = ["::/0"] + +[realtime] +enabled = true +# Bind realtime via either IPv4 or IPv6. (default: IPv4) +# ip_version = "IPv6" +# The maximum length in bytes of HTTP request headers. (default: 4096) +# max_header_length = 4096 + +[studio] +enabled = true +# Port to use for Supabase Studio. +port = 54323 +# External URL of the API server that frontend connects to. +api_url = "http://127.0.0.1" +# OpenAI API Key to use for Supabase AI in the Supabase Studio. +openai_api_key = "env(OPENAI_API_KEY)" + +# Email testing server. Emails sent with the local dev setup are not actually sent - rather, they +# are monitored, and you can view the emails that would have been sent from the web interface. +[inbucket] +enabled = true +# Port to use for the email testing server web interface. +port = 54324 +# Uncomment to expose additional ports for testing user applications that send emails. +# smtp_port = 54325 +# pop3_port = 54326 +# admin_email = "admin@email.com" +# sender_name = "Admin" + +[storage] +enabled = true +# The maximum file size allowed (e.g. "5MB", "500KB"). +file_size_limit = "50MiB" + +# Image transformation API is available to Supabase Pro plan. +# [storage.image_transformation] +# enabled = true + +# Uncomment to configure local storage buckets +# [storage.buckets.images] +# public = false +# file_size_limit = "50MiB" +# allowed_mime_types = ["image/png", "image/jpeg"] +# objects_path = "./images" + +[auth] +enabled = true +# The base URL of your website. Used as an allow-list for redirects and for constructing URLs used +# in emails. +site_url = "http://127.0.0.1:3000" +# A list of *exact* URLs that auth providers are permitted to redirect to post authentication. +additional_redirect_urls = ["https://127.0.0.1:3000"] +# How long tokens are valid for, in seconds. Defaults to 3600 (1 hour), maximum 604,800 (1 week). +jwt_expiry = 3600 +# Path to JWT signing key. DO NOT commit your signing keys file to git. +# signing_keys_path = "./signing_keys.json" +# If disabled, the refresh token will never expire. +enable_refresh_token_rotation = true +# Allows refresh tokens to be reused after expiry, up to the specified interval in seconds. +# Requires enable_refresh_token_rotation = true. +refresh_token_reuse_interval = 10 +# Allow/disallow new user signups to your project. +enable_signup = true +# Allow/disallow anonymous sign-ins to your project. +enable_anonymous_sign_ins = true +# Allow/disallow testing manual linking of accounts +enable_manual_linking = false +# Passwords shorter than this value will be rejected as weak. Minimum 6, recommended 8 or more. +minimum_password_length = 6 +# Passwords that do not meet the following requirements will be rejected as weak. Supported values +# are: `letters_digits`, `lower_upper_letters_digits`, `lower_upper_letters_digits_symbols` +password_requirements = "" + +[auth.rate_limit] +# Number of emails that can be sent per hour. Requires auth.email.smtp to be enabled. +email_sent = 2 +# Number of SMS messages that can be sent per hour. Requires auth.sms to be enabled. +sms_sent = 30 +# Number of anonymous sign-ins that can be made per hour per IP address. Requires enable_anonymous_sign_ins = true. +anonymous_users = 30 +# Number of sessions that can be refreshed in a 5 minute interval per IP address. +token_refresh = 150 +# Number of sign up and sign-in requests that can be made in a 5 minute interval per IP address (excludes anonymous users). +sign_in_sign_ups = 30 +# Number of OTP / Magic link verifications that can be made in a 5 minute interval per IP address. +token_verifications = 30 +# Number of Web3 logins that can be made in a 5 minute interval per IP address. +web3 = 30 + +# Configure one of the supported captcha providers: `hcaptcha`, `turnstile`. +# [auth.captcha] +# enabled = true +# provider = "hcaptcha" +# secret = "" + +[auth.email] +# Allow/disallow new user signups via email to your project. +enable_signup = true +# If enabled, a user will be required to confirm any email change on both the old, and new email +# addresses. If disabled, only the new email is required to confirm. +double_confirm_changes = true +# If enabled, users need to confirm their email address before signing in. +enable_confirmations = false +# If enabled, users will need to reauthenticate or have logged in recently to change their password. +secure_password_change = false +# Controls the minimum amount of time that must pass before sending another signup confirmation or password reset email. +max_frequency = "1s" +# Number of characters used in the email OTP. +otp_length = 6 +# Number of seconds before the email OTP expires (defaults to 1 hour). +otp_expiry = 3600 + +# Use a production-ready SMTP server +# [auth.email.smtp] +# enabled = true +# host = "smtp.sendgrid.net" +# port = 587 +# user = "apikey" +# pass = "env(SENDGRID_API_KEY)" +# admin_email = "admin@email.com" +# sender_name = "Admin" + +# Uncomment to customize email template +# [auth.email.template.invite] +# subject = "You have been invited" +# content_path = "./supabase/templates/invite.html" + +[auth.sms] +# Allow/disallow new user signups via SMS to your project. +enable_signup = false +# If enabled, users need to confirm their phone number before signing in. +enable_confirmations = false +# Template for sending OTP to users +template = "Your code is {{ .Code }}" +# Controls the minimum amount of time that must pass before sending another sms otp. +max_frequency = "5s" + +# Use pre-defined map of phone number to OTP for testing. +# [auth.sms.test_otp] +# 4152127777 = "123456" + +# Configure logged in session timeouts. +# [auth.sessions] +# Force log out after the specified duration. +# timebox = "24h" +# Force log out if the user has been inactive longer than the specified duration. +# inactivity_timeout = "8h" + +# This hook runs before a new user is created and allows developers to reject the request based on the incoming user object. +# [auth.hook.before_user_created] +# enabled = true +# uri = "pg-functions://postgres/auth/before-user-created-hook" + +# This hook runs before a token is issued and allows you to add additional claims based on the authentication method used. +# [auth.hook.custom_access_token] +# enabled = true +# uri = "pg-functions:////" + +# Configure one of the supported SMS providers: `twilio`, `twilio_verify`, `messagebird`, `textlocal`, `vonage`. +[auth.sms.twilio] +enabled = false +account_sid = "" +message_service_sid = "" +# DO NOT commit your Twilio auth token to git. Use environment variable substitution instead: +auth_token = "env(SUPABASE_AUTH_SMS_TWILIO_AUTH_TOKEN)" + +# Multi-factor-authentication is available to Supabase Pro plan. +[auth.mfa] +# Control how many MFA factors can be enrolled at once per user. +max_enrolled_factors = 10 + +# Control MFA via App Authenticator (TOTP) +[auth.mfa.totp] +enroll_enabled = false +verify_enabled = false + +# Configure MFA via Phone Messaging +[auth.mfa.phone] +enroll_enabled = false +verify_enabled = false +otp_length = 6 +template = "Your code is {{ .Code }}" +max_frequency = "5s" + +# Configure MFA via WebAuthn +# [auth.mfa.web_authn] +# enroll_enabled = true +# verify_enabled = true + +# Use an external OAuth provider. The full list of providers are: `apple`, `azure`, `bitbucket`, +# `discord`, `facebook`, `github`, `gitlab`, `google`, `keycloak`, `linkedin_oidc`, `notion`, `twitch`, +# `twitter`, `slack`, `spotify`, `workos`, `zoom`. +[auth.external.apple] +enabled = false +client_id = "" +# DO NOT commit your OAuth provider secret to git. Use environment variable substitution instead: +secret = "env(SUPABASE_AUTH_EXTERNAL_APPLE_SECRET)" +# Overrides the default auth redirectUrl. +redirect_uri = "" +# Overrides the default auth provider URL. Used to support self-hosted gitlab, single-tenant Azure, +# or any other third-party OIDC providers. +url = "" +# If enabled, the nonce check will be skipped. Required for local sign in with Google auth. +skip_nonce_check = false +# If enabled, it will allow the user to successfully authenticate when the provider does not return an email address. +email_optional = false + +# Allow Solana wallet holders to sign in to your project via the Sign in with Solana (SIWS, EIP-4361) standard. +# You can configure "web3" rate limit in the [auth.rate_limit] section and set up [auth.captcha] if self-hosting. +[auth.web3.solana] +enabled = false + +# Use Firebase Auth as a third-party provider alongside Supabase Auth. +[auth.third_party.firebase] +enabled = false +# project_id = "my-firebase-project" + +# Use Auth0 as a third-party provider alongside Supabase Auth. +[auth.third_party.auth0] +enabled = false +# tenant = "my-auth0-tenant" +# tenant_region = "us" + +# Use AWS Cognito (Amplify) as a third-party provider alongside Supabase Auth. +[auth.third_party.aws_cognito] +enabled = false +# user_pool_id = "my-user-pool-id" +# user_pool_region = "us-east-1" + +# Use Clerk as a third-party provider alongside Supabase Auth. +[auth.third_party.clerk] +enabled = false +# Obtain from https://clerk.com/setup/supabase +# domain = "example.clerk.accounts.dev" + +# OAuth server configuration +[auth.oauth_server] +# Enable OAuth server functionality +enabled = false +# Path for OAuth consent flow UI +authorization_url_path = "/oauth/consent" +# Allow dynamic client registration +allow_dynamic_registration = false + +[edge_runtime] +enabled = true +# Supported request policies: `oneshot`, `per_worker`. +# `per_worker` (default) — enables hot reload during local development. +# `oneshot` — fallback mode if hot reload causes issues (e.g. in large repos or with symlinks). +policy = "per_worker" +# Port to attach the Chrome inspector for debugging edge functions. +inspector_port = 8083 +# The Deno major version to use. +deno_version = 2 + +# [edge_runtime.secrets] +# secret_key = "env(SECRET_VALUE)" + +[analytics] +enabled = true +port = 54327 +# Configure one of the supported backends: `postgres`, `bigquery`. +backend = "postgres" + +# Experimental features may be deprecated any time +[experimental] +# Configures Postgres storage engine to use OrioleDB (S3) +orioledb_version = "" +# Configures S3 bucket URL, eg. .s3-.amazonaws.com +s3_host = "env(S3_HOST)" +# Configures S3 bucket region, eg. us-east-1 +s3_region = "env(S3_REGION)" +# Configures AWS_ACCESS_KEY_ID for S3 bucket +s3_access_key = "env(S3_ACCESS_KEY)" +# Configures AWS_SECRET_ACCESS_KEY for S3 bucket +s3_secret_key = "env(S3_SECRET_KEY)" + +[functions.test_flow_worker] +enabled = true +verify_jwt = false +import_map = "./functions/test_flow_worker/deno.json" +# Uncomment to specify a custom file path to the entrypoint. +# Supported file extensions are: .ts, .js, .mjs, .jsx, .tsx +entrypoint = "./functions/test_flow_worker/index.ts" +# Specifies static files to be bundled with the function. Supports glob patterns. +# For example, if you want to serve static HTML pages in your function: +# static_files = [ "./functions/test_flow_worker/*.html" ] diff --git a/apps/demo/supabase/config.toml.backup b/apps/demo/supabase/config.toml.backup new file mode 100644 index 000000000..eb114f3ce --- /dev/null +++ b/apps/demo/supabase/config.toml.backup @@ -0,0 +1,349 @@ +# For detailed configuration reference documentation, visit: +# https://supabase.com/docs/guides/local-development/cli/config +# A string used to distinguish different Supabase projects on the same host. Defaults to the +# working directory name when running `supabase init`. +project_id = "demo" + +[api] +enabled = true +# Port to use for the API URL. +port = 54321 +# Schemas to expose in your API. Tables, views and stored procedures in this schema will get API +# endpoints. `public` and `graphql_public` schemas are included by default. +schemas = ["public", "graphql_public", "pgflow"] +# Extra schemas to add to the search_path of every request. +extra_search_path = ["public", "extensions"] +# The maximum number of rows returns from a view, table, or stored procedure. Limits payload size +# for accidental or malicious requests. +max_rows = 1000 + +[api.tls] +# Enable HTTPS endpoints locally using a self-signed certificate. +enabled = false +# Paths to self-signed certificate pair. +# cert_path = "../certs/my-cert.pem" +# key_path = "../certs/my-key.pem" + +[db] +# Port to use for the local database URL. +port = 54322 +# Port used by db diff command to initialize the shadow database. +shadow_port = 54320 +# The database major version to use. This has to be the same as your remote database's. Run `SHOW +# server_version;` on the remote database to check. +major_version = 17 + +[db.pooler] +enabled = false +# Port to use for the local connection pooler. +port = 54329 +# Specifies when a server connection can be reused by other clients. +# Configure one of the supported pooler modes: `transaction`, `session`. +pool_mode = "transaction" +# How many server connections to allow per user/database pair. +default_pool_size = 20 +# Maximum number of client connections allowed. +max_client_conn = 100 + +# [db.vault] +# secret_key = "env(SECRET_VALUE)" + +[db.migrations] +# If disabled, migrations will be skipped during a db push or reset. +enabled = true +# Specifies an ordered list of schema files that describe your database. +# Supports glob patterns relative to supabase directory: "./schemas/*.sql" +schema_paths = [] + +[db.seed] +# If enabled, seeds the database after migrations during a db reset. +enabled = true +# Specifies an ordered list of seed files to load during db reset. +# Supports glob patterns relative to supabase directory: "./seeds/*.sql" +sql_paths = ["./seed.sql"] + +[db.network_restrictions] +# Enable management of network restrictions. +enabled = false +# List of IPv4 CIDR blocks allowed to connect to the database. +# Defaults to allow all IPv4 connections. Set empty array to block all IPs. +allowed_cidrs = ["0.0.0.0/0"] +# List of IPv6 CIDR blocks allowed to connect to the database. +# Defaults to allow all IPv6 connections. Set empty array to block all IPs. +allowed_cidrs_v6 = ["::/0"] + +[realtime] +enabled = true +# Bind realtime via either IPv4 or IPv6. (default: IPv4) +# ip_version = "IPv6" +# The maximum length in bytes of HTTP request headers. (default: 4096) +# max_header_length = 4096 + +[studio] +enabled = true +# Port to use for Supabase Studio. +port = 54323 +# External URL of the API server that frontend connects to. +api_url = "http://127.0.0.1" +# OpenAI API Key to use for Supabase AI in the Supabase Studio. +openai_api_key = "env(OPENAI_API_KEY)" + +# Email testing server. Emails sent with the local dev setup are not actually sent - rather, they +# are monitored, and you can view the emails that would have been sent from the web interface. +[inbucket] +enabled = true +# Port to use for the email testing server web interface. +port = 54324 +# Uncomment to expose additional ports for testing user applications that send emails. +# smtp_port = 54325 +# pop3_port = 54326 +# admin_email = "admin@email.com" +# sender_name = "Admin" + +[storage] +enabled = true +# The maximum file size allowed (e.g. "5MB", "500KB"). +file_size_limit = "50MiB" + +# Image transformation API is available to Supabase Pro plan. +# [storage.image_transformation] +# enabled = true + +# Uncomment to configure local storage buckets +# [storage.buckets.images] +# public = false +# file_size_limit = "50MiB" +# allowed_mime_types = ["image/png", "image/jpeg"] +# objects_path = "./images" + +[auth] +enabled = true +# The base URL of your website. Used as an allow-list for redirects and for constructing URLs used +# in emails. +site_url = "http://127.0.0.1:3000" +# A list of *exact* URLs that auth providers are permitted to redirect to post authentication. +additional_redirect_urls = ["https://127.0.0.1:3000"] +# How long tokens are valid for, in seconds. Defaults to 3600 (1 hour), maximum 604,800 (1 week). +jwt_expiry = 3600 +# Path to JWT signing key. DO NOT commit your signing keys file to git. +# signing_keys_path = "./signing_keys.json" +# If disabled, the refresh token will never expire. +enable_refresh_token_rotation = true +# Allows refresh tokens to be reused after expiry, up to the specified interval in seconds. +# Requires enable_refresh_token_rotation = true. +refresh_token_reuse_interval = 10 +# Allow/disallow new user signups to your project. +enable_signup = true +# Allow/disallow anonymous sign-ins to your project. +enable_anonymous_sign_ins = true +# Allow/disallow testing manual linking of accounts +enable_manual_linking = false +# Passwords shorter than this value will be rejected as weak. Minimum 6, recommended 8 or more. +minimum_password_length = 6 +# Passwords that do not meet the following requirements will be rejected as weak. Supported values +# are: `letters_digits`, `lower_upper_letters_digits`, `lower_upper_letters_digits_symbols` +password_requirements = "" + +[auth.rate_limit] +# Number of emails that can be sent per hour. Requires auth.email.smtp to be enabled. +email_sent = 2 +# Number of SMS messages that can be sent per hour. Requires auth.sms to be enabled. +sms_sent = 30 +# Number of anonymous sign-ins that can be made per hour per IP address. Requires enable_anonymous_sign_ins = true. +anonymous_users = 30 +# Number of sessions that can be refreshed in a 5 minute interval per IP address. +token_refresh = 150 +# Number of sign up and sign-in requests that can be made in a 5 minute interval per IP address (excludes anonymous users). +sign_in_sign_ups = 30 +# Number of OTP / Magic link verifications that can be made in a 5 minute interval per IP address. +token_verifications = 30 +# Number of Web3 logins that can be made in a 5 minute interval per IP address. +web3 = 30 + +# Configure one of the supported captcha providers: `hcaptcha`, `turnstile`. +# [auth.captcha] +# enabled = true +# provider = "hcaptcha" +# secret = "" + +[auth.email] +# Allow/disallow new user signups via email to your project. +enable_signup = true +# If enabled, a user will be required to confirm any email change on both the old, and new email +# addresses. If disabled, only the new email is required to confirm. +double_confirm_changes = true +# If enabled, users need to confirm their email address before signing in. +enable_confirmations = false +# If enabled, users will need to reauthenticate or have logged in recently to change their password. +secure_password_change = false +# Controls the minimum amount of time that must pass before sending another signup confirmation or password reset email. +max_frequency = "1s" +# Number of characters used in the email OTP. +otp_length = 6 +# Number of seconds before the email OTP expires (defaults to 1 hour). +otp_expiry = 3600 + +# Use a production-ready SMTP server +# [auth.email.smtp] +# enabled = true +# host = "smtp.sendgrid.net" +# port = 587 +# user = "apikey" +# pass = "env(SENDGRID_API_KEY)" +# admin_email = "admin@email.com" +# sender_name = "Admin" + +# Uncomment to customize email template +# [auth.email.template.invite] +# subject = "You have been invited" +# content_path = "./supabase/templates/invite.html" + +[auth.sms] +# Allow/disallow new user signups via SMS to your project. +enable_signup = false +# If enabled, users need to confirm their phone number before signing in. +enable_confirmations = false +# Template for sending OTP to users +template = "Your code is {{ .Code }}" +# Controls the minimum amount of time that must pass before sending another sms otp. +max_frequency = "5s" + +# Use pre-defined map of phone number to OTP for testing. +# [auth.sms.test_otp] +# 4152127777 = "123456" + +# Configure logged in session timeouts. +# [auth.sessions] +# Force log out after the specified duration. +# timebox = "24h" +# Force log out if the user has been inactive longer than the specified duration. +# inactivity_timeout = "8h" + +# This hook runs before a new user is created and allows developers to reject the request based on the incoming user object. +# [auth.hook.before_user_created] +# enabled = true +# uri = "pg-functions://postgres/auth/before-user-created-hook" + +# This hook runs before a token is issued and allows you to add additional claims based on the authentication method used. +# [auth.hook.custom_access_token] +# enabled = true +# uri = "pg-functions:////" + +# Configure one of the supported SMS providers: `twilio`, `twilio_verify`, `messagebird`, `textlocal`, `vonage`. +[auth.sms.twilio] +enabled = false +account_sid = "" +message_service_sid = "" +# DO NOT commit your Twilio auth token to git. Use environment variable substitution instead: +auth_token = "env(SUPABASE_AUTH_SMS_TWILIO_AUTH_TOKEN)" + +# Multi-factor-authentication is available to Supabase Pro plan. +[auth.mfa] +# Control how many MFA factors can be enrolled at once per user. +max_enrolled_factors = 10 + +# Control MFA via App Authenticator (TOTP) +[auth.mfa.totp] +enroll_enabled = false +verify_enabled = false + +# Configure MFA via Phone Messaging +[auth.mfa.phone] +enroll_enabled = false +verify_enabled = false +otp_length = 6 +template = "Your code is {{ .Code }}" +max_frequency = "5s" + +# Configure MFA via WebAuthn +# [auth.mfa.web_authn] +# enroll_enabled = true +# verify_enabled = true + +# Use an external OAuth provider. The full list of providers are: `apple`, `azure`, `bitbucket`, +# `discord`, `facebook`, `github`, `gitlab`, `google`, `keycloak`, `linkedin_oidc`, `notion`, `twitch`, +# `twitter`, `slack`, `spotify`, `workos`, `zoom`. +[auth.external.apple] +enabled = false +client_id = "" +# DO NOT commit your OAuth provider secret to git. Use environment variable substitution instead: +secret = "env(SUPABASE_AUTH_EXTERNAL_APPLE_SECRET)" +# Overrides the default auth redirectUrl. +redirect_uri = "" +# Overrides the default auth provider URL. Used to support self-hosted gitlab, single-tenant Azure, +# or any other third-party OIDC providers. +url = "" +# If enabled, the nonce check will be skipped. Required for local sign in with Google auth. +skip_nonce_check = false +# If enabled, it will allow the user to successfully authenticate when the provider does not return an email address. +email_optional = false + +# Allow Solana wallet holders to sign in to your project via the Sign in with Solana (SIWS, EIP-4361) standard. +# You can configure "web3" rate limit in the [auth.rate_limit] section and set up [auth.captcha] if self-hosting. +[auth.web3.solana] +enabled = false + +# Use Firebase Auth as a third-party provider alongside Supabase Auth. +[auth.third_party.firebase] +enabled = false +# project_id = "my-firebase-project" + +# Use Auth0 as a third-party provider alongside Supabase Auth. +[auth.third_party.auth0] +enabled = false +# tenant = "my-auth0-tenant" +# tenant_region = "us" + +# Use AWS Cognito (Amplify) as a third-party provider alongside Supabase Auth. +[auth.third_party.aws_cognito] +enabled = false +# user_pool_id = "my-user-pool-id" +# user_pool_region = "us-east-1" + +# Use Clerk as a third-party provider alongside Supabase Auth. +[auth.third_party.clerk] +enabled = false +# Obtain from https://clerk.com/setup/supabase +# domain = "example.clerk.accounts.dev" + +# OAuth server configuration +[auth.oauth_server] +# Enable OAuth server functionality +enabled = false +# Path for OAuth consent flow UI +authorization_url_path = "/oauth/consent" +# Allow dynamic client registration +allow_dynamic_registration = false + +[edge_runtime] +enabled = true +# Supported request policies: `oneshot`, `per_worker`. +# `per_worker` (default) — enables hot reload during local development. +# `oneshot` — fallback mode if hot reload causes issues (e.g. in large repos or with symlinks). +policy = "per_worker" +# Port to attach the Chrome inspector for debugging edge functions. +inspector_port = 8083 +# The Deno major version to use. +deno_version = 2 + +# [edge_runtime.secrets] +# secret_key = "env(SECRET_VALUE)" + +[analytics] +enabled = true +port = 54327 +# Configure one of the supported backends: `postgres`, `bigquery`. +backend = "postgres" + +# Experimental features may be deprecated any time +[experimental] +# Configures Postgres storage engine to use OrioleDB (S3) +orioledb_version = "" +# Configures S3 bucket URL, eg. .s3-.amazonaws.com +s3_host = "env(S3_HOST)" +# Configures S3 bucket region, eg. us-east-1 +s3_region = "env(S3_REGION)" +# Configures AWS_ACCESS_KEY_ID for S3 bucket +s3_access_key = "env(S3_ACCESS_KEY)" +# Configures AWS_SECRET_ACCESS_KEY for S3 bucket +s3_secret_key = "env(S3_SECRET_KEY)" diff --git a/apps/demo/supabase/functions/test_flow_worker/.npmrc b/apps/demo/supabase/functions/test_flow_worker/.npmrc new file mode 100644 index 000000000..48c638863 --- /dev/null +++ b/apps/demo/supabase/functions/test_flow_worker/.npmrc @@ -0,0 +1,3 @@ +# Configuration for private npm package dependencies +# For more information on using private registries with Edge Functions, see: +# https://supabase.com/docs/guides/functions/import-maps#importing-from-private-registries diff --git a/apps/demo/supabase/functions/test_flow_worker/deno.json b/apps/demo/supabase/functions/test_flow_worker/deno.json new file mode 100644 index 000000000..b7a100606 --- /dev/null +++ b/apps/demo/supabase/functions/test_flow_worker/deno.json @@ -0,0 +1,15 @@ +{ + "imports": { + "@pgflow/core": "../_vendor/@pgflow/core/index.ts", + "@pgflow/core/": "../_vendor/@pgflow/core/", + "@pgflow/dsl": "../_vendor/@pgflow/dsl/index.ts", + "@pgflow/dsl/": "../_vendor/@pgflow/dsl/", + "@pgflow/dsl/supabase": "../_vendor/@pgflow/dsl/src/supabase.ts", + "@pgflow/edge-worker": "../_vendor/@pgflow/edge-worker/index.ts", + "@pgflow/edge-worker/": "../_vendor/@pgflow/edge-worker/", + "@pgflow/edge-worker/_internal": "../_vendor/@pgflow/edge-worker/_internal.ts", + "postgres": "npm:postgres@3.4.5", + "@henrygd/queue": "jsr:@henrygd/queue@^1.0.7", + "@supabase/supabase-js": "jsr:@supabase/supabase-js@^2.49.4" + } +} diff --git a/apps/demo/supabase/functions/test_flow_worker/deno.lock b/apps/demo/supabase/functions/test_flow_worker/deno.lock new file mode 100644 index 000000000..0a49da32d --- /dev/null +++ b/apps/demo/supabase/functions/test_flow_worker/deno.lock @@ -0,0 +1,118 @@ +{ + "version": "3", + "packages": { + "specifiers": { + "jsr:@henrygd/queue@^1.0.7": "jsr:@henrygd/queue@1.0.7", + "jsr:@supabase/supabase-js@^2.49.4": "jsr:@supabase/supabase-js@2.58.0", + "npm:@supabase/auth-js@2.72.0": "npm:@supabase/auth-js@2.72.0", + "npm:@supabase/functions-js@2.5.0": "npm:@supabase/functions-js@2.5.0", + "npm:@supabase/node-fetch@2.6.15": "npm:@supabase/node-fetch@2.6.15", + "npm:@supabase/postgrest-js@1.21.4": "npm:@supabase/postgrest-js@1.21.4", + "npm:@supabase/realtime-js@2.15.5": "npm:@supabase/realtime-js@2.15.5", + "npm:@supabase/storage-js@2.12.2": "npm:@supabase/storage-js@2.12.2", + "npm:postgres@3.4.5": "npm:postgres@3.4.5" + }, + "jsr": { + "@henrygd/queue@1.0.7": { + "integrity": "98cade132744bb420957c5413393f76eb8ba7261826f026c8a89a562b8fa2961" + }, + "@supabase/supabase-js@2.58.0": { + "integrity": "4d04e72e9f632b451ac7d1a84de0b85249c0097fdf06253f371c1f0a23e62c87", + "dependencies": [ + "npm:@supabase/auth-js@2.72.0", + "npm:@supabase/functions-js@2.5.0", + "npm:@supabase/node-fetch@2.6.15", + "npm:@supabase/postgrest-js@1.21.4", + "npm:@supabase/realtime-js@2.15.5", + "npm:@supabase/storage-js@2.12.2" + ] + } + }, + "npm": { + "@supabase/auth-js@2.72.0": { + "integrity": "sha512-4+bnUrtTDK1YD0/FCx2YtMiQH5FGu9Jlf4IQi5kcqRwRwqp2ey39V61nHNdH86jm3DIzz0aZKiWfTW8qXk1swQ==", + "dependencies": { + "@supabase/node-fetch": "@supabase/node-fetch@2.6.15" + } + }, + "@supabase/functions-js@2.5.0": { + "integrity": "sha512-SXBx6Jvp+MOBekeKFu+G11YLYPeVeGQl23eYyAG9+Ro0pQ1aIP0UZNIBxHKNHqxzR0L0n6gysNr2KT3841NATw==", + "dependencies": { + "@supabase/node-fetch": "@supabase/node-fetch@2.6.15" + } + }, + "@supabase/node-fetch@2.6.15": { + "integrity": "sha512-1ibVeYUacxWYi9i0cf5efil6adJ9WRyZBLivgjs+AUpewx1F3xPi7gLgaASI2SmIQxPoCEjAsLAzKPgMJVgOUQ==", + "dependencies": { + "whatwg-url": "whatwg-url@5.0.0" + } + }, + "@supabase/postgrest-js@1.21.4": { + "integrity": "sha512-TxZCIjxk6/dP9abAi89VQbWWMBbybpGWyvmIzTd79OeravM13OjR/YEYeyUOPcM1C3QyvXkvPZhUfItvmhY1IQ==", + "dependencies": { + "@supabase/node-fetch": "@supabase/node-fetch@2.6.15" + } + }, + "@supabase/realtime-js@2.15.5": { + "integrity": "sha512-/Rs5Vqu9jejRD8ZeuaWXebdkH+J7V6VySbCZ/zQM93Ta5y3mAmocjioa/nzlB6qvFmyylUgKVS1KpE212t30OA==", + "dependencies": { + "@supabase/node-fetch": "@supabase/node-fetch@2.6.15", + "@types/phoenix": "@types/phoenix@1.6.6", + "@types/ws": "@types/ws@8.18.1", + "ws": "ws@8.18.3" + } + }, + "@supabase/storage-js@2.12.2": { + "integrity": "sha512-SiySHxi3q7gia7NBYpsYRu8gyI0NhFwSORMxbZIxJ/zAVkN6QpwDRan158CJ+UdzD4WB/rQMAGRqIJQP+7ccAQ==", + "dependencies": { + "@supabase/node-fetch": "@supabase/node-fetch@2.6.15" + } + }, + "@types/node@18.16.19": { + "integrity": "sha512-IXl7o+R9iti9eBW4Wg2hx1xQDig183jj7YLn8F7udNceyfkbn1ZxmzZXuak20gR40D7pIkIY1kYGx5VIGbaHKA==", + "dependencies": {} + }, + "@types/phoenix@1.6.6": { + "integrity": "sha512-PIzZZlEppgrpoT2QgbnDU+MMzuR6BbCjllj0bM70lWoejMeNJAxCchxnv7J3XFkI8MpygtRpzXrIlmWUBclP5A==", + "dependencies": {} + }, + "@types/ws@8.18.1": { + "integrity": "sha512-ThVF6DCVhA8kUGy+aazFQ4kXQ7E1Ty7A3ypFOe0IcJV8O/M511G99AW24irKrW56Wt44yG9+ij8FaqoBGkuBXg==", + "dependencies": { + "@types/node": "@types/node@18.16.19" + } + }, + "postgres@3.4.5": { + "integrity": "sha512-cDWgoah1Gez9rN3H4165peY9qfpEo+SA61oQv65O3cRUE1pOEoJWwddwcqKE8XZYjbblOJlYDlLV4h67HrEVDg==", + "dependencies": {} + }, + "tr46@0.0.3": { + "integrity": "sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw==", + "dependencies": {} + }, + "webidl-conversions@3.0.1": { + "integrity": "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ==", + "dependencies": {} + }, + "whatwg-url@5.0.0": { + "integrity": "sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw==", + "dependencies": { + "tr46": "tr46@0.0.3", + "webidl-conversions": "webidl-conversions@3.0.1" + } + }, + "ws@8.18.3": { + "integrity": "sha512-PEIGCY5tSlUt50cqyMXfCzX+oOPqN0vuGqWzbcJ2xvnkzkq46oOpz7dQaTDBdfICb4N14+GARUDw2XV2N4tvzg==", + "dependencies": {} + } + } + }, + "remote": {}, + "workspace": { + "dependencies": [ + "jsr:@henrygd/queue@^1.0.7", + "jsr:@supabase/supabase-js@^2.49.4", + "npm:postgres@3.4.5" + ] + } +} diff --git a/apps/demo/supabase/functions/test_flow_worker/index.ts b/apps/demo/supabase/functions/test_flow_worker/index.ts new file mode 100644 index 000000000..47f163928 --- /dev/null +++ b/apps/demo/supabase/functions/test_flow_worker/index.ts @@ -0,0 +1,4 @@ +import { EdgeWorker } from '@pgflow/edge-worker'; +import TestFlow from './test_flow.ts'; + +EdgeWorker.start(TestFlow); diff --git a/apps/demo/supabase/functions/test_flow_worker/test_flow.ts b/apps/demo/supabase/functions/test_flow_worker/test_flow.ts new file mode 100644 index 000000000..e8d638547 --- /dev/null +++ b/apps/demo/supabase/functions/test_flow_worker/test_flow.ts @@ -0,0 +1,6 @@ +import { Flow } from '@pgflow/dsl'; + +export default new Flow<{ message: string }>({ slug: 'test_flow' }).step( + { slug: 'greet' }, + (input) => `Hello, ${input.run.message}!` +); diff --git a/apps/demo/supabase/migrations/20251031133754_20250429164909_pgflow_initial.sql b/apps/demo/supabase/migrations/20251031133754_20250429164909_pgflow_initial.sql new file mode 100644 index 000000000..69150cd89 --- /dev/null +++ b/apps/demo/supabase/migrations/20251031133754_20250429164909_pgflow_initial.sql @@ -0,0 +1,579 @@ +-- Add new schema named "pgflow" +CREATE SCHEMA "pgflow"; +-- Add new schema named "pgmq" +CREATE SCHEMA IF NOT EXISTS "pgmq"; +-- Create extension "pgmq" +CREATE EXTENSION IF NOT EXISTS "pgmq" WITH SCHEMA "pgmq"; +-- Create "read_with_poll" function +CREATE FUNCTION "pgflow"."read_with_poll" ("queue_name" text, "vt" integer, "qty" integer, "max_poll_seconds" integer DEFAULT 5, "poll_interval_ms" integer DEFAULT 100, "conditional" jsonb DEFAULT '{}') RETURNS SETOF pgmq.message_record LANGUAGE plpgsql AS $$ +DECLARE + r pgmq.message_record; + stop_at TIMESTAMP; + sql TEXT; + qtable TEXT := pgmq.format_table_name(queue_name, 'q'); +BEGIN + stop_at := clock_timestamp() + make_interval(secs => max_poll_seconds); + LOOP + IF (SELECT clock_timestamp() >= stop_at) THEN + RETURN; + END IF; + + sql := FORMAT( + $QUERY$ + WITH cte AS + ( + SELECT msg_id + FROM pgmq.%I + WHERE vt <= clock_timestamp() AND CASE + WHEN %L != '{}'::jsonb THEN (message @> %2$L)::integer + ELSE 1 + END = 1 + ORDER BY msg_id ASC + LIMIT $1 + FOR UPDATE SKIP LOCKED + ) + UPDATE pgmq.%I m + SET + vt = clock_timestamp() + %L, + read_ct = read_ct + 1 + FROM cte + WHERE m.msg_id = cte.msg_id + RETURNING m.msg_id, m.read_ct, m.enqueued_at, m.vt, m.message; + $QUERY$, + qtable, conditional, qtable, make_interval(secs => vt) + ); + + FOR r IN + EXECUTE sql USING qty + LOOP + RETURN NEXT r; + END LOOP; + IF FOUND THEN + RETURN; + ELSE + PERFORM pg_sleep(poll_interval_ms::numeric / 1000); + END IF; + END LOOP; +END; +$$; +-- Create composite type "step_task_record" +CREATE TYPE "pgflow"."step_task_record" AS ("flow_slug" text, "run_id" uuid, "step_slug" text, "input" jsonb, "msg_id" bigint); +-- Create "is_valid_slug" function +CREATE FUNCTION "pgflow"."is_valid_slug" ("slug" text) RETURNS boolean LANGUAGE plpgsql IMMUTABLE AS $$ +begin + return + slug is not null + and slug <> '' + and length(slug) <= 128 + and slug ~ '^[a-zA-Z_][a-zA-Z0-9_]*$' + and slug NOT IN ('run'); -- reserved words +end; +$$; +-- Create "flows" table +CREATE TABLE "pgflow"."flows" ("flow_slug" text NOT NULL, "opt_max_attempts" integer NOT NULL DEFAULT 3, "opt_base_delay" integer NOT NULL DEFAULT 1, "opt_timeout" integer NOT NULL DEFAULT 60, "created_at" timestamptz NOT NULL DEFAULT now(), PRIMARY KEY ("flow_slug"), CONSTRAINT "opt_base_delay_is_nonnegative" CHECK (opt_base_delay >= 0), CONSTRAINT "opt_max_attempts_is_nonnegative" CHECK (opt_max_attempts >= 0), CONSTRAINT "opt_timeout_is_positive" CHECK (opt_timeout > 0), CONSTRAINT "slug_is_valid" CHECK (pgflow.is_valid_slug(flow_slug))); +-- Create "steps" table +CREATE TABLE "pgflow"."steps" ("flow_slug" text NOT NULL, "step_slug" text NOT NULL, "step_type" text NOT NULL DEFAULT 'single', "step_index" integer NOT NULL DEFAULT 0, "deps_count" integer NOT NULL DEFAULT 0, "opt_max_attempts" integer NULL, "opt_base_delay" integer NULL, "opt_timeout" integer NULL, "created_at" timestamptz NOT NULL DEFAULT now(), PRIMARY KEY ("flow_slug", "step_slug"), CONSTRAINT "steps_flow_slug_step_index_key" UNIQUE ("flow_slug", "step_index"), CONSTRAINT "steps_flow_slug_fkey" FOREIGN KEY ("flow_slug") REFERENCES "pgflow"."flows" ("flow_slug") ON UPDATE NO ACTION ON DELETE NO ACTION, CONSTRAINT "opt_base_delay_is_nonnegative" CHECK ((opt_base_delay IS NULL) OR (opt_base_delay >= 0)), CONSTRAINT "opt_max_attempts_is_nonnegative" CHECK ((opt_max_attempts IS NULL) OR (opt_max_attempts >= 0)), CONSTRAINT "opt_timeout_is_positive" CHECK ((opt_timeout IS NULL) OR (opt_timeout > 0)), CONSTRAINT "steps_deps_count_check" CHECK (deps_count >= 0), CONSTRAINT "steps_step_slug_check" CHECK (pgflow.is_valid_slug(step_slug)), CONSTRAINT "steps_step_type_check" CHECK (step_type = 'single'::text)); +-- Create "deps" table +CREATE TABLE "pgflow"."deps" ("flow_slug" text NOT NULL, "dep_slug" text NOT NULL, "step_slug" text NOT NULL, "created_at" timestamptz NOT NULL DEFAULT now(), PRIMARY KEY ("flow_slug", "dep_slug", "step_slug"), CONSTRAINT "deps_flow_slug_dep_slug_fkey" FOREIGN KEY ("flow_slug", "dep_slug") REFERENCES "pgflow"."steps" ("flow_slug", "step_slug") ON UPDATE NO ACTION ON DELETE NO ACTION, CONSTRAINT "deps_flow_slug_fkey" FOREIGN KEY ("flow_slug") REFERENCES "pgflow"."flows" ("flow_slug") ON UPDATE NO ACTION ON DELETE NO ACTION, CONSTRAINT "deps_flow_slug_step_slug_fkey" FOREIGN KEY ("flow_slug", "step_slug") REFERENCES "pgflow"."steps" ("flow_slug", "step_slug") ON UPDATE NO ACTION ON DELETE NO ACTION, CONSTRAINT "deps_check" CHECK (dep_slug <> step_slug)); +-- Create index "idx_deps_by_flow_dep" to table: "deps" +CREATE INDEX "idx_deps_by_flow_dep" ON "pgflow"."deps" ("flow_slug", "dep_slug"); +-- Create index "idx_deps_by_flow_step" to table: "deps" +CREATE INDEX "idx_deps_by_flow_step" ON "pgflow"."deps" ("flow_slug", "step_slug"); +-- Create "runs" table +CREATE TABLE "pgflow"."runs" ("run_id" uuid NOT NULL DEFAULT gen_random_uuid(), "flow_slug" text NOT NULL, "status" text NOT NULL DEFAULT 'started', "input" jsonb NOT NULL, "output" jsonb NULL, "remaining_steps" integer NOT NULL DEFAULT 0, "started_at" timestamptz NOT NULL DEFAULT now(), "completed_at" timestamptz NULL, "failed_at" timestamptz NULL, PRIMARY KEY ("run_id"), CONSTRAINT "runs_flow_slug_fkey" FOREIGN KEY ("flow_slug") REFERENCES "pgflow"."flows" ("flow_slug") ON UPDATE NO ACTION ON DELETE NO ACTION, CONSTRAINT "completed_at_is_after_started_at" CHECK ((completed_at IS NULL) OR (completed_at >= started_at)), CONSTRAINT "completed_at_or_failed_at" CHECK (NOT ((completed_at IS NOT NULL) AND (failed_at IS NOT NULL))), CONSTRAINT "failed_at_is_after_started_at" CHECK ((failed_at IS NULL) OR (failed_at >= started_at)), CONSTRAINT "runs_remaining_steps_check" CHECK (remaining_steps >= 0), CONSTRAINT "status_is_valid" CHECK (status = ANY (ARRAY['started'::text, 'failed'::text, 'completed'::text]))); +-- Create index "idx_runs_flow_slug" to table: "runs" +CREATE INDEX "idx_runs_flow_slug" ON "pgflow"."runs" ("flow_slug"); +-- Create index "idx_runs_status" to table: "runs" +CREATE INDEX "idx_runs_status" ON "pgflow"."runs" ("status"); +-- Create "step_states" table +CREATE TABLE "pgflow"."step_states" ("flow_slug" text NOT NULL, "run_id" uuid NOT NULL, "step_slug" text NOT NULL, "status" text NOT NULL DEFAULT 'created', "remaining_tasks" integer NOT NULL DEFAULT 1, "remaining_deps" integer NOT NULL DEFAULT 0, "created_at" timestamptz NOT NULL DEFAULT now(), "started_at" timestamptz NULL, "completed_at" timestamptz NULL, "failed_at" timestamptz NULL, PRIMARY KEY ("run_id", "step_slug"), CONSTRAINT "step_states_flow_slug_fkey" FOREIGN KEY ("flow_slug") REFERENCES "pgflow"."flows" ("flow_slug") ON UPDATE NO ACTION ON DELETE NO ACTION, CONSTRAINT "step_states_flow_slug_step_slug_fkey" FOREIGN KEY ("flow_slug", "step_slug") REFERENCES "pgflow"."steps" ("flow_slug", "step_slug") ON UPDATE NO ACTION ON DELETE NO ACTION, CONSTRAINT "step_states_run_id_fkey" FOREIGN KEY ("run_id") REFERENCES "pgflow"."runs" ("run_id") ON UPDATE NO ACTION ON DELETE NO ACTION, CONSTRAINT "completed_at_is_after_started_at" CHECK ((completed_at IS NULL) OR (completed_at >= started_at)), CONSTRAINT "completed_at_or_failed_at" CHECK (NOT ((completed_at IS NOT NULL) AND (failed_at IS NOT NULL))), CONSTRAINT "failed_at_is_after_started_at" CHECK ((failed_at IS NULL) OR (failed_at >= started_at)), CONSTRAINT "started_at_is_after_created_at" CHECK ((started_at IS NULL) OR (started_at >= created_at)), CONSTRAINT "status_and_remaining_tasks_match" CHECK ((status <> 'completed'::text) OR (remaining_tasks = 0)), CONSTRAINT "status_is_valid" CHECK (status = ANY (ARRAY['created'::text, 'started'::text, 'completed'::text, 'failed'::text])), CONSTRAINT "step_states_remaining_deps_check" CHECK (remaining_deps >= 0), CONSTRAINT "step_states_remaining_tasks_check" CHECK (remaining_tasks >= 0)); +-- Create index "idx_step_states_failed" to table: "step_states" +CREATE INDEX "idx_step_states_failed" ON "pgflow"."step_states" ("run_id", "step_slug") WHERE (status = 'failed'::text); +-- Create index "idx_step_states_flow_slug" to table: "step_states" +CREATE INDEX "idx_step_states_flow_slug" ON "pgflow"."step_states" ("flow_slug"); +-- Create index "idx_step_states_ready" to table: "step_states" +CREATE INDEX "idx_step_states_ready" ON "pgflow"."step_states" ("run_id", "status", "remaining_deps") WHERE ((status = 'created'::text) AND (remaining_deps = 0)); +-- Create "step_tasks" table +CREATE TABLE "pgflow"."step_tasks" ("flow_slug" text NOT NULL, "run_id" uuid NOT NULL, "step_slug" text NOT NULL, "message_id" bigint NULL, "task_index" integer NOT NULL DEFAULT 0, "status" text NOT NULL DEFAULT 'queued', "attempts_count" integer NOT NULL DEFAULT 0, "error_message" text NULL, "output" jsonb NULL, "queued_at" timestamptz NOT NULL DEFAULT now(), "completed_at" timestamptz NULL, "failed_at" timestamptz NULL, PRIMARY KEY ("run_id", "step_slug", "task_index"), CONSTRAINT "step_tasks_flow_slug_fkey" FOREIGN KEY ("flow_slug") REFERENCES "pgflow"."flows" ("flow_slug") ON UPDATE NO ACTION ON DELETE NO ACTION, CONSTRAINT "step_tasks_run_id_fkey" FOREIGN KEY ("run_id") REFERENCES "pgflow"."runs" ("run_id") ON UPDATE NO ACTION ON DELETE NO ACTION, CONSTRAINT "step_tasks_run_id_step_slug_fkey" FOREIGN KEY ("run_id", "step_slug") REFERENCES "pgflow"."step_states" ("run_id", "step_slug") ON UPDATE NO ACTION ON DELETE NO ACTION, CONSTRAINT "attempts_count_nonnegative" CHECK (attempts_count >= 0), CONSTRAINT "completed_at_is_after_queued_at" CHECK ((completed_at IS NULL) OR (completed_at >= queued_at)), CONSTRAINT "completed_at_or_failed_at" CHECK (NOT ((completed_at IS NOT NULL) AND (failed_at IS NOT NULL))), CONSTRAINT "failed_at_is_after_queued_at" CHECK ((failed_at IS NULL) OR (failed_at >= queued_at)), CONSTRAINT "only_single_task_per_step" CHECK (task_index = 0), CONSTRAINT "output_valid_only_for_completed" CHECK ((output IS NULL) OR (status = 'completed'::text)), CONSTRAINT "valid_status" CHECK (status = ANY (ARRAY['queued'::text, 'completed'::text, 'failed'::text]))); +-- Create index "idx_step_tasks_completed" to table: "step_tasks" +CREATE INDEX "idx_step_tasks_completed" ON "pgflow"."step_tasks" ("run_id", "step_slug") WHERE (status = 'completed'::text); +-- Create index "idx_step_tasks_failed" to table: "step_tasks" +CREATE INDEX "idx_step_tasks_failed" ON "pgflow"."step_tasks" ("run_id", "step_slug") WHERE (status = 'failed'::text); +-- Create index "idx_step_tasks_flow_run_step" to table: "step_tasks" +CREATE INDEX "idx_step_tasks_flow_run_step" ON "pgflow"."step_tasks" ("flow_slug", "run_id", "step_slug"); +-- Create index "idx_step_tasks_message_id" to table: "step_tasks" +CREATE INDEX "idx_step_tasks_message_id" ON "pgflow"."step_tasks" ("message_id"); +-- Create index "idx_step_tasks_queued" to table: "step_tasks" +CREATE INDEX "idx_step_tasks_queued" ON "pgflow"."step_tasks" ("run_id", "step_slug") WHERE (status = 'queued'::text); +-- Create "poll_for_tasks" function +CREATE FUNCTION "pgflow"."poll_for_tasks" ("queue_name" text, "vt" integer, "qty" integer, "max_poll_seconds" integer DEFAULT 5, "poll_interval_ms" integer DEFAULT 100) RETURNS SETOF "pgflow"."step_task_record" LANGUAGE sql SET "search_path" = '' AS $$ +with read_messages as ( + select * + from pgflow.read_with_poll( + queue_name, + vt, + qty, + max_poll_seconds, + poll_interval_ms + ) +), +tasks as ( + select + task.flow_slug, + task.run_id, + task.step_slug, + task.task_index, + task.message_id + from pgflow.step_tasks as task + join read_messages as message on message.msg_id = task.message_id + where task.message_id = message.msg_id + and task.status = 'queued' +), +increment_attempts as ( + update pgflow.step_tasks + set attempts_count = attempts_count + 1 + from tasks + where step_tasks.message_id = tasks.message_id + and status = 'queued' +), +runs as ( + select + r.run_id, + r.input + from pgflow.runs r + where r.run_id in (select run_id from tasks) +), +deps as ( + select + st.run_id, + st.step_slug, + dep.dep_slug, + dep_task.output as dep_output + from tasks st + join pgflow.deps dep on dep.flow_slug = st.flow_slug and dep.step_slug = st.step_slug + join pgflow.step_tasks dep_task on + dep_task.run_id = st.run_id and + dep_task.step_slug = dep.dep_slug and + dep_task.status = 'completed' +), +deps_outputs as ( + select + d.run_id, + d.step_slug, + jsonb_object_agg(d.dep_slug, d.dep_output) as deps_output + from deps d + group by d.run_id, d.step_slug +), +timeouts as ( + select + task.message_id, + coalesce(step.opt_timeout, flow.opt_timeout) + 2 as vt_delay + from tasks task + join pgflow.flows flow on flow.flow_slug = task.flow_slug + join pgflow.steps step on step.flow_slug = task.flow_slug and step.step_slug = task.step_slug +) + +select + st.flow_slug, + st.run_id, + st.step_slug, + jsonb_build_object('run', r.input) || + coalesce(dep_out.deps_output, '{}'::jsonb) as input, + st.message_id as msg_id +from tasks st +join runs r on st.run_id = r.run_id +left join deps_outputs dep_out on + dep_out.run_id = st.run_id and + dep_out.step_slug = st.step_slug +cross join lateral ( + -- TODO: this is slow because it calls set_vt for each row, and set_vt + -- builds dynamic query from string every time it is called + -- implement set_vt_batch(msgs_ids bigint[], vt_delays int[]) + select pgmq.set_vt(queue_name, st.message_id, + (select t.vt_delay from timeouts t where t.message_id = st.message_id) + ) +) set_vt; +$$; +-- Create "add_step" function +CREATE FUNCTION "pgflow"."add_step" ("flow_slug" text, "step_slug" text, "deps_slugs" text[], "max_attempts" integer DEFAULT NULL::integer, "base_delay" integer DEFAULT NULL::integer, "timeout" integer DEFAULT NULL::integer) RETURNS "pgflow"."steps" LANGUAGE sql SET "search_path" = '' AS $$ +WITH + next_index AS ( + SELECT COALESCE(MAX(step_index) + 1, 0) as idx + FROM pgflow.steps + WHERE flow_slug = add_step.flow_slug + ), + create_step AS ( + INSERT INTO pgflow.steps (flow_slug, step_slug, step_index, deps_count, opt_max_attempts, opt_base_delay, opt_timeout) + SELECT add_step.flow_slug, add_step.step_slug, idx, COALESCE(array_length(deps_slugs, 1), 0), max_attempts, base_delay, timeout + FROM next_index + ON CONFLICT (flow_slug, step_slug) + DO UPDATE SET step_slug = pgflow.steps.step_slug + RETURNING * + ), + insert_deps AS ( + INSERT INTO pgflow.deps (flow_slug, dep_slug, step_slug) + SELECT add_step.flow_slug, d.dep_slug, add_step.step_slug + FROM unnest(deps_slugs) AS d(dep_slug) + ON CONFLICT (flow_slug, dep_slug, step_slug) DO NOTHING + RETURNING 1 + ) +-- Return the created step +SELECT * FROM create_step; +$$; +-- Create "add_step" function +CREATE FUNCTION "pgflow"."add_step" ("flow_slug" text, "step_slug" text, "max_attempts" integer DEFAULT NULL::integer, "base_delay" integer DEFAULT NULL::integer, "timeout" integer DEFAULT NULL::integer) RETURNS "pgflow"."steps" LANGUAGE sql SET "search_path" = '' AS $$ +-- Call the original function with an empty array + SELECT * FROM pgflow.add_step(flow_slug, step_slug, ARRAY[]::text[], max_attempts, base_delay, timeout); +$$; +-- Create "calculate_retry_delay" function +CREATE FUNCTION "pgflow"."calculate_retry_delay" ("base_delay" numeric, "attempts_count" integer) RETURNS integer LANGUAGE sql IMMUTABLE PARALLEL SAFE AS $$ select floor(base_delay * power(2, attempts_count))::int $$; +-- Create "maybe_complete_run" function +CREATE FUNCTION "pgflow"."maybe_complete_run" ("run_id" uuid) RETURNS void LANGUAGE sql SET "search_path" = '' AS $$ +-- Update run status to completed and set output when there are no remaining steps + -- All done in a single declarative SQL statement + UPDATE pgflow.runs + SET + status = 'completed', + completed_at = now(), + output = ( + -- Get outputs from final steps (steps that are not dependencies for other steps) + SELECT jsonb_object_agg(st.step_slug, st.output) + FROM pgflow.step_tasks st + JOIN pgflow.step_states ss ON ss.run_id = st.run_id AND ss.step_slug = st.step_slug + JOIN pgflow.runs r ON r.run_id = ss.run_id AND r.flow_slug = ss.flow_slug + WHERE st.run_id = maybe_complete_run.run_id + AND st.status = 'completed' + AND NOT EXISTS ( + SELECT 1 + FROM pgflow.deps d + WHERE d.flow_slug = ss.flow_slug + AND d.dep_slug = ss.step_slug + ) + ) + WHERE pgflow.runs.run_id = maybe_complete_run.run_id + AND pgflow.runs.remaining_steps = 0 + AND pgflow.runs.status != 'completed'; +$$; +-- Create "start_ready_steps" function +CREATE FUNCTION "pgflow"."start_ready_steps" ("run_id" uuid) RETURNS void LANGUAGE sql SET "search_path" = '' AS $$ +WITH ready_steps AS ( + SELECT * + FROM pgflow.step_states AS step_state + WHERE step_state.run_id = start_ready_steps.run_id + AND step_state.status = 'created' + AND step_state.remaining_deps = 0 + ORDER BY step_state.step_slug + FOR UPDATE +), +started_step_states AS ( + UPDATE pgflow.step_states + SET status = 'started', + started_at = now() + FROM ready_steps + WHERE pgflow.step_states.run_id = start_ready_steps.run_id + AND pgflow.step_states.step_slug = ready_steps.step_slug + RETURNING pgflow.step_states.* +), +sent_messages AS ( + SELECT + started_step.flow_slug, + started_step.run_id, + started_step.step_slug, + pgmq.send(started_step.flow_slug, jsonb_build_object( + 'flow_slug', started_step.flow_slug, + 'run_id', started_step.run_id, + 'step_slug', started_step.step_slug, + 'task_index', 0 + )) AS msg_id + FROM started_step_states AS started_step +) +INSERT INTO pgflow.step_tasks (flow_slug, run_id, step_slug, message_id) +SELECT + sent_messages.flow_slug, + sent_messages.run_id, + sent_messages.step_slug, + sent_messages.msg_id +FROM sent_messages; +$$; +-- Create "complete_task" function +CREATE FUNCTION "pgflow"."complete_task" ("run_id" uuid, "step_slug" text, "task_index" integer, "output" jsonb) RETURNS SETOF "pgflow"."step_tasks" LANGUAGE plpgsql SET "search_path" = '' AS $$ +begin + +WITH run_lock AS ( + SELECT * FROM pgflow.runs + WHERE pgflow.runs.run_id = complete_task.run_id + FOR UPDATE +), +step_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug + FOR UPDATE +), +task AS ( + UPDATE pgflow.step_tasks + SET + status = 'completed', + completed_at = now(), + output = complete_task.output + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index + RETURNING * +), +step_state AS ( + UPDATE pgflow.step_states + SET + status = CASE + WHEN pgflow.step_states.remaining_tasks = 1 THEN 'completed' -- Will be 0 after decrement + ELSE 'started' + END, + completed_at = CASE + WHEN pgflow.step_states.remaining_tasks = 1 THEN now() -- Will be 0 after decrement + ELSE NULL + END, + remaining_tasks = pgflow.step_states.remaining_tasks - 1 + FROM task + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug + RETURNING pgflow.step_states.* +), +-- Find all dependent steps if the current step was completed +dependent_steps AS ( + SELECT d.step_slug AS dependent_step_slug + FROM pgflow.deps d + JOIN step_state s ON s.status = 'completed' AND d.flow_slug = s.flow_slug + WHERE d.dep_slug = complete_task.step_slug + ORDER BY d.step_slug -- Ensure consistent ordering +), +-- Lock dependent steps before updating +dependent_steps_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug IN (SELECT dependent_step_slug FROM dependent_steps) + FOR UPDATE +), +-- Update all dependent steps +dependent_steps_update AS ( + UPDATE pgflow.step_states + SET remaining_deps = pgflow.step_states.remaining_deps - 1 + FROM dependent_steps + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = dependent_steps.dependent_step_slug +) +-- Only decrement remaining_steps, don't update status +UPDATE pgflow.runs +SET remaining_steps = pgflow.runs.remaining_steps - 1 +FROM step_state +WHERE pgflow.runs.run_id = complete_task.run_id + AND step_state.status = 'completed'; + +PERFORM pgmq.archive( + queue_name => (SELECT run.flow_slug FROM pgflow.runs AS run WHERE run.run_id = complete_task.run_id), + msg_id => (SELECT message_id FROM pgflow.step_tasks AS step_task + WHERE step_task.run_id = complete_task.run_id + AND step_task.step_slug = complete_task.step_slug + AND step_task.task_index = complete_task.task_index) +); + +PERFORM pgflow.start_ready_steps(complete_task.run_id); + +PERFORM pgflow.maybe_complete_run(complete_task.run_id); + +RETURN QUERY SELECT * +FROM pgflow.step_tasks AS step_task +WHERE step_task.run_id = complete_task.run_id + AND step_task.step_slug = complete_task.step_slug + AND step_task.task_index = complete_task.task_index; + +end; +$$; +-- Create "create_flow" function +CREATE FUNCTION "pgflow"."create_flow" ("flow_slug" text, "max_attempts" integer DEFAULT 3, "base_delay" integer DEFAULT 5, "timeout" integer DEFAULT 60) RETURNS "pgflow"."flows" LANGUAGE sql SET "search_path" = '' AS $$ +WITH + flow_upsert AS ( + INSERT INTO pgflow.flows (flow_slug, opt_max_attempts, opt_base_delay, opt_timeout) + VALUES (flow_slug, max_attempts, base_delay, timeout) + ON CONFLICT (flow_slug) DO UPDATE + SET flow_slug = pgflow.flows.flow_slug -- Dummy update + RETURNING * + ), + ensure_queue AS ( + SELECT pgmq.create(flow_slug) + WHERE NOT EXISTS ( + SELECT 1 FROM pgmq.list_queues() WHERE queue_name = flow_slug + ) + ) +SELECT f.* +FROM flow_upsert f +LEFT JOIN (SELECT 1 FROM ensure_queue) _dummy ON true; -- Left join ensures flow is returned +$$; +-- Create "fail_task" function +CREATE FUNCTION "pgflow"."fail_task" ("run_id" uuid, "step_slug" text, "task_index" integer, "error_message" text) RETURNS SETOF "pgflow"."step_tasks" LANGUAGE plpgsql SET "search_path" = '' AS $$ +begin + +WITH run_lock AS ( + SELECT * FROM pgflow.runs + WHERE pgflow.runs.run_id = fail_task.run_id + FOR UPDATE +), +step_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug + FOR UPDATE +), +flow_info AS ( + SELECT r.flow_slug + FROM pgflow.runs r + WHERE r.run_id = fail_task.run_id +), +config AS ( + SELECT + COALESCE(s.opt_max_attempts, f.opt_max_attempts) AS opt_max_attempts, + COALESCE(s.opt_base_delay, f.opt_base_delay) AS opt_base_delay + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN flow_info fi ON fi.flow_slug = s.flow_slug + WHERE s.flow_slug = fi.flow_slug AND s.step_slug = fail_task.step_slug +), + +fail_or_retry_task as ( + UPDATE pgflow.step_tasks as task + SET + status = CASE + WHEN task.attempts_count < (SELECT opt_max_attempts FROM config) THEN 'queued' + ELSE 'failed' + END, + failed_at = CASE + WHEN task.attempts_count >= (SELECT opt_max_attempts FROM config) THEN now() + ELSE NULL + END, + error_message = fail_task.error_message + WHERE task.run_id = fail_task.run_id + AND task.step_slug = fail_task.step_slug + AND task.task_index = fail_task.task_index + AND task.status = 'queued' + RETURNING * +), +maybe_fail_step AS ( + UPDATE pgflow.step_states + SET + status = CASE + WHEN (select fail_or_retry_task.status from fail_or_retry_task) = 'failed' THEN 'failed' + ELSE pgflow.step_states.status + END, + failed_at = CASE + WHEN (select fail_or_retry_task.status from fail_or_retry_task) = 'failed' THEN now() + ELSE NULL + END + FROM fail_or_retry_task + WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug + RETURNING pgflow.step_states.* +) +UPDATE pgflow.runs +SET status = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN 'failed' + ELSE status + END, + failed_at = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN now() + ELSE NULL + END +WHERE pgflow.runs.run_id = fail_task.run_id; + +-- For queued tasks: delay the message for retry with exponential backoff +PERFORM ( + WITH retry_config AS ( + SELECT + COALESCE(s.opt_base_delay, f.opt_base_delay) AS base_delay + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN pgflow.runs r ON r.flow_slug = f.flow_slug + WHERE r.run_id = fail_task.run_id + AND s.step_slug = fail_task.step_slug + ), + queued_tasks AS ( + SELECT + r.flow_slug, + st.message_id, + pgflow.calculate_retry_delay((SELECT base_delay FROM retry_config), st.attempts_count) AS calculated_delay + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'queued' + ) + SELECT pgmq.set_vt(qt.flow_slug, qt.message_id, qt.calculated_delay) + FROM queued_tasks qt + WHERE EXISTS (SELECT 1 FROM queued_tasks) +); + +-- For failed tasks: archive the message +PERFORM ( + WITH failed_tasks AS ( + SELECT r.flow_slug, st.message_id + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'failed' + ) + SELECT pgmq.archive(ft.flow_slug, ft.message_id) + FROM failed_tasks ft + WHERE EXISTS (SELECT 1 FROM failed_tasks) +); + +return query select * +from pgflow.step_tasks st +where st.run_id = fail_task.run_id + and st.step_slug = fail_task.step_slug + and st.task_index = fail_task.task_index; + +end; +$$; +-- Create "start_flow" function +CREATE FUNCTION "pgflow"."start_flow" ("flow_slug" text, "input" jsonb) RETURNS SETOF "pgflow"."runs" LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_created_run pgflow.runs%ROWTYPE; +begin + +WITH + flow_steps AS ( + SELECT steps.flow_slug, steps.step_slug, steps.deps_count + FROM pgflow.steps + WHERE steps.flow_slug = start_flow.flow_slug + ), + created_run AS ( + INSERT INTO pgflow.runs (flow_slug, input, remaining_steps) + VALUES ( + start_flow.flow_slug, + start_flow.input, + (SELECT count(*) FROM flow_steps) + ) + RETURNING * + ), + created_step_states AS ( + INSERT INTO pgflow.step_states (flow_slug, run_id, step_slug, remaining_deps) + SELECT + fs.flow_slug, + (SELECT run_id FROM created_run), + fs.step_slug, + fs.deps_count + FROM flow_steps fs + ) +SELECT * FROM created_run INTO v_created_run; + +PERFORM pgflow.start_ready_steps(v_created_run.run_id); + +RETURN QUERY SELECT * FROM pgflow.runs where run_id = v_created_run.run_id; + +end; +$$; +-- Create "workers" table +CREATE TABLE "pgflow"."workers" ("worker_id" uuid NOT NULL, "queue_name" text NOT NULL, "function_name" text NOT NULL, "started_at" timestamptz NOT NULL DEFAULT now(), "stopped_at" timestamptz NULL, "last_heartbeat_at" timestamptz NOT NULL DEFAULT now(), PRIMARY KEY ("worker_id")); +-- Create index "idx_workers_queue_name" to table: "workers" +CREATE INDEX "idx_workers_queue_name" ON "pgflow"."workers" ("queue_name"); diff --git a/apps/demo/supabase/migrations/20251031133755_20250517072017_pgflow_fix_poll_for_tasks_to_use_separate_statement_for_polling.sql b/apps/demo/supabase/migrations/20251031133755_20250517072017_pgflow_fix_poll_for_tasks_to_use_separate_statement_for_polling.sql new file mode 100644 index 000000000..74f0e91c4 --- /dev/null +++ b/apps/demo/supabase/migrations/20251031133755_20250517072017_pgflow_fix_poll_for_tasks_to_use_separate_statement_for_polling.sql @@ -0,0 +1,101 @@ +-- Modify "poll_for_tasks" function +CREATE OR REPLACE FUNCTION "pgflow"."poll_for_tasks" ("queue_name" text, "vt" integer, "qty" integer, "max_poll_seconds" integer DEFAULT 5, "poll_interval_ms" integer DEFAULT 100) RETURNS SETOF "pgflow"."step_task_record" LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + msg_ids bigint[]; +begin + -- First statement: Read messages and capture their IDs + -- This gets its own snapshot and can see newly committed messages + select array_agg(msg_id) + into msg_ids + from pgflow.read_with_poll( + queue_name, + vt, + qty, + max_poll_seconds, + poll_interval_ms + ); + + -- If no messages were read, return empty set + if msg_ids is null or array_length(msg_ids, 1) is null then + return; + end if; + + -- Second statement: Process tasks with fresh snapshot + -- This can now see step_tasks that were committed during the poll + return query + with tasks as ( + select + task.flow_slug, + task.run_id, + task.step_slug, + task.task_index, + task.message_id + from pgflow.step_tasks as task + where task.message_id = any(msg_ids) + and task.status = 'queued' + ), + increment_attempts as ( + update pgflow.step_tasks + set attempts_count = attempts_count + 1 + from tasks + where step_tasks.message_id = tasks.message_id + and status = 'queued' + ), + runs as ( + select + r.run_id, + r.input + from pgflow.runs r + where r.run_id in (select run_id from tasks) + ), + deps as ( + select + st.run_id, + st.step_slug, + dep.dep_slug, + dep_task.output as dep_output + from tasks st + join pgflow.deps dep on dep.flow_slug = st.flow_slug and dep.step_slug = st.step_slug + join pgflow.step_tasks dep_task on + dep_task.run_id = st.run_id and + dep_task.step_slug = dep.dep_slug and + dep_task.status = 'completed' + ), + deps_outputs as ( + select + d.run_id, + d.step_slug, + jsonb_object_agg(d.dep_slug, d.dep_output) as deps_output + from deps d + group by d.run_id, d.step_slug + ), + timeouts as ( + select + task.message_id, + coalesce(step.opt_timeout, flow.opt_timeout) + 2 as vt_delay + from tasks task + join pgflow.flows flow on flow.flow_slug = task.flow_slug + join pgflow.steps step on step.flow_slug = task.flow_slug and step.step_slug = task.step_slug + ) + select + st.flow_slug, + st.run_id, + st.step_slug, + jsonb_build_object('run', r.input) || + coalesce(dep_out.deps_output, '{}'::jsonb) as input, + st.message_id as msg_id + from tasks st + join runs r on st.run_id = r.run_id + left join deps_outputs dep_out on + dep_out.run_id = st.run_id and + dep_out.step_slug = st.step_slug + cross join lateral ( + -- TODO: this is slow because it calls set_vt for each row, and set_vt + -- builds dynamic query from string every time it is called + -- implement set_vt_batch(msgs_ids bigint[], vt_delays int[]) + select pgmq.set_vt(queue_name, st.message_id, + (select t.vt_delay from timeouts t where t.message_id = st.message_id) + ) + ) set_vt; +end; +$$; diff --git a/apps/demo/supabase/migrations/20251031133756_20250609105135_pgflow_add_start_tasks_and_started_status.sql b/apps/demo/supabase/migrations/20251031133756_20250609105135_pgflow_add_start_tasks_and_started_status.sql new file mode 100644 index 000000000..7f696c8b5 --- /dev/null +++ b/apps/demo/supabase/migrations/20251031133756_20250609105135_pgflow_add_start_tasks_and_started_status.sql @@ -0,0 +1,371 @@ +-- Create index "idx_workers_heartbeat" to table: "workers" +create index "idx_workers_heartbeat" on "pgflow"."workers" ("last_heartbeat_at"); +-- Modify "step_tasks" table +alter table "pgflow"."step_tasks" drop constraint "valid_status", +add constraint "valid_status" check ( + status = ANY(array['queued'::text, 'started'::text, 'completed'::text, 'failed'::text]) +), +add constraint "completed_at_is_after_started_at" check ( + (completed_at is null) or (started_at is null) or (completed_at >= started_at) +), +add constraint "failed_at_is_after_started_at" check ( + (failed_at is null) or (started_at is null) or (failed_at >= started_at) +), +add constraint "started_at_is_after_queued_at" check ((started_at is null) or (started_at >= queued_at)), +add column "started_at" timestamptz null, +add column "last_worker_id" uuid null, +add constraint "step_tasks_last_worker_id_fkey" foreign key ("last_worker_id") references "pgflow"."workers" ( + "worker_id" +) on update no action on delete set null; +-- Create index "idx_step_tasks_last_worker" to table: "step_tasks" +create index "idx_step_tasks_last_worker" on "pgflow"."step_tasks" ("last_worker_id") where (status = 'started'::text); +-- Create index "idx_step_tasks_queued_msg" to table: "step_tasks" +create index "idx_step_tasks_queued_msg" on "pgflow"."step_tasks" ("message_id") where (status = 'queued'::text); +-- Create index "idx_step_tasks_started" to table: "step_tasks" +create index "idx_step_tasks_started" on "pgflow"."step_tasks" ("started_at") where (status = 'started'::text); +-- Modify "complete_task" function +create or replace function "pgflow"."complete_task"( + "run_id" uuid, "step_slug" text, "task_index" integer, "output" jsonb +) returns setof "pgflow"."step_tasks" language plpgsql set "search_path" += '' as $$ +begin + +WITH run_lock AS ( + SELECT * FROM pgflow.runs + WHERE pgflow.runs.run_id = complete_task.run_id + FOR UPDATE +), +step_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug + FOR UPDATE +), +task AS ( + UPDATE pgflow.step_tasks + SET + status = 'completed', + completed_at = now(), + output = complete_task.output + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index + AND pgflow.step_tasks.status = 'started' + RETURNING * +), +step_state AS ( + UPDATE pgflow.step_states + SET + status = CASE + WHEN pgflow.step_states.remaining_tasks = 1 THEN 'completed' -- Will be 0 after decrement + ELSE 'started' + END, + completed_at = CASE + WHEN pgflow.step_states.remaining_tasks = 1 THEN now() -- Will be 0 after decrement + ELSE NULL + END, + remaining_tasks = pgflow.step_states.remaining_tasks - 1 + FROM task + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug + RETURNING pgflow.step_states.* +), +-- Find all dependent steps if the current step was completed +dependent_steps AS ( + SELECT d.step_slug AS dependent_step_slug + FROM pgflow.deps d + JOIN step_state s ON s.status = 'completed' AND d.flow_slug = s.flow_slug + WHERE d.dep_slug = complete_task.step_slug + ORDER BY d.step_slug -- Ensure consistent ordering +), +-- Lock dependent steps before updating +dependent_steps_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug IN (SELECT dependent_step_slug FROM dependent_steps) + FOR UPDATE +), +-- Update all dependent steps +dependent_steps_update AS ( + UPDATE pgflow.step_states + SET remaining_deps = pgflow.step_states.remaining_deps - 1 + FROM dependent_steps + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = dependent_steps.dependent_step_slug +) +-- Only decrement remaining_steps, don't update status +UPDATE pgflow.runs +SET remaining_steps = pgflow.runs.remaining_steps - 1 +FROM step_state +WHERE pgflow.runs.run_id = complete_task.run_id + AND step_state.status = 'completed'; + +-- For completed tasks: archive the message +PERFORM ( + WITH completed_tasks AS ( + SELECT r.flow_slug, st.message_id + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.task_index = complete_task.task_index + AND st.status = 'completed' + ) + SELECT pgmq.archive(ct.flow_slug, ct.message_id) + FROM completed_tasks ct + WHERE EXISTS (SELECT 1 FROM completed_tasks) +); + +PERFORM pgflow.start_ready_steps(complete_task.run_id); + +PERFORM pgflow.maybe_complete_run(complete_task.run_id); + +RETURN QUERY SELECT * +FROM pgflow.step_tasks AS step_task +WHERE step_task.run_id = complete_task.run_id + AND step_task.step_slug = complete_task.step_slug + AND step_task.task_index = complete_task.task_index; + +end; +$$; +-- Modify "fail_task" function +create or replace function "pgflow"."fail_task"( + "run_id" uuid, "step_slug" text, "task_index" integer, "error_message" text +) returns setof "pgflow"."step_tasks" language plpgsql set "search_path" += '' as $$ +begin + +WITH run_lock AS ( + SELECT * FROM pgflow.runs + WHERE pgflow.runs.run_id = fail_task.run_id + FOR UPDATE +), +step_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug + FOR UPDATE +), +flow_info AS ( + SELECT r.flow_slug + FROM pgflow.runs r + WHERE r.run_id = fail_task.run_id +), +config AS ( + SELECT + COALESCE(s.opt_max_attempts, f.opt_max_attempts) AS opt_max_attempts, + COALESCE(s.opt_base_delay, f.opt_base_delay) AS opt_base_delay + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN flow_info fi ON fi.flow_slug = s.flow_slug + WHERE s.flow_slug = fi.flow_slug AND s.step_slug = fail_task.step_slug +), + +fail_or_retry_task as ( + UPDATE pgflow.step_tasks as task + SET + status = CASE + WHEN task.attempts_count < (SELECT opt_max_attempts FROM config) THEN 'queued' + ELSE 'failed' + END, + failed_at = CASE + WHEN task.attempts_count >= (SELECT opt_max_attempts FROM config) THEN now() + ELSE NULL + END, + started_at = CASE + WHEN task.attempts_count < (SELECT opt_max_attempts FROM config) THEN NULL + ELSE task.started_at + END, + error_message = fail_task.error_message + WHERE task.run_id = fail_task.run_id + AND task.step_slug = fail_task.step_slug + AND task.task_index = fail_task.task_index + AND task.status = 'started' + RETURNING * +), +maybe_fail_step AS ( + UPDATE pgflow.step_states + SET + status = CASE + WHEN (select fail_or_retry_task.status from fail_or_retry_task) = 'failed' THEN 'failed' + ELSE pgflow.step_states.status + END, + failed_at = CASE + WHEN (select fail_or_retry_task.status from fail_or_retry_task) = 'failed' THEN now() + ELSE NULL + END + FROM fail_or_retry_task + WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug + RETURNING pgflow.step_states.* +) +UPDATE pgflow.runs +SET status = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN 'failed' + ELSE status + END, + failed_at = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN now() + ELSE NULL + END +WHERE pgflow.runs.run_id = fail_task.run_id; + +-- For queued tasks: delay the message for retry with exponential backoff +PERFORM ( + WITH retry_config AS ( + SELECT + COALESCE(s.opt_base_delay, f.opt_base_delay) AS base_delay + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN pgflow.runs r ON r.flow_slug = f.flow_slug + WHERE r.run_id = fail_task.run_id + AND s.step_slug = fail_task.step_slug + ), + queued_tasks AS ( + SELECT + r.flow_slug, + st.message_id, + pgflow.calculate_retry_delay((SELECT base_delay FROM retry_config), st.attempts_count) AS calculated_delay + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'queued' + ) + SELECT pgmq.set_vt(qt.flow_slug, qt.message_id, qt.calculated_delay) + FROM queued_tasks qt + WHERE EXISTS (SELECT 1 FROM queued_tasks) +); + +-- For failed tasks: archive the message +PERFORM ( + WITH failed_tasks AS ( + SELECT r.flow_slug, st.message_id + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'failed' + ) + SELECT pgmq.archive(ft.flow_slug, ft.message_id) + FROM failed_tasks ft + WHERE EXISTS (SELECT 1 FROM failed_tasks) +); + +return query select * +from pgflow.step_tasks st +where st.run_id = fail_task.run_id + and st.step_slug = fail_task.step_slug + and st.task_index = fail_task.task_index; + +end; +$$; +-- Modify "poll_for_tasks" function +create or replace function "pgflow"."poll_for_tasks"( + "queue_name" text, + "vt" integer, + "qty" integer, + "max_poll_seconds" integer default 5, + "poll_interval_ms" integer default 100 +) returns setof "pgflow"."step_task_record" language plpgsql set "search_path" += '' as $$ +begin + -- DEPRECATED: This function is deprecated and will be removed in a future version. + -- Please update pgflow to use the new two-phase polling approach. + -- Run 'npx pgflow install' to update your installation. + raise notice 'DEPRECATED: poll_for_tasks is deprecated and will be removed. Please update pgflow via "npx pgflow install".'; + + -- Return empty set - no tasks will be processed + return; +end; +$$; +-- Create "start_tasks" function +create function "pgflow"."start_tasks"( + "flow_slug" text, "msg_ids" bigint [], "worker_id" uuid +) returns setof "pgflow"."step_task_record" language sql set "search_path" += '' as $$ +with tasks as ( + select + task.flow_slug, + task.run_id, + task.step_slug, + task.task_index, + task.message_id + from pgflow.step_tasks as task + where task.flow_slug = start_tasks.flow_slug + and task.message_id = any(msg_ids) + and task.status = 'queued' + ), + start_tasks_update as ( + update pgflow.step_tasks + set + attempts_count = attempts_count + 1, + status = 'started', + started_at = now(), + last_worker_id = worker_id + from tasks + where step_tasks.message_id = tasks.message_id + and step_tasks.flow_slug = tasks.flow_slug + and step_tasks.status = 'queued' + ), + runs as ( + select + r.run_id, + r.input + from pgflow.runs r + where r.run_id in (select run_id from tasks) + ), + deps as ( + select + st.run_id, + st.step_slug, + dep.dep_slug, + dep_task.output as dep_output + from tasks st + join pgflow.deps dep on dep.flow_slug = st.flow_slug and dep.step_slug = st.step_slug + join pgflow.step_tasks dep_task on + dep_task.run_id = st.run_id and + dep_task.step_slug = dep.dep_slug and + dep_task.status = 'completed' + ), + deps_outputs as ( + select + d.run_id, + d.step_slug, + jsonb_object_agg(d.dep_slug, d.dep_output) as deps_output + from deps d + group by d.run_id, d.step_slug + ), + timeouts as ( + select + task.message_id, + task.flow_slug, + coalesce(step.opt_timeout, flow.opt_timeout) + 2 as vt_delay + from tasks task + join pgflow.flows flow on flow.flow_slug = task.flow_slug + join pgflow.steps step on step.flow_slug = task.flow_slug and step.step_slug = task.step_slug + ) + select + st.flow_slug, + st.run_id, + st.step_slug, + jsonb_build_object('run', r.input) || + coalesce(dep_out.deps_output, '{}'::jsonb) as input, + st.message_id as msg_id + from tasks st + join runs r on st.run_id = r.run_id + left join deps_outputs dep_out on + dep_out.run_id = st.run_id and + dep_out.step_slug = st.step_slug + cross join lateral ( + -- TODO: this is slow because it calls set_vt for each row, and set_vt + -- builds dynamic query from string every time it is called + -- implement set_vt_batch(msgs_ids bigint[], vt_delays int[]) + select pgmq.set_vt(t.flow_slug, st.message_id, t.vt_delay) + from timeouts t + where t.message_id = st.message_id + and t.flow_slug = st.flow_slug + ) set_vt +$$; diff --git a/apps/demo/supabase/migrations/20251031133757_20250610180554_pgflow_add_set_vt_batch_and_use_it_in_start_tasks.sql b/apps/demo/supabase/migrations/20251031133757_20250610180554_pgflow_add_set_vt_batch_and_use_it_in_start_tasks.sql new file mode 100644 index 000000000..1e8585a5a --- /dev/null +++ b/apps/demo/supabase/migrations/20251031133757_20250610180554_pgflow_add_set_vt_batch_and_use_it_in_start_tasks.sql @@ -0,0 +1,127 @@ +-- Create "set_vt_batch" function +CREATE FUNCTION "pgflow"."set_vt_batch" ("queue_name" text, "msg_ids" bigint[], "vt_offsets" integer[]) RETURNS SETOF pgmq.message_record LANGUAGE plpgsql AS $$ +DECLARE + qtable TEXT := pgmq.format_table_name(queue_name, 'q'); + sql TEXT; +BEGIN + /* ---------- safety checks ---------------------------------------------------- */ + IF msg_ids IS NULL OR vt_offsets IS NULL OR array_length(msg_ids, 1) = 0 THEN + RETURN; -- nothing to do, return empty set + END IF; + + IF array_length(msg_ids, 1) IS DISTINCT FROM array_length(vt_offsets, 1) THEN + RAISE EXCEPTION + 'msg_ids length (%) must equal vt_offsets length (%)', + array_length(msg_ids, 1), array_length(vt_offsets, 1); + END IF; + + /* ---------- dynamic statement ------------------------------------------------ */ + /* One UPDATE joins with the unnested arrays */ + sql := format( + $FMT$ + WITH input (msg_id, vt_offset) AS ( + SELECT unnest($1)::bigint + , unnest($2)::int + ) + UPDATE pgmq.%I q + SET vt = clock_timestamp() + make_interval(secs => input.vt_offset), + read_ct = read_ct -- no change, but keeps RETURNING list aligned + FROM input + WHERE q.msg_id = input.msg_id + RETURNING q.msg_id, + q.read_ct, + q.enqueued_at, + q.vt, + q.message + $FMT$, + qtable + ); + + RETURN QUERY EXECUTE sql USING msg_ids, vt_offsets; +END; +$$; +-- Modify "start_tasks" function +CREATE OR REPLACE FUNCTION "pgflow"."start_tasks" ("flow_slug" text, "msg_ids" bigint[], "worker_id" uuid) RETURNS SETOF "pgflow"."step_task_record" LANGUAGE sql SET "search_path" = '' AS $$ +with tasks as ( + select + task.flow_slug, + task.run_id, + task.step_slug, + task.task_index, + task.message_id + from pgflow.step_tasks as task + where task.flow_slug = start_tasks.flow_slug + and task.message_id = any(msg_ids) + and task.status = 'queued' + ), + start_tasks_update as ( + update pgflow.step_tasks + set + attempts_count = attempts_count + 1, + status = 'started', + started_at = now(), + last_worker_id = worker_id + from tasks + where step_tasks.message_id = tasks.message_id + and step_tasks.flow_slug = tasks.flow_slug + and step_tasks.status = 'queued' + ), + runs as ( + select + r.run_id, + r.input + from pgflow.runs r + where r.run_id in (select run_id from tasks) + ), + deps as ( + select + st.run_id, + st.step_slug, + dep.dep_slug, + dep_task.output as dep_output + from tasks st + join pgflow.deps dep on dep.flow_slug = st.flow_slug and dep.step_slug = st.step_slug + join pgflow.step_tasks dep_task on + dep_task.run_id = st.run_id and + dep_task.step_slug = dep.dep_slug and + dep_task.status = 'completed' + ), + deps_outputs as ( + select + d.run_id, + d.step_slug, + jsonb_object_agg(d.dep_slug, d.dep_output) as deps_output + from deps d + group by d.run_id, d.step_slug + ), + timeouts as ( + select + task.message_id, + task.flow_slug, + coalesce(step.opt_timeout, flow.opt_timeout) + 2 as vt_delay + from tasks task + join pgflow.flows flow on flow.flow_slug = task.flow_slug + join pgflow.steps step on step.flow_slug = task.flow_slug and step.step_slug = task.step_slug + ), + -- Batch update visibility timeouts for all messages + set_vt_batch as ( + select pgflow.set_vt_batch( + start_tasks.flow_slug, + array_agg(t.message_id order by t.message_id), + array_agg(t.vt_delay order by t.message_id) + ) + from timeouts t + ) + select + st.flow_slug, + st.run_id, + st.step_slug, + jsonb_build_object('run', r.input) || + coalesce(dep_out.deps_output, '{}'::jsonb) as input, + st.message_id as msg_id + from tasks st + join runs r on st.run_id = r.run_id + left join deps_outputs dep_out on + dep_out.run_id = st.run_id and + dep_out.step_slug = st.step_slug +$$; diff --git a/apps/demo/supabase/migrations/20251031133758_20250614124241_pgflow_add_realtime.sql b/apps/demo/supabase/migrations/20251031133758_20250614124241_pgflow_add_realtime.sql new file mode 100644 index 000000000..353ade1ca --- /dev/null +++ b/apps/demo/supabase/migrations/20251031133758_20250614124241_pgflow_add_realtime.sql @@ -0,0 +1,501 @@ +-- Modify "step_states" table +ALTER TABLE "pgflow"."step_states" ADD COLUMN "error_message" text NULL; +-- Create index "idx_step_states_run_id" to table: "step_states" +CREATE INDEX "idx_step_states_run_id" ON "pgflow"."step_states" ("run_id"); +-- Modify "maybe_complete_run" function +CREATE OR REPLACE FUNCTION "pgflow"."maybe_complete_run" ("run_id" uuid) RETURNS void LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_completed_run pgflow.runs%ROWTYPE; +begin + -- Update run status to completed and set output when there are no remaining steps + WITH run_output AS ( + -- Get outputs from final steps (steps that are not dependencies for other steps) + SELECT jsonb_object_agg(st.step_slug, st.output) as final_output + FROM pgflow.step_tasks st + JOIN pgflow.step_states ss ON ss.run_id = st.run_id AND ss.step_slug = st.step_slug + JOIN pgflow.runs r ON r.run_id = ss.run_id AND r.flow_slug = ss.flow_slug + WHERE st.run_id = maybe_complete_run.run_id + AND st.status = 'completed' + AND NOT EXISTS ( + SELECT 1 + FROM pgflow.deps d + WHERE d.flow_slug = ss.flow_slug + AND d.dep_slug = ss.step_slug + ) + ) + UPDATE pgflow.runs + SET + status = 'completed', + completed_at = now(), + output = (SELECT final_output FROM run_output) + WHERE pgflow.runs.run_id = maybe_complete_run.run_id + AND pgflow.runs.remaining_steps = 0 + AND pgflow.runs.status != 'completed' + RETURNING * INTO v_completed_run; + + -- Only send broadcast if run was completed + IF v_completed_run.run_id IS NOT NULL THEN + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:completed', + 'run_id', v_completed_run.run_id, + 'flow_slug', v_completed_run.flow_slug, + 'status', 'completed', + 'output', v_completed_run.output, + 'completed_at', v_completed_run.completed_at + ), + 'run:completed', + concat('pgflow:run:', v_completed_run.run_id), + false + ); + END IF; +end; +$$; +-- Modify "start_ready_steps" function +CREATE OR REPLACE FUNCTION "pgflow"."start_ready_steps" ("run_id" uuid) RETURNS void LANGUAGE sql SET "search_path" = '' AS $$ +WITH ready_steps AS ( + SELECT * + FROM pgflow.step_states AS step_state + WHERE step_state.run_id = start_ready_steps.run_id + AND step_state.status = 'created' + AND step_state.remaining_deps = 0 + ORDER BY step_state.step_slug + FOR UPDATE +), +started_step_states AS ( + UPDATE pgflow.step_states + SET status = 'started', + started_at = now() + FROM ready_steps + WHERE pgflow.step_states.run_id = start_ready_steps.run_id + AND pgflow.step_states.step_slug = ready_steps.step_slug + RETURNING pgflow.step_states.* +), +sent_messages AS ( + SELECT + started_step.flow_slug, + started_step.run_id, + started_step.step_slug, + pgmq.send(started_step.flow_slug, jsonb_build_object( + 'flow_slug', started_step.flow_slug, + 'run_id', started_step.run_id, + 'step_slug', started_step.step_slug, + 'task_index', 0 + )) AS msg_id + FROM started_step_states AS started_step +), +broadcast_events AS ( + SELECT + realtime.send( + jsonb_build_object( + 'event_type', 'step:started', + 'run_id', started_step.run_id, + 'step_slug', started_step.step_slug, + 'status', 'started', + 'started_at', started_step.started_at, + 'remaining_tasks', 1, + 'remaining_deps', started_step.remaining_deps + ), + concat('step:', started_step.step_slug, ':started'), + concat('pgflow:run:', started_step.run_id), + false + ) + FROM started_step_states AS started_step +) +INSERT INTO pgflow.step_tasks (flow_slug, run_id, step_slug, message_id) +SELECT + sent_messages.flow_slug, + sent_messages.run_id, + sent_messages.step_slug, + sent_messages.msg_id +FROM sent_messages; +$$; +-- Modify "complete_task" function +CREATE OR REPLACE FUNCTION "pgflow"."complete_task" ("run_id" uuid, "step_slug" text, "task_index" integer, "output" jsonb) RETURNS SETOF "pgflow"."step_tasks" LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_step_state pgflow.step_states%ROWTYPE; +begin + +WITH run_lock AS ( + SELECT * FROM pgflow.runs + WHERE pgflow.runs.run_id = complete_task.run_id + FOR UPDATE +), +step_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug + FOR UPDATE +), +task AS ( + UPDATE pgflow.step_tasks + SET + status = 'completed', + completed_at = now(), + output = complete_task.output + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index + AND pgflow.step_tasks.status = 'started' + RETURNING * +), +step_state AS ( + UPDATE pgflow.step_states + SET + status = CASE + WHEN pgflow.step_states.remaining_tasks = 1 THEN 'completed' -- Will be 0 after decrement + ELSE 'started' + END, + completed_at = CASE + WHEN pgflow.step_states.remaining_tasks = 1 THEN now() -- Will be 0 after decrement + ELSE NULL + END, + remaining_tasks = pgflow.step_states.remaining_tasks - 1 + FROM task + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug + RETURNING pgflow.step_states.* +), +-- Find all dependent steps if the current step was completed +dependent_steps AS ( + SELECT d.step_slug AS dependent_step_slug + FROM pgflow.deps d + JOIN step_state s ON s.status = 'completed' AND d.flow_slug = s.flow_slug + WHERE d.dep_slug = complete_task.step_slug + ORDER BY d.step_slug -- Ensure consistent ordering +), +-- Lock dependent steps before updating +dependent_steps_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug IN (SELECT dependent_step_slug FROM dependent_steps) + FOR UPDATE +), +-- Update all dependent steps +dependent_steps_update AS ( + UPDATE pgflow.step_states + SET remaining_deps = pgflow.step_states.remaining_deps - 1 + FROM dependent_steps + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = dependent_steps.dependent_step_slug +) +-- Only decrement remaining_steps, don't update status +UPDATE pgflow.runs +SET remaining_steps = pgflow.runs.remaining_steps - 1 +FROM step_state +WHERE pgflow.runs.run_id = complete_task.run_id + AND step_state.status = 'completed'; + +-- Get the updated step state for broadcasting +SELECT * INTO v_step_state FROM pgflow.step_states +WHERE pgflow.step_states.run_id = complete_task.run_id AND pgflow.step_states.step_slug = complete_task.step_slug; + +-- Send broadcast event for step completed if the step is completed +IF v_step_state.status = 'completed' THEN + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:completed', + 'run_id', complete_task.run_id, + 'step_slug', complete_task.step_slug, + 'status', 'completed', + 'output', complete_task.output, + 'completed_at', v_step_state.completed_at + ), + concat('step:', complete_task.step_slug, ':completed'), + concat('pgflow:run:', complete_task.run_id), + false + ); +END IF; + +-- For completed tasks: archive the message +PERFORM ( + WITH completed_tasks AS ( + SELECT r.flow_slug, st.message_id + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.task_index = complete_task.task_index + AND st.status = 'completed' + ) + SELECT pgmq.archive(ct.flow_slug, ct.message_id) + FROM completed_tasks ct + WHERE EXISTS (SELECT 1 FROM completed_tasks) +); + +PERFORM pgflow.start_ready_steps(complete_task.run_id); + +PERFORM pgflow.maybe_complete_run(complete_task.run_id); + +RETURN QUERY SELECT * +FROM pgflow.step_tasks AS step_task +WHERE step_task.run_id = complete_task.run_id + AND step_task.step_slug = complete_task.step_slug + AND step_task.task_index = complete_task.task_index; + +end; +$$; +-- Modify "fail_task" function +CREATE OR REPLACE FUNCTION "pgflow"."fail_task" ("run_id" uuid, "step_slug" text, "task_index" integer, "error_message" text) RETURNS SETOF "pgflow"."step_tasks" LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + v_run_failed boolean; +begin + +WITH run_lock AS ( + SELECT * FROM pgflow.runs + WHERE pgflow.runs.run_id = fail_task.run_id + FOR UPDATE +), +step_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug + FOR UPDATE +), +flow_info AS ( + SELECT r.flow_slug + FROM pgflow.runs r + WHERE r.run_id = fail_task.run_id +), +config AS ( + SELECT + COALESCE(s.opt_max_attempts, f.opt_max_attempts) AS opt_max_attempts, + COALESCE(s.opt_base_delay, f.opt_base_delay) AS opt_base_delay + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN flow_info fi ON fi.flow_slug = s.flow_slug + WHERE s.flow_slug = fi.flow_slug AND s.step_slug = fail_task.step_slug +), +fail_or_retry_task as ( + UPDATE pgflow.step_tasks as task + SET + status = CASE + WHEN task.attempts_count < (SELECT opt_max_attempts FROM config) THEN 'queued' + ELSE 'failed' + END, + failed_at = CASE + WHEN task.attempts_count >= (SELECT opt_max_attempts FROM config) THEN now() + ELSE NULL + END, + started_at = CASE + WHEN task.attempts_count < (SELECT opt_max_attempts FROM config) THEN NULL + ELSE task.started_at + END, + error_message = fail_task.error_message + WHERE task.run_id = fail_task.run_id + AND task.step_slug = fail_task.step_slug + AND task.task_index = fail_task.task_index + AND task.status = 'started' + RETURNING * +), +maybe_fail_step AS ( + UPDATE pgflow.step_states + SET + status = CASE + WHEN (select fail_or_retry_task.status from fail_or_retry_task) = 'failed' THEN 'failed' + ELSE pgflow.step_states.status + END, + failed_at = CASE + WHEN (select fail_or_retry_task.status from fail_or_retry_task) = 'failed' THEN now() + ELSE NULL + END, + error_message = CASE + WHEN (select fail_or_retry_task.status from fail_or_retry_task) = 'failed' THEN fail_task.error_message + ELSE NULL + END + FROM fail_or_retry_task + WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug + RETURNING pgflow.step_states.* +), +-- Send broadcast event for step failed if necessary +broadcast_step_failed AS ( + SELECT + realtime.send( + jsonb_build_object( + 'event_type', 'step:failed', + 'run_id', fail_task.run_id, + 'step_slug', fail_task.step_slug, + 'status', 'failed', + 'error_message', fail_task.error_message, + 'failed_at', now() + ), + concat('step:', fail_task.step_slug, ':failed'), + concat('pgflow:run:', fail_task.run_id), + false + ) + FROM maybe_fail_step + WHERE maybe_fail_step.status = 'failed' +) +-- Only decrement remaining_steps, don't update status +UPDATE pgflow.runs +SET status = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN 'failed' + ELSE status + END, + failed_at = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN now() + ELSE NULL + END +WHERE pgflow.runs.run_id = fail_task.run_id +RETURNING (status = 'failed') INTO v_run_failed; + +-- Send broadcast event for run failure if the run was failed +IF v_run_failed THEN + DECLARE + v_flow_slug text; + BEGIN + SELECT flow_slug INTO v_flow_slug FROM pgflow.runs WHERE pgflow.runs.run_id = fail_task.run_id; + + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:failed', + 'run_id', fail_task.run_id, + 'flow_slug', v_flow_slug, + 'status', 'failed', + 'error_message', fail_task.error_message, + 'failed_at', now() + ), + 'run:failed', + concat('pgflow:run:', fail_task.run_id), + false + ); + END; +END IF; + +-- For queued tasks: delay the message for retry with exponential backoff +PERFORM ( + WITH retry_config AS ( + SELECT + COALESCE(s.opt_base_delay, f.opt_base_delay) AS base_delay + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN pgflow.runs r ON r.flow_slug = f.flow_slug + WHERE r.run_id = fail_task.run_id + AND s.step_slug = fail_task.step_slug + ), + queued_tasks AS ( + SELECT + r.flow_slug, + st.message_id, + pgflow.calculate_retry_delay((SELECT base_delay FROM retry_config), st.attempts_count) AS calculated_delay + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'queued' + ) + SELECT pgmq.set_vt(qt.flow_slug, qt.message_id, qt.calculated_delay) + FROM queued_tasks qt + WHERE EXISTS (SELECT 1 FROM queued_tasks) +); + +-- For failed tasks: archive the message +PERFORM ( + WITH failed_tasks AS ( + SELECT r.flow_slug, st.message_id + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'failed' + ) + SELECT pgmq.archive(ft.flow_slug, ft.message_id) + FROM failed_tasks ft + WHERE EXISTS (SELECT 1 FROM failed_tasks) +); + +return query select * +from pgflow.step_tasks st +where st.run_id = fail_task.run_id + and st.step_slug = fail_task.step_slug + and st.task_index = fail_task.task_index; + +end; +$$; +-- Create "get_run_with_states" function +CREATE FUNCTION "pgflow"."get_run_with_states" ("run_id" uuid) RETURNS jsonb LANGUAGE sql SECURITY DEFINER AS $$ +SELECT jsonb_build_object( + 'run', to_jsonb(r), + 'steps', COALESCE(jsonb_agg(to_jsonb(s)) FILTER (WHERE s.run_id IS NOT NULL), '[]'::jsonb) + ) + FROM pgflow.runs r + LEFT JOIN pgflow.step_states s ON s.run_id = r.run_id + WHERE r.run_id = get_run_with_states.run_id + GROUP BY r.run_id; +$$; +-- Create "start_flow" function +CREATE FUNCTION "pgflow"."start_flow" ("flow_slug" text, "input" jsonb, "run_id" uuid DEFAULT NULL::uuid) RETURNS SETOF "pgflow"."runs" LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_created_run pgflow.runs%ROWTYPE; +begin + +WITH + flow_steps AS ( + SELECT steps.flow_slug, steps.step_slug, steps.deps_count + FROM pgflow.steps + WHERE steps.flow_slug = start_flow.flow_slug + ), + created_run AS ( + INSERT INTO pgflow.runs (run_id, flow_slug, input, remaining_steps) + VALUES ( + COALESCE(start_flow.run_id, gen_random_uuid()), + start_flow.flow_slug, + start_flow.input, + (SELECT count(*) FROM flow_steps) + ) + RETURNING * + ), + created_step_states AS ( + INSERT INTO pgflow.step_states (flow_slug, run_id, step_slug, remaining_deps) + SELECT + fs.flow_slug, + (SELECT created_run.run_id FROM created_run), + fs.step_slug, + fs.deps_count + FROM flow_steps fs + ) +SELECT * FROM created_run INTO v_created_run; + +-- Send broadcast event for run started +PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:started', + 'run_id', v_created_run.run_id, + 'flow_slug', v_created_run.flow_slug, + 'input', v_created_run.input, + 'status', 'started', + 'remaining_steps', v_created_run.remaining_steps, + 'started_at', v_created_run.started_at + ), + 'run:started', + concat('pgflow:run:', v_created_run.run_id), + false +); + +PERFORM pgflow.start_ready_steps(v_created_run.run_id); + +RETURN QUERY SELECT * FROM pgflow.runs where pgflow.runs.run_id = v_created_run.run_id; + +end; +$$; +-- Create "start_flow_with_states" function +CREATE FUNCTION "pgflow"."start_flow_with_states" ("flow_slug" text, "input" jsonb, "run_id" uuid DEFAULT NULL::uuid) RETURNS jsonb LANGUAGE plpgsql SECURITY DEFINER AS $$ +DECLARE + v_run_id UUID; +BEGIN + -- Start the flow using existing function + SELECT r.run_id INTO v_run_id FROM pgflow.start_flow( + start_flow_with_states.flow_slug, + start_flow_with_states.input, + start_flow_with_states.run_id + ) AS r LIMIT 1; + + -- Use get_run_with_states to return the complete state + RETURN pgflow.get_run_with_states(v_run_id); +END; +$$; +-- Drop "start_flow" function +DROP FUNCTION "pgflow"."start_flow" (text, jsonb); diff --git a/apps/demo/supabase/migrations/20251031133759_20250619195327_pgflow_fix_fail_task_missing_realtime_event.sql b/apps/demo/supabase/migrations/20251031133759_20250619195327_pgflow_fix_fail_task_missing_realtime_event.sql new file mode 100644 index 000000000..9782f5393 --- /dev/null +++ b/apps/demo/supabase/migrations/20251031133759_20250619195327_pgflow_fix_fail_task_missing_realtime_event.sql @@ -0,0 +1,185 @@ +-- Modify "fail_task" function +CREATE OR REPLACE FUNCTION "pgflow"."fail_task" ("run_id" uuid, "step_slug" text, "task_index" integer, "error_message" text) RETURNS SETOF "pgflow"."step_tasks" LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + v_run_failed boolean; + v_step_failed boolean; +begin + +WITH run_lock AS ( + SELECT * FROM pgflow.runs + WHERE pgflow.runs.run_id = fail_task.run_id + FOR UPDATE +), +step_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug + FOR UPDATE +), +flow_info AS ( + SELECT r.flow_slug + FROM pgflow.runs r + WHERE r.run_id = fail_task.run_id +), +config AS ( + SELECT + COALESCE(s.opt_max_attempts, f.opt_max_attempts) AS opt_max_attempts, + COALESCE(s.opt_base_delay, f.opt_base_delay) AS opt_base_delay + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN flow_info fi ON fi.flow_slug = s.flow_slug + WHERE s.flow_slug = fi.flow_slug AND s.step_slug = fail_task.step_slug +), +fail_or_retry_task as ( + UPDATE pgflow.step_tasks as task + SET + status = CASE + WHEN task.attempts_count < (SELECT opt_max_attempts FROM config) THEN 'queued' + ELSE 'failed' + END, + failed_at = CASE + WHEN task.attempts_count >= (SELECT opt_max_attempts FROM config) THEN now() + ELSE NULL + END, + started_at = CASE + WHEN task.attempts_count < (SELECT opt_max_attempts FROM config) THEN NULL + ELSE task.started_at + END, + error_message = fail_task.error_message + WHERE task.run_id = fail_task.run_id + AND task.step_slug = fail_task.step_slug + AND task.task_index = fail_task.task_index + AND task.status = 'started' + RETURNING * +), +maybe_fail_step AS ( + UPDATE pgflow.step_states + SET + status = CASE + WHEN (select fail_or_retry_task.status from fail_or_retry_task) = 'failed' THEN 'failed' + ELSE pgflow.step_states.status + END, + failed_at = CASE + WHEN (select fail_or_retry_task.status from fail_or_retry_task) = 'failed' THEN now() + ELSE NULL + END, + error_message = CASE + WHEN (select fail_or_retry_task.status from fail_or_retry_task) = 'failed' THEN fail_task.error_message + ELSE NULL + END + FROM fail_or_retry_task + WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug + RETURNING pgflow.step_states.* +) +-- Update run status +UPDATE pgflow.runs +SET status = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN 'failed' + ELSE status + END, + failed_at = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN now() + ELSE NULL + END +WHERE pgflow.runs.run_id = fail_task.run_id +RETURNING (status = 'failed') INTO v_run_failed; + +-- Check if step failed by querying the step_states table +SELECT (status = 'failed') INTO v_step_failed +FROM pgflow.step_states +WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug; + +-- Send broadcast event for step failure if the step was failed +IF v_step_failed THEN + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:failed', + 'run_id', fail_task.run_id, + 'step_slug', fail_task.step_slug, + 'status', 'failed', + 'error_message', fail_task.error_message, + 'failed_at', now() + ), + concat('step:', fail_task.step_slug, ':failed'), + concat('pgflow:run:', fail_task.run_id), + false + ); +END IF; + +-- Send broadcast event for run failure if the run was failed +IF v_run_failed THEN + DECLARE + v_flow_slug text; + BEGIN + SELECT flow_slug INTO v_flow_slug FROM pgflow.runs WHERE pgflow.runs.run_id = fail_task.run_id; + + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:failed', + 'run_id', fail_task.run_id, + 'flow_slug', v_flow_slug, + 'status', 'failed', + 'error_message', fail_task.error_message, + 'failed_at', now() + ), + 'run:failed', + concat('pgflow:run:', fail_task.run_id), + false + ); + END; +END IF; + +-- For queued tasks: delay the message for retry with exponential backoff +PERFORM ( + WITH retry_config AS ( + SELECT + COALESCE(s.opt_base_delay, f.opt_base_delay) AS base_delay + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN pgflow.runs r ON r.flow_slug = f.flow_slug + WHERE r.run_id = fail_task.run_id + AND s.step_slug = fail_task.step_slug + ), + queued_tasks AS ( + SELECT + r.flow_slug, + st.message_id, + pgflow.calculate_retry_delay((SELECT base_delay FROM retry_config), st.attempts_count) AS calculated_delay + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'queued' + ) + SELECT pgmq.set_vt(qt.flow_slug, qt.message_id, qt.calculated_delay) + FROM queued_tasks qt + WHERE EXISTS (SELECT 1 FROM queued_tasks) +); + +-- For failed tasks: archive the message +PERFORM ( + WITH failed_tasks AS ( + SELECT r.flow_slug, st.message_id + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'failed' + ) + SELECT pgmq.archive(ft.flow_slug, ft.message_id) + FROM failed_tasks ft + WHERE EXISTS (SELECT 1 FROM failed_tasks) +); + +return query select * +from pgflow.step_tasks st +where st.run_id = fail_task.run_id + and st.step_slug = fail_task.step_slug + and st.task_index = fail_task.task_index; + +end; +$$; diff --git a/apps/demo/supabase/migrations/20251031133800_20250627090700_pgflow_fix_function_search_paths.sql b/apps/demo/supabase/migrations/20251031133800_20250627090700_pgflow_fix_function_search_paths.sql new file mode 100644 index 000000000..09d325c00 --- /dev/null +++ b/apps/demo/supabase/migrations/20251031133800_20250627090700_pgflow_fix_function_search_paths.sql @@ -0,0 +1,6 @@ +-- Add "calculate_retry_delay" function configuration parameter +ALTER FUNCTION "pgflow"."calculate_retry_delay" SET "search_path" = ''; +-- Add "is_valid_slug" function configuration parameter +ALTER FUNCTION "pgflow"."is_valid_slug" SET "search_path" = ''; +-- Add "read_with_poll" function configuration parameter +ALTER FUNCTION "pgflow"."read_with_poll" SET "search_path" = ''; diff --git a/apps/demo/supabase/migrations/20251031133801_20250707210212_pgflow_add_opt_start_delay.sql b/apps/demo/supabase/migrations/20251031133801_20250707210212_pgflow_add_opt_start_delay.sql new file mode 100644 index 000000000..28e05deb8 --- /dev/null +++ b/apps/demo/supabase/migrations/20251031133801_20250707210212_pgflow_add_opt_start_delay.sql @@ -0,0 +1,103 @@ +-- Modify "steps" table +ALTER TABLE "pgflow"."steps" ADD CONSTRAINT "opt_start_delay_is_nonnegative" CHECK ((opt_start_delay IS NULL) OR (opt_start_delay >= 0)), ADD COLUMN "opt_start_delay" integer NULL; +-- Modify "start_ready_steps" function +CREATE OR REPLACE FUNCTION "pgflow"."start_ready_steps" ("run_id" uuid) RETURNS void LANGUAGE sql SET "search_path" = '' AS $$ +WITH ready_steps AS ( + SELECT * + FROM pgflow.step_states AS step_state + WHERE step_state.run_id = start_ready_steps.run_id + AND step_state.status = 'created' + AND step_state.remaining_deps = 0 + ORDER BY step_state.step_slug + FOR UPDATE +), +started_step_states AS ( + UPDATE pgflow.step_states + SET status = 'started', + started_at = now() + FROM ready_steps + WHERE pgflow.step_states.run_id = start_ready_steps.run_id + AND pgflow.step_states.step_slug = ready_steps.step_slug + RETURNING pgflow.step_states.* +), +sent_messages AS ( + SELECT + started_step.flow_slug, + started_step.run_id, + started_step.step_slug, + pgmq.send( + started_step.flow_slug, + jsonb_build_object( + 'flow_slug', started_step.flow_slug, + 'run_id', started_step.run_id, + 'step_slug', started_step.step_slug, + 'task_index', 0 + ), + COALESCE(step.opt_start_delay, 0) + ) AS msg_id + FROM started_step_states AS started_step + JOIN pgflow.steps AS step + ON step.flow_slug = started_step.flow_slug + AND step.step_slug = started_step.step_slug +), +broadcast_events AS ( + SELECT + realtime.send( + jsonb_build_object( + 'event_type', 'step:started', + 'run_id', started_step.run_id, + 'step_slug', started_step.step_slug, + 'status', 'started', + 'started_at', started_step.started_at, + 'remaining_tasks', 1, + 'remaining_deps', started_step.remaining_deps + ), + concat('step:', started_step.step_slug, ':started'), + concat('pgflow:run:', started_step.run_id), + false + ) + FROM started_step_states AS started_step +) +INSERT INTO pgflow.step_tasks (flow_slug, run_id, step_slug, message_id) +SELECT + sent_messages.flow_slug, + sent_messages.run_id, + sent_messages.step_slug, + sent_messages.msg_id +FROM sent_messages; +$$; +-- Create "add_step" function +CREATE FUNCTION "pgflow"."add_step" ("flow_slug" text, "step_slug" text, "deps_slugs" text[], "max_attempts" integer DEFAULT NULL::integer, "base_delay" integer DEFAULT NULL::integer, "timeout" integer DEFAULT NULL::integer, "start_delay" integer DEFAULT NULL::integer) RETURNS "pgflow"."steps" LANGUAGE sql SET "search_path" = '' AS $$ +WITH + next_index AS ( + SELECT COALESCE(MAX(step_index) + 1, 0) as idx + FROM pgflow.steps + WHERE flow_slug = add_step.flow_slug + ), + create_step AS ( + INSERT INTO pgflow.steps (flow_slug, step_slug, step_index, deps_count, opt_max_attempts, opt_base_delay, opt_timeout, opt_start_delay) + SELECT add_step.flow_slug, add_step.step_slug, idx, COALESCE(array_length(deps_slugs, 1), 0), max_attempts, base_delay, timeout, start_delay + FROM next_index + ON CONFLICT (flow_slug, step_slug) + DO UPDATE SET step_slug = pgflow.steps.step_slug + RETURNING * + ), + insert_deps AS ( + INSERT INTO pgflow.deps (flow_slug, dep_slug, step_slug) + SELECT add_step.flow_slug, d.dep_slug, add_step.step_slug + FROM unnest(deps_slugs) AS d(dep_slug) + ON CONFLICT (flow_slug, dep_slug, step_slug) DO NOTHING + RETURNING 1 + ) +-- Return the created step +SELECT * FROM create_step; +$$; +-- Drop "add_step" function +DROP FUNCTION "pgflow"."add_step" (text, text, integer, integer, integer); +-- Drop "add_step" function +DROP FUNCTION "pgflow"."add_step" (text, text, text[], integer, integer, integer); +-- Create "add_step" function +CREATE FUNCTION "pgflow"."add_step" ("flow_slug" text, "step_slug" text, "max_attempts" integer DEFAULT NULL::integer, "base_delay" integer DEFAULT NULL::integer, "timeout" integer DEFAULT NULL::integer, "start_delay" integer DEFAULT NULL::integer) RETURNS "pgflow"."steps" LANGUAGE sql SET "search_path" = '' AS $$ +-- Call the original function with an empty array + SELECT * FROM pgflow.add_step(flow_slug, step_slug, ARRAY[]::text[], max_attempts, base_delay, timeout, start_delay); +$$; diff --git a/apps/demo/supabase/migrations/20251031133802_20250719205006_pgflow_worker_deprecation.sql b/apps/demo/supabase/migrations/20251031133802_20250719205006_pgflow_worker_deprecation.sql new file mode 100644 index 000000000..783852022 --- /dev/null +++ b/apps/demo/supabase/migrations/20251031133802_20250719205006_pgflow_worker_deprecation.sql @@ -0,0 +1,2 @@ +-- Rename a column from "stopped_at" to "deprecated_at" +ALTER TABLE "pgflow"."workers" RENAME COLUMN "stopped_at" TO "deprecated_at"; diff --git a/apps/demo/supabase/migrations/20251031133803_20251006073122_pgflow_add_map_step_type.sql b/apps/demo/supabase/migrations/20251031133803_20251006073122_pgflow_add_map_step_type.sql new file mode 100644 index 000000000..ed359ea57 --- /dev/null +++ b/apps/demo/supabase/migrations/20251031133803_20251006073122_pgflow_add_map_step_type.sql @@ -0,0 +1,1244 @@ +-- Modify "step_task_record" composite type +ALTER TYPE "pgflow"."step_task_record" ADD ATTRIBUTE "task_index" integer; +-- Modify "step_states" table - Step 1: Drop old constraint and NOT NULL +ALTER TABLE "pgflow"."step_states" + DROP CONSTRAINT "step_states_remaining_tasks_check", + ALTER COLUMN "remaining_tasks" DROP NOT NULL, + ALTER COLUMN "remaining_tasks" DROP DEFAULT, + ADD COLUMN "initial_tasks" integer NULL; +-- AUTOMATIC DATA MIGRATION: Prepare existing data for new constraints +-- This runs AFTER dropping NOT NULL but BEFORE adding new constraints +-- All old steps had exactly 1 task (enforced by old only_single_task_per_step constraint) + +-- Backfill initial_tasks = 1 for all existing steps +-- (Old schema enforced exactly 1 task per step, so all steps had initial_tasks=1) +UPDATE "pgflow"."step_states" +SET "initial_tasks" = 1 +WHERE "initial_tasks" IS NULL; + +-- Set remaining_tasks to NULL for 'created' status +-- (New semantics: NULL = not started, old semantics: 1 = not started) +UPDATE "pgflow"."step_states" +SET "remaining_tasks" = NULL +WHERE "status" = 'created' AND "remaining_tasks" IS NOT NULL; +-- Modify "step_states" table - Step 2: Add new constraints +ALTER TABLE "pgflow"."step_states" + ADD CONSTRAINT "initial_tasks_known_when_started" CHECK ((status <> 'started'::text) OR (initial_tasks IS NOT NULL)), + ADD CONSTRAINT "remaining_tasks_state_consistency" CHECK ((remaining_tasks IS NULL) OR (status <> 'created'::text)), + ADD CONSTRAINT "step_states_initial_tasks_check" CHECK ((initial_tasks IS NULL) OR (initial_tasks >= 0)); +-- Modify "step_tasks" table +ALTER TABLE "pgflow"."step_tasks" DROP CONSTRAINT "only_single_task_per_step", DROP CONSTRAINT "output_valid_only_for_completed", ADD CONSTRAINT "output_valid_only_for_completed" CHECK ((output IS NULL) OR (status = ANY (ARRAY['completed'::text, 'failed'::text]))); +-- Modify "steps" table +ALTER TABLE "pgflow"."steps" DROP CONSTRAINT "steps_step_type_check", ADD CONSTRAINT "steps_step_type_check" CHECK (step_type = ANY (ARRAY['single'::text, 'map'::text])); +-- Modify "maybe_complete_run" function +CREATE OR REPLACE FUNCTION "pgflow"."maybe_complete_run" ("run_id" uuid) RETURNS void LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_completed_run pgflow.runs%ROWTYPE; +begin + -- ========================================== + -- CHECK AND COMPLETE RUN IF FINISHED + -- ========================================== + -- ---------- Complete run if all steps done ---------- + UPDATE pgflow.runs + SET + status = 'completed', + completed_at = now(), + -- Only compute expensive aggregation when actually completing the run + output = ( + -- ---------- Gather outputs from leaf steps ---------- + -- Leaf steps = steps with no dependents + -- For map steps: aggregate all task outputs into array + -- For single steps: use the single task output + SELECT jsonb_object_agg( + step_slug, + CASE + WHEN step_type = 'map' THEN aggregated_output + ELSE single_output + END + ) + FROM ( + SELECT DISTINCT + leaf_state.step_slug, + leaf_step.step_type, + -- For map steps: aggregate all task outputs + CASE WHEN leaf_step.step_type = 'map' THEN + (SELECT COALESCE(jsonb_agg(leaf_task.output ORDER BY leaf_task.task_index), '[]'::jsonb) + FROM pgflow.step_tasks leaf_task + WHERE leaf_task.run_id = leaf_state.run_id + AND leaf_task.step_slug = leaf_state.step_slug + AND leaf_task.status = 'completed') + END as aggregated_output, + -- For single steps: get the single output + CASE WHEN leaf_step.step_type = 'single' THEN + (SELECT leaf_task.output + FROM pgflow.step_tasks leaf_task + WHERE leaf_task.run_id = leaf_state.run_id + AND leaf_task.step_slug = leaf_state.step_slug + AND leaf_task.status = 'completed' + LIMIT 1) + END as single_output + FROM pgflow.step_states leaf_state + JOIN pgflow.steps leaf_step ON leaf_step.flow_slug = leaf_state.flow_slug AND leaf_step.step_slug = leaf_state.step_slug + WHERE leaf_state.run_id = maybe_complete_run.run_id + AND leaf_state.status = 'completed' + AND NOT EXISTS ( + SELECT 1 + FROM pgflow.deps dep + WHERE dep.flow_slug = leaf_state.flow_slug + AND dep.dep_slug = leaf_state.step_slug + ) + ) leaf_outputs + ) + WHERE pgflow.runs.run_id = maybe_complete_run.run_id + AND pgflow.runs.remaining_steps = 0 + AND pgflow.runs.status != 'completed' + RETURNING * INTO v_completed_run; + + -- ========================================== + -- BROADCAST COMPLETION EVENT + -- ========================================== + IF v_completed_run.run_id IS NOT NULL THEN + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:completed', + 'run_id', v_completed_run.run_id, + 'flow_slug', v_completed_run.flow_slug, + 'status', 'completed', + 'output', v_completed_run.output, + 'completed_at', v_completed_run.completed_at + ), + 'run:completed', + concat('pgflow:run:', v_completed_run.run_id), + false + ); + END IF; +end; +$$; +-- Modify "start_ready_steps" function +CREATE OR REPLACE FUNCTION "pgflow"."start_ready_steps" ("run_id" uuid) RETURNS void LANGUAGE plpgsql SET "search_path" = '' AS $$ +begin +-- ========================================== +-- GUARD: No mutations on failed runs +-- ========================================== +IF EXISTS (SELECT 1 FROM pgflow.runs WHERE pgflow.runs.run_id = start_ready_steps.run_id AND pgflow.runs.status = 'failed') THEN + RETURN; +END IF; + +-- ========================================== +-- HANDLE EMPTY ARRAY MAPS (initial_tasks = 0) +-- ========================================== +-- These complete immediately without spawning tasks +WITH empty_map_steps AS ( + SELECT step_state.* + FROM pgflow.step_states AS step_state + JOIN pgflow.steps AS step + ON step.flow_slug = step_state.flow_slug + AND step.step_slug = step_state.step_slug + WHERE step_state.run_id = start_ready_steps.run_id + AND step_state.status = 'created' + AND step_state.remaining_deps = 0 + AND step.step_type = 'map' + AND step_state.initial_tasks = 0 + ORDER BY step_state.step_slug + FOR UPDATE OF step_state +), +-- ---------- Complete empty map steps ---------- +completed_empty_steps AS ( + UPDATE pgflow.step_states + SET status = 'completed', + started_at = now(), + completed_at = now(), + remaining_tasks = 0 + FROM empty_map_steps + WHERE pgflow.step_states.run_id = start_ready_steps.run_id + AND pgflow.step_states.step_slug = empty_map_steps.step_slug + RETURNING pgflow.step_states.* +), +-- ---------- Broadcast completion events ---------- +broadcast_empty_completed AS ( + SELECT + realtime.send( + jsonb_build_object( + 'event_type', 'step:completed', + 'run_id', completed_step.run_id, + 'step_slug', completed_step.step_slug, + 'status', 'completed', + 'started_at', completed_step.started_at, + 'completed_at', completed_step.completed_at, + 'remaining_tasks', 0, + 'remaining_deps', 0, + 'output', '[]'::jsonb + ), + concat('step:', completed_step.step_slug, ':completed'), + concat('pgflow:run:', completed_step.run_id), + false + ) + FROM completed_empty_steps AS completed_step +), + +-- ========================================== +-- HANDLE NORMAL STEPS (initial_tasks > 0) +-- ========================================== +-- ---------- Find ready steps ---------- +-- Steps with no remaining deps and known task count +ready_steps AS ( + SELECT * + FROM pgflow.step_states AS step_state + WHERE step_state.run_id = start_ready_steps.run_id + AND step_state.status = 'created' + AND step_state.remaining_deps = 0 + AND step_state.initial_tasks IS NOT NULL -- NEW: Cannot start with unknown count + AND step_state.initial_tasks > 0 -- Don't start taskless steps + -- Exclude empty map steps already handled + AND NOT EXISTS ( + SELECT 1 FROM empty_map_steps + WHERE empty_map_steps.run_id = step_state.run_id + AND empty_map_steps.step_slug = step_state.step_slug + ) + ORDER BY step_state.step_slug + FOR UPDATE +), +-- ---------- Mark steps as started ---------- +started_step_states AS ( + UPDATE pgflow.step_states + SET status = 'started', + started_at = now(), + remaining_tasks = ready_steps.initial_tasks -- Copy initial_tasks to remaining_tasks when starting + FROM ready_steps + WHERE pgflow.step_states.run_id = start_ready_steps.run_id + AND pgflow.step_states.step_slug = ready_steps.step_slug + RETURNING pgflow.step_states.* +), + +-- ========================================== +-- TASK GENERATION AND QUEUE MESSAGES +-- ========================================== +-- ---------- Generate tasks and batch messages ---------- +-- Single steps: 1 task (index 0) +-- Map steps: N tasks (indices 0..N-1) +message_batches AS ( + SELECT + started_step.flow_slug, + started_step.run_id, + started_step.step_slug, + COALESCE(step.opt_start_delay, 0) as delay, + array_agg( + jsonb_build_object( + 'flow_slug', started_step.flow_slug, + 'run_id', started_step.run_id, + 'step_slug', started_step.step_slug, + 'task_index', task_idx.task_index + ) ORDER BY task_idx.task_index + ) AS messages, + array_agg(task_idx.task_index ORDER BY task_idx.task_index) AS task_indices + FROM started_step_states AS started_step + JOIN pgflow.steps AS step + ON step.flow_slug = started_step.flow_slug + AND step.step_slug = started_step.step_slug + -- Generate task indices from 0 to initial_tasks-1 + CROSS JOIN LATERAL generate_series(0, started_step.initial_tasks - 1) AS task_idx(task_index) + GROUP BY started_step.flow_slug, started_step.run_id, started_step.step_slug, step.opt_start_delay +), +-- ---------- Send messages to queue ---------- +-- Uses batch sending for performance with large arrays +sent_messages AS ( + SELECT + mb.flow_slug, + mb.run_id, + mb.step_slug, + task_indices.task_index, + msg_ids.msg_id + FROM message_batches mb + CROSS JOIN LATERAL unnest(mb.task_indices) WITH ORDINALITY AS task_indices(task_index, idx_ord) + CROSS JOIN LATERAL pgmq.send_batch(mb.flow_slug, mb.messages, mb.delay) WITH ORDINALITY AS msg_ids(msg_id, msg_ord) + WHERE task_indices.idx_ord = msg_ids.msg_ord +), + +-- ---------- Broadcast step:started events ---------- +broadcast_events AS ( + SELECT + realtime.send( + jsonb_build_object( + 'event_type', 'step:started', + 'run_id', started_step.run_id, + 'step_slug', started_step.step_slug, + 'status', 'started', + 'started_at', started_step.started_at, + 'remaining_tasks', started_step.remaining_tasks, + 'remaining_deps', started_step.remaining_deps + ), + concat('step:', started_step.step_slug, ':started'), + concat('pgflow:run:', started_step.run_id), + false + ) + FROM started_step_states AS started_step +) + +-- ========================================== +-- RECORD TASKS IN DATABASE +-- ========================================== +INSERT INTO pgflow.step_tasks (flow_slug, run_id, step_slug, task_index, message_id) +SELECT + sent_messages.flow_slug, + sent_messages.run_id, + sent_messages.step_slug, + sent_messages.task_index, + sent_messages.msg_id +FROM sent_messages; + +end; +$$; +-- Create "cascade_complete_taskless_steps" function +CREATE FUNCTION "pgflow"."cascade_complete_taskless_steps" ("run_id" uuid) RETURNS integer LANGUAGE plpgsql AS $$ +DECLARE + v_total_completed int := 0; + v_iteration_completed int; + v_iterations int := 0; + v_max_iterations int := 50; +BEGIN + -- ========================================== + -- ITERATIVE CASCADE COMPLETION + -- ========================================== + -- Completes taskless steps in waves until none remain + LOOP + -- ---------- Safety check ---------- + v_iterations := v_iterations + 1; + IF v_iterations > v_max_iterations THEN + RAISE EXCEPTION 'Cascade loop exceeded safety limit of % iterations', v_max_iterations; + END IF; + + -- ========================================== + -- COMPLETE READY TASKLESS STEPS + -- ========================================== + WITH completed AS ( + -- ---------- Complete taskless steps ---------- + -- Steps with initial_tasks=0 and no remaining deps + UPDATE pgflow.step_states ss + SET status = 'completed', + started_at = now(), + completed_at = now(), + remaining_tasks = 0 + FROM pgflow.steps s + WHERE ss.run_id = cascade_complete_taskless_steps.run_id + AND ss.flow_slug = s.flow_slug + AND ss.step_slug = s.step_slug + AND ss.status = 'created' + AND ss.remaining_deps = 0 + AND ss.initial_tasks = 0 + -- Process in topological order to ensure proper cascade + RETURNING ss.* + ), + -- ---------- Update dependent steps ---------- + -- Propagate completion and empty arrays to dependents + dep_updates AS ( + UPDATE pgflow.step_states ss + SET remaining_deps = ss.remaining_deps - dep_count.count, + -- If the dependent is a map step and its dependency completed with 0 tasks, + -- set its initial_tasks to 0 as well + initial_tasks = CASE + WHEN s.step_type = 'map' AND dep_count.has_zero_tasks + THEN 0 -- Empty array propagation + ELSE ss.initial_tasks -- Keep existing value (including NULL) + END + FROM ( + -- Aggregate dependency updates per dependent step + SELECT + d.flow_slug, + d.step_slug as dependent_slug, + COUNT(*) as count, + BOOL_OR(c.initial_tasks = 0) as has_zero_tasks + FROM completed c + JOIN pgflow.deps d ON d.flow_slug = c.flow_slug + AND d.dep_slug = c.step_slug + GROUP BY d.flow_slug, d.step_slug + ) dep_count, + pgflow.steps s + WHERE ss.run_id = cascade_complete_taskless_steps.run_id + AND ss.flow_slug = dep_count.flow_slug + AND ss.step_slug = dep_count.dependent_slug + AND s.flow_slug = ss.flow_slug + AND s.step_slug = ss.step_slug + ), + -- ---------- Update run counters ---------- + -- Only decrement remaining_steps; let maybe_complete_run handle finalization + run_updates AS ( + UPDATE pgflow.runs r + SET remaining_steps = r.remaining_steps - c.completed_count + FROM (SELECT COUNT(*) AS completed_count FROM completed) c + WHERE r.run_id = cascade_complete_taskless_steps.run_id + AND c.completed_count > 0 + ) + -- ---------- Check iteration results ---------- + SELECT COUNT(*) INTO v_iteration_completed FROM completed; + + EXIT WHEN v_iteration_completed = 0; -- No more steps to complete + v_total_completed := v_total_completed + v_iteration_completed; + END LOOP; + + RETURN v_total_completed; +END; +$$; +-- Modify "complete_task" function +CREATE OR REPLACE FUNCTION "pgflow"."complete_task" ("run_id" uuid, "step_slug" text, "task_index" integer, "output" jsonb) RETURNS SETOF "pgflow"."step_tasks" LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_step_state pgflow.step_states%ROWTYPE; + v_dependent_map_slug text; + v_run_record pgflow.runs%ROWTYPE; + v_step_record pgflow.step_states%ROWTYPE; +begin + +-- ========================================== +-- GUARD: No mutations on failed runs +-- ========================================== +IF EXISTS (SELECT 1 FROM pgflow.runs WHERE pgflow.runs.run_id = complete_task.run_id AND pgflow.runs.status = 'failed') THEN + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index; + RETURN; +END IF; + +-- ========================================== +-- LOCK ACQUISITION AND TYPE VALIDATION +-- ========================================== +-- Acquire locks first to prevent race conditions +SELECT * INTO v_run_record FROM pgflow.runs +WHERE pgflow.runs.run_id = complete_task.run_id +FOR UPDATE; + +SELECT * INTO v_step_record FROM pgflow.step_states +WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug +FOR UPDATE; + +-- Check for type violations AFTER acquiring locks +SELECT child_step.step_slug INTO v_dependent_map_slug +FROM pgflow.deps dependency +JOIN pgflow.steps child_step ON child_step.flow_slug = dependency.flow_slug + AND child_step.step_slug = dependency.step_slug +JOIN pgflow.steps parent_step ON parent_step.flow_slug = dependency.flow_slug + AND parent_step.step_slug = dependency.dep_slug +JOIN pgflow.step_states child_state ON child_state.flow_slug = child_step.flow_slug + AND child_state.step_slug = child_step.step_slug +WHERE dependency.dep_slug = complete_task.step_slug -- parent is the completing step + AND dependency.flow_slug = v_run_record.flow_slug + AND parent_step.step_type = 'single' -- Only validate single steps + AND child_step.step_type = 'map' + AND child_state.run_id = complete_task.run_id + AND child_state.initial_tasks IS NULL + AND (complete_task.output IS NULL OR jsonb_typeof(complete_task.output) != 'array') +LIMIT 1; + +-- Handle type violation if detected +IF v_dependent_map_slug IS NOT NULL THEN + -- Mark run as failed immediately + UPDATE pgflow.runs + SET status = 'failed', + failed_at = now() + WHERE pgflow.runs.run_id = complete_task.run_id; + + -- Archive all active messages (both queued and started) to prevent orphaned messages + PERFORM pgmq.archive( + v_run_record.flow_slug, + array_agg(st.message_id) + ) + FROM pgflow.step_tasks st + WHERE st.run_id = complete_task.run_id + AND st.status IN ('queued', 'started') + AND st.message_id IS NOT NULL + HAVING count(*) > 0; -- Only call archive if there are messages to archive + + -- Mark current task as failed and store the output + UPDATE pgflow.step_tasks + SET status = 'failed', + failed_at = now(), + output = complete_task.output, -- Store the output that caused the violation + error_message = '[TYPE_VIOLATION] Produced ' || + CASE WHEN complete_task.output IS NULL THEN 'null' + ELSE jsonb_typeof(complete_task.output) END || + ' instead of array' + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index; + + -- Mark step state as failed + UPDATE pgflow.step_states + SET status = 'failed', + failed_at = now(), + error_message = '[TYPE_VIOLATION] Map step ' || v_dependent_map_slug || + ' expects array input but dependency ' || complete_task.step_slug || + ' produced ' || CASE WHEN complete_task.output IS NULL THEN 'null' + ELSE jsonb_typeof(complete_task.output) END + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug; + + -- Archive the current task's message (it was started, now failed) + PERFORM pgmq.archive( + v_run_record.flow_slug, + st.message_id -- Single message, use scalar form + ) + FROM pgflow.step_tasks st + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.task_index = complete_task.task_index + AND st.message_id IS NOT NULL; + + -- Return empty result + RETURN QUERY SELECT * FROM pgflow.step_tasks WHERE false; + RETURN; +END IF; + +-- ========================================== +-- MAIN CTE CHAIN: Update task and propagate changes +-- ========================================== +WITH +-- ---------- Task completion ---------- +-- Update the task record with completion status and output +task AS ( + UPDATE pgflow.step_tasks + SET + status = 'completed', + completed_at = now(), + output = complete_task.output + WHERE pgflow.step_tasks.run_id = complete_task.run_id + AND pgflow.step_tasks.step_slug = complete_task.step_slug + AND pgflow.step_tasks.task_index = complete_task.task_index + AND pgflow.step_tasks.status = 'started' + RETURNING * +), +-- ---------- Step state update ---------- +-- Decrement remaining_tasks and potentially mark step as completed +step_state AS ( + UPDATE pgflow.step_states + SET + status = CASE + WHEN pgflow.step_states.remaining_tasks = 1 THEN 'completed' -- Will be 0 after decrement + ELSE 'started' + END, + completed_at = CASE + WHEN pgflow.step_states.remaining_tasks = 1 THEN now() -- Will be 0 after decrement + ELSE NULL + END, + remaining_tasks = pgflow.step_states.remaining_tasks - 1 + FROM task + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug = complete_task.step_slug + RETURNING pgflow.step_states.* +), +-- ---------- Dependency resolution ---------- +-- Find all child steps that depend on the completed parent step (only if parent completed) +child_steps AS ( + SELECT deps.step_slug AS child_step_slug + FROM pgflow.deps deps + JOIN step_state parent_state ON parent_state.status = 'completed' AND deps.flow_slug = parent_state.flow_slug + WHERE deps.dep_slug = complete_task.step_slug -- dep_slug is the parent, step_slug is the child + ORDER BY deps.step_slug -- Ensure consistent ordering +), +-- ---------- Lock child steps ---------- +-- Acquire locks on all child steps before updating them +child_steps_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = complete_task.run_id + AND pgflow.step_states.step_slug IN (SELECT child_step_slug FROM child_steps) + FOR UPDATE +), +-- ---------- Update child steps ---------- +-- Decrement remaining_deps and resolve NULL initial_tasks for map steps +child_steps_update AS ( + UPDATE pgflow.step_states child_state + SET remaining_deps = child_state.remaining_deps - 1, + -- Resolve NULL initial_tasks for child map steps + -- This is where child maps learn their array size from the parent + -- This CTE only runs when the parent step is complete (see child_steps JOIN) + initial_tasks = CASE + WHEN child_step.step_type = 'map' AND child_state.initial_tasks IS NULL THEN + CASE + WHEN parent_step.step_type = 'map' THEN + -- Map->map: Count all completed tasks from parent map + -- We add 1 because the current task is being completed in this transaction + -- but isn't yet visible as 'completed' in the step_tasks table + -- TODO: Refactor to use future column step_states.total_tasks + -- Would eliminate the COUNT query and just use parent_state.total_tasks + (SELECT COUNT(*)::int + 1 + FROM pgflow.step_tasks parent_tasks + WHERE parent_tasks.run_id = complete_task.run_id + AND parent_tasks.step_slug = complete_task.step_slug + AND parent_tasks.status = 'completed' + AND parent_tasks.task_index != complete_task.task_index) + ELSE + -- Single->map: Use output array length (single steps complete immediately) + CASE + WHEN complete_task.output IS NOT NULL + AND jsonb_typeof(complete_task.output) = 'array' THEN + jsonb_array_length(complete_task.output) + ELSE NULL -- Keep NULL if not an array + END + END + ELSE child_state.initial_tasks -- Keep existing value (including NULL) + END + FROM child_steps children + JOIN pgflow.steps child_step ON child_step.flow_slug = (SELECT r.flow_slug FROM pgflow.runs r WHERE r.run_id = complete_task.run_id) + AND child_step.step_slug = children.child_step_slug + JOIN pgflow.steps parent_step ON parent_step.flow_slug = (SELECT r.flow_slug FROM pgflow.runs r WHERE r.run_id = complete_task.run_id) + AND parent_step.step_slug = complete_task.step_slug + WHERE child_state.run_id = complete_task.run_id + AND child_state.step_slug = children.child_step_slug +) +-- ---------- Update run remaining_steps ---------- +-- Decrement the run's remaining_steps counter if step completed +UPDATE pgflow.runs +SET remaining_steps = pgflow.runs.remaining_steps - 1 +FROM step_state +WHERE pgflow.runs.run_id = complete_task.run_id + AND step_state.status = 'completed'; + +-- ========================================== +-- POST-COMPLETION ACTIONS +-- ========================================== + +-- ---------- Get updated state for broadcasting ---------- +SELECT * INTO v_step_state FROM pgflow.step_states +WHERE pgflow.step_states.run_id = complete_task.run_id AND pgflow.step_states.step_slug = complete_task.step_slug; + +-- ---------- Handle step completion ---------- +IF v_step_state.status = 'completed' THEN + -- Cascade complete any taskless steps that are now ready + PERFORM pgflow.cascade_complete_taskless_steps(complete_task.run_id); + + -- Broadcast step:completed event + -- For map steps, aggregate all task outputs; for single steps, use the task output + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:completed', + 'run_id', complete_task.run_id, + 'step_slug', complete_task.step_slug, + 'status', 'completed', + 'output', CASE + WHEN (SELECT s.step_type FROM pgflow.steps s + WHERE s.flow_slug = v_step_state.flow_slug + AND s.step_slug = complete_task.step_slug) = 'map' THEN + -- Aggregate all task outputs for map steps + (SELECT COALESCE(jsonb_agg(st.output ORDER BY st.task_index), '[]'::jsonb) + FROM pgflow.step_tasks st + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.status = 'completed') + ELSE + -- Single step: use the individual task output + complete_task.output + END, + 'completed_at', v_step_state.completed_at + ), + concat('step:', complete_task.step_slug, ':completed'), + concat('pgflow:run:', complete_task.run_id), + false + ); +END IF; + +-- ---------- Archive completed task message ---------- +-- Move message from active queue to archive table +PERFORM ( + WITH completed_tasks AS ( + SELECT r.flow_slug, st.message_id + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = complete_task.run_id + AND st.step_slug = complete_task.step_slug + AND st.task_index = complete_task.task_index + AND st.status = 'completed' + ) + SELECT pgmq.archive(ct.flow_slug, ct.message_id) + FROM completed_tasks ct + WHERE EXISTS (SELECT 1 FROM completed_tasks) +); + +-- ---------- Trigger next steps ---------- +-- Start any steps that are now ready (deps satisfied) +PERFORM pgflow.start_ready_steps(complete_task.run_id); + +-- Check if the entire run is complete +PERFORM pgflow.maybe_complete_run(complete_task.run_id); + +-- ---------- Return completed task ---------- +RETURN QUERY SELECT * +FROM pgflow.step_tasks AS step_task +WHERE step_task.run_id = complete_task.run_id + AND step_task.step_slug = complete_task.step_slug + AND step_task.task_index = complete_task.task_index; + +end; +$$; +-- Modify "fail_task" function +CREATE OR REPLACE FUNCTION "pgflow"."fail_task" ("run_id" uuid, "step_slug" text, "task_index" integer, "error_message" text) RETURNS SETOF "pgflow"."step_tasks" LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + v_run_failed boolean; + v_step_failed boolean; +begin + +-- If run is already failed, no retries allowed +IF EXISTS (SELECT 1 FROM pgflow.runs WHERE pgflow.runs.run_id = fail_task.run_id AND pgflow.runs.status = 'failed') THEN + UPDATE pgflow.step_tasks + SET status = 'failed', + failed_at = now(), + error_message = fail_task.error_message + WHERE pgflow.step_tasks.run_id = fail_task.run_id + AND pgflow.step_tasks.step_slug = fail_task.step_slug + AND pgflow.step_tasks.task_index = fail_task.task_index + AND pgflow.step_tasks.status = 'started'; + + -- Archive the task's message + PERFORM pgmq.archive(r.flow_slug, ARRAY_AGG(st.message_id)) + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.message_id IS NOT NULL + GROUP BY r.flow_slug + HAVING COUNT(st.message_id) > 0; + + RETURN QUERY SELECT * FROM pgflow.step_tasks + WHERE pgflow.step_tasks.run_id = fail_task.run_id + AND pgflow.step_tasks.step_slug = fail_task.step_slug + AND pgflow.step_tasks.task_index = fail_task.task_index; + RETURN; +END IF; + +WITH run_lock AS ( + SELECT * FROM pgflow.runs + WHERE pgflow.runs.run_id = fail_task.run_id + FOR UPDATE +), +step_lock AS ( + SELECT * FROM pgflow.step_states + WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug + FOR UPDATE +), +flow_info AS ( + SELECT r.flow_slug + FROM pgflow.runs r + WHERE r.run_id = fail_task.run_id +), +config AS ( + SELECT + COALESCE(s.opt_max_attempts, f.opt_max_attempts) AS opt_max_attempts, + COALESCE(s.opt_base_delay, f.opt_base_delay) AS opt_base_delay + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN flow_info fi ON fi.flow_slug = s.flow_slug + WHERE s.flow_slug = fi.flow_slug AND s.step_slug = fail_task.step_slug +), +fail_or_retry_task as ( + UPDATE pgflow.step_tasks as task + SET + status = CASE + WHEN task.attempts_count < (SELECT opt_max_attempts FROM config) THEN 'queued' + ELSE 'failed' + END, + failed_at = CASE + WHEN task.attempts_count >= (SELECT opt_max_attempts FROM config) THEN now() + ELSE NULL + END, + started_at = CASE + WHEN task.attempts_count < (SELECT opt_max_attempts FROM config) THEN NULL + ELSE task.started_at + END, + error_message = fail_task.error_message + WHERE task.run_id = fail_task.run_id + AND task.step_slug = fail_task.step_slug + AND task.task_index = fail_task.task_index + AND task.status = 'started' + RETURNING * +), +maybe_fail_step AS ( + UPDATE pgflow.step_states + SET + status = CASE + WHEN (select fail_or_retry_task.status from fail_or_retry_task) = 'failed' THEN 'failed' + ELSE pgflow.step_states.status + END, + failed_at = CASE + WHEN (select fail_or_retry_task.status from fail_or_retry_task) = 'failed' THEN now() + ELSE NULL + END, + error_message = CASE + WHEN (select fail_or_retry_task.status from fail_or_retry_task) = 'failed' THEN fail_task.error_message + ELSE NULL + END + FROM fail_or_retry_task + WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug + RETURNING pgflow.step_states.* +) +-- Update run status +UPDATE pgflow.runs +SET status = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN 'failed' + ELSE status + END, + failed_at = CASE + WHEN (select status from maybe_fail_step) = 'failed' THEN now() + ELSE NULL + END +WHERE pgflow.runs.run_id = fail_task.run_id +RETURNING (status = 'failed') INTO v_run_failed; + +-- Check if step failed by querying the step_states table +SELECT (status = 'failed') INTO v_step_failed +FROM pgflow.step_states +WHERE pgflow.step_states.run_id = fail_task.run_id + AND pgflow.step_states.step_slug = fail_task.step_slug; + +-- Send broadcast event for step failure if the step was failed +IF v_step_failed THEN + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'step:failed', + 'run_id', fail_task.run_id, + 'step_slug', fail_task.step_slug, + 'status', 'failed', + 'error_message', fail_task.error_message, + 'failed_at', now() + ), + concat('step:', fail_task.step_slug, ':failed'), + concat('pgflow:run:', fail_task.run_id), + false + ); +END IF; + +-- Send broadcast event for run failure if the run was failed +IF v_run_failed THEN + DECLARE + v_flow_slug text; + BEGIN + SELECT flow_slug INTO v_flow_slug FROM pgflow.runs WHERE pgflow.runs.run_id = fail_task.run_id; + + PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:failed', + 'run_id', fail_task.run_id, + 'flow_slug', v_flow_slug, + 'status', 'failed', + 'error_message', fail_task.error_message, + 'failed_at', now() + ), + 'run:failed', + concat('pgflow:run:', fail_task.run_id), + false + ); + END; +END IF; + +-- Archive all active messages (both queued and started) when run fails +IF v_run_failed THEN + PERFORM pgmq.archive(r.flow_slug, ARRAY_AGG(st.message_id)) + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = fail_task.run_id + AND st.status IN ('queued', 'started') + AND st.message_id IS NOT NULL + GROUP BY r.flow_slug + HAVING COUNT(st.message_id) > 0; +END IF; + +-- For queued tasks: delay the message for retry with exponential backoff +PERFORM ( + WITH retry_config AS ( + SELECT + COALESCE(s.opt_base_delay, f.opt_base_delay) AS base_delay + FROM pgflow.steps s + JOIN pgflow.flows f ON f.flow_slug = s.flow_slug + JOIN pgflow.runs r ON r.flow_slug = f.flow_slug + WHERE r.run_id = fail_task.run_id + AND s.step_slug = fail_task.step_slug + ), + queued_tasks AS ( + SELECT + r.flow_slug, + st.message_id, + pgflow.calculate_retry_delay((SELECT base_delay FROM retry_config), st.attempts_count) AS calculated_delay + FROM pgflow.step_tasks st + JOIN pgflow.runs r ON st.run_id = r.run_id + WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'queued' + ) + SELECT pgmq.set_vt(qt.flow_slug, qt.message_id, qt.calculated_delay) + FROM queued_tasks qt + WHERE EXISTS (SELECT 1 FROM queued_tasks) +); + +-- For failed tasks: archive the message +PERFORM pgmq.archive(r.flow_slug, ARRAY_AGG(st.message_id)) +FROM pgflow.step_tasks st +JOIN pgflow.runs r ON st.run_id = r.run_id +WHERE st.run_id = fail_task.run_id + AND st.step_slug = fail_task.step_slug + AND st.task_index = fail_task.task_index + AND st.status = 'failed' + AND st.message_id IS NOT NULL +GROUP BY r.flow_slug +HAVING COUNT(st.message_id) > 0; + +return query select * +from pgflow.step_tasks st +where st.run_id = fail_task.run_id + and st.step_slug = fail_task.step_slug + and st.task_index = fail_task.task_index; + +end; +$$; +-- Modify "start_flow" function +CREATE OR REPLACE FUNCTION "pgflow"."start_flow" ("flow_slug" text, "input" jsonb, "run_id" uuid DEFAULT NULL::uuid) RETURNS SETOF "pgflow"."runs" LANGUAGE plpgsql SET "search_path" = '' AS $$ +declare + v_created_run pgflow.runs%ROWTYPE; + v_root_map_count int; +begin + +-- ========================================== +-- VALIDATION: Root map array input +-- ========================================== +WITH root_maps AS ( + SELECT step_slug + FROM pgflow.steps + WHERE steps.flow_slug = start_flow.flow_slug + AND steps.step_type = 'map' + AND steps.deps_count = 0 +) +SELECT COUNT(*) INTO v_root_map_count FROM root_maps; + +-- If we have root map steps, validate that input is an array +IF v_root_map_count > 0 THEN + -- First check for NULL (should be caught by NOT NULL constraint, but be defensive) + IF start_flow.input IS NULL THEN + RAISE EXCEPTION 'Flow % has root map steps but input is NULL', start_flow.flow_slug; + END IF; + + -- Then check if it's not an array + IF jsonb_typeof(start_flow.input) != 'array' THEN + RAISE EXCEPTION 'Flow % has root map steps but input is not an array (got %)', + start_flow.flow_slug, jsonb_typeof(start_flow.input); + END IF; +END IF; + +-- ========================================== +-- MAIN CTE CHAIN: Create run and step states +-- ========================================== +WITH + -- ---------- Gather flow metadata ---------- + flow_steps AS ( + SELECT steps.flow_slug, steps.step_slug, steps.step_type, steps.deps_count + FROM pgflow.steps + WHERE steps.flow_slug = start_flow.flow_slug + ), + -- ---------- Create run record ---------- + created_run AS ( + INSERT INTO pgflow.runs (run_id, flow_slug, input, remaining_steps) + VALUES ( + COALESCE(start_flow.run_id, gen_random_uuid()), + start_flow.flow_slug, + start_flow.input, + (SELECT count(*) FROM flow_steps) + ) + RETURNING * + ), + -- ---------- Create step states ---------- + -- Sets initial_tasks: known for root maps, NULL for dependent maps + created_step_states AS ( + INSERT INTO pgflow.step_states (flow_slug, run_id, step_slug, remaining_deps, initial_tasks) + SELECT + fs.flow_slug, + (SELECT created_run.run_id FROM created_run), + fs.step_slug, + fs.deps_count, + -- Updated logic for initial_tasks: + CASE + WHEN fs.step_type = 'map' AND fs.deps_count = 0 THEN + -- Root map: get array length from input + CASE + WHEN jsonb_typeof(start_flow.input) = 'array' THEN + jsonb_array_length(start_flow.input) + ELSE + 1 + END + WHEN fs.step_type = 'map' AND fs.deps_count > 0 THEN + -- Dependent map: unknown until dependencies complete + NULL + ELSE + -- Single steps: always 1 task + 1 + END + FROM flow_steps fs + ) +SELECT * FROM created_run INTO v_created_run; + +-- ========================================== +-- POST-CREATION ACTIONS +-- ========================================== + +-- ---------- Broadcast run:started event ---------- +PERFORM realtime.send( + jsonb_build_object( + 'event_type', 'run:started', + 'run_id', v_created_run.run_id, + 'flow_slug', v_created_run.flow_slug, + 'input', v_created_run.input, + 'status', 'started', + 'remaining_steps', v_created_run.remaining_steps, + 'started_at', v_created_run.started_at + ), + 'run:started', + concat('pgflow:run:', v_created_run.run_id), + false +); + +-- ---------- Complete taskless steps ---------- +-- Handle empty array maps that should auto-complete +PERFORM pgflow.cascade_complete_taskless_steps(v_created_run.run_id); + +-- ---------- Start initial steps ---------- +-- Start root steps (those with no dependencies) +PERFORM pgflow.start_ready_steps(v_created_run.run_id); + +-- ---------- Check for run completion ---------- +-- If cascade completed all steps (zero-task flows), finalize the run +PERFORM pgflow.maybe_complete_run(v_created_run.run_id); + +RETURN QUERY SELECT * FROM pgflow.runs where pgflow.runs.run_id = v_created_run.run_id; + +end; +$$; +-- Modify "start_tasks" function +CREATE OR REPLACE FUNCTION "pgflow"."start_tasks" ("flow_slug" text, "msg_ids" bigint[], "worker_id" uuid) RETURNS SETOF "pgflow"."step_task_record" LANGUAGE sql SET "search_path" = '' AS $$ +with tasks as ( + select + task.flow_slug, + task.run_id, + task.step_slug, + task.task_index, + task.message_id + from pgflow.step_tasks as task + join pgflow.runs r on r.run_id = task.run_id + where task.flow_slug = start_tasks.flow_slug + and task.message_id = any(msg_ids) + and task.status = 'queued' + -- MVP: Don't start tasks on failed runs + and r.status != 'failed' + ), + start_tasks_update as ( + update pgflow.step_tasks + set + attempts_count = attempts_count + 1, + status = 'started', + started_at = now(), + last_worker_id = worker_id + from tasks + where step_tasks.message_id = tasks.message_id + and step_tasks.flow_slug = tasks.flow_slug + and step_tasks.status = 'queued' + ), + runs as ( + select + r.run_id, + r.input + from pgflow.runs r + where r.run_id in (select run_id from tasks) + ), + deps as ( + select + st.run_id, + st.step_slug, + dep.dep_slug, + -- Aggregate map outputs or use single output + CASE + WHEN dep_step.step_type = 'map' THEN + -- Aggregate all task outputs ordered by task_index + -- Use COALESCE to return empty array if no tasks + (SELECT COALESCE(jsonb_agg(dt.output ORDER BY dt.task_index), '[]'::jsonb) + FROM pgflow.step_tasks dt + WHERE dt.run_id = st.run_id + AND dt.step_slug = dep.dep_slug + AND dt.status = 'completed') + ELSE + -- Single step: use the single task output + dep_task.output + END as dep_output + from tasks st + join pgflow.deps dep on dep.flow_slug = st.flow_slug and dep.step_slug = st.step_slug + join pgflow.steps dep_step on dep_step.flow_slug = dep.flow_slug and dep_step.step_slug = dep.dep_slug + left join pgflow.step_tasks dep_task on + dep_task.run_id = st.run_id and + dep_task.step_slug = dep.dep_slug and + dep_task.status = 'completed' + and dep_step.step_type = 'single' -- Only join for single steps + ), + deps_outputs as ( + select + d.run_id, + d.step_slug, + jsonb_object_agg(d.dep_slug, d.dep_output) as deps_output, + count(*) as dep_count + from deps d + group by d.run_id, d.step_slug + ), + timeouts as ( + select + task.message_id, + task.flow_slug, + coalesce(step.opt_timeout, flow.opt_timeout) + 2 as vt_delay + from tasks task + join pgflow.flows flow on flow.flow_slug = task.flow_slug + join pgflow.steps step on step.flow_slug = task.flow_slug and step.step_slug = task.step_slug + ), + -- Batch update visibility timeouts for all messages + set_vt_batch as ( + select pgflow.set_vt_batch( + start_tasks.flow_slug, + array_agg(t.message_id order by t.message_id), + array_agg(t.vt_delay order by t.message_id) + ) + from timeouts t + ) + select + st.flow_slug, + st.run_id, + st.step_slug, + -- ========================================== + -- INPUT CONSTRUCTION LOGIC + -- ========================================== + -- This nested CASE statement determines how to construct the input + -- for each task based on the step type (map vs non-map). + -- + -- The fundamental difference: + -- - Map steps: Receive RAW array elements (e.g., just 42 or "hello") + -- - Non-map steps: Receive structured objects with named keys + -- (e.g., {"run": {...}, "dependency1": {...}}) + -- ========================================== + CASE + -- -------------------- MAP STEPS -------------------- + -- Map steps process arrays element-by-element. + -- Each task receives ONE element from the array at its task_index position. + WHEN step.step_type = 'map' THEN + -- Map steps get raw array elements without any wrapper object + CASE + -- ROOT MAP: Gets array from run input + -- Example: run input = [1, 2, 3] + -- task 0 gets: 1 + -- task 1 gets: 2 + -- task 2 gets: 3 + WHEN step.deps_count = 0 THEN + -- Root map (deps_count = 0): no dependencies, reads from run input. + -- Extract the element at task_index from the run's input array. + -- Note: If run input is not an array, this will return NULL + -- and the flow will fail (validated in start_flow). + jsonb_array_element(r.input, st.task_index) + + -- DEPENDENT MAP: Gets array from its single dependency + -- Example: dependency output = ["a", "b", "c"] + -- task 0 gets: "a" + -- task 1 gets: "b" + -- task 2 gets: "c" + ELSE + -- Has dependencies (should be exactly 1 for map steps). + -- Extract the element at task_index from the dependency's output array. + -- + -- Why the subquery with jsonb_each? + -- - The dependency outputs a raw array: [1, 2, 3] + -- - deps_outputs aggregates it into: {"dep_name": [1, 2, 3]} + -- - We need to unwrap and get just the array value + -- - Map steps have exactly 1 dependency (enforced by add_step) + -- - So jsonb_each will return exactly 1 row + -- - We extract the 'value' which is the raw array [1, 2, 3] + -- - Then get the element at task_index from that array + (SELECT jsonb_array_element(value, st.task_index) + FROM jsonb_each(dep_out.deps_output) + LIMIT 1) + END + + -- -------------------- NON-MAP STEPS -------------------- + -- Regular (non-map) steps receive ALL inputs as a structured object. + -- This includes the original run input plus all dependency outputs. + ELSE + -- Non-map steps get structured input with named keys + -- Example output: { + -- "run": {"original": "input"}, + -- "step1": {"output": "from_step1"}, + -- "step2": {"output": "from_step2"} + -- } + -- + -- Build object with 'run' key containing original input + jsonb_build_object('run', r.input) || + -- Merge with deps_output which already has dependency outputs + -- deps_output format: {"dep1": output1, "dep2": output2, ...} + -- If no dependencies, defaults to empty object + coalesce(dep_out.deps_output, '{}'::jsonb) + END as input, + st.message_id as msg_id, + st.task_index as task_index + from tasks st + join runs r on st.run_id = r.run_id + join pgflow.steps step on + step.flow_slug = st.flow_slug and + step.step_slug = st.step_slug + left join deps_outputs dep_out on + dep_out.run_id = st.run_id and + dep_out.step_slug = st.step_slug +$$; +-- Create "add_step" function +CREATE FUNCTION "pgflow"."add_step" ("flow_slug" text, "step_slug" text, "deps_slugs" text[] DEFAULT '{}', "max_attempts" integer DEFAULT NULL::integer, "base_delay" integer DEFAULT NULL::integer, "timeout" integer DEFAULT NULL::integer, "start_delay" integer DEFAULT NULL::integer, "step_type" text DEFAULT 'single') RETURNS "pgflow"."steps" LANGUAGE plpgsql SET "search_path" = '' AS $$ +DECLARE + result_step pgflow.steps; + next_idx int; +BEGIN + -- Validate map step constraints + -- Map steps can have either: + -- 0 dependencies (root map - maps over flow input array) + -- 1 dependency (dependent map - maps over dependency output array) + IF COALESCE(add_step.step_type, 'single') = 'map' AND COALESCE(array_length(add_step.deps_slugs, 1), 0) > 1 THEN + RAISE EXCEPTION 'Map step "%" can have at most one dependency, but % were provided: %', + add_step.step_slug, + COALESCE(array_length(add_step.deps_slugs, 1), 0), + array_to_string(add_step.deps_slugs, ', '); + END IF; + + -- Get next step index + SELECT COALESCE(MAX(s.step_index) + 1, 0) INTO next_idx + FROM pgflow.steps s + WHERE s.flow_slug = add_step.flow_slug; + + -- Create the step + INSERT INTO pgflow.steps ( + flow_slug, step_slug, step_type, step_index, deps_count, + opt_max_attempts, opt_base_delay, opt_timeout, opt_start_delay + ) + VALUES ( + add_step.flow_slug, + add_step.step_slug, + COALESCE(add_step.step_type, 'single'), + next_idx, + COALESCE(array_length(add_step.deps_slugs, 1), 0), + add_step.max_attempts, + add_step.base_delay, + add_step.timeout, + add_step.start_delay + ) + ON CONFLICT ON CONSTRAINT steps_pkey + DO UPDATE SET step_slug = EXCLUDED.step_slug + RETURNING * INTO result_step; + + -- Insert dependencies + INSERT INTO pgflow.deps (flow_slug, dep_slug, step_slug) + SELECT add_step.flow_slug, d.dep_slug, add_step.step_slug + FROM unnest(COALESCE(add_step.deps_slugs, '{}')) AS d(dep_slug) + WHERE add_step.deps_slugs IS NOT NULL AND array_length(add_step.deps_slugs, 1) > 0 + ON CONFLICT ON CONSTRAINT deps_pkey DO NOTHING; + + RETURN result_step; +END; +$$; +-- Drop "add_step" function +DROP FUNCTION "pgflow"."add_step" (text, text, integer, integer, integer, integer); +-- Drop "add_step" function +DROP FUNCTION "pgflow"."add_step" (text, text, text[], integer, integer, integer, integer); diff --git a/apps/demo/supabase/migrations/20251031133804_demo_anon_permissions.sql b/apps/demo/supabase/migrations/20251031133804_demo_anon_permissions.sql new file mode 100644 index 000000000..384f0cd7f --- /dev/null +++ b/apps/demo/supabase/migrations/20251031133804_demo_anon_permissions.sql @@ -0,0 +1,14 @@ +-- Grant anon role access to start flows +GRANT USAGE ON SCHEMA pgflow TO anon; +GRANT EXECUTE ON FUNCTION pgflow.start_flow TO anon; + +-- Grant anon role read access to pgflow tables for real-time updates +GRANT SELECT ON pgflow.flows TO anon; +GRANT SELECT ON pgflow.runs TO anon; +GRANT SELECT ON pgflow.steps TO anon; +GRANT SELECT ON pgflow.step_states TO anon; +GRANT SELECT ON pgflow.deps TO anon; + +-- Enable real-time for anon role +ALTER PUBLICATION supabase_realtime ADD TABLE pgflow.runs; +ALTER PUBLICATION supabase_realtime ADD TABLE pgflow.step_states; diff --git a/apps/demo/supabase/migrations/20251031142456_create_test_flow_flow.sql b/apps/demo/supabase/migrations/20251031142456_create_test_flow_flow.sql new file mode 100644 index 000000000..fd7e400a9 --- /dev/null +++ b/apps/demo/supabase/migrations/20251031142456_create_test_flow_flow.sql @@ -0,0 +1,2 @@ +SELECT pgflow.create_flow('test_flow'); +SELECT pgflow.add_step('test_flow', 'greet'); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 150d26826..926edc910 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -114,6 +114,9 @@ importers: apps/demo: dependencies: + '@pgflow/client': + specifier: workspace:* + version: link:../../pkgs/client '@supabase/supabase-js': specifier: ^2.78.0 version: 2.81.0