Pull-request checks

What a check is

A check is a matrix: every flow × every device your visitors use × a few people, run against the pull request’s preview and against the base (production, or whatever you pass as base). The people are the same on both sides, in the same order, so the difference is the build and not the luck of the draw. The verdict is per flow and per device: a flow that works on desktop and fails on a phone is reported as a regression on mobile, not averaged away.

The journeys come from your workspace: written from the repository’s code when the repository is connected to the GitHub App, or mapped by simulathic map (and edited with simulathic flows). Nothing is kept in the repository. One comment is posted, and updated in place on later pushes: the verdict, how the people fared on base and on the PR per journey and per device, what went wrong with the evidence and a fix prompt where the cause is clear, and links to the recordings.

Exit codes: 0 when nothing regressed, 2 on a regression, 1 when the check itself could not run.

The GitHub App

The simplest path. Install the App once from Settings → GitHub in the Studio, connect the repository to the project, and every pull request gets a check run named Simulathic (branch protection can require it) and one comment. No workflow file, no token in the repository. Draft pull requests wait until they are ready for review.

When you connect the repository you say how a pull request becomes a URL. Preview deployments from Vercel, Netlify, Cloudflare Pages or Render start the check when the deployment for the PR’s commit succeeds (a protected Vercel preview needs the project’s bypass secret, pasted once). A URL pattern, for previews you build yourselves, starts it on the push, after the minutes you say a deploy takes, once the URL answers. One staging site deployed from a branch starts it on the push to that branch and reports on the pull requests that push merged. For a native app, CI builds the app and runs simulathic upload; the check starts when the build for the PR’s commit arrives and compares it with the newest build uploaded from the base branch.

/simulathic rerun as a comment on the pull request runs the last check again; /simulathic all runs every journey whatever the repository’s policy. Uninstalling the App on GitHub stops the checks and changes nothing else in the workspace. The GitHub App page has the details: what runs, how journeys are written from the code, and what ends a check neutral.

GitHub Actions, without the App

The GitHub App needs none of this; the path below is for a repository that runs the check from its own workflow instead.

Mint a token for CI with the CLI and store it as the repository secret SIMULATHIC_TOKEN; with the GitHub CLI that is one line, simulathic token --name github-ci | gh secret set SIMULATHIC_TOKEN. It is the only secret the Actions path needs; nothing else goes in the repository. The token is printed once, is separate from your own login, and lasts 90 days. Then add a workflow that runs when the preview deployment is ready. With Vercel that is the deployment_status event; note that GitHub only fires it from the default branch, so the workflow file has to be merged before it runs on PRs.

name: Simulathic check
on:
  deployment_status:
jobs:
  check:
    if: github.event.deployment_status.state == 'success' && github.event.deployment.environment == 'Preview'
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    env:
      SIMULATHIC_TOKEN: ${{ secrets.SIMULATHIC_TOKEN }}
      PREVIEW_URL: ${{ github.event.deployment_status.target_url || github.event.deployment_status.environment_url }}
    steps:
      - run: curl -fsSL https://app.simulathic.com/cli/install.sh | sh
      - run: |
          ~/.simulathic/cli/simulathic ci \
            --url "$PREVIEW_URL" \
            --base "https://app.yourproduct.com" \
            --project ws_xxxxxx --users 3 --fail-on high \
            --repo "$GITHUB_REPOSITORY" --sha "$GITHUB_SHA"

--repo and --sha let the check find the pull request for the deployment and post the comment itself. On other CI systems pass --pr with the number, or use --out comment.mdand --json and post them however you like.

Options

Size and cost

A web check is flows × devices × people × 2 sessions. Six flows on two devices with three people is 72 sessions, about two dollars and a few minutes. --users 2 halves the cost, and --only sends people through the flows you name alone. The check prints its plan and estimate before it starts.

Native apps: Mac, Android and iOS

A native app gets the same check — a Mac app (Swift, Electron, Tauri or Flutter, built for Apple Silicon), an Android app (an .apk with arm64-v8a: native, React Native, Expo, Flutter) or an iOS app (a simulator build, the .app from xcodebuild -sdk iphonesimulator; an .ipa is a device build and cannot run). Nothing runs on your CI box but the build: the PR’s build is uploaded, compared with the newest build uploaded from the base branch, and both run on a Simulathic worker — a Mac, a pristine Android 14 phone per run, or an iPhone simulator whose app data is wiped between people — each person starting from a just-installed app. The journeys come from mapping the app once (simulathic map --app build/MyApp.app, or --app app-release.apk): a worker explores it, typing into its fields and pressing what that enables, and writes the journeys into your workspace, so there is nothing to keep in your repo. With the GitHub App connected, only the upload step below is needed; the check starts when the build for the PR’s commit arrives. Without it, two commands in CI, after whatever builds the app:

      - run: curl -fsSL https://app.simulathic.com/cli/install.sh | sh
      # on every push to main: the base for later checks
      - if: github.event_name == 'push'
        run: ~/.simulathic/cli/simulathic upload build/MyApp.app
      # on every pull request: this build against the newest main build
      - if: github.event_name == 'pull_request'
        run: ~/.simulathic/cli/simulathic ci --app build/MyApp.app --base-branch main --users 2 --out comment.md

      - run: ./gradlew assembleRelease
      - run: curl -fsSL https://app.simulathic.com/cli/install.sh | sh
      - if: github.event_name == 'push'
        run: ~/.simulathic/cli/simulathic upload app/build/outputs/apk/release/app-release.apk
      - if: github.event_name == 'pull_request'
        run: ~/.simulathic/cli/simulathic ci --app app/build/outputs/apk/release/app-release.apk --base-branch main --users 2 --out comment.md

      - run: xcodebuild -workspace App.xcworkspace -scheme App -configuration Release -sdk iphonesimulator -destination 'generic/platform=iOS Simulator' -derivedDataPath build CODE_SIGNING_ALLOWED=NO
      - run: curl -fsSL https://app.simulathic.com/cli/install.sh | sh
      - if: github.event_name == 'push'
        run: ~/.simulathic/cli/simulathic upload build/Build/Products/Release-iphonesimulator/App.app
      - if: github.event_name == 'pull_request'
        run: ~/.simulathic/cli/simulathic ci --app build/Build/Products/Release-iphonesimulator/App.app --base-branch main --users 2 --out comment.md

upload reads the bundle id or package, version and architecture from the app itself (no Xcode, aapt or Java needed on the box) and the commit, branch and PR from the CI it runs on (GitHub Actions, Bitrise, GitLab, CircleCI, Xcode Cloud, EAS); the same bytes twice are not sent again. It refuses what could not run on our phones — an x86-only APK, or a React Native / Expo development build with no JavaScript bundled (upload a release build) — and notes whether the app hard-requires Google Play services (Maps, Billing, Google Sign-In), which the phone tier does not have. On a phone, a crash of the app during a person’s session (Android: the logcat crash buffer and the app’s own error lines; iOS: the simulator’s crash report) is reported as a defect with the trace as the fix prompt, and so is an app that goes blank under the person, which is how React Native fails in release builds. The comment is in comment.md for your workflow to post; --journeys <file> overrides the workspace’s journeys for one run. Not on GitHub Actions? The two commands run anywhere the build exists.

A cross-platform app keeps one journey set under its bundle id, mapped from whichever platform you ran map --app on last; each platform’s map is kept. See Simulate for what runs where today.