Server-Side Integrations

This guide covers the recommended backend pattern for ZeyOS: keep OAuth tokens on the server, refresh them automatically, and isolate business sync logic behind small job functions.

When to Use This Pattern

Use a server-side integration when you are building:

• scheduled sync jobs
• background workers
• webhook handlers
• internal APIs that proxy or enrich ZeyOS data
• automation services that should not depend on a live browser session

1. Store State Safely

A sync job keeps two pieces of state on disk: OAuth tokens and its checkpoint. Both need the same guarantees, so write them through one helper. The helper tells a missing file apart from malformed data and from a permission failure, and it replaces files atomically with private permissions. A crash during a write leaves the previous version intact.

JavaScript

// state-file.mjs
import { randomBytes } from 'node:crypto';
import { open, readFile, rename, rm } from 'node:fs/promises';
import { basename, dirname, join } from 'node:path';

export class StateFileError extends Error {
  constructor(kind, path, action) {
    super(`${kind} state file ${basename(path)}: ${action}`);
    this.name = 'StateFileError';
    this.kind = kind; // 'permission' | 'malformed' | 'unreadable'
  }
}

/** Resolves to undefined only when the file does not exist. */
export async function readStateFile(path) {
  let text;
  try {
    text = await readFile(path, 'utf8');
  } catch (err) {
    if (err.code === 'ENOENT') return undefined;
    if (err.code === 'EACCES' || err.code === 'EPERM') {
      throw new StateFileError('permission', path, 'grant the job user read access.');
    }
    throw new StateFileError('unreadable', path, `check the file (${err.code}).`);
  }
  try {
    return JSON.parse(text);
  } catch {
    // JSON parser messages can quote the input, and this file holds secrets.
    throw new StateFileError('malformed', path, 'restore it from a backup or re-authenticate.');
  }
}

/** Temp file in the same directory, flushed, then renamed over the target. */
export async function writeStateFile(path, value) {
  const suffix = `${process.pid}.${randomBytes(6).toString('hex')}.tmp`;
  const tempPath = join(dirname(path), `.${basename(path)}.${suffix}`);
  try {
    const handle = await open(tempPath, 'wx', 0o600);
    try {
      await handle.writeFile(`${JSON.stringify(value, null, 2)}\n`);
      await handle.sync();
    } finally {
      await handle.close();
    }
    await rename(tempPath, path);
  } catch (err) {
    await rm(tempPath, { force: true });
    if (err.code === 'EACCES' || err.code === 'EPERM') {
      throw new StateFileError('permission', path, 'grant the job user write access to the directory.');
    }
    throw err;
  }
}

The client requires only a get() and set() pair from a token store. This one is for development. In production, back the same two methods with a secrets manager or database.

JavaScript

// token-store.mjs
import { rm } from 'node:fs/promises';
import { normalizeTokenSet } from '@zeyos/client';
import { readStateFile, StateFileError, writeStateFile } from './state-file.mjs';

export class JsonFileTokenStore {
  constructor(path) {
    this.path = path;
  }

  async get() {
    const stored = await readStateFile(this.path);
    if (stored === undefined) return null; // Not signed in yet.

    const tokenSet = normalizeTokenSet(stored);
    if (!tokenSet) {
      throw new StateFileError('malformed', this.path, 'it holds no token set; re-authenticate.');
    }
    return tokenSet;
  }

  async set(tokenSet) {
    if (!tokenSet) {
      await rm(this.path, { force: true }); // client.auth.clearTokenSet()
      return;
    }
    await writeStateFile(this.path, tokenSet);
  }
}

Only a missing file means "not signed in". A corrupt or unreadable store throws, and the error reaches the caller of the API method. Treating those cases as empty would discard a working refresh token and send the job to a confusing 401.

Keep the file outside your repository and do not log its contents.

Sharing a Token Store Between Workers

The client coordinates refreshes only inside one client instance: concurrent requests share a single refresh. Nothing coordinates separate processes. The client has no lock, no compare-and-swap and no hook around a refresh, and a token store sees only the resulting get() and set() calls. A lock therefore cannot be added inside a store.

The client does read the store at the start of every operation, so a worker picks up tokens written by another process on its next call. What remains is the race between two workers refreshing at the same moment. Choose one approach:

Approach	How it works	Use when
Single refresher (recommended)	Workers set autoRefresh: false and only read the store. One process calls client.oauth2.refreshToken() before expiry.	Any multi-process deployment
Lock around an explicit refresh	Each worker sets autoRefresh: false and takes a lock before calling client.oauth2.refreshToken(), as shown below.	There is no natural owner for a refresher
Accept duplicate refreshes	Leave autoRefresh on and let workers race.	Only after confirming your instance keeps the previous refresh token valid

The third approach depends on server behavior this client does not establish. Its code handles a response with no new refresh token by keeping the old one, and also handles a rotated one, but ZeyOS's rotation policy is not documented here. If refresh tokens rotate, the losing worker presents a used token, its refresh fails, and the failed call surfaces as an error even though the store already holds valid tokens. Test this against a sandbox instance before relying on it.

JavaScript

// refresh.mjs: run this from workers configured with autoRefresh: false.
import { withLockFile } from './lock-file.mjs';

const REFRESH_MARGIN_SECONDS = 120;

export async function refreshIfDue(client, lockPath) {
  await withLockFile(lockPath, async () => {
    // Re-read inside the lock: another worker may have finished a refresh.
    const tokenSet = await client.auth.getTokenSet();
    const now = Math.floor(Date.now() / 1000);
    if (tokenSet?.expiresAt == null || tokenSet.expiresAt > now + REFRESH_MARGIN_SECONDS) return;
    await client.oauth2.refreshToken();
  });
}

A failed set() needs attention in every approach. The server has already issued new tokens at that point, and if it rotated the refresh token, the job has lost the only copy. A failed write during a refresh before the request is also swallowed by the client, which continues with the expired token, so monitor store write failures separately.

The lock uses an exclusive create, which either succeeds or reports who holds it:

JavaScript

// lock-file.mjs
import { open, readFile, rm, stat } from 'node:fs/promises';

export class LockHeldError extends Error {
  constructor({ pid, ageSeconds, stale }) {
    super(`${stale ? 'Stale lock' : 'Another run holds the lock'}: pid ${pid ?? 'unknown'}, ${ageSeconds}s old.`);
    this.name = 'LockHeldError';
    this.stale = stale; // true: no live run owns it, so nothing will clear it.
  }
}

function isRunning(pid) {
  try {
    process.kill(pid, 0); // Signal 0 tests for the process without touching it.
    return true;
  } catch (err) {
    return err.code === 'EPERM'; // It exists under another user.
  }
}

async function describeLock(path, maxAgeMs) {
  try {
    const [text, { mtimeMs }] = await Promise.all([readFile(path, 'utf8'), stat(path)]);
    const pid = Number.parseInt(text, 10);
    const ageMs = Date.now() - mtimeMs;
    const owned = pid > 0;
    return {
      pid: owned ? pid : null,
      ageSeconds: Math.round(ageMs / 1000),
      // An empty file is a holder between create and write, until it ages out.
      stale: (owned && !isRunning(pid)) || ageMs > maxAgeMs,
    };
  } catch (err) {
    if (err.code === 'ENOENT') return { pid: null, ageSeconds: 0, stale: false }; // Just released.
    throw err;
  }
}

export async function withLockFile(path, task, { maxAgeMs = 60 * 60_000 } = {}) {
  let handle;
  try {
    handle = await open(path, 'wx', 0o600);
  } catch (err) {
    if (err.code === 'EEXIST') throw new LockHeldError(await describeLock(path, maxAgeMs));
    throw err;
  }
  try {
    await handle.writeFile(`${process.pid}\n`);
    return await task();
  } finally {
    await handle.close();
    await rm(path, { force: true });
  }
}

A process that is killed leaves its lock behind. The next run reports the lock as stale, because its process is gone or it is older than maxAgeMs, and exits non-zero so a monitor sees that the job has stopped. It does not remove the lock itself: confirm that the recorded process is gone, then delete the file. Set maxAgeMs above your longest run, because a process ID can be reused after a reboot. This lock works on one host's local filesystem. Across hosts, use a lock from your database or scheduler.

2. Build the Client

The code in Sections 2, 3 and 4 forms one module, sync.mjs.

JavaScript

import { createZeyosClient } from '@zeyos/client';
import { JsonFileTokenStore } from './token-store.mjs';

const tokenStore = new JsonFileTokenStore('/var/lib/my-sync/tokens.json');

const client = createZeyosClient({
  platform: 'https://cloud.zeyos.com/demo/',
  validate: true,
  timeoutMs: 30_000,
  auth: {
    mode: 'oauth',
    oauth: {
      clientId: process.env.ZEYOS_CLIENT_ID,
      clientSecret: process.env.ZEYOS_CLIENT_SECRET,
      tokenStore,
      autoRefresh: true,
    },
  },
});

• validate: true rejects unknown fields, invalid enum values and invalid join paths before any request is sent. Validation is off by default in JavaScript.
• timeoutMs bounds each attempt. Without it, a stalled connection can wait forever and the retry logic never runs.
• autoRefresh: true refreshes an expired access token before a request, or after a bearer 401. Your token store must persist the result.
• If the job names individual extdata keys, add a reviewed customFields contract bound to this instance.

3. Fetch One Page

ZeyOS has no server-side cursor. The API offers limit (up to 10000), offset, sort, filters and count, and a sync job builds its position from those.

Do not page a changing dataset with offset, and that includes client.paginate(), which is offset-based. When a record on an earlier page changes, it moves to the end of a lastmodified ordering, every later row shifts up by one, and the next page silently skips a record.

Instead, use keyset pagination: sort by lastmodified with ID as the tie-breaker, and ask for rows strictly after the last one you processed. A row that changes during the run moves further ahead of the position, so it is read later and never skipped.

JavaScript

import { normalizeListResult } from '@zeyos/client';

export const PAGE_SIZE = 500;
const TICKET_FIELDS = ['ID', 'ticketnum', 'name', 'status', 'priority', 'visibility', 'lastmodified'];

/** Rows strictly after (lastmodified, ID) in the sort order below. */
function ticketPageQuery(after) {
  return {
    fields: TICKET_FIELDS,
    // No visibility filter: archiving a ticket must reach the downstream system.
    filters: {
      1: ['OR',
        { lastmodified: { '>': after.lastmodified } },
        { 2: ['AND',
          { lastmodified: { '=': after.lastmodified } },
          { ID: { '>': after.ID } },
        ] },
      ],
    },
    sort: ['+lastmodified', '+ID'],
    limit: PAGE_SIZE,
  };
}

export async function fetchTicketPage(after, { signal } = {}) {
  const result = await client.api.listTickets(ticketPageQuery(after), { signal });
  return normalizeListResult(result).data;
}

The ID tie-breaker matters because lastmodified has one-second resolution. A bulk update can give more rows the same timestamp than fit on one page, and a lastmodified-only position would stall or lose rows at that boundary.

Keep each sync unit focused: one resource, one query shape, one return shape. If you use aliases in fields, keep lastmodified and ID under their real names, because the position is read from them.

4. Run a Resumable Sync

A pass reads every row after its starting point, and may span several runs. The checkpoint stores two positions and is saved only after the downstream system has accepted the rows:

• through is where the last complete pass ended. The next pass starts one overlap window behind it.
• position is how far the current pass has got, or null between passes. A run that stops early leaves it set, and the next run continues from there.

The overlap applies only when a new pass starts. If every run began behind through, a window holding more rows than one run can process would use up the budget each time, and the job would never get past those rows. Because a resumed pass continues from position, which only moves forward, every run that processes a row makes lasting progress.

JavaScript

import { readStateFile, writeStateFile } from './state-file.mjs';

const SYNC_PLATFORM = 'https://cloud.zeyos.com/demo';
const OVERLAP_SECONDS = 300;
const START = { lastmodified: 0, ID: 0 };

const isAfter = (a, b) =>
  a.lastmodified > b.lastmodified || (a.lastmodified === b.lastmodified && a.ID > b.ID);

function positionOf(row) {
  if (!Number.isSafeInteger(row?.lastmodified) || !Number.isSafeInteger(row?.ID)) {
    throw new Error(`Ticket ${row?.ID} has no usable sync position.`);
  }
  return { lastmodified: row.lastmodified, ID: row.ID };
}

async function loadCheckpoint(path) {
  const saved = await readStateFile(path);
  if (saved === undefined) return { through: START, position: null };
  if (saved?.platform !== SYNC_PLATFORM || saved?.resource !== 'tickets') {
    throw new Error('Checkpoint belongs to another instance or resource.');
  }
  return {
    through: positionOf(saved.through),
    position: saved.position === null ? null : positionOf(saved.position),
  };
}

function saveCheckpoint(path, checkpoint) {
  return writeStateFile(path, {
    platform: SYNC_PLATFORM,
    resource: 'tickets',
    ...checkpoint,
    savedAt: new Date().toISOString(), // For operators; the sync never reads it.
  });
}

export async function runTicketSync({ checkpointPath, upsert, signal, maxPages = 10_000 }) {
  const { through, position: resumed } = await loadCheckpoint(checkpointPath);

  // Only a new pass re-reads the overlap window, so a crash or late commit is covered
  // without letting that window hold the job back.
  let position = resumed ?? { lastmodified: Math.max(0, through.lastmodified - OVERLAP_SECONDS), ID: 0 };

  // Offline preflight: fail on a bad query before the first request.
  const preflight = client.schema.validate('listTickets', ticketPageQuery(position));
  if (!preflight.valid) throw new Error(`Invalid sync query: ${preflight.errors[0].code}`);

  let records = 0;
  try {
    for (let page = 0; page < maxPages; page += 1) {
      const rows = await fetchTicketPage(position, { signal });

      for (const row of rows) {
        signal?.throwIfAborted();
        const next = positionOf(row);
        // Stop rather than advance past rows if the server ignored the ordering.
        if (!isAfter(next, position)) throw new Error(`Ticket ${row.ID} arrived out of order.`);
        await upsert(row, { signal });
        position = next;
        records += 1;
      }

      if (rows.length < PAGE_SIZE) {
        // The overlap can end behind `through`, so never move it backwards.
        const reached = isAfter(position, through) ? position : through;
        await saveCheckpoint(checkpointPath, { through: reached, position: null });
        return { complete: true, records, checkpoint: reached };
      }
      await saveCheckpoint(checkpointPath, { through, position });
    }
  } catch (err) {
    // Keep the rows already accepted, so a slow page cannot repeat forever. The
    // original failure matters more than a failed save here.
    await saveCheckpoint(checkpointPath, { through, position }).catch(() => {});
    throw err;
  }
  return { complete: false, records, checkpoint: position }; // Budget reached: resume next run.
}

Run it behind a lock, so scheduled runs cannot overlap, and with a budget for the whole job:

JavaScript

import { describeFailure } from './failures.mjs';
import { LockHeldError, withLockFile } from './lock-file.mjs';

export async function main(upsert) {
  try {
    const summary = await withLockFile('/var/lib/my-sync/tickets.lock', () => runTicketSync({
      checkpointPath: '/var/lib/my-sync/tickets.checkpoint.json',
      upsert,
      signal: AbortSignal.timeout(15 * 60_000), // Budget for all pages and retries.
    }));
    console.log(JSON.stringify({ job: 'ticket-sync', ...summary }));
  } catch (err) {
    // An overlapping run is normal. A stale lock falls through and fails the job,
    // because every later run would otherwise stop here without anyone noticing.
    if (err instanceof LockHeldError && !err.stale) {
      console.log(JSON.stringify({ job: 'ticket-sync', skipped: err.message }));
      return;
    }
    console.error(JSON.stringify(describeFailure(err, { resource: 'tickets', operationId: 'listTickets' })));
    process.exitCode = 1;
  }
}

Crash and Change Behavior

Event	Result
Hard crash during a page	The checkpoint still holds the end of the previous page. The next run repeats this page, so rows may be delivered twice.
Timeout or error during a page	The accepted rows are saved first, and the next run continues after them. A row may be delivered twice if upsert committed but did not return.
Crash after a complete pass	The next run starts a new pass, which re-reads the overlap window.
Record changes during the run	Its lastmodified moves ahead of the position, so it is read later in this run or in the next one.
Record changes repeatedly while the job runs	It may be delivered several times. The downstream write keeps the newest version.
Write commits late with an older timestamp	Covered if it lands within the overlap window behind the point where the pass ended.
Budget or maxPages reached	The run stops with saved progress. A maxPages stop returns complete: false and a budget stop throws. The next run continues from position, without repeating the overlap.
More rows share a timestamp, or sit in the overlap, than one run can process	Several runs finish the pass. Each continues inside the same timestamp, because the position includes ID.
Rows change faster than the job processes them	The position still advances, but the pass may not finish until the rate drops.
A killed run leaves its lock	The next run exits non-zero with a stale-lock failure and does not sync until an operator clears it.

Make Downstream Writes Idempotent

Every row can arrive more than once, by design. upsert must therefore key on the instance and ID, and must refuse to replace newer data. It receives the job's signal as upsert(row, { signal }): pass that to your database or HTTP call, so a slow downstream write cannot outlive the job's budget.

SQL

INSERT INTO zeyos_tickets (instance, id, name, status, visibility, lastmodified)
VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (instance, id) DO UPDATE
SET name = excluded.name,
    status = excluded.status,
    visibility = excluded.visibility,
    lastmodified = excluded.lastmodified
WHERE excluded.lastmodified >= zeyos_tickets.lastmodified;

Treat visibility as data: 1 is archived and 2 is deleted. Resources without that column need a different rule, and client.schema.describe('<resource>') shows what each one has.

Reconcile Deletions

An incremental sync sees only rows that still exist and match. A hard-deleted record simply stops appearing, and whether archiving updates lastmodified is not guaranteed here. Rows with an empty lastmodified never match the filter either. Run a periodic ID pass, for example nightly, to catch all three:

JavaScript

export async function listAllTicketIds({ signal } = {}) {
  const ids = [];
  let after = 0;
  for (;;) {
    const result = await client.api.listTickets({
      fields: ['ID'],
      filters: { ID: { '>': after } },
      sort: ['+ID'],
      limit: PAGE_SIZE,
    }, { signal });
    const { data: rows } = normalizeListResult(result);
    ids.push(...rows.map((row) => row.ID));
    if (rows.length < PAGE_SIZE) return ids;
    after = rows.at(-1).ID;
  }
}

Compare this list with your downstream IDs. IDs missing downstream need a getTicket and an upsert. Before removing a record that is missing upstream, require a pass that finished without errors, because an incomplete list looks exactly like a mass deletion. Keep that removal explicit and reviewable.

What "Complete" Means

complete: true means one thing: a pass ended on a short page, so at that moment no row after the position was visible to this credential. complete: false means the pass continues on the next run. It does not mean the downstream system equals ZeyOS.

• It is point-in-time. Records can change immediately afterwards, and the next run picks them up.
• It is scoped to the credential. Rows the authenticated user cannot read are not returned, and nothing reports them as absent.
• count is not a completeness check. It returns only a number, ignores sort, offset and limit, and changes while the job runs.
• Timestamps are Unix seconds in UTC. They carry no timezone, so no local conversion belongs in the position. Only compare lastmodified with other server values, never with your own clock.
• The overlap is a tuning value. Five minutes is an example, not a guarantee about commit delays on your instance. It is measured from where a pass ended, so a pass that chases live changes for longer than the overlap narrows that cover.
• Joined data is not tracked. A change to a linked contact does not necessarily update the ticket's lastmodified. The composite filter example shows how to include a joined timestamp, but the two-field position in this tutorial does not cover that case.
• Check this query on a sandbox instance. The composite filter and two-field sort follow the API reference, and positionOf and the ordering check stop the job if a server behaves differently. Bundled metadata does not list lastmodified as indexed, so test page times on large tables before choosing PAGE_SIZE.

5. Use client.request() for Advanced Cases

Use the generated methods first. Drop to client.request() when you need a custom path or an operation outside your current helper layer.

By operation ID, which keeps the validation and read classification of the generated method:

JavaScript

const response = await client.request({
  service: 'api',
  operationId: 'listTickets',
  body: {
    filters: { visibility: 0, priority: 4 },
    limit: 25,
  },
});

By explicit path and method:

JavaScript

const response = await client.request({
  service: 'api',
  method: 'POST',
  path: '/tickets/',
  body: {
    filters: { visibility: 0, status: 4 },
    limit: 25,
  },
});

A method and path call has no operation schema, which costs you two protections:

• No preflight validation, even with validate: true. Unknown fields, invalid enum values, join paths and custom-field keys all reach the server unchecked.
• No read classification. A POST list sent this way counts as a possible write, so it is not retried after a network error, a timeout or a 503 unless you pass retryOnNetworkError: true.

6. Report Failures Without Leaking Data

Log what an operator can act on: the resource, the operation, a category and the next step. err.body and err.message can contain record data from the response, so neither belongs in a log line.

JavaScript

// failures.mjs
import { ZeyosApiError, ZeyosValidationError } from '@zeyos/client';
import { LockHeldError } from './lock-file.mjs';
import { StateFileError } from './state-file.mjs';

const NEXT_STEP = {
  validation: 'Fix the query; no request was sent.',
  authentication: 'Re-authenticate and check the token store.',
  permission: 'Grant the integration user access to this resource.',
  not_found: 'Reconcile the record ID.',
  rate_limited: 'Reduce request volume; the next run resumes from the checkpoint.',
  upstream: 'The next run resumes from the checkpoint. Reconcile any write first.',
  request: 'Review the request against the API reference.',
  timeout: 'Check connectivity. Reconcile any write before retrying it.',
  cancelled: 'The job budget ended or a caller aborted. The next run resumes.',
  state_file: 'Repair the state file named in the message.',
  stale_lock: 'Confirm the recorded process is gone, then delete the lock file.',
  network: 'Check connectivity. Reconcile any write before retrying it.',
};

function categorize(err) {
  if (err instanceof ZeyosValidationError) return 'validation';
  if (err instanceof StateFileError) return 'state_file';
  if (err instanceof LockHeldError) return 'stale_lock';
  if (err instanceof ZeyosApiError) {
    if (err.status === 401) return 'authentication';
    if (err.status === 403) return 'permission';
    if (err.status === 404) return 'not_found';
    if (err.status === 429) return 'rate_limited';
    return err.status >= 500 ? 'upstream' : 'request';
  }
  if (err.isTimeout) return 'timeout';
  if (err.name === 'AbortError' || err.name === 'TimeoutError') return 'cancelled';
  return 'network';
}

export function describeFailure(err, context) {
  const category = categorize(err);
  return {
    ...context,
    category,
    ...(err.status ? { status: err.status } : {}),
    ...(err instanceof ZeyosValidationError
      ? { problems: err.errors.map(({ field, code }) => ({ field, code })) }
      : {}),
    ...(err instanceof StateFileError || err instanceof LockHeldError ? { detail: err.message } : {}),
    next: NEXT_STEP[category],
  };
}

JavaScript

import { describeFailure } from './failures.mjs';

try {
  await client.api.updateTicket({ ID: 42, body: { status: 7 } });
} catch (err) {
  console.error(JSON.stringify(describeFailure(err, { resource: 'tickets', operationId: 'updateTicket', ID: 42 })));
  throw err;
}

Reads are safe to run again, which is why the sync can simply stop and resume. Writes are not: the updateTicket call above may have succeeded even if it threw. See Retries and Rate Limiting for what the client retries, and uncertain write outcomes before you retry one.

7. Know the Limits

• Prefer filters in client code.
• Use visibility: 0 on resources that have the column — accounts, contacts, tickets, tasks, projects, items, documents, notes, opportunities, appointments, campaigns, mailinglists, storages, contracts, ledgers — unless you want archived (1) or deleted (2) records, as a sync usually does. Most other resources have no visibility column — transactions (and every billing/procurement entity), payments, messages, actionsteps, addresses, users, prices, dunning — and filtering on it there returns an opaque HTTP 400. Check with zeyos describe <entity> when unsure.
• Use explicit body objects for update operations that pass ID.
• Treat count-enabled responses defensively and normalize them in one place inside your backend.

Next Steps

• Integration Architecture
• Making Requests
• Authentication