Sync External Content

Keep FoxNose in sync with your external data sources — documentation repositories, CMSes, product catalogs, or any system with stable identifiers. This guide shows how to use external_id and the upsert API to build reliable, idempotent sync pipelines.

What you'll learn:

Why external_id matters for content synchronization
How to upsert individual resources via the Python or JavaScript SDK
How to sync a batch of items in parallel
Two real-world examples: syncing Markdown docs from a repository and importing products from an external API

Prerequisites:

A FoxNose environment with a collection folder and a published schema
A Management API key with resource-write permissions
Python: Python 3.9+ with the foxnose-sdk package (version 0.3.0 or later)
JavaScript: Node.js 18+ with the @foxnose/sdk package (version 0.1.0 or later)

npm install @foxnose/sdk

Why external_id?

When you sync content from an external system, each item already has a natural identifier — a file path, a database ID, a SKU. The external_id field lets you map that identifier to a FoxNose resource so you can:

Create or update in one call — the upsert endpoint checks whether a resource with the given external_id already exists. If it does, a new revision is created; otherwise a new resource is created.
Run syncs idempotently — re-running the same script produces the same result without duplicates.
Avoid maintaining a mapping table — no need to store FoxNose keys externally; the external_id is the link.

external_id is scoped to a folder. Two resources in different folders can share the same external_id.

SDK Setup

All examples below use the ManagementClient. Initialize the client and retrieve the target folder by its path — this gives you a FolderSummary object that you can pass directly to all resource methods:

import { ManagementClient, SimpleKeyAuth } from '@foxnose/sdk'

const client = new ManagementClient({
  baseUrl: 'https://api.foxnose.net',
  environmentKey: process.env.FOXNOSE_ENVIRONMENT_KEY,
  auth: new SimpleKeyAuth(
    process.env.FOXNOSE_PUBLIC_KEY,
    process.env.FOXNOSE_SECRET_KEY,
  ),
})

// Retrieve the folder by its path — no need to know the internal key
const folder = await client.getFolderByPath('content/blog-posts')

Store your credentials in a .env file:

FOXNOSE_PUBLIC_KEY=your-public-key
FOXNOSE_SECRET_KEY=your-secret-key
FOXNOSE_ENVIRONMENT_KEY=your-environment-key

getFolderByPath() / get_folder_by_path() accepts the folder's hierarchical path (e.g. "content/blog-posts" or "content/articles"). The returned FolderSummary object can be passed to upsertResource(), batchUpsertResources(), createResource(), and any other method that expects a folder reference.

Upsert a Single Resource

The simplest case — push one item and let FoxNose decide whether to create or update:

const folder = await client.getFolderByPath('content/blog-posts')

const resource = await client.upsertResource(
  folder,
  {
    title: 'Getting Started with FoxNose',
    body: 'This guide walks you through...',
    author: 'engineering',
  },
  { externalId: 'docs/getting-started.md' },
)

console.log(`${resource.key} — revision ${resource.current_revision}`)

If docs/getting-started.md doesn't exist in the folder → 201 Created (new resource + first revision).
If it already exists → 200 OK (new revision, previous one automatically unpublished).

The returned ResourceSummary always reflects the latest state.

Batch Upsert

When you have tens, hundreds, or thousands of items, batchUpsertResources() / batch_upsert_resources() fans out upsert calls concurrently:

This is an SDK helper, not a separate Management API endpoint. Under the hood it executes concurrent Upsert Resource requests (PUT /v1/:env/folders/:folder/resources/?external_id=<value>).

const folder = await client.getFolderByPath('content/articles')

const items = [
  {
    external_id: 'article-1',
    payload: { title: 'First Article', body: '...' },
  },
  {
    external_id: 'article-2',
    payload: { title: 'Second Article', body: '...' },
  },
  {
    external_id: 'article-3',
    payload: { title: 'Third Article', body: '...' },
  },
]

const result = await client.batchUpsertResources(
  folder,
  items,
  { maxConcurrency: 10 },
)

console.log(`OK: ${result.succeeded.length}, Failed: ${result.failed.length}`)

Key parameters

JS / Python	Default	Description
`maxConcurrency` / `max_concurrency`	`5`	Maximum parallel requests. Increase for throughput, lower if you hit rate limits.
`failFast` / `fail_fast`	`false`	When `true`, stops on the first error and raises the exception. When `false`, processes all items and collects failures.
`onProgress` / `on_progress`	`undefined` / `None`	Callback `(completed, total)` called after each item finishes.

Handling failures

With failFast / fail_fast set to false (the default), inspect the result to handle partial failures:

const result = await client.batchUpsertResources(folder, items)

if (result.failed.length > 0) {
  for (const error of result.failed) {
    console.log(`  [${error.index}] ${error.external_id}: ${error.error.message}`)
  }
} else {
  console.log(`All ${result.succeeded.length} items synced successfully`)
}

With failFast / fail_fast set to true, the first error raises immediately — useful for CI pipelines where any failure should stop the build:

try {
  const result = await client.batchUpsertResources(folder, items, {
    failFast: true,
  })
} catch (err) {
  console.error(`Sync aborted: ${err.message}`)
  throw err
}

Example: Sync Markdown Documentation

A documentation repository stores .md files. Each file path is a natural external_id. Here's a complete script that walks a directory, parses front matter, and pushes everything to FoxNose:

import { readdir, readFile } from 'node:fs/promises'
import { join, relative, basename, extname } from 'node:path'
import { ManagementClient, SimpleKeyAuth } from '@foxnose/sdk'

const client = new ManagementClient({
  baseUrl: 'https://api.foxnose.net',
  environmentKey: process.env.FOXNOSE_ENVIRONMENT_KEY,
  auth: new SimpleKeyAuth(
    process.env.FOXNOSE_PUBLIC_KEY,
    process.env.FOXNOSE_SECRET_KEY,
  ),
})

const DOCS_DIR = './docs'
const FOLDER_PATH = 'content/documentation'

function parseFrontmatter(text) {
  const match = text.match(/^---\n([\s\S]+?)\n---\n([\s\S]*)/)
  if (!match) return [{}, text]

  const meta = {}
  for (const line of match[1].split('\n')) {
    const idx = line.indexOf(':')
    if (idx === -1) continue
    const key = line.slice(0, idx).trim()
    const value = line.slice(idx + 1).trim().replace(/^['"]|['"]$/g, '')
    meta[key] = value
  }

  return [meta, match[2].trim()]
}

async function collectFiles(dir) {
  const entries = await readdir(dir, { withFileTypes: true, recursive: true })
  return entries
    .filter((e) => e.isFile() && e.name.endsWith('.md'))
    .map((e) => join(e.parentPath, e.name))
    .sort()
}

async function main() {
  const folder = await client.getFolderByPath(FOLDER_PATH)
  const files = await collectFiles(DOCS_DIR)

  const items = await Promise.all(
    files.map(async (filePath) => {
      const relativePath = relative(DOCS_DIR, filePath).replaceAll('\\', '/')
      const text = await readFile(filePath, 'utf-8')
      const [meta, body] = parseFrontmatter(text)

      return {
        external_id: relativePath,
        payload: {
          title: meta.title ?? basename(filePath, extname(filePath)),
          body,
          path: relativePath,
          category: meta.category ?? 'general',
        },
      }
    }),
  )

  console.log(`Found ${items.length} documents to sync`)

  const result = await client.batchUpsertResources(folder, items, {
    maxConcurrency: 10,
    onProgress: (done, total) =>
      process.stdout.write(`\r  Progress: ${done}/${total}`),
  })

  console.log(
    `\nDone — ${result.succeeded.length} synced, ${result.failed.length} failed`,
  )

  for (const error of result.failed) {
    console.log(`  FAILED ${error.external_id}: ${error.error.message}`)
  }
}

main()

Running in CI

This script is idempotent — run it on every push or merge to main:

# .github/workflows/sync-docs.yml (excerpt)
- name: Sync docs to FoxNose
  env:
    FOXNOSE_PUBLIC_KEY: ${{ secrets.FOXNOSE_PUBLIC_KEY }}
    FOXNOSE_SECRET_KEY: ${{ secrets.FOXNOSE_SECRET_KEY }}
    FOXNOSE_ENVIRONMENT_KEY: ${{ secrets.FOXNOSE_ENVIRONMENT_KEY }}
  run: node sync_docs.mjs

Because each file's path is the external_id, only changed files create new revisions. Unchanged content still produces a revision (the API does not diff payloads), so if you want to skip unchanged files, compare checksums locally before calling upsert.

Example: Import Products from an External API

A common integration: an e-commerce platform exposes a product catalog via REST API, and you want to mirror it into FoxNose for search, localization, or headless delivery.

import { ManagementClient, SimpleKeyAuth } from '@foxnose/sdk'

const client = new ManagementClient({
  baseUrl: 'https://api.foxnose.net',
  environmentKey: process.env.FOXNOSE_ENVIRONMENT_KEY,
  auth: new SimpleKeyAuth(
    process.env.FOXNOSE_PUBLIC_KEY,
    process.env.FOXNOSE_SECRET_KEY,
  ),
})

const PRODUCTS_API = 'https://api.example.com/v1/products'
const FOLDER_PATH = 'catalog/products'

async function fetchProducts() {
  const products = []
  let url = PRODUCTS_API

  while (url) {
    const response = await fetch(url)
    if (!response.ok) throw new Error(`HTTP ${response.status}`)
    const data = await response.json()
    products.push(...data.results)
    url = data.next ?? null
  }

  return products
}

async function sync() {
  const folder = await client.getFolderByPath(FOLDER_PATH)
  const products = await fetchProducts()
  console.log(`Fetched ${products.length} products from external API`)

  const items = products.map((p) => ({
    external_id: `product-${p.id}`,
    payload: {
      name: p.name,
      description: p.description,
      price: p.price,
      sku: p.sku,
      category: p.category,
      image_url: p.image_url,
      in_stock: p.inventory > 0,
    },
  }))

  const result = await client.batchUpsertResources(folder, items, {
    maxConcurrency: 15,
    onProgress: (done, total) =>
      process.stdout.write(`\r  ${done}/${total} products synced`),
  })

  console.log(
    `\nSync complete: ${result.succeeded.length} OK, ${result.failed.length} failed`,
  )

  for (const error of result.failed) {
    console.log(`  FAILED ${error.external_id}: ${error.error.message}`)
  }
}

sync()

The product ID from the external system (product-{id}) serves as the external_id. Running this script hourly or on a webhook keeps FoxNose up to date without maintaining any mapping state.

Async Variant

For high-throughput pipelines or async frameworks, the Python SDK provides AsyncManagementClient — the API is identical to the synchronous client. The JavaScript SDK is already fully asynchronous (all methods return Promises), so no separate client is needed.

// The JavaScript SDK is async by default — no separate client needed.
import { ManagementClient, SimpleKeyAuth } from '@foxnose/sdk'

const client = new ManagementClient({
  baseUrl: 'https://api.foxnose.net',
  environmentKey: 'YOUR_ENVIRONMENT_KEY',
  auth: new SimpleKeyAuth('PUBLIC_KEY', 'SECRET_KEY'),
})

const folder = await client.getFolderByPath('content/articles')

const result = await client.batchUpsertResources(
  folder,
  [{ external_id: 'a-1', payload: { title: 'Async Article' } }],
  { maxConcurrency: 20 },
)

console.log(`Done: ${result.succeeded.length} OK, ${result.failed.length} failed`)

Tips

Choose stable external IDs. Use identifiers that don't change across syncs — file paths, database primary keys, SKUs. Avoid auto-generated UUIDs that rotate.
Tune concurrency. Start with maxConcurrency / max_concurrency set to 5 and increase until you see 429 rate-limit errors, then back off slightly.
Use failFast / fail_fast in CI. If any item fails, fail the build so you can fix the data before deploying.
Use failFast: false / fail_fast=False for bulk imports. Sync as much as possible and fix failures separately.
Scope by folder. external_id is unique per folder. The same ID can exist in different folders without conflict.
Schema validation. By default, data is validated against the published schema. Pass validate_data=False in the payload if you want to skip validation for drafts.

API Reference

Upsert Resource — PUT endpoint details, query parameters, error codes
Create Resource — POST with optional external_id
Resource Object — full field reference including external_id

What's Next?

Your sync pipeline is ready. Two directions from here:

Expose synced content via API — create a Flux API so your apps can search and retrieve the content you've synced.

Expose Content via Flux API

Build AI features on top — use the synced content as a knowledge base for RAG pipelines and LLM agents.

RAG Pipeline with Python RAG Pipeline with JS / Node.js