Module 5: Credential And Secrets Security

8. Project: harden credentials and secrets

Next →

Description

By the end of this lesson you'll have transformed Terra Market's instance from a messy keyring into a governed one, in six phases with verifiable deliverables. You'll build the complete inventory and run the audit; you'll confirm —not assume— that the encryption key is backed up and that it restores; you'll reduce erp_api from administrator to its minimum scope without breaking production; you'll get out the secrets that were left written in the workflows; you'll harden the instance at the application level with the access review and the surface trimming; and you'll set the budget, the alerts, and the runbook for llm_token. In the end you'll have six artifacts that can be shown and an instance that is materially harder to compromise, not just better documented.

This matters because it's where the module stops being knowledge. The previous seven lessons gave you a mental model, procedures, and plan honesty; none changed a single credential. This project does. And it's designed to be run on an instance that's running, with 4,000 daily executions that can't be stopped, because that's the real condition of any operation in production and it's what separates a lab exercise from a defensible piece of work.

Connection to the module: this project closes module 5 and uses all seven lessons in order. Phase 1 is lesson 1's inventory plus lesson 3's audit. Phase 2 applies lesson 2's storage model —and refers to the Self-Hosting and Operations guide, module 3, lesson 5 for the key setup, which isn't repeated here. Phase 3 is lesson 3's least-privilege and rotation procedure. Phase 4 applies lesson 5's rule —secrets go in credentials, the code sees nothing— with lesson 4's honest decision already made. Phase 5 is all of lesson 6, at the application level, with network hardening delegated to the Self-Hosting guide, module 4, lesson 6. And phase 6 is lesson 7. Looking ahead, module 6 will scale this instance with queue mode —where the encryption key you secured in phase 2 has to reach all the workers— and module 7 will fine-tune the AI budget you set in phase 6.

The goal, in one sentence

You're not buying "security" in the abstract. You're buying two concrete and measurable things:

First: that the worst possible case be smaller. Today, if Terra Market's ERP key leaks, the worst case is that someone deletes or modifies the system that governs the company's orders, inventory, and prices. When you finish, the worst case will be that someone reads inventory and creates fake orders. It's still bad, and it's an order of magnitude less bad. That's defense 3 from lesson 1 made real.

Second: that things stop depending on someone remembering. Today rotation happens when someone thinks of it, AI spending is reviewed when the bill arrives, and former employees' accounts are deleted if someone notices. When you finish, each one will have a named owner, a date, and —where possible— a mechanism that warns on its own.

And a note before starting: this project isn't done in an afternoon, and it shouldn't be done in an afternoon. Phases 1, 2, 5, and 6 fit in a day. Phases 3 and 4 touch workflows that are running, and their correct procedure includes observation periods. A reasonable pace is two weeks, with most of the time being waiting, not work. If your instance is a practice one, you can compress it; if it's production, respect the waits.

Each phase is a complete worked example and has the same structure: 📍 Where you are, the goal, the steps with their What to expect, and the deliverable. And this applies to all six: you run everything in your instance, on your server, and in your providers' panels. The guide runs nothing.

Phase 1 — Inventory and audit

📍 Where you are: at the beginning. You haven't changed anything yet, and that's fine: the first phase changes nothing on purpose. What isn't known can't be protected, and in an inherited instance almost nothing is known.

Goal of the phase: to have the complete and honest list of which credentials exist, who uses them, what permissions they have, when they were rotated, and who answers for them.

Step 1 — List the instance's credentials

In the n8n interface, open the credentials section and note all of them: name, type, identifier.

What to expect: a list longer than you remembered, with at least one credential whose purpose nobody remembers. It's normal, and it's precisely the finding.

Step 2 — Run the built-in audit

According to the documentation, you have three paths. The most direct from the server:

n8n audit

The other two: a POST call to the API's /audit endpoint authenticating as the instance owner, or the n8n node inside a workflow with Resource > Audit and Operation > Generate.

What to expect: a report with five sections —credentials, database, filesystem, nodes, and instance. Save it as is, with a date. It's your baseline, and in phase 5 you'll come back to it.

Of this report, for this phase you're interested in the credentials section, which according to the documentation points out three things: credentials that no workflow uses, that no active workflow uses, and that no recently active workflow uses.

Step 3 — Answer the four questions per credential

For each row: which workflows use it, what permissions it has in the external system (this one is answered by getting into that system's panel, not in n8n), when it was rotated, and who is its owner.

What to expect: several cells with "to be confirmed" and the owner column empty in every row. Write the gaps as gaps: an honest inventory with six "to be confirmed" is worth more than a complete but invented one, because each gap is a task and a guess is false reassurance.

Step 4 — Assign owners

Each credential needs a person with a name responsible for rotating it, deciding who uses it, and answering when something odd shows up. "The operations team" is not an owner.

What to expect: a twenty-minute conversation and a column that gets filled. And the rule Terra Market adopts from today: a credential without an owner doesn't go into production.

Deliverable 1 — CREDENTIAL-INVENTORY.md

CREDENTIAL INVENTORY — Terra Market — production
Date: <today>   ·   Surveyed by: <name>   ·   Next review: <today + 3 months>

| Credential  | Type        | Used by                      | Permissions it HAS   | Permissions it USES | Gap    | Last rotation | Owner    |
|-------------|-------------|------------------------------|----------------------|---------------------|--------|---------------|----------|
| erp_api     | Header Auth | order-sync, inventory-update | Administrator (all)  | Read + write        | HUGE   | Never         | <name>   |
| carrier_api | Header Auth | shipment-notify              | Read shipments       | Read                | none   | 14 months ago | <name>   |
| llm_token   | API Key     | 2 classification workflows   | All models           | One model           | medium | Never         | <name>   |
| store_api   | OAuth2      | order-sync, inventory-update | To be confirmed      | Read + write        | ?      | To be confirmed | <name> |
| smtp_notify | SMTP        | shipment-notify              | Send email           | Send email          | none   | 8 months ago  | <name>   |
| (orphan)    | Header Auth | none                         | To be confirmed      | —                   | ?      | To be confirmed | (to revoke) |

FINDINGS
  F1  erp_api with administrator permissions and a huge gap             → Phase 3
  F2  1 orphan credential that no workflow uses                         → Phase 1, step 5
  F3  store_api with unconfirmed permissions                           → open task
  F4  llm_token with no spending cap or alert                          → Phase 6
  F5  No credential had an owner (fixed in this phase)                 → closed

Step 5 — Close the cheap one: the orphans

Before moving to phase 2, resolve the cheapest finding. Ask the team with a deadline —"if nobody claims these credentials by Friday, I revoke them"— and then, in this order: revoke the key in the external system, and then delete the credential from n8n.

The order matters: deleting the credential from n8n without revoking the key leaves the key alive in the external system and now with no record that it existed. It's the worst of both worlds.

What to expect: one or two fewer credentials and a risk eliminated in twenty minutes.

Phase 2 — The encryption key, verified

📍 Where you are: you have the inventory and you've already eliminated the orphans. Now you're going to secure the piece that all the credentials on the list depend on: the key that encrypts them. If this one fails, the whole inventory becomes unreadable.

Goal of the phase: to confirm —not assume— that the encryption key exists explicitly, that it's backed up outside the server, and that the credentials survive a restart.

Boundary. The setup of the key —generating it with openssl, placing it in the .env before the first startup, what exactly happens if it's lost or changed— is developed in depth in the Self-Hosting and Operations guide, module 3, lesson 5, and isn't repeated here. This phase verifies that that work is done well in your instance and adds what belongs to this module.

Step 1 — Confirm that the key is explicit

Review your stack's .env and confirm that N8N_ENCRYPTION_KEY is present with a real value.

What to expect: a line with a long, random string. If the variable isn't there, your instance is using the automatic key that n8n generates and stores in ~/.n8n —according to the documentation— and you have a significant problem: you don't know your key, so you can't back it up. That case is resolved with the Self-Hosting guide's procedure; don't improvise it, because changing the key over already-encrypted credentials makes them unreadable.

Step 2 — Verify the backup, for real

This is the part of this phase that provides new value, and it doesn't consist of asking whether there's a backup: it consists of using it.

Go to the place where the team says the key is backed up —the password manager, the team's vault— and compare it character by character with the one in the .env.

What to expect: that they match exactly. If they don't match, or if you can't find the entry, you just discovered the most serious finding possible: your database backup restores nothing, because it's a safe with no combination. Fix it today, by copying the key from the .env to the manager with a clear label.

Step 3 — Test that the credentials survive a restart

docker compose down
docker compose up -d

Get into n8n and open any credential. Then manually run one of the workflows.

What to expect: the credentials are still there and the workflow authenticates without problems. That's the confirmation that the key in the .env is the same one they were encrypted with. If you saw "could not decrypt the credential" errors, stop: someone changed the key at some point, and the recovery procedure is in the Self-Hosting guide.

Step 4 — Decide on encryption key rotation

According to the documentation, n8n offers encryption key rotation on self-hosted instances, enabled by the owner with N8N_ENV_FEAT_ENCRYPTION_KEY_ROTATION=true on all instances (main process and workers). The model is two-key: the instance one protects the data key, which is the one that gets rotated.

And the warnings, which the documentation marks clearly: it's a one-way change, it requires a full database backup before, and once enabled, disabling the variable or downgrading the version leaves the data permanently inaccessible.

For this project, the reasonable decision is to document the decision, not to run it lightly. If Terra Market wants it, it's scheduled in a window with a verified backup. What this phase does produce is the written decision, with a date and a person responsible.

Deliverable 2 — ENCRYPTION-KEY-BACKUP.md

ENCRYPTION KEY VERIFICATION — Terra Market
Date: <today>   ·   Verified by: <name>

[x] N8N_ENCRYPTION_KEY explicit in the .env (not automatic)
[x] Backup located in <team's password manager>, entry "<label>"
[x] Character-by-character comparison: MATCHES
[x] Restart test: credentials decrypt correctly
[x] Manual run of order-sync after the restart: OK
[x] The .env is NOT in the repository (verified with git status)

ENCRYPTION KEY ROTATION
    Status: not enabled
    Decision: <enable in the <date> window / not enable for now>
    Responsible: <name>
    Prerequisite: full database backup, verified by restoring

OPERATIONS REMINDER
    In queue mode (module 6), ALL workers must receive the same
    N8N_ENCRYPTION_KEY. Verify when scaling.

Phase 3 — Least privilege on erp_api

📍 Where you are: the inventory is done and the key that encrypts everything is verified. Now you go to the most serious finding: the credential with administrator permissions that only needs to read and create orders. This is the phase that most reduces the worst case, and the one that requires the most care because it touches production workflows.

Goal of the phase: for erp_api to go from administrator to its minimum scope, without order-sync or inventory-update failing a single execution.

Step 1 — Inventory the real operations

Go through the two workflows and note every call to the ERP: method and path.

Observed operations of erp_api

order-sync
  POST /api/v1/orders             → create order
  GET  /api/v1/customers/{id}     → read customer

inventory-update
  GET  /api/v1/inventory          → read stock
  GET  /api/v1/products/{sku}     → read product

Total: 1 write, 3 reads. Zero deletes. Zero configuration.

What to expect: a short list. If you have observability of the outbound calls (module 4), cross-check this list against what actually got called in the last month: it covers the branches that only run in rare cases and that reading the workflow doesn't reveal.

Step 2 — Translate into the ERP's vocabulary

Proposed minimum scope:
  orders:write · customers:read · inventory:read · products:read

Removed relative to the administrator role:
  orders:delete, inventory:write, products:write, customers:write,
  settings:*, users:*, and everything else in the system.

Step 3 — Create a NEW credential, don't modify the old one

In the ERP, generate a key with the scope from step 2. In n8n, create a new credential with it, called erp_api_scoped. If the ERP allows service accounts, take the chance to create it from a dedicated identity —svc-n8n-erp— instead of from someone's personal account.

What to expect: two credentials in the list and no workflow using the new one yet. That intermediate state is correct: the intact old key is your rollback and it costs nothing to keep it.

Step 4 — Migrate the lower-risk workflow and observe

Change only inventory-update —which only reads— to use erp_api_scoped.

What to expect: the executions stay green and stock keeps updating in the store. If something fails, you'll see an ERP authorization error (typically a 403) that tells you which operation is missing. That's not a failure: it's the discovery of an operation that your step 1 inventory didn't include. Add that scope to the new key and test again.

Let at least one full operating cycle go by before continuing.

Step 5 — Migrate order-sync and observe more closely

Now the one that writes. Change its nodes to erp_api_scoped.

What to expect: executions green and —this is what really has to be verified— the orders reaching the ERP. A call that returns 200 without creating anything is a silent failure; module 3 gave you the tools to detect it.

Step 6 — Revoke the old key

With the two workflows stable, get into the ERP and revoke the previous key. Don't disable it "just in case": revoke it. Then delete the erp_api credential from n8n.

What to expect: the executions stay green, because nobody used the old one since step 4. And an extra benefit that closes the module's finding 1: the key that eight people had in their chat just stopped working, without needing to ask anyone to delete it. If something went down when revoking, you just discovered a consumer of that key outside n8n, and that's a valuable finding on its own.

Deliverable 3 — Inventory update and log

SCOPE-REDUCTION LOG — erp_api
<date>  Operations inventoried: 1 write, 3 reads
<date>  New key created in the ERP with minimum scope (svc-n8n-erp)
<date>  inventory-update migrated to erp_api_scoped
<date>  48h observation: no incidents
<date>  order-sync migrated; orders verified in the ERP
<date>  48h observation: no incidents
<date>  Old key REVOKED in the ERP · erp_api credential deleted from n8n
        Intended side effect: the key that circulated over chat is now useless

| Credential     | Permissions it HAS                                          | Permissions it USES | Gap  | Last rotation | Owner  |
|----------------|-------------------------------------------------------------|---------------------|------|---------------|--------|
| erp_api_scoped | orders:write, customers:read, inventory:read, products:read | the same ones       | none | <date>        | <name> |

Phase 4 — Getting the secrets out of the workflows

📍 Where you are: the credentials are inventoried, the key that encrypts them is verified, and the most dangerous one is already bounded. But all that work is voided if a secret is written outside a credential: in a node field, in a Code node, or —the worst case— in an item that stays stored in the execution history.

Goal of the phase: for no Terra Market secret to live outside an n8n credential or the .env, and for none to pass through the data.

Step 1 — Look for secrets written by hand in the workflows

Export the workflows and look in their JSON for strings that look like keys: in header fields, in query parameters, in URLs, in Edit Fields nodes, in Code nodes.

# Export the workflows to files so you can search their text.
n8n export:workflow --all --output=/tmp/wf-review.json

What to expect: with luck, nothing. What's usual is to find one or two, almost always in an API that authenticates in a nonstandard way and where someone pasted the key into the headers field "for now."

What to do with each finding. The fix is a generic credential: Header Auth if the secret goes in a header, Query Auth if it goes as a URL parameter. And the correct split: the secret to the credential, the non-secret —an account number, a base URL— to the node field. In Terra Market, carrier_api is the typical case: the key goes in the Header Auth credential and the account number goes as a normal parameter.

And an important warning: if you found a secret in a workflow that's under version control, that secret is considered compromised, because Git doesn't forget. Besides moving it to a credential, you have to rotate it.

Step 2 — Look for secrets that pass through the data

Open recent executions of each workflow and go through the nodes' outputs looking for strings that look like keys. Pay special attention to the Code nodes and the Edit Fields.

What to expect: any field like auth_header, token, api_key, or Bearer … in a node's output is a finding. Remember from lesson 2: the execution history isn't encrypted, so a secret there is in text, replicated in every execution.

The fix isn't to hide it better: it's to take it out of the code. In n8n 2.0 the Code node doesn't make HTTP calls, so it never needs the token. The one that calls is the HTTP Request node or the service's node, with its credential.

// ============================================================
// Node: Code — "Build ERP payload"
// Mode: Run Once for All Items
//
// INPUT:   store orders
// OUTPUT:  one item per order, ready for the ERP
// NOTE:    this node does NOT know any token or any URL.
//          They're handled by the next HTTP Request node with the
//          erp_api_scoped credential.
// ============================================================

return $input.all().map((item, i) => ({
  json: {
    external_id: item.json.order_id,
    customer: item.json.customer_name,
    total: item.json.order_total,
  },
  pairedItem: i,
}));

What to expect after the fix: the output panel shows the order data and no string that looks like a key. That absence is the sign of success.

And if you found secrets in the history, the fix doesn't end at the workflow: delete the affected executions and rotate the secret, because it was visible.

Step 3 — Protect the .env

Verify lesson 5's three pieces:

# .gitignore
.env
.env.*
!.env.example

Confirm that the .env.example exists with the variable names and no values, and adopt the reflex: git status before each commit, confirming that no .env appears in the list.

What to expect: that git status shows no .env. If your repository already had one in some commit, the secret is considered compromised and has to be rotated: deleting it from the last commit doesn't delete it from the history.

Step 4 — Verify $env's behavior in your instance

Remember the contradiction from lesson 5: the documentation's security variables page lists N8N_BLOCK_ENV_ACCESS_IN_NODE with a default value of false, and the version 2.0 changes page says it became true. Both are official. The instance decides.

// Diagnostic Code node. Run it ONCE and delete the node.
return [{ json: { env_access: typeof $env } }];

What to expect: "undefined" or an access error means the block is active, which is the safe stance. "object" means it's allowed, and that's a finding: anyone with edit permission can dump the entire process environment —including the encryption key— to the output panel. In that case, set it explicitly to true instead of depending on an ambiguous default value. And delete the node after the test: if access turned out to be allowed, leaving it saved leaves the dump in the history.

Deliverable 4 — SECRETS-IN-WORKFLOWS-AUDIT.md

SECRETS OUTSIDE CREDENTIALS — Terra Market
Date: <today>   ·   Reviewed by: <name>

SEARCH IN THE JSON OF THE 18 WORKFLOWS
  Findings: <N>
  <workflow> · <node> · <field>  → moved to credential <type>  · ROTATED: yes/no

SEARCH IN THE EXECUTION HISTORY
  Findings: <N>
  <workflow> · <node> · <field>  → node fixed · executions deleted · ROTATED: yes/no

.env PROTECTION
  [x] .gitignore covers .env and .env.* with an exception for .env.example
  [x] .env.example present, with no values
  [x] git status clean of .env
  [ ] The repository had a .env in the history → if so, rotate EVERYTHING it contained

$env BEHAVIOR IN THIS INSTANCE
  Result of `typeof $env` in a Code node: <undefined / object>
  Decision: N8N_BLOCK_ENV_ACCESS_IN_NODE set explicitly to <true/false>
  Note: the official documentation contradicts itself about the default value.
        This result is OUR instance's and wins over the docs.

Phase 5 — Harden the instance at the n8n level

📍 Where you are: the secrets are where they should be, with the scope they should have, and they don't pass through the data. The other half is missing: who can reach them from inside the application.

Goal of the phase: for only the people who should to have an account, for the accounts to be protected, and for the application's surface to be the minimum.

Boundary. The hardening of network and server —closing port 5678, the reverse proxy with HTTPS, the firewall, the security headers, the SSH tunnel— is the Self-Hosting and Operations guide, module 4, lesson 6, and isn't done here. This phase is application hardening. If that other half isn't done in your instance, do it: it's a prerequisite, not an alternative.

Step 1 — Access review

List the accounts and answer for each one: is this person still in the area? do they need to edit automations? do they have 2FA? when did they last log in?

What to expect: the three classic patterns —accounts of people who are gone (deleted today), shared area accounts that destroy traceability, and test accounts that survived for years. It's normal to end up with half the accounts.

And don't forget lesson 3's trigger: if someone who left knew the value of any credential, besides deleting the account you have to rotate that credential.

Step 2 — Protect the owner's account

Long, unique password stored in a manager. 2FA activated. And the rule Terra Market adopts: the owner's account isn't shared, ever.

Remember lesson 6's plan honesty: according to the documentation, the Admin role is on the Pro and Enterprise plans, and RBAC is on all plans except Community. In Community there are owner and members, and nothing in between. That friction is real; the solution isn't to share the owner's account, it's to document the friction as an argument for when the team grows.

What to expect: an instance where a single person can log in as owner, with a second factor.

Step 3 — Trim the application's surface

# .env — turn off what Terra Market doesn't use.

# n8n's public API, if nobody uses it:
N8N_PUBLIC_API_DISABLED=true
N8N_PUBLIC_API_SWAGGERUI_DISABLED=true

# Nodes that give access to the server where n8n runs.
# The value is a JSON array written as a string.
NODES_EXCLUDE="[\"n8n-nodes-base.executeCommand\", \"n8n-nodes-base.readWriteFile\"]"

What to expect after restarting the stack: the blocked nodes stop appearing in the editor's node search, and the public API stops responding. Verify both things instead of assuming them.

Before blocking, check with the audit report whether any workflow uses those nodes. If any uses them, it has to be redesigned first —very often what they do is resolved outside n8n— and blocked afterward.

Step 4 — Close the audit findings

Go back to the phase 1 report and go through the five sections. Credentials is already closed in phase 1. Filesystem and nodes are the candidates from step 3, plus the installed community nodes —third-party code with your instance's access, which deserves a conscious decision. Database lists expressions in SQL query fields, which are injection risk. And Instance lists the unprotected webhooks: each one is an open door, because anyone who discovers the address triggers the workflow. In Terra Market, an unauthenticated shipment-notify would allow injecting fake confirmations and sending messages to real customers; authenticate them with the node's own options and validate the signature when the provider sends it.

Step 5 — Automate the audit

Set up a workflow that runs the audit on the first Monday of each month and sends the report to where the team looks. With the n8n node: Resource > Audit, Operation > Generate.

What to expect: a monthly report that arrives on its own. This turns a task done when someone remembers into one that happens on its own, and it's probably the most durable deliverable in the whole project.

Deliverable 5 — ACCESS-REVIEW.md and the checklist

ACCESS REVIEW — Terra Market — <date>   ·   Next: <date + 3 months>

| Account  | Role   | In the area? | Needs to edit? | 2FA | Last login    | Decision  |
|----------|--------|--------------|----------------|-----|---------------|-----------|
| ...      | ...    | ...          | ...            | ... | ...           | ...       |

Accounts before: <N>   ·   Accounts after: <M>   ·   Deleted: <N-M>
Credentials rotated due to staff departures: <list>

n8n-LEVEL HARDENING
  [x] No accounts of people who are gone
  [x] One account per person (no area accounts)
  [x] Owner account with a unique password, not shared
  [x] 2FA active on the owner's account
  [x] Public API turned off (N8N_PUBLIC_API_DISABLED)
  [x] Risky nodes blocked (NODES_EXCLUDE) and verified in the editor
  [x] Webhooks from the instance report authenticated
  [x] Automated audit: monthly workflow → <destination>

  Not available in Community (documented as known friction):
  [ ] Mandatory 2FA for everyone (Business/Enterprise)
  [ ] Granular projects and roles (all plans except Community)
  [ ] Sharing credentials without revealing their value (Cloud / Business+ self-hosted)
  [ ] Execution data redaction (Enterprise)

NETWORK HARDENING — outside this module's scope
  See Self-Hosting and Operations guide, module 4, lesson 6.
  Status in our instance: <done / pending>

Phase 6 — AI budget and runbook

📍 Where you are: everything before this protects keys that open data. The one that opens an account that charges per use is missing, and today it has no cap, no alert, and no spending owner.

Goal of the phase: for llm_token to have a known ceiling, an alert that reaches someone who reads it, and a written runbook for when it fires.

Step 1 — Measure current consumption

In the provider's panel, review the consumption over the last few months and note the monthly average P.

What to expect: a figure. If there isn't enough history, estimate from the volume (how many of the 4,000 daily executions touch the model) and treat it as a hypothesis to review in a month.

Step 2 — Configure the three numbers

Alert 1 (early):  at half of what's expected for that date in the month
Alert 2 (serious): at 100% of P
Hard limit:        around 2 × P

The names of these options vary between providers —"usage limits," "budgets," "billing alerts"; look for them in the billing section of your panel and verify there, not in a tutorial.

What to expect: a confirmation in the panel that all three became active. And a conversation with finance about the hard limit's trade-off: if legitimate use reaches it, the classification workflows stop. It's better to say it before than to explain it after.

Step 3 — Test that the alert arrives

Temporarily lower alert 1's threshold below current consumption, wait for it to fire, confirm the notice arrived where someone reads it, and raise it again.

What to expect: a notice in minutes or hours. This step is the one that separates "we have alerts" from "we have alerts that work," and it's the one almost nobody does.

Configure at least two recipients and, better still, make it arrive at the channel where the team already looks at module 4's operational alerts.

Step 4 — Separate the keys by use and narrow the scope

If the same token is used by production and experiments, separate them: llm_token_prod and llm_token_experiments, with different budgets. And if the provider allows restricting a key to specific models, restrict the production one to the model your workflows actually use.

What to expect: being able to answer "where does this spending come from?" by looking at the panel, in minutes instead of hours.

Step 5 — Rotate llm_token and observe what breaks

Take advantage of being here. Generate a new key, update the credential, observe for a day, and revoke the old one.

What to expect: the executions stay normal. And pay attention to the provider's panel after revoking: if any authentication error from the revoked key shows up, someone else was using it from somewhere else. That finding, on an AI token, justifies the rotation on its own.

Step 6 — Review the prompts

Open an execution of each AI workflow and read the complete prompt the way someone from outside would read it.

What to expect: that there's no secret and that the personal data is the minimum the task needs. To classify a message you almost never need the customer's name, phone, or email. Remember that the prompt travels to the provider and is stored in the execution history, which isn't encrypted, and that data redaction is Enterprise: in Community the defense is not putting them in.

Deliverable 6 — AI-BUDGET-AND-RUNBOOK.md

AI SPENDING CONTROL — Terra Market — <date>

CONSUMPTION
  Observed monthly average (P): <amount>   ·   Basis: <months of history / estimate>

CAPS AND ALERTS
  Alert 1: <amount> → <recipients / channel>   ·   Tested on <date>: ARRIVED
  Alert 2: <amount> → <recipients / channel>
  Hard limit: <amount>  (effect: the provider stops serving; classification
              workflows stop. Communicated to finance on <date>.)

KEYS
  llm_token_prod         → production workflows · restricted to <model> · owner <name>
  llm_token_experiments  → team tests · low budget · owner <name>
  Rotation: <date> · next: <date + 4 months>
  Finding when revoking the old key: <none / external consumer detected>

PROMPTS REVIEWED
  [x] No secret in any prompt
  [x] Personal data reduced to the minimum needed for the task

RUNBOOK — suspected AI key leak
  0–10 min   1. REVOKE the key in the provider's panel. Before investigating.
                Every minute of analysis with the key alive is additional spending.
             2. Confirm the hard limit is still active.
  10–30 min  3. Generate a new key, update the credential, verify executions.
  30–90 min  4. Review consumption: how much, since when, which models, from where.
             5. Look for the exit route: repository (history, not just the current
                state), decrypted export, token in a ticket or chat, account of someone
                who's gone, token in a prompt or in an item.
  90–120 min 6. Rotate everything that could have gotten out by the same route.
             7. Write down what happened and notify finance with the figure.

  Responsible: <name>   ·   Backup: <name>

Acceptance criteria

The project is finished when you can answer yes to all of this, with evidence:

Inventory and governance

  • Complete inventory with the two permissions columns and the gap between them.
  • Each credential has a named owner and a next-rotation date.
  • No orphan credential: the ones that were there were revoked in the external system and deleted from n8n, in that order.

Encryption key

  • The key is explicit in the .env, not automatic.
  • The backup was verified by comparing it character by character, not taken for granted.
  • The credentials survive a stack restart, verified.
  • The decision about encryption key rotation is written, with a person responsible.

Least privilege

  • erp_api was reduced to its minimum scope and the old key is revoked.
  • The migration was done with a new credential, one workflow at a time, with observation between steps.
  • The gap between "has" and "uses" is none where it could be confirmed; the rest is marked as an open task with a person responsible.

Secrets out of the data

  • No secret written by hand in the JSON of any workflow.
  • No secret in the output of any node, verified by opening real executions.
  • Every secret found outside a credential was rotated, not just moved.
  • The .env is protected with .gitignore (including the variants) and there's a .env.example.
  • $env's behavior was tested in the instance and set explicitly.

Instance

  • No accounts of people who are gone, and the credentials they knew were rotated.
  • One account per person; no area accounts. The owner's with 2FA and not shared.
  • Public API turned off if not used and risky nodes blocked, verified in the editor.
  • The unprotected webhooks from the report are authenticated.
  • The audit runs on its own and reaches someone.
  • What isn't available in Community is documented as known friction.

AI

  • llm_token has a hard limit and two alerts, and the alert was tested and arrived.
  • The keys are separated by use and narrowed to the model that's used.
  • No prompt contains secrets, and the personal data is the minimum.
  • The runbook is written, with a person responsible and a backup.

Common mistakes

Producing documents instead of changes (conceptual, and the one that ruins the project). What happens: the six deliverables are written in detail, and when you finish erp_api still has administrator permissions and llm_token still has no cap. The project feels done because there are artifacts. Why it happens: documenting is convenient and visible; changing a credential in production is scary. How to spot it: ask yourself what changed in the system, not in the documentation folder. If the answer is "nothing," this is it. How to fix it: the deliverables are the record of the work, not the work. The proof of this project isn't the file: it's that the old ERP key no longer works and that the spending alert arrived when you tested it.

Doing the six phases in a day (practical, and it causes incidents). What happens: everything is compressed into one day, the two ERP workflows are migrated back to back without observing in between, and a failure that only shows up on a certain branch goes unnoticed until the following week. Why it happens: the sense of finishing is strong and the observation periods feel like dead time. How to spot it: if you migrated order-sync the same day as inventory-update, this is it. How to fix it: phases 3 and 4 touch production and their value is in the waiting periods. Two weeks with most of the time being observation is the correct pace. Rushing here buys nothing and can cost a night.

Moving a secret without rotating it (practical, and it leaves the risk intact). What happens: a key written in a node field is found, the corresponding credential is created, it's deleted from the field, and it's considered resolved. But that key was in the workflow's JSON —which was probably versioned, exported, or shared at some point. Why it happens: moving the secret feels like the complete fix. How to spot it: if your phase 4 record has findings with "ROTATED: no," this is it. How to fix it: a secret that was outside a credential is considered exposed. The fix has two halves: moving it and rotating it. The first fixes the design; the second fixes the risk.

Revoking a key before confirming the new one works (practical, and it causes downtime). What happens: the old key is revoked the same moment the new one is created, and something in the migration wasn't right; order-sync stops working in the middle of operation. How to spot it: if your procedure doesn't have an observation period between "the new one works" and "I revoke the old one," this is it. How to fix it: new key → migrate → observe a full cycle → revoke. The intact old key during that period is your rollback, and it costs nothing.

Declaring the project finished with the network half pending (conceptual, and about the boundary). What happens: the six phases are run impeccably and port 5678 is still open to the internet with no reverse proxy. The instance with hardened credentials is exposed to the entire internet. Why it happens: this module is about the application and it's easy to forget the other half exists. How to spot it: check from outside whether your instance responds on 5678 and whether the editor is served over HTTPS. How to fix it: the Self-Hosting and Operations guide, module 4, is a prerequisite for this work, not an optional complement. Do it and note it in deliverable 5, where there's a line precisely for that.

Exercises

Exercise 1 — Plan the calendar. Terra Market gives you two weeks for this project, with the constraint that the workflows can't stop and that the ERP administrator only handles requests on Tuesdays. Write the calendar, day by day, indicating which phase advances and where the waits are.

See solution

A defensible calendar:

Week 1. Monday: phase 1 complete, and —key— prepare the request to the ERP administrator with the minimum scope, to hand it in tomorrow. Tuesday: hand in the request (the only day they handle them) and run phase 2 while it's being processed. Wednesday: phase 4, steps 1 and 2 —search for secrets in the JSON and in the history, fix and rotate. Thursday: phase 4, steps 3 and 4, plus revoking the orphans from phase 1. Friday: phase 5, steps 1 and 2 —access review, delete accounts, owner's 2FA.

Week 2. Monday: with the key already handed in, phase 3 steps 3 and 4: create erp_api_scoped and migrate only inventory-update. Observation begins. Tuesday: observation; meanwhile, phase 6 steps 1 to 3 —and the ERP administrator is available in case a 403 revealed a missing scope. Wednesday: phase 3 step 5, migrate order-sync and verify the orders in the ERP; meanwhile, phase 6 steps 4 to 6. Thursday: observation; meanwhile, phase 5 steps 3 to 5, with the stack restart at a low-activity time. Friday: revoke the old ERP key, verify, and go through the acceptance criteria.

The three decisions that make the calendar good:

  1. The ERP request is prepared on Monday and handed in on Tuesday, because the administrator only handles them that day. If you wait until you need it, you lose a whole week.
  2. The observations aren't dead time: each one overlaps with a phase that doesn't touch production. That way everything fits in two weeks without compressing any wait.
  3. The revocation goes at the end, when the two migrations have been stable for days.

Why it works: in a production project, the constraint that rules is almost never the technical work —it's the availability of third parties and the observation times. A calendar that identifies those two things first and arranges the work around them is the one that gets done.

Exercise 2 — Decide under pressure. You're on Wednesday of week 2. You migrated order-sync to erp_api_scoped this morning and the executions are green. At 16:00, operations reports that orders have been missing from the ERP since noon. What do you do, in what order, and what do you NOT do?

See solution

First: restore operation, don't diagnose. There are customer orders that aren't reaching the ERP, and every minute there are more. The rollback exists precisely for this and it's a one-minute one: change order-sync's nodes back to the old erp_api credential. It still exists and is still valid —that's why the procedure says the revocation goes at the end. What to expect: the orders start reaching again; confirm it in the ERP, not just in the n8n panel.

Second: recover what was lost. Identify the orders from noon to 16:00 that didn't arrive and reprocess them, with module 3's tools.

Third, now without pressure: diagnose. The main hypothesis is that there's an operation that the step 1 inventory didn't capture. Notice the detail that guides the search: the executions were green, so it probably wasn't a noisy 403 but a silent failure —an accepted response that didn't produce the effect. It's exactly the scenario that phase 3's step 5 warns about when it asks to verify the orders in the ERP and not just the execution's color.

Fourth: fix and retry the following Tuesday, when the ERP administrator can add the missing scope.

What you do NOT do:

  • You don't revoke the old key. It's your rollback and it's the only thing that let you resolve this in a minute.
  • You don't "fix" the problem by giving the new key administrator permissions. That undoes all of phase 3's work and, worse, leaves the sense that least privilege "didn't work." What was missing was a concrete operation; you add that one.
  • You don't diagnose before restoring. With customer orders not reaching the ERP, every minute of analysis has a business cost. Diagnose when the operation is running.

Why it works: the exercise tests whether you understood why the procedure has the shape it has. The three elements that save you —the intact old credential, observation as part of the plan, and verifying the effect and not just the state— are exactly the steps that on a calm day seem like excessive care. This is the day they were for.

Exercise 3 — Present the result. When you finish, Terra Market's director asks you to explain in five minutes what changed. They're not a technical person. Write what you'd say, without a task list: with the before and after in terms of business risk.

See solution

A possible version:

"I'll tell it with two questions: what's the worst that could happen to us before, and what's the worst that can happen to us now.

Before. The key our automations use to get into the ERP had administrator permissions: it could read, write, delete, and change the configuration of the whole system. And it had circulated over chat until it reached eight people, some of whom are no longer in the area. The worst that could happen to us was that someone with that key deleted or altered the ERP: orders, inventory, prices. On top of that we had an artificial intelligence key that charges per use, with no cap and no alert, so if it leaked we'd find out when the bill arrived, up to a month later.

Now. The ERP key can only do the four operations the automations actually run: create orders and read three types of information. It can't delete anything or change configuration. The worst that can happen to us is still bad —someone could read inventory and create fake orders— but it's reversible, and it's much less than losing the ERP. And the old key, the one that circulated over chat, is revoked. On the artificial intelligence one, it now has a ceiling and two warnings before reaching it, which we tested to make sure they reach someone who reads them: we went from finding out in a month to finding out in hours.

And what I most want to stick. Before, all these things depended on someone remembering. Now each key has a named owner, a rotation date, and there's an automatic monthly review that warns us if something gets out of order. That's what keeps this fine a year from now.

What's still pending and I want you to know. There are three things our current n8n plan doesn't include: separating by area who sees what, forcing the second factor on everyone, and automatically hiding sensitive data in the history. Today we cover them with agreements and with keeping few accounts, and it works with this team size. If we grow, it's worth revisiting, and I'll prepare the numbers whenever you want."

Why it works: the answer doesn't list tasks —the director can't evaluate them— but translates the work into worst case before versus worst case after, which is the unit risk is thought about in. It quantifies what's quantifiable (from a month to hours), says what changed structurally, and —most importantly— states what's still pending instead of selling the work as complete. That last part is what makes them believe the first.

Summary and next step

In this lesson you ran the project that closes the module. In six phases you turned Terra Market's messy keyring into a governed one: you built the inventory with the gap between the permissions each credential has and the ones it uses, ran the built-in audit, and assigned owners; you verified —comparing character by character— that the encryption key is backed up and that the credentials survive a restart; you reduced erp_api from administrator to its minimum scope with a new credential, migration of one workflow at a time, observation between steps, and revocation at the end —which incidentally made the key that eight people had in their chat useless; you got out the secrets written outside the credentials and the ones that passed through the data, rotating each because moving it isn't enough; you hardened the instance at the application level with the access review, the owner's 2FA, the surface trimming, and the automated audit; and you gave llm_token a ceiling, two tested alerts, keys separated by use, and a runbook whose first step is to revoke before investigating.

Before moving on you should be able to: go through your instance's acceptance criteria and answer yes with evidence; explain why the revocation goes at the end of each migration; justify why a secret found outside a credential is rotated in addition to moved; and present the result to someone non-technical in terms of worst case before and after.

Module 6 scales this instance. You'll move Terra Market to queue mode, where besides the main process several workers run that share out the executions —what's needed when 4,000 daily executions start to become 40,000. And there the piece you secured in phase 2 will reappear, with more weight: all the workers have to share the same encryption key, because each one needs to decrypt credentials to execute. The work you just did isn't a closed episode: it's the ground on which what comes next is built.

Resources

← Previous7. Protecting API keys and LLM tokensNext →1. Module introduction: when the workflow is the bottleneck