Customer balances were right. Stock was right. The cash drawer was right. Only the trial balance knew. It had no sales at all, while every purchase posted at once. On paper, the business bought and never sold.

FinyX-ERP is the Arabic ERP and point-of-sale system I design, build and sell alone. This article is the machinery that keeps it correct without a team, and the exact reason 372 passing tests could not see that bug. By the end you will have a save gate you can copy into any EF Core app, an audit trail that notices edits made straight on the database file, and a checklist for the bugs your tests are blind to by design.

A rule in a screen is a rule the next screen forgets

FinyX began as accounting software for Windows. Today the same repository holds a WPF till and back office, an ASP.NET Core server for each restaurant, a Next.js web store and three phone apps from one Expo codebase. One person cannot remember every rule in every one of those places, so the design must not depend on anyone remembering.

The desktop app and the restaurant server compile the same Domain, Application and Infrastructure source files. Every business service is registered in one method, AddFinyXBusinessServices(), and both hosts call it. A rule written once runs in both.

That settles where a rule is written. The harder question is where it is enforced. My answer: at the one place every change to the data has to pass through.

The gate every save walks through

FinyX treats each business event as one database transaction. An invoice, its stock movements and its journal entry commit together or not at all. Every save goes through the unit of work, and the unit of work refuses anything that would rewrite history:

public async Task<int> SaveChangesAsync(CancellationToken ct = default)
{
    EnsureAppendOnlyAccountingAndAuditRecords(); // journal entries and lines, payments, cash vouchers, audit: insert only
    EnsurePostedDocumentsAreImmutable();         // a posted invoice is corrected by a new document
    await EnsureNoClosedFinancialPeriodWritesAsync(ct);

    var audits = BusinessAuditCapture.Capture(_context, _operationId, _user.CurrentUserKey);

    await using var tx = await _context.Database.BeginTransactionAsync(ct);
    var changed = await _context.SaveChangesAsync(ct);

    var records = audits.Select(audit => audit.Materialize()).ToArray();
    AuditChain.Link(records, await AuditChain.ReadTailHashAsync(_context, ct));
    _context.BusinessAuditRecords.AddRange(records);
    await _context.SaveChangesAsync(ct);

    await tx.CommitAsync(ct);
    return changed;
}

This is shortened. The real method joins an outer transaction when one exists, and it tells open screens to refresh only after the commit, so no screen ever reads a write that might still roll back.

Three checks, three promises.

Append-only. Audit records, journal entries, journal lines, payments and cash vouchers are never modified or deleted through the gate. A mistake is corrected by a reversing entry, and both stay in the history.

Posted means fixed. A posted invoice, return or expense cannot change. You correct it with a new document.

Closed periods stay closed. The comment above that check explains why it lives here: "correctness must not depend on every current or future service remembering to call one." It reads both the original and the new date of a changed row, so a historical entry cannot be moved out of a closed period, or into one. It also locks everything dated before the end of a closed period, including a date before the first period or in a gap between two. The next opening balance is the sum of everything before it, so a row slipped into a gap changes a number someone has already signed.

When the gate was too strict

A gate can fail in the other direction, just as quietly. Archiving a sales invoice required it to be posted, and the posted-document check then refused the save because the invoice was posted. Archiving never succeeded once. The print counter failed the same way after every print of a posted invoice, so it stayed at zero and the invoice read "never printed".

The fix is a closed allow-list. Four filing fields may change on a posted invoice:

private static readonly string[] SaleInvoiceFilingProperties =
[
    nameof(SaleInvoice.IsArchived),
    nameof(SaleInvoice.ArchivedAtUtc),
    nameof(SaleInvoice.ArchivedBy),
    nameof(SaleInvoice.PrintCount)
];

If a single field outside that list changes in the same save, the whole save is refused. Anyone hoping to edit a posted amount finds no door here.

An audit trail that notices edits behind its back

My audit log used to promise one thing: it could not be edited from the app. That promise is weak. Anyone who opens the SQLite file with a database tool can delete a sale and its audit row together, and nothing is left to show it.

So every audit record now carries a SHA-256 hash of its own content plus the hash of the record before it. Change one character and its hash stops matching. Delete a record and the next one points at a hash that no longer exists.

public static string ComputeEntryHash(BusinessAuditRecord record, string previousHash)
{
    var payload = string.Join("\u001F",            // a separator no value contains
        record.OperationId.ToString("N"),
        record.OccurredAtUtc.ToString("yyyy-MM-ddTHH:mm:ss.fff", CultureInfo.InvariantCulture),
        record.UserKey, record.Action, record.EntityType, record.EntityKey,
        record.ChangedProperties, record.BeforeJson, record.AfterJson,
        record.ApplicationVersion, record.MachineName,
        previousHash);

    return Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(payload))).ToLowerInvariant();
}

Twelve lines. Four traps hid inside them.

Timestamps that come back different. FinyX runs on SQLite, SQL Server or PostgreSQL. SQLite hands a DateTime back with no Kind, and PostgreSQL rounds it to microseconds. Hash the full value and a record written on SQLite fails verification once the data moves to PostgreSQL. The hash takes milliseconds with no kind suffix, the one precision every provider returns intact.

The payload is a storage format. Field order and separator are frozen. Change either and every hash ever written becomes unverifiable, so treat any change there as a data migration.

Two writers, one tail. On a shared server, two tills can save at the same instant and both link to the same previous record. That is honest work. A verifier that demands one straight line would accuse the customer every day, and an alarm that rings every day gets switched off. So verification ignores order and checks three things: every hash matches its content, every previous hash points to a record that exists, and the chain has exactly one start.

The tail has no witness. Delete the last record and no later record points at it. The chain alone cannot see that, so the record count and the last hash are also kept outside the database. A log that got shorter shows up as a smaller count.

The order inside the gate matters too. The "before" values are captured before the first save, while EF Core still holds the original values. The "after" values and any new primary keys are read after it. The tail hash is read inside the same transaction, so the tail I link to is the tail that commits. Because both saves share that transaction, a business change never commits without its evidence.

The test does exactly what a tamperer would do. It writes to the database file with raw SQL, from outside the app:

// edit a record outside the app: detected
ExecuteOutsideTheSystem(db, "UPDATE BusinessAuditRecords SET UserKey = 'someone.else' WHERE Id = (SELECT MIN(Id) FROM BusinessAuditRecords);");
Require(!Verify(db).IsIntact, "An edited audit record went unnoticed.");

// put the value back: intact again, the check holds no grudge
ExecuteOutsideTheSystem(db, "UPDATE BusinessAuditRecords SET UserKey = 'audit.chain.tests' WHERE Id = (SELECT MIN(Id) FROM BusinessAuditRecords);");
Require(Verify(db).IsIntact, "Restoring the original value left the trail broken.");

// delete one from the middle: detected
ExecuteOutsideTheSystem(db, "DELETE FROM BusinessAuditRecords WHERE Id = (SELECT MIN(Id) + 1 FROM BusinessAuditRecords);");
Require(!Verify(db).IsIntact, "A deleted audit record went unnoticed.");

On PostgreSQL, triggers guard 11 tables as well, so a posted entry or an audit record stays fixed even for someone holding an admin tool. A test compares that list with the SQL Server list, so moving a customer to a server never weakens the protection.

Tests that open real databases

FinyX does not use xUnit. Its tests are one executable harness. Each test is a method named as an English sentence that states the promise it keeps, and the ones that touch data run against a real SQLite database in their own temporary folder. Each test prints a METRIC line with the numbers it proved, then PASS, FAIL or SKIP:

.\AccountsSystem.Tests.exe
METRIC TRIAL_BALANCE_ROWS=3;BALANCED=TRUE;SHORTAGE=100.00;CASH=900.00
PASS TrialBalance_BalancesAndNamesWhoBearsTheCashShortage
METRIC AUDIT_CHAIN_EDIT_DETECTED=PASS;AUDIT_CHAIN_DELETE_DETECTED=PASS;CLEAN_TRAIL_NOT_ACCUSED=PASS
PASS AuditTrail_DetectsAnEditedOrDeletedRecordMadeOutsideTheSystem

A METRIC line turns a green test into evidence. You can read what it proved without opening the code. SKIP exists for one reason: a test the environment cannot run is announced and counted, never reported as a failure, because a red that means nothing trains you to ignore red. The release script runs the whole desktop suite before it publishes anything, and stops if any test fails.

With all of that in place, the suite said 372 of 373. Here is what it could not see.

What 372 green tests could not see

The sales screen shows how many invoices are waiting to be posted, with a button that posts them. In the view model, that list required every invoice to be approved. Approval is a policy, SalesRequireApprovalBeforePosting, and it ships switched off. So with the default settings, no invoice ever reached that list, and the button had nothing to post.

Why did nobody notice? Because everything a cashier or an owner checks every day was right. Customer balances, stock and the cash drawer all include unposted documents on purpose, so they stay correct before the accountant posts. Only the reports that read the ledger itself, the trial balance and the income statement, were missing every sale.

Why did 372 tests not notice? This is the part worth stealing. The service already had the right function, GetUnpostedSaleInvoicesAsync. It was tested. And nothing called it. The screen built its own list from the rows it happened to be showing, with its own copy of the rule. The tests proved a function the product never ran.

The same mistake had a second symptom. Because the list came from the visible rows, it changed with the screen's date filter. With the range set to "today", yesterday's invoice could not be posted at all.

The rule moved into the service, which now reads the same policy as the posting itself, and the screen takes its list from the service:

// AccountingJournalService.GetUnpostedSaleInvoicesAsync
var requireApproval = (await _operationalPolicyService.GetAsync(ct))
    .SalesRequireApprovalBeforePosting;

if (requireApproval)
{
    query = query.Where(invoice => invoice.ApprovedBy != null && invoice.ApprovedBy != "");
}

What the screen offers for posting is now exactly what the service accepts.

Cancel must never touch the ledger

The same audit found that pressing Edit on a posted invoice removed its journal entry first and opened the editor second. Press Cancel and the invoice left the books: revenue, tax, cost and the receivable gone, the goods still out of stock, the invoice still in the list. No message.

Now the editor opens first. If it closes without confirmation, the ledger is never touched. The invoice is unposted only after confirmation, right before the change is applied, and any failure after that restores the posting. If even the restore fails, a window names the invoice and says its entry is missing from the ledger.

How do you test a decision that lives behind a modal dialog, in a suite with no screen? You test the order in the source:

var body = MethodBody(File.ReadAllText(salesViewModelPath), "private async Task EditSelectedInvoiceAsync()");
var dialogGuard = body.IndexOf("editor.ShowDialog() != true", StringComparison.Ordinal);
var unpost = body.IndexOf("UnpostSaleInvoiceAsync", StringComparison.Ordinal);

Require(dialogGuard >= 0, "The editor is no longer opened before the invoice is amended.");
Require(unpost > dialogGuard, "The ledger is stripped before the editor is confirmed.");
Require(body.Contains("RestorePostingAfterAbandonedEditAsync"),
    "Nothing restores the entry when the amend fails after the invoice was unposted.");

It is crude, and it measures the exact thing that broke.

The escape hatch that walked around the gate

Unposting an invoice has to remove its journal entry, which the gate forbids. So unposting and administrative deletion had a path of their own: raw SQL, with the database triggers suspended. That path never passed through the unit of work, so it never met the closed-period check. An entry from a month already closed could be erased.

The fix runs the period check before the connection opens and before any trigger is suspended. A request against a closed period is refused before a single row is touched.

Every path around a gate has to carry the gate's checks with it. The fastest way to find yours is to search the codebase for raw SQL.

The red test everyone learns to skip

The one failing test of the 373 failed on every normal run. It needed a database prepared beforehand by a separate script, so for anyone else it was always red. A permanent failure in an otherwise green list teaches its reader to skip it, and the first real failure gets skipped with it. The test now builds its own database in a fresh temporary folder and removes it afterwards. After the fixes the suite ran 376 of 376.

Less than a month later, the desktop suite has 475 tests.

The checklist I keep

• Put every invariant that protects money in the save path. Screens may warn early. The save path decides.
• A screen offers only what the service accepts. Read the list from the service, never from the rows on display.
• Make money records append-only, and correct them with reversing documents.
• In a period check, test the original date and the new one, and lock the gaps before a closed period's end.
• Write exceptions to an invariant as closed allow-lists of fields.
• Chain your audit records: fixed field order, a separator no value contains, a timestamp precision every database returns, verification that tolerates concurrent writers, and a witness for the tail outside the database.
• In tests, write to the database file from outside the app, and prove you notice.
• Search for raw SQL. Every path around the gate must repeat the gate's checks.
• Search for tested service methods that nothing calls.
• Allow no permanently red test. Skip loudly, and count the skips.
• Audit with one rule: reading code is not a test run, and a successful build proves nothing about the accounting.

If I started FinyX again tomorrow, I would write the save gate before the first screen. The audit's first findings were two rules that lived in a screen and one raw SQL path around the gate. Put the rule where the data changes, then write the test that proves the screen uses it.

It is the same discipline I bring to the business systems I build for clients.

It was an Xprinter XP-370B on a till in real use, loaded with 76 mm roll, and it fed paper until the roll ran out. Measured on the bench without printing, the job asked for 124.7 mm. The job was bounded. The printer was not.

The cause was five bytes. Every row of the receipt image was 72 bytes wide, on a head that prints 67.

FinyX-ERP prints every receipt in Arabic, with a QR code at the bottom, on 203 DPI printers that cost little and forgive nothing. It never sends the printer a single Arabic character. By the end of this article you will be able to print clean Arabic on any ESC/POS printer, with a hard ceiling on paper, and you will know exactly how five bytes turn pixels into commands.

Why the printer cannot write Arabic

A receipt printer's text mode was built for Latin script: fonts in ROM, and code pages that map one byte to one glyph. Arabic asks for more. Each letter takes up to four shapes depending on its neighbours. Lam and alef fuse into one glyph. A line runs right to left while the numbers inside it run left to right. I wrote about those joining rules in why per-letter text effects break Arabic. Text mode does none of it.

In practice, firmware code pages disconnect the letters, reverse the run, or substitute glyphs that have nothing to do with Arabic, and two units of the same model can behave differently. I cannot support behaviour I cannot predict.

So in FinyX-ERP, Arabic never reaches a printer as text. The computer shapes it with a real text engine, the whole receipt becomes a black-and-white picture, and the printer only ever receives raster images. Barcode labels follow the same rule: on the TSPL label printers, the Arabic product name goes as a bitmap too.

Pictures solve Arabic completely. They also carry the risk at the heart of this story, because to the printer a picture is just more bytes in the command stream.

Five steps from a sale to burned dots

QuestPDF composes the receipt right to left on a continuous page with no fixed height, exactly as wide as the print area. The page is rendered at 203 DPI, so one pixel is one printer dot, and from there it is bits all the way down.

The encoder is short. Three lines in it do the real work:

// Rec. 601 luma, in integer form.
var luminance = (77 * red + 150 * green + 29 * blue) >> 8;

// Composite transparent pixels onto white first. Without it, an unpainted
// background reads as black and the receipt prints as a black rectangle.
luminance = (luminance * alpha + 255 * (255 - alpha)) / 255;

if (luminance < InkThreshold)   // 180
    raster[y * rowBytes + x / 8] |= (byte)(0x80 >> (x % 8));

Each dot is one bit, and the leftmost dot of a byte is its highest bit, so dot x lands in byte x / 8 under the mask 0x80 >> (x % 8). A row of 536 dots is 67 bytes. Hold on to that number.

The bands go out with GS v 0, 128 rows at a time. That command advances the paper by exactly the image height, so bands meet with no seam; a line feed between them would open a white stripe. Below the last inked row, everything except a margin of about 2 mm is dropped, so no hand's length of blank paper follows the receipt.

The spooler call is plain winspool with data type RAW, in 64 KB chunks. The Windows driver installed for this printer is a label driver, and RAW keeps it from turning ESC/POS into its own TSPL. If anything fails halfway, the code calls AbortPrinter and never retries: an ambiguous partial job sent twice can duplicate a receipt or empty the roll.

Seven commands, every byte

A receipt needs seven commands at most. Here they are with the exact bytes for 76 mm paper on the XP-370B. Two-byte numbers are little-endian.

Look at the GS W row. 536 is the most important number in this article, and I did not choose it.

The paper is wider than the print

At 203 DPI a millimetre is about eight dots, and only at 203 DPI. So millimetres become dots in exactly one function, against the printer profile's own resolution, rounding half away from zero so that a barcode module never shrinks.

The first two are the generic profiles. The XP-370B numbers were measured: the paper is 607 dots wide, the head marks 536 of them, and the first 24 are mechanically out of reach. Using the full width clipped the right-hand column. On an Arabic receipt the right edge is where every line starts, which makes it the worst edge to lose.

The settings offer five print areas (384, 512, 536, 576 and 608 dots), and the default is the conservative 536. A narrower area always prints. A wider one is what breaks things, and soon you will see how badly.

The biggest quality gain came from one decision. My first receipts were composed 80 mm wide and scaled down to 536 dots by the encoder, a factor of 0.84. Scaling a 1-bit image is what makes Arabic look ragged: a shrunken stroke lands between dots, and the threshold has to guess. Now the page is composed at the print area itself:

// 536 dots at 203 DPI = 67.07 mm. One PDF pixel becomes one printed dot.
internal static float PageWidthMm(DocumentPrintSettings settings) =>
    (float)(settings.ReceiptPrintAreaDots * 25.4d / RasterDpi);

That 536 came from a roll on the floor, and we will get there. First, the Arabic itself needs rules, because one bit per dot is a harsh medium.

Making Arabic survive one bit

A 203 DPI head has no grey. Each dot burns or it does not, and a stroke thinner than one dot simply does not print. Four rules follow from that.

Choose the font by test. I rendered the same invoice in Ping AR + LT and in Cairo at real size, through the same threshold, and compared. Ping is narrower and more regular, so a line fits a longer item name, and its stems stay a full dot wide at 7.5 pt. Cairo starts to crumble at those sizes. Ping became the thermal default.

Medium weight for body text. At 8 pt, the joins of a regular weight drawn for backlit screens fall below one dot and break, which reads as a ragged letter.

A floor of 6.5 pt. That is about 18 dots tall. Below it, the dots that tell ب from ت from ث start merging into the letter.

No colour and no grey. Everything collapses to black or white at the threshold, so a light shaded band prints either nothing or a black block over its own text. The document title is white text reversed out of a solid black band.

Body text is 8 pt, labels 7.5 pt, the grand total 13 pt. Sizes that small work only because nothing is scaled on the way to the head.

Digits and direction

With an Arabic culture, the text stack swaps in Arabic-Indic digits, so 76,400.00 prints as ٧٦٬٤٠٠٫٠٠. Readable, but a customer compares the price with a shelf label and a supplier reads barcode numbers back, and both want Western digits. So Western digits are the default and Arabic-Indic is a setting. In the WPF renderer that draws labels, the switch is one object:

_numbers = new NumberSubstitution(
    NumberCultureSource.Override,
    CultureInfo.GetCultureInfo(arabicIndicDigits ? "ar-SA" : "en-US"),
    arabicIndicDigits ? NumberSubstitutionMethod.NativeNational : NumberSubstitutionMethod.European);

A date with a time, such as 2026/09/08 14:35, holds two Latin runs. Inside an Arabic paragraph they are laid out right to left, so the time reaches the paper before the date. Values like that go inside a left-to-right isolate:

// U+2066 LEFT-TO-RIGHT ISOLATE … U+2069 POP DIRECTIONAL ISOLATE
internal static string Ltr(string value) =>
    string.IsNullOrEmpty(value) ? value : $"⁦{value}⁩";

An isolate fixes the character order and leaves placement to the right-to-left flow. A left-to-right container would also push the value to the left edge of its cell.

Never put Arabic text inside the isolate: in there, the Yemeni rial symbol «ر.ي» reverses its letters. So currency prints as its ISO code (YER, SAR), once, on the grand total. Two bugs from this family reached paper. A payment line built without the isolate printed YER 75,000.00 reversed, and a negative balance printed as 50,000.00- because the minus sign sat outside the isolate. Both now go through one formatter.

Shaped, sized and ordered, the receipts finally looked right. Then a till in real use printed one and kept going.

The roll on the floor

The first theory looked solid. The driver advertised 38.1 mm media while the receipt was 76 mm wide, so the printer seemed stuck in label mode, hunting for a gap that continuous paper does not have.

I disproved it with a PowerShell script. It sent raw ESC/POS to the same printer with the same settings, printed one clean copy, and stopped. A RAW job bypasses the driver completely, so the driver's media size could not be the cause. The difference had to be in the bytes.

So I measured the job without sending it. A diagnostic runs the real composer and encoder and writes out the command stream instead of paper:

QuestPDF: 1 page, 639 × 1037 px = 129.8 mm
ESC @ · ESC a 1 · GS L 0 · GS W 576 · 8 × GS v 0 (72 bytes × 128 rows) · ESC d 3
paper requested: 124.7 mm

Twelve centimetres, and not one stray byte. What differed from the working script was the geometry:

There are the five bytes. At offset zero, 576 dots is wider than this head prints on 76 mm paper. Given raster rows wider than its print area, the firmware resynchronises mid-image, and from then on it reads image bytes as commands. Raster data is full of bytes that mean "feed". That is how a 12 cm receipt becomes a roll on the floor.

ESC d made it worse. It feeds in lines, and some label-capable firmware reads it as "advance to the next form", which continuous paper never has.

The fix had three parts:

• Every raster row is exactly as wide as GS W declares, rowBytes = printAreaDots / 8, and anything beyond is cropped. Printer and encoder can never disagree on the byte count, and a test asserts rowBytes × 8 == width exactly.
• ESC J in dots replaces ESC d, split into steps of at most 255 because it takes a single byte.
• ESC a 1 is gone. The raster fills the whole area, so alignment only added another way for the two sides to disagree.

After the fix, the stream matches the working script command for command:

ESC @ · GS L 24 · GS W 536 · 7 × GS v 0 (67 bytes × 128 rows) · ESC J 216
paper requested: 134.9 mm

Then I made sure the next layout fault would end as an error message. A receipt may use at most 800 mm of paper, feed included, checked before the first byte leaves; the old ceiling was 40,000 rows, about five metres. A button in the printing settings empties the printer's queue, and a script does the same without the app:

# Pause the queue so nothing new starts, then cancel what is already in it.
Add-Type -AssemblyName System.Printing, ReachFramework
$server = New-Object System.Printing.LocalPrintServer
$queue  = $server.GetPrintQueues() | Where-Object FullName -eq 'Xprinter XP-370B'
$queue.Pause(); $queue.Commit()
foreach ($job in $queue.GetPrintJobInfoCollection()) { $job.Cancel(); $job.Dispose() }

Know its limit. Paper already moving comes from the printer's own buffer and stops when that drains. If it keeps going after that, the printer is feeding on its own, and only the power switch or the media setting will stop it.

The same error log held a second way to waste a roll. Process.Start with the print verb sends a PDF to the Windows default printer. On a till where that is the thermal printer, an A4 page goes through the thermal driver onto a roll with no page length. Now the PDF opens, and a person picks the printer.

The QR, the feed, the cut and the drawer

The QR. The QR is drawn as SVG: one square per module, shape-rendering="crispEdges", no smoothing. A raster QR that gets resized has modules that bleed into each other, and scanners give up. A vector is drawn straight onto the printer's own grid.

var matrix = new QRCodeWriter().encode(content, BarcodeFormat.QR_CODE, 0, 0,
    new Dictionary<EncodeHintType, object>
    {
        [EncodeHintType.ERROR_CORRECTION] = ErrorCorrectionLevel.M,
        [EncodeHintType.MARGIN] = 0,   // the quiet zone is added by hand: 4 modules
    });
// then one "M x,y h1v1h-1z" square per dark module

Error correction is level M. The receipt's QR carries about 550 characters, level H would make it too big to print at a readable size, and M survives an ordinary scratch or ink spot. It prints 38 × 38 mm, centred between the totals and the footer.

The feed. After the last line the paper advances 27 mm, so the footer clears the tear bar. ESC J counts motion units, one dot on these printers, so 27 mm is 216 dots.

The cut. Only when the printer has a cutter, because a cut sent to one without a cutter jams it or prints as characters. The XP-370B's driver offers None, Tear Off and Peel Off, and no Cut, so the cashier tears. Elsewhere, GS V makes one partial cut after the whole receipt. The live path still recognises the XP-370B by name; the profile-driven encoder I am moving to reads the cutter from the printer's profile.

The drawer. ESC p pulses the cash drawer. It lives in the profile-driven encoder and is off by default, because a pulse on a printer wired to something else is at best a click.

Before you print a single receipt

• Treat Arabic as pixels from the first day.
• Measure the print area on the real printer, keep it per printer, and declare it in every job with GS L and GS W.
• Make every raster row exactly printAreaDots / 8 bytes, and crop anything wider.
• Compose at the final width. Never scale a 1-bit image.
• Composite transparency onto white before the threshold.
• Feed in dots with ESC J, and cap the paper before sending anything.
• Preview the bits the head will burn. My preview tool runs pages through the real encoder and saves PNGs; it caught wrapping, reversed text and extra length before they cost paper.
• Never retry a print automatically.

Once Arabic became pixels, every problem left was geometry: how wide to print, how far to feed, where to stop. Five bytes of it cost a whole roll.

No balance moved by a single rial. The database refused every one of those inserts.

And every one of those requests passed through an idempotency layer, built for exactly this: phones that resend.

This is how FinyxWatt bills electricity customers from phones that lose the network. It covers the id that follows a payment from the phone to the receipt, the reason bill totals must never be added, the outbox that decides what to retry, and the rule I was missing that night. By the end you will be able to build an offline queue for money where a retry can never charge twice and a refusal can never loop.

Where the network drops, the phone keeps the work

A private electricity station runs on a chain of small steps. A collector reads the meter, a bill is issued, the customer gets an SMS, someone collects the money, and the cash moves up to the manager. FinyxWatt is the platform I built for that chain: four Flutter apps on one NestJS API over PostgreSQL. The station app is the one in the collectors' hands, and stations lose the network often.

So the app never waits for the network. It shows saved copies first, and every write goes into an outbox on the phone. When the connection returns, one coordinator uploads in dependency order: new customers first, then the bills and payments that may refer to them, then chat. A payment for a bill that so far exists only on the phone waits its turn. After a failed run the coordinator waits 30 seconds, doubling up to five minutes, so a struggling server is not hammered.

The first version ran on Firebase. Moving it to my own backend while stations kept working left me with one rule: the phone proposes, the server decides.

Retries are the whole design, which makes two questions unavoidable. If a retry is normal, what stops a retry from charging twice? And what stops a retry from never ending? The first has a clean answer. The second is the night of 18 October.

Give the command its id at birth

Every command gets a random UUID the moment it is first saved on the phone, and keeps it across retries, app restarts and lost responses. The id is also the identity of what it creates: a queued bill is created on the server under that id, and so is a payment.

/// A new random (version 4) id, created when a command is first queued.
String newOperationId() {
  final random = Random.secure();
  final bytes = List<int>.generate(16, (_) => random.nextInt(256));
  bytes[6] = (bytes[6] & 0x0f) | 0x40;
  bytes[8] = (bytes[8] & 0x3f) | 0x80;
  return _format(bytes);
}

/// The id stored with a queued command, or a derived one for older items.
String operationIdFor(Map<String, dynamic> payload, String queueItemId) {
  final stored = payload['operationId'];
  if (stored is String && isCanonicalUuid(stored)) return stored.toLowerCase();
  return derivedOperationId(queueItemId);
}

The second function handles the trap that shows up after an upgrade. A new version of the app can land on a phone that still holds commands queued by an older version, which stored no id. Generate a fresh id on every retry and you have built the double charge you set out to prevent. So an old item gets an id derived from its own permanent queue id. It is the same value on every retry, and it is marked as UUID version 8 so anyone reading it knows it was derived.

A second trap hides inside that derivation. It uses 32-bit FNV-1a, and the multiplication by the FNV prime is spelled out as shifts:

hash = (hash + (hash << 1) + (hash << 4) + (hash << 7) + (hash << 8) + (hash << 24)) & 0xffffffff;

The same Dart code also builds for the browser, where integers are doubles. A plain multiplication there can pass 2^53 and lose its low bits, and the browser would derive a different id from the same queue item than the phone does.

Claim the id before doing the work

On the server the id arrives as the Idempotency-Key header, and the API claims it before it touches any money. The claim and the command share one transaction:

const claimed = await trx.insertInto('idempotency_receipts').values({
  tenant_id, actor_identity_id, idempotency_key: key, operation,
  request_hash: sha256(stableStringify(request)),
  status: 'processing',
}).onConflict((c) => c.columns(['tenant_id', 'actor_identity_id', 'idempotency_key']).doNothing())
  .returning('idempotency_key').executeTakeFirst()

if (claimed === undefined) {
  const existing = await readReceiptForUpdate(trx, key) // SELECT … FOR UPDATE
  if (existing === undefined) throw conflict('IDEMPOTENCY_RECEIPT_UNAVAILABLE') // fail closed
  if (existing.operation !== operation || existing.request_hash !== requestHash) throw conflict('IDEMPOTENCY_KEY_REUSED')
  if (existing.status !== 'completed') throw conflict('OPERATION_IN_PROGRESS')
  return { replayed: true, status: existing.response_status, body: existing.response_body }
}

const result = await execute(trx)       // first time: do the work
await markCompleted(trx, key, result)   // and store its response in the same transaction
return { replayed: false, ...result }

Each line closes a door.

• Two copies of the same request arriving together cannot both run. PostgreSQL makes the second ON CONFLICT wait for the first transaction, and the loser reads the finished receipt.
• A receipt that exists but cannot be read fails closed. Running the command a second time is the one thing this code must never do.
• A known key sent with a different body is a client bug, and it gets a 409 instead of a replay. The body is hashed after a stringify that sorts object keys, so the same command always hashes the same.
• A key still in processing answers OPERATION_IN_PROGRESS, and the phone files that under "try again later".

One trap here took a migration to fix. My first receipts table was keyed by tenant, actor, operation and key. It looked tidy, and it allowed one key to be used once per endpoint. If a retry were ever routed to the wrong endpoint, the same key could create a second effect. Migration 0015 made each key single-use per actor across every command type.

A lost response is now harmless. Hold on to one detail from that code, though: the response is stored in the same transaction as the work. It matters later.

The server does the arithmetic

An idempotent request is only safe if it carries no money the phone made up. The phone sends a meter reading. The server does the rest.

A bill's charge is (current reading − previous reading) × price per kWh, rounded to two decimals in PostgreSQL numeric. The previous reading comes from the meter row, which the server locks for the length of the transaction. The price is the customer's own while it is within its dates, otherwise the station's. A price sent by a phone counts only for a station that has never stored one, so a phone that has been offline for weeks cannot bill at an old price.

Side effects need the same discipline. The customer's SMS is created in the same transaction as the bill, under the key invoice:<id>:issued:v1, with a unique index on source, template and recipient. A replayed bill cannot queue a second message.

Never add bill totals

Every amount a customer owes lives in one table, ledger_entries. Bill charges and payment reversals are debits. Payments and voids are credits. The balance is the sum. The API's database role has only SELECT and INSERT on that table, and a trigger refuses the rest:

CREATE TRIGGER ledger_entries_append_only
BEFORE UPDATE OR DELETE ON ledger_entries
FOR EACH ROW EXECUTE FUNCTION app_private.prevent_ledger_mutation();
-- raises: 'ledger entries are append-only; post a reversal entry instead'

That design still let one bug through, and an accounting audit caught it. A bill's total includes the customer's whole balance as arrears, which is what the customer needs to see on the bill. Yet every older unpaid bill stayed payable, and each payment was checked against its own bill only.

Bill A: 1,000, unpaid. Bill B: 500 + 1,000 arrears = 1,500. A payment of 1,000 on A and a payment of 1,500 on B were both accepted, for a real debt of 1,500. Two meters, or a voided bill, opened the same hole.

Migration 0080 changed three rules. Issuing a bill carries every open bill of the customer into it, and a carried bill takes no payment. A payment is capped at the bill's remaining amount and at the customer's ledger balance. After any money change, the database sets every bill's status from the ledger, oldest debt first, so paid or unpaid is never worked out from paid_amount against total_amount.

The same mistake was hiding in a dashboard. The website's dashboard added up a page of 250 rows to show what was billed and owed. Now the database sums every row the member is allowed to see, and the dashboard reads that one number.

Your status codes are your retry policy

Back to the outbox. When an upload fails, the phone must decide what the failure means. It sorts every error into four kinds:

Read the 500 in that table once more. Now the night of 18 October.

The receipt carried a payment provider's transaction number, the reference of a transfer. That number was already on another live receipt from the same provider. A unique index, payments_provider_reference_unique, refused the insert, exactly as designed. The API had no mapping for that error, so it surfaced as a 500. The phones read a 500 as a passing failure, and sent the receipt again.

Now the question from the opening. Why did the idempotency layer not stop it?

Because a key replays successes. The claim lives in the same transaction as the work, and that is exactly what makes it safe: the work and its stored response commit together or not at all. When the insert failed, PostgreSQL aborted the transaction and the claim vanished with it. Nothing remembered the failure, so every resend was a first attempt that failed the same way. Storing failures would be worse, because a failure that is only passing, a deadlock or a restart, would then be replayed forever.

The key did its job. The status code was the bug. The fix is one function: that unique violation becomes a 409, which the station app and the website explain in plain Arabic. On the phone a 409 is blocked, shown for review and never resent.

function duplicateReference(error: unknown): never {
  const failure = error as { code?: unknown; constraint?: unknown } | null
  if (failure?.code === '23505' && failure.constraint === 'payments_provider_reference_unique') {
    throw new ConflictException({ code: 'PAYMENT_REFERENCE_DUPLICATE' })
  }
  throw error
}

The commit that fixed it is timestamped 22:29 the same night. Its integration test sends the same command twice with the same key and expects the same 409 both times. It also checks that a different provider may carry the same number.

Late writes land in the open period

Offline phones raise one more question. What if a phone syncs a receipt dated inside a period the station has already closed? Closing a period writes, for every customer, a closing entry and an opening entry that sum to zero, and the closed period has to keep summing to zero. So a trigger moves a late entry to the start of the open period:

CREATE FUNCTION app_private.ledger_entry_into_open_period() RETURNS trigger AS $$
DECLARE open_from timestamptz;
BEGIN
  PERFORM pg_advisory_xact_lock_shared(hashtextextended('archive:' || NEW.tenant_id::text, 0));
  open_from := app_private.archive_boundary(NEW.tenant_id);
  IF NEW.effective_at < open_from AND NEW.entry_kind::text <> 'period_close' THEN
    NEW.effective_at := open_from;
  END IF;
  RETURN NEW;
END $$ LANGUAGE plpgsql;

Closing takes the same advisory lock exclusively, per station. A close waits for the writes in progress, and new writes wait for the close.

The checklist for offline money

• Give each command its id when it is first saved on the device, and keep it until the server confirms.
• Give commands queued before you had ids a derived, stable id. Never a fresh one per retry.
• Make that id the identity of what the command creates: the bill's id, the payment's id.
• Claim the key before the work, in the same transaction, and store the response with it.
• Make a key single-use per actor across every endpoint, and reject it when the body differs.
• Fail closed when a receipt exists and cannot be read.
• Let the phone send facts, such as a meter reading. Let the server compute every amount.
• Give side effects their own keys, like invoice:<id>:issued:v1.
• Keep balances in an append-only ledger, and never add bill totals.
• Map every unique constraint to a 4xx. Your status codes are your retry policy.
• Move late entries into the open period and leave the closed one closed.

The network will keep dropping, and phones will keep resending. The balance has to be right anyway, and a refusal has to stay a refusal.

This is how I build mobile apps that have to work offline.

So nothing depends on me remembering it.

This is the field manual for how FinyxWatt and the server behind this site are protected: six layers, each built on the assumption that the layer above it has already failed. Every config comes from the real files, trimmed, without secrets or hostnames. By the end you can make PostgreSQL refuse a read your code forgot to filter, run containers that can barely do anything, give a phone an identity it cannot fake, and prove every week that your backup restores.

Three traps wait along the way: a PostgreSQL setting that comes back as an empty string, a header anyone could forge if one container joined the wrong network, and a retry default that lets a broken backup hold its own test hostage for hours.

Layer 1: rows that refuse the wrong station

The usual tenant boundary is a WHERE clause repeated in every query. It holds until one query, written in a hurry, leaves it out, and the bug is silent: the page works, it just shows too much.

FinyxWatt puts the boundary inside the database. Row-level security is enabled and forced on 88 of the 94 tables in its migrations. The six without it hold the permission catalogue, schema metadata and the bookkeeping of a one-time import. Every policy checks the row's station and that the user is an active member of it:

ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
ALTER TABLE invoices FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_scope_api ON invoices FOR ALL TO platform_api
  USING (
    tenant_id = app_private.current_tenant_id()
    AND EXISTS (
      SELECT 1 FROM tenant_memberships m
      WHERE m.tenant_id   = app_private.current_tenant_id()
        AND m.identity_id = app_private.current_identity_id()
        AND m.status = 'active'
    )
  )
  WITH CHECK ( /* the same condition */ );

The migration that added the membership check opens with its reason: "a future coding error must not turn a verified identity from tenant A into tenant B merely by setting a wrong transaction setting." A wrong station id finds no active membership, and the query returns nothing.

The API sets both values at the start of every transaction, from the verified token:

async withTenant<T>(tenantId: string, identityId: string, callback: (trx: Transaction<Database>) => Promise<T>) {
  return this.db.transaction().execute(async (trx) => {
    await sql`select set_config('app.tenant_id', ${tenantId}, true)`.execute(trx)
    await sql`select set_config('app.identity_id', ${identityId}, true)`.execute(trx)
    return callback(trx)
  })
}

Three details decide whether this works.

The final true makes the setting local to the transaction. A plain SET stays on the connection, and the pool hands it, still pointing at station A, to the next request.

The empty string is the first trap. On a new connection, a setting that was never defined reads as NULL. On a pooled connection that has already served a request, it reads as '', and ''::uuid throws. The helper every policy calls handles both:

CREATE FUNCTION app_private.current_tenant_id() RETURNS uuid
LANGUAGE sql STABLE PARALLEL SAFE AS $function$
  SELECT nullif(current_setting('app.tenant_id', true), '')::uuid;
$function$;

NULL equals nothing, so a request without context sees zero rows. It fails closed.

FORCE matters because a table's owner skips its policies by default. Superusers and BYPASSRLS roles skip them always. That is layer 2's job.

CI applies every migration to a fresh PostgreSQL and runs adversarial SQL tests as the API's own role. The core of one, ready to copy:

SET ROLE platform_api;
SET LOCAL app.tenant_id   = '<station A>';
SET LOCAL app.identity_id = '<an active member of station A>';

DO $test$
BEGIN
  IF (SELECT count(*) FROM customers) <> 1 THEN
    RAISE EXCEPTION 'RLS expected one station-A customer';
  END IF;

  BEGIN
    INSERT INTO customers (tenant_id, subscriber_number, display_name)
    VALUES ('<station B>', 'ILLEGAL', 'Cross-tenant write');
    RAISE EXCEPTION 'RLS accepted a cross-tenant write';
  EXCEPTION WHEN insufficient_privilege THEN NULL;
  END;
END $test$;

All of this assumes the API connects as a role the policies apply to.

Layer 2: logins that cannot switch it off

Connect as the owner without FORCE, as a superuser or as a BYPASSRLS role, and every policy above is decoration.

FinyxWatt's database creates its logins on first start, "before the API image is ever allowed to connect", in the words of the script. Trimmed; the real one takes names from the environment and passwords from secret files:

CREATE ROLE migrator  LOGIN NOSUPERUSER NOCREATEDB CREATEROLE   NOREPLICATION NOBYPASSRLS;
CREATE ROLE api_login LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS;

ALTER DATABASE app OWNER TO migrator;
REVOKE CONNECT ON DATABASE app FROM PUBLIC;
GRANT  CONNECT ON DATABASE app TO migrator, api_login;

GRANT platform_api TO api_login;   -- the role every policy is written for

Only the migration login owns the schema. No runtime login holds a migration password, and the API runs with RUN_MIGRATIONS: "false". Five runtime roles split the work:

The ledger and the audit log refuse changes twice:

GRANT SELECT, INSERT ON ledger_entries TO platform_api;   -- no UPDATE, no DELETE

CREATE FUNCTION app_private.prevent_ledger_mutation() RETURNS trigger
LANGUAGE plpgsql SECURITY INVOKER SET search_path = pg_catalog, public AS $function$
BEGIN
  RAISE EXCEPTION 'ledger entries are append-only; post a reversal entry instead'
    USING ERRCODE = 'integrity_constraint_violation';
END;
$function$;

CREATE TRIGGER ledger_entries_append_only
BEFORE UPDATE OR DELETE ON ledger_entries
FOR EACH ROW EXECUTE FUNCTION app_private.prevent_ledger_mutation();

A later migration can widen a grant by accident. The trigger still refuses, and its message tells the next developer what to do instead.

-- Logins that skip row-level security
SELECT rolname FROM pg_roles
WHERE rolcanlogin AND (rolsuper OR rolbypassrls);

-- Tables without enabled AND forced row-level security
SELECT relname FROM pg_class
WHERE relkind = 'r' AND relnamespace = 'public'::regnamespace
  AND NOT (relrowsecurity AND relforcerowsecurity);

Roles decide what a login may do. Which station it acts for comes from a token.

Layer 3: an API that believes only what it verified

FinyxWatt signs people in with Keycloak. The API checks every token's signature, issuer, audience and expiry against Keycloak's published keys:

this.jwks ??= createRemoteJWKSet(new URL(this.config.keycloakJwksUri))
const { payload } = await jwtVerify(token, this.jwks, {
  issuer: this.config.keycloakIssuer,
  audience: this.config.keycloakAudience,
})

Then it reads the verified azp claim to learn which client obtained the token. The comment above that check is the whole lesson of this layer: "Never accept a client-type header: a caller could forge one to route an Android bearer around a cookie-only endpoint."

The station follows the same rule. The API accepts only memberships of the verified identity, then reads that membership again under row-level security. Sessions are short and single-use:

"accessTokenLifespan": 300,
"revokeRefreshToken": true,
"refreshTokenMaxReuse": 0,
"bruteForceProtected": true,
"failureFactor": 5

Access tokens live five minutes. A refresh token works once, so a copied one is spent the moment either side uses it. Five wrong passwords make the account wait.

So far the attacker has been a bug in my code. The next layers assume the attacker is already inside a container.

Layer 4: containers that can barely do anything

This site runs in one Caddy container, defined like this:

web:
  image: caddy:2.10.2-alpine
  ports: ['80:80', '443:443', '443:443/udp']
  volumes:
    - ./Caddyfile:/etc/caddy/Caddyfile:ro
    - /srv/moghalis:/srv/moghalis:ro
    - caddy_data:/data
    - caddy_config:/config
  read_only: true
  tmpfs: [/tmp]
  cap_drop: [ALL]
  cap_add: [NET_BIND_SERVICE]
  security_opt: [no-new-privileges:true]
  mem_limit: 128m

Nothing can be written outside /tmp and the certificate volumes, every Linux capability is gone except binding low ports, no process can gain new privileges, and memory stops at 128 MB.

FinyxWatt runs twenty services, each with no-new-privileges; its Caddy is read-only with every capability dropped. The stronger protection is the network map:

networks:
  edge:
  application:   { internal: true }
  database:      { internal: true }
  observability: { internal: true }
  sms_gateway:   { internal: true }
  egress:

An internal network has no route out, and each service joins only what it needs:

The front door has no route to the database, and the services in the last three rows have no route to the internet.

FinyxWatt's production stack needs 28 secrets. Each is a separate file with its own random value, mounted as a Docker secret only into the containers that need it:

api:
  environment:
    DB_USER: ${PLATFORM_API_USER}
    DB_PASSWORD_FILE: /run/secrets/platform_api_password
  secrets:
    - platform_api_password

The secrets guide puts the rule in one sentence: "Do not put secrets in command-line arguments, .env files, shell history, container images, or source control." An environment variable shows up in docker inspect; a file under /run/secrets exists only where it was mounted.

Layer 5: a front door that answers less

Caddy on this site's server renews HTTPS certificates and adds these headers to every response:

header {
	Strict-Transport-Security "max-age=31536000; includeSubDomains"
	X-Content-Type-Options "nosniff"
	X-Frame-Options "SAMEORIGIN"
	Referrer-Policy "strict-origin-when-cross-origin"
	Permissions-Policy "camera=(), microphone=(), geolocation=(), payment=()"
	-Server
}

FinyxWatt's front door also closes what it does not serve:

{
	admin off
}

(restricted_console) {
	@outside_trusted_networks not remote_ip {$ADMIN_TRUSTED_CIDRS}
	respond @outside_trusted_networks 403
	basic_auth {
		{$OBSERVABILITY_USER} {$OBSERVABILITY_PASSWORD_HASH}
	}
}

admin off turns off Caddy's own admin API. Monitoring consoles answer only from trusted networks, then ask for a password the server holds only as a bcrypt hash. /metrics returns 404 from the internet, and the public phone-enrolment route accepts 32 KB at most, because it never receives a file. Each security header comes from one place: the API sets X-Frame-Options itself, so Caddy's block for the API leaves it out.

The front door's most interesting job is checking a passport for every phone that sends bills.

A passport for every phone

FinyxWatt sends each customer's bill by SMS from dedicated Android phones at the station. A phone that sends messages in a station's name needs more than a password, so it gets a client certificate.

The database keeps only an HMAC of the code, made with a secret pepper, and the schema enforces the limits itself:

-- HMAC-SHA-256(code-id || secret), never the raw enrollment code.
code_digest  bytea NOT NULL CHECK (octet_length(code_digest) = 32),
expires_at   timestamptz NOT NULL,
max_attempts smallint NOT NULL DEFAULT 5 CHECK (max_attempts BETWEEN 1 AND 10),
attempts     smallint NOT NULL DEFAULT 0 CHECK (attempts BETWEEN 0 AND 10),
CONSTRAINT sms_device_enrollment_codes_attempts_bound CHECK (attempts <= max_attempts),
CONSTRAINT sms_device_enrollment_codes_expiry CHECK (expires_at > created_at)

Approval goes through the main API, which refuses the code's creator with SMS_ENROLLMENT_SELF_APPROVAL_FORBIDDEN and binds a certificate from a private authority whose public key matches the key the phone proved it holds. From then on:

tls {
	client_auth {
		mode require_and_verify
		trust_pool file /run/secrets/sms_device_ca_pem
	}
}
reverse_proxy sms-gateway:8081 {
	header_up X-Verified-Client-Certificate-Fingerprint {tls_client_fingerprint}
}

Now the second trap. The gateway believes that fingerprint header without question, and that is safe for one reason: it sits on an internal network that only Caddy joins, and Caddy overwrites the header after verifying the certificate. The gateway's own code spells out how to break it: "Do not expose the gateway container port or add arbitrary containers to that network; doing either would make this header forgeable." One ports: line added for a quick debug, and anyone could send the header.

The proxy hides one more trap. Every request reaches the gateway with Caddy's address, so a per-IP rate limit would give all phones one shared budget. The gateway keys its limit on the verified certificate:

await app.register(fastifyRateLimit, {
  max: 120,
  timeWindow: '1 minute',
  keyGenerator: (request) => {
    const fingerprint = request.headers['x-verified-client-certificate-fingerprint']
    const normalized = typeof fingerprint === 'string' ? fingerprint.trim().toLowerCase().replaceAll(':', '') : ''
    return /^[0-9a-f]{64}$/.test(normalized) ? `device:${normalized}` : `addr:${request.ip}`
  },
})

The authority's private key stays offline, and each phone's certificate is revoked when the device is.

Layer 6: a server you cannot lock yourself out of

This site's server was set up by one script, and its SSH part is careful about order:

if [ -s "/home/$USER_NAME/.ssh/authorized_keys" ]; then
  cat >/etc/ssh/sshd_config.d/10-moghalis.conf <<'EOF'
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
EOF
  sshd -t && systemctl reload ssh
else
  echo "!! No SSH key for $USER_NAME yet: password login left ON."
fi

ufw default deny incoming
ufw allow OpenSSH
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 443/udp
ufw --force enable

Password login goes off only when a key is in place, SSH reloads only when sshd -t accepts the config, and the firewall allows SSH before it goes up. Get any of the three out of order on a fresh VPS and your next stop is the provider's rescue console.

Six layers, each assuming the one above it fails. The backup assumes they all did.

Beside the stack: a backup that proves it restores

FinyxWatt's PostgreSQL archives its write-ahead log at least every five minutes and takes a full base backup every night with WAL-G, whose binary is pinned and checksum-verified. Every file is compressed and encrypted before it is written. The owner's office computer pulls new files every four hours and keeps 45 days of them.

The encryption key is the backup: lose it and every file is noise. It is sealed with the other server secrets in a password-protected file, on a USB drive and in a second place, with the password kept apart from both.

None of that proves anything until a restore works. Every Friday a script restores the newest base backup and every archived log after it into a throwaway volume. Trimmed:

timeout --kill-after=60 2700 docker run --rm --network none --name "$name" \
  -e PITR_STORAGE_MODE=local -e WALG_DOWNLOAD_FILE_RETRIES=2 \
  -v "$name:/restore" \
  -v "$repository:/local-pitr:ro" \
  -v "$key:/run/secrets/pitr_encryption_key:ro" \
  --entrypoint bash "$image" /restore-check.sh

Inside, the copy's config gets these lines before PostgreSQL starts:

cat >> /restore/data/postgresql.auto.conf <<'CONF'
restore_command = '/usr/local/bin/pitr-walg wal-fetch %f %p'
archive_mode = off
listen_addresses = ''
CONF
touch /restore/data/recovery.signal

Each line is a guard. No network, so the copy cannot touch the live database. A read-only repository and key. No archiving, no listeners. And $image is read from the live database container, so the check uses the exact image production runs. When recovery finishes, it counts what came back:

SELECT json_build_object(
  'migrations',   (SELECT count(*) FROM schema_migrations),
  'stations',     (SELECT count(*) FROM tenants),
  'customers',    (SELECT count(*) FROM customers),
  'bills',        (SELECT count(*) FROM invoices),
  'receipts',     (SELECT count(*) FROM payments),
  'newestChange', (SELECT max(changed_at) FROM change_log));

The counts say the backup holds what it should; the newest change says how fresh it is. Then everything is deleted, whatever happened. A failed check, or no check for nine days, raises an alert in my developer app.

Now the third trap, and the reason for WALG_DOWNLOAD_FILE_RETRIES=2. WAL-G retries a file it cannot read 15 times, minutes apart. Point it at a damaged or undecryptable backup and the check does not fail. It waits, for hours. Two retries and a 45-minute ceiling turn that wait into a clear failure with a reason.

A watchdog runs every ten minutes and reports what quietly breaks backups: a disk at 85%, a stopped service, a full backup older than 26 hours, stuck WAL archiving, an office computer that has not pulled for three days, a certificate close to expiry.

The gate in front of every release

The layers protect the running system. The gate protects it from the next release.

A repository check fails the build on private keys, tokens, JWTs, database URLs with a password, real emails and phone numbers, and any tracked .env or secrets/ file:

['private key material', /-----BEGIN(?: [A-Z]+)? PRIVATE KEY-----/],
['GitHub access token', /\bgh[pousr]_[A-Za-z0-9_]{30,}\b/],
['database URI with embedded password', /\bpostgres(?:ql)?:\/\/[^\s:@/${]+:[^\s@/${]{8,}@/i],

It lists files with git ls-files --cached --others --exclude-standard, so run locally it catches a file before you add it.

Then CI runs every migration on a fresh PostgreSQL with the tenant-isolation tests, the backend tests, integration tests under the restricted database role, and npm audit --omit=dev --audit-level=high. Automatic deployment waits for all of it:

on:
  workflow_run:
    workflows: [Independent Platform CI]
    types: [completed]
    branches: [main]

concurrency:
  group: production-deploy
  cancel-in-progress: false

jobs:
  deploy:
    if: github.event.workflow_run.conclusion == 'success'
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.workflow_run.head_sha }}

Three traps live in those lines. workflow_run fires when CI completes, pass or fail, so without the if a red build deploys. Checking out main instead of head_sha can ship a commit CI never tested. And a deployment cancelled halfway leaves a half-migrated server. The job also pins the server's host-key fingerprint before its first SSH command.

Attack it yourself, on paper

A written review attacked FinyxWatt through its code and with passive requests to the live servers, recording each result with the file that proves it:

The checklist

• Row-level security forced on every tenant table; policies check an active membership.
• Tenant context per transaction with set_config(..., true), from the token, read through nullif.
• No application login is owner, superuser or BYPASSRLS; one login per job.
• Ledger and audit tables: no UPDATE or DELETE grants, plus a trigger that refuses both.
• Secrets as files, mounted only where needed; CI fails on keys, tokens and tracked .env files.
• Containers read-only, capability-free, memory-capped, on internal networks.
• Identity headers trusted only on a closed network; rate limits keyed on that identity.
• Encrypted backups, the key kept apart, a copy off the server, a timed restore check with an alert.
• Deploys only after green CI, from the tested commit, never cancelled halfway.

None of these layers is clever on its own. The strength is the assumption under all of them: the layer above has already failed, and this one has to hold anyway. Write that assumption into your configs, and the day one of them fails becomes an ordinary day.

If you would rather have someone run this for your system, that is what hosting and support covers.

Some of the business owners who write to me have sat on the other side of that week. They paid for software and never received it in a state their staff could use.

I run projects so that week never comes. This is the method, written for developers and freelancers. By the end you will have the outline of a proposal that settles most arguments before they start, a phase loop that puts working software on the client's server from the first phase, a way to make your tests part of what the client buys, a handover pack, and the questions good clients ask, answered in writing before they ask.

Two things below do more work than the rest: one section of the proposal that heads off more arguments than any other, and one line in the terms that makes the final payment the easy one.

The first call is a requirements interview

A short description is enough to start: what the business does, who will use the system, and what is not working today. My contact form also asks for the type of project and a rough budget range, and "not sure yet" is one of the answers.

Then I map the business. Three questions carry most of the weight:

• Who uses the system? Reception, accountants, branch managers, drivers, customers.
• What data matters? Invoices, stock, patient files, readings, cash boxes.
• Who may do what? Who can cancel an invoice, who sees cost prices, who approves a refund.

The third question decides more of the design than any screen. On a network operations platform I built, the answer became 49 separate permissions grouped into four roles. On an ordering platform for a clothing retailer, the central rule was that a customer must never see another tier's price, and that rule is enforced inside the database with its own tests.

Then ask to see the tools they use today: the shared spreadsheet, the paper form, the old program. They show the real workflow better than anyone's description of it.

The proposal: six sections, and the one that prevents arguments

Nothing is built before the client holds a written proposal. Mine follow one outline:

The section that heads off more arguments than any other is the "not included" list. Server and domain fees, a WhatsApp Business account, the app stores' fees: all of them are paid to other companies. Written down before the client signs, none of them can arrive later as a surprise with your name on it.

The "included" list earns its place too. In my template it reads: design, programming, automated tests, deployment on the client's server, training for the reception team, and a support period after the final handover. Automated tests are a line the client reads. That matters later.

Each phase has its own cost, and my template ties each payment to the start of its phase. The client never pays more than one phase ahead of software they have used, and you never work more than one phase ahead of the money.

A private page with a link nobody can guess

The client receives the proposal as a page on my site, in Arabic or English, with a reference number and the date it stays valid until. Nothing on the site links to it. If you build the same, three rules from my build are worth copying.

The file name is the link, so the build refuses any name without a random tail of at least six characters:

const SLUG = /^[a-z0-9]+(?:-[a-z0-9]+)*-[a-z0-9]{6,}$/

The prerender stops if a proposal page is not noindex. Proposals stay out of the sitemap, robots.txt disallows their paths, and the list of proposals is never bundled for the browser.

The server keeps search engines away and keeps the link out of referrers when the client clicks onward:

@proposals path /p/* /ar/p/*
header @proposals {
	X-Robots-Tag "noindex, nofollow, noarchive"
	Referrer-Policy "no-referrer"
}

Design while a change costs minutes

Before code, we agree on four things: the data the system keeps, the permissions, the connections to other systems, and the main screens.

This is the cheapest moment for anyone to change their mind. Moving a field on a sketch takes minutes. Changing how data is stored once the system is live is real engineering: when I moved FinyxWatt, my own platform for electricity and water stations, from Firebase to PostgreSQL, it took 107 ordered and checksummed migrations to do it while the stations kept working.

Every phase ends live on the client's server

The work is split into phases, and each phase is useful on its own. In my proposal template, a clinic system is split like this:

Each phase ends with a version running on the client's server, or the hosting the proposal names, and their team tests it before the next phase begins. Inside a phase I work in short iterations, with working software, tests and a staging environment.

That order protects both sides. The client sees working software from the first phase. Their staff find the problems that matter to them while those are still cheap to fix. If priorities change, the next phase changes with them. And every payment after the first follows software the client has already used.

Delivering every phase to a real server only works when a release is cheap and reversible. The network operations platform's deploy script takes a verified backup before it stops a single container, refuses to go on if the backup fails, runs the migrations and waits for the health check. The heart of this site's rollback is five lines:

CURRENT=$(readlink -f "$ROOT/current")
PREVIOUS=$(ls -1dt "$ROOT"/releases/*/ | sed 's:/$::' | grep -vx "$CURRENT" | head -n 1 || true)
[ -n "$PREVIOUS" ] || { echo "!! No earlier release to go back to."; exit 1; }
ln -sfn "$PREVIOUS" "$ROOT/current.new"
mv -Tf "$ROOT/current.new" "$ROOT/current"

It takes under a second. When going back costs that little, putting a phase on the client's server stops being a gamble.

Tests are part of what the client buys

An automated test is a small program that checks one rule of the business every time the code changes. Across twenty products I have written over four thousand of them.

Tests are easy to treat as a cost hidden inside the estimate. In my template they sit in the "included" list, where the client reads them. The trick is to describe each one in the client's words.

In FinyX-ERP, my accounting and point-of-sale product, the owner can delete a posted invoice through an administrative tool. A set of tests checks that the deletion happens, that the protections which stop anyone from editing posted amounts are switched back on afterwards, and that they come back even when the deletion fails halfway. The comment at the top of the file explains the stakes: the owner asked for the deletion, so the deletion is fine. The danger is a ledger left without its guards, silently, until someone edits a posted amount.

Its first check is worth copying into every suite:

Require(before > 0, "The database carries no integrity triggers, so the test proves nothing.");

A test that cannot fail proves nothing, so this one confirms it can before it checks anything else. And "posted amounts can never be edited, even after an administrative delete" is a sentence an owner wants to read in a proposal.

On larger projects the tests run before any release goes out. On FinyxWatt, the app suites run on every push and pull request, and an automatic server deployment starts only after the server's suites have passed in CI. On the Bin Masoud store, every backend change runs through lint, type checks, unit tests with coverage, the database migrations and end-to-end tests.

The acceptance report

The client's own testing comes on top, and it deserves a document. For the network operations platform, the release candidate came with a written acceptance report:

The last two rows build more trust than all the others. A developer who writes down his own limitations is believed about everything else.

The handover pack

Now the line in the terms. My template says the source code belongs to the client once the final payment is made. By then their team has used every phase on their own server, so the final payment transfers ownership of a system they already run. It is the easiest payment to ask for, because nothing in it is taken on trust.

What they receive:

The documentation and script lists are what the network operations platform's repository holds. The test of a pack is simple: could the next developer run the system without calling you? A client who knows they could leave is a client who can afford to trust you.

Scope changes, in writing, every time

Projects change. The term in my template takes one line: anything outside the agreed scope is estimated and approved in writing before work starts. That keeps the proposal the client signed meaningful, and it keeps you from doing free work in silence.

Phases make it easier. A change can be estimated and placed in the next phase, and when priorities shift, the next phase shifts with them. A written change needs five things: what changes, why, the estimate, the phase it lands in, and the client's approval.

Support after go-live

Real use always finds things testing did not. Deployment, backups, monitoring and those fixes are part of the work, and a support period after the final handover is written into the proposal. I run seven live systems today: their releases, servers, security and the fixes that keep them up.

What moves the estimate

I give no price before the first conversation, because the same words describe very different systems. These move the estimate most:

• Roles and permissions. Two roles with full access is a small system. Many roles with rules per branch is a larger one.
• Integrations. Payment gateways such as Moyasar, WhatsApp, Salla, SMS. Each has its own rules, test environment and approval steps.
• Platforms. Web only, or also Android, iOS and a Windows desktop app.
• Working offline. An app that keeps working without a connection needs a safe way to sync when it returns.
• Existing data. Importing years of spreadsheets or an old database is a task of its own.
• Reports. Each report is a query, a screen and an export.
• Operations. Backups, monitoring, and who looks after the server.

Phases help here too. A first phase can be sized to the client's budget and still be useful on its own.

The questions good clients ask

Good clients ask these before they sign. Put every answer in the proposal, and you will never have to give one in an argument.

• Will I get a written proposal with phases, and a list of what is not included?
• Where will the system run, and whose name are the server and the domain in?
• Will I test each phase on my own server before the next one starts?
• Who owns the source code, and from when?
• How many automated tests are there, and do they run before every release?
• How are backups made, and when was a restore last tested?
• What documentation will I receive at handover?
• How are changes outside the scope priced and approved?
• If a release goes wrong, how is it undone, and how long does that take?
• After go-live, who fixes problems, and for how long?

The final payment is easy when nothing about it is new. The client has used every phase on their own server, read the not-included list before signing, approved every change in writing, and holds a pack that lets them run the system without you. That is also why they come back.

Every project on my services page runs this way.

On a phone, the same page stuttered.

I expected a heavy image or a fat bundle. Chrome showed me something stranger: with nobody touching the screen, the main thread was busy 98% of every second.

So I started switching animations off to find the guilty one. Nothing changed. That dead end turned out to be the best clue of the whole hunt.

By the end of this article you'll know why one animation can tax all the others, how to make Chrome name the one doing it, and you'll have two scripts that measure it for you.

98% busy, and nobody was touching it

The test was simple. Load the page in a phone profile (390×844, 3× pixel density, CPU slowed 4×), let it finish loading, touch nothing, and ask Chrome how much of each second the main thread spent running tasks.

At 98%, every tap and every scroll queues behind work the user can't see. The page looks alive and feels dead.

Frames per second hid all of it. Most of my motion ran on the compositor, a separate thread that kept moving layers smoothly while the main thread drowned. The FPS meter was measuring the healthy thread.

The number comes from one DevTools Protocol metric, TaskDuration: the total seconds the main thread has spent on tasks. Read it, wait three seconds, read it again, divide. The full script, with A/B runs and medians, is near the end.

One warning before any number: my machine was often busy with other work, so single runs lied. Every comparison in this article was A/B, interleaved, and the median of several runs.

The clue: switching animations off changed nothing

I turned animations off in groups. Removed the canvas: 98%. Removed the rings: 98%.

No single group mattered, so I was asking the wrong question. The cost behaved like a fixed fee, and a few animations were making the whole page pay it.

A composited animation is a layer the GPU moves on its own. The main thread hands it over once and goes quiet. A non-composited animation needs the main thread to produce a fresh frame every time the screen refreshes, and that frame is a full one: style, layout bookkeeping, paint and layer updates, for the whole page.

So 200 animations, most of them already on the GPU, paid that fixed cost because a handful of others kept asking for a frame. Removing one group did nothing while another was still asking.

The real question became: what forces a main-thread frame? I listed six suspects and replaced every one of them. When I re-tested each alone, five were guilty and one was innocent. The first had a comment at the top of its file promising it would cost almost nothing.

The six suspects I chased, and what replaced each

1. The canvas that was "built to cost almost nothing"

That's what the old file said, and it looked responsible. The rune rain was one transparent canvas at CSS pixel size, capped at 24 frames a second on phones and 30 on desktop, paused when the tab was hidden.

Here is its loop:

const tick = (t: number) => {
  raf = requestAnimationFrame(tick)
  if (t - last < frameMs) return // too soon: skip this frame
  last = t
  draw()
}

The skip is the trap. Every one of those callbacks, including the ones that return at once, is a request for a main-thread frame. The frame cap saved drawing time and nothing else.

The replacement draws nothing at runtime. Twenty-six columns of glyphs, each string picked once from a fixed seed (so the prerendered HTML and the browser agree), each one moved down by a CSS transform:

.rune-col {
  position: absolute;
  top: 0;
  left: var(--x);
  width: 1ch;
  word-break: break-all;           /* one glyph per line: a column */
  mask-image: linear-gradient(transparent, #000 80%); /* the tail fades */
  transform: translate3d(0, -110%, 0);
  animation: rune-fall var(--dur) linear infinite;
  animation-delay: var(--delay);   /* negative: already falling on first paint */
}

@keyframes rune-fall {
  to { transform: translate3d(0, calc(100vh + 10%), 0); }
}

@media (max-width: 760px) {
  .rune-col:nth-child(2n) { display: none; } /* half the streams on a phone */
}

Each column gets its own --x, a duration between 11 and 24 seconds, and a negative delay. Same look. Zero script per frame.

2. React state that ticked like a clock

The terminal typed by calling setState on a timer: every 42 to 82 ms while typing, every 18 ms while erasing. Each call meant a render, a style recalculation and a layout, because the text changed width every time.

Now the typing is pure CSS. A window one character wider than the command slides in one-character steps, while the text inside it slides back by the same amount. Each letter appears exactly where it stands, and only transforms move.

.tw {
  display: inline-block;
  animation: tw-in var(--type) steps(var(--steps), jump-start) both;
}
.tw-window {
  position: relative;
  display: block;
  width: calc(var(--steps) * 1ch);
  height: 1.5em;
  overflow: hidden;
  animation: tw-out var(--erase) steps(var(--steps), jump-start) calc(var(--type) + var(--hold)) forwards;
}
.tw-text-out {
  display: block;
  animation: tw-text-out var(--erase) steps(var(--steps), jump-start) calc(var(--type) + var(--hold)) forwards;
}
.tw-text {
  display: block;
  width: calc(var(--steps) * 1ch);
  line-height: 1.5em;
  animation: tw-text-in var(--type) steps(var(--steps), jump-start) both;
}

@keyframes tw-in       { from { transform: translate3d(-100%, 0, 0); } }
@keyframes tw-out      { to   { transform: translate3d(-100%, 0, 0); } }
@keyframes tw-text-in  { from { transform: translate3d(100%, 0, 0); } }
@keyframes tw-text-out { to   { transform: translate3d(100%, 0, 0); } }

.caret {                          /* rides the window's last cell */
  position: absolute;
  top: 50%;
  right: 0;
  width: 1ch;
  height: 1.15em;
  translate: 0 -50%;
  background: currentColor;
  animation: caret-blink 1s steps(1) infinite;
}
@keyframes caret-blink { 50% { opacity: 0; } }

React sets the timing once per command:

import type { CSSProperties } from 'react'

const TYPE_MS = 55
const ERASE_MS = 18
const HOLD_MS = 1700

export function TypeLine({ text }: { text: string }) {
  const timing = {
    ['--steps' as string]: text.length + 1, // one step per character, one for the caret
    ['--type' as string]: `${text.length * TYPE_MS}ms`,
    ['--hold' as string]: `${HOLD_MS}ms`,
    ['--erase' as string]: `${text.length * ERASE_MS}ms`,
  } as CSSProperties
  // The key remounts the layers, so a new command restarts every animation.
  return (
    <span key={text} className="tw" style={timing}>
      <span className="tw-window">
        <span className="tw-text-out">
          <span className="tw-text">{text}</span>
        </span>
        <span className="caret" />
      </span>
    </span>
  )
}

It needs a monospace font, since every step is 1ch. Swap the text after text.length * (TYPE_MS + ERASE_MS) + HOLD_MS plus a short gap. That's one setState every few seconds, down from dozens a second.

Look at those four layers. The first CSS version had fewer, and that is cause number six.

3. A light that slid by background-position

Every section label had a gold rule with a light running along it: a gradient three times wider than the rule, animated with background-position. The compositor can't move a background inside an element, so it meant a repaint every frame, for every label.

/* Before: a repaint every frame */
.eyebrow::before {
  background: linear-gradient(90deg, var(--gold) 0 35%, var(--gold-fill-2) 50%, var(--gold) 65% 100%);
  background-size: 300% 100%;
  animation: rule-scan 3.4s linear infinite;
}

/* After: the rule draws itself out and back */
.eyebrow::before {
  transform-origin: left center;
  animation: rule-draw 3.4s ease-in-out infinite alternate;
}
html[dir='rtl'] .eyebrow::before {
  transform-origin: right center; /* grow from the side the text starts on */
}
@keyframes rule-draw {
  from { transform: scaleX(0.35); opacity: 0.55; }
  to   { transform: none; opacity: 1; }
}

It's a different effect, a rule that breathes in and out, and it costs nothing per frame.

4. The innocent suspect: a blinking line inside an SVG

The cursor under the M in my logo blinked with opacity on a path inside the SVG. I believed that repainted the SVG on every frame, so I moved the blink to an HTML element laid exactly where the SVG would draw it. The numbers come from the SVG's own geometry: a 16-unit line with 2.6-unit round caps, in a 64-unit viewBox drawn at 30px, is (16 + 2.6) × 30 ÷ 64 = 8.7px wide.

.logo-mark { position: relative; }

.logo-caret {
  position: absolute;
  top: 27.8px;
  left: 50%;
  width: 8.7px;
  height: 1.3px;
  translate: -50% 0;
  border-radius: 1px;
  background: var(--blue);
  animation: caret-blink 1.1s steps(1) infinite;
}

@keyframes caret-blink { 50% { opacity: 0; } }

Then, for this article, I re-tested every suspect on its own page. The old logo, blinking alone, caused no main-thread frames in Chrome 153 or in Edge 155, and the trace reported no failure for it. This one was innocent.

The HTML caret stays, since opacity on an HTML element is exactly the case the compositor is built for. The lesson is the part worth keeping: test each suspect alone, or you will credit a fix that did nothing.

5. A conic border spun by @property

The contact form had a light travelling round its border: a conic gradient whose angle was a registered custom property, animated from 0 to 360 degrees.

/* Before: computed on the main thread every frame */
@property --spin {
  syntax: '<angle>';
  inherits: false;
  initial-value: 0deg;
}
.contact-panel::before {
  background: conic-gradient(from var(--spin), transparent 0 68%, var(--gold) 84%, transparent 100%);
  animation: spin-border 6s linear infinite;
}

@property makes the angle animatable, and the compositor still can't run it. Chrome recomputes the style and repaints the gradient on every frame.

The same look works as a big square of conic gradient that rotates with a transform, under a 1px ring mask that never moves:

/* The parent needs position: relative and isolation: isolate. */
.spin-ring {
  position: absolute;
  inset: -1px;
  z-index: -1;
  overflow: hidden;
  padding: 1px;
  border-radius: inherit;
  /* Keep only the 1px of padding: a ring. */
  -webkit-mask: linear-gradient(#000 0 0) content-box, linear-gradient(#000 0 0);
  -webkit-mask-composite: xor;
  mask-composite: exclude;
  pointer-events: none;
}
.spin-ring > span {
  position: absolute;
  top: 50%;
  left: 50%;
  width: 150%;
  aspect-ratio: 1;
  translate: -50% -50%;
  background: conic-gradient(transparent 0 68%, var(--gold) 84%, transparent 100%);
  animation: spin-ring 6s linear infinite;
}
@keyframes spin-ring { to { transform: rotate(360deg); } }

The gradient is rasterised once. Only its rotation changes, and rotation is the compositor's job.

6. Two animations, one element, and the number 64

I would never have found this one by reading code. It came out of a Chrome trace, with a number attached: 64.

In the first CSS terminal, one element carried both tw-in (slide in while typing) and tw-out (slide out while erasing), and tw-out only started after tw-in had finished moving. Chrome refused to put either of them on the compositor. Both ran on the main thread, every frame, for as long as the terminal was on screen.

That 64 is a bitmask from Chromium's source (compositor_animations.h), and bit 6 is kTargetHasIncompatibleAnimations. Chromium does allow several transform-related animations on one element when each targets a different property, say rotate on one and scale on another. Mine both animated transform.

The fix is the stack you saw above: one transform animation per element, nested four deep.

So how do you get Chrome to hand you that number?

Make Chrome name the culprit

Chrome records why it couldn't composite each animation. You only have to ask. Record a trace with the blink.animations category and read the Animation events:

• the begin event (ph: 'b') carries args.data.displayName (your @keyframes name; name comes back empty for CSS animations), nodeName and nodeId;
• the instant events (ph: 'n') carry args.data.compositeFailed, a bitmask, and args.data.unsupportedProperties, the CSS properties the compositor couldn't run;
• both share id2.local, which is how you pair them.

Here it is as a script. Give it a URL and it prints every animation the compositor refused, with the reason in words.

// composite-audit.mjs   usage: node composite-audit.mjs http://localhost:4173/
import { chromium } from 'playwright'

// Chromium's CompositorAnimations::FailureReason bits (compositor_animations.h)
const REASONS = {
  [1 << 3]: 'unsupported timing parameters',
  [1 << 5]: 'target has invalid compositing state',
  [1 << 6]: 'target has incompatible animations',
  [1 << 10]: 'transform cannot be accelerated on this target',
  [1 << 12]: 'filter may move pixels',
  [1 << 13]: 'unsupported CSS property',
  [1 << 17]: 'no visible change',
  [1 << 18]: 'animates an !important property',
  [1 << 19]: 'SVG target has an independent transform property',
}
const decode = (mask) =>
  Array.from({ length: 22 }, (_, bit) => 1 << bit)
    .filter((flag) => mask & flag)
    .map((flag) => REASONS[flag] ?? `bit ${Math.log2(flag)}`)
    .join(', ')

const browser = await chromium.launch()
const context = await browser.newContext({ viewport: { width: 390, height: 844 }, deviceScaleFactor: 3, isMobile: true })
const page = await context.newPage()
await browser.startTracing(page, { categories: ['devtools.timeline', 'blink.animations'] })
await page.goto(process.argv[2], { waitUntil: 'load' })
await page.waitForTimeout(5000)
const { traceEvents } = JSON.parse((await browser.stopTracing()).toString())
await browser.close()

const idOf = (e) => e.id2?.local ?? e.args?.data?.id
const started = new Map()
for (const e of traceEvents) if (e.name === 'Animation' && e.ph === 'b') started.set(idOf(e), e.args.data)

const printed = new Set()
for (const e of traceEvents) {
  const mask = e.name === 'Animation' && e.ph === 'n' ? e.args.data?.compositeFailed : 0
  if (!mask) continue
  const { displayName = '?', nodeName = '?' } = started.get(idOf(e)) ?? {}
  const props = e.args.data.unsupportedProperties?.join(', ')
  const line = `${nodeName}  ${displayName}  ${mask}: ${decode(mask)}${props ? `  [${props}]` : ''}`
  if (!printed.has(line)) {
    printed.add(line)
    console.log(line)
  }
}

// Scripts never appear above. Count main-thread work in the last 3 seconds instead.
const end = traceEvents.reduce((max, e) => Math.max(max, e.ts ?? 0), 0)
const recent = (name) => traceEvents.filter((e) => e.name === name && e.ts > end - 3e6).length
console.log('last 3 s:', {
  FireAnimationFrame: recent('FireAnimationFrame'),
  TimerFire: recent('TimerFire'),
  UpdateLayoutTree: recent('UpdateLayoutTree'),
})

I ran it on a test page holding the "before" CSS from this article and the old canvas loop. This is what Chrome 153 printed:

DIV class='contact-panel'  spin-border  8192: unsupported CSS property  [--spin]
SPAN class='tw-window'  tw-in  64: target has incompatible animations
SPAN class='tw-window'  tw-out  64: target has incompatible animations
SPAN class='eyebrow-rule'  rule-scan  8224: target has invalid compositing state, unsupported CSS property  [background-position-x]
last 3 s: { FireAnimationFrame: 180, TimerFire: 0, UpdateLayoutTree: 180 }

Read it line by line:

• [--spin] and [background-position-x] come from unsupportedProperties. Note the second one: Chrome names the longhand, not the shorthand you wrote.
• Both terminal animations fail with 64, which is why neither one ran on the GPU.
• 180 FireAnimationFrame callbacks in 3 seconds is the canvas loop, frame cap and all: one per vsync. The cap never reduced the callbacks.
• UpdateLayoutTree is Chrome's style recalculation. 180 in 3 seconds means one on every frame.

The same page with the "after" CSS printed no failures, no animation frames, and 11 style updates in 3 seconds.

Two of the causes never show up as Animation events, because they are scripts: the canvas loop and the timer behind setState. FireAnimationFrame counts requestAnimationFrame callbacks and TimerFire counts timers. On a page at rest, both should be close to zero.

Two traps that looked like good ideas

All six were replaced and the number was down. Then one elegant selector doubled it.

:has() over a subtree that animates. The headline's gold sweep had to wait until the headline had finished decoding. The elegant way to say that:

.hero-headline:has(.decode:not(.is-done)) .sheen-band {
  animation-play-state: paused;
}

It worked, and it doubled idle CPU on phones. With animations running inside the headline, Chrome rechecked the :has() on every frame.

The fix is boring. React adds a class once, when the decode settles, and CSS reads the class:

.hero-headline:not(.is-decoded) .sheen-band,
.hero-headline:not(.is-decoded) .sheen-copy {
  animation-play-state: paused;
}

Backdrop blur over a moving background. The desktop header had backdrop-filter: blur(18px) saturate(140%) over a 78% background. Behind it the runes never stop falling, so the blur was recomputed every frame. I raised the bar to 94% and dropped the blur. At that opacity nobody can tell the difference. Phones already had it off.

Pause what nobody can see

Animations off screen still cost something, and a long page keeps most of its animations off screen. One IntersectionObserver marks every section outside the viewport, and CSS pauses everything inside it:

[data-offscreen]::before,
[data-offscreen]::after,
[data-offscreen] *,
[data-offscreen] *::before,
[data-offscreen] *::after {
  animation-play-state: paused !important;
}

import { useEffect } from 'react'

const TARGETS = 'section, footer'

export default function OffscreenPause() {
  useEffect(() => {
    if (!('IntersectionObserver' in window)) return
    let io: IntersectionObserver | undefined
    let mo: MutationObserver | undefined
    let cancelled = false
    const each = (node: Node, fn: (el: Element) => void) => {
      if (!(node instanceof Element)) return
      if (node.matches(TARGETS)) fn(node)
      node.querySelectorAll(TARGETS).forEach(fn)
    }
    const start = () => {
      if (cancelled) return
      const observer = (io = new IntersectionObserver(
        (entries) => entries.forEach((e) => e.target.toggleAttribute('data-offscreen', !e.isIntersecting)),
        { rootMargin: '160px 0px' }, // start moving just before a section scrolls in
      ))
      document.querySelectorAll(TARGETS).forEach((el) => observer.observe(el))
      // Route changes add and remove sections.
      mo = new MutationObserver((records) =>
        records.forEach((r) => {
          r.addedNodes.forEach((n) => each(n, (el) => observer.observe(el)))
          r.removedNodes.forEach((n) => each(n, (el) => observer.unobserve(el)))
        }),
      )
      mo.observe(document.body, { childList: true, subtree: true })
    }
    // Prerendered HTML: wait until React has taken the page over, or it sees attributes it never rendered.
    const kick = () =>
      window.requestIdleCallback ? window.requestIdleCallback(start, { timeout: 2500 }) : window.setTimeout(start, 300)
    if (document.readyState === 'complete') kick()
    else window.addEventListener('load', kick, { once: true })
    return () => {
      cancelled = true
      window.removeEventListener('load', kick)
      mo?.disconnect()
      io?.disconnect()
    }
  }, [])
  return null
}

Mount it once, anywhere. Two details took me a while:

• The 160px margin. Sections start moving before they scroll into view, so nobody ever sees one start.
• CSS pauses animations, and only animations. Your timers keep running. The terminal's timer checks closest('[data-offscreen]') and, if its section is paused, waits a second and checks again.

Measure it yourself: the whole script

This is the idle measurement from the top of the article. Build the two versions you want to compare, serve them on two ports, and run one command.

// idle-busy.mjs   usage: node idle-busy.mjs http://localhost:4173/ http://localhost:4174/
import { chromium } from 'playwright'

const RUNS = 5          // per URL, interleaved: A, B, A, B...
const SETTLE_MS = 4000  // let loading finish; we want the cost at rest
const WINDOW_MS = 3000

const browser = await chromium.launch()

async function idleBusy(url) {
  const context = await browser.newContext({
    viewport: { width: 390, height: 844 },
    deviceScaleFactor: 3,
    isMobile: true,
    hasTouch: true,
  })
  const page = await context.newPage()
  const cdp = await context.newCDPSession(page)
  await cdp.send('Emulation.setCPUThrottlingRate', { rate: 4 })
  await cdp.send('Performance.enable')
  await page.goto(url, { waitUntil: 'load' })
  await page.waitForTimeout(SETTLE_MS)

  const taskSeconds = async () =>
    (await cdp.send('Performance.getMetrics')).metrics.find((m) => m.name === 'TaskDuration').value
  const before = await taskSeconds()
  await page.waitForTimeout(WINDOW_MS)
  const after = await taskSeconds()
  await context.close()
  return ((after - before) / (WINDOW_MS / 1000)) * 100
}

const urls = process.argv.slice(2)
const runs = Object.fromEntries(urls.map((url) => [url, []]))
for (let i = 0; i < RUNS; i++) for (const url of urls) runs[url].push(await idleBusy(url))
await browser.close()

const median = (xs) => [...xs].sort((a, b) => a - b)[Math.floor(xs.length / 2)]
for (const [url, xs] of Object.entries(runs)) {
  console.log(`${median(xs).toFixed(0)}% busy at rest  ${url}  (runs: ${xs.map((x) => x.toFixed(0)).join(', ')})`)
}

Install with npm i -D playwright and npx playwright install chromium.

What changed

Same page, same animations, measured on the same phone profile:

The load-time gains came from three other changes. I dropped framer-motion (42 KB gzipped) for a few CSS keyframes. The home page stopped bundling the full case-study texts, which took the main script from 416 KB to 324 KB. And the web fonts stopped holding up the app script.

Not one animation was removed.

The checklist

• Animate transform and opacity on HTML elements. Nothing else, ever, in a loop.
• No script that runs for ever: no requestAnimationFrame loops, no canvas redraws, no state updates on a timer.
• One transform animation per element. Nest a wrapper when you need two.
• No :has() over a subtree that animates. Set a class once instead.
• No backdrop-filter over moving content.
• Pause what's off screen, and make your timers wait with it.
• Run the trace audit after every new effect. Every line it prints is a main-thread frame you pay for.
• Test each suspect alone before you credit a fix. One of my six changed nothing.
• Measure the idle main thread, A/B, with medians. Frames per second only tell you about the healthy thread.

Motion was never the expensive part. The expensive part was the one animation quietly asking the main thread for a frame, and the whole page paying for it.

That headline with the gold sweep broke in Arabic in a way no trace could show. It's the other half of this story.

Then I switched the page to Arabic.

Before the effect had even finished, my eye caught on one word: «كاملًا». The «لا» at its end, a shape every Arabic reader knows at a glance, stood there as two separate letters. A second later the gold sweep passed over the headline, and every word looked doubled.

One word, two bugs, and a third across the whole line. The fix every developer reaches for, Intl.Segmenter, cures exactly one of them.

And when I rebuilt the bug on a test page for this article, it got stranger. The same spans were fine in an ordinary paragraph. It took one CSS property on my headline to break them.

By the end you'll know which property, why the word is the smallest unit you can safely animate in Arabic, how to hand the line back so anything layered on it lines up to the pixel, and you'll have a component you can paste.

The version everyone writes first

Split the text into characters, wrap each one in a span, keep them transparent until their turn, and draw a random glyph on top while they wait:

{Array.from(text).map((char, i) => (
  <span key={i} className="dc">{char}</span>
))}

.dc { color: transparent; }
.dc.is-scr::after { content: attr(data-g); position: absolute; /* … */ }
.dc.is-on { color: inherit; }

One part of this is right, and it stayed. Every character is in the DOM from the first frame, so the line already has its final width and wraps where it will wrap. Nothing reflows while the effect runs.

Everything else had to change.

What broke, one layer at a time

Bug 1: the marks fell off their letters

Array.from() splits a string by code point. «كاملًا» has six:

const hex = (s) => [...s].map((c) => c.codePointAt(0).toString(16).toUpperCase().padStart(4, '0'))

hex('كاملًا')
// ['0643', '0627', '0645', '0644', '064B', '0627']   six code points, six spans

That 064B is the tanween, the mark that sits on the lam. As its own code point it got its own span, separated from the letter it belongs to.

Don't assume your strings are free of marks. Interface Arabic drops most of them, but tanween on adverbs like «كاملًا» and «أيضًا» is standard spelling, and a shadda turns up in names like «محمّد».

What a reader sees as one character is a grapheme cluster, and the browser can segment those for you:

const seg = new Intl.Segmenter(undefined, { granularity: 'grapheme' })

Array.from(seg.segment('كاملًا'), (part) => hex(part.segment).join('+'))
// ['0643', '0627', '0645', '0644+064B', '0627']   five graphemes: the mark rides with its lam

Bug 1, fixed. Now look at the last two entries.

Bug 2: «لا» split in two, and Intl.Segmenter can't stop it

0644+064B is the lam with its mark. 0627 is the alef. Two graphemes, so still two spans.

Lam followed by alef is the one ligature Arabic requires. Unicode keeps the finished shape only for compatibility (ﻻ, U+FEFB). Real text stores two letters, and the font draws a single shape when it shapes them together.

Grapheme clusters describe what a reader perceives as one character. The ligature is bigger than that: two characters the font fuses into one glyph. No grapheme split can keep it whole, because it spans two graphemes.

So when does the split actually show? I rebuilt the headline on a test page (IBM Plex Sans Arabic at weight 600, in Chrome 153 and Edge 155, which agreed to the pixel) and measured the line in four setups:

In an ordinary line, Chrome shapes across inline spans and the word survives. My headline had text-wrap: balance, and with it the same spans lost the ligature, several joins, and 43 pixels of width. The inline-block row is what you get from any effect that moves letters one by one, since a transform needs a box: every letter stands alone.

Even in the case that survives, there's a catch. With my font, the ligature is a single glyph, and Chrome paints it in the colour of the lam's span. In my test, the whole «لا» appeared the moment the lam landed, and while the lam was still transparent the alef was invisible too. Half a glyph can't be animated.

Bug 3: the gold sweep doubled

My headline has a gold sweep on top: a second, plain-text copy of the headline, revealed by a moving band. Plain text and spans now disagreed by 43 pixels on the widest line, so they spaced and balanced differently, and the gold letters no longer sat on the white ones.

The word is the smallest safe unit

A word is the smallest piece of Arabic that carries its own shaping: its joins, its marks and its ligatures. So Arabic decodes word by word. English keeps character-by-character decoding, which reads well in Latin script and costs nothing extra.

const ARABIC = /\p{Script=Arabic}/u
const byWord = ARABIC.test(text)
// Spaces stay as plain text between the words.
const units = byWord ? text.split(/( )/).filter(Boolean) : graphemes(text)

My site tests for the main Arabic block, U+0600 to U+06FF, which covers its strings. \p{Script=Arabic} is the safer choice for text you don't control: it also matches the supplement and extended blocks and the presentation forms.

A string with any Arabic in it goes by word, Latin words included. «مهندس برمجيات Full Stack» decodes as four words.

The rhythm needed two adjustments, both in the component below:

A word takes three steps, so it doesn't flash past in the time of a single letter. And while it waits, it shows a short burst of glyphs about its own length, so the line keeps its texture.

End on plain text, or the gold won't line up

Per-word spans match plain text to within 0.03px on my test page. Plain text matches it exactly, with one text node in place of a span per word. So when the effect ends, the spans go away entirely.

The settled headline is now shaped exactly like plain text, ligatures and all, and the gold sweep, also plain text, lines up with it pixel for pixel.

if (settled) return <span className="decode is-done">{text}</span>

The sweep waits for that moment with a class, never a :has(). Why a :has() there doubled my idle CPU is in the performance article.

The sweep itself is two transforms. A band 30% of the headline wide slides one way, and inside it a full-width plain-text copy slides back by the same distance on the same clock, so the gold letters stay on the white ones. Its geometry uses physical left on purpose. In Arabic, only the text inside should flip. The band's path stays put.

Right to left, for free

The decode runs right to left in Arabic without a single special case.

The units are in DOM order, which is logical order: the order you type and read. Index 0 is the first word a reader sees, on the right. The loop walks the indexes, so the cursor moves the way the reader's eye does.

The glyph overlay needs no direction rules either. It sits at left: 50% with translate: -50% 0, centred on its unit in either direction.

Numbers are the one place to be explicit. Digits inside Arabic text run left to right, so the figures under my headline sit in their own dir="ltr" elements and decode through digit glyphs instead of code symbols.

Keeping it cheap

This effect runs a requestAnimationFrame loop, which my performance article treats as a warning sign. The difference is that this loop ends.

• One loop per text, touching only the units inside the scrambling window. It breaks out at the first unit beyond the window, and it stops when the last one lands.
• New glyphs every 60 ms, about 16 a second: fast enough to read as noise, slow enough to see the symbols.
• Only the units scrambling at that moment get position: relative, which the glyph needs. A positioned span becomes its own paint layer, and a hundred of them made every frame of the decode heavy.
• Burst lengths for Arabic words are worked out once. Running Intl.Segmenter on every shuffle for every waiting word showed up in the profile.
• With reduced motion, the text settles at once.

Measured A/B on a phone profile with the CPU slowed four times, the whole effect costs about 0.35 s of main-thread work spread over the first two seconds, and nothing after that.

The component, ready to paste

Here is the component from my site, trimmed to the parts that matter. It needs TypeScript's ES2022 lib or later for Intl.Segmenter, and it falls back to code points where the browser lacks it.

import { useEffect, useMemo, useRef, useState } from 'react'

const GLYPHS = '{}[]()<>/=+-*&|!?#$%@;:01λΣ'
const ARABIC = /\p{Script=Arabic}/u

/** User-perceived characters, so a letter keeps its marks. */
const graphemes = (text: string): string[] =>
  typeof Intl.Segmenter === 'function'
    ? Array.from(new Intl.Segmenter(undefined, { granularity: 'grapheme' }).segment(text), (s) => s.segment)
    : Array.from(text)

type Props = {
  text: string
  delay?: number // ms before the first unit starts
  step?: number // ms per character; an Arabic word takes three steps
  window?: number // how many units ahead of the cursor are scrambling
  onSettled?: () => void
}

export default function DecodeText({ text, delay = 0, step = 20, window: span = 5, onSettled }: Props) {
  const ref = useRef<HTMLSpanElement>(null)
  const onSettledRef = useRef(onSettled)
  useEffect(() => {
    onSettledRef.current = onSettled
  })
  const byWord = ARABIC.test(text)
  // Arabic by word (spaces stay plain text between words), anything else by grapheme.
  const units = useMemo(() => (byWord ? text.split(/( )/).filter(Boolean) : graphemes(text)), [text, byWord])
  const [settledText, setSettledText] = useState<string | null>(null)
  const settled = settledText === text

  useEffect(() => {
    const root = ref.current
    if (!root || settled) return
    const cells = Array.from(root.querySelectorAll<HTMLElement>('.dc'))
    const settle = () => {
      setSettledText(text)
      onSettledRef.current?.()
    }
    if (window.matchMedia('(prefers-reduced-motion: reduce)').matches) {
      settle()
      return
    }

    const noise = (n: number) => Array.from({ length: n }, () => GLYPHS[(Math.random() * GLYPHS.length) | 0]).join('')
    // A word's burst is about its own length: worked out once, never per shuffle.
    const burst = cells.map((c) => (byWord ? Math.min(7, Math.max(2, graphemes(c.textContent ?? '').length)) : 1))
    const pace = byWord ? step * 3 : step
    const start = performance.now() + delay
    let raf = 0
    let timer = 0
    let lastShuffle = 0

    const tick = (now: number) => {
      const t = now - start
      const shuffle = now - lastShuffle > 60 // about 16 new glyphs a second
      if (shuffle) lastShuffle = now
      let landed = 0
      for (let i = 0; i < cells.length; i++) {
        const cell = cells[i]
        const at = i * pace
        if (t >= at) {
          if (!cell.classList.contains('is-on')) {
            cell.classList.remove('is-scr')
            cell.classList.add('is-on')
          }
          landed++
        } else if (t >= at - span * pace) {
          if (shuffle || !cell.classList.contains('is-scr')) {
            cell.dataset.g = noise(burst[i])
            cell.classList.add('is-scr')
          }
        } else break // everything after this is still waiting
      }
      if (landed < cells.length) raf = requestAnimationFrame(tick)
      else timer = window.setTimeout(settle, 200) // then hand back plain text
    }
    raf = requestAnimationFrame(tick)
    return () => {
      cancelAnimationFrame(raf)
      window.clearTimeout(timer)
    }
  }, [text, units, byWord, delay, step, span, settled])

  const className = settled ? 'decode is-done' : 'decode'
  // Settled: one plain string, shaped like any other text on the page.
  if (settled) return <span className={className}>{text}</span>
  // Keyed by the text, so a language switch mounts fresh, hidden units.
  return (
    <span key={text} ref={ref} className={className}>
      {units.map((unit, i) => (unit === ' ' ? ' ' : <span key={i} className="dc">{unit}</span>))}
    </span>
  )
}