DEV Community

Roberto Luna
Roberto Luna

Posted on

Fixing Unicode Handling in the Content‑Automation Pipeline: From Silent Failure to Robust JSON Generation

#playadev #buildinpublic #productivity

Fixing Unicode Handling in the Content‑Automation Pipeline: From Silent Failure to Robust JSON Generation

TL;DR: I discovered that a stray “ñ” in a markdown file broke the content‑automation pipeline without throwing an error. By normalizing all inputs to UTF‑8 and adding explicit encoding checks, the generator now produces valid JSON for Medium, Substack, and Bluesky every day.

The Problem

My daily content‑automation script pulls markdown files from content/2026/11/07/content-automation/, converts them to platform‑specific payloads (Medium, Substack, Bluesky), and pushes the JSON to the respective APIs. On 2026‑11‑07 the bot stopped publishing the Spanish post; the console showed nothing, and the generated JSON files (bluesky_es.json, medium_es.md, etc.) were empty.

The root cause was a single Unicode character – the letter “ñ” in the headline of content/2026/11/07/content-automation/medium_es.md – that got written to the file with a UTF‑16 BOM while the rest of the pipeline expected UTF‑8. When Node’s fs.readFileSync read the file without specifying an encoding, it returned a Buffer that contained the BOM bytes (0xFF 0xFE). The subsequent JSON.stringify call produced an invalid payload that the API silently rejected, leaving no stack trace.

Error‑like symptom (captured from the API logs after the fix):

POST https://api.medium.com/v1/posts 400 Bad Request
Response: {"errors":[{"message":"Invalid JSON payload"}]}
Enter fullscreen mode Exit fullscreen mode

Before the fix there was no error logged because the pipeline never reached the HTTP request – the JSON file never got written.

What I Tried First

My first instinct was to wrap the file read in a try/catch and log any exception:

try {
  const raw = fs.readFileSync(mdPath);
  const content = raw.toString(); // <-- implicit UTF‑8 conversion
  // …
} catch (err) {
  console.error('Read error:', err);
}
Enter fullscreen mode Exit fullscreen mode

Running this on the problematic file still produced an empty string (content === ''). No exception was thrown, so the catch block never executed. I also tried forcing UTF‑8:

const raw = fs.readFileSync(mdPath, 'utf8');
Enter fullscreen mode Exit fullscreen mode

But the BOM remained at the start of the string ('\uFEFF'), and the downstream markdown parser (markdown-it) failed to recognize the heading, resulting in an empty HTML fragment that later got stripped out.

I then attempted to strip the BOM manually:

if (content.charCodeAt(0) === 0xFEFF) {
  content = content.slice(1);
}
Enter fullscreen mode Exit fullscreen mode

That worked for this single file, but it was a band‑aid: any future file with a different BOM (UTF‑16LE/BE) would still break, and the code was duplicated across three platform generators.

The Implementation

1. Centralize File Loading with Normalization

I created a utility src/utils/readMarkdown.js that guarantees UTF‑8, removes any BOM, and validates that the result is non‑empty:

// src/utils/readMarkdown.js
const fs = require('fs');

/**
 * Reads a markdown file, normalizes encoding to UTF‑8,
 * strips BOM if present, and returns a trimmed string.
 * Throws if the file is empty after normalization.
 */
function readMarkdown(filePath) {
  // Explicitly read as UTF‑8 buffer
  const buffer = fs.readFileSync(filePath, { encoding: 'utf8' });

  // Remove any leading BOM (U+FEFF)
  const content = buffer.replace(/^\uFEFF/, '').trim();

  if (!content) {
    throw new Error(`Markdown file ${filePath} is empty after normalization`);
  }

  return content;
}

module.exports = { readMarkdown };
Enter fullscreen mode Exit fullscreen mode

All generators now import this helper:

// src/pipeline/generatePayload.js
const { readMarkdown } = require('../utils/readMarkdown');

function buildMediumPayload(mdPath, metadata) {
  const markdown = readMarkdown(mdPath);
  // … continue with markdown-it conversion
}
Enter fullscreen mode Exit fullscreen mode

2. Update Platform Generators

Medium 

--- a/src/pipeline/medium.js
+++ b/src/pipeline/medium.js
@@ -1,7 +1,9 @@
 const { readMarkdown } = require('../utils/readMarkdown');
 const MarkdownIt = require('markdown-it');
 const md = new MarkdownIt();

 function generateMediumPost(mdPath, meta) {
-  const raw = fs.readFileSync(mdPath, 'utf8');
-  const html = md.render(raw);
+  const markdown = readMarkdown(mdPath);
+  const html = md.render(markdown);
   return {
     title: meta.title,
     contentFormat: 'html',
Enter fullscreen mode Exit fullscreen mode

Substack 

--- a/src/pipeline/substack.js
+++ b/src/pipeline/substack.js
@@ -3,8 +3,9 @@
 const fs = require('fs');
 const { readMarkdown } = require('../utils/readMarkdown');

 function generateSubstackEmail(mdPath, meta) {
-  const raw = fs.readFileSync(mdPath, 'utf8');
-  const body = raw.replace(/\n/g, '<br>');
+  const markdown = readMarkdown(mdPath);
+  const body = markdown.replace(/\n/g, '<br>');
   return {
     subject: meta.subject,
     body,
Enter fullscreen mode Exit fullscreen mode

Bluesky 

--- a/src/pipeline/bluesky.js
+++ b/src/pipeline/bluesky.js
@@ -2,7 +2,8 @@
 const { readMarkdown } = require('../utils/readMarkdown');

 function generateBlueskyPost(mdPath, meta) {
-  const raw = fs.readFileSync(mdPath, 'utf8');
+  const markdown = readMarkdown(mdPath);
+  const raw = markdown; // keep variable name for downstream API shape
   return [
     {
       type: 'post',
Enter fullscreen mode Exit fullscreen mode

3. Enforce UTF‑8 in Git

To prevent future BOMs, I added a .gitattributes entry:

*.md text eol=lf
*.json text eol=lf
Enter fullscreen mode Exit fullscreen mode

Git now normalizes line endings and forces UTF‑8 text handling on checkout.

4. Update metadata.json Flags

The pipeline tracks which platforms have been generated for the day. After the fix I set the flags correctly in content/2026/11/07/content-automation/metadata.json:

{
  "date": "2026-11-07",
  "medium_generated": true,
  "substack_generated": true,
  "bluesky_generated": true,
  "pull_requests": 0,
  "releases": 0,
  "closed_issues": 0
}
Enter fullscreen mode Exit fullscreen mode

The diff that landed with the commit:

--- a/content/2026/11/07/content-automation/metadata.json
+++ b/content/2026/11/07/content-automation/metadata.json
@@ -12,8 +12,8 @@
   "pull_requests": 0,
   "releases": 0,
   "closed_issues": 0,
-  "medium_generated": false,
-  "substack_generated": false,
+  "medium_generated": true,
+  "substack_generated": true,
   "bluesky_generated": true
 }
Enter fullscreen mode Exit fullscreen mode

5. Add Unit Test for BOM Stripping 


js
// tests/readMarkdown.test.js
const { readMarkdown } = require('../src/utils/readMarkdown');
const fs = require('fs');
const path = require('path');

test('readMarkdown removes BOM and returns trimmed content', () => {
  const fixture = path.join(__dirname, 'fixtures',

---

*Part of my [Build in Public](https://dev.to/zaerohell) series — sharing the real process of building SaaS projects from Playa del Carmen, México.*

*Repo: `zaerohell/content-automation` · 2026-11-08*

\#playadev #buildinpublic
Enter fullscreen mode Exit fullscreen mode

Top comments (0)