• Home
  • Blog
  • Projects
  • Bookshelf
  • About
← Back to all posts

Building Better Beacon

Published 📅: ....
Last modified 📝: ....

Share this post on BlueskySee discussion on Bluesky

Over the past year or so we've been rebuilding a core library that supports tracking business KPI and metrics across our various web and native app experiences.

This involved quite a bit of work from a variety of teams, I acted as one of the lead architects in designing the web integration for the new system, and worked with my team to build it out and get the first iteration in production.

It had been smooth sailing for several months across the first few experiences that adopted the new version of the core library, at least until the other week where we identified a show stopping bug.

Background

The library leverages navigator.sendBeacon to post event payloads to one of our internal processing gateways, we use this data to get an overall sense of how users are browsing and interacting with our sites, and it powers key business metrics like our visits, add to cart rate and our overall conversion.

We chose sendBeacon because of it's slightly stronger guarantee that the browser won't drop the requests when a page is unloaded or navigated away from. This gave us confidence that we could send richer content (via JSON payloads) than our previous system, and also allow us to track key interactions before they resulted in the customer navigating to another page.

Our previous iteration of this system, which was about 6 years old or so, heavily relied upon cookies and sending data via image requests (which greatly inhibited the size and fidelity of the data we could collect).

The Bug

We've been adopting this new core library on several experiences as of late, and on one of them we noticed that we were queuing up a large number of these events via sendBeacon, so many in fact that the browser would effectively stall and no longer send the requests.

In the DevTools Network panel they'd all show up as (pending) without any clear indicator on why they weren't being flushed.

At first, we thought it could be due to some limit on the total number of connections being made, but there seemed to be no other network requests that would similarly get stuck.

We did some research and found out a painful little caveat with sendBeacon (and one that also applies to fetch keepalive which we considered as an alternative also):

  1. The payload to be sent must be smaller that 64kb (for Chrome, seems other browsers also use a similar limit) source
  2. The total size of the queued payloads to be sent must be smaller than 64kb

We checked our pending requests and none of them got close to the 64kb limit, we then tallied up our total requested payload size and got a bit worried.

Our first read through of the second limitation was that it was 64kb in total for the lifetime of a page. In that case it'd only take about a hundred or so requests (with an average size of ~500b) before we hit that cap.

Honestly, we were pretty concerned that we'd need to go back to the drawing board and consider alternative ways to send these payloads.

Good Fortune

Fortunately however, some of my teammates re-read the limitations of sendBeacon, and found further evidence that the second limitation only applied to queued requests, not completed requests. So we would only hit this pending state for requests if there are other requests in-flight and the size of their payload in total is over that 64kb limit.

This is where the teammates had a bright idea, we could retry these requests a little bit later if they get stuck in the pending state. Fortunately again, the navigator.sendBeacon api returns a boolean indicating if the browser can send the request, if it is false then that means we have hit either size limitation.

So if we encounter any such request, we can wait a little bit (via setTimeout, requestAnimationFrame, etc), and then retry it.

We opted for a queue, so we can enqueue any pending requests, and otherwise pop them from the queue if they can be sent.

Resolution

In the end, we opted for an implementation that seemed to work quite well.

I've taken the patterns we worked on internally and implemented them within this public NPM package so others don't run into the same issue as we did: better-beacon. The code for the library is available on my GitHub as well.

Tags:

DevelopmentWeb DevelopmentProjects

Loading...

Related Posts

Development

Onboarding Your New AI Teammate

Published: ....

Are we reinventing the wheel when it comes to onboarding new AI agents to a codebase, when we already have primitives available for onboarding humans?

Tip: GitHub Created Date Filtering

Published: ....

Progressive Identifiers

Published: ....

Roundup Notes in Obsidian with Dataview

Published: ....

A quick tip for creating roundup or summary notes based on other notes in Obsidian using the Dataview plugin!

Quick Tip - Theme Aware Images

Published: ....

Have you ever found the need to change the image you render on a web page based on the current preferred color scheme of your theme?

Website Redesign v10

Published: ....

I recently launched a rewrite and redesign of this personal website, I figured I'd talk a bit about the changes and new features that I added along the way!

Quick Tip - Specific Local Module Declarations

Published: ....

A quick tip outlining how to provide specific TypeScript type definitions for a local module!

You're Building Software Wrong

Published: ....

Slicing software: why vertical is better than horizontal.

Single File Web Apps

Published: ....

What if you could author an entire web application in a single file?

The AI Development Conundrum

Published: ....

Is it a good or a bad thing to offload writing code to AI agents and Large Language Models?

A Quick Look at Import Maps

Published: ....

A brief look at Import Maps and package.json#imports to support isomorphic JavaScript applications!

Recommended Tech Talks

Published: ....

A collection of tech talks that I regularly re-watch and also recommend to everyone!

Request for a (minimal) RSC Framework

Published: ....

Some features and functionality that I'd like within a React Server Component compatible framework.

Bluesky Tips and Tools

Published: ....

A (running) collection of Bluesky tips, tools, packages, and other misc things!

Building a Custom Ghostty Theme

Published: ....

How to generate a custom Ghostty theme based on any iterm2 theme!

Offload Complexity, Don't Offload Learning

Published: ....

A rough mental model for how you should be leveraging AI as a tool for your own growth

More Thoughts on Dogfooding

Published: ....

Even more thoughts on dogfooding!

Dogfooding

Published: ....

The secret to excellent product teams is using your own product, and often!

Git Notes as a Tool for Thought

Published: ....

(Ab)using Git as yet another tool for thought!

My Current Dev Setup

Published: ....

A quick look at the applications and tools that I (generally) use day to day for web development!

There Is No Standard Markdown

Published: ....

There are a variety of different markdown "standards" out there, and sometimes they're not all that consistent

Tip: Request and Response Headers

Published: ....

There's a common gotcha when creating Web Request and Response instances with Headers!

Using Feature Toggles to De-risk Refactors

Published: ....

Feature toggles are often underused by most software development teams, and yet offer so much value during not only feature development but also refactors

Hohoro

Published: ....

A quick introduction to my new side project, hohoro. An incremental JS/TS library build tool!

Funport: True Dynamic Imports in webpack

Published: ....

webpack, and tools built on it like Next.js, don't support true dynamic imports, but I found a way to trick the system!

My Current AI Stack

Published: ....

I've been using a variety of AI tools as of late, I figured I'd document the ones I'm primarily using!

Configuring Cloudflare Domains with Vercel

Published: ....

I've started to use Cloudflare to manage my domains for several side projects, have had to jump through the same hooks multiple times that I figured I should document them here!

React Error Boundaries: Revisited

Published: ....

Revising my previous blog post on React Error Boundaries and my preferred go-to implementation!

Custom Favicon Recipes

Published: ....

Two neat tricks for enhancing your site's favicon!

Corporate Sponsored OSS

Published: ....

The various risks and pitfalls of open source software run by corporations.

The Library-Docs Monorepo Template

Published: ....

A monorepo template for managing a library and documentation together.

Churn Anxiety

Published: ....

When did semver major changes become so scary?

Stop Snacking

Published: ....

No I don't mean those Milano cookies you keep taking from the office snack wall either (although you should probably stop snacking on those as often as well).

Pair Programming

Published: ....

Pair programming can be good sometimes - but not all the time

Taking a Break

Published: ....

A few quick thoughts on burn out and taking a break

Web Development

Onboarding Your New AI Teammate

Published: ....

Are we reinventing the wheel when it comes to onboarding new AI agents to a codebase, when we already have primitives available for onboarding humans?

Quick Tip - Theme Aware Images

Published: ....

Have you ever found the need to change the image you render on a web page based on the current preferred color scheme of your theme?

Async Class Creation In JavaScript

Published: ....

Have you ever wanted to create a class in JavaScript or TypeScript but also have the initialization be async? Here's a quick tip on a pattern that I've used in the past!

Projects

Hohoro

Published: ....

A quick introduction to my new side project, hohoro. An incremental JS/TS library build tool!

The Library-Docs Monorepo Template

Published: ....

A monorepo template for managing a library and documentation together.

Project Deep Dive: Tails

Published: ....

A(nother) deep dive into one of my recent side projects; tails - a plain and simple cocktail recipe app.

Project Deep Dive: Microfibre
Website Redesign v10

Published: ....

I recently launched a rewrite and redesign of this personal website, I figured I'd talk a bit about the changes and new features that I added along the way!

Server Side Rendering Compatible CSS Theming

Published: ....

A quick tip to implementing CSS theming that's compatible with Server Side Rendered applications!

Podcasting By Hand

Published: ....

A brief overview on how we launched The Bikeshed Podcast, including a deep dive in our recording and distribution workflows!

Quick Tip - Specific Local Module Declarations

Published: ....

A quick tip outlining how to provide specific TypeScript type definitions for a local module!

On File-System Routing Conventions

Published: ....

Some rough thoughts on building a file-system routing based web application

You're Building Software Wrong

Published: ....

Slicing software: why vertical is better than horizontal.

Single File Web Apps

Published: ....

What if you could author an entire web application in a single file?

Resetting Controlled Components in Forms

Published: ....

A quick way to handle resetting internal state in components when a parent form is submitted!

A Quick Look at Import Maps

Published: ....

A brief look at Import Maps and package.json#imports to support isomorphic JavaScript applications!

Recommended Tech Talks

Published: ....

A collection of tech talks that I regularly re-watch and also recommend to everyone!

Request for a (minimal) RSC Framework

Published: ....

Some features and functionality that I'd like within a React Server Component compatible framework.

Bluesky Tips and Tools

Published: ....

A (running) collection of Bluesky tips, tools, packages, and other misc things!

The Bookkeeping Pattern

Published: ....

A quick look at a small but powerful pattern I've been leveraging as of late!

TSLite

Published: ....

A proposal for a minimal variant of TypeScript!

Monorepo Tips and Tricks

Published: ....

Sharing a few core recommendations when working within monorepos to make your life easier!

Next.js with Deno v2

Published: ....

This is a quick post noting that Next.js should now work with Deno v2!

Don't Break the Implicit Prop Contract

Published: ....

React components have a fundamental contract that is often unstated in their implementation, and you should know about it!

A Better useSSR Implementation

Published: ....

Replace that old useState and useEffect combo for a new and better option!

My Current Dev Setup

Published: ....

A quick look at the applications and tools that I (generally) use day to day for web development!

There Is No Standard Markdown

Published: ....

There are a variety of different markdown "standards" out there, and sometimes they're not all that consistent

Abstract Your API

Published: ....

Proposing a solution for sharing core "business" logic across services!

Tip: Request and Response Headers

Published: ....

There's a common gotcha when creating Web Request and Response instances with Headers!

Using Feature Toggles to De-risk Refactors

Published: ....

Feature toggles are often underused by most software development teams, and yet offer so much value during not only feature development but also refactors

Hohoro

Published: ....

A quick introduction to my new side project, hohoro. An incremental JS/TS library build tool!

Custom Favicon Recipes

Published: ....

Two neat tricks for enhancing your site's favicon!

Corporate Sponsored OSS

Published: ....

The various risks and pitfalls of open source software run by corporations.

The Library-Docs Monorepo Template

Published: ....

A monorepo template for managing a library and documentation together.

Project Deep Dive: Tails

Published: ....

A(nother) deep dive into one of my recent side projects; tails - a plain and simple cocktail recipe app.

Churn Anxiety

Published: ....

When did semver major changes become so scary?

Service Monitors and Observability

Published: ....

Leveraging service monitors properly to improve service observability.

On Adopting CSS-in-JS

Published: ....

A brief recap of how Wayfair changed it's CSS approach not once but twice in the span of 5 years!

Project Deep Dive: Microfibre

Published: ....

A deep dive into one of my recent side projects; microfibre - a minimal text posting application

Pair Programming

Published: ....

Pair programming can be good sometimes - but not all the time

Suspense Plus GraphQL

Published: ....

A few thoughts on using Suspense with GraphQL to optimize application data loading

You've Launched, Now What?

Published: ....

A few thoughts on what to do after you launch a new project

Taking a Break

Published: ....

A few quick thoughts on burn out and taking a break

Managing Complex UI Component State

Published: ....

A few thoughts on managing complex UI component state within React

Understanding React 16.3 Updates

Published: ....

A quick overview of the new lifecycle methods introduced in React 16.3

CSS in JS

Published: ....

A few thoughts and patterns for using styled-jsx or other CSS-in-JS solutions

Redesign v6

Published: ....

A few thoughts on the redesign of my personal site, adopting Next.js and deploying via Now

JavaScript Weirdness

Published: ....

A few weird things about JavaScript

Calendar

Published: ....

Building a calendar web application

Published: ....

A deep dive into one of my recent side projects; microfibre - a minimal text posting application