Learning Modern Package Development: Monorepos and Backends

This article is a work in progress. Feel free to read it, but some sections are incomplete.

Introduction

Unlighthouse is an open-source package I built to scan your entire site using Google Lighthouse.

Building it was chaotic, with day-long bugs, constant refactoring and endless documentation reading.

Through building it, I learnt modern development practices, making use of the vast ecosystem of packages and tools.

Background: Why build Unlighthouse?

As a freelancer I keep on top of my clients organic growth with Google Search Console.

Was a day like any other, looking at one of my clients' dashboard. Seemingly out of nowhere, I saw the trend of page position, clicks and page views in free fall. My clients' income was based on organic traffic, not good.

Isolating the reason for the falling page rank wasn't easy. The site had issues, but what was causing the free fall? There was no easy way to know.

To diagnose the issue, I used Google Lighthouse. I went through all pages of the site, noticing quite a number of issues. I spent a couple of days fixing them all up and improving the general performance of the site.

What happened next? Things started turning around. I was able to invert the graph. Organic growth doubled in the next few months. Happy client.

Now that was out of the way, how could I make it easier to stay on top of the health of the sites I manage?

Deciding on the stack

I needed to build a tool that would run Google Lighthouse on an entire site with just the home page URL.

I had a plan of attack for the build.

The backend would be build using Typescript and Node.

The frontend client would be built using Vue and Vite.

But how would I be able to design this in a way that was easy to build and maintain?

I had seen the amazing work coming out of the UnJS ecosystem and knew that they could solve some of my problems.

With that, the package would be known as Un (inspired by Unjs) Lighthouse.

Keeping a keen eye on other modern packages coming out, I took some of the best practices and tools I saw implemented.

The stack was split into three core parts:

Monorepo: containing the dependencies to build, test and deploy the code
Frontend: displaying searchable, filtering and sortable results
Backend: generating the frontend, running the scans and providing an API for the frontend

Monorepo

Implementing a monorepo is a keystone for a large project which will ship multiple packages. It allows you to group up logic, dependencies and documentation into a single repository.

PNPM

PNPM is the new kid on the block of node package managers and has gained a large following quickly, for good reason. It is the most performant package manager and has first class support for monorepos.

There are many benefits to using a monorepo for a package. My personal favourite is it allows me to easily isolate logic and dependencies for your package, letting you write simpler code. Allowing end users to pull any specific part of your package that they want to use.

Vitest

Vitest is also the new kid on the block of testing. It's original aim was to be a testing framework specifically for Vite, but it has ended up being a possible replacement for Jest entirely.

Vitest makes writing your logic and tests a breeze and I'd recommend checking it out for any project.

Unbuild

This package is described as a "A unified javascript build system".

In reality, it's a minimal config way to build your package code to ESM and CJS.

One of the amazing features of unbuild is stubbing. This allows you can run source code from your dist folder, meaning it transpiles just-in-time.

This allows you to completely cut out the build step when you're iterating and testing integrations on your package.

It's as simple as unbuild --stub.

build.config.ts

import { defineBuildConfig } from 'unbuild'

export default defineBuildConfig({
  entries: [
    { input: 'src/index' },
    { input: 'src/process', outDir: 'dist/process', builder: 'mkdist', declaration: false },
  ],
})

GitHub Repo Link

Server

Lighthouse Binary

Unlighthouse wouldn't be possible if Google hadn't published Lighthouse as its own NPM binary.

To make Unlighthouse fast, I combined the binary with the package puppeteer-cluster, which allows for multi-threaded lighthouse scans.

Unctx

It's amazing that a simple pattern like composition has evaded Node packages for so long.

With the introduction of Vue 3, composition became cool. And with that, unctx is composition for your own package.

unctx allows you to define a scope where there's only a single instance of something that is globally accessible. This is incredibly useful for building packages, as you no longer need to be juggling core state. You can build your logic out as composables that interact with the core.

import { createContext } from 'unctx'

const engineContext = createContext<UnlighthouseContext>()

export const useUnlighthouse = engineContext.use as () => UnlighthouseContext

export async function createUnlighthouse(userConfig: UserConfig, provider?: Provider) {
  // ...
  engineContext.set(ctx, true)
}

unctx GitHub Repo

Hookable

For Nuxt.js users, you might be familiar with the concept of frameworks hooks. A way for you to modify or do something with the internal logic of Nuxt.

Building a package, I knew that this was a useful feature, not just for end-users, but for me as a way to organise logic.

Having a core which is hookable means you can avoid baking logic in that may be better suited elsewhere.

For example, I wanted to make sure that Unlighthouse didn't start for integrations until they visited the page.

I simply set a hook for it to start only when they visit the client.

hooks.hookOnce('visited-client', () => {
  ctx.start()
})

Hookable GitHub Repo

Unconfig

Unconfig is a universal solution for loading configurations. This let me allow the package to load in a configuration from unlighthouse.config.ts or a custom path, with barely any code.

import { loadConfig } from 'unconfig'

const configDefinition = await loadConfig<UserConfig>({
  cwd: userConfig.root,
  sources: [
    {
      files: [
        'unlighthouse.config',
        // may provide the config file as an argument
        ...(userConfig.configFile ? [userConfig.configFile] : []),
      ],
      // default extensions
      extensions: ['ts', 'js'],
    },
  ],
})
if (configDefinition.sources?.[0]) {
  configFile = configDefinition.sources[0]
  userConfig = defu(configDefinition.config, userConfig)
}

Unconfig GitHub Repo

ufo

Dealing with URLs in Node isn't nice. For Unlighthouse I needed to deal with many URLS, I needed to make sure they were standardised no matter how they were formed.

This meant using the ufo package heavily. The slash trimming came in handy and the origin detection.

export const trimSlashes = (s: string) => withoutLeadingSlash(withoutTrailingSlash(s))

const site = new $URL(url).origin

ufo GitHub Repo

Unrouted

I needed an API for the client to communicate with the Node server to fetch the status of the scan and submit re-scans.

The current JS offerings were a bit lackluster. I wanted something that just worked and had a nice way to use it.

I ended up building unrouted as a way to solve that.

group('/api', () => {
  group('/reports', () => {
    post('/rescan', () => {
      // ...
      return true
    })

    post('/:id/rescan', () => {
      const report = useReport()
      const { worker } = useUnlighthouse()

      if (report)
        worker.requeueReport(report)
    })
  })

  get('__launch', () => {
    const { file } = useQuery<{ file: string }>()
    if (!file) {
      setStatusCode(400)
      return false
    }
    const path = file.replace(resolvedConfig.root, '')
    const resolved = join(resolvedConfig.root, path)
    logger.info(`Launching file in editor: \`${path}\``)
    launch(resolved)
  })

  get('ws', req => ws.serve(req))

  get('reports', () => {
    const { worker } = useUnlighthouse()

    return worker.reports().filter(r => r.tasks.inspectHtmlTask === 'completed')
  })

  get('scan-meta', () => createScanMeta())
})

Unrouted GitHub Repo

Learning Modern Package Development: Monorepos and Backends - Part 1