Building My First PWA

I’ve been looking for time and a sandbox to sit down and learn how to build offline-capable/first web sites or “Progressive Web Apps” (PWA) for a while. I learn best with a hands-on approach. ColorMe is a site I maintain and a perfect candidate for offline experimentation. This post details the steps I took, the issues I ran into, and things I learned building my first PWA.

Quick note. This isn’t a general “How to make a PWA” post. It may not even be a good intro. There are plenty of articles and tutorials to get the basics. This post is specific to the work involved in making ColorMe a PWA.

In a lot of what I’ve read about PWAs, there’ve been common steps for building them. A simple-on-the-surface looking process. Here’s one list from Jeremy:

  1. switch over to HTTPS,
  2. add a JSON manifest file with your metacrap, and
  3. add a service worker.
Jeremy Keith, “Progressing the Web.”

OK, that seems easy enough. Except that last step. That seems like it could be a lot to unpack. He does offer a disclaimer:

That last step can be tricky if you’re new to service workers, but it’s not unsurmountable.

I’ll ignore the service worker part of the process for now and focus on the first two items.

Using HTTPS

ColorMe already has HTTPS in place. I host it on S3 and serve it through CloudFront. I used Amazon’s Certificate Manager to add an SSL certificate. HTTPS is in place and has been since launch.

Adding a Manifest

This is where things got more interesting. I built ColorMe with Create React App (CRA) so I made a manifest.json file in the public directory. The same directory as the favicon and index.html. Anything in the public directory gets copied to the build directory as is. That’s what we need for manifest.json.

Doesn’t CRA do PWA stuff out of the box?

Yes. CRA added built-in support for PWAs in version 1.0.0. ColorMe is still on version 0.8.4. That was the latest version when I created the project and haven’t had a reason to update.

I could have updated CRA to use the built-in PWA, but I didn’t want to miss the opportunity to learn step-by-step. Doing this myself, taking the long, “dumb” way, helped internalize the why and the how of each step. On future CRA-built projects, I’ll use the latest version with built-in PWA support. With that out of the way, here’s all my “metacrap.”

I knew there was “stuff” that went in the manifest, but I wasn’t sure about specifics. What keys can I use? What are example values for each key? And what key-values am I supposed to have? The best resource I found for questions number one and two is the MDN Web App Manifest documentation. It lists available keys and example values.

Lighthouse

For the third question, I turned to Lighthouse audits in Chrome dev tools. Before adding anything to the manifest, I ran a PWA audit. It reported items needed to meet the minimum requirements for a PWA (according to the audit).

A pre-pwa audit of colorme.io

With the audit report as a starting point, I hit each item on the list. Most important, I referenced the manifest in the head of index.html. This uses the CRA-specific %PUBLIC_URL%. 

<link rel="manifest" href="%PUBLIC_URL%/manifest.json">

Most items in the manifest are straightforward enough so I won’t go line-by-line. But I will call out a couple items that took a bit more work. You can see the complete file on GitHub and below: 

{
  "background_color": "#ffffff",
  "theme_color": "#B50003",
  "display": "standalone",
  "short_name": "ColorMe",
  "name": "ColorMe",
  "start_url": "/",
  "icons": [
    {
      "src": "launcher-icon-48x48.png",
      "type": "image/png",
      "sizes": "48x48"
    },
    {
      "src": "launcher-icon-96x96.png",
      "type": "image/png",
      "sizes": "96x96"
    },
    {
      "src": "launcher-icon-192x192.png",
      "type": "image/png",
      "sizes": "192x192"
    },
    {
      "src": "launcher-icon-256x256.png",
      "type": "image/png",
      "sizes": "256x256"
    },
    {
      "src": "launcher-icon-384x384.png",
      "type": "image/png",
      "sizes": "384x384"
    },
    {
      "src": "launcher-icon-512x512.png",
      "type": "image/png",
      "sizes": "512x512"
    }
  ]
}

Icon sizes

As you can see in the manifiest, I included six different icon sizes. The audit requires two sizes; 192x192 and 512x512. The former for the homescreen icon on Android, the latter as an icon for a splash screen on Android.

I’m not sure if the other four sizes are necessary, but I saw those in examples so I figured it wouldn’t hurt to include them.

Theme color

A requirement–per the audit–was to add a theme-color meta tag to index.html: 

<meta name="theme-color" content="#B50003">

The theme-color meta tag ensures that the address bar is branded when a user visits your site as a normal webpage.

noscript

Another failing audit was “Contains Some Content When JavaScript Is Not Available.” For that, I added noscript content. It doesn’t do anything except apologize for not working without JavaScript. It would be better to have some type of useful experience without JS, but I’ll save that for another time. It’s interesting to think how to make a site like this provide value without JavaScript.

“According to the audit”

I’m using specific language like; “per the audit” and “according to the audit” on purpose here. Some of these aren’t universal. For example, the theme_color property and the theme-color meta tag have no effect on Mobile Safari or Mobile Chrome on iOS as far as I can tell. For the purposes of this exercise, I’m working towards 100% on the audit. I’m sure not every project needs every item. As usual, it depends.

The Chrome Dev Tools Manifest Tab

Every time I’d make a change to the manifest I’d re-run the PWA audit to check the results. This was slow. I didn’t realize there was a tab in Chrome dev tools for inspecting the manifest.json results. I found it early enough in the process that it helped speed things along. It also has an “Add to homescreen” button to test that mechanism. That’s much appreciated because I don’t have access to an Android device for proper testing. I wrote this in case someone else also doesn’t know about the manifest tab in Google Chrome Dev Tools.

At this point I still don’t have a PWA. The audit turns up one last failure:

Failures: Site does not register a Service Worker, Manifest start_url is not cached by a Service Worker.

With the basics of the manifest in place, I turned my attention to the service worker.

The Service Worker

Like I mentioned above, this was the biggest mystery for me. I understood the general concept of service workers, but I didn’t understand what the goals of a service worker for a PWA were. Sure, it’s JS, it runs in the background, but what’s that JS supposed to do? After spending time with tutorials, examples, and fiddling, I got a clearer picture.

The goals of ColorMe’s service worker:

  • Store the site’s static files–HTML, CSS, JavaScript, and images–in the window.caches object
  • Intercept all network requests. If the name of the requested file is in window.caches, respond with the cached file instead of making a request to the server
  • Delete stale caches when the cache key changes

One thing that stuck out early was the caches member on the global window scope. When I first saw caches in use in example service workers, I thought it was a global only in the service worker context. That’s not the case. window.caches is available from any JS.

Here’s a quick example of caches. Go to colorme.io. Open the developer console and run this snippet: 

caches.keys().then(names => {console.log(names)});

That should output ["colorme-v7"] (the version number might be different). Not much to look at, but you can see that window.caches is a thing in this context. That means you can access caches from any client side JavaScript, not only service workers. That’s pretty cool.

Goal 1: Cache Static Files

For ColorMe to work offline it needs to cache all critical static files. It’s a single page site so there are only a few; index.html, main.css, main.js, manifest.json, an svg image, and a Google Fonts stylesheet.

The CRA build process creates or renames the CSS, JS, and image files. That made things difficult and I’ll describe my process for fixing it later. For now, I’ll pretend the file names are what they are and walk through the code.

I created service-worker.js in the public directory. The full file is available on GitHub. 

const STATIC_CACHE_NAME = "colorme-v1";
const STATIC_URLS = [
  "/",
  "/index.html",
  "/manifest.json",
  "/static/css/main.css",
  "/static/js/main.js",
  "/static/media/bgTransparent.svg",
  "https://fonts.googleapis.com/css?family=Cousine:400|Karla:400,700"
];
self.addEventListener("install", event => {
  event.waitUntil(
    caches.open(STATIC_CACHE_NAME).then(cache => {
      return cache.addAll(STATIC_URLS);
    }).then(() => self.skipWaiting())
  );
});

STATIC_CACHE_NAME is a unique key for this cache. STATIC_URLS is the list of files to cache. I’ll explain how I update this list later to account for dynamic file names.

In broad strokes, the next lines say:

  1. when the service worker finishes the install process,
  2. find or create a cache with our name,
  3. and put the files we specified in that cache.

There are full descriptions of the install event and waitUntil available. MDN is a great one.

Skip Waiting?

I’ve read docs about skipWaiting, but I’m still a bit hazy on what it does or if I even need it in this context. Enough examples I saw recommended it, so I went with it for now. I’ll learn more about it as I work with it.

With those lines, ColorMe’s static assets are snug in a cache. There’s still work to be done though.

Goal 2: Serve Cached Files

Putting static files in caches isn’t enough on its own. For ColorMe to work offline, we need to tell the browser to look in the cache for those files. 

self.addEventListener("fetch", event => {
  event.respondWith(
    caches.match(event.request)
      .then(response => response || fetch(event.request))
  );
});

This snippet listens for all HTTP requests from the user’s browser. event.respondWith prevents the browser’s default fetch handling. That allows us to check if the requested URL–event.request–is in the cache. If it is, respond with the cached file. If it’s not cached, continue with the request to the server using fetch().

ColorMe now works with or without an Internet connection.

A post-pwa audit of colorme.io

Goal 3: Delete Stale Caches

If I stopped here ColorMe works offline, but I’d have no way to release updates to users. Cached items have to be deleted, they never expire. The service worker should remove stale caches. 

self.addEventListener("activate", event => {
  event.waitUntil(
    caches.keys().then(cacheNames => {
      return Promise.all(
        cacheNames
          .filter(name => name.includes("colorme") && name !== STATIC_CACHE_NAME)
          .map(name => caches.delete(name))
      )
    }).then(() => self.clients.claim())
  );
});

This is the most involved-looking code in the service worker, but it doesn’t do much.

  1. get an array of all cache keys with caches.keys(),
  2. remove any keys that don’t contain our name and remove a cache with the exact STATIC_CACHE_NAME,
  3. for each of the remaining keys, delete that cache.

When the service worker activates–every time the page loads–check to see if there are stale caches. If there are, delete them. I determine if a cache is stale with #2 above.

In the first code snippet I defined the cache key; const STATIC_CACHE_NAME = "colorme-v1". When I make changes to any of the cached files and deploy the site, I also change that version number. If a user visited the site when colorme-v1 was latest, then visits again when colorme-v7 is the latest, the service worker deletes v1 and caches v7.

Caching Dynamic Filenames

At this point, I’ve met all the goals for ColorMe’s service worker and have a functioning PWA. If you visit ColorMe then turn off your network connection, you should be able to refresh the page and use it as normal.

There is one issue I mentioned above. The filenames for the CSS, JS, and image are incorrect. In the earlier snippet, I cached files named;

  • "/static/css/main.css"
  • "/static/js/main.js"
  • "/static/media/bgTransparent.svg"

The CRA build process fingerprints those filenames to something like;

  • "/static/css/main.2ebebc14.css"
  • "/static/js/main.7e7a1a8f.js"
  • "/static/media/bgTransparent.e6317315.svg"

This is common for cachebusting and not only for CRAs. I’ve used and built tons of different asset pipelines over the years that use fingerprinting.

I got hung up on this big time. How am I going to tell the service worker to cache files when I have no control over their names? I didn’t find much info on the topic. This issue raises the question and the discussion helped me figure out an approach. I’d need to somehow generate the list of filenames during the build process. I had a few options, a couple of them not good ones;

  1. Run the CRA eject script and change the default webpack build process to generate the service worker
  2. Update to latest CRA and use their built-in PWA support
  3. Walk away from computer, sleep on it, think about a better solution the next morning while walking my dog

I didn’t want to eject from CRA, because it didn’t seem worth it. I mentioned above, updating to the latest CRA would take away the opportunity to learn. Option #3 sounded best. After getting away from the computer, I realized I had more options than I realized and a solution that should work. It was’t going to be elegant or scalable, but it’d do the job.

Piggybacking on the CRA build script

The default CRA build script builds the project according to the baked in webpack config. I can’t change the config–without ejecting–but I can add to the build script in package.json. I’d already modified the script to include NODE_PATH=src. That makes it easier to import modules without referencing the full path. ColorMe’s starting build script looked like: 

NODE_PATH=src react-scripts build

I knew a couple things. I knew the build process creates a file named asset-manifest.json. The contents of that file include the full fingerprinted names of all static assets used in the site. Example contents of the manifest file: 

{
  "main.css": "static/css/main.2ebebc14.css",
  "main.css.map": "static/css/main.2ebebc14.css.map",
  "main.js": "static/js/main.7e7a1a8f.js",
  "main.js.map": "static/js/main.7e7a1a8f.js.map",
  "static/media/bgTransparent.svg": "static/media/bgTransparent.e6317315.svg"
}

Those are the full filenames I need to cache with my service worker. I need to get those filenames into the service worker file.

The first thing I did was add to the build script. Back in package.json I updated the script to look like: 

NODE_PATH=src react-scripts build && npm run generate-sw

This says; “run the normal build process, when you’re done with that run this other npm script”. That script looks like: 

"generate-sw": "node scripts/generate-sw.js"

To make sure that worked, I created /scripts/generate-sw.js and added a single line; console.log('hello'). Then I ran the build script npm run build to make sure the project built and I saw “hello” in my terminal output. So far so good.

The generate script

I need to get the filenames out of asset-manifest.json and into the array of filenames to cache in service-worker.js. My plan was to not get fancy with this. I only need to take strings from one file and write them into another file. The fact that the target file is JavaScript is immaterial to this process.

The full file is available on GitHub and I’ll go through the code in detail here. 

const manifest = require("../build/asset-manifest.json");
const fs = require("fs");
const swPath = "build/service-worker.js";

First is setup. asset-manifest is JSON so I require it here for use as an object. I’ll use the fs package for reading and writing files. I store the path of the service worker for convenience. 

const manifest = require("../build/asset-manifest.json");
const fs = require("fs");
const swPath = "build/service-worker.js";
const urlsCSV = Object.keys(manifest)
  .filter(k => !k.includes(".map"))
  .map(k => manifest[k]);

Getting more interesting, but still not fancy. The goal of this chunk of code is to build an array of filenames. First, use Object.keys to get the keys from the manifest JSON to loop over an array.

Next, use filter to remove keys that include the string “.map.” If you look at asset-manifest.json you’ll see source maps. We don’t want to cache those. I’m not sure if there’s a best practice for or against that, but I decided it didn’t seem right for this project.

Now that we only have keys for the files we want to cache, use map to create the array of filenames stored as urlsCSV; 

[
  "static/css/main.2ebebc14.css",
  "static/js/main.7e7a1a8f.js",
  "static/media/bgTransparent.e6317315.svg"
]

A short tangent. Given the code above, you might be asking; “why didn’t you just use Object.values instead of Object.keys plus map?” That’s a great question with a quick answer. As of this writing, I’m running Node.js version 6.9.1. Object.values is not supported without the --harmony flag until version 7.0.0. I didn’t want to upgrade Node.js for this. I’ll do that another time. That’s all.

I need to get that array of filenames into the service worker file. Again, this isn’t meant to be fancy or scalable. It’s meant to do the work. 

const manifest = require('../build/asset-manifest.json');
const fs = require('fs');
const swPath = 'build/service-worker.js';
const urlsCSV = Object.keys(manifest)
  .filter(k => !k.includes('.map'))
  .map(k => manifest[k]);

fs.readFile(swPath, "utf8", (err, data) => {
  if (err) { return console.log("Error trying to read SW file", err); }

  const result = data.replace("%MANIFESTURLS%", JSON.stringify(urlsCSV));

  fs.writeFile(swPath, result, "utf8", err => {
    if (err) { return console.log("Error trying to write SW file", err); }
  });
});

Let’s break this down. First, open the service worker file (swPath) for reading. The error condition isn’t important. I included it to be nice to myself in case something odd happens during a build.

The next line is the point of this script. It searches the contents of the service worker file (data) for the unique string “%MANIFESTURLS%”. When found, it’s replaced with a JSON stringified version of our filenames array, urlsCSV. Then, the updated contents are written back to the service worker file.

Updates to service-worker.js

As mentioned above the generate script needs to find “%MANIFESTURLS%” in service-worker.js. I went back and updated the script to account for that. 

const STATIC_CACHE_NAME = "colorme-v1";
const BASE_STATIC_URLS = [
  "/",
  "/index.html",
  "/manifest.json",
  "https://fonts.googleapis.com/css?family=Cousine:400|Karla:400,700"
];
const STATIC_URLS = BASE_STATIC_URLS.concat(JSON.parse('%MANIFESTURLS%'));

// The install handler is the same as when we started.
self.addEventListener("install", event => {
event.waitUntil(
caches.open(STATIC_CACHE_NAME).then(cache => {
return cache.addAll(STATIC_URLS);
}).then(() => self.skipWaiting())
);
});

Here’s what I've done. I moved the filenames I know about to BASE_STATIC_URLS. I don’t fingerprint those file names, so they’re safe to hard-code. The important change is next. STATIC_URLS still ends up being an array of filenames, but now it’s a combination of two arrays. The filenames we know about and the generated array of filenames written to this file.

STATIC_URLS ends up looking something like this; 

[
  "/",
  "/index.html",
  "/manifest.json",
  "https://fonts.googleapis.com/css?family=Cousine:400|Karla:400,700",
  "static/css/main.2ebebc14.css",
  "static/js/main.7e7a1a8f.js",
  "static/media/bgTransparent.e6317315.svg"
]

When the project builds the fingerprinted filenames and asset-manifest changes. Then the service worker gets updated with the new filenames.

That all adds up to a build process that handles caching of fingerprinted files and one offline-capable PWA.

A New Normal

I’ve seen folks talking at length about how transformative PWAs are, but I couldn’t grasp it until I went through this process. This felt like building a responsive design for the first time. It’s the realization that this isn’t going to be a gimmick or extra or nice-to-have. It’ll take time for habit to kick in and browsers to catch up, but this will become my default when building sites.