Differences between static and dynamic imports in ESM

This article has probably be written hundreds of times already, so I don't think I'm adding anything new to the overall tech knowledge online, but hey writing it will probably help me remember it better.

In modern (ie. ESM) JavaScript, you no longer use the old require() method to load your dependencies; you use import instead. What I didn't really realize at first when I started migrating my codebase to ESM was that import comes in two flavors: static imports and dynamic imports.

Let's assume the following two dependency files:

// defaultExport.js
export default {
  name: 'Tim',
};

// namedExport.js
export const greetings = function greetings(name) {
  console.info(`Hello ${name}!`);
};

The first one is doing what we call a default export (using export default). The other is doing a named export (using export {something}).

Static imports

import config from './defaultExport.js';
import { greetings } from './namedExport.js';

greetings(config.name); // Hello Tim!

Using static imports, I can import either the default export with the import name from ‘./path.js syntax, or a named export with import { name } from ‘./path.js (the difference is in the { } wrapping the name).

The good: They sit at the top of the file, defining the needed dependencies. They are a staple of ESM, and allow for tree shaking your dependencies (ie. removing all dead code). They should be the most common kind of imports.

The bad: As they load dependencies statically, they won't "fail" in case of a cyclic dependency by default (you'll only realize something is broken in production, when suddenly one of your deps is undefined). Thankfully ESLint can help you catch this.

Dynamic imports

const config = (await import('./defaultExport.js')).default;
const { greetings } = await import('./namedExport.js');

greetings(config.name); // Hello Tim!

Using dynamic imports, I can still import both default exports and named imports, but there are a few subtle differences to keep in mind. The await import() call returns an object that contains _all _the exports of the module, both default and named.

*The good: *If you need to access named imports, the change in syntax is trivial. The { } now acts as an object destructuring, and allows you to access and define only the keys you're interested in.

*The bad: *Access default exports though requires you to specifically target the .default key of the returned object. Because import is asynchronous, it also requires you to wrap your await import() call in ( ) to actually return the .default key of what the promise returns, rather than the .default key of the promise itself (that doesn't actually exist).

What to use when?

I tend to use static imports 90% of the time.

I only ever use dynamic imports when:

• I need the module to specifically be loaded at a certain point in time (maybe because I don't yet have all the config ready before that)
• The module is slow to load (it does a lot of stuff when imported) and I want to delay that to a more appropriate time.
• I have a wrapper function that can do a bunch of different things, and I want to only load the required dependency modules based on what the function will actually need.

When this happens, I make sure that I don't forget to grab the .default key (but as I tend to prefer named exports anyway, most of the time the change doesn't require much thought).

Fetch this: Illegal invocation in Cloudflare Workers

If you write anything in JavaScript, you've certainly encountered the this keyword, and wondered what is this? Even if you're an experienced JS dev, sometimes this can bite you in unexpected ways.

This happened to me today, while porting a codebase from the got library, to the builtin fetch method. I had a method to write that was doing a bunch of API calls (using fetch), massaging the data I got from one endpoint, and sending it to another.

To unit-test that method, I didn't want to do actual API calls, but rather mock the calls, and return fake data to simulate a real call exchange. My test needed to assert that one of the calls in the pipeline was using the right headers.

// main.js
export default {
  async run() {
        // Here I do a bunch of chained API calls
        // [...]
  },
  __fetch: fetch
}

// __tests__/main.js
vi.spyOn(main, '__fetch').mockReturnValue({...});

Instead of using fetch directly in my run method, I added the __fetch key into my object, and would call this.__fetch instead. That, way, I could spyOn the method and mock the returned value. As main.__fetch would become a mocked function, I could also run asserts on it to check if it had been called enough times and with the right number of arguments.

That works perfectly and my tests all pass \o/

But it fails in production

Weirdly enough, even if tests are passing, the code doesn't actually work in production once deployed to Cloudflare workers. What I got instead was a nice TypeError: Illegal invocation: function called with incorrect this reference error.

Turns out it's a well known and documented type of error on Cloudflare Workers. Their documentation even has a dedicated part about it.

What's happening is that once the code is bundled with esbuild before being sent to Cloudflare workers, the reference to this inside of fetch is lost.

The solution

The fix is surprisingly simple. Instead of attributing __fetch as a reference to fetch, all I had to do was to define a new function that would simply call fetch.

export default {
  async init() {
        // Here I do a bunch of chained API calls
        // [...]
  },
  __fetch: async function(url, options) {
        return await fetch(url, options)
  }
}

This creates a wrapper around fetch, so its this isn't lost.

This might look like repetitive code, but the wrapper actually has a use. I made sure to add a comment around that wrapper so future me doesn't inadvertently "optimize" the code and break it.

The hidden complexity of "just a few lines of code"

We're often hosting meetups in the Algolia Paris office. To make our lives easier, I decided to build an API that would allow us to fetch metadata about the event easily (like description, date, list of attendees, etc). I decided to host that on Cloudflare Workers.

The premise was simple. Pass the url of the event to the API, and metadata about the event returned in a nicely formatted JSON. My function would crawl the URL, parse it with cheerio, extract relevant information and return it. Easy.

It's just gonna be a few lines of code

I could grab the description easily, write a bit more complex selectors and get the date. Great.

Then I realized that the meetup.com markup had some specific JSON blocks, called JSON-LD, that had most of the data I needed. Ok, so let's understand the schema, and extract what I need from that. Some keys are easier to access from this JSON (like the startDate and endDate) while others are still better in the HTML (description is truncated in the JSON for example). No problem, let's build an hybrid approach.

{
  "@context": "https://schema.org",
  "@type": "Event",
  "name": "HumanTalks Paris Novembre 2024",
  "url": "https://www.meetup.com/humantalks-paris/events/304412456/",
  "description": "Hello everyone!\nThis month we wanted to thank **Algolia** for hosting us.",
  "startDate": "2024-11-12T18:45:00+01:00",
  "endDate": "2024-11-12T21:30:00+01:00",
  "eventStatus": "https://schema.org/EventScheduled",
  "image": [
        "https://secure-content.meetupstatic.com/images/classic-events/524553157/676x676.jpg",
        "https://secure-content.meetupstatic.com/images/classic-events/524553157/676x507.jpg",
        "https://secure-content.meetupstatic.com/images/classic-events/524553157/676x380.jpg"
  ],
  "eventAttendanceMode": "https://schema.org/OfflineEventAttendanceMode",
  "location": {
        "@type": "Place",
        "name": "Algolia",
        "address": {
          "@type": "PostalAddress",
          "addressLocality": "Paris",
          "addressRegion": "",
          "addressCountry": "fr",
          "streetAddress": "5 rue de Bucarest, Paris"
        },
        "geo": {
          "@type": "GeoCoordinates",
          "latitude": 48.880684,
          "longitude": 2.326596
        }
  },
  "organizer": {
        "@type": "Organization",
        "name": "HumanTalks Paris",
        "url": "https://www.meetup.com/humantalks-paris/"
  }
}

Ok, now for the list of attendees. Oh, it's not in the markup, it's actually fetched from the front-end dynamically and added to the page. It seems to be doing a GraphQL query and getting the list of all attendees. All I have to do is replicate that query from my Cloudflare Worker. Thankfully, both the URL and the required query ID are available in the initial page. I just need to handle pagination of the results (as they are only returned by chunks of 50).

Testing

At that point, my code was obviously a bit more than just a few lines of code. I had to extract data from three different sources: HTML source, JSON LD and a GraphQL API.

That's when I started to add some unit tests. The number of edge cases was starting to increase, and I wanted to be sure that when I fixed one, I didn't end up breaking another.

Error Handling

As I spent a few days working on that, I also realized that meetups that have passed are no longer accessible (unless you're logged in). That means I also needed to add some error handling for those cases. Also, some meetups can disable the fetching of the attendee list, even if the meetup itself is public. I also needed to handle that case gracefully.

More sources

The project worked well, and really helped us internally plan for meetups. But then we had to host a meetup hosted on Eventbrite. And one on Luma. So I had to go back to my code again and see how to make my code work with those other platforms.

I had to re-organize my code, to have some shared helpers (like for parsing HTML with cheerio or for extracting JSON LD), but still keep per-source unit test and logic.

I also had to handle some source-specific issues. For example Eventbrite has no visible attendee list, and the HTML returned by Luma is different based on the User-Agent passed. Many things are the same, but many others are different. That's when having unit tests really started to shine. I could be sure that I could fix something for Eventbrite, without breaking Luma. This would have been a nightmare to test manually.

What I learned

An important lesson I learned was to make endpoints that were as generic as possible. Initially I had an endpoint for getting the description, another for getting the dates, etc. But I realized it was much easier (from a code POV, as well as a user POV), to have one endpoint that returned everything.

I still kept the list of attendees behind a flag (so, off by default, but if you need them you turn it on on the request), as it was the slowest part of the request and also the most prone to fail (on Eventbrite, or on past Meetup pages for example).

This "Oh, it's just going to be a simple proxy to grab 2 or 3 pieces of data" turned into a much more complex beast, but I like it. From the outside, I managed to keep it simple (just one API endpoint), and from the inside I have enough tests and shared code that it's relatively easy to make it evolve.

My journey through the ESM Tree Shaking forest

I had to work with Cloudflare Workers recently, and everything worked well until one day one of the HTTP calls I was doing started to fail.

When I ran the same piece of code locally it worked (obviously!). But pushed and ran through Cloudflare Workers, it failed. This was the first step in what then became a day-long trip into the rabbit hole of debugging. After a couple of hours of debugging "live" (by pushing my code, hitting the server, and checking logs), I finally discovered that my issue was that the HTTP endpoint I targeted had a rate limit, based on the originating IP. And when doing calls from Cloudflare, sharing the same IP with other workers, the IP already had hit the limit and my calls would fail.

That made me start digging into better ways to locally test CFW production code, and I discovered the wrangler dev mode. At first I thought this would spin my code on a real remote server and broadcast the console.log locally to my terminal, but no, it's a minimal version of CFW that runs locally. Not exactly the same as a staging env, but pretty close.

The main difference with running my scripts locally through unit tests is that when using wrangler dev, my code is bundled with esbuild and the bundled version is executed. This opened a whole new category of problems and questions to me.

First, I realized that the bundle size was way too big for what my function was actually doing. I had ~100 lines of code at most, but my bundle was several megabytes of minified code. Surely, something wasn't right. By inspecting the bundled code I realized that it had bundled all my dependencies and subdependencies.

But, isn't it supposed to do tree shaking?

I had read that esbuild was the new hotness, and that it should do tree shaking my dependencies automatically, keeping only what I would actually use. But somehow, it didn't seem to work.

What I learned is that tree-shaking is not possible through the virtue of esbuild alone. Modules have to be in ESM as well (so, basically using import rather than require) for it to actually work. So I updated my dependencies to their latest versions; most of them are now ESM-compliant. I managed to upgrade all my deps to ESM, and with that, esbuild was now able to tree shake the final bundle, reducing my filesize footprint to something 10 times smaller \o/.

ESMify all the things

One of the dependencies was actually one of my own modules, firost, and let me tell you that converting a CommonJS module to ESM is not a trivial task. It's certainly doable, but it does take some time, especially when you have several intertwined modules, some in CommonJS and others in ESM.

I especially had to be careful to use named exports rather than God Objects in my files, to avoid pulling all dependencies with a greedy import. The restructuring of files and import was tedious and long. I also had to ditch Jest (that does not support ESM) in favor of Vitest. I also updated ESLint to its latest version, which finally also supports ESM!

Lodash, you're next

The only dependency I didn't manage to shave off was lodash. I really like lodash, especially the _.chain().value() syntax, which I think makes expressing complex pipelines easier. But lodash still seems to be loaded as a monolithic block, even though I'm only using a few of its methods. I didn't dig too much into how to load it in a more clever way, but that's on my TODO list.

I also needed to include cheerio (because my worker is doing some scraping + HTML extraction), but couldn't find a way to load a leaner alternative (domjs is roughly the same size, and I prefer the API from cheerio)

Minimal .zshrc for remote servers

Recently, I found myself connecting to remote machines quite often. I have to debug remote servers for work or connect through ssh to an emulation handheld console I just bought (more posts coming on that later).

But I've configured my local zsh so much that when I connect to a bare remote server I feel a bit lost. No colors to differentiate folders and files. No tabbing through completion. Even simple things like backspace or delete key do not always work.

So this time, I built a very minimal .zshrc that I intend to scp to a remote server whenever I need to do some work there, just to make my life easier. I tried it on the aforementioned console, and it helped a lot, so I'm gonna share it here.

Fix the keyboard

export TERM=xterm-256color
bindkey "^[[3~" delete-char            # Delete
bindkey "^?" backward-delete-char # Backspace
bindkey "^[OH" beginning-of-line  # Start of line
bindkey "^[OF" end-of-line             # End of line

Starting with the basics, I ensure my terminal type is set to xterm-256color. It should fix most keyboard issues. But just to be sure I actually did define the keycodes for delete, backspace as well as start and end of line.

The ^[ and ^? chars here are not real characters, but escape characters. In vim, you have to press Ctrl-V, followed by the actual key (so, backspace, or delete, etc) to input it correctly. I found that various servers had this mapped differently, so you might have to manually change it if it doesn't work for you.

Completion

autoload -Uz compinit
compinit
zstyle ':completion:*' menu select

This will enable a much better completion than the default one. Now, whenever there are several possible solutions when pressing tab, the list of possibilities will be displayed, and you can tab through them as they are getting highlighted.

Colors

autoload -U colors && colors
export COLOR_GREEN=2
export COLOR_PURPLE=21
export COLOR_BLUE=94
export LS_COLORS="di=38;5;${COLOR_GREEN}:ow=38;5;${COLOR_GREEN}:ex=4;38;5;${COLOR_PURPLE}:ln=34;4;${COLOR_BLUE}"
zstyle ':completion:*' list-colors ${(s.:.)LS_COLORS}

I then added a bit of color. I defined a few variables to better reference the colors. Those are mapped based on the color palette I'm using in my local kitty. They would probably be different on your machine, so you should also adapt it.

The LS_COLORS definition sets the directories in green, executable files in purple and symlinks in blue. This simple change already makes everything much easier to grok. The zstyle line also applies those colors to the tab completions \o/.

Aliases

alias v='vi'
alias ls='ls -lhN --color=auto'
alias la='ls -lahN --color=auto'
alias ..='cd ..'
function f() {
        find . -name "*$1*" | sed 's|^./||'
}

I added some very minimal aliases; those that are embedded in my muscle memory. I have much more locally, but I went for the minimal amount of aliases to make me feel at home. I also didn't want to have to install any third party (even if exa, fd and bat would sure would have been nice).

v is twice as fast to type as vi. Some better ls and la (for hidden files). A quick way to move back one level in the tree structure, and a short alias to find files based on a pattern. Those are simple, but very effective.

Prompt

PS1="[%m] %{[38;5;${COLOR_GREEN}m%}%~/%{[00m%} "

And finally a left-side prompt to give more information of where I am. It starts with the name of the current machine, so I can easily spot if I'm on a remote session or locally, then the current directory (in green, once again, as is my rule for directories).

The wrapping %{ and %} are needed around color espace sequences, to tell zsh that what's inside doesn't take any space on the screen. If you omit them, you'll see that what you type is offset on the right by a large amount.

I actually like to replace the %m with a machine-specific prefix, to more easily see where I am.

Here, for example, you can see I'm connected to my handheld console (I added the SNES-like colored button), currently in the /roms2/ directory and I'm tabbing through completions in the ./n64/games/ folder.

And that's it. A very minimal .zshrc for when I need to get my bearings on a new remote server and still be able to do what I want quickly.