Hvordan lykkes med smidig transformasjon?

In today's post I'll share key lessons from my journey in implementing Anonymous Tokens and integrating it in Norway's contact tracing app "Smittestopp". Privacy and transparency, especially in government IT, is vital for gaining citizens' trust - and is here to stay. Therefore I'll share some success factors and my takeaways with you. 

This post is a follow-up on "Anonymous Tokens for Private Contact Tracing" that Tjerand Silde, Martin Strand and I wrote for last year's calendar on 22nd of December, where we tell the story on how we improved privacy in Norway's contact tracing app "Smittestopp" by designing a protocol for anonymity.

Through the work on Anonymous Tokens and integrating our contribution to Smittestopp, I've summarized some success factors and what I want those of us who build digital services to take away from our journey.

Key factors for success:

cross competency

deep skills and overlap in skills

good communication skills

open source enabled transparency and trust

Key takeaways:

ensure built-in privacy

have a data minimisation mindset

trust through transparency

Next I'll bring you up to speed on what's happened since our last post, and share how things went after our work was done. 

We're live!

"Anonymous Tokens" went live in Smittestopp on April 8th 2021! 🎉 

Anonymous Tokens was the culmination of a huge collaboration between the development team at The Norwegian Health Authority (FHI), other contributors and Martin, Tjerand and myself. It truly was a nation-wide open source "dugnad" to get our contribution into Norway's contact tracing app "Smittestopp" on GitHub!  

The tale has been told

The story about the collaboration between contributors to Smittestopp has been told in written form by Den norske dataforening (DND) and Johannes Brodwall on Kode24.no. Johannes highlights the value of FHI's willingness to move towards transparency and their openness to external contributions. You can also hear a podcast-episode from Utviklerpodden with Johannes on this topic.

Our work on Anonymous Tokens has also been mentioned in several Norwegian news media: forskning.no, Grannar.no and Computer World.

We won a privacy-award!

We won the "Built-in privacy"-award from the Norwegian Data Protection Authority (Datatilsynet)! This took us by surprise, to say the least. It seems they really wanted to get the message across that this is something they want to encourage and see more of. We agree! 😊

So... what did we learn from all of this?

I'm writing this post almost exactly a year after we started our work on Anonymous Tokens, and I reflect on how we managed to develop it in just 4 weeks. How did we manage that in such a short time? We've also observed challenges in privacy in contact tracing apps around the world. What can be done to move forwards in this space to ensure citizens' and users' privacy are respected in (government) IT services?

Key factors for our success

Cross-competency

A team of cross-competency skill sets will move faster by being able to do more and have the required key competency to reach their goals. This encourages team members to develop deep skills in their areas of expertise and collaborate with others, rather than individuals focusing on their own work.

Deep skills and overlap between skills

When you have a team composed of people with expertise in their own subjects, but who can look over to other subjects just enough to get their head above water, the team will understand each other better and move quicker. This was certainly true for us where coding and cryptography were two subjects where we each had expertise and understood enough of each other's subjects to understand, communicate and collaborate.

Good communication skills

When people communicate clearly, concisely and bring just enough context to the discussion to provide for a valuable dialog - you are able to progress faster. Poor communication where people bring in foreign elements such as unfamiliar abbreviations and terms, with a lot of out-of-context information, clutters the dialog and makes it cognitively harder to follow the discussions. A healthy dose of empathy doesn't go amiss.

In both open source and projects I've worked on I've seen both ends of the scale. I wish focus on communication skills and empathy were a larger part of software development.

Open source enables transparency & trust

GitHub provided tooling we could use to construct software built-in measures for security, trust and quality. By using open source we ensured that our contribution to Smittestopp was transparent for public scrutiny and open for external contributions. These benefits can also be used by government IT services that are built with an open source mindset.

FHI's change of direction towards transparency for Smittestopp 2 enabled a nation-wide collaboration and made all of this possible! 

Key takeaways for those who build digital services

Ensure built-in privacy

If you want to know more about how you can implement built-in privacy in software development you should have a look at this handbook by Datatilsynet. They've written guidelines and best-practices on privacy in software development focusing on requirements, design, testing, coding, maintenance and more. Share this with your team!

Have a data minimisation mindset

Large datasets with rich user-data are honeypots for threat actors. Threat actors aren't just external hackers. Your own employees, your company and even yourself can unknowingly be threat actors that should be modelled and mitigatigated.

Handling user-data with person identifiable information (PII) is a risk. How big of a risk it is depends on how well you've secured your user's data. By minimizing the amount of data you handle and store, you also minimize the damage potential if data is exposed by hackers and leaks. 

Ask yourself: do I really need this data for my service to provide value, and do I have a legitimate reason for storing this data? If you answered no to either question, then you might risk being in breach of GDPR-regulations. Moving towards a data minimisation mindset will please privacy-aware users and mitigate data-processing risks for your service.

Ensure trust through transparency

Trust is a frail thing. Without it, you risk users not adopting your service. Building digital services on an open source platform is a great way of being transparent. This is true not just for commercial companies but especially for governments who build digital services for their citizens. 

Transparency builds trust. Transparency in government IT can reveal inequality and injustice. Citizens can also validate if automated processes within these digital services are compliant to laws. This is a win-win scenario!

Could transparency in government IT even lead to stronger democracies?



Key learnings from working on privacy in contact tracing

We've all been there: we created our first react project and was introduced to the lock file. Some took the time to understand it whereas others was scared of it and instinctively added it to the .gitignore file. This post will give a short description of the lock file, how we should treat it and use it correctly.

🔍 What is a lock file?

While the package.json 👇 file of your project contains dependencies. The lock file describes the exact version of packages to be installed, including the dependencies of the dependencies. Wait what? Do dependencies have dependencies? 😅

Yes! Whenever you add a dependency to your project through npm, yarn, or append it to the package.json file, the new dependency for the project almost certainly has dependencies. The exact versions of the dependencies of the dependency are described in the lock file. This ensures that the lock file is the single source of truth in terms of package versions.

The block above 👆is how the dependency react-dom is described in the lock file. The entry contain different properties about the package, such as the dependencies of the package along with: 

🎯  Version: The exact installed version of the package in the project.

🚚  Resolved: Where the package manager resolved the package from.

💯  Integrity: The computed sha512 hash of the package to ensure that no files of the dependency has changed since the package was published.

📦  Dependencies: Dependencies for the package along with the versions supported by the package.

🤝  PeerDependencies: A list of packages along with their versions that the package ensures to be compatible with if it is a plugin to a host package.

The lock file for a project is generated automatically by your package manager when a fresh install of all dependencies of the project is performed. If yarn is your preferred package manager the lock file will be named yarn.lock, and contrary, if npm is your preferred package manager the lock file will be named package-lock.json.

💡 npm 7 can use a yarn.lock file as the source of dependencies for a project 

As mentioned, the lock file is generated when running the install command with your preferred package manager. Even if the package.json file hasn't changed, a subsequent install may update the lock file. This happens because of what you may have prefixed the dependencies with. Take a look at the dependencies from a package.json file below 👇 and note the characters ^ and ~. 

These characters are used to specify the latest compatible version of the dependency the package manager should install. The ~ character specifies that it should install the latest possible version with fixes. The ^ character tells the package manager that it should install the latest non-breaking version of the dependency. Therefore an install command without changing the package.json file may result in an updated lock file. Listing a dependency without any of these characters means that the version is pinned to the exact version and any later install won't install a later version of the package. 

🤔 Should lock files be committed?

The lock file is intended to be included in repositories. As described above, the lock file gives a single source of truth of the exact version of all dependencies for a project. Including the lock file in repositories ensures that any other fresh installation of the packages for the project uses the same versions of all dependencies.

Another benefit of including the lock file in the repository is that it gives the possibility to "time-travel". Time-travel in this context means that the lock file makes it possible to install all packages at various states of the project.

 👀 Sounds great, but what can go wrong?

There aren’t many things that could go wrong in terms of the lock file, but as with any process where humans are involved, accidents happen 🤷‍♂️

Imagine that a package is added to or updated in the project. As we learned in the first section, this will modify the lock file accordingly. If the updated lock file is forgotten to be committed along with the package.json file to the repository, a situation where different dependencies are installed could occur. This is usually not a huge problem, but it means that the versions of the dependencies installed locally could differ from the versions used when e.g. building the project for production in a continuous integration process. NPM and yarn have different approaches to avoid such a situation.

NPM

NPM have a special command to avoid such a situation:

npm ci

The command is meant to be used in continuous integration environments. It ensures that the project has an existing lock file and that the versions of the dependencies specified in the lock file does match the ones specified in package.json, otherwise an error will be thrown. Another benefit of this command is that it will automatically delete the node_modules folder if it exists which will ensure that a clean install of the dependencies is performed.

Yarn

Yarn avoids the described situation by offering an flag to the install command:

yarn install --immutable

The flag make the install process abort with an exit code if the lock file was to be modified because of a clean install. In continuous integration environments this flag is on by default.

💡 If you are using Yarn v1 --frozen-lockfile is the equivalent to --immutable

🙌🏻 Thats it

Hopefully you learned something new or refreshed your knowledge about lock files. You probably want to double check your continuous integration pipelines to ensure that they use the correct commands for installing the dependencies for your project 👀 If you want to dive even deeper into lock files you should checkout (depending on your package manager) NPMs or Yarns documentation.

Thanks for reading - Happy holidays! 🎅🎄⛄️

Something about lock files

Finn.no, the marketplace of possibilities, might be Norway's biggest sale platform.
They also happen to publish a lot of the tools they create and use as open source tools, such as the feature toggle service Unleash.
Let's try to recreate their own page with their tools! 🙌
Since Finn.no wasn't built in a day, we will restrict ourselves to the frontpage.
Also, to make the article readable in a sensible amount of time, I will not go through the entire architecture, but reading this tutorial should give you the necessary means to build it all yourself.

All the code I wrote here is pushed to my try-finn-no Github repo, feel free to check it out if you want to copy-paste or believe I made a typo here.

Finn.no's architecture: Podium

Finn.no consists of many small frontends, known as a microfrontend architecture.
To orchestrate these frontends and sew them together, the developers at Finn.no created Podium.
Podium composes the different frontends into a single page server side.

Podium defines the "maestro" in charge of composition as a Layout. Each microfrontend is called a Podlet.
So first, let's create a Podlet.
The Podium documentation has an illustration of which part of their page is a separate frontend. This illustration is however outdated, but trustable sources have verified that the header is still it's own Podlet.

Header as a React App ⚛️

I will recreate the header in React with TypeScript, but Podium claims to be framework agnostic as long as the result is HTML, CSS and JS, so you are free to use anything you'd like 💪

To create our new app, simply run:
npx create-react-app header --template typescript

Taking a quick glimpse at Finn.no, we see that the header consists of 5 elements:

Logo

Notifications ("Varsler")

New ad ("Ny annonse")

Messages ("Meldinger")

Login ("Logg inn")

Except for the logo, all the other elements consists of an icon and a text, which is a hyperlink.
We can then create a simple component for these links.

We now have a clickable link, with all content being clickable.
However, we still need the most important part: the icons 🧑‍🎨
Luckily for us, Finn.no's icons are also available through their design system, Fabric.
Let's install the icons in the header-project.
npm install @fabric-ds/icons

We can now use these icons in our LinkButton.
However, since Fabric's React module is still in a beta state, we will have to import the svg's directly as they are currently only exported as Vue Components.
Since I am using Create React App, I import the svg as ReactComponent.

Now, repeat this for "Ny annonse", "Meldinger" and "Logg inn", with the icons "circle-plus", "messages" and "circle-user".
You can find an overview of all the available icons at Finn's icon overview page.

PS: You might encounter a TypeScript error related to SVG types here. 
Cannot find module '@fabric-ds/icons/dist/32/bell.svg' or its corresponding type declarations 
This means that you probably ran npm install @fabric-ds/icons in the wrong folder - SVGs with TypeScript should just work™️ in Create React App 🙏

Now, with a little bit of styling and combining, we have our header 🎨

If you run Create React Apps development server with npm start, you should now see the header at localhost:3000!


If you want to get the logo correct as well, you can sneak over to Finn.no and replace the content of our src/logo.svg with the svg of their logo.

Layout 🎨

To serve our webpage, we will need a "maestro", a Layout server.
Podium's docs cover this step very well, but I'll repeat it here for you anyways.

First, initialize a new project. It doesn't really matter what you name it, and we can just go ahead with all the defaults ✨

Next, we add our Podiums dependencies.
npm install express @podium/layout
and lets add the types for Express as well.
npm install --save-dev @types/express

We can then go ahead by creating our index.ts and instantiate a new Express app.

This is still just a regular Express app, so let's initialize our layout as well.

Then we add the Podium layout middleware to the Express app, which does its magic and adds data to our responses 🪄

All that is left is to configure our default endpoint and set the port to listen to.

All summarized, our index.ts should look something like this:

To run our Layout server easily, we can use nodemon. Since we're using TypeScript, we will need ts-node as well, which nodemon will use under the hood.
npm install --save-dev nodemon ts-node.
We also have to add a start script to our package.json.

Start our Layout server with npm start and you should see our beautiful "Bonjour" at localhost:7000 👋


Last I checked however, Finn.no is more than just a white page stating "Bonjour", so lets move on to serving our Header React App as a Podlet!

Header as a Podlet

To serve our Podlet, we could in theory include a static manifest.json-file with the necessary information the Layout server needs to get the content.
However, Podium also includes a module like the layout-module which helps us a lot in creating and serving our Podlet.
npm install express @podium/podlet.

We can then go ahead and create a new file in our header-project called podletServer.js.
I would have preferred to use TypeScript here as well, but I took a shortcut by using JavaScript so that I do not need to edit my tsconfig.json to properly compile the podlet server outside of the rest of the project.
We will use a Express app to serve our Podlet just like our Layout server, but with the Podlet module instead.

This looks fairly similar to our Layout-server 🔎
We also have to tell Express to serve our build-folder as static files, to actually serve the bundled output of Create React Apps npm run build.

The last thing we need is to add the two paths the Layout server expects: the path to our content and the path to the manifest.
We can get these paths automatically from the Podlet.
To generate the manifest, we can simply just send the podlet as a response, and it returns itself as a manifest.

The content fetch will only get the HTML-file, so we have to register the js and css output of our build as well.
These are currently dynamically named on build with a hash based on the build timestamp through Create React Apps included Webpack configuration.
We thus have to make changes to our webpack config ⚙️
To make it easy however, let's not eject our app entirely.
Instead, I'll use Craco to inject our changes in the Create React Apps default config.
npm install @craco/craco --save
Next, we add our Craco config 🔧
We could write these changes ourself, but NAV has created a small plugin set exactly for this.
I'll take a shortcut and use these tools, but feel free to take a look at the code itself. It is only 92 lines.
npm i @navikt/craco-plugins

Finally, we replace our start & build scripts using react-scripts in our package.json with craco.

With our new static bundle names, we can easily register the build output with our Podlet.



Last, all we need is to tell the Express app which port it should listen to.
All together, the Header Podlet will look something like this:



Let us install nodemon to run our Podlet server. Note that we don't need ts-node now, since our podletServer is written in JavaScript and not TypeScript.
npm install --save-dev nodemon

Also, let's add a script to run our Podlet server in our package.json as well.
"start:podlet": "nodemon podletServer.js"

If we start our Podlet, npm run start:podlet, we should now see our header server through Express by visiting localhost:7100.



We can also check the manifest at localhost:7100/manifest.json.
Here we can see that the manifest defined by Create React App for webworkers is prioritized before our Express-path.
Since we do not use webworkers in this article, you can go ahead and delete the manifest.json in the public-folder so that our dynamically generated manifest will be shown.
Rebuilding the app, npm run build and possibly restarting the Podlet server, npm run start:podlet, should now show us the correct manifest at localhost:7100/manifest.json, which should look more or less like this:

It all comes together 🧵

We can now register our Podlet with the Layout server.
Open index.ts in our layout-server-folder.
Then, register the Podlet through the layout.client.register-method.

Now we can change our output from simply "Bonjour" to our header.
First, fetch the registered Podlet and pass it some context named incoming which we get from the Layout middleware.
Add the podlets to our incoming object. This magically adds all CSS and JS to our <head>.
Last, use the content in our HTML response.
Also, React expects to be loaded after the DOM tree is finished, so let's load it to the end of our DOM by using the built-in toHTML-method.



All in all, our final Layout server looks like this:



Let's spin it up all together!
Start our Podlet server if it is not already running through npm run start:podlet in the header-project.
Then, run our Layout server with npm start and ...🥁

Et voila!
The Header Podlet is displayed through our Layout server!

We now have the structure for the Finn.no.
Simply add the rest of the owl and you've got our own Finn.no competitor!


Finally, I'd like to thank Pål-Edward Larsen, developer at Finn.no for answering my questions whenever I failed to understand the docs or simply turned blind from reading the docs repeatedly. 🎉

FAQ

How do you have frequently asked questions on a newly posted article?

I lied, I don't 🤷 I just predict what you might want to know after reading all this.

Are we not loading our JS bundle twice in the Layout server?

Yes, we are 🔎 The keen reader might have noticed that the Podlet JS bundle will be loaded twice, one in the body and one in the head.
There are multiple ways to avoid that, either by editing the document template of the layout server or by stripping it from the response before we add it the Podlets array incoming.podlets.
I did not include it in the article to keep it short and to the point, but feel free to change it in your Finn.no frontpage!

Can I just copy and paste the header-project many times and plug them into the Layout-server to have a full webpage?

Almost 🙌 In our header-project, we use the default DOM-id root. This will obviously not work if multiple projects use root.
Thus, we need to set this for each app. This can easily be done through changing the id in public/index.html as well as in src/index.tsx's line document.getElementById('root'). It would probably also be smart to use unique bundle names. You can see an example on my multiple-podlets-branch on the try-finn-no-repo.
It will probably look a bit stupid though, I would suggest you change the content a bit 🤔



15-Dec

Today's posts

Hvordan lykkes med smidig transformasjon?

Key learnings from working on privacy in contact tracing

Something about lock files

Recreating Finn.no's frontpage with their own tools