The rise of cloud platforms has provided a number of application hosting platforms, like Heroku and Firebase, to name a few. Being old news in themselves, they provide a fast and effortless way of setting up an application. Combined with some creativity, platforms like these may provide great value for your team, even if they’re not part of your main platform infrastructure.

Have you ever had the need of a clickable prototype of your application for demoing purposes or to go a step beyond mere illustrations? Do you wish that your users could play around with your product to be familiar with its capabilities or to learn to know it without messing with real data? Does your organisation have customer support or other internal staff that need to know the look, feel and functions for end users, or need a playground for training? All of these are cases where using a (more or less) fake instance of your application may come in handy, and cloud platforms are great for setting up stuff like this in a flexible way.

Couldn’t we just use the good old test / QA environment for taking care of these needs? Sure we could, at least for some of them. But they usually come with a number of limitations. In particular:

They are often tightly integrated with other systems, and require some real test data set up in other systems

They often require some kind of user login, either like the production system or some test logins/users that need to be set up in other internal systems

They may run in environments that are hard to reach. For random external users in particular, but not rarely also for various groups of internal users

These limitations may relatively easy be overcome by the combination of:

Decoupling your application, or parts of it, from it’s run-time dependencies and integrations

Tailor your application to fit your needs.

Running it in a publicly available environment with easy set-up and deployment. This is where application hosting platforms come in, if your regular infrastructure have shortcomings

Decoupling your application: Most applications depend on one or more other systems or features. Some are in the hands of the core developer team, while others are maintained by other internal teams or even external vendors. A common web application architecture at least consists of a frontend application (e.g. a javascript / react app) communicating with a backend through an API for looking up or updating data. The backend, in turn may have further dependencies, e.g. to a database or to one or more other systems. Removing (or faking) these dependencies will make it easier to run your application stand-alone. The most light-weight approach could be faking your backend API, making the frontend the only unit to deploy. A more heavy-weight approach is deploying your backend and fake the deeper dependencies. The database could be a real stand-alone database, a simple replacement (like an in-memory db), or mock data provided in code.

Tailoring your application could be things like letting the users choose their own login and user group (i.e. internal, external or admin user), providing different behaviour or data for various users (e.g. user names starting with “xyz” always get a certain data or behaviour), or have some way of switching on and off new beta features.

Running it in a publicly available environment. It might well be that your regular infrastructure is already a public cloud, providing a simple setup of new, stand-alone environments. If so, use this platform for deploying your demo app. If not, Heroku, Firebase or similar platforms come in handy.

Let’s finish with a few examples: My current team is working with digitizing and improving NAV’s (Norwegian Labour and Welfare Administration) services for corporations. Some of the services are deployed as stand-alone mock versions for demo and training purposes.

Companies that need to lay off employees are required to report this to Nav. The arrival of Covid-19 brought a need for rapidly digitizing this process. A mock version of the service is set up at Heroku: https://permittering-skjema.herokuapp.com/permittering/

Another service from NAV is to partly cover the salary for an employee, or to let an unemployed act as a trainee to test their work ability in a company. This service is supported by a three-way digital contract, shared by NAV, the company and the employee. The contract has a nifty online demo, letting the users “log in” as either NAV, a corporation representative or the employee. It is backed by an in-memory database for short-lived persistency, and is recently moved from Heroku to NAV’s “labs” domain: https://arbeidsgiver.labs.nais.io/tiltaksgjennomforing/

Finally, the open source feature toggle system Unleash provide a great demo of their admin system. Anyone may log in password-less and try out creation of feature toggles and activation strategies. Beware that everyone sees everything :) The demo is hosted at Heroku: https://unleash.herokuapp.com/#/features

Happy demo-ing in the cloud!

Demoing and training in the cloud

There are many ways of getting started with learning Kotlin, different people learn best from different approaches. Whether it's reading books, solving code puzzles, getting practical experience or watching talks I hope this blogpost helps you find some interesting resources.

Documentation

One good way to both start learning Kotlin and honing your already existing skills is to use the official Kotlin documentation. The getting started guide is a great place to start, while the Kotlin reference page is a good tool to get an overview of subjects, and often points you in the right direction on where to dig for more information. An example of this is the Kotlin coding conventions that can help you if you’re a little unsure if you've structured your code in a good way.

Coding and puzzles

If you’re learning more by doing practical stuff another way of honing your Kotlin skills is to do code puzzles, or koans, to get familiar with the language. The official Kotlin Koans comes with an online editor making the setup and learning curve minimal.

If you have some more time to spare, doing workshops can be a great way to get started with a new language. There are multiple workshops to choose from but I can recommend both the Kotlin Workshop by Jetbrains and the Kotlin Workshop we’ve created at Bekk.

Another great way of learning Kotlin is to have a side project where you can experiment with everything you find interesting and explore the language without having any hard requirements you need to meet.

Videos

There are many talks about Kotlin online so it can be a little hard to find the perfect talk for your needs, but if you’re unsure where to look for some great content the recorded talks from Kotlinconf is a good place to start. Below are some suggestions about talks that I’ve found are worth while watching.

Introduction to Kotlin by Hadi is a great introduction video if you’re new to Kotlin.

If you feel you’ve learned the basics of Kotlin but are unsure which new shiny syntax to use, and which syntax that might cause more headaches down the line you should check out Putting Down the Golden Hammer by Huyen.

If you feel like you’ve mastered the basics of Kotlin The Power of types by Danny is a great talk exploring how you can use Kotlin types to your benefit. It goes in depth on how types can help us make our code more readable and less error prone.

Testing: Testing by Christina is a great talk about testing in Kotlin and how Kotlin types help you reduce some of the mundane parts of testing code.

If you want to go beyond the basics and learn more about some of the more advanced topics Beyond the Basics by Hadi is a great place to start.

Summary

There are loads of resources for you to get started out there, so I hope this post has helped you find some new resources that can be of help when learning the Kotlin programming language.

Resources for Learning Kotlin

LiveOverflow is a german hacker running a very popular YouTube channel where he posts videos related to Capture The Flag competitions and IT security. I had the great honor and privilege of talking to him about his experiences with CTF competitions among other things.

Talking CTFs with LiveOverflow 

2020 har vært et uforutsigbart år preget av pandemi, nedstengning, smittevernstiltak, smitterisiko, og kohorter, men også et år der tilpasningsdyktighet, digital omstilling, nye arbeidsmetoder, børsrekorder, reduserte klimautslipp og et demokratisk gjennomført valg i USA har vært sentralt. Det er følgelig et år som vanskelig kan oppsummeres med få ord, og derfor skal dette innlegget, som en slags litterær årskavalkade, gi lesere inspirasjon til faglig relevant, aktuell eller passende litteratur som både kan representere og oppsummere året som var. Hensikten er at du som leser kan få inspirasjon til ny og spennende lesning, samt se tilbake på 2020 gjennom nye briller. Innlegget beskriver 12 bøker som hver tilskrives en måned i 2020 basert på periodisk relevans og innhold. God lesing!

For nærmere et år siden er vi gledelig uvitende om året som kommer. Vi har hørt om en pangolin og en flaggermus i Kina, men dette virker aldeles fjernt og irrelevant for oss i lille Norge. Hverdagen går som normalt og starten på det nye året er tilsynelatende lik som alle andre år. Vi skulle bare visst. Hadde vi bare hatt evner og verktøy til å se frem i tid kunne mye vært avverget og vi kunne tatt de riktige valgene hele veien. Dessverre er slike verktøy kun forbeholdt spåkoner og overtroiske, men som tidligere professor Eric Siegel skriver i sin bok Predictive Analytics, har vi faktisk evner til å forutse mye, dog på et lavere nivå. Siegel beskriver hvordan analyse av store mengder data kan avdekke handlingsmønstre og hjelpe oss til å bedre forstå hvordan konsumenter oppfører seg. Selv om boken tar for seg kompliserte konsepter som maskinlæring og kunstig intelligens, er den lettskrevet med fokus på forståelige eksempler fra virkeligheten.

Februar er preget av usikkerhet. Coronaviruset begynner virkelig å få fotfeste flere steder i Europa, og vi innser gradvis at viruset før eller siden kan komme til Norge. Når og i hvor stor grad er fortsatt uvisst, men spørsmålet om hvordan en potensiell nasjonal epidemi vil påvirke samfunnet diskuteres stadig oftere. I boken The Black Swan fra 2007 beskriver tidligere finansmann Nassim Taleb hvordan store, uventende, og innvirkningsfulle hendelser – som corona – påvirker oss. Taleb definerer slike hendelser som sorte svaner, hendelser som er spesielt uvanlige, har ekstrem innvirkning, men som også kan forklares i ettertid. Bokens hensikt er ikke å beskrive hvordan man forutser slike hendelser, men heller å skape bedre forståelse rundt de og hvordan man kan posisjonere seg selv for å bedre dra nytte av det uventede. Nyttig lesning for den som ønsker å stille forberedt til neste pandemi.

Torsdag 12. mars stenges Norge ned og de mest inngripende tiltakene i fredstid innføres. Vi går alle en uventet og mørk tid i møte, børsene raser, folk hamstrer, hysteriet sprer seg og mange rømmer utenbys. I noen dager var usikkerheten og hysteriet av apokalyptisk preg, og det så lenge ut som at verden slik vi skjente den var historie. Heldigvis ble ikke de dystopiske forestillingene virkelighet, og verden gikk videre – det gjorde den ikke i George Orwells 1984. Det er kanskje i overkant dramatisk å sammenlikne Norge anno 2020 med den fiktive, mørke og dypt dystopiske verdenen beskrevet av Orwell, men boken kan likevel representere stemningen mange opplevde i mars. Skrevet allerede i 1949, ønsket Orwell å beskrive den ytterste konsekvens av totalitærisme og en splittet verden. I boken følger man Winston Smith og deler av hans liv som borger av London i nasjonen Oceania i året 1984. Dette er dog ikke et London fra den virkelig verden, men et trist, mørkt og dystopisk sted dyrket frem av et totalitært parti som undertrykker og styrer livene til innbyggere.

I april begynner det å synke inn; vi er rammet av en dødelig pandemi som vanskelig kan kontrolleres og som potensielt kan vare lenge. Deler av næringslivet er i knestående, folk permitteres og hjemmekontoret gjør sitt definitive inntog i det norske hjem. Det blir tydelig at gamle vaner og arbeidsmetodikk må legges bort på ubestemt tid, og at omstilling og nye verktøy blir viktige i tiden fremover. Mens kunder flykter og inntektsstrømmer stadig faller bort, blir også bedrifter tvunget til å omstille seg, ta grep og tilpasse strategier for å overleve. Mange opplever likevel en form for optimisme der tiltak innføres, og man virkelig tar tak for å finne tilpassede løsninger til denne spesielle situasjonen. I slike tider blir stå-på-vilje, mot og strategisk tankegang viktig, og man kan se mot Sun Tzu og hans The Art of War for inspirasjon om fundamentale strategiske prinsipper. Selv om boken ble skrevet for mer enn 2500 år siden og kun beskriver militær strategi og krigføring, er den fortsatt populær i dag, da flere av prinsippene enkelt kan anvendes i andre disipliner for å fremme effektive handlinger og gode beslutninger.

Vi begynner stadig å vende oss til den nye normalen, og børser og næringsliv løftes gradvis opp av mørket. Likevel føler mange et sterkt savn etter kollegaer og fysisk samvær, og motivasjonen er for mange sterkt preget av en mindre spennende hverdag på hjemmekontor. I slike tider kan boken Drive være et nyttig verktøy for å tenne gnisten og for å forstå hvordan vi mennesker virkelig motiveres. Advokat og Al Gores tidligere taleskriver, Daniel H. Pink, utforsker i denne boken hva som har motivert mennesker gjennom historien, hvordan den moderne verden er preget av et utdatert jag etter gevinster, og hvordan man kan finne og utnytte den indre motivasjonen mennesker har som barn. Nyttig innsikt og viktige poenger i en tid som krever litt ekstra motivasjon.

Smittetallene reduseres stadig, tiltakene lettes gradvis og det er en generell positiv holdning i vårt vidstrakte land – er dette slutten på pandemien? Dessverre viser det seg å være en tidkrevende prosess med stadig mindre tilbakefall og smittemønstre som er krevende å kartlegge. Det kan virke som at vi er fanget i et begrenset samfunn der det normale livet kun er en fjern virkelighet. I slike tider er det viktig å bevare positive holdninger og holde motet oppe. Et godt verktøy som kan bidra til å se helheten og å sette verden i et bedre perspektiv er Hans Roslings Factfulness. I denne boken analyserer den svenske professoren Hans Rosling sammenhenger i makroøkonomiske og demografiske data for å avkrefte typiske misforståelser ved verdens utvikling, samt hvordan vi kan rette opp i disse feilene. Denne lettleste boken gir håp og bidrar til å snu et generelt negativt perspektiv på verdens utvikling til noe positivt. Hans Rosling gikk dessverre bort før boken ble utgitt, men hans sønn Ola Rosling og svigerdatteren Anna Rosling fullførte og utga boken til hans ære, året etter hans død i 2018.

På tross av fortsatt strenge tiltak og et begrenset samfunn er det duket for sommerferie i Norge. Hjemmekontoret legges til side, PCen skrus permanent av og Ola Nordmann legger seg pent til rette på en solstol i skjærgården. Samfunnet logges på mange måter av og vi får tid til å tenke på andre ting enn corona og smittefare. I en slik tid kan et historisk tilbakeblikk være på sin plass for å utvide perspektiver og for å forstå at historien er større enn 2020. I den anledning leverer den britiske historikeren Peter Frankopan gjennom Silk Roads et mesterverk av en oppsummering av verdenshistorien fra år 2000 f.kr. til i dag. Gjennom boken ønsker forfatteren å vri det tradisjonelle synet om at verden har utviklet seg rundt vesten, til at Silkeveiene har vært limet som har tilrettelagt for verdens utvikling de siste 4000 årene. Et spennende perspektiv, en påminnelse om at verden har sett langt verre historiske kriser og et historisk påfyll som burde vært pensum i skolen.

I august slår realiteten oss i ansiktet, ferien er over og vi må tilbake til hjemmekontoret. Det som i mars var en ny realitet har blitt en ny normal; det er ikke lenger uvanlig å stå opp 07:50 for å rekke et møte klokken 08:00 og skjorter har ikke vært strøket siden mars. Sakte, men sikkert, har vi vendt oss til situasjonen og etablert nye vaner og rutiner. Det er fascinerende hvor tilpasningsdyktige vi mennesker er, og hvordan vi over tid både bevisst og ubevisst kan tilpasse handlingsmønstrene våre. I boken The Power of Habit drøfter journalisten Charles Duhigg mye av denne logikken, og utforsker hvordan vi mennesker tillegger oss vaner og rutiner, og hvordan vi kan bruke dette som virkningsfulle verktøy. Interessant lesning i en periode der ting stadig begynner å gå i ett og hverdagen har blitt ren automatikk.

Samfunnet er drastisk forandret, omgangskretser er sterkt begrenset og organisasjoner har virkelig måttet tilpasse seg en ny hverdag. I næringslivet er det kanskje viktigere enn noen gang å gjøre gode målinger for å bevare kontroll og oversikt i en turbulent tid. For i slike tider kan riktige målinger og tilhørende verktøy være til stor hjelp for å sikre at man iverksetter virkningsfulle og nødvendige tiltak, samt tilrettelegger for fleksibilitet i organisasjonen. OKR er et slikt verktøy. Med utgangspunkt i tre hovedelementer; Objectives, Key Results og Initiatives representerer OKR et fleksibelt målstyringsverktøy der man kontinuerlig kan definere, tilpasse og følge opp mål og tilhørende resultater. I 2017 ga forretningsmannen John Doerr ut boken Measure What Matters, der OKR for første gang kommersielt beskrives. Boken forklarer rammeverket i detalj og åpner med det en viktig verktøykasse for organisasjoner verden over. Hvis du ikke ønsker å bruke juleferien på å lese om målstyringsverktøy, anbefaler jeg deg likevel å lese min kollega Martins innlegg for en lettlest introduksjon til OKR.

Smittetallene øker, tiltakene strammes ytterligere inn og det er tydelig at Norge står overfor en ny smittebølge. Men, på tross av all denne usikkerheten og risikoen, handles paradoksalt nok flere verdipapirer og aksjer til toppnoteringer på børsen. Samtidig er belåningsgraden i norske husstander rekordhøy og antallet privatinvestorer på børsen er større en noen gang. For mange kan dette være skremmende tendenser som tyder på et usunt pengemarked, mens det for den eventyrlystne kan øyne om gode investeringsmuligheter. Som en inspirasjon for både den risikoaverse og den eventyrlystne anbefales derfor boken The Man Who Solved the Market: How Jim Simons Launched the Quant Revolution av Gregory Zuckerman. Boken tar for seg suksesshistorien til den amerikanske matematikeren og hedgefondforvalteren Jim Simons, som grunnla det hemmelighetsfulle investeringsselskapet Renaissance Technologies i 1982, og som siden har hatt enorm suksess med høyfrekvente investeringsstrategier drevet av finansielle data. I perioden mellom 1994 og 2014 oppnådde selskapets største fond, Medallion, en gjennomsnittlig årlig avkastning på 71.8%, noe som har gjort Simons til en av USA rikeste personer, med en nåværende formue på hele 23.5 milliarder dollar. Da Jim Simons er en svært mediesky og lukket finansprofil, er boken skrevet som en biografi basert på gravende journalistikk av Gregory Zuckerman.

Vintermørket faller på, det blir stadig kaldere, smittetallene øker drastisk og det er tydelig at Norge befinner seg i pandemiens andre bølge. Heldigvis kommer det stadige positive vaksinenyheter og mye kan tyde på at pandemien kan være over allerede i 2021. Selv om nyhetene er veldig usikre og lite tyder på en rask vaksinasjonsprosess fortsetter børser verden over å sette nye rekorder. Hva er det som gir denne veksten på verdens børser? For på tross av gode vaksinenyheter og enkel tilgang på kapital, har det vært noe særegent irrasjonelt ved den økonomiske utviklingen det siste året. For å bedre forstå hvordan vi mennesker gang på gang tar irrasjonelle beslutninger, kan boken Predictably Irrational forklare mye. I denne lettleste og svært interessante boken beskriver professoren Dan Ariely hvordan skjulte krefter driver menneskers beslutninger. Det er en gripende tankevekker som får deg til å reflektere over egne og andres valg, og som i tillegg kan hjelpe deg å bli mer rasjonell i egne handlinger.

Desember og 2020 er snart over, og vi har lagt et år med usikkerhet, muligheter, læring, tilpasninger og omstilling bak oss. Et unikt år vi vil se tilbake på som en rar reise med både oppturer og nedturer. Med stadig flere land som igangsetter vaksinering, er vi forhåpentligvis på god vei ut av pandemien, og vi kan gradvis returnere til våre vanlige, frie liv. Fordelen med slike spesielle år og hendelser er at verden utvikles; digitale hjelpemidler har fått et enormt løft, vi har blitt mer renslige og vi reiser langt mindre - listen er lang, og erfaringene er mange. Det er uansett alltid fornuftig å reflektere over tiden som har gått og ta med seg viktige læringspunkter videre. Et godt eksempel på slike refleksjoner er Robert Iger og hans biografi The Ride of a Lifetime. I denne boken reflekterer Iger over sin karriere som forretningsmann, med et spesielt fokus på hans tid som CEO i The Walt Disney Company fra 2005 til 2020. Det er en meget gripende bok om lederskap, om Disney som et moderne selskap og en beretning om hvordan enorme organisasjoner fatter store beslutninger. For gjennom Igers tid som CEO i Disney, ble selskapet for alvor et av verdens største mediekonsern gjennom oppkjøp av blant annet Pixar, Marvel, Lucasfilm og 21st Century Fox. Alt i alt en inspirerende bok om en dyktig leder, forretningsmann og forhandler som leder et selskap vi alle kjenner så godt.

Inn i det ukjente

Som bredden i den beskrevne litteraturen tilsier, har det vært et innholdsrikt år preget av utfordringer, usikkerhet og omstilling. På tross av dette har vi har lært, tilpasset og utviklet oss, etablert nye rutiner og funnet en delvis ro i en ny normal. Nå nærmer vi oss 2021, og vi skal på ny inn i det ukjente - heldigvis har du 1 juleferie å lese deg opp på!

Hvordan lese seg gjennom 2020 med strategibrillene på?

Kultur spiser strategi til frokost. Særlig når den omfavner psykologisk trygghet blant de ansatte. La oss se på hvorfor vi trenger dette, og se på noen enkle verktøy vi kan bruke for å metodisk skape denne kulturen i teamene vi sitter i.

Å designe en kultur med psykologisk trygghet

Earlier this year I tried using three.js within a React app. It wasn't straightforward, and I ran into quite a few problems. Since then, I've been eager to try out this library a co-worker told me about: react-three-fiber. Apparently, it could solve all my problems (ok, maybe only a few) and make it super easy to create cool animations with few lines of code. Say no more!

I am far from an expert in three.js, but I want to share my experience and give you a quick introduction to react-three-fiber. If you've caught yourself staring longingly at amazing animations, thinking "oh, I wish I could make that!", but not quite found the courage to start, then this might be something for you. Since it's component based, it's mainly plug-and-play! 🥳

Getting Started

react-three-fiber is a React renderer for three.js, which in turn is built on the WebGL API. The ninth article in our JavaScript calendar was about animation in three.js, written by my colleague Holger. We are going to recreate his example in the article using react-three-fiber. If you haven't read it already, hop over and do so before you continue reading this article.

There are a couple of things to know before you get started. To display the animations, three.js renders a HTML <canvas> element with your content in a loop inside it. We need to prepare this canvas before we can "paint" on it (or animate, if you will). For this, three.js requires a scene, camera and renderer. In basic three.js, you would need to define and configure these yourself (as you probably saw in Holger's article). Luckily, react-three-fiber does this for us, which means even less boilerplate code! We can use the Canvas-component directly:

react-three-fiber provides us with some neat default values, which makes the setup easy. For example, the camera will look at position [0,0,0] (the center of the canvas), but a little bit zoomed out (z-position = 5). It gives us a good bird's eye view of the scene. Now, when our content is placed in the center by default, we are able to see it (because we aren't "standing" in the middle of the content). Of course, there are a great deal of properties available in the Canvas-component if you want to customise things later.

Comparable Cubes

To illustrate how easy react-three-fiber is to use, compared to the good old three.js way, we are going to create the same red, shaking cube as in the article mentioned above.

First, we'll need to change the position of the camera in the canvas to reflect the example we are reproducing. To achieve this, we simply set the camera property in the Canvas component, using x, y and z coordinates: camera={{ position: [2, 2, 2] }}. Now we are looking at the scene a little sideways.

Then, we can add objects to our scene inside the canvas. An object is created using a mesh, which is given a geometry (a cube) and a material (the appearance of the cube; red 🟥), which we render in the scene. In react-three-fiber, we just need to put it inside the canvas element, along with some light to actually see it:

Now we've got a red box in our canvas! You can try and change some of the numerical values and see what happens. Maybe even try to remove the light?

Shake It Up

Next, we're going to make the box shake when we click on it. We need to use the rotation property of the mesh to make it move.

We start by separating the cube into its own component, in order to use a react-three-fiber hook called useFrame(). It must be called from inside the Canvas, so we can't define it directly inside our Component. The hook will execute the code inside the callback just before every frame render (60 times per second). If we use the callback to change the rotation of the box, we can make it look like it's shaking.

We have a box, and we have an initial rotation (which is zero). Our hook doesn't do much yet. In his article, Holger explained the details of the shaking animation he made. Let's borrow that nice code from Holger to shake it, baby!

And there it is! With a few lines of code we made something pretty cool, right? Take a look at the final result in the sandbox below:

To compare the code to the version made using only three.js, you can find Holger's sandbox here.

If you want to play with animations in react-three-fiber, the holidays might be the perfect opportunity for a new hobby project. I'll be linking some useful resources below to get started. And while you're at it, why not try out the brand new Create React App 4? That's a combination I can vouch for :raised_hands:

Happy Holidays, and happy coding! :christmas_tree: 🤗

Deck the Halls With React Three Fiber

I'm one of those people who want to learn and build stuff but has way to many things on the wish list and then most of them never happen. 😅 Trying embedded programming has been on the list for a long time, and as I got into functional programming it seemed to fall even further down on the list. From my point of view embedded programming involved C or other lower level language which did not appeal to me at all.[^rpi] Fighting for attention with FP hot shot languages as Elm and Haskell, embedded projects as I thought of it just never got far enough up on my list.

Then an event spurred some more research. By January 1, 2019, every house and aparment in Norway where reqiured to have a smart meter[^smartmeter] with the capability for the owner to read out the power usage programatically.^ams This data is available through a port on the smart meter called the HAN (Home Area Network) port. Power usage data is published to this port on regular intervals in a standard format that all the smart meter vendors implement.

This seemed like a great opportunity to get some embedded experience and after some research I found the Nerves project. Nerves is an Elixir based project, which for many probably begs the question "is a functional language like Elixir really a good fit for embedded projects?". As it turns out, with the right tooling Elixir is an excelent candidate for this.

Why Elixir?

I will not cover Elixir in detail in this post, but check out my previous post, The case for Elixir, for an introduction.

There is a few features that makes Elixir great for embedded/IoT projects:

Binary pattern matching

Elixir has (inherited from Erlang) a syntax for pattern matching on binary data. This makes it very easy to parse binary protocols (hey there, HAN-port data!). In other languages you would have to do bit shifting which, for people not used to it (and maybe when it is not new), is much harder than just pattern matching out the different parts of the binary data.

As a simple example, lets say you have a binary protocol which represents the a date as "yearmonthday" ("20201223") and is encapsuled by a byte, 0x7E (01111110), on both side. The binary data may be represented like this: <<0x7E, 0x32, 0x30, 0x32, 0x30, 0x31, 0x32, 0x32, 0x33, 0x7E>>. To retrieve the date and the individual parts you could do a simple pattern matching:

This retrieves the individual parts of the date into sperate variables.

Fault tolerance

Elixirs, or rather Erlang VM/BEAMs process and supervisor model makes it easy to create fault tolerent, isolated applications that fit together and can handle the turbulent connectivity that might be present where you deploy your embeded system. Even just for internal error handeling, having the runtime there to kill a failing proccess and start it again with a known good state is better than the alternative in other languages with might in some instances be to completely crash.

OTP releases

Nerves leverages OTP releases to create a deployable binary that is optimized and compiled for the target system. This is similar to (but not the same as) compiled languages like Go and Rust that produces a statically linked binary.

What is Nerves?

Nerves simplifies the process of taking an Elixir application and packing it up into a firmware that can easily be deployed to almost any computer that can boot from a SD card or hard drive.

Booting into goodness

The output of a Nerves build is a firmware which basically is a custom linux distro that boots straight into the Erlang VM. Nerves leverages the Linux kernel for drivers so most peripherals will work out of the box.

The firmware written to the SD-card has different partitions, most notably two separate root file system partitions. This is to support a blue/green deployment where on update of the firmware the new firmware will be deployed to the unused partiton. The device then reboots into this new partition. If for some reason this new version of the firmware does not work, the device will reboot back into the previous and working version. This system prevents bricking devices with bad firmware and makes the remote updating functionality that Nerves support a practical and safe feature.

All the partitions except from the application data partition are read-only. This will prevent filesystem corruption which can happen if the device writes to the filesystem while loosing power.

A newly generated, minimal Nerves project firmware size will be about 40mb.

Hardware support

Nerves support all a lot of different hardware:

Every Raspberry Pi ever made

Beaglebone Black, Green, Blue and PocketBeagle

x86_64

GRiSP 2

Custom hardware and most boards that can run embedded Linux (via Buildroot)

The most common for getting started is probably the Raspberry Pi line of boards but other boards are supported and with some extra work you can make it work on hardware that kan run embedded Linux.

Nerves uses Buildroot to create a custom firmware for the device you are targeting. The common boards has premade "images" so unless you are doing some custom board or peripherals, you will not need to learn Buildroot. Just build your application and run a command to get the firmware for you target system. Buildroot takes care of cross-compilation for other architectures like ARM.

But Nerves do not stop giving there. How about a system to manage a fleet of Nerves deployments? Enter NervesHub!

NervesHub

NervesHub is an open source firmware update server. It enables you to do over-the-air updates to the devices deployed with the platform. NervesHub can be self hosted or used as a managed version at nerves-hub.org.

Security is a primary concern of NervesHub and it has several features to secure deployments and update:

Cryptographic signatures on firmware updates provide end-to-end authentication

Client-side and server-side SSL provide link authentication and encryption

Access control within organizations to firmware updates

If you enable it NervesHub can give you remote console/iex access directly to the device in the browser.

What is Nerves used for?

So what is Nerves used for? Lets look at some different examples!

Hobby projects

Even in small hobby projects Nerves can simplify deployment and maintenance. One of my projects is a dashboard on a Raspberry Pi touch screen, nerves-pidash. For the first version the backend was written in Go. This worked well for a while. Go produces a staticly linked binary which makes deployments easy compared to other interpeted langugage where you need to make sure all dependencies is available.

The hassle comes when the device loose power and I have to manually go and start the backend and open the browser to the correct URL. This became tedious and of course I had the well known SD card corruption issue because of power loss.

Nerves' way of handling the filesystem and updates makes sure that power loss does not cause corruption and with the nerves-web-kiosk/kiosk_system_rpi3 the device starts a browser with a predefined url. Changes to the system is a breeze with OTA updates.

With the Circuits library you can interact with many different sensors with support for GPIO, UART, I2C and SPI.

For my smart meter project the hard part was actually to find out how the binary protocol works. Reading the data from the device is easy with Circuits, and implemented in just a few lines of code. Admittedly this project is not in a working state and some code has been lost. Remember to check in your code! That is not something Nerves will do for you. 😅 For the interested the project can be found at https://github.com/ringvold/han_ams.

But there are more serious projects out there. Some are even basing their business on devices powered by Nerves!

The more serious projects

The Nerves project web site has several show cases where you can read about how Le Tote used Nerves to increasing warehouse efficiency, how Hop used Nerves to build the world's first self-serve beer kiosk powered by facial recognition technology or how Nerves enables open source smart farming through FarmBot.

One of my favorite real world cases is Realtime Vehicle Tracking with Elixir and Phoenix by James Smith from ElixirConf 2017.

Wrapping up

For a good introduction I recommend checking out a talk by one of the co-authors of Nerves, Justin Schneck: Functional IoT with Elixir and Nerves[^fpiot]. It goes a bit more into detail in some areas and have some awesome demos at the end!

If you have a project involving a Raspberry Pi or another embedded device I would recommend you to check out Nerves. It simplifies a lot of the hard parts about embedded programming and helps you focus on solving the problem at hand and not having to build you own deployment schemes. And you can do it all with functional programming! 😄

[^rpi]: Raspberry Pi have made this more aproachable with python libaries, but that still did not tickle my fancy enough to get started. [^smartmeter]: Smart meters, AMS (Advanced Metering System), energy meters, digital electricity meters. They have many names!

Functional IoT with Elixir and Nerves

Smidige virksomheter har for lengst tatt til seg hyppige læringssykluser og hypotesetesting som metode. Det prøves og feiles og læres raskt om alt fra kundebehov, organisering, teamdynamikk og løsninger til strategi og eierstrukturer.

Men hvor blir det av akademia i alt mylderet?

Pia Solheim, Moquan Chen og Jens Andreas Huseby tok en prat

Ligger akademia bakpå?

Do you get convinced by spreadsheets with inarguable and objective facts? Imagine a talk where the presenter lists facts, from behind a lectern, at a comedy conference. It is hard to picture this would sway the listeners. What does it take to create a compelling and convincing talk?

Consider the following talk, commenting on the Covid-response

When Dr. Michael J. Ryan starts talking, we believe him. We trust what he is saying. He doesn’t list facts or numbers, but his very being, his tone of voice, his demeanour and his authenticity, all of this works to his advantage to create a compelling argument.

We believe him, even though most of us has never heard of him before, let alone heard him talk or know about his credentials as a world health official.

The purpose of presenting or giving talks, is to convey a thought or opinion for the listener to take to heart.

In order to create compelling arguments and convince the audience, the presenter must seem credible. Credibility comes from competence on the subject, his/her perceived moral character and the good will of the listeners.

Ethos

Ethos is the perceived moral character of you as the presenter, in the eyes of the audience. Consider Dr. Michael J. Ryan. Even before you started watching the above video, you start making an opinion about the person talking. We look for identifiers and markers by instinct. The person in the above video is introduced as Dr. Michael J. Ryan, he is wearing a blue shirt and a dark suit. What he is about to say is in all likelihood important, as the banner at the bottom states, in capital letters, that this is breaking news. He is sitting in front of a blue backdrop with the WHO-logo.

This introductory ethos, all that we ascribe to the presenter, is a part of your argument or your talk. The listeners will have preconceptions based on how the conference introduces you, your title, your employer, and so on, so forth.

There are many ways to establish ethos, such as describing your years of working within the field, the number of hours volunteered, or personal experience with the issue at hand.

Logos

Next you need your logical appeal, all that can be counted, measured and otherwise perceived to be true. Here you are turning to sense and reason. This involves taking your audience with you on a journey to see the same facts and truths as those you hold, helping them see your way of reasoning.

When Dr. Michael J. Ryan says "you need to react quickly", he is basing that on the learnings from dealing with the Ebola outbreak. He is effectively saying that what worked the last time will also work this time. He is able to do this because of his credibility as a source, his ethos, but also because he begins with saying "what we have learned from the Ebola outbreaks is that…".

For this to work, he needs the audience to remember the Ebola epidemic and the consequences of that. It requires that the audience shares a frame of reference and has at least some knowledge about the current state of the Ebola outbreak.

Pathos

Pathos is the appeal to emotions and where you move to action. Creating will to act. This works to balance out the logos-part, requiring you to appeal both to reason and to emotions.

When people are talking about winning the “hearts and minds”, pathos is the heart. Stirring emotions about the subject at hand, in order to encourage action. Be it to act fast or dare to be wrong.

There is little doubt that Dr. Ryan is passionate about the topic at hand.

Kairos

The setting of the presentation, the meeting, the talk. The time and place of it all. Kairos is about being aware of the rhetorical situation you are in.

A press conference, as in the clip above, is an entirely different setting from an informal meeting with a new client. This matters, and has great impact on what your listeners are expecting, and what they are willing to accept.

This also means that you are able to deliver your talk with good rhythm and in tune with your audience.

Dr. Michael J. Ryan, starts the clip above with running through a list of bullet points in fast order.

You need to react quickly

You need to go after the virus

You need to stop the chain of transmission

You need to engage with communities […]

You need to be coordinated

You need to be coherent

Before changing pace (about 43 seconds in)

Anyone who is involved in emergency response will know this [short pause] If you need to be right before you move [longer pause] you will never win.

The change in pace underlines the importance of what he is about to say. There is no doubt that what comes after the short, first pause, will be important. As listeners, we edge to the front of our seats, awaiting what comes next.

While this can seem daunting and like many things to consider, remember that your credibility is one of the most important factors in all this. As we have seen in the example above, the process of establishing credibility begins before uttering a word, or even entering the stage.

Being authentic and with a genuin passion for what you are talking about is probably the best advice to create compelling arguments and a convincing talk.

The building blocks of compelling arguments

TypeScript is great because it lets you write statically verified code. You'll never try to access a property on an object that doesn't have it, and you'll never get a undefined is not a function. Except there are holes.

Have you ever written code like this?

This sucks, because we are just hoping and praying the response from the /users endpoint matches our definition of the User type, which might look something like the following.

Usually it does, but there is never a guarantee - and especially not a guarantee that the data from the endpoint never changes. It's even more insidiuous if the data mostly matches, but sometimes it returns something slightly different, because you'll probably not catch it when developing and maybe not even when testing. And that's how you ship a bug to production that should (and could!) have been caught at compile time.

Alright but how do you catch that at compile time? Without some kind of integration of your backend and frontend types, how do you statically verify the shape of your (possibly external) API's data? Well you can't really do that easily, but you can verify that the data actually does match the type you say it has, as it comes in. So that's the idea, verify early and fail hard. Instead of mindlessly casting (using TypeScripts as operator) where you introduce the possibility of error, we would like to actually test and make sure the data actually conforms to our type and instead reject the Promise if it doesn't. Which I think makes sense, there is no practical difference to your app whether the external endpoint is down or it returns to you data that you don't understand.

But this seems annoying, should we write parsers or validators for every type that comes in from an external source? That takes a lot of time, and is doubly annoying to maintain as our app and our APIs grow and change. That's why I wrote TypeScript Json Decoder. It is a library that automatically creates what is often known as "decoders", functions that make sure your data looks the way you say it does, based on your types. It has no external dependencies, is lightweight, and one of its core values is that it resembles and can be swapped in, in place of your existing TypeScript type definitions without any modifications or limitations. I want this to be idiomatic, regular TypeScript with as little friction as possible. The API surface of the library is designed to be intuitive and as small as possible, so that it's not a case of "yet another library" for people to learn and manage in their code. This is merely how I wish TypeScript already worked.

Check out the GitHub page to see lots of examples and explanations of the more advanced features needed to express everything a TypeScript type can.

Let me give you a basic introduction to the underlying idea. If we wanted to replace our User type with a decoder for that type, we would instead write the following.

And to use it we replace the cast with a call to the decoder.

Or rather a decoder of an array of userDecoders, assuming our endpoint returns a list of users. Notice we have to parse the json first using a call to fetch's standard Json parser, x.json(). This is because although the library is called typescript-json-decoder, it actually operates on plain old JavaScript values, such as objects, arrays, numbers and strings. It's maybe a bit of a misnomer, but it mostly makes sense to think about it this way. The nice thing about it is that you can pass any valid JS value to a decoder, and it will either return the decoded value or throw an error. The error reporting actually just throws a string describing the error, which rejects the promise if you use it in an async context. The decoder itself isn't async or aware of promises or anything like that.

At the cost of one line of boilerplate this gives you complete type safety when handling external data. I think this pattern should be used every place you receive untrusted data, and that includes endpoints you own. I at least don't trust myself to get these things right, there are too many opportunities to mess up APIs, even if it's just misspelling a field name when typing in the type.

How it works

The way this works under the hood is pretty interesting. If you pay close attention you see that what we have defined is not a type at all, it's actually a JavaScript object that kind of looks like a type definition. This is because it's actually impossible in TypeScript (without hooking into the compiler) to do the kind of metaprogramming where you inspect the type itself. So instead we do the opposite, we create a regular value which represents the type we wish to generate, and from that we generate both the type itself (and call it User) and the decoder. The special TypeScript operator typeof (named after a similar, but different, JavaScript operator with the same name) is used to extract the type of the decoder from the JavaScript object that defines it, in our case userDecoder. This type is then transformed to the corresponding type which the decoder is supposed to decode to.

The idea is that the library supplies all the primitive decoders such as number, string, and even array (which takes another decoder as a parameter), and then we combine these to build bigger decoders. The way these decoders are defined is really simple, take a look at the following definition of the string decoder.

Note that the library refers to regular JavaScript objects as Pojos. This decoder doesn't actually do anything! It just returns the string it's passed, if it is a string, or if not, throws. So this is actually the identity function for strings.

The complexity in the library is elsewhere; the core idea is that JavaScript values are considered to be decoders of themselves. I call these literal decoders. For example, the string "hello" is a decoder of the literal value "hello", and only that value. That might not seem so useful, but once you extend it to records, it becomes quite powerful. A record containing decoders, is a decoder of a record with the same fields. In a sense, it is a decoder of itself. In other words, it decodes a thing that looks like itself. And this is defined recursively, which allows us to nest objects, or have arrays of objects, or arrays containing objects of objects containing arrays, and so on and so on. In this way you can define your own "custom" decoders and compose them arbitrarily.

The way the decoders are evaluated, then, is a pretty straightforward recursive traversal of the tree structure of decoders where it applies the decoders it finds. What is much more interesting, and honestly where all the complexity of this library resides, is in the innocent looking decode type level function, which is responsible for taking a decoder and producing the type of the thing that decoder decodes.

What. A type level function? So TypeScript actually has an incredibly sophisticated type system. I come from a background of having fallen in love with the intricacies of Haskell's type system, but TypeScript is in many ways more advanced. This comes from the necessity of being able to express a lot of the patterns commonly found in JavaScript, which is by nature very dynamic. Anyway, a type level function is a function, at the type level. Bear with me. decode in the example above is not actually itself a type, rather it is something that when given a type, returns a type. A function of types! A function from a type to a type? Can types have types? This is sometimes referred to as "the kind" of a type, but if we don't want to confuse ourselves too much, we'll just think of this as something that you can give an existing type to, and get another, new, type out of.

If you think about it, a decoder by necessity does not have the same type as the thing it decodes. Instead maybe it has the following type: (x: Pojo) => User. So it takes any plain old JavaScript object, and returns a User (if it can, that is; TypeScript doesn't have checked exceptions, so you won't see the failure case in the type). The type this decoder decodes, is User. So the following type expression: decode<(x: Pojo) => User> evaluates to the type User, or generally whatever the decoder decodes. This all is complicated further by the fact previously mentioned that not all decoders are functions; some decoders, for instance a record decoder, has a literal form.

However, let's ignore that and first think about how to extract the type that a (function) decoder will decode.

Now this is the kind of metaprogramming that gets me going. What in the world is going on. Well, first of all we have a ternary - essentially an if test on types. The thing we are testing on is the decoder extends (x: Pojo) => (infer T) part, which is a subtype test. The extend keyword, I think, is a horribly chosen name in TypeScript, mostly carried over from other contexts. What it means is "is a subtype of". It is a question asked of the type parameter decoder, are you a subtype of the type (x: Pojo) => (infer T)? Which begs the question, what is infer T? Well, it is whatever it needs to be to satisfy the subtype test. If decoder is the function type defined above, (x: Pojo) => User, then T would need to be User for the one to be the subtype of the other - at least if you consider being the same type as being a subtype. The keyword infer is used to introduce a new type variable. In the first branch of the ternary we return the type T if we have a match (that is, the decoder is a function type), and in the second branch we return TypeScript's bottom type never, indicating this should never happen. If this does happen, and we try to use the resulting never type for anything, we get a compiler error. never is the empty set, if you are inclined to think about types as sets. There is no value of this type.

This is actually the definition of the type level function for extracting the return type of a function (that's a mouthful), and is in fact available in the standard library under the name ReturnType! So that was a lot of type theory for very little. We need to go further. I mentioned that the essence of this library is to understand literal values as decoders of themselves, and to define this recursively in the case of records. So let's add records!

Here I've formatted the ternary to make it resemble a classical if block a bit more. It's kind of elegant in a way. It's the classic recursive pattern, a base case where we break the recursion when there is no more work to do, and a body where we recursively call ourselves with a smaller set of the problem. Here we use TypeScript's well understood mapped types to iterate, or map I guess, over the keys and values of the record. For every key we evaluate decode on its value. So the following type, which would be the type of the decoder of our User type,

would evaluate to what we expect, namely the User type itself.

Now hopefully it makes sense to think about the series of steps it would take to evaluate a nested type defintion. So if we were to add an address to our User:

we would get the following evaluation steps.

And that's basically it! If you're curious to see the actual implementation I suggest checking out the source code on GitHub. It's a bit more involved, but I have tried to keep it pretty clean. I would very much welcome PR's and suggestions for improvements. Thanks for reading!

Twitter: @_tskj_

Json decoders in TypeScript

Now that you've learned how to define both incomming and outgoing ports, you might wonder how many of them you need. It might seem reasonable to require two ports for each javascript function you wish to call, but what if I told you'll need, at most, two for the entire application?

Now that you know about ports you might be wondering how you'd make use of them in an actual application. For the sake of this article, lets pretend that we're going to implement a todo application. We dont't want our todos to dissapear every time you leave the application, so we've decided to write a little bit of javascript to store our todos in local storage:

To access this code from the Elm side of our application, we'll need to use ports. One way to wrap this javascript code looks like this:

And on the Elm side of our code we need to define our ports:

For the sake of brevity, I'll leave out the Elm code which shows the actual implementation. We should have covered that in the past two days.

While this works, you're probably thinking that this is a lot of work for wrapping three functions. Is this really how javascript interop is supposed to work?

The answer to that is yes, and no. Yes, all javascript integration is supposed to happen through ports but no, it's not supposed to be this much work. What we need to do is to think a bit differently about how we should interop with our javascript code. Instead of wrapping each and every function, what should do instead is to treat javascript as a service. A service with a well defined API, and which responds by sending us messages about important things.

Take a look at the javascript code again. Is it really important to get a message each time one function has successfully run? Probably not. In fact, the only thing we really care about is the state of our todos. A better setup of ports is:

So you see that by changing our mindset a little, we can reduce the number of ports we need. This is good for several reasons:

Ports are not namespaced. Two ports with the same name, even when defined in different Elm modules, will cause one to overwrite the other.

Having many ports might make it difficult to keep track of how an application makes use of javascript.

Actually, when looking at the one subscription we have left, you might realize that we don't actually need more ports than two: one for outgoing messages and one for incomming messages.

Since you can send and receive json with ports, you can send any number of complex messages into and out of Elm. You also get greater control when encoding/decoding json. You can ignore values or be more strict, or even less strict, about what you expect from incomming values.

Conclusion

When working with ports it's important to not be too specific. Clearly define your API, and only implement messages that actually matter to your application. At the same time, having many ports do not scale well. Try to keep to as few ports as you can. Remember, you only ever need two.

23-Dec

Today's posts