While using Java or Kotlin, have you ever needed the actual type of the type parameter in a generic function? Meet reified!

Search for reified on github, and you'll get about 247k hits. Limit it to Kotlin, and it's around 50k. But what is it?

A generic function

First, here's the signature of a generic function

The function allows us to perform some operation on an Int which should be able to return any type. However, as we don't have any information about the type returned, the function is severly limited. Functions with a signature like this is often just some CRUD operation on a generic container type(e.g. List).

The KClass parameter

To be able to do something based on the type of the generic parameter, we introduce the KClass parameter.

Usually, the KClass argument is used to reveal the actual type to the function. By doing it this way, the type can be revealed without the call-site having to instantiate and pass in a dummy object. In the example, this is seen to be one way that the FasterXML/jackson project offers to deserialize from some supported format to an object. A similar solution is also offered by other libraries, like GSON.

A downside with the current implementation is that it doesn't support generics very well. The class passed will be that of e.g. List, but it won't have any information about the type of the List elements. And isn't that extra parameter a bit nasty?

Reified type

Enter the reified keyword.

reify: to consider or represent (something abstract) as a material or concrete thing : to give definite content and form to (a concept or idea)

With this defintion from Merriam-Webster in mind, lets look at an example. One caveat though; reified requires it to be an inline function, denoted by the inline keyword. Inline, simply put, means that the compiler will copy the function and paste it in place at the call site for us.

By adding reified to the type parameter, we get access to the generic type like it was an actual type. No more passing ::class parameters, just call the function with the type parameter!

To sweeten the deal, remember that Kotlin has type inference. Let's trim it down just a litle bit extra. If you want Jackson to support generics and their element type as well, such as List<Name> instead of just List, replace T::class.java with object : TypeReference<T>(){}.

Now look at that. One could even suspect that someone had thought of functions like this already!

When we need a concrete generic

In this post, I'll show that C# implements the functional concepts map and bind. I'll further show that these are useful for other contexts than just collections with an example implementation for Task<T>. We'll then clean up a process by using map and bind to compose a new process of several smaller processes. Hopefully this will show that functional concepts both exists and are useful even in non-functional languages.

The function map will transform some value in a context, potentially changing the type, but stay in the same context. Most languages doesn't have a way to talk about "some context", and will implement the function for each context, and the only similarity between them will be the name map as the language isn't able to express this function.

C# is such a language, but even if we're unable to create a generic interface that states "this context implements map", we can still implement a method with the same name which operates the same way.

In C#, we have IEnumerable<T> which is an interface that supports enumerating over the elements in a collection. The mapping function for this interface is implemented as an extension method, and is named Select rather than map.

This could have been called Map instead, but Microsoft implemented this as a part of the Language Integrated Query feature (iirc), and modeled it after SQL where it's called select and where rather than map and filter. I'll use the functional name and C#s name interchangeably, so if you wonder what I'm talking about, refer to this table

If C# supported the "some context" feature, the above code could look more like the following pseudo-C#

So we're unable to express this, but let's revisit what map actually does; it lets you transform something in some context, but stay in the same context. This is a feature we should be able to implement for various different contexts, and languages which supports this actually does so (it's called Functor in other languages).

One useful context I've implemented map for is Task. A Task is an asynchronous operation that you can start in the background and wait for the result when you need it. Often you'll also want to post-process the result before continuing as is the case when fetching data from a database or a web API. C# doesn't implement map on Task, but they have implemented a more powerful function called ContinueWith we can use to build our map.

This is our implementation of map. It will transform the value within the Task in the same way the operation with the same name does on a collection.

Well, this was a bit of an anti-climax :/ But fear not as we'll use our "new" functionality in order to implement another very useful operation called bind!

Let's look at another feature of IEnumerable called SelectMany. Much like Select, it will transform each value in the enumerable, but instead of returning a simple value, it returns the value wrapped in another enumerable. It will then flatten (also called join) the result so we don't have IEnumerable<IEnumerable<TResult>>.

But in the same way we wanted to generalize Select for TSomeContext, SelectMany could also have been generalized.

Let's look at an example where we usually could get a TSomeContext<TSomeContext<T>>.

We could wrap the specific things inside a new Task so we get all code specific to this process in one place and not interleaved with unrelated code.

But if we squint a little, we see that we transform the output from the first, and then use this to create a new Task. This sounds a bit like map followed by join. And the combination of these two is called bind (or SelectMany).

We start by creating a function to extract the value from the task. This already exists as Result, so we just wrap it for kicks.

join / flatten will just remove one layer of the context.

And we can implement SelectMany in terms of these

Our example thus becomes

Tada! We have shown that Select is the same as map and SelectMany is the same as bind, and we've implemented these for Task to allow composing operations the same way as we usually do for IEnumerable.

Now try to implement Select and SelectMany for Nullable<T>!

PS: Notice that we implemented map (Select) in terms of bind (ContinueWith). It is possible to implement map in terms of bind for anything which implements bind.

PPS: Task.FromResult (often called pure or return, which would be confusing in C#) will put something in a task which would be returned when Result is called. This way you can mix in other processes in code that operates on Task. This method, together with bind (ContinueWith) pretty much makes Task into a Monad without looking like scary category theory.

Map and bind - A hidden functional concept in C#

One of the many branches of machine learning is Natural Language Processing, or NLP, where computers are trained to understand and generate text and speech. However, while discussing potential use cases for the technology, we realized the need for an overview of language processing resources for Norwegian.

To explore this, we did a conversation analysis PoC – analyzing and presenting the contents and metadata of a conversation between two or more people talking in Norwegian. This blog post will give a quick intro to some of the most useful resources for Norwegian NLP, to help you get started with your own NLP project!

Analysis output of a test discussion revolving around NLP product development.

Transcription

Our first task was transcribing the recorded speech. After evaluating several options, the Google Speech-to-Text API emerged as the best alternative, scoring as low as 7 % normalized Levenshtein distance and 17 % Word Error Rate on test samples. This corresponds well to the results of recent tests1 evaluating different speech-to-text APIs in English.

Diarization

A key part of speech analysis is speaker identification, known in the field as diarization. As opposed to most NLP methods, this should theoretically be language indifferent. However, as Google only supports diarization of English (as of Dec 2019), we instead employed one of the many available Python libraries, pyAudioAnalysis, for diarization, achieving at best 96 % accuracy in takes with different gender speakers.

Summarization

As for the summary of the text, we used the Gensim library, which offers an extractive summarization model based on the TextRank algorithm. In our experience, the function is effective when input text data is of high quality, but becomes equivalently confused by low-grade transcriptions. We experimented with abstractive summarization models for English (translating the transcribed speech back and forth), but any meaningful insight was clearly lost in translation and/or transcription.

Keyword extraction

To extract keywords from the transcription, we implemented a modified tf-idf algorithm, aided by a Norwegian Snowball stemmer provided by NLTK2. With a careful optimization of the TF/IDF weights, stopword deletion and outlier removal, the program was able to extract highly relevant keywords (for Norwegian speakers: top left box in the analysis output above).

POS tagging

One of the most sophisticated libraries we encountered was spaCy. It has a trained convolutional neural network model for Norwegian which enables context-based recognition of named entities, part-of-speech tagging and even dependency parsing. To demonstrate its POS tagging abilities, we let spaCy analyze the following sentence:

or in English, "This year, Bekk is publishing twelve Advent calendars". The main challenge posed here is the fact that bekk, apart from being our company name, also means brook or creek in Norwegian. While brook and creek are clearly common nouns, bekk, in this case, is intented as a company name – that is, a proper noun. We run the following code:

receiving the output:

As we see, spaCy understands from the context that bekk is in fact a proper noun! This is a very helpful tool in extraction of named entities, but also structural sentence analysis or at some point even abstractive approaches.

This has hopefully been a useful intro to some of the many available resources for Norwegian NLP! We tackled several other challenges, including sentiment analysis, speech time mapping and sociogram generation, which may be covered some other time. Feel free to drop me an email if you're interested in hearing more✉

1https://www.rev.ai/blog/how-to-calculate-word-error-rate/, https://medium.com/descript/which-automatic-transcription-service-is-the-most-accurate-2018-2e859b23ed19

2When combining stemming with tf-idf, we recommend grouping words on their stem, setting the stem's TF to the sum of each word's TF, and the stem's IDF to the lowest of each word's IDF.

Conversation analysis with Python

People like to understand what's happening around them, whether it's in real life or in an app on their phone. Let us have a little peak into how we can help users of our React Native app.

A lot of apps have lists. Lists are great. They can display information in a vertical or horizontal way. But they can also be hard to understand sometimes. In this case we'll take a look at living lists where you can remove items off of it.

Let's say my hypothetical kid, Ove, has written a wish list for Christmas. Ove is a good kid, but he has a hard time deciding what he wants. So, when compiling his list, he's constantly discarding items off said list. Being the developer dad that I apparently am, I made him an app. This might look a little something like this:

The problem here lies when Ove taps an item, it disappears. Thanks to the colouring we might understand what just happened. But before we blink the item has vanished. Ove is crying, he didn't understand. This is where React Native's lovely LayoutAnimation API comes in handy. With just a few lines of code we can help him out.

Our code to remove an item from the wish list might look like this:

Our problem is that the item vanishes in an instant. To remedy this, we really just need two lines of code. Firstly, we need to import LayoutAnimation itself by using the standard import { LayoutAnimation } from 'react-native' To actually use it, all we need to do is adding a configuration for when we remove an item. So, by extending the above code block to

We tell React Native that we want to animate whatever happens on the next render.

The result being:

It really is that easy. Personally, I find this a lot clearer. And there's quite a few more things we could do. The above example is just a preset. If you want to have more control, you can supply LayoutAnimation with a configuration object for what happens when creating, updating or deleting objects. We can also define how the animation should look with different curves.

Read more about that here

Micro animations to the rescue

I used to think that feature switches were clunky if-statements in my code that would require re-compilation or at least restart of my app. Launch Darkly proved me totally wrong, swinging me from cynical scepticism to deep love!

Feature toggling as a service is best explained by a live demo. This 60 second sample from our enterprise app shows how we can instantly enable/disable a feature. And it shows how we can also do that for a specific user of our app!

Your browser does not support the video tag.

Targeting and logging

Although not shown in the video, Launch Darkly obviously also allows you to target groups of users based on whatever criteria you have available. It's also backed by great audit logging and statistics, so you can see how feature configuration has changed over time and what values have been served to the users over time.

Getting started

Launch Darkly has a nice SDK, allowing you to integrate with all popular programming language, including packages for both general JavaScript and React apps. There are also 3rd party libraries like Flopflip, which smoothly mirrors the flag states to your Redux store. This is what we do, and it makes it really easy to debug issues related to feature flags, e.g. using time-travel.

Show me the code

In your React app, you just add this snippet around your code:

and then for the specific component that you want to toggle:

With plain Javascript you initialise like this:

and can access the value like this:

Is that all?

You can of course do more complex logic based on your feature-switches too, e.g. have a fallback component which is rendered in the case the feature is OFF. Finally, I should mention that feature flags can also be string values. This opens up for multi-variation scenarios and we have "misused" this as a low-effort solution for showing localised warning messages whenever we have system issues.

Launch Darkly also provides a rich, modern API which allows you to build your own tooling if you prefer. That can include e.g. automated monitoring of toggles that need to be archived: when adding toggles is this easy, you quickly end up with a lot of them.

Empowering the stakeholders

The ultimate proof that Launch Darkly adds real value to our development and deployment process, is what has started to happen recently. Our (non-technical) product owner is not only in charge of deciding when new features should be enabled, he is actually doing it himself, without even asking us. The first time it happened, I was baffled, but it is really how things should be — empower people to take the decision that are important to them.

Feature toggling as a service – ❤️at first sight

Have you ever found it hard to debug Streams in Java? Allow me to introduce you to a very powerful tool in IntelliJ IDEA; Mr. Stream Trace.

The introduction of Streams in JDK 1.8 was a welcome change in the Java community. It allows developers to express their intent in code better than the good ol’ loop, encourages less mutability and works well with a functional mindset. There are, however, some disadvantages. One thing that bothered me was how hard it is to debug.

Let's say we have the following (admittedly slightly artificial) piece of code:

Are you able to instantly see what the result of this expression will be? No? Lets try debugging and stepping through each chain in the pipeline.

When learning how to use the Stream API, I had recently learned a super power in IntelliJ: Evaluate expression. I’d mark e.g. the first two lines of code and use my new favorite tool:

That was … underwhelming. Where are my numbers?! The only thing looking remotely interesting is this$0. Surely this are my numbers, right?

No 🤦‍♀️

This is because, as we all know, Stream operations are lazy:

Streams are lazy; computation on the source data is only performed when the terminal operation is initiated, and source elements are consumed only as needed. https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html

So I guess I have to perform a terminal operation:

Well, yes, this works. But I can no longer just select the interesting code, press a shortcut and see the results. If I want to figure out what the next step in the pipeline does, I have to close the dialog, increase my selection to the next line, press the Evaluate Expression shortcut and add .toArray() again. I don’t have time for that!

Enter my new favorite feature of IntelliJ: «Trace Current Stream Chain». It’s not the most visible button, but you can find it in the Debug pane:

Simply press the magic button (or assign a shortcut!) and you’ll see the following panel:

This gives you an overview of the entire pipeline and you can now see the input, every step of the chain and the end result. You can even click an item to see how it changes throughout the pipeline:

Here I’ve selected the number 2 in the leftmost column. By selecting e.g. the number 4 in the rightmost column you can see how we ended up with that exact number:

If you have a more complex pipeline with a lot of steps, this flat representation might be too much to take in. In order to concentrate on one single step, press «Split mode» in the bottom left corner and choose the operation you want to have a closer look at:

If you, like me, have found it hard to debug a Stream pipeline I highly suggest you try out the Stream debugger in IntelliJ next time you get lost in the mysterious woods of flatmap and friends.

Happy debugging!

Debugging Streams in IntelliJ

How does Digipost contribute to open source to speed up their migration to the public cloud?

What is Digipost?

Digipost is one of two providers of digital mailboxes, a more secure alternative than regular email. This enables central and local government as well as private companies to send secure letters to Norwegian citizens. As a consultant for Bekk I currently help Digipost with migrating their systems to a public cloud – Microsoft Azure.

When migrating to the cloud, there are several known patterns one can use. The simplest is perhaps doing a lift and shift, where VMs and data are copied and the infrastructure is replicated as much as possible (using only IaaS). Even though this approach has benefits such as reduced migration time, we have not chosen this path. At Digipost, we wanted to take advantage of what the cloud providers have to offer in terms of managed services and new approaches to running infrastructure. That meant that we had to adapt our applications to take advantage of new technologies available in Azure, such as Blob Storage and Key Vault and make the application behave well under Kubernetes. This effort requires broad support in the API surface available to us in Terraform providers.

Terraform for Infrastructure as code (IaC)

At our current infrastructure/hosting partner, we use Terraform to manage our resources. This tool has served us well and we wanted to use it with Azure, for everything related to infrastructure. Luckily for us, two Terraform providers exists to help us out: azuread (Azure Active Directory (AAD)) and azurerm (everything else).

As we began testing out these providers, we soon came across some limitations and pains. These limitations were roadblocks for our migration process. We needed to figure out what to do about these issues. The first option is to do nothing, hoping that the issues will fix themselves. The second option is to do some local hacks from Terraform with shell scripts, the Azure CLI or PowerShell. Both of these options are valid in some contexts, but for our most pressing issues we decided that the best option is to directly contribute to the relevant providers. This helps us avoid fragile hacks, spaghetti code, save time and hopefully help other Terraform users on Azure.

The following pain points related to Azure AD were fixed upstream:



The first thing we stumbled upon was a limitation with the password length restricted to max 16 chars. This was not acceptable security-wise for us and a quick fix after Microsoft added API support.



For avoiding single users with wide permissions, we helped contribute support for creating group memberships and mass-assignment of users to groups, together with @tiwood: #63, #100



For licensing/legal reasons, we added a simple extra usage_location field for AAD users

Having these issues sorted out we got our IAM in order. The next step was to address some of the limitations we stumbled upon in the azurerm provider:



Enable auto-growing of disk storage for Azure-managed PostgreSQL



Some of our partners required that we come from a whitelisted IP (or range), so we needed support for outbound IPs when using an Azure standard Load Balancer. We started an implementation in #4400 (which was closed for technical reasons) and we eagerly await the merging of #4472

In addition to these pull requests, we've opened some issues in both of these providers and in other repositories related to our migration effort.

Summary

With the help of the community and two very helpful maintainers (@katbyte, @tombuildsstuff) we feel welcome as contributors to the Azure Terraform providers. Our contributions at Digipost help us migrate faster to Azure and helps the community in the process.

Accelerating cloud migration by contributing to Terraform providers

We are generating astounding amounts of data. Estimates show that the amount of data worldwide will increase by 60% per year, reaching 175 zettabytes by 2025, up from 33 in 2018. Roughly half of it will be stored in public clouds and the rest in other types of data centers.

The digital economy requires a massive infrastructure

Today's digital economy comprises of services such as Uber, Office365 and Netflix built on top of the data we store in the cloud. But what is the cloud? Storing, processing and transporting information across all digital services we employ requires a massive infrastructure. The cloud is nothing more than a vast amount of data centers scattered around the world covering large masses of land which in sum operate millions of servers, processing data on behalf of those services, in terms serving billions of users.

Aerial view of a large scale data center in San Diego (Source: Wordpress)

The need to store and process this exponentially increasing amount of data has led to a golden period of development in the data center industry. The number of hyperscale datacenters in the world powering the cloud has experienced a 100% increase the last 5 years alone. Today there are 8 million datacenters and 500 hyperscale datacenters in the world, but the total amount of hyperscale data centers needed to manage the expected amounts of data in the years to come is three times today's amount.

Why is that a problem?

There are 5 billion Google searches performed each day, each resulting in 6-8 servers to be activated. In terms of greenhouse gases, each search is equivalent to about 0.2 grams of CO2. Despacito going viral, reaching 6.3 billion views, burned alone as much energy as 40 000 U.S. homes.

Add on top of this all the other services powered by the cloud such as internet banking, ticketing services, Netflix, Spotify, Uber and Office365, you'll soon realize that data centers powering the cloud have become as mission critical and essential as water and electricity.

We depend on it at work to be able to send and receive email, share and collaborate on documents, code, schematics and files. We use it to buy shoes and other merchandise online, and we communicate with our colleagues, friends and family both near and far via services that run on the cloud. We've become dependent on the cloud to be able to perform tasks both at work and at home. When cloud services go down, the world halts. This begs for a number of questions with regards to environmental impact.

What is the problem?

In addition to the actual housing of the data centers, you need to fill each data center with lots and lots of hardware, hundreds of miles of cable, refrigerant and water for cooling, batteries and diesel generators to mitigate power outages, and last but not least - an enormous amount of energy to operate the hardware. I'll describe some of the issues.

Hardware

It is hard to imagine how much data 175 ZB represents, but if you were able to store that onto BluRay discs, then you’d have a stack of discs that can get you to the moon 23 times. To store it on hard drives, you would need 11 billion drives(!). Imagine the resources necessary to manufacture only the disk drives, and you will begin to realize that data centers consume a lot of resources.

Since data centers require so much hardware, it is natural to assess the lifecycle impact of the hardware used in the data centers. For data center operations, hardware needs to be manufactured, shipped, installed, operated and decommissioned. Each of these processes requires resources. The manufacturing process for computer electronics for instance requires the mining and extraction of both regular and rare earth metals such as neodymium, which is used in traditional hard drives, and terbium which is used in solid state electronics. Although rare earth metals is a somewhat misleading term, the extraction and processing of ore to produce concentrates usually involves usage of lots of heavy machinery and chemicals and leaves significant impacts on the environment via the open pits the mines imprint on the geography.

Hardware also usually has a fairly limited life span. Usually hardware is replaced every 3-5 years to keep up with performance requirements and reliability concerns for operations. Decomissioning hardware might entail both hazardous materials such as refrigerant used in cooling systems, electronic waste and recycling electronics. Especially recycling electronics will have a big impact on the life cycle assessment of the hardware.

Real estate

Data centers require a lot of space. Hyperscale data centers require hundreds of acres of land in order to accomodate the hardware. One of the issues are that cloud providers usually want data centers to be as close as possible to most end users, in order to provide best performance and lowest latency. This has led to a competition for real estate between data centers and humans, and in Amsterdam, which houses 30% of all hyperscale data centers in Europe, it led to a temporary ban on building new ones.

Energy

Servers in data centers are on 24/7/365. As each data center can have hundreds of thousands of servers, they naturally consume vast amounts of energy. According to estimates from the International Energy Agency, data centers in the world account for 200 TWh/year energy consumption. This is 1% of the total worldwide energy consumption.

1% might not sound like a big deal, but that is the equivalent of the total energy consumption of Indonesia, a country of 250 million citizens and the 4th most populated country in the world. What is more important, the source of this energy might come from fossil fuels such as coal. If all this energy were produced by coal plants, it would result in an annual emission of 1.2 billion metrics tonnes of CO2. In comparison, the total aviation industry emits roughly 0.9 billion metric tonnes of CO2.

Although many cloud providers have pledged to decarbonize their data centers, none have ditched fossil fuels entirely, and most of them rely on renewable energy credits rather than directly utilizing renewable energy sources such as solar or wind power. Greenpeace has been following up on the cloud providers pledges, and there are big differences, as laid out in this recent Wired article.

Some alarmist predictions indicate that due to the increasing number of data centers in the world, the total energy consumption of data centers could rise to as much as 8% of total world energy consumption by 2030.

That's why environmental impact of the cloud first and foremost is dictated by the amount and source of its energy.

Is there some light at the end of the tunnel?

Maybe. In order to answer that, we need to look at what has been done until now.

Increased energy efficiency on component level

Although various sources report an increased energy consumption due to the increased number of data centers, the International Energy Agency refutes this, claiming the total energy consumption of data center operations worldwide will be stable for at least the next three years despite a projected 80% increase in data center traffic and 50% increase in data center workloads.

This requires some explanation. Enter Moore's Law. For the last 50 years, driven by manufacturing process improvements, we've been able to reduce the size of transistors at an exponentially steady state each 18 months. In turn, that has resulted in an exponentially larger number of transistors in each new processor generation, substantially improving computing power. But the scaling has also reduced power consumption with the same rate. So, even though we've seen a rapid increase in number of data centers, the technology improvement has been reducing power consumption by an equal amount.

However, these efforts have lately come to a halt. We are getting close to physical boundaries with regards to how small each transistor can be manufactured. The transistors today measure 7nm. In comparison, human DNA is 2.5nm. The processor industry has solved this by increasing the number of cores in each processor, and improving computing power by parallelization. Today, you can find processors with up to 64 cores, and GPUs with hundreds of cores, enabling parallellization of specific tasks. But not all computations are parallelizable, and this too will hit a rooftop in the future.

A third technology improvement has been dynamic scaling based on demand. Since servers must always be on, being able to automatically scale down processor speed can have a tremendous impact on power consumption when the need for computing is low. Most processors and servers nowadays have the ability to idle or throttle down when not in use, consuming only a fraction of energy.

Lastly, replacing older hard drives with SSD drives reduces the energy consumption by half.

Increased energy efficiency on data center level

Energy is the single largest expense for a data center operation, this is in particular true for hyperscale operators of public clouds. These companies have invested heavily in improving their infrastructure in orde to reduce power bills. A standard measure used in industry is power usage effectiveness (PUE) of a data center - the ratio of total power required to run an entire facility versus the direct power involved in compute and storage. While smaller data centers are still being measured with PUE values greater than 2, large hyperscale cloud data centers have over the past 10 years decreased this value, beginning to record PUE value of 1.1 or less, which is very close to the theoretically perfect PUE of 1.0.

What will the future bring?

In the past decade, manufacturing improvement and targeted efforts towards improving energy efficiency on both component level and data center level, via reductions in PUE, has been ensuring us an efficient offset in total energy consumption in data centers, even due to strong growth of both hyperscale and regular data centers.

No more low-hanging fruits

However, now that we are starting to hit some physical and theoretical limitations, these low-hanging fruits are gone. The shift away from small, inefficient data centers towards much larger cloud and hyperscale data centers seem sevident. The Lawrence Berkeley National Laboratory estimated that if 80 percent of servers in the U.S. were moved over to optimized hyperscale facilities, this would result in a 25 percent drop in their energy usage. A prediction by the IAE is that this trend is already on its way, as illustrated in the chart below.

IEA, "Global data centre energy demand by data centre type", IEA, Paris https://www.iea.org/data-and-statistics/charts/global-data-centre-energy-demand-by-data-centre-type

Continued efforts on improving energy efficiency

Meanwhile, these hyperscale operators continue to innovate. Google for instance entered into a collaboration with DeepMind to improve data center cooling via Machine Learning and just recently launched a fully automated solution for their data centers, rendering a PUE of 1.06 on certain facilities.

A typical day of PUE (power usage effectiveness) with ML turned on and off. Source: DeepMind

The cloud vendors also continue to improve on runtimes, virtualization, compression and software that runs our workloads on top of their hardware improving the overall computation density. For instance, Google recently launched a new task scheduler which assigns resources dynamically, hence increasing hardware utilization in massive-multiparallell environments. Microsoft has done substantial work to improve performance in their .NET Core libraries for the same reasons.

Why is this important and what can you do?

Resistance to both the cloud and the Borg is futile. Limiting viral videos, Google searches and users from using online services is obviously to no purpose.

However, for us in the industry of making such services, there lies a responsibility to inform and acquaint our leaders, customers and decision makers about environmental impacts of their decisions and where our workloads run.

As a part of this advent calendar, our CTO wrote an article about private PaaS being considered harmful. He was mainly arguing the benefits of public clouds versus private clouds and data centers. I hope this article has contributed to illuminate a new perspective. Not only are private clouds considered harmful from an innovation perspective, but from an environmental aspect as well, where hyperscale clouds continue to innovate not only on the breath of services, but also on energy efficiency on a scale.

However, there lies a responsibility on all of us developers as well to utilize libraries, coding techniques and compression algorithms which consume less storage and less energy. Mobile developers are well aware of the power restrictions due to batteries. It's time the rest of us follows and give their contribution to lowering data storage and processing for our workloads. The benefit? Reducing storage, memory and CPU for your workloads has an economic benefit - you pay less money to the cloud vendor. Win-win.

How clouds impact environment

Når vi utvikler digitale løsninger så tar vi gjerne mange antagelser om hva som vil løse problemet vi jobber med. Vi ser et problem, vi kommer på en idé om hvordan vi kan løse det, vi lager det og setter det ut i prod.

Og mange ganger: så er vi fornøyde med det. Vi har jo levert en løsning som vi har tro på ut til brukerne, og det ligger mange andre problemer tilgjengelig som en kan ta tak i nå. Men har vi egentlig løst problemet? Eller har vi bare antatt at det vi laget faktisk løste problemet?

Vi har tatt en antagelse. Og antagelser er skumle - rett og slett fordi de kan være feil!

Jeg jobber som interaksjonsdesigner i et tverrfaglig team som utvikler digitale løsninger, og det siste året har vi jobbet hypotesedrevet. I dette innlegget skal jeg fortelle litt om hvordan vi har brukt testkort som utgangspunkt for gode diskusjoner og litt om hvor vanskelig det er å leve med at vi faktisk har antagelser.

Slik bruker vi «testkort»

For å bli mer bevisst våre antagelser så har vi jobbet med hypoteser gjennom såkalte «testkort», hentet fra Osterwalders Test card.

«En hypotese er en gjetning, antagelse eller forklaring som synes rimelig ut fra foreliggende kunnskap» snl.no

Målet med å få opp en hypotese er å kjøre et eksperiment der man prøver å bekrefte eller avkrefte hypotesen. For så å lære om hva som faktisk er riktig.

Vi har typisk jobbet med hypotesene i arbeidsmøter der alle fagfelt er med: design, utvikling og produkteier. Da har vi diskutert hvilke antagelser vi har om løsningen, hva vi vil finne ut av og hvordan vi kan finne ut av det.

Testkortet

Formuleringen vi har brukt:

Vi tror at … [fyll inn det dere tenker staå er nå, hva er problemet dere ønsker å løse?]



Så hvis vi … [fyll inn det dere tenker er løsningen, hva skal dere «tilby»?]



Så vil vi se at … [fyll inn det dere tenker brukeren vil oppleve som forbedret? Hva er utfallet dere ønsker skal skje?]



Og dette kan vi vite ved å måle ... [hvordan skal dere teste dette? Hvordan skal eksperimentet være?]

Selve hypotesen er de første tre delene:

Problemet vi ønsker å løse. Hva er «ståa nå»? Noen ganger er dette noe en antar, og noe ganger er dette noe en faktisk vet er et problem. Noen ganger har vi nok innsikt på problemet og da har vi bare strøket over og sagt «Vi vet at – det og det – er et problem for ...»

Løsningen: hva tenker vi å gjøre for å løse problemet. Kanskje er vi enige om hvordan vi tror vi bør løse det, kanskje trenger vi å jobbe lenge med å finne en løsning vi har tro på. Kanskje det skisseres mye. Kanskje det kjøres store workshop for å generere nye innovative løsninger. På lik linje med at problemet kan være et stort og fluffy problem eller et lite konkret problem, så kan også løsningen noen ganger være innovativ og kjempespennende, eller veldig konkret og lite tidkrevende.

Utfall: Hva vil løsningen føre til. Her kunne vi kanskje endret teksten til «Så tror vi at vi vil se ...». Det er gjerne her den største usikkerheten ligger. Det er dette utfallet vi prøver å teste og lære om er riktig eller ikke. I diskusjon med teamet så er det veldig tydelig at hvis vi ikke har et godt svar her, så må vi kanskje revurdere hele løsningen.

Eksperimentet:



Måling: hvordan kan vi se at utfallet vi tror vil skje, faktisk skjer? Her tenker jeg på ordet «måle» i en bred forståelse: det kan være både kvalitative og kvantitative målinger. Her må en designe «et eksperiment» for å avkrefte eller bekrefte hypotesen, og det kan gjøres på så utrolig mange måter! Og det er dette som er det virkelig vanskelige. Kanskje er det å sette en liten løsning ut i prod, og kanskje er det å lese forskningsartikler eller gjøre en større test på folk?

Et eksempel på et utfylt testkort for en A/B-test

Og del 2: læringskortet

Læringskortet

Her oppsummeres det dere trodde (altså selve hypotesen) og hva dere lærte. Ble hypotesen bekreftet eller avkreftet? Og hvorfor?

Noen erfaringer og refleksjoner

Testkortene henger oppe på veggen og minner meg på i hverdagen hvor viktig det er å vite og ikke bare anta.

De gode diskusjonene

Er ikke disse testkortene et veldig strengt format? Nei. Dette er ikke en mal som skal følges til punkt og prikke. Jeg tror ikke det har noen verdi å bare fylle ut testkortet. Det bør være et grunnlag for å styre teamets diskusjoner. Hvis testkortet blir brukt som diskusjonsgrunnlag så har det hjulpet meg i arbeidsmøter med teamet til å fokusere på hvorfor lager vi det vi lager. Vi skal ikke snakke om «vi skal lage dette eller dette». Men at vi istedenfor ordlegger oss i formen «Vi tror at vi bør lage dette fordi det og det blir bedre».

Få med alle fagfelt

Å forstå og skrive hypoteser er ikke vanskelig, og derfor noe alle i teamet kan samle seg rundt. I et autonomt team skal alle, både produkteier, designer og utvikler, ta ansvar for å vurdere om vi jobber med det som er riktig. Disse testkortene har hjulpet i arbeidsmøter til å styre diskusjonene inn på det som er viktig å snakke om. Det er så mye mindre viktig å snakke om hva jeg tror og hva du tror, i motsetning til å snakke om hvordan vi faktisk kan finne ut og vite.

Læringskortet gjør at vi tar oss tid til å reflektere over læringen

Å sette av tid og få en rutine for å samle teamet til å analysere og snakke om hva vi FAKTISK lærte, har vært veldig spennende. Det skaper en felles forståelse i teamet og tydeliggjør hva veien videre blir. Å ta seg tid til læringskortet kan være skikkelig vanskelig, men jeg har erfart igjen og igjen at dette er veldig nyttig. Læringskortet er også fint for å enkelt dele lærdom med andre, for eksempel til ledelse eller andre team.

Tips om testkortet

Gi deg selv en påminnelse om når du skal sjekke hypotesen. Har du for eksempel lansert noe til prod og satt på målinger, så er gjerne oppgaven satt til «done» i Jira/Trello, og da er det så lett å glemme å se på den igjen. Mitt triks ble da å lage en påminnelse i kalenderen som sier ifra hvilken hypotese jeg skal se på etter for eksempel to-tre uker.

Å ikke vite er ubehagelig

Å endre mentaliteten rundt at vi ikke vet, men at vi tror, kan være vanskelig! Fordi plutselig så blir en mer bevisst på at en lever i en stor usikkerhet. Da prøver jeg å minne meg på at det er helt ok, fordi vi jobber jo med ting som skal lages i fremtiden og det er ikke noen andre som heller vet hva som er riktig. Husk på at det er helt ok å ikke vite, men da må en i hvert fall bruke tid og arbeidskraft på å utforske og lære! Og det har jeg erfart at hypoteser kan hjelpe å sette fokus på.

Å være bevisst sine antagelser

The Code Book

If you want a history lesson about the fascinating story of cryptography, look no further, the book by Simon Singh from 2000 has you covered. From ancient codes to modern cryptography, Singh gives an insightful, yet page turning tale of politics, war and bright minds. Throughout the book the reader is taken through both the history and some of the math behind this ever-challenging topic. Singh clearly has a good understanding of both the theory and the application of codes. The book even has a young readers version, maybe this could be a gift for an aspiring code breaker. https://simonsingh.net/books/the-code-book/

Liars and Outliers: Enabling the Trust that Society Needs to Thrive

Bruce Schneier describes himself as working in the intersection between technology, security and people. In the book Liars and Outliers Schneier discusses how technology and trust affects our society and how our society has affected technology. At the core of this book, is trust and how we can establish trust in various relations. Trust is essential both in secure communication, but also a key component is how we conduct business and It has been praised for its appeal outside the tech-scene and is still relevant, even after its 2011 release.

https://www.schneier.com/books/liars_and_outliers/

GCHQ Puzzle Book I and II

The GCHQ Puzzle Book is published in corporation with the GCHQ, the British intelligence service. The book features puzzles that range from solvable to "how can this possibly have an answer". Luckily, there are some introductory problems that help you get used to their style of questions.

There is no hiding that these puzzles are hard, but they are a great way to test your wit against some of the best code breakers in the world.

https://www.penguin.co.uk/books/300160/the-gchq-puzzle-book/9780718185541.html and https://www.penguin.co.uk/books/311109/the-gchq-puzzle-book-ii/9780241365434.html

Hackers: Heroes of the Computer Revolution

We cannot believe that it is 25 years since this book was released, many of us were hardly born yet. Steven Levy tells the story of the early hackers, mostly around MIT and Silicon Valley milieu. From the hobbyist that funded computer clubs, via hardware hackers with their own ethics to game developers and the start of the home computer revolution. The book is filled with facts and interviews with the people that actually where there during the infancy of our business. This updated edition contains new afterword with new interviews.

https://www.stevenlevy.com/index.php/books/hackers

The Geek Atlas: 128 Places Where Science and Technology Come Alive

If you still have not made plan on where to go next year, we might suggest you have a look at this book. The Geek Atlas covers famous, and not so famous destinations, all related to technology. John Graham-Cumming has selected a diverse range of locations where “science happened”. Each place is described and includes interesting facts about the sites significance in history. On the top of our list is Bletchley Park, where Alan Turing worked as a code breaker in the Government Code and Cipher School (GC & CS).

https://jgc.org

Happy reading...

Stuff we read - sunday reading

Did you know that you might be required by law to create accessible websites?

In many countries, government agencies has been required to create websites that comply with WCAG 2.0 for a long time, and in for instance Norway, this requirement also apply for private companies.

For a general introduction to accessibility, I recommend reading the article from day 6 in our UX Christmas calendar, and to see all the technical tidbits involved, I recommend going through the checklist found at https://www.wuhcag.com/wcag-checklist/.

As this is a CSS Christmas calendar, let’s focus on the stuff we control using CSS, and today, it is all about keeping focus!

:focus

How many of you have done this before, I know I have:

Great, we just made our site useless for those who really need and rely on this feature, and at the same time made our site less usable for the rest of our users. And, if we do not offer an alternative way of showing if an element is in focus, a fine might be one its way, since we are not complying with Success Criterion 2.4.7 Focus Visible!

Ok, but I didn’t actually add that line of CSS you say, it is part of my CSS Reset you say. Ok, but then it might be time to update your css reset, as the most used CSS Reset from Eric Meyer removed this css in 2011...

Controlling the focus

I highly recommend reading Focusing on focus styles over at CSS tricks for detailed information about everything focus. Different designs require different considerations when choosing how you are showing focus, but whatever you end up choosing, remember to take into account all the other WCAG Success Criterion to make sure you do not break compliance with more rules.

Don’t forget to focus on your fancy designs

We love fancy design, or at least our users and designers love fancy design, and especially custom radio, checkboxes and dropdowns. As developers though, we especially hate styling these three form elements, due to the fact that they are not customisable out of the box. Cue hacks!

Aditya Bhandari goes through all the steps necessary for creating a custom accessible checkbox, and the same technique applies for radio buttons and dropdowns. To give you a hint; ::before and ::after is your friend :)

Not all custom checkboxes are created equal though, and it is depressing how little care is given to accessibility when browsing for custom css forms. When searching for “custom checkbox” on codepen.io, almost all top examples did not have proper accessibility.

And it really isn't that hard! To make sure you custom css forms are accessible:

Do not remove the initial input element from the DOM.

You might be tempted to apply input[type="checkbox"] { display: none; } to your checkbox. You just lost screen readers, and the possibility of controlling the focus state. The better way is to use opacity: 0; and move the original checkbox out of the way so that it does not take up any space.

Style your ::before or ::after based on the focus state

The work is almost done, it is just a matter of adding some focus style:

Thanks for reading, hope it was useful, and hope you have time to verify that your focus is correctly handled on your website :)

CSS Accessibility - keeping focus

Vi diskuterer spørsmålet om autonomi i produktutviklingsteam har gått for langt. Og hvordan krysser du egentlig en elv?

15-Dec

Today's posts