Such a simple question. But does it have an easy answer? Opening up your code base can seem a bit scary. Everyone can see your (bad) code and commit messages, and it's easier for evil people to find weaknesses or make fake copies of your product. But there are many good reasons for open-sourcing a project. Let's go through some of them.

You seek contributions from the community

This is maybe the most obvious reason. If your code is open sourced, outsiders can contribute to the development of the project. This is a strong argument for libraries, frameworks and tools where the target user group is developers. But this goes also for other kinds of products. If you want users of the product to be able to find bugs, and even correct them, that's possible when the source code is open. And if you're good at publishing issues, project plans, you might get help in driving the development forward.

Transparency

One reason for open sourcing your code is to achieve transparency. You might want – or are required to – give people the possibility to read the code. This is a common argument for public companies, where the tax payers should be able to have insight into what is made, and therefore, what they pay for. NAV, Digipost and Entur are examples of Norwegian public companies that have taken this principle to heart.

Showcase and teaching

If you are a data or API provider or creator of a framework or library, you can create open source projects that work as a showcase for your main product. This shows both what is possible to do with your APIs/data/library and developers can see how to do it.

Norgeskart is an interactive map of Norway created by Kartverket (The Norwegian Mapping Authority). It's on GitHub. I asked Kartverket why they open sourced this project, and Senior Engineer Håvard Vidme at Kartverket could tell me: "Norgeskart is our so-called 'showcase' where we show some of what we have. If someone wants to create their own map client with or data, that's great, and we happily display what we have done." So Norgeskart is not Kartverket's main product – their APIs and services are. Norgeskart is more a side-project that encourages the use of the APIs and services.

Another example is Entur Tavla. It serves as an example of how one can create cool things from Entur's data, and teaches developers how their SDK can be used to accomplish this.

It's your business model

Open source does not mean free, as you might have read this December 18th. Highcharts is a good example of this. Highcharts is a chart library, and it's open source. Their licensing model makes it free to use non-commercially, which has ensured wide and fast adaptation. But it's not free to use commercially. This has made Highsoft, the company behind it, one of Norway's most profitable companies.

Security

Remember the first paragraph in this article? Since evil persons can read the whole code base, this sounds scary. But the thing is, there are many good people out there as well. When a project is open source, there is a greater possibility that vulnerabilities are found and fixed, depending on the size of its community.

A lot of automated processes and tools can also help in finding vulnerabilities in open source software, like Snyk and npm audit. Also, GitHub Security is a very interesting new initiative.

And the answer is...

Does our simple question have a simple answer? No. There are many possible reasons to open-source your project, and they depend on the characteristics of the project, and the principles and philosophy of your team and company. We haven't talked about any of the reasons not to open-source your projects, but hopefully this article has helped enlighten some of the benefits.

Why did you open source your project? I'd like to know! Email me at mats.byrkjeland@bekk.no.

Why should you open source your product?

We all know and appreciate Java Generics. It enables us to say "oh, this is a List of Strings", "oh, this is a Comparator for Integers", and the compiler will forbid you to add numbers to your list, or sorting your List of Strings by comparing Integers. Because it does not make sense. Before Generics was introduced in Java 5.0 in 2004, nothing was stopping you from expressing these nonsensical actions, and things would blow up when running your program, instead of being told already when writing the code that there is no way this will work.

When we are introduced to Java Generics, parameterized types, or what you choose to call it, it is usually in the context of container classes. The most well-known containers in Java are probably the ones offered by the Collections Framework. Parameterized types is an intuitive fit for managing lists, sets, maps, and other generic collections in a safe way. The alternative before Generics was introduced, was scattering collections around which you can add anything to, and must remember which type you must manually cast to when retrieving elements. Generic collections is an easy concept to grasp and use. It is almost as if Java Generics were introduced solely for collection classes.

But what are those extends and super words for? They always introduce problems. They seem to be related to why we can't pass a List<Car> to a method accepting a Collection<Vehicle>, even though a Listis a Collection and a Caris a Vehicle. We can fix this by having the method accept a Collection<? extends Vehicle> instead. Ok, so we can settle with the fact that we can assign the List to its super type of Collection, but this does not work with those types inside the < and >. In some cases, and for some reason, we have to be so darn verbose by saying "oh, this is a Collection of something which is a Vehicle of some particular type".

And that super word. That's just weird.

The extends wildcard

The extends keyword used with a type parameter is probably the most familiar wildcard type annotation in Java. With collections in Java they may still be confusing at first. For instance, why is this not possible:

It is a list of anything which extends the Vehicle class and Car extends Vehicle, so why can I not add a Car? Is it bug in the type system?

It is actually quite easy to get an understanding to why being able to add a Car to this list would be a hole in the type system. Consider adding some more context to the code above:

The list of boats, which is passed as a list of some particular type of vehicles, can not have cars added to it, and the type system prevents us from having such an error in our program. We can not trick the type system by assigning the List<Boat> to the lesser specific type List<? extends Vehicle>, which is what we do by passing it as an argument to the method.

The following code example should make this very evident:

There is only one list being instantiated, and it is an ArrayList<Boat>. The list instance is then (correctly) assigned as a List<? extends Vehicle>, but that does not change what the list originally was instantiated as, and that integrity is maintained by the type system, which refuses to add a car to the list of boats.

If the list was indeed instantiated as an ArrayList<Vehicle>, then it would be possible to add both cars and boats to it. And that is also why it is not allowed to assign a List<Boat> to a List<Vehicle>, as they exhibit different behaviour in what type of elements you may add to them.

You are allowed to assign a List<Vehicle> as a, also in this case lesser specific, type of List<? extends Vehicle>, and you will lose the ability to add anything to the list through the new reference:

In summary

In the previous examples, List<? extends Vehicle> is a parameterized type, and ? extends Vehicle is a wildcard type.

Types describe properties of instances passed around in our program, and the type system must always be able to guarantee the integrity of what an instance was originally created as. When an instance is passed as a compatible less specific type than what it was originally declared, we lose some options for what we may do with this instance. As in the examples with the lists of vehicles, cars and boats, it makes sense that when we don't know exactly of which specific type a list was originally declared as, the type system can not allow us to add anything to it, because it may be list of boats, or a list of cars. It may also be a list of vehicles, which can hold both cars and boats, but we don't know as long as we have a List<? extends Vehicle>.

But how does the compiler know that the add(..) method is not safe? Having a List<? extends Vehicle> or List<Vehicle> does not stop us from retrieving the vehicle instances contained in the list.

Generics with extends — in general

The compiler and type system does not know anything about the semantics of lists and what it means to add elements to them. The compiler knows about a certain property of the signature of the add(..)-method: it accepts an argument of the same type as the declared type parameter for the List: add(T element). We can not pass arguments to the add method when the type states that T may have been declared as either Boat or Car. It may even be declared as Vehicle, making the list capable to hold both boats and cars, but the wildcard type has lost this information.

💡 If we do not know exactly which type T is, all methods accepting arguments of this type are effectively useless.

The extends wildcard have a distinct relation to method arguments, in that it renders any method with parameters of the wildcard type as simply not invocable. The compiler will not allow it. Now this seems like a rather annoying restriction. Why would anyone want to disable perfectly usable functionality? But this is what a type system is about, to impose restrictions on what you may express in code in order to provide certain guarantees on correctness.

So what do we gain with declaring wildcard types?

💡 The extends wildcard type widens the scope of assignable references for the parameterized type, e.g. what you may pass as a method argument, at the cost of effectively shutting off all methods of the parameterized type accepting the wildcard type as input parameter.

So for collections, methods retrieving elements from a given collection, but does not mutate it by adding new elements, will typically benefit from accepting a more flexible parameterized type declared with the extends wildcard. Using our cars and boats, and say they need to be registered in some vehicle registry, the following code skeleton should demonstrate how to be able to accept collections of different types of vehicles:

The register(..)-method is able to accept collections of cars and boats because of the wildcard type, but are not able to add elements to the given list.

So what about super?

As the extends wildcard puts restrictions on the use of methods having the wildcard type in their parameters, the super keyword puts restrictions on methods returning the wildcard type. Not in a way that you can not invoke such methods, but you can only ever get a value of type Object when invoking methods returning a wildcard type declared with super.

Continuing with our types of vehicles, consider the following example:

We create a list for containing vehicles, but somewhat strangely assign it to a list of something that may be cars or any super-class of Car. This has, what is probably expected, the implication that you can not add a Boat to the list, as it is not a super-class of Car. But it also has the somewhat strange effect that the only thing that is safe to add to a List<? super Car> is specifically a Car. We can not add a Vehicle. Perhaps even more non-intuitive is that if we have a Volvo class which extends Car, we can add a Volvo to a list declared to contain ? super Car!

This is weird, but still valid code. And again, if your experience with Generics is mostly by using them with the Collections Framework, the super keyword will seem very esoteric. Granted, the super keyword is not meant to be used as shown above, and does not yield any advantages.

super-generics — in general

With the more elaborate assessment of the extends wildcard fresh in memory, let's look at the effects of the super wildcard from a general point of view as well.

The super wildcard is all about flexibility when consuming values of specific types, since a wildcard type of ? super T enables the parameterized type to accept any arguments of type T as method arguments. But a type parameter without a wildcard already enables this capability.

💡 The super wildcard type widens the scope of assignable references for the parameterized type, e.g. what you may pass as a method argument, at the cost of effectively making the wildcard type unusable as a return type for all methods of the parameterized type.

So analogous to the extends wildcard, super is also about enabling flexibility when passing references of parameterized types. And in the case of the super wildcard, the parameterized type is used for consuming values of the wildcard type.

Some super awkward collection manipulation

Passing a list of some type to a method, which then mutates the list by adding elements to it as a side effect, is rarely a natural way of solving how to populate a list. But to bring some closure to our use of the Collections Framework as examples for applying wildcard types, consider this code:

The implementation of putExpiredInto(Collection)-method is just a matter of invoking expired.add(..) for applicable cars in the parking lot, and not affected by the Collection being parameterized with a type with or without a super wildcard. But the wildcard adds more flexibility in which types of collections that are accepted as input parameters by the method, but still being type safe. Code using putExpiredInto(Collection) may actually not be interested in having the expired parked cars put into a collection of cars, but a more general kind, say a List<Vehicle>, or even a Set<Object>, and that should not matter to the ParkingLot. What does matter though, is that it is not accepted to add the cars to a collection of for instance boats, and the type system is able to enforce that.

The ? super Car wildcard type enables to express that the method needs to be able to add cars to the given collection, but the caller is free to choose any level of compatible generality of the collection. Without the super wildcard, it would not be possible to pass any of the collections instantiated in the last example. With the wildcard, only the non-sensical operation of adding cars to a collections of boats is prohibited.

Applying the general super powers

The super wildcard has arguably become more applicable with the introduction of facilities for functional programming in the Java language. Functions, and especially passing functions to other functions, creates a more natural motivation for designing methods accepting arguments with flexible means for consuming values. Functions are notorious for consuming values.

So finally leaving the Collections Framework, let us apply the general understanding we now should have on where the two types of wildcard apply to method signatures:

extends adds flexibility to return types, but disables method parameter types

super adds flexibility to method parameter types, but makes return types not useful

So which simple concept can we apply this knowledge to in an actual useful way? The new facilities introduced in Java 8: Functional Interfaces. You are probably already passing lambdas and method references to various methods in APIs of the JDK and third parties which utilizes both types of wildcards. The Function<T, R> interface is a perfect fit for both super and extends, because its two type parameters, T and R, assigns types to respectively a method input parameter type, and a return type.

Say that every vehicle has a name, so we add a method for getting it to Vehicle. Then, with an instance of another ubiquitous type introduced in Java 8, an Optional, we wish to map the Car to a String containing the car's name:

This is simple enough, nothing controversial. But what is the type of Vehicle::getName? Let's assign it to a reference to make it more explicit:

Here we see that the signature of the method reference is Function<Vehicle, String>, so the .map(..)-method is able to accept a Function which is parameterized with a super-type of what is contained in the Optional. The signature of the method is declared like this:

And we recognize the super wildcard being used for the type parameter for what the function declares as it's input type. It is perfectly valid to do that since the contained value of the Optional is passed to the given mapping function, and it is up to the caller of .map() to pass a function of appropriate level of generality. It may be as specific as only accepting Car, but not anything more specific. I.e. a subclass of Car will not be allowed, because the type system can not guarantee the value is anything more specific than a Car.

The effect of the extends wildcard is a bit more subtle. It enables the function to return a more specific type than the resulting type parameter of the returned Optional<U>. This kind of flexibility is important, as the resulting Optional could very likely be used as a return value from a method which signature is already defined by an interface, or for some other reason needs to be assigned to an Optional of a less specific type. And, as already demonstrated, an Optional<A> is never compatible with Optional<B> even though if B is a subclass of A. But the extends wildcard enables the function to for instance return a value of type String, and if the result of the mapping must be of type Optional<CharSequence>, the compiler is inspecting the target type to verify it is parameterized with a super-type of the type returned from the function, making it in fact assignable.

Recalling the signature of Optional.map(..), the compiler verifies that String, when substituted with U such that ? extends String, is indeed a type which extends the target type parameter of CharSequence. This holds, and the compiler can safely allow this type transformation.

Summary

Types are about guarantees of the constructs we are working on in code. For a type system to be sound it must maintain the integrity of values starting out as some specific type, and being passed around and referred to as potentially less specific types by code on various levels of generality. Treating values as less specific parameterized types than what it was originally conceived as, imposes certain restrictions on how the parameter types may be used.

The intention of this article has been to give an understanding and appreciation for how types enables to prove certain properties of our programs, automatically, and how maintaining the integrity of these proofs requires the type system to be very strict about how hierarchies of types are managed, and how working with parameterized types in hierarchies naturally imposes restrictions on what operations are possible to safely perform with generic types. The use of familiar examples instead of introducing theoretical concepts on type theory, should hopefully help the reader being able to begin to internalize more advanced use of Java Generics.

For readers interested in further understanding some of the theory behind types, I can recommend the article on Covariance and contravariance on Wikipedia. In particular, it is interesting to gain an understanding on how type parameterization differ in C# and Java, using respectively declaration-site and use-site variance type annotations. One can also argue that the keywords used in C# for variance, in and out, offer a more intuitive mnemonic for the effect of the variance than the overloaded words extends and super in Java.

Extend your super knowledge on Generics

Er Kolonial.no mer enn bare en app og nettside? Hva skjer på baksiden? Og hva skal til for å unngå brune flekker på bananene? I denne kaffepraten har vi Hilde Johannessen fra Kolonial.no og tidligere Bekker. I tillegg har vi Hilde Marie Flesland Torall, Pia Solheim og Moquan Chen.

Er Kolonial.no en smidig organisasjon?

Helping you pick the top team for Boxing Day!

If you’re a below-average Fantasy Premier League performer and an above-average data science enthusiast, we have at least two things in common – and you’ve found exactly the right article!

Picking the right Fantasy team with data science is no new subject,1 but we thought we’d give it a go and compare two classic prediction models: linear regression and a basic neural network. We’ll train the models on historical data, evaluate their performance, and finally set up our ultimate team for the pinnacle of Premier League – Boxing Day⚽

Both models are based on three steps. First, they are trained to predict expected amount of Fantasy points achieved by each Premier League player in any round, based on a set of input data. Second, the models try to predict the points scored by each player in an out-of-sample round. Third, the simplex algorithm is used to solve the LP problem of constructing a team of 11 players fulfilling the constraints given by the Fantasy rules, maximizing the number of expected points. Still hanging on? Let’s dive in!

1. Training the models

Input data

The models are trained using data provided by GitHub user Vaastav Anand, who publishes all Fantasy results and stats after each gameweek. The input parameteres for each individual player in any given round are:

Position (four variables)

Team (20 variables)

Opponent team (20 variables)

Home/away game (one variable)

Form the last 5 games (Fantasy's own ICT index) (five variables)

This results in 50 input variables in total, with actual Fantasy points achieved in each round as the dependent variable. All data from previous rounds are used in the training phase to predict a given round (i.e., predicting results in round n, the training sample contains data from rounds n-1, n-2, ... , 1). Predicting round 19, data from rounds 1 to 17 was used.2

Designing the models

The linear regression is set up with the assumed weakest team as baseline on the team variable, and the assumed strongest team as baseline on the opponent team variable. An ordinary least squares regression is performed. The neural net uses four layers: input layer (50 neurons), two middle layers (50 and 30 neurons) and finally an output layer (one neuron). All layers use the relu activation function, except the output which uses a linear activation function. There is also added a dropout layer between the two middle layers with a dropout probability of 0.2. The model uses mean squared error as loss function and the Adam optimizer.

2. Predicting points

After the models are fitted, they predict points achieved by all the Premier League players in the out-of-sample round – in our case, round 19. Injured or suspended players are not predicted.

3. Selecting the XI

After predicting points scored by all players, the simplex algorithm is used to construct a team maximizing total expected points. The official Fantasy rules are set as constraints, including the £100m budget (however, as we will see, the models do not necessarily spend all their money). The cost of substitutes are accounted for by leaving at least enough money in the budget to buy the cheapest possible players filling up the 15-man squad, but they are not specifically selected (and consequently, no players are subbed on should any in the first XI not play). The player with the highest amount of expected points is set as captain, and the second highest as vice captain.

Results from earlier rounds

To examine the strength of our models, we let them predict already played rounds and compare their results with the average score of all human Fantasy players.3 The number of points achieved by the models in gameweek 13 to 17 is displayed in the table below:

As we see, the regression is remarkably consistent, beating the average score with at least 10 % each round. The neural network appears slighty more risk-seeking, resulting in highly varying scores ranging from a staggering round 16 score of 89 points to an equivalently disastrous performance in the previous round. However, both models outperform the average player over the course of five rounds. Let's have a look at their bets for Boxing Day!

Gameweek 19 predictions

The Regressed Registas. Cost: £79.5m

 |

The Neural Netters. Cost: £63.2m

As we see, the two teams are fundamentally different. The regression settles on a conventional 3-5-2 formation, with familiar faces such as Vardy, Lundstram and De Bruyne, which were in fact the three most selected players in GW 18. A great performance against Aston Villa in round 17 paired with a supposedly managable home match against Watford supports the addition of Lundstram's teammates Fleck and Stevens. Vardy and Maddison will hope to shorten the Liverpool lead, while Maupay needs to bounce back from a disappointing home performance against Sheffield United on Saturday. A slightly weaker opponent is likely what gives De Bruyne the edge as captain over Mohamed Salah. The price of the team leaves a comfortable £20.5m for filling up the four spots on the sideline.

The neural network has opted for a refreshing 5-2-3 formation, with large emphasis on Sheffield and Villa defenders. Doherty and Neves will face a tough task in Pep Guadiola's men on Thursday, while Christian Benteke is still waiting for his first goal of the 19/20 season. Trusted with the captain's band, Abraham should have plenty of chances to reignite his old scoring form against the most conceding team of the Premier League season so far. Notably, all players but Martinelli have home matches, which could indicate a substantial preference for this by the model. Finally, despite boasting a rather speculative team, the model remains financially sober and leaves plenty of room in the budget for some decent benchwarmers.

Further work

These models have been built to show the potential value of adding machine learning capabilities to solve problems which require consideration of many factors. The models themselves are fairly basic, and there are several ways to improve them – perhaps by augmenting the input data or through some ensamble learning approach. However, a more interesting discussion is whether this is in fact a problem worth solving with machine learning.

Modeling all possible variables influencing a player's performance in a given Premier League round is not feasible. The problem of setting up a Fantasy draft is, as most complex problems of this sort, an area where expert humans are likely to continue outperforming automated efforts. Nevertheless, the value of quantitative methods as decision support is indisputable. Letting machines do the dirty work – crunching big data sets and discovering seemingly indiscernible patterns – and use this insight as guidance for making human decisions, is probably a more rational approach.

Enjoy Boxing Day, and may the second half of the Fantasy Premier League season treat you well!

1See links below

2The GitHub data is usually published 2-3 days after the last game of the round (which was played yesterday, on the 22nd). Further, since Liverpool-West Ham was postponed, the form data would be incomplete.

3We compare results from our models with the average human score for each round. This implicitly relies on the false premise that all human players can pick a brand-new squad (in practice, use a wildcard) every single round, so the machine scores should ideally be slightly devaluated.

Fantasy Premier League with data science

You’ve got this. Your colleague just opened up for the ultimate discussion: “Can you explain this ‘design’ thing to me?” You’ve waited a long time for this moment, biting your tongue every time your whole field is reduced to “making it pretty”. After many years of study and hard work you know what this design thing is now, right? Yet the words escape you, and all those perfect quotes you found online just don’t seem to cut it. You may fall back to describing the value of design, but describing exactly what design is... that’s harder.

There is an ancient Indian proverb about the blind men and the elephant. The men have never seen an elephant before, and when one is placed in front of them each starts experiencing a different part of the elephant…

“It’s a rope!” says one blind man, feeling the slender length of the elephant’s tail. “No, it’s definitely a tree,” says another, arms around one of the legs. “You’re all mistaken,” says yet another near the tusk, “This is definitely a spear.”

And so on and so forth. The proverb is often used to illustrate how good we are at claiming absolute truth based on our own limited experiences, while dismissing the truths of others. I find this idea of multiple truths interesting in the pursuit of defining something so complex as design. In researching for this article, I found a variety of approaches to describing design where the perspective, context, and the goals of the author vary. Many designers have attempted to describe design in a simple elevator pitch, yet these simple descriptions only describe a part of the beast. But consider them together, and a greater picture starts to form.

Here are the four descriptions of design I found most useful:

1. The design curriculum

Perhaps the most straight-forward way of describing design would be the way it was described to the designer – through their education. Though design training varies, there are some common themes:

Project-based learning\ Most design schools have a focus on learning design by doing design. It is a hands-on major where students learn by model projects requiring them frame the challenge, ideate and present designs, then receive and give feedback through “Critiques”.

Methodology\ Techniques for generating and exploring numerous concepts, then methodically narrowing down to the best solution.

Insight, research and co-creation\ Designers are taught that the answer always lies with the user. By learning interview techniques, co-creation, facilitation and mapping techniques, designers grow their empathy for users and train to move past their own assumptions and biases.

Storytelling\ Humans respond best to stories – it is how we naturally process and store information. Designers learn to harness this both in text, user flows and to sell concepts.

Aesthetics\ How can shape, line, color, and type affect how we receive and respond to a visual message? How can one lead the eye through a composition, or create different feelings through visual elements?

Visualization\ With a foundation in aesthetics, designers practice to be able to quickly and efficiently turn complex ideas and problems into easy-to-understand visuals through simple sketches and diagrams.

Prototyping\ Prototyping allows designers to test and iterate on concepts in a fast and cheap way.

Even after formal education, designers are often still measured by these attributes. Want to understand a designer? Understand the little professor-voice in their head still nagging about these things.

2. Not engineering or art, but close.

Zooming way out, another approach to describing design is by contrasting it to related disciplines. In the Netflix series Abstract, designer and renaissance woman Neri Oxman presents a diagram of her team’s process, with an interesting perspective on design:

Neri Oxman's Krebs Cycle of Creativity. Source: MIT Spectrum

Oxman’s work with Bio-architecture is at the intersection of design, science, engineering and art, so for her the blurred boundaries of these disciplines has been especially enlightening. She describes how the fields bleed into and inform one another:

“Science converts information into knowledge. Engineering converts knowledge into utility. Design converts utility into cultural behavior in context. Art takes that cultural behavior and questions our perception of the world.” – Neri Oxman

In her team’s work, design is defined by the input it needs from neighboring disciplines, and the output it gives to others.

Similarly, way back in 1965 designer Bruce Archer mused about a question that still confuses many today: what is the difference between art and design? Though they may seem similar, they are quite fundamentally different. Think about it… an architect preparing plans for a house – that is design. So is a typographer preparing a page for print. But what about a sculptor, shaping a figure out of clay? Is that design? Archer would argue no – an artist creates, they do not design. For an activity to qualify as “design” there are two requirements:

A prior formulation of a prescription or model

The hope or expectation of ultimate embodiment as an artifact

So basically, according to Archer, design is about making plans for something you hope gets made in the future, while art creates the actual thing then and there.

3. The many facets of design

Perhaps one of the greatest challenges in finding one definition of design is the sheer breadth and diversity of design activities, problems being solved, and outputs. At the broadest level, designers could be classified by the medium they work with: physical objects, digital interfaces, visual systems, environments, experiences, etc. While some designers aim to be generalists and deliver all types of design, some specialize. Others find a subset of design they love doing, but can do the rest as well. A multitude of titles have emerged to describe these profiles and the facets of design they work with, with mixed success. This variety can create confusion for their non-design counterparts, especially in trying to find the right designer(s) for their team.

As visual communicators, describing this variety becomes a design problem. Many designers have attempted to describe these different types of designers by visualizing them on a spectrum to provide some clarity. Here are some examples from digital product design, one of many design disciplines:

Jasper Stephenson maps designer titles over a timeline of the project phases they are most involved in. Source: The spectrum of design roles in 2018 | UX Collective

The UX spectrum shows a wide variety of profiles even within UX, depending on their strengths in certain skills. Source: Vitamintalent

4. Design is the questions we ask.

Some would argue that there is one sure-fire way to find the designer on a team – the questions they ask. Your designer is probably the one asking questions like:

Who is our target audience? What are they trying to achieve? How can I get their input?

How do people react to, interact with, or use this? What doesn’t work right now?

What are we trying to achieve? How does this help our users?

What are we trying to communicate? Is there a way to do it faster, better or in a more unique way?

How can I get input from subject-matter experts?

What dependencies (other services, technical systems, laws and regulations, etc.) affect the experience?

Questions are an interesting angle in defining design, as questions are a tangible output of the typical problems design aims to solve – not to mention the core values and philosophies of a designer. Yet they don’t require long training or special tools. They can be asked by anyone, not just the designer. So per this definition, design can still happen without it’s advocate, as long as questions like these are being asked. Interesting.

---

And there you have it – four definitions, each shedding a bit of light on what this “design” thing is. Four not enought? Check out even more below.

Four Definitions of Design

Serverless computing has been with us for some years now, and has been "production ready" for quite a while. Services like AWS Lambda, Google Cloud Functions, and Azure Functions allow us to create highly scalable services with minimal overhead where you only pay for what you actually need.

Azure Durable Functions is an extension of the Azure Functions family that lets you create stateful functions. This is useful for a lot of different scenarios, including the fan-out/fan-in pattern, which we will look into in this blog post.

Fan-out/fan-in

The fan-out/fan-in pattern is useful in scenarios where you have a root process with multiple child processes that can run in parallel. Each of these child processes may also have more child processes. This is the "fan out" part of fan-out/fan-in, and can easily be implemented using regular functions and some kind of queue.

However, if you want to have control of your system, you probably want to keep track of when all processes have completed as well as the result of each child process, in other words fanning back in. Azure Durable Functions lets you do this using Orchestrators, SubOrchestrators, and Activities. In addition, you need some kind of Trigger function to get the fun started.

In the following sections, I will show an example of creating a Lotto ticket generator using Azure Durable Functions. A Lotto ticket consists of 6 numbers. This Lotto ticket generator takes a list of people who should get a Lotto ticket, and stores the Lotto tickets someplace safe. Our (somewhat contrived) example also pretends to use an external number generator service to simulate real world challenges.

The example uses C#, Azure Functions 3, and Durable Functions 2.

Activities

The activity function is where the actual work happens when using Durable Functions. An activity works the same way as a regular Azure Function, except that it is kicked off using an [ActivityTrigger] which can be any kind of object. Note that the activity function only may take one argument, so if you want to pass multiple arguments, you either need to create a complex type, or use tuples. The return type must be a Task containing the return value.

So there it is. This activity takes an int input, calls an external number generator function, and returns the result.

When creating activity functions, it is important to be aware that they should be extremely simple and fast. There are different hosting plans that you may use for your functions, but unless you are using a consumption plan, you aren't really running serverless, and you are missing out on the good parts. This means that your activity duration should never, ever, exceed 5 minutes. (It is possible to increase the maximum duration to 10 minutes, but if you need to do that, you are probably doing too much work in the function.)

Sub-orchestrators

The sub-orchestrator is responsible for kicking off child processes and returning the results to its parent. Sub-orchestrators may be chained to create multiple levels of fan out, but in our example, we'll only use one sub-orchestrator.

The sub-orchestrator has to take an [OrchestrationTrigger], which must be of the type IDurableOrchestrationContext. To get the input to the function, you call context.GetInput<T>(), where T can be any type. The return type, like activities, needs to be a Task containing the result.

In this example, the sub-orchestrator takes a name as input and generates 6 numbers by invoking 6 instances of the CreateNumberActivity function. These numbers are then returned along with with the name of the ticket's owner.

Orchestrators

The orchestrator is the manager of the entire process, and is responsible for starting the child processes, and knowing when they all have completed. In practice, orchestrators are implemented the same way as sub-orchestrators (sub-orchestrators are orchestrators).

In this orchestrator, a list of ticket owners is provided, and the ticket generator sub-orchestrator is kicked off for each owner. When all tickets have been created, the orchestrator stores the tickets (probably using an activity), and the function completes.

When running orchestrators (and sub-orchestrators), the process is paused when you have kicked off activities or other sub-orchestrators, so you don't need to worry about the 5 minute execution time limit. However, each time a child process returns, the entire function is re-executed from the start to rebuild the local state. The state of child activities and sub-orchestrators are stored so that they are not invoked multiple times. But you need to make sure that the internals of the orchestrator are deterministic. In other words, calls to methods like DateTime.Now or Guid.NewGuid() must be avoided.

Triggers

The final part of the puzzle is some way to start everything. Azure Functions contain a lot of different Triggers, which respond to different types of events. In this example, we will use the [HttpTrigger], which simply creates an HTTP endpoint, and responds to requests to that endpoint.

To start a Durable Function, the activity needs to take a [DurableClient] parameter of type IDurableOrchestrationClient. This client is then used to start the Orchestrator function.

Error handling

So, now we have a functioning function, able to scale indefinitely. But what happens when an error occurs, as we all know will happen? After all, computers do have a mind of their own, especially in the cloud. There are a couple of different ways to handle this, as explained in a previous post. But first, we need to discern what kind of errors we can expect.

Transient errors

These are errors that typically occur as a consequence of communication between services. Examples of these are database connectivity issues, unresponsive web services, or other unavailable resources. Typically, these kinds of errors can be handled by trying again later.

Application errors

These are errors that occur because of an error in the application code. For example, the input can be invalid, or cause an invalid state in the function. Retrying the function a thousand times will cause the same error every time.

Retries

Durable functions provides out-of-the box support for retrying failed functions and orchestrators. You simply pass a RetryOptions object when invoking the function, specifying the number of retries and the amount of time to wait before retrying. You may also specify the backoff coefficient for exponential backoff.

However, when using retries, you should be aware of the error types specified above. If an activity throws an application error every time it is invoked, and the sub-orchestrator retries 10 times, you have 10 retries which will fail every time. Furthermore, if an orchestrator also retries the sub-orchestrator 10 times, you suddenly have 100 retries which fail every time. And if the sub-orchestrator calls other activities, those activities will be retried as well.

Exception handling

So, if the error is an application error, you need a way to tell the function to simply accept it as a fact of life. RetryOptions also contains a callback for determining whether exceptions should cause a retry or not. This, combined with a good old try-catch, give you fine-grained control over how different types of errors in your functions should be handled.

This way, the (sub-)orchestrator will not fail even if one of its activities fails, while still using retries for transient errors. And you are able to control how to handle the application error. Of course, you may still pass information about the failed activity to the orchestrator using exceptions or status objects.

Full example

Fan-out/fan-in for high scalability with Durable Functions

Making a small Haskell application

Haskell is a programming language that a lot of people use to learn functional programming and solve programming exercises. But not as many take the leap to use Haskell for practical projects. To try to help to bridge this gap, I will walk through how to create a "practical" Haskell application. This is of course not the only way to do it and there are alternatives to every approach I am using, but this is what I find most suitable for a small easy-to-understand project.

The application we are going to make is pretty simple, it should just output the current Bitcoin rate. Even though it is not a complex project, it will hopefully give an overview of how to use Haskell in other contexts than programming exercises.

To do this, we will take a look at the following aspects

How to create and build a Stack project with added dependencies

How to make a simple HTTP request in Haskell

How to use lenses to extract values inside a nested JSON object

First we need to create and set up our project.

Stack

We will be using Stack, a cross-platform tool for developing Haskell projects.

Installation instructions are available if you don't have stack installed and want to follow along.

New stack project

stack new bitcoin simple-hpack creates a new project for us, named bitcoin, based on a template which uses hpack.

This will create a bare bones project for us.

We can ignore most of these files for now. The source code of our project resides inside src

If we look at our main file src/Main.hs, it is a simple "hello world" application where main just prints the string "hello world".

To build our project we run stack build. If you haven't used stack before, this will download an appropriate GHC version. After this is done, we can execute the project by stack exec bitcoin

As expected, the output is "hello world".

Now, lets add some dependencies.

Dependencies

package.yaml is the configuration file of our project, like package.json is in npm projects.

We see here that our only dependency is base, the Haskell standard library. To fetch data using HTTP and work with JSON data, we need to add the following packages to our dependencies list:



lens and lens-aeson : To easily work with JSON data using lenses



bytestring and text: Different string representations. Needed to work with http requests and JSON



http-conduit : HTTP client

After we have added these dependencies to package.yaml, run stack build to install all the dependencies.

Rapid development

One of my favorite Haskell tools is ghcid. It provides an easy and fast way to compile and run Haskell projects when you save any changes, which helps speed up development time. To install ghcid : run stack install ghcid.

For our project we can use the command ghcid -r , which will compile the project and run the main function when you save any changes.

A decent alternative if you don't want to install ghcid, albeit not as fast, is

Now we are ready to start developing our application. Let's start by making an HTTP request!

Making an HTTP request

The purpose of our application is to output the current Bitcoin rate. The API endpoint with the data we want is https://api.coindesk.com/v1/bpi/currentprice.json, and is provided by Coindesk.

It responds with JSON on this form:

Lets start by fetching JSON from this endpoint in our Haskell program.

Lets walk through the code :

The language extension OverloadedStrings makes literal strings more flexible. In our case it makes so that our URL is automaticly parsed into a Request, which simplifies our HTTP request.

Then we have some imports that allows us to make HTTP requests and work with bytestrings.

fetchJSON fetches the JSON from the URL by making an HTTP request and then returns the response body

in main, we use fetchJSON to get our JSON as a ByteString, and then we print it

This will output :

We have managed to fetch and print JSON. Great!

Now we need to extract the correct values.

Working with JSON

We want to get the Bitcoin rate in USD. In Javascript this would be something like json.bpi.USD.rate.

To do that in Haskell, we will use lenses.

Extracting data from our JSON object using Lenses

Data.Lens.Aeson lets us work with JSON data[1], by providing lenses[2] for a Haskell representation of JSON objects.

Data.Lens.Aeson provides key, which focuses on a property value. So we start by focusing on bpi, by using key "bpi". preview allows us to "get" a value that is possibly missing.

Lenses are a complex subject, and the details of lenses are outside the scope of this short blog post. The basic idea however can be seen as functional getters and setters, and an intro to this was posted earlier in this series, The Lens Pattern in TypeScript. Even though it uses TypeScript, and not Haskell, it can help to understand the motivations behind lenses.

Our use of lenses will probably become clearer with some actual code. So lets try to work our way into the value we want:

The json bytestring is now converted to a Haskell representation of JSON, but if we look really hard, we can see that we have the value for the bpi property.

We now want to go into the USD property value. To do that, we compose key "bpi" with key "USD"[3]:

Then into the rate property value:

We see here that we get String "7,364.1200", but we want "7,364.1200". We can then use _String.

We made it! Now we have successfully extracted the bitcoin rate in USD from the json bytestring. We put this functionality into a function getRate and then the whole program is now:

which outputs

Finishing up the application and error handling

To account for the possibility of the rate not being present, we do a pattern match using a case expression. If we get Nothing, which means that we could not find the Bitcoin rate, we print an error message. If we succesfully get Just rate, we print a string along with the rate.



We need to use putStrLn from Data.Text.IO because rate is of type Text and putStrLn from Prelude only works with String.



<> is used to append the three Text values together.

We can now execute our final program.

It works as expected :)

Which means that we are done.

Conclusion

I hope this walk through gave you some insight into how to you can use Haskell in a somewhat practical context. And perhaps inspire you to create a personal project using Haskell.

Github

The project source code is available at Github.

Footnotes

[1] : For more robust JSON decoding, a better alternative is aeson

[2] : The optics used in this article are not actually lenses, but traversals and prisms

Making a small Haskell application

If you haven't lived under a rock the last couple of years, the term Ransomware isn't something new. It grinds the largest corporations to a complete halt and can take months to recover from. But how does it really work? And how should you protect yourself?

The anatomy of ransomware

As with any malicious software, there are no one size fits all description of ransomware. But many of the more famous, and more devastating ones have generally had these traits:

Exploitation of existing weakness

The first problem for the attacker is of course getting into the target system. This is usually accomplished by exploiting an existing vulnerability. This might be a published and known weakness in the system, or an unknown bug. In the latter case the exploit is known as a zero-day vulnerability, and these are obviously preferred, as any victim will be defenseless. On the other side, they are rare, difficult to find, and expensive to buy, making known vulnerabilities far more common to exploit. The weakness used can vary greatly. It can be a flaw in a service running on a publicly accessible port, making direct entry into the system possible. Or a flaw in an email reader causing malicious code in certain attachments to automatically execute when the message is loaded. Or another application is fitted with a backdoor that is used to deliver the payload of the malware. The only limitations here are probably the imagination (and endurance) of the attacker.

Encrypting files

The thing that distinguishes ransomware from other malware is that it does not intend to destroy its target system permanently, nor is its main intent to steal data. Instead, it seeks to make reversible damage. This is usually done by encrypting parts of the contents of the infected machine. Naively encrypting all files would most likely encrypt some files required by the operating system, causing the system to be unrecoverable. Instead, the ransomware targets files based on type and location. Some search for all files of type .docx, others may target all files contained within the My Documents or User directories. It also searches the network for other machines to infect and attempts to encrypt files on network disks. To be able to efficiently encrypt (and later decrypt) a large number of files, most ransomware uses a hybrid combination of symmetric and asymmetric encryption. The attacker generates an asymmetric key-pair and bundles the public key with the ransomware. When the ransomware needs to encrypt data, it randomly generates a symmetric encryption key, uses this to encrypt all the files, and encrypts the key with the public key of the asymmetric encryption key. The encrypted key can now be sent to the attacker for decryption when payment is made. This method makes sure that the attacker does not have to decrypt all the files themselves, while also making sure that one person paying the ransom cannot share their decryption key with anyone else.

Asking for ransom

Yeah, that's kind of obvious from the name. Ransomware asks for ransom! In exchange for money, the bad actors behind the ransomware promises you to decrypt all your files and hand your systems back to you. The price varies, but in large scale attacks aimed at the general population the price is usually not that high. The WannaCry ransomware asked for between $300 and $600. The goal is to set a sum that is likely to be paid by enough people. If the ransomware on the other hand targets a more specific target, say an international organisation, the ransom could be much higher. Ransomed is usually asked to be paid in some form of crypto currency to make it difficult to trace the money. Gift cards for different web shops are also used, but probably by less experienced attackers, as you would face significant risk buying 200 TVs on Amazon should your ransomware suddenly be very successful. On the other hand, most people have no understanding of how to buy and send crypto currency, so this is a tradeoff for the attacker.

Unlocking your files (if you are lucky)

After paying the attacker, all you can do is wait, hoping that the evil person who encrypted your files to extort you for money will hold up their end of the bargain. This might not be the best prospect. While you could argue that an attacker has the incentive to prove willing to unlock people’s files, there is little stopping them from taking your money without unlocking the files. In addition to this, paying will provide a market for more ransomware. This entails that for the greater good, you should refrain from paying. We will look into alternatives to paying later.

Examples

Let's look at some examples. All these attacks have of course been described in much greater detail before, but this can give you a glimpse into what happens to a company when ransomware attacks.

WannaCry

Perhaps the best publicly known ransomware attack, WannaCry was launched in May 2017, and exploited a vulnerability in the SMB protocol for Windows systems. The exploit had been identified by the NSA, but not reported to Microsoft, leaving it unpatched for a long time. A fix was made available by Microsoft in March 2017, meaning that any well patched system was at the time of the attack protected. The attack followed the standard method as described above but had an interesting feature that was used to limit the attack. Before encrypting the files, the ransomware checked for the existence of randomly generated domain name the authors could be sure was not registered (www.iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com). Including this feature may seem counterintuitive at first but was done for a good reason. When malware is examined in a laboratory setting, all network requests are usually simulated as successful. If a request to a known bad domain was answered, the malware could be certain that it was being prodded by researchers and would stop working to make analysis more difficult. The attack was greatly limited when a security researcher purchased the domain in question just hours after the attack had started. This however did not stop the attack from having massive consequences, especially for the British health services, which had to cancel operations and revert to emergency procedures with pen and paper.

Hydro attack

Norsk Hydro is one of the world’s largest producers of aluminium, as well as other metal products. In March 2019 they were hit by a ransomware called LockerGoga. This was seen as a directed attack against Hydro, as LockerGoga is not usually self-replicating, meaning that its spread has to be intentionally controlled. The malware was sent to Hydro as part of what seemed like a legitimate email from a known customer. The biggest danger for Hydro was the possibility of a shutdown of the metal melting furnaces. This could cause the metal to set inside the furnace, possibly destroying it. Luckily, these operations were able to continue with manual operations. The rest of the company also attempted to revert to manual procedures, though this slowed down production significantly. The Hydro attack was special because it was extremely targeted, but also because it was handled very well by the company. The decision to not pay ransom was made early, as was the decision to publicly state that they were under attack. A full system restore was conducted within about a month, with downtime causing significant economic impact, however information loss appears to have been mitigated by good backups. Because of their openness about the attack, the Norwegian National Security Authority (NSM) was able to stop similar attacks on other companies in Norway, though none of these companies have been willing to come forward.

Maersk

Maersk is an international shipping giant, handling a large portion of the worlds freight. In 2017, they were struck by the NotPetya ransomware. While not very different from the previous attacks, there are two noteworthy points about the attack on Maersk. First, it was not an attack directly aimed at Maersk, but was instead released to spread and cause maximal damage. Maersk was not the only company to be hit, but it did suffer greatly. The second noteworthy thing is that Maersk mapped their entire organisation in their domain controllers. These communicated each other worldwide, making it possible for the ransomware to spread to every controller in the world. The problem was that Maersk was unable to find a backup of the domain controller. They were saved by the most amazing coincidence. The Maersk office in Ghana had lost power before the attack and had not been able to come back online. This meant that their copy of the domain controller was not infected and could be used to restore the network. An employee flew the backup directly to London, as the bandwidth was to slow to effectively transfer it over the internet. This one offline copy of the domain controller saved Maersk from total disaster.

Advice

What lessons should we take from this? I have a few takeaways, though there may be more, and these may be contested.

Do not pay the ransom

This is perhaps the most controversial advice I will give you. The FBI has in several cases publicly encouraged people to just pay the ransom when there is no possibility to recover from backups. But this may not be the best solution. You will be contributing to keeping ransomware a lucrative business for criminals. There is also no guarantee that your files will be unlocked, nor that the perpetrators are actually able to decrypt your files. There have been cases where only parts of the data have been decrypted after payment, and a new ransom has been made for the rest of the data. Paying also makes you a prime target for a new attack, as it is now known that you are willing to pay this kind of ransom. You should consider carefully whether paying is the best option you have.

Keep offline backups

While many earlier ransomwares only encrypted files on the system they infected, most of the more serious attacks have actively sought out backups to ensure that you have no choice but to pay the ransom. You need to ensure that you have valid, up-to-date backups in offline storage at all times. This will enable you to do a full system restore, however clever the ransomware you were hit with was. And while we are on the topic of backups, you should regularly attempt to do a system restore. This not only validates the backup, but also ensures that you know how to restore your system. There is not a lot of value in backups if you are not able to restore from them.

Patch everything always

Most ransomware use some kind of known vulnerability to enter its target systems. You can stay ahead by having an efficient regime for patching all parts of your system. Running services in the cloud will of course limit your exposure, as most cloud providers will patch as soon as patches are released. The WannaCry attack was based on a SMB vulnerability for which a patch had existed for 2 months. This was totally avoidable by simply applying the patch in time, but thousands of systems evidently were not up to date.

Have detection systems in place

Aside from having up to date virus protection, you can also put into place some detection systems specifically aimed at ransomware. Since we know that the ransomware will most likely encrypt your files, we can monitor for file extensions that are associated with encrypted files. Some ransomware even chooses to use their own file extension, making it much easier to spot. You can also monitor the amount of file renames, or general activity on your disks. If this spikes beyond normal rates, it could be a sign of ransomware.

Ransomware, an introduction

The world of front end frameworks is always evolving. For years it was dominated by jQuery and Backbone, but these days we hear most talk about React and Vue. However, there is another framework that hasn't been talked about so much, namely Svelte. Although this framework is actually three years old, it had a rebirth in April this year with the release of its third major version.

Rethinking reactivity

Svelte was created by Rich Harris, a graphics editor in The New York Times. While working with data visualization with loads of nodes and lots of small changes he encountered problems with performance. Frameworks like React and Vue use a virtual DOM on which the framework can perform diffing and calculations to perform effecient updates of the DOM, seemingly in a reactive manner. This mostly works great, until a point where diffing and calculations actually becomes a bottle neck. Rather than trying to improve this technique Rich wanted to change the way we think of reactive changes.

Rich took his inspiration from good old spreadsheets. When you change the value of a cell in a spreadsheet, the program instantly knows to update all cells referencing the changed cell. In order to achieve this Rich didn't just create a framework – he created a compiler. Svelte knows all dependencies between your variables and DOM nodes at build time. This means it can create code which surgically updates specific nodes in your DOM. Hence the virtual DOM is no longer needed. The DOM is actually really fast when you know exactly what needs to be done. And what's more, this will create smaller and faster applications.

Svelte at a glance

Are you intrigued? Good. You should be. Now you know the core idea of what makes Svelte different, but I'm sure you're wondering what it looks like using Svelte. Much like React you will create components and each component has its own file. The basic structure of a Svelte component looks like this:

Because this is a component all code in your <script>- and <style>-tags are scoped. This means that the Javascript in this component won't affect other components, and your styling won't affect nodes from other components. Outside the scripts and styling we have some templated HTML.

What's with let?

Well let me explain. By default Svelte expects data to be mutable. What if we wanted to add an input field to the code above to allow the user to specify his name? With React you would need to create a state and run a set-function every time you wanted to update it. With Svelte you simply bind your mutable value to an input field and voilà:

For simplicity I removed the <style>-tag in this example.

Reactive declarations

One more thing you should know to truly make your Svelte code reactive is how to deal with reactive declarations. Say you want to show the users name in capital letters. Could you just make another let?

If you run this code you would see the text Your name in capital letters: DAVID, but sadly this would not react to changes in the input field. In functional React components all your code runs on every render. In Svelte components your code only runs once.

When the value of the input field changes, there's no need to re-render the component. What happens is very straightforward:

the value of name changes

the text in the h1 node is updated

But Svelte has no way of knowing you also have an expression that depends on name. To solve this, Svelte borrows the Javascript label syntax to introduce the reactive declaration syntax$:.

With this in hand it really is a piece of cake to make reactive declarations:

Is this all?

No, no. That was just the basics. Svelte is actually rich on features. Both in terms of possibilities in the template syntax and helpful features you can import to your Javascript.

What I personally loved the most when I started making something with Svelte was how easy it was to add transitions. Although I am capable of making transitions with CSS, I loved how I could just throw in an element directive, and it just worked:

I could go on for a while about all the built-in functionality of Svelte, but I'd rather you go try it for yourself.

P.S. if you're intrigued and curious, but feel guilty for betraying your current framework of choice, don't worry. Svelte was actually awarded the Prediction Award in this years state of JS, so you're not alone.

New kid on the block

Pseudo classes are used to select elements of a specific state. The state of an element can either be something that is applied to it by the user, such as hover, or it is based on the element's position in the markup, for example it is the first of its type.

We can use pseudo classes to change the styling of some elements based on user interactions. For example, if we want to give the user feedback that an element is clickable, we can use psuedo classes like; ::hover, ::focus or ::checked.

Pseudo classes give you the flexibility to add specific styling to certain elements, while still keeping your markup nice and clean.

The fun begins when we start using pseudo elements. Pseudo elements are elements that will act as if they were added to your markup. With ::before and ::after we can add elements with a specified content.

Let’s start with an example. I want to make an interactive Christmas card! To make it more interesting let's hide a secret message using pseudo elements.

I start of with a nice Christmas color and some boxes where today's date is written. The secret message is hidden in the html! Can you see it?

Now let's hide the message on the page. If we imagine that the date boxes are cubes we can hide the message on the side facing down and turn them around on mouse over. I start off by adding pseudo element to each box using ::after and position them right below the box. Then I insert the secret message using attr().

Now I can fold the pseudo element beneath the box to create the illusion of a cube. I use transformto rotate the pseudo element 90 degrees along the X-axis. The boxes will turn on hover. To get the effect of a turning box I rotate the parent element and change the color of the pseudo element to make it look like shadow. I set the transition time to 0.3 seconds.

I'm not quite happy with the hover effects. The boxes are turning too quickly and I cannot grasp the secret message. I want the boxes to turn on click. In order to achieve this I have to change the boxes into checkboxes. If I check a checkbox, it will rotate.

To style the checkbox to look like the cubes we had before I need to add a label. It is actually the label that will be styled as a box. When we have achieved this we can hide the actual checkbox by setting width: 0. Here’s the result.

Merry christmas everyone!

Pseudo classes? More like pseudo awesome

When you’re developing similar apps for the web and native platforms, it's hard to find a project structure that works well. Let's take a look at how our team is approaching this challenge for maintaining different view layers, while sharing most of the business logic.

The Web and the Native

We are working on the web and native applications for Entur, which provides a journey planner for public transportation in Norway. These applications let the user plan their journey, buy tickets, create a profile, add payment options, and much more. To be able to create all of these features, we're using React for web and React Native for iOS and Android.

We have the (dis)advantage that the web and native applications mostly (but not completely) contain the same functionality. And as you may have guessed, this creates two very similar, yet different, code bases. We wanted to have the same business logic for both applications, but didn't want it to introduce unnecessary complexity to our view layer. We wanted to separate the views from the business logic, but at the same time minimize the duplication of the code. In addition, we wanted to be able to implement features for both platforms at the same time. Therefore, we ended up with a monorepo.

The Monorepo

A monorepo is just a repository containing several projects, and how you structure it will affect the developer experience.

When developing Feature X for the web application, where 95 % of the code will be identical to the native one, where does it make the most sense to put your components? There's no easy answer here, and it's totally independent of the developer. Just check out Mathilde's article from yesterday! However, we decided to divide our code into subfolders that contains the code for both web and native, called domain packages.

The Domain Packages

We create subfolders that contains all the files for web and native and we separate them by the domain we feel they belong to. For instance, we have created a subfolder for everything regarding the profile page, and another folder for things that applies to the travel suggestions (called journey). By doing it this way we're able to easier compare the components and share the business logic across web and native without too much overhead.

In addition to the domain packages we have a client-app and client-web folder which are the entry points for the different applications. Here is a rough snippet of our structure:

Actions, reducers, sagas and helper files are typically files we use to share between the applications. They are put in separate files and folders in the current domain packages, and since both the web and app components are in short distance of each other, locating it is a breeze.

There are both positive and negative sides to working with a monorepo. So far, we like the domain package structure we are going for. At first look it seems a little bit chaotic, but when you get used to the domain of our application it's pretty easy to follow! Of course, it's not easy to know when to create a new package, and when to not, or even when to delete them. For example, in our case the product we are creating is a journey planner, and it is super hard not to put everything in the journey domain package! 😅 And of course, we have that one folder, the utils-folder, where all the stuff we don't know where to put is placed!

When you are done developing a feature for one of the platforms, it's often relatively straight forward to develop it for the other. Since all the logic, design and architecture is taken care of already, most of the work left is to "port" the views from native to web or vice versa.

Of course it isn't always as easy to convert one view from native to web, because there are some corner cases in the business logic, device capabilities, and the styling when it comes to vertical vs. horizontal design, but all in all, it's a pretty pleasurable experience.

The Setup

So, a monorepo helps us with shipping features this way, but how do you configure this?

If deciding on a project structure is not difficult itself, getting the tooling and supported frameworks to work is another uphill battle.

First, importing internal and external modules. Needless to say, we do not want to worry about whether a module is part of the monorepo or from an external repository, like NPM. Luckily, Yarn has a feature tailored towards monorepos called workspaces. It solves the issues of symlinking modules together, de-duplicating dependencies across packages, and keeping it all in a single yarn.lock file. As a result dependency resolution works just as you would expect. Need to access an exported module from another package? Just add import { something } from "@entur/profile". Need to add a dependency to one of your packages, just run yarn add date-fns. The only difference from a traditional JavaScript repository is that you have one package.json for every package, in addition to the root package.json.

Bundling

Secondly, bundle everything. Building these applications requires two different build systems. For web we use Webpack and for React Native - Metro. Both systems implement the same concept, one main entry file with references (directly or indirectly) to the entire application, which is then traced and bundled into a single file. When separate implementations are required for the different applications, a .web.js or .native.js file extension can be added to expose the file to only one of the build systems. Metro has built in support for .native.js, and Webpack can be configured to also accept .web.js.

Third Parties

Thanks to Yarn workspace module resolution works out of the box. However, library developers sometimes requires files to exist in a certain location. When there is no way to override the config, or get a pull-request accepted, the last resort is to patch source files. If you’re new to patching, check out Mats Byrkjeland’s post about patching your node_modules. The advantage of patching is that you can make the change as specific as you need. The disadvantage is now you have to support the patch for every library update. Luckily not all files changes with every release, and for us patching is rare.

Other Possibilities

A monorepo is not the only solution when developing for native and web. When these applications was created, we had absolutely all the business logic on the client side, which affected the choice of having one repository. Over the years we have moved more of our logic over to cloud functions (see the article written by Carl Joachim about Firebase features in Entur), that could have influenced the decision of a monorepo vs. several repositories today.

Another possibility when developing for native and web, is the react-native-web. It provides much of the logic and code of React Native for the web. We tried to use this as a part of our project, but decided to remove it due to the lack of semantics. The div-o-rama that goes along with the native code bleeds over to the web applications. Another example is the repetitive styling you need for React Native components, which is separate styling for each view due to the lack of inheritance. In addition we had rarely a 100 % one-to-one connection between native and web, which in most cases would create separate views.

So, all-in-all, it is hard to give a correct answer of what you should use when developing for web and native applications. However, even with the complexity of the setup we have been through, we are satisfied with our structure and would recommend it to others struggling with the decision of the setup of your projects!

Two Applications, One Repository

The Kotlin library comes with several container types. Two of these are List and Sequence. At the first glance these two look quite similar, but we will look at the differences in this article.

A List implements the Collection interface, which again implements the Iterable interface. A List is an ordered collection of elements, and every element can be accessed by an index.

The listOf() method returns an immutable list, and as a result you can't add or remove objects. If you want to work with a mutable list, you can specify that when creating the list:

Note that although the list is created as a constant with the val keyword, you can still add or remove objects. The reason for that is that write operations modify the object and not the object reference.

A Sequence offers the same functionality as Iterable, but implements another approach to multi-step collection processing.

You can also create a sequence from an Iterable:

On both sequences and lists you can operate upon collection of elements, like you can see in this example:

Now let's look at the difference between sequences and lists. If a sequence operation returns another sequence, it’s an intermediate function. If it doesn’t return a sequence, it’s terminal. Sequences are lazy, so intermediate functions for sequences don’t do any calculations. All the calculations are added to the sequence, and they are not executed until a terminal operation is called. On a list, however, the intermediate function does the calculation and returns a new collection.

If we expand the example above with some print statements, we can see how the program executes:

The output:

We can see that the map() function is executed immediately after filter() for Sweden and Germany when the terminal function toList() is called.

In this next example I want to get the three first numbers bigger than 10 in a list. First I can execute the functions directly on the list:

As you can see on the output, all the filter() functions are executed before we take the three first numbers:

Let's try the same on the sequence:

The output:

Because the intermediate function filter() isn't called before the terminal function toList() is called, the compiler knows that only the first three numbers bigger than 10 are included, and as a result the filter() function is only executed four times.

If you are dealing with big data sets, or doing lots of processing on the data set, it might be smart to consider sequences instead of a normal list.

23-Dec

Today's posts