Tech stack #10YearChallenge

#10YearChallenge has been trending for a while, so I thought it would be fun to do a 10 year challenge for programming and take a look at the technology I used back in 2010.

2010

10 years ago covers my final year in high school, and my first year in university. Both used completely different programming languages and tech stacks, so it’s an interesting place to look back at.

I was running Windows on my personal machine, but the computers in the engineering department at my university were running Linux (SUSE if I recall correctly). It wasn’t my first exposure to Linux, but I was still more comfortable using Windows at this point.

VB.NET

I started learning how to program in my final few years of high school. My computer science teacher started us off with Visual Basic .NET. We were actually the first year group to use this stack. Previously my school used Delpi and Pascal, so it was new to everyone.

For my final year project, I built a system for a hairdresser complete with appointment scheduling, customer database, and inventory management!

MATLAB

The first week of university we got thrown into the deep end with a week-long Lego Mindstorms coursework project. There were no real limitations except for your imagination… and your MATLAB skills. In the end, our team built a robot with an automatic gearbox.

Despite MATLAB’s reputation for not being a ‘real’ programming language, I used it a lot throughout all four years at university, including for my Master’s thesis! I’d really recommend MATLAB Cody if you’re looking to improve your MATLAB skills.

C++

Still one of the favourite languages for teaching undergraduates. C++ was used extensively, but one of my proudest pieces of work in C++ is still the logic simulator I wrote for a coursework project.

I really got to cut my teeth on C++ during my first ever internship, working on an H.265/HEVC video encoder at Cisco. To this day, it was some of the most challenging (in a good way) work I’ve done. Or to use someone else’s words “H.264 is Magic”.

2020

Flash forward to 2020 and I’ve been programming professionally for almost 6 years now. In that time, I’ve used a lot of different languages including Java, Python, and even a year working in X++ (despite my attempts to forget it!).

Even though I work at Microsoft, I’ve been running Arch Linux as my daily driver for over 3 years. Yes, I still need to use Windows in a VM from time to time, but the fact that I can achieve my developer workflow almost entirely from Linux just goes to show that Microsoft ♥ Linux isn’t just an empty platitude.

C#

It’s only in the last year or so that I’ve come back to working on a .NET stack, but already I’ve deployed applications on Azure Functions, ASP.NET Core running in Kubernetes, and most recently Service Fabric. C# is a real breath of fresh air coming from 4 years of Java, and I am really excited to see where the language goes after C# 8 and .NET 5.

TypeScript

If you’re doing front-end work nowadays, I think TypeScript is the best way to do it. It papers over the cracks of JavaScipt, and gives you much more confidence, especially when working in a large codebase. The most common stack I work in now is React + TypeScript, and it is a million times better than the jQuery days.

I’ve also used TypeScript for some back-end work too – most notably for Renovate. The type system really lends itself well to these sorts of back-end tasks, and I wouldn’t discount it over some of the more conventional stacks.

DevOps

Okay, so this one isn’t a programming language, but it’s definitely something that has changed the way I work. In this context, DevOps means a couple of things to me: testing, continuous integration/continuous delivery (CI/CD) and monitoring.

In 2010, testing meant manual testing. I remember for my hairdresser management system I had to document my manual test plan. It was a requirement of the marking scheme. Nowadays, it’s easier to think of testing as a pyramid with unit tests at the base, integration tests and E2E tests in the middle, and a small number of manual tests at the top. Ham Vocke’s The Practical Test Pyramid is the definitive guide for testing in 2020.

CI/CD has been one of my favourite topics lately. Even though the agile manifesto talked about it almost 20 years ago, only recently has the barrier to entry gotten so low. Between Github Actions, Gitlab CI, Travis CI and all the rest it’s a no-brainer. I use GitHub Actions in almost every side project I build.

Monitoring is such an important tool for running a successful service. You can use it to pre-emptively fix problems before they become problems or choose what areas to work on based on customer usage. Like CI/CD it’s become so easy now. For most platforms all you need to do is include an SDK!

2030?

Who knows what 2030 will bring? Maybe Rust will replace C++ everywhere? Maybe AI will have replaced programmers? Maybe Go will finally get generics?

Common async pitfalls—part two

Following on from part one, here’s some more of the most common pitfalls I’ve come across—either myself, colleagues and friends, or examples in documentation—and how to avoid them.

‘Fake’-sync is not async

If the method you are calling is synchronous, even in an async method, then call it like any other synchronous method. If you want to yield the thread, then you should use Task.Yield in most cases. For UI programming, see this note about Task.Yield from the .NET API documentation.

Delegates

Here’s a common pitfall when passing actions as method parameters:

The implicit type conversion from the async function to Action is, surprisingly, not a compiler error! This happens because the function doesn’t have a return value, so it’s converted to a method with an async void signature. In this example the side effects aren’t bad, but in a real application this could be terrible as it violates the expected execution contract.

Synchronization

Synchronizing asynchronous code is slightly more complicated than synchronizing synchronous code. Mostly, this is because awaiting a task will result in switching to a different thread. This means that the standard synchronization primitives, which require the same thread to acquire and release a lock, won’t work when used in an async state machine.

Therefore, you must take care to use thread safe synchronization primitives in async methods. For example, using lock, will block the current thread while your code waits to gain exclusive access. In asynchronous code, threads should only block for a short amount of time.

In general, it’s not a good idea to perform any I/O under a lock. There’s usually a much better way to synchronize access in asynchronous programming.

Lazy Initialization

Imagine you need to lazy initialize some object under a lock.

When converting RetrieveData to run asynchronously, you might try to rewrite Initialize a few different ways:

But there are a few issues:

1. You shouldn’t call external code under a lock. The caller has no idea what work the external code will do, or what assumptions it has made.
2. You shouldn’t perform I/O under a lock. Code sections under a lock should execute as quickly as possible, to reduce contention with other threads. As soon as you perform I/O under a lock, avoiding contention isn’t possible.

SemaphoreSlim

If you absolutely must perform asynchronous work which limits the number of callers, .NET provides SemaphoreSlim which support asynchronous, non-blocking, waiting.

You still need to take care when converting from a synchronous locking construct. Semaphores, unlike monitor locks, aren’t re-entrant.

IDisposable

IDisposible is used to finalize acquired resources. In some cases, you need to dispose of these resources asynchronously, to avoid blocking. Unfortunately, you can’t do this inside Dispose().

Thankfully, .NET Core 3.0 provides the new IAsyncDisposible interface, which allows you to handle asynchronous finalization like so:

IEnumerable and IEnumerator

Usually you would implement IEnumerable or IEnumerator so you can use syntactic sugar, like foreach and LINQ-to-Objects. Unfortunately, these are synchronous interfaces that can only be used on synchronous data sources. If your underlying data source is actually asynchronous, you shouldn’t expose it using these interfaces, as it will lead to blocking.

With the release of .NET Core 3.0 we got the IAsyncEnumerable and IAsyncEnumerator interfaces, which allow you to enumerate asynchronous data sources:

Prefer the compiler-generated state machine

There are some valid cases for using Task.ContinueWith, but it can introduce some subtle bugs if not used carefully. It’s much easier to avoid it, and just use async and await instead.

TaskCompletionSource

TaskCompletionSourc<T> allows you to support manual completion in asynchronous code. In general, this class should not be used… but when you have to use it you should be aware of the following behaviour:

Common async pitfalls—part one

The .NET Framework provides a great programming model that enables high performance code using an easy to understand syntax. However, this can often give developers a false sense of security, and the language and runtime aren’t without pitfalls. Ideally static analysers, like the Microsoft.VisualStudio.Threading.Analyzers Roslyn analysers, would catch all these issues at build time. While they do help catch a lot of mistakes, they can’t catch everything, so it’s important to understand the problems and how to avoid them.

Here’s a collection of some of the most common pitfalls I’ve come across—either myself, colleagues and friends, or examples in documentation—and how to avoid them.

Blocking calls

The main benefit of asynchronous programming is that the thread pool can be smaller than a synchronous application while performing the same amount of work. However, once a piece of code begins to block threads, the resulting thread pool starvation can be ugly.

If I run a small test, which makes 5000 concurrent HTTP requests to a local server, there are dramatically different results depending on how many blocking calls are used.

% blocking shows the number of calls that use Task.Result, which blocks the thread. All other requests use await.

The increased total duration when using blocking calls is due to the thread pool growth, which happens slowly. You can always tune the thread pool settings to achieve better performance, but it will never match the performance you can achieve with non-blocking calls.

Streams

Like all other blocking calls, any methods from System.IO.Stream should use their async equivalents: Read to ReadAsync, Write to WriteAsync, Flush to FlushAsync, etc. Also, after writing to a stream, you should call the FlushAsync method before disposing the stream. If not, the Dispose method may perform some blocking calls.

CancellationToken

You should always propagate cancellation tokens to the next caller in the chain. This is called a cooperative cancellation model. If not, you can end up with methods that run longer than expected, or even worse, never complete.

To indicate to the caller that cancellation is supported, the final parameter in the method signature should be a CancellationToken object.

Linked tokens

If you need to put a timeout on an inner method call, you can link one cancellation token to another. For example, you want to make a service-to-service call, and you want to enforce a timeout, while still respecting the external cancellation.

Cancelling uncancellable operations

Sometimes you may find the need to call an API which does not accept a cancellation token, but your API receives a token and is expected to respect cancellation. In this case the typical pattern involves managing two tasks and effectively abandoning the un-cancellable operation after the token signals.

Constructors

Occasionally, you may find yourself wanting to perform asynchronous work during initialization of a class instance. Unfortunately, there is no way to make constructors async.

There are a couple of different ways to solve this. Here’s a pattern I like:

1. A public static creator method, which publicly replaces the constructor
2. A private async member method, which does the work the constructor used to do
3. A private constructor, so callers can’t directly instantiate the class by mistake

So, if I apply the same pattern to the class above the class becomes:

And we can instantiate the class by calling var foo = await Foo.CreateAsync(1, 2);.

In cases where the class is part of an inheritance hierarchy, the constructor can be made protected and InitializeAsync can be made protected virtual, so it can be overridden and called from derived classes. Each derived class will need to have its own CreateAsync method.

Parallelism

Avoid premature optimization

It might be very tempting to try to perform parallel work by not immediately awaiting tasks. In some cases, you can make significant performance improvements. However, if not used with care you can end up in debugging hell involving socket or port exhaustion, or database connection pool saturation.

Using async everywhere generally pays off without having to make any individual piece of code faster via parallelization. When threads aren’t blocking you can achieve higher performance with the same amount of CPU.

Avoid Task.Factory.StartNew, and use Task.Run only when needed

Even in the cases where not immediately awaiting is safe, you should avoid Task.Factory.StartNew, and only use Task.Run when you need to run some CPU-bound code asynchronously.

The main way Task.Factory.StartNew is dangerous is that it can look like tasks are awaited when they aren’t. For example, if you async-ify the following code:

be careful because changing the delegate to one that returns Task, Task.Factory.StartNew will now return Task<Task>. Awaiting only the outer task will only wait until the actual task starts, not finishes.

Normally what you want to do, when you know delegates are not CPU-bound, is to just use the delegates themselves. This is almost always the right thing to do.

However, if you are certain the delegates are CPU-bound, and you want to offload this to the thread pool, you can use Task.Run. It’s designed to support async delegates. I’d still recommend reading Task.Run Etiquette and Proper Usage for a more thorough explanation.

If, for some extremely unlikely reason, you really do need to use Task.Factory.StartNew you can use Unwrap() or await await to convert a Task<Task> into a Task that represents the actual work. I’d recommend reading Task.Run vs Task.Factory.StartNew for a deeper dive into the topic.

Null conditionals

Using the null conditional operator with awaitables can be dangerous. Awaiting null throws a NullReferenceException.

Instead, you must do a manual check first.

A Null-conditional await is currently under consideration for future versions of C#, but until then you’re stuck with manually checking.

Zwift on Linux

Getting Zwift to run on Linux was a journey I started just over a year ago. I didn’t get very far with my effort, but since then a lot of progress has been made by the Wine developers and others in the community, and Zwift is now (mostly) playable on Linux. I’ll admit there are some workarounds required. Like having to use the Zwift companion app to connect sensors. But on the whole, it works well. So I wanted to summarise the process for anyone who wants to try it for themselves.

I’m using Lutris, a gaming client for Linux, to script out all the steps needed to make games playable on Linux. If you’ve never used it before, I’d really recommend it for gaming on Linux in general. First things first, you’re going to have to download and install Lutris for your Linux distribution. Thankfully Lutris has a great help page explaining how to do this for most distributions.

Installation

Once you’ve got Lutris installed, installing Zwift is pretty easy. In Lutris search for Zwift, select the only result, and click the “Install” button to start the installation process. You can also start the installer from the command line by running lutris install/zwift-windows.

This might take a while, and depending on your internet speed could be anywhere from 10 minutes to around an hour.

Once the Zwift launcher has finished downloading and updating, we’ve hit the first hurdle that can’t be scripted with Lutris.

The launcher will appear as a blank white window. Actually, the launcher is displaying a web page, but Wine can’t render properly. Thankfully all the files are already downloaded, so all you need to do is quit the launcher window, and exit Zwift from the Wine system menu. After that, the Lutris installer should complete.

Running Zwift

Zwift requires the Launcher to be running all the time while in-game. However, Lutris only allows 1 application to launch from the “Play” button. So before you hit the play button, first you need to click “Run EXE inside wine prefix” and browse to drive_c\Program Files (x86)\Zwift\ZwiftLauncher. You should see that familiar blank white screen.

Finally, you can hit the “Play” button and Ride On 👍

How to host your Helm chart repository on GitHub

Since the release of Helm 3, the official helm/charts repository has been deprecated in favour of Helm Hub. While it’s great for decentralization and the long term sustainability of the project, I think there’s a lot more that is lost. Where is the best place to go for of the expert advice now? Installing Helm now requires you to manually add each repository you use. And there’s now some added friction to hosting your Helm charts.

Thankfully GitHub has all the tools required, in the form of GitHub Pages and GitHub Actions, to host a fully automated build pipeline and to host a repository for your Helm charts. Also, we can use some of the tools from the community to ensure our charts are high quality.

GitHub Pages

First you need to go ahead and create a gh-pages branch in your repository. As I’m writing this there’s an issue open to do this automatically, but to do it manually you can run the following:

git checkout --orphan gh-pages
git rm -rf .
git commit -m "Initial commit" --allow-empty
git push

Once you’ve done that, you need to enable GitHub Pages in your repository. Go to the settings page on your repository and set the source branch to the gh-pages branch you just created.

Now you’ve configured GitHub Pages, it will act as your Helm repository. Next, you need to configure GitHub Actions to publish to there.

GitHub Actions

You’re going to use GitHub Actions to create two workflows: one for pull requests, and one for commits to master. Your pull request workflow will deal with linting and testing your chart using a collection of automated tooling. While this isn’t a direct replacement for the expert advice offered by the Helm community, it’s better than nothing. Your master branch workflow will deal with releasing your charts using GitHub pages, meaning you never have to do it manually.

First up let’s look at the pull request workflow.

Pull requests

For each pull request in your chart repository, you want to run a series of different validation and linting tools to catch any avoidable mistakes in your Helm charts. To do that, go ahead and create a workflow in your repository by creating a file at .github/workflows/ci.yaml and add the following YAML to it:

This will run the workflow on any pull request that changes files under the charts directory.

That’s the skeleton of the workflow sorted, next onto the tools that you’re going to use.

Chart Testing

The Helm project created Chart Testing, AKA ct, as a comprehensive linting tool for Helm charts. To use it in your pull request build, you’ll go ahead and add the following job:

Where ct.yaml is:

For a full list of configuration options check out this sample file.

The lint action for Chart Testing is a bit of a catch-all that helps you prevent a lot of potential bugs or mistakes in your charts. That includes:

• Version checking
• YAML schema validation on Chart.yaml
• YAML linting on Chart.yaml and values.yaml
• Maintainer validation on changed charts

Helm-docs

Helm-docs isn’t strictly a linting tool, but it makes sure that your documentation stays up-to-date with the current state of your chart. It requires that you create a README.md.gotmpl in each chart repository using the available templates, otherwise it will create a README.md for you using a default template.

To use it as part of your pull request build, you need to add the following job:

Where helm-docs.sh is:

This runs Helm-docs against each chart in your repository and generates the README.md for each one. Then, using git, you’ll fail the build if there are any differences. This ensures that you can’t check in any changes to your charts without also updating the documentation.

Kubeval

Next up is Kubeval. It validates the output from Helm against schemas generated from the Kubernetes OpenAPI specification. You’re going to add it to your pull request, and use it to validate across multiple different versions of Kubernetes. Add the following job:

Where kubeval.sh is:

This script is a bit longer, but if you break it down step-by-step it’s essentially:

1. Get a list of charts that have been changed between this PR and master branch
2. Install Kubeval
3. For each chart:
   1. Generate the Kubernetes configuration using Helm
   2. Validatate the configuration using Kubeval

You’re doing this for each version of Kubernetes you’ve defined in the job, so if you’re using an API that isn’t available in all versions, Kubeval will fail the build. This help keep backwards compatibility for all of your charts, and makes sure you’re not releasing breaking changes accidentally.

This doesn’t guarantee that the chart will actually install successfully on Kubernetes—but that’s where Kubernetes in Docker comes in.

Kubernetes in Docker (KIND)

Finally you’re going to use Chart Testing again to install your Helm charts on a Kubernetes cluster running in the GitHub Actions runner using Kubernetes in Docker (KIND). Like Kubeval, you can create clusters for different versions of Kubernetes.

KIND doesn’t publish Docker images for each version of Kubernetes, so you need to look at the Docker image tags. That’s why the Kubernetes versions in this job won’t necessarily match the versions used for the Kubeval job.

So you got a temporary Kubernetes cluster, installed your charts on it, and ran any helm tests (that you definitely wrote 🙄). This is the ultimate test of your Helm chart—installing and running it. If this passes, and you merge your pull request, you’re ready to release!

Releasing

Remember that gh-pages branch you created earlier? Now you can use it to publish your fully tested Helm chart to.

You’re going to create another GitHub workflow, this time at .github/workflows/release.yaml. This one is going to be significantly simpler:

It will check out the repository, set the configuration of Git to the user that kicked-off the workflow, and run the chart releaser action. The chart releaser action will package the chart, create a release from it, and update the index.yaml file in the gh-pages branch. Simple!

But one thing you still need to do is create a secret in your repository, CR_TOKEN, which contains a GitHub personal access token with repo scope. This is due to a GitHub Actions bug, where GitHub Pages is not deployed when pushing from GitHub Actions.

Once that’s all configured, any time a change under the charts directory is checked in, like from a pull request, your Github workflow will run and your charts will be available almost instantly!

Next steps

From here you’ll want to add your repository to Helm so you can use it, and share it on Helm Hub so others can too. For the former, you’ll need to run:

helm repo add renovate https://<username>.github.io/<repository>/
helm repo update

And for the latter, the Helm project have written a comprehensive guide that I couldn’t possibly top.

If you want to see all these pieces working together checkout the renovatebot/helm-charts repository, or our page on Helm Hub. And if you would like some help please reach out to me on Twitter at @Jamie_Magee.