May 5, 2023 · 1 minute readWhen it comes to Linux containers, there are plenty of tools out there that can scan container images, generate Software Bill of Materials (SBOM), or list vulnerabilities. However, Windows container images are more like the forgotten stepchild in the container ecosystem. And that means we’re forgetting the countless developers using Windows containers, too.
Instead of allowing this gap to widen further, container tool authors—especially SBOM tools and vulnerability scanners—need to add support for Windows container images.
In my presentation at Container Plumbing Days 2023 I showed how to extract version information from Windows containers images that can be used to generate SBOMs, as well as how to integrate with the Microsoft Security Updates API which can provide detailed vulnerability information.
Watch on YouTube: "Container Plumbing Days 2023: Windows Containers" May 4, 2023 · 7 minute readIs your Jest test suite failing you? You might not be using the testing framework’s full potential, especially when it comes to preventing state leakage between tests. The Jest settings clearMocks, resetMocks, restoreMocks, and resetModules are set to false by default. If you haven’t changed these defaults, your tests might be fragile, order-dependent, or just downright wrong. In this blog post, I’ll dig into what each setting does, and how you can fix your tests.
clearMocks
First up is clearMocks:
Automatically clear mock calls, instances, contexts and results before every test. Equivalent to calling jest.clearAllMocks() before each test. This does not remove any mock implementation that may have been provided.
Every Jest mock has some context associated with it. It’s how you’re able to call functions like mockReturnValueOnce instead of only mockReturnValue. But if clearMocks is false by default, then that context can be carried between tests.
Take this example function:
1export function randomNumber() {
2 return Math.random();
3}
And this simple test for it:
1jest.mock('.');
2
3const { randomNumber } = require('.');
4
5describe('tests', () => {
6 randomNumber.mockReturnValue(42);
7
8 it('should return 42', () => {
9 const random = randomNumber();
10
11 expect(random).toBe(42);
12 expect(randomNumber).toBeCalledTimes(1)
13 });
14});
The test passes and works as expected. However, if we add another test to our test suite:
1jest.mock('.');
2
3const { randomNumber } = require('.');
4
5describe('tests', () => {
6 randomNumber.mockReturnValue(42);
7
8 it('should return 42', () => {
9 const random = randomNumber();
10
11 expect(random).toBe(42);
12 expect(randomNumber).toBeCalledTimes(1)
13 });
14
15 it('should return same number', () => {
16 const random1 = randomNumber();
17 const random2 = randomNumber();
18
19 expect(random1).toBe(42);
20 expect(random2).toBe(42);
21
22 expect(randomNumber).toBeCalledTimes(2)
23 });
24});
Our second test fails with the error:
1Error: expect(jest.fn()).toBeCalledTimes(expected)
2
3Expected number of calls: 2
4Received number of calls: 3
And even worse, if we change the order of our tests:
1jest.mock('.');
2
3const { randomNumber } = require('.');
4
5describe('tests', () => {
6 randomNumber.mockReturnValue(42);
7
8 it('should return same number', () => {
9 const random1 = randomNumber();
10 const random2 = randomNumber();
11
12 expect(random1).toBe(42);
13 expect(random2).toBe(42);
14
15 expect(randomNumber).toBeCalledTimes(2)
16 });
17
18 it('should return 42', () => {
19 const random = randomNumber();
20
21 expect(random).toBe(42);
22 expect(randomNumber).toBeCalledTimes(1)
23 });
24});
We get the same error as before, but this time for should return 42 instead of should return same number.
Enabling clearMocks in your Jest configuration ensures that every mock’s context is reset between tests. You can achieve the same result by adding jest.clearAllMocks() to your beforeEach() functions. But this isn’t a great idea as it means you have to remember to add it to each test file to make your tests safe, instead of using clearMocks to make them all safe by default.
resetMocks
Next up is resetMocks:
Automatically reset mock state before every test. Equivalent to calling jest.resetAllMocks() before each test. This will lead to any mocks having their fake implementations removed but does not restore their initial implementation.
resetMocks takes clearMocks a step further, by clearing the implementation of any mocks. However, you need to use it in addition to clearMocks.
Going back to my first example again, I’m going to move the mock setup inside the first test case randomNumber.mockReturnValue(42);.
1describe('tests', () => {
2 it('should return 42', () => {
3 randomNumber.mockReturnValue(42);
4 const random = randomNumber();
5
6 expect(random).toBe(42);
7 expect(randomNumber).toBeCalledTimes(1)
8 });
9
10 it('should return 42 twice', () => {
11 const random1 = randomNumber();
12 const random2 = randomNumber();
13
14 expect(random1).toBe(42);
15 expect(random2).toBe(42);
16
17 expect(randomNumber).toBeCalledTimes(2)
18 });
19});
Logically, you might expect this to fail, but it passes! Jest mocks are global to the file they’re in. It doesn’t matter what describe, it, or test scope you use. And if I change the order of tests again, they fail. This makes it very easy to write tests that leak state and are order-dependent.
Enabling resetMocks in your Jest context ensures that every mock implementation is reset between tests. Like before, you can also add jest.resetAllMocks() to beforeEach() in every test file. But it’s a much better idea to make your tests safe by default instead of having to opt-in to safe tests.
restoreMocks
Next is restoreMocks:
Automatically restore mock state and implementation before every test. Equivalent to calling jest.restoreAllMocks() before each test. This will lead to any mocks having their fake implementations removed and restores their initial implementation.
restoreMocks takes test isolation and safety to the next level.
Let me rewrite my example a little bit, so instead of mocking the function directly, I’m mocking Math.random() instead.
1const { randomNumber } = require('.');
2
3const spy = jest.spyOn(Math, 'random');
4
5describe('tests', () => {
6 it('should return 42', () => {
7 spy.mockReturnValue(42);
8 const random = randomNumber();
9
10 expect(random).toBe(42);
11 expect(spy).toBeCalledTimes(1)
12 });
13
14 it('should return 42 twice', () => {
15 spy.mockReturnValue(42);
16
17 const random1 = randomNumber();
18 const random2 = randomNumber();
19
20 expect(random1).toBe(42);
21 expect(random2).toBe(42);
22
23 expect(spy).toBeCalledTimes(2)
24 });
25});
With clearMocks and resetMocks enabled, and restoreMocks disabled, my tests pass. But if I enable restoreMocks both tests fail with an error message like:
1Error: expect(received).toBe(expected) // Object.is equality
2
3Expected: 42
4Received: 0.503533695686772
restoreMocks has restored the original implementation of Math.random() before each test, so now I’m getting an actual random number instead of my mocked return value of 42. This forces me to be explicit about not only the mocked return values I’m expecting, but the mocks themselves.
To fix my tests I can set up my Jest mocks in each individual test.
1describe('tests', () => {
2 it('should return 42', () => {
3 const spy = jest.spyOn(Math, 'random').mockReturnValue(42);
4 const random = randomNumber();
5
6 expect(random).toBe(42);
7 expect(spy).toBeCalledTimes(1)
8 });
9
10 it('should return 42 twice', () => {
11 const spy = jest.spyOn(Math, 'random').mockReturnValue(42);
12
13 const random1 = randomNumber();
14 const random2 = randomNumber();
15
16 expect(random1).toBe(42);
17 expect(random2).toBe(42);
18
19 expect(spy).toBeCalledTimes(2)
20 });
21});
resetModules
Finally, we have resetModules:
By default, each test file gets its own independent module registry. Enabling resetModules goes a step further and resets the module registry before running each individual test. This is useful to isolate modules for every test so that the local module state doesn’t conflict between tests. This can be done programmatically using jest.resetModules().
Again, this builds on top of clearMocks, resetMocks, and restoreMocks. I don’t think this level of isolation is required for most tests, but I’m a completionist.
Let’s take my example from above and expand it to include some initialization that needs to happen before I can call randomNumber. Maybe I need to make sure there’s enough entropy to generate random numbers? My module might look something like this:
1let isInitialized = false;
2
3export function initialize() {
4 isInitialized = true;
5}
6
7export function randomNumber() {
8 if (!isInitialized) {
9 throw new Error();
10 }
11
12 return Math.random();
13}
I also want to write some tests to make sure that this works as expected:
1const random = require('.');
2
3describe('tests', () => {
4 it('does not throw when initialized', () => {
5 expect(() => random.initialize()).not.toThrow();
6 });
7
8 it('throws when not initialized', () => {
9 expect(() => random.randomNumber()).toThrow();
10 });
11});
initialize shouldn’t throw an error, but randomNumber should throw an error if initialize isn’t called first. Great! Except it doesn’t work. Instead I get:
1Error: expect(received).toThrow()
2
3Received function did not throw
That’s because without enabling resetModules, the module is shared between all tests in the file. So when I called random.initialize() in my first test, isInitialized is still true for my second test. But once again, if I were to switch the order of my tests in the file, they would both pass. So my tests are order-dependent again!
Enabling resetModules will give each test in the file a fresh version of the module for each test. Though, this might actually be a case where you want to use jest.resetAllModules() in your beforeEach() instead of enabling it globally. This kind of isolation isn’t required for every test. And if you’re using import instead of require, the syntax can get very awkward very quickly if you’re trying to avoid an 'import' and 'export' may only appear at the top level error.
TL;DR reset all of the things
By default, Jest tests are only isolated at the file level. If you really want to make sure your tests are safe and isolated, add this to your Jest config:
1{
2 clearMocks: true,
3 resetMocks: true,
4 restoreMocks: true,
5 resetModules: true // It depends
6}
There is a suggestion to make this part of the default configuration. But until then, you’ll have to do it yourself.
Mar 16, 2023 · 5 minute readOne big advantage that Arch Linux has over other distributions, apart from being able to say “BTW I use Arch.”, is the Arch User Repository (AUR). It’s a community-driven repository with over 80,000 packages. If you’re looking for a package, chances are you’ll find it in the AUR.
Keeping all those packages up to date, takes a lot of manual effort by a lot of volunteers. People have created and used tools, like urlwatch and aurpublish, to let them know when upstream releases are cut and automate some parts of the process. I know I do. But I wanted to automate the entire process. I think Renovate can help here.
Updating versions with Renovate
Renovate is an automated dependency update tool. You might have seen it opening pull requests on GitHub and making updates for npm or other package managers, but it’s a lot more powerful than just that.
Renovate has a couple of concepts that I need to explain first: datasources and managers. Datasources define where to look for new versions of a dependency. Renovate comes with over 50 different datasources, but the one that is important for AUR packages is the git-tags datasource. Managers are the Renovate concept for package managers. There isn’t an AUR or PKGBUILD manager, but there is a regex manager that I can use.
I can create a renovate.json configuration with the following regex manager configuration:
1{
2 "regexManagers": [
3 {
4 "fileMatch": ["(^|/)PKGBUILD$"],
5 "matchStrings": [
6 "pkgver=(?<currentValue>.*) # renovate: datasource=(?<datasource>.*) depName=(?<depName>.*)"
7 ],
8 "extractVersionTemplate": "^v?(?<version>.*)$"
9 }
10 ]
11}
Breaking that down:
- The
fileMatch setting tells Renovate to look for any PKGBUILD files in a repository - The
matchStrings is the regex format to extract the version, datasource, and dependency name from the PKGBUILD - The
extractVersionTemplate is to handle a “v” in front of any version number that is sometimes added to Git tags
And here’s an extract from the PKGBUILD for the bicep-bin AUR package that I maintain:
1pkgver=0.15.31 # renovate: datasource=github-tags depName=Azure/bicep
Here I’m configuring Renovate to use the github-tags datasource and to look in the Azure/bicep GitHub repository for new versions. That means it’ll look in the list of tags for the Azure/bicep repository for any new versions. If Renovate finds any new versions, it’ll automatically update the PKGBUILD and open a pull request with the updated version.
So I’ve automated the PKGBUILD update, but that’s only half of the work. The checksums and .SRCINFO must be updated before pushing to the AUR. Unfortunately, Renovate can’t do that (yet, see Renovate issue #16923), but GitHub Actions can!
Updating checksums and .SRCINFO with GitHub Actions
Updating the checksums with updpkgsums is easy, and generating an updated .SRCINFO with makepkg --printsrcinfo > .SRCINFO is straightforward too. But doing that for a whole repository of AUR packages is going to be a little trickier. So let me build up the GitHub actions workflow step-by-step.
First, I only want to run this workflow on pull requests targeting the main branch.
1on:
2 pull_request:
3 types:
4 - opened
5 - synchronize
6 branches:
7 - main
Next, I’m going to need to check out the entire history of the repository, so I can compare the files changed in the latest commit with the Git history.
1jobs:
2 updpkgsums:
3 runs-on: ubuntu-latest
4 steps:
5 - name: Checkout
6 uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # v3.3.0
7 with:
8 fetch-depth: 0
9 ref: ${{ github.ref }}
Getting the package that changed in a pull request requires a little bit of shell magic.
1- name: Find updated package
2 run: |
3 #!/usr/bin/env bash
4 set -euxo pipefail
5
6 echo "pkgbuild=$(git diff --name-only origin/main origin/${GITHUB_HEAD_REF} "*PKGBUILD" | head -1 | xargs dirname)" >> $GITHUB_ENV
Now I’ve found the package that changed in the Renovate pull request, I can update the files.
This step in the workflow uses a private GitHub Action that I have in my aur-packages repository. I’m not going to break it down here, but at its core it runs updpkgsums and makepkg --printsrcinfo > .SRCINFO with a little extra configuration required to run Arch Linux on GitHub Actions runners. You can check out the full code on GitHub.
1- name: Validate package
2 if: ${{ env.pkgbuild != '' }}
3 uses: ./.github/actions/aur
4 with:
5 action: 'updpkgsums'
6 pkgname: ${{ env.pkgbuild }}
Finally, once the PKGBUILD and .SRCINFO are updated I need to commit that change back to the pull request.
1- name: Commit
2 if: ${{ env.pkgbuild != '' }}
3 uses: stefanzweifel/git-auto-commit-action@3ea6ae190baf489ba007f7c92608f33ce20ef04a # v4.16.0
4 with:
5 file_pattern: '*/PKGBUILD */.SRCINFO'
Check out this pull request for bicep-bin where Renovate opened a pull request, and my GitHub Actions workflow updated the b2sums in the PKGBUILD and updated the .SRCINFO.
But why stop there? Let’s talk about publishing.
Publishing to the AUR
Each AUR package is its own Git repository. So to update a package in the AUR, I only need to push a new commit with the updated PKGBUILD and .SRCINFO. Thankfully, KSXGitHub created the github-actions-deploy-aur GitHub Action to streamline the whole process.
If I create a new GitHub Actions workflow to publish to the AUR, I can reuse the first two steps from my previous workflow to check out the repository and find the updated package. Then all I need to do is to use the github-actions-deploy-aur GitHub Action:
1- name: Publish package
2 uses: KSXGitHub/github-actions-deploy-aur@065b6056b25bdd43830d5a3f01899d0ff7169819 # v2.6.0
3 if: ${{ env.pkgbuild != '' }}
4 with:
5 pkgname: ${{ env.pkgbuild }}
6 pkgbuild: ${{ env.pkgbuild }}/PKGBUILD
7 commit_username: ${{ secrets.AUR_USERNAME }}
8 commit_email: ${{ secrets.AUR_EMAIL }}
9 ssh_private_key: ${{ secrets.AUR_SSH_PRIVATE_KEY }}
All together now
If you own any AUR packages and want to automate some of the maintenance burden, check out my AUR packages template GitHub repository. It contains all of the steps I showed in this blog post. And if you want to see how it works in practice, check out my AUR packages GitHub repository.
Jan 2, 2023 · 5 minute readWhen it comes to Linux containers, there are plenty of tools out there that can scan container images, generate Software Bill of Materials (SBOM), or list vulnerabilities. However, Windows container images are more like the forgotten stepchild in the container ecosystem. And that means we’re forgetting the countless developers using Windows containers, too.
I wanted to see what I’d need to make scanning tools for Windows container images. Turns out it’s pretty easy. So easy, in fact, I think the existing container tools should add support for Windows container images.
What version of Windows am I running?
The first question I needed to answer was: what version of Windows was the container image based on? This tells me what date the container image is from, what updates are applicable, and what vulnerabilities it has.
Container images are really just tar files, and Windows container images are no different. So first I saved a Windows container image locally using skopeo:
1$ skopeo --insecure-policy --override-os windows copy docker://mcr.microsoft.com/windows/nanoserver:ltsc2022 dir:///tmp/nanoserver
2$ ls /tmp/nanoserver
30db1879370e5c72dae7bff5d013772cbbfb95f30bfe1660dcef99e0176752f1c 7d843aa7407d9a5b1678482851d2e81f78b08185b72c18ffb6dfabcfed383858 manifest.json version
Next, I inspected the manifest using jq to find the layer that had the Windows files.
1$ jq . manifest.json
2{
3 "schemaVersion": 2,
4 "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
5 "config": {
6 "mediaType": "application/vnd.docker.container.image.v1+json",
7 "size": 638,
8 "digest": "sha256:0db1879370e5c72dae7bff5d013772cbbfb95f30bfe1660dcef99e0176752f1c"
9 },
10 "layers": [
11 {
12 "mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar",
13 "size": 304908800,
14 "digest": "sha256:7d843aa7407d9a5b1678482851d2e81f78b08185b72c18ffb6dfabcfed383858"
15 }
16 ]
17}
I then extracted the layer and fixed the permissions.
1$ mkdir layer
2$ tar -xf 7d843aa7407d9a5b1678482851d2e81f78b08185b72c18ffb6dfabcfed383858 -C ./layer/
3$ sudo find ./layer -type f -exec chmod 0644 {} \;
4$ sudo find ./layer -type d -exec chmod 0755 {} \;
5$ ls -lah layer/
6total 16K
7drwxr-xr-x 4 jamie users 4.0K Dec 28 15:05 .
8drwxr-xr-x 3 jamie users 4.0K Dec 28 15:00 ..
9drwxr-xr-x 5 jamie users 4.0K Dec 9 01:18 Files
10drwxr-xr-x 3 jamie users 4.0K Dec 9 01:22 UtilityVM
11$ ls -lah layer/Files/
12total 28K
13drwxr-xr-x 5 jamie users 4.0K Dec 9 01:18 .
14drwxr-xr-x 4 jamie users 4.0K Dec 28 15:05 ..
15-rw-r--r-- 1 jamie users 5.6K Dec 9 01:18 License.txt
16drwxr-xr-x 4 jamie users 4.0K May 7 2021 ProgramData
17drwxr-xr-x 6 jamie users 4.0K Dec 9 01:19 Users
18drwxr-xr-x 20 jamie users 4.0K Dec 9 01:19 Windows
Inside the extracted layer there are two directories: Files and UtilityVM. Files had the filesystem of the Windows container image, while UtilityVM is used by Hyper-V behind the scenes. So I just needed to focus on Files.
How did I figure out the specific version of Windows the container is running? From the registry of course! The SOFTWARE registry hive contained information about installed software, including Windows itself, and was found at Files/Windows/System32/config/SOFTWARE.
Thankfully, there’s a great NuGet package called Registry that let me easily load and parse the registry, but there are also packages for Go, Rust, and even Node.js.
1using Registry;
2
3var registryHive = new RegistryHive("/tmp/nanoserver/layer/Files/Windows/System32/config/SOFTWARE");
4registryHive.ParseHive();
5var currentVersion = registryHive.GetKey(@"Microsoft\Windows NT\CurrentVersion");
6var fullVersion =
7 $"{currentVersion.GetValue("CurrentMajorVersionNumber")}.{currentVersion.GetValue("CurrentMinorVersionNumber")}.{currentVersion.GetValue("CurrentBuildNumber")}.{currentVersion.GetValue("UBR")}";
8Console.WriteLine(fullVersion);
Running this code, I got version 10.0.20348.1366 which was apparently released on 13th December 2022.
What about Windows updates?
The version of Windows doesn’t tell the whole story. There are also updates that can be applied on top. You might have seen them referred to by their KB number, for example KB1234567. Information on what updates have been applied is also stored in the registry.
By extending my earlier code, I can find out what updates this container image has.
1var packages = registryHive.GetKey(@"Microsoft\Windows\CurrentVersion\Component Based Servicing\Packages");
2var updatePackageRegex = new Regex(@"^Package_\d+_for_(KB\d+)~\w{16}~\w+~~((?:\d+\.){3}\d+)$");
3
4var updates = new Dictionary<string, string>();
5foreach (var packageKey in packages.SubKeys)
6{
7 if (!updatePackageRegex.IsMatch(packageKey.KeyName))
8 {
9 continue;
10 }
11
12 var currentState = packageKey.Values.Find(v => v.ValueName == "CurrentState")?.ValueData;
13
14 // Installed
15 if (currentState == "112")
16 {
17 var groups = updatePackageRegex.Match(packageKey.KeyName).Groups;
18 updates[groups[1].Value] = groups[2].Value;
19 }
20}
21
22foreach (var update in updates)
23{
24 Console.WriteLine($"{update.Key}: {update.Value}");
25}
Running this gave me a single update: KB5020373: 20348.1300.1.0. Searching online for KB5020373 led me to the documentation for the update. It’s the November 2022 security update for .NET Framework and has a fix for CVE-2022-41064.
Done! …Now what if we scaled this?
It turns out it’s not that difficult to find out info about Windows container images. It took me a couple of hours to figure out, but that’s only because no one seems to have done this before. The actual code is only about 30 lines.
Windows containers are widely used for legacy applications, like .NET Framework applications, that haven’t been rewritten but could benefit from the cloud. All of the big three cloud providers offer managed Kubernetes services that support Windows nodes out of the box (yes, Kubernetes supports Windows nodes). There is clearly a demand for Windows containers, but there is a gap in the kind of container tooling that has sprung up for Linux containers.
Instead of allowing this gap to widen further, I think that container tool authors—especially SBOM tools and vulnerability scanners—should add support for Windows container images. These tools should then correlate the extracted information with the Microsoft Security Research Center (MSRC) API. MSRC publishes information every month on security updates. Comparing the Windows version from a container image with the fixed versions provided by the MSRC API, you could easily see your container image’s security vulnerabilities.
As my proof-of-concept has shown, it’s low-hanging fruit. A small addition that would have a big impact for the many forgotten developers and the applications they work on.
Jul 26, 2022 · 6 minute readThe GitHub documentation has a lot of good advice about rate limits for its API, and how to make the most of them. However, since using the GitHub API, there are some things I’ve discovered that the documentation doesn’t cover, or doesn’t cover so well.
Conditional requests
This topic is actually covered very well in the GitHub documentation. To summarise, all REST API requests will return ETag headers, and most will return Last-Modified. You can make use of these by making subsequent requests with the If-None-Match and If-Modified-Since headers respectively. If the resource hasn’t been modified, you’ll get back a 304 Not Modified response, and the request won’t count against your rate limit.
To show you what I mean, here’s a short example:
1$ curl -I -H "Authorization: token ..." "https://api.github.com/user
2< HTTP/2 200
3< etag: "0c05f6422602a76a6671b28fc70af0ff9775ee41c80aca7d527814bb79a0fc2c"
4< last-modified: Mon, 21 Feb 2022 17:25:59 GMT
5< x-ratelimit-limit: 5000
6< x-ratelimit-remaining: 4993
7< x-ratelimit-reset: 1645482669
8
9$ curl -I -H "If-None-Match: \"0c05f6422602a76a6671b28fc70af0ff9775ee41c80aca7d527814bb79a0fc2c\"" -H "Authorization: token ..." "https://api.github.com/user"
10< HTTP/2 304
11< etag: "0c05f6422602a76a6671b28fc70af0ff9775ee41c80aca7d527814bb79a0fc2c"
12< last-modified: Mon, 21 Feb 2022 17:25:59 GMT
13< x-ratelimit-limit: 5000
14< x-ratelimit-remaining: 4993
15< x-ratelimit-reset: 1645482669
16
17$ curl -I -H "If-Modified-Since: Mon, 21 Feb 2022 17:25:59 GMT" -H "Authorization: token ..." "https://api.github.com/user"
18< HTTP/2 304
19< last-modified: Mon, 21 Feb 2022 17:25:59 GMT
20< x-ratelimit-limit: 5000
21< x-ratelimit-remaining: 4993
22< x-ratelimit-reset: 1645482669
The first request uses one request of my rate limit, taking it from 4994 to 4993. But the next two requests use If-None-Match and If-Modified-Since headers, so my rate limit is still 4993.
Unfortunately, conditional requests are only available for the REST API. HTTP caching over GraphQL is not a simple problem, and it’s unlikely that GitHub will ever support it.
Prefer If-Modified-Since
The GitHub REST API documentation covers conditional requests pretty well. The reason I’m mentioning it? Well, the documentation says that you can use ETag or If-Modified-Since interchangeably—but they’re not equivalent. Take a look at this example:
1$ curl -I -H "Authorization:token ..." "https://api.github.com/repos/renovatebot/renovate/releases/latest"
2< HTTP/2 200
3< etag: "70eb55000ec3e69bc2d88079714612000a955d4afaf02643b6602d99fb60dd8d"
4< last-modified: Mon, 21 Feb 2022 21:47:30 GMT
And if I make the same request a little bit later…
1$ curl -I -H "Authorization:token ..." "https://api.github.com/repos/renovatebot/renovate/releases/latest"
2< HTTP/2 200
3< etag: "85f04330d7bca80e6e0d62ac1b41b6d57e2ff11744565655e46732d44736dba6"
4< last-modified: Mon, 21 Feb 2022 21:47:30 GMT
The ETag is different but the Last-Modified time is still the same as before. Based on this StackOverflow question, it appears as if this has been an issue for a while. So if a response has both an ETag and a Last-Modified time, I’d recommend using the Last-Modified time to make conditional requests.
Both REST and GraphQL
Saying “rate limit” isn’t really accurate. What I actually mean is “rate limits”. GitHub actually has nine different rate limits. Some are for very specific use cases, like integration_manifest for the GitHub App Manifest code conversion endpoint. But the two that are most useful are core (AKA REST) and graphql.
If I make a request to the rate limit endpoint, you can see all the different rate limits.
1{
2 "resources": {
3 "core": {
4 "limit": 5000,
5 "used": 0,
6 "remaining": 5000,
7 "reset": 1656981763
8 },
9 "search": {
10 "limit": 30,
11 "used": 0,
12 "remaining": 30,
13 "reset": 1656978223
14 },
15 "graphql": {
16 "limit": 5000,
17 "used": 38,
18 "remaining": 4962,
19 "reset": 1656979534
20 },
21 "integration_manifest": {
22 "limit": 5000,
23 "used": 0,
24 "remaining": 5000,
25 "reset": 1656981763
26 },
27 "source_import": {
28 "limit": 100,
29 "used": 0,
30 "remaining": 100,
31 "reset": 1656978223
32 },
33 "code_scanning_upload": {
34 "limit": 1000,
35 "used": 0,
36 "remaining": 1000,
37 "reset": 1656981763
38 },
39 "actions_runner_registration": {
40 "limit": 10000,
41 "used": 0,
42 "remaining": 10000,
43 "reset": 1656981763
44 },
45 "scim": {
46 "limit": 15000,
47 "used": 0,
48 "remaining": 15000,
49 "reset": 1656981763
50 },
51 "dependency_snapshots": {
52 "limit": 100,
53 "used": 0,
54 "remaining": 100,
55 "reset": 1656978223
56 }
57 },
58 "rate": {
59 "limit": 5000,
60 "used": 0,
61 "remaining": 5000,
62 "reset": 1656981763
63 }
64}
The REST API has a rate limit of 5000 requests per hour. Separately, the GraphQL API has a rate limit of 5000 points per hour.
Depending on what API calls you want to make, you can intelligently split them across the REST and GraphQL APIs to achieve a higher overall limit. For example, if a GraphQL call is going to cost a lower number of points than the number of REST calls required to get the same data, you should make those calls via the GraphQL API. You should also bear in mind that you can make conditional requests to the REST API, but not to the GraphQL API.
Maximise page size
Whenever you’re making a request to an endpoint with pagination, you should check what the maximum results per page are and set your query parameter to that size.
The default size for most endpoints is 30 results, but the maximum size is often 100. If you forget to set this you might need to make four times as many requests to get the same number of results.
Use sorting
Most API calls allow you to sort them based on a date field when querying an endpoint. If you use this—and do some caching on your end as well—you can avoid having to fetch all pages for a request whenever you have a cache request.
For example, if you need to fetch the most recently changed pull requests for a repository, you should be sorting by updated and storing a local cache of pull requests. That way a conditional request cache miss won’t require you to fetch all the pages of a request. You can compare each page to your local cache, and only fetch the next page if required.
Use HEAD requests
This tip isn’t strictly about rate limits, but is useful when you’re eking out every last bit of performance. Nearly all GitHub REST API endpoints support HEAD requests, in addition to the other HTTP verbs. If you’re already using conditional requests, you can avoid having the body of a request sent over the wire by sending a HEAD request instead.
For example, here’s the header and body size for a GET request:
1$ curl -w \%{size_header}:\%{size_download} -s -o /dev/null -H "Authorization:token ..." "https://api.github.com/repos/renovatebot/renovate/releases"
21448:137229
And here’s the header and body size for the HEAD equivalent:
1$ curl -w \%{size_header}:\%{size_download} -s -o /dev/null -H "Authorization:token ..." "https://api.github.com/repos/renovatebot/renovate/releases"
21448:0
By making a HEAD request instead of a GET request, you can avoid being sent 137KB.
There is a trade-off, though. If you use conditional requests and have a cache miss, you’ll have to make the GET request anyway.
Summing up
Using these methods I’ve managed to eke out every bit of performance of the GitHub API for my integrations. Let me know what methods you use, or if there’s anything I’ve missed.