How I, a software engineer, secured The Razor Crest

The cast: myself and my 10 year old daughter, both fans of The Mandalorian and of LEGO

The background: my former team gives me a generous LEGO gift card as a leaving gift, that would cover acquiring LEGO 75292, The Razor Crest, as coveted by my daughter and I

The problem: the set is sold out almost everywhere; occasionally stock becomes available on the official LEGO shop, but it sells out within minutes...

And now, the solution...

As software engineers we talk alot about our professional engineering, our side-projects, we take on coding challenges and katas, we chastise ourselves publicly for writing our own blogging software, but we rarely talk about how we can use our skills in small ways to help out in day-to-day "real life".

By this, I don't mean providing technical support to relatives who can't connect their new printer to Windows or homebrewing your own blogging software (for which it has become a cliché to publicly chastise ourselves), just using the skills in small ways to make some other task easier, the way that people who work in kitchens often know great little tips to enhance day-to-day cooking.

When it works out it can be immensely satisfying - I remain proud of the time that, with my knowledge of e-commerce operations and how localisation and white-labelling works, I managed to successfully order an item that was only available on the Italian website of a clothing chain via the German website for delivery to my partner here in Berlin using some slight tinkering with cookies and the Firefox browser console. She still reminds me of this from time to time.

So my intention for this blog post is not a how-to guide on keeping tabs on the stock levels of e-commerce sites, rather a celebration of the fact that these are the kinds of things we can so easily do. It can be so easy to feel burnt out and jaded as an engineer, it's not always easy to find genuinely purposeful work, that it must be healthy to acknowledge that you can do some pretty cool stuff as well, which has some small purpose if only to yourself and those around you.

So for the record, here's how I did it.

I needed to get some kind of alert when something on a web page changed. Yes, there exist paid services to do this for you. Maybe even open source projects you can deploy yourself. But that's not the point here!

For me, the first thing to look into here is how to get a useful alert. Checking for a change in a webpage is a problem I know I can solve, so best to put the initial effort into where the greatest unknowns are.

I have a MacBook Pro, and know full well that there's a notification centre used by plenty of apps. It didn't take much googling to find that this is very, very easy to use from the CLI:

osascript -e 'display notification "Hello, World!"'

This might not be enough to immediately get my attention though, especially when I'm away from the machine. Fortunately, it is easy to add sound to:

osascript -e 'display notification "Hello, World!" sound name "Blow.aiff"'

(the standard sound files can be found in ls /System/Library/Sounds, at least on my somewhat ailing machine)

This is trivial to call from a bash script. Now I need to work out how to check the availability from a bash script.

First thing is to know what it is you need to look for, so I did a little digging into the LEGO shop website using the Firefox web developer tools. Inspecting the text where it tells you the availability, under the price, shows there's a nice consistently identifiable element available:

<p color="green" data-test="product-overview-availability" class="Text__BaseText-sc-178efqu-0 fNGpzM ProductOverviewstyles__AvailabilityStatus-sc-1a1az6h-11 ejRirH">
    <span class="Markup__StyledMarkup-ar1l9g-0 hlipzx">Available now</span>
</p>

There's some horribly named classes on there that I wouldn't want to rely on, but data-test="product-overview-availability" looks just the ticket.

Another nice thing about the LEGO site is that it isn't some JavaScript-framework generated SPA monstrosity - the HTML is there when you view source. Sure, it's heavily augmented by JS, but the information I need is available without (just as it should be). In other words, I won't need to run it in a browser, headless or otherwise, I can just grab the source. As a dev I have httpie installed because it's much nicer than using curl:

http GET https://www.lego.com/en-de/product/default-the-mandalorian-bounty-hunter-transport-au-the-razor-crest-nz-the-razor-crest-75292

I could jsut grep the result for the text Available now but bitter experience has taught me to be more precise, so I'm determined to check that specific part of the page for the desired text. Another CLI tool I have installed for interacting with HTML is pup:

pup 'p[data-test="product-overview-availability"] span text{}'

This will get me the text content of the element I'm interested in. Connecting this up with http is easy:

http GET https://www.lego.com/en-de/product/the-mandalorian-bounty-hunter-transport-75292 \
| pup 'p[data-test="product-overview-availability"] span text{}'

I also discovered by browsing around that there are other statuses e.g. Back orders accepted, will be shipped by ... which are entirely acceptable for me placing an order. All I really know is that I don't want Temporarily out of stock. This is easily checked with grep:

grep -qv "Temporarily out of stock"

(-q keep it quiet, -v invert match). Chain this all together and stick it in a bash script:

#!/usr/bin/env bash
set -euo pipefail

http GET https://www.lego.com/en-de/product/the-mandalorian-bounty-hunter-transport-75292 \
| pup 'p[data-test="product-overview-availability"] span text{}' \
| grep -qv "Temporarily out of stock" \
&& osascript -e 'display notification "THE RAZOR CREST IS ON OUR SCOPES!" sound name "Blow.aiff"'

Now I just need to run this periodically as CRON job, not so frequently that LEGO thinks I'm an attacker but often enough that I'm not going to miss the short availability windows it has been having.

*/2 * * * * cd /Users/jimkinsey/scripts && ./rc-checker.sh

And so, on Friday 29th January 2021 at 06:14AM (yes, it woke me up)...

Then literally the following day thanks to LEGO's phenomenally fast shipping, et voila

It's a lovely set, fun to build (and time-consuming!) and I will be forever grateful to my former team for enabling me in this, as I'd never have spent that kind of money on a LEGO set otherwise.

Looking back over this now I can see deficiencies in how I've implemented it, but again that's not really the point - it was a one-off task that did it's job leveraging my knowledge as an engineer. And that's pretty cool! (also: I got LEGO)