Dylan Staley

What's On My Desk: 2024 Edition

Sun, 23 Jun 2024 12:00:00 -0700

Next month is my seven-year anniversary of working from home. Since then, I've learned a lot about having a workspace that is functional, minimizes annoyance, and allows me to do my best work. The biggest thing I've learned is that while I really enjoy having a separate work and personal computer, I don't like the idea of having separate anything else. The idea of having to switch between a work keyboard and personal keyboard is something that I find deeply unacceptable. I'm sure it works for some people, but it's not something I could manage myself. I want one set of peripherals, shared between two computers. I want to buy high quality hardware just once and get that benefit both personally and professionally with the push of a single button. While it's taken a while to get the implementation details down, I think I've finally found something that gets me incredibly close to that ideal. I've been using most of this setup for the past two-and-a-half years and have recently made a few adjustments to eliminate some of the pain points I've run into in the past. This is my best attempt yet at a completely ludicrous work-from-home setup that is easily shared between a personal and a work computer at the push of a button.

The centerpiece of the whole setup is a Level1Techs KVM, specifically the DisplayPort 1.4 Single Monitor – Two Computer</a> 10Gbps model. This wonderful little piece of hardware allows me to share my 4K 144Hz monitor and USB peripherals between both my work MacBook Pro and my personal desktop. Many other KVMs, especially the more affordable options, omit full DisplayPort 1.4 compatibility, which would limit me to a paltry 60Hz refresh rate. As anyone who has used a high-refresh rate display can attest, it's a difficult comfort to give up.

</a>
My Level1Techs KVM sitting on top of the Dell WD19TBS Thunderbolt 3 dock. On the left is the Drop BMR1 speaker. Also pictured here is an Ecobee room sensor, which helps keep my office at a tolerable temperature.</figcaption> </figure>
Switching between computers is as easy as pressing a single button, which moves my monitor, keyboard, mouse, microphone, teleprompter, and Stream Deck between computers. If you read my post from a year ago</a>, you'll notice an interesting omission: my webcam. While resetting the USB hub every time I switched computers worked, it wasn't exactly an elegant solution. That's where the recently released Elgato Game Capture Neo comes in.
The Game Capture Neo</a> works similarly to the Elgato CamLink</a>: it takes in an HDMI signal and presents that to the host computer as a UVC device (which most software understands as a webcam). The neat extra feature it has that the CamLink doesn't is HDMI passthrough; since this is designed to capture gaming footage, having the ability to pass the HDMI signal along to a display is useful. For me though, this allows me to connect my HDMI webcam to two computers simultaneously, without having the awkward dance of resetting the USB hub to get the CamLink working when I switch computers. I've used this setup for a week at my new job and it seems to be working reliably. There is one small issue though: the Game Capture Neo advertises a variety of compatible resolutions, both in 4:3 and 16:9 aspect ratios even when the source aspect ratio is 16:9. This means that when software requests a 640x480 resolution, the Game Capture Neo happily squishes my 16:9 video into a 4:3 container. Thankfully I've only run into one application where this is the case, but I work around this using Camo</a>. Think of it as a more user-friendly OBS. Camo allows me to request only the 720p output of the Game Capture Neo and is smart enough to correctly scale that when another piece of software requests a 640x480 video stream. I use Camo in applications that expect a 640x480 output from the webcam, and the Game Capture Neo in everything else.
Speaking of video calls, I've also added an Elgato Prompter</a> to the mix. I really like the idea of being able to make eye contact while on a video call (something that companies like Microsoft are going so far as to use AI to fake</a>), and the Elgato Prompter allows that by putting a specially designed reflective mirror in front of your webcam. It has a small screen that acts as an external display that you can put your video call on, which allows you to look directly into the camera lens. It's not without its faults though; I had to add a circular polarizer</a> to my camera to eliminate a glow that came from the screen, which darkens the image a bit. This is easily rectified with the use of two Elgato Key Lights</a>, which also help combat the darkness of my home office. The Prompter uses a technology called DisplayLink</a>, which allows you to connect displays over non-display protocols like USB and TCP/IP; this helps when you have a device that doesn't have any available display out ports. I've found that this isn't the most reliable thing in the world, but it's a small price to pay for the benefit I feel that it brings.
</a>
Before and after turning on my Elgato Key Lights. Notice how the colors are significantly warmer and brighter.</figcaption> </figure>
For a webcam, I use a Sony a6000</a> with a Sigma 30mm f/1.4 DC DN Contemporary lens</a>. This was the camera I originally upgraded to when I became unsatisfied with the quality of typical webcams, but I switched to the Sony a5100 for its pop-up display; since the Elgato Prompter blocks the camera I decided to switch back. The a6000 also gives me the ability to control the camera via an IR remote</a>, which is useful for adjusting settings without having to unmount the camera from the prompter to be able to see the display. I also have the camera plugged into a Philips Hue Smart Plug</a>, which allows it to be turned on and off without physically moving the power dial on the camera.
I've also upgraded my audio setup, switching from the Mackie CR3 monitors to the Drop BMR1 Nearfield monitors</a>. These diminutive speakers don't have incredible reviews, but I mainly got them for their size, not their performance characteristics. This is mostly because I've recently outfitted my office with a stereo pair of the Sonos Era 100s</a>, which I use to listen to music (which is the main reason I'd want good speakers anyway). The BMR1s have been sufficient for the occasional YouTube video and work wonderfully for video calls. Plus, they take up significantly less space on my desk compared to the CR3s.
Previously, I was able to connect both my work and personal computers directly to the CR3s due to them having multiple inputs. The BMR1s only have one line-in, but this is trivially solved with the Rolls MX42 Stereo Mini Mixer</a>, which is a passive device that can take in up to four stereo signals and output them over one output. This allows me to have both computers outputting simultaneously to the same set of speakers. This is convenient since it allows me to get notifications from both computers at the same time.
</a>
The ROLLS MX42 Stereo Mini Mixer. Shortly after taking this photo, I realized that I had forgotten to switch the left and right channels since the Drop BMR1s expect the active unit to be on the right, but I have it placed on my left.</figcaption> </figure>
For keyboard and mouse, I use the Logitech MX Mechanical Mini</a> and the MX Master 3S</a>. Both connect to Logitech's wireless Bolt receiver, which allows them to switch between computers with the rest of the USB peripherals. This setup wouldn't work with Bluetooth devices, since those are bound to a specific host. I also love that the MX keyboards are designed for both Windows and macOS, with appropriate labels on the keys. Lots of other keyboards offer a Mac/PC switch, but this is usually a physical toggle on the device itself; the Logitech keyboards change their behavior based on what OS the receiver is connected to. This means that the keyboard switches between Mac and PC layouts automatically when the KVM switches between computers. No manual intervention required. This is important because the position of the Super key (Start on PC keyboards and Command on Mac keyboards) is immediately to the left and right of the spacebar on a Mac layout, whereas that key position is used for the Alt key on a PC layout. Once I get used to the layout switching, I'll probably explore building my own keyboard that has similar behavior; while I think the MX Mechanical Mini is a great keyboard, I do miss the loud "thock" of my previous keyboard, the Drop ALT.
</a>
The Logitech MX Mechanical Mini keyboard, which has labels for both the Mac and PC key layout.</figcaption> </figure>
Another new piece of desk gadgetry that I've acquired recently is the Elgato Stream Deck +</a>. I've owned the Stream Deck Mini for several years, but I never really loved using it because it felt so chintzy; pressing the keys required more force than pushing it across my desk, meaning I had to hold it in place and press buttons. The Stream Deck + is heavier, which results in button presses requiring less force than moving it, which makes it feel more stable. The larger model also has some lovely tactile knobs that I use to control the brightness and temperature of my Key Lights, along with my system volume. I have actions setup to put my office into video call mode, which turns the webcam on, turns the Key Lights on, dims my overhead lights, and turns on a light in my hallway for some additional contrast. I'm still finding ways to integrate it into my setup; I'm pretty sure I can figure out a way to have it trigger the KVM, which would allow me to mount the KVM underneath my desk. The touchscreen to switch pages is also a nice feature, although I'd love the ability to switch pages with a knob instead.
</a>
The Elgato Stream Deck +. I'm still finding good uses for the buttons, hence why some are empty.</figcaption> </figure>
At my previous job I used the Dell WD19TBS Thunderbolt 3 dock</a> with the Dell laptop they supplied. I was surprised to see that it works wonderfully with a MacBook Pro, including full 4K 144Hz DisplayPort output over the Thunderbolt 3 port. The dock doesn't have audio out, but that's fine since I usually try to use the Apple USB-C to 3.5mm Headphone Jack adapter</a> anyway. Its digital-to-analog converter is excellent, but it seems to have issues being plugged into a USB-A to USB-C adapter, which results in the DAC going completely haywire and outputting full-volume static every few hours. I switched to the Creative Sound BlasterX G1</a>, which uses USB-A, and haven't had any issues.
Overall, I'm happy with the setup I've managed to assemble given the quality it results in. I get to use a great 4K 144Hz monitor, a mirrorless DSLR webcam, teleprompter, wireless keyboard and mouse, and more between both my work and personal computers, all with the push of a single button.
</a>
All of the hardware in action. You'll notice that the underside of my desk is conspicuously missing; that's because, like all eldritch horrors, merely looking upon it is enough to drive anyone mad. The wallpaper is "Chatbots" by David Lanham.</figcaption> </figure>

Hardware Used</h2></summary>

Level1Techs DisplayPort 1.4 KVM</a></li>
Anker 7-Port USB 3.0 Hub</a></li>
Sony INZONE M9 4K HDR 144Hz Gaming Monitor</a></li>
Ergotron NX Monitor Arm</a></li>
Camera

Sony a6000</a></li>
Sigma 30mm f/1.4 DC DN Contemporary Lens</a></li>
Amazon Basics Circular Polarizer Lens Filter</a></li>
Elgato CamLink 4K</a></li>
Elgato Game Capture Neo</a></li>
Elgato Prompter</a></li>
Elgato Key Light</a></li>
Philips Hue Smart Plug</a></li>
Sony Wireless Remote</a></li> </ul> </li>
Audio

Rolls MX42 Stereo Mini Mixer</a></li>
Drop BMR1 Nearfield Monitors</a></li>
Sonos Era 100</a></li>
Apple USB-C to 3.5 mm Headphone Jack Adapter</a></li>
Creative Sound BlasterX G1 USB DAC</a></li>
Elgato Wave:1 (discontinued, replaced by the Wave:3</a>)</li>
RODE PSA1 Studio Arm</a></li> </ul> </li>
Logitech MX Keys Mechanical</a></li>
Logitech MX Master 3S</a></li>
Elgato Stream Deck +</a></li>
Dell WD19TBS Thunderbolt Dock</a></li> </ul> </details>

Dynamic Open Graph Images with @vercel/og and Zola

Thu, 20 Jun 2024 12:00:00 -0700

The website you're currently reading is a static site, built with the static site generator Zola</a>. I've always been a fan of static sites. There's something lovely about the idea that it's all just a collection of HTML and CSS files that you can toss on basically any hosting service and have a website going. One of the neat things about hosting on Vercel, though, is that I can opt-in to some of their more dynamic features, even when using a static site. So, inspired by Zach Leatherman</a>, I wanted to see if I could add support for dynamic Open Graph image generation using Vercel's @vercel/og</code></a> library. Spoiler alert: you can!
Whether you're using a static site generator like Zola or Hugo</a>, or a JavaScript framework like Next.js</a> or Astro</a>, Vercel gives you access to API routes</a>, which are basically functions that can run in response to an incoming request. Frameworks usually have much nicer implementations (including a fancy file-based router), but static sites still can have API routes by simply dropping some JavaScript into a folder named api</code> in the root of your project. When deployed to Vercel, their build system will bundle those files into separate functions and deploy them, making them available at /api</code>.
However, what about local development? zola serve</code> isn't going to magically understand how to invoke HTTP endpoints written in JavaScript. That's where the Vercel CLI</a> comes into play. If you're using a known framework, running vercel dev</code> inside your project will start two servers, one being your framework on a random port, and the other being a proxy server that knows how to route requests to either your API endpoints or to your framework's server. For instance, when I run vercel dev</code> in the folder for this website, vercel</code> runs the proxy server on port 3000, and starts the zola</code> binary on port 51579. This allows me to run both Zola and Vercel Functions locally for development.
The next question that you probably have is how can you install JavaScript dependencies in a static site project that doesn't use a JavaScript package manager? Well, the answer is pretty unexciting. It's simply "use a JavaScript package manager". Even if your project is configured using a static site framework preset, Vercel is smart enough to install dependencies when it sees a package.json</code> file. This is admittedly a bit weird, especially considering that the "Install Command" listed in my project's "Build & Development Settings" configuration is "None", but I'm thankful for the behavior as it allows me to install @vercel/og</code>, which is the magical piece that allows me to support dynamic Open Graph images like this one:

So now that we know how to create API routes and install JavaScript dependencies, let's generate some Open Graph images. Dropping the following example from Vercel into /api/og.jsx</code> should make the /api/og</code> URL return a static "Hello, world!" image.
import { ImageResponse } from '@vercel/og' export function GET() { return new ImageResponse( ( <div style={{ display: 'flex', fontSize: 128, background: 'white', width: '100%', height: '100%', }} > Hello, World! </div> ) ) } </code></pre>
However, if you run this locally and try to access /api/og</code>, you'll get a 404 NOT_FOUND</code> error. This is because the Vercel CLI by itself doesn't generate API routes from files ending in .jsx</code>. If you try to simply rename the file to og.js</code>, you'll get Error: Unexpected token '<'</code>. This makes sense given that we have JSX in a .js</code> file. Ultimately, I wasn't able to figure out how to get the Vercel CLI to work well with JSX. However, at the end of the day JSX is transformed to plain JavaScript anyway, so we can just perform the transform manually. Satori</a>, the underlying engine used by @vercel/og</code> that renders markup to images, expects objects with a type</code> property that contains a string value and a props</code> property containing, well, props. This is also where the children</code> prop goes. Without JSX, our example from above looks like this:
import { ImageResponse } from '@vercel/og' export function GET() { return new ImageResponse( { type: 'div', props: { style: { display: 'flex', fontSize: 128, background: 'white', width: '100%', height: '100%', }, children: 'Hello, World!' } } ) } </code></pre> With this code we're now able to dynamically generate Open Graph images even though we're using a static site! There's a few other nice things I've added to my own Open Graph images. The first is a simple helper function named h()</code> (real ones know) that allows me to build the image using a slightly nicer syntax than plain objects.
function h(tag, props) { return { type: tag, props }; } </code></pre> With h()</code>, the above example looks like this:
export function GET() { return new ImageResponse( h('div', { style: { display: 'flex', fontSize: 128, background: 'white', width: '100%', height: '100%', }, children: 'Hello, World!' }) ) } </code></pre> Satori also supports custom fonts if you provide a font file, but I didn't want to go through the trouble of storing a font file within my repo. Thankfully, Google Fonts will happily give you a compatible font file if you ask nicely (read: using the older /css</code> endpoint with a non-browser User-Agent</code>). This means that with a small helper function, I'm able to load the TTF data for any font hosted by Google Fonts:
function extractTTFURL(css) { const urlRegex = /src:\s*url$([^)]+)$/; const match = css.match(urlRegex); if (match && match[1]) { return match[1]; } return null; } async function loadGoogleFont(name) { const url = new URL("https://fonts.googleapis.com/css"); url.searchParams.append("family", name); const css = await fetch(url).then((res) => res.text()); const ttfUrl = extractTTFURL(css); if (!ttfUrl) { throw new Error(`unable to determine TTF URL from ${css}`); } const data = await fetch(ttfUrl).then((res) => res.arrayBuffer()); return { name, data }; } </code></pre> Finally, I didn't want to pass in the title via a URL parameter, mainly because I didn't want people to be able to generate arbitrary Open Graph images with my face on them (especially since generating images isn't free!). So, with a small tweak to vercel.json</code> to bundle the Markdown files for posts with the function's code, I'm now able to read the frontmatter during function invocation. This means I can trivially configure how the function works via frontmatter, instead of creating an ever-growing list of URL parameters.
{ "$schema": "https://openapi.vercel.sh/vercel.json", "functions": { "api/og/index.js": { "includeFiles": "content/**/*.md" } } } </code></pre> As someone who spends a lot of time working with sites deployed on Vercel using frameworks like Next.js, it's nice to see that there's still some benefit to hosting static sites on Vercel. While I wish the documentation was better, I'm glad that Vercel makes functionality like Vercel Functions available to static sites.
If you're curious to see the full implementation, it's all open source</a>.

Do It With Kindness; or, Why a Skateboard Can (Sometimes) Be Faster Than a Ferrari

Mon, 10 Jun 2024 12:00:00 -0700

This post was originally shared with my team at HashiCorp on my last day. It has been edited to account for a broader audience.

I spent a lot of time at HashiCorp thinking about how I could help others level up their engineering skills. With that in mind, I wanted to make one final attempt to share some of the things that I think engineers can do to distinguish themselves apart from the rest. This, of course, isn't an exhaustive list, as any attempt to do so could probably fill several libraries, but I do think it captures some of the qualities that I've seen in the engineers I look up to and admire.
Read the error message/code/documentation</h2>
One of the really awesome things about software engineering is that if you don't know how something works, you can simply go read the code (well, when you have access to the code that is!). Get really, really good at that. In high school, my computer science teacher frequently told us that "computers don't do anything you haven't told them to do", which was a really nice way of telling us to stop complaining and just read the code that we had written. In large codebases it's unreasonable to expect engineers to understand off the top of their heads how everything works, so being able to open an unfamiliar piece of code and read it top-to-bottom and understand what it's doing is a skill you'll benefit from.
Similarly, get good at reading error messages and documentation. I can't count the number of times when an error message contained the specific piece of knowledge that I needed to go solve the issue. Even when an error message is terrible, it often comes with a stack trace, so you can see all the lines at play that resulted in a specific error. Jump into the code and walk backwards, and you'll likely figure out what the issue was, even if the error message was unhelpful.
Documentation is another thing that you should be good at reading. Frameworks like Next.js have tons of documentation detailing their features and functionality. If you haven't read through them at least once I highly suggest doing so, if only to drop breadcrumbs of knowledge that you can pick up when you run into issues.
Get good at searching</h2>
Learning how to search for things, whether it be on Google, Slack, GitHub, Datadog, or some other service, is such a superpower. In just a few seconds you can search across an unfathomably large amount of information, and being able to do so effectively can really boost your productivity. I find this most useful when searching after encountering an unfamiliar error message. One thing I really find useful is searching for the portion of the error message that is unlikely to appear in anything other than the error message itself but is also applicable enough that it will match against all instances of the error. In these situations, performing exact searches (usually by using quotations) is also a great strategy. Most search features have additional filters to narrow down the results, so make sure you're learning about and using those. For example, I frequently used the org:</code>, language:</code>, and path:</code> filters in GitHub Search.
Getting good at search is often just a matter of being able to intuit the pieces that are likely contributing to the error. As an example, a documentation author at HashiCorp was having issues with Docker on their MacBook where one of my team's Node applications would simply output "Segmentation fault" and exit. However, another user running the exact same branch on their MacBook did not encounter the same error. The first user was using an Intel Mac, whereas the second user was using Apple Silicon. With that information I Googled "docker node segmentation fault intel mac" and the first result was this issue from over a year ago</a>, which had some helpful configuration changes that we were able to make to get the Intel user back up and running.
Being good at search isn't just for error messages though. In organizations that use tools like Slack and Google Workspace, you can dig up some incredible things by using search. I frequently used Slack to look up previous conversations to get background on why certain decisions were made, especially when the people who made them were no longer at HashiCorp. Tools like Google Cloud Search (which allows you to search for text in anything hosted in Gmail or Google Drive) were similarly useful in finding old documents and slide decks, something I found myself referencing a lot, especially when it came to decisions made outside my team.
Make minimal reproductions</h2> When it comes to debugging, being able to make a minimal reproduction of the issue is basically the holy grail. Many open source projects won't even look at a bug report without one. The reason they're so good is that they boil the issue down to the pieces that are necessary to make it happen, and nothing else. This allows you to narrow your focus to just the pieces at play, and easily confirm if your changes resolve the issue. During my time at HashiCorp, I made dozens of minimal reproductions that ultimately resulted in pinpointing the source of an issue. These reproductions also helped the Next.js and Vercel teams fix issues on their end, ultimately benefiting all of their customers. Whether you start from scratch or start pulling pieces out one by one, being able to end up with the problem distilled to its most minimal form is a powerful skill to have. To give an example, about a year ago one of our team members ran into an issue with dynamic routes after updating Next.js where pages would render with data from the incorrect page, but only when our application was deployed to Vercel. They opened a support case and worked with Vercel to try and track the issue down, but after a month of back and forth, were unable to narrow down the issue enough to come to a resolution. When it began to block our ability to update to Next.js v14, I sat down with my coworker's notes and took a few hours to build a reproduction that was just a handful of files and reliably demonstrated the issue. When I shared that with Vercel, they were able to deploy a fix in just 10 days. Try to figure it out yourself first</h2> When you run into a problem, make a decent attempt at figuring it out on your own before you call in reinforcements. Don't spend too much time, especially if you're truly stuck, but you should at least try to solve the problem yourself. This allows you to build up experience with unfamiliar systems, and helps you develop your own understanding. Pairing with other people is a productivity boost, but it can also easily become a crutch holding you back from developing your own skills, especially if your pairing partner isn't experienced enough to help guide you to the solution, and instead simply tells you what the answer is. Align how long it takes to make a decision with how long it takes to unmake it</h2> Allen Pike has a great article titled "Do Something, So We Can Change It"</a> which I highly recommend reading. I often like to think about technical decisions in terms of how long it takes to undo a change. If it takes half an hour, there's no reason to spend two weeks trying to come up with a perfect solution. If it will take six months to undo, maybe make sure you're really confident that your approach is the correct one. Don't spend tons of time worrying over small, easily reversible decisions. Save that time for the big, complicated, thorny problems. It's okay to be wrong, but don't become known for it</h2> The best engineers I've ever worked with were the ones who weren't ashamed to admit they didn't know something. Instead of making something up or guessing, they were honest that they had no idea. As we become more experienced, there's this desire to maintain our prestige in the eyes of others, which often leads us to feeling discomfort when admitting we don't know something. That's totally okay! I can't recall a single instance where an engineer admitted they didn't know something and I thought less of them. However, there have been times in which an engineer confidently answered a question with incorrect information, without any qualification that their understanding wasn't complete. This is always an awkward, uncomfortable situation because correcting misinformation in a kind way takes a large amount of skill. It's always easier to just admit you don't know something, because you don't want to build a reputation for giving unreliable information. Be (a little) lazy</h2> Look, I understand the allure of building wonderful, beautiful, complicated systems, but sometimes the best thing is the laziest approach. Instead of trying to build something that covers every possible edge case, build the thing that covers the majority of the non-edge cases, and iterate from there. At HashiCorp, we talked a lot about "building a skateboard", which originates from a talk by Henrik Kniberg</a>, instead of jumping straight to building a car. We also started using the term "bus ticket" for something that currently exists and can be used without building something new. This way of thinking allows you to avoid spending a lot of time doing something that isn't quite the right thing. </a> "General Problem". XKCD.</figcaption> </figure> Knowing when to go with the easy approach and when to spend the time building the "right" approach is a skill that only comes with time and experience, but you can start by asking yourself "What is the easiest way to solve this problem?" Start from that solution, and expand outwards, instead of jumping straight to the solution to solve all instances of an entire class of problems. So, what does the lazy solution look like? Sometimes it's just copying and pasting code instead of trying to fit two use cases into the same API. Sometimes it's copying in a small dependency and modifying it slightly instead of making a pull request and waiting for a change to be merged. Sometimes it's accepting that you're only going to be able to handle a specific set of circumstances automatically and being okay with handling the other cases manually. All this being said, don't take this as advice to be bad at your job, or to take shortcuts that result in worse outcomes. Being lazy has nothing to do with not doing the work that needs to be done. It just means avoiding work that doesn't really need to be done. Be okay saying "No"</h2> You don't need to immediately say yes to every idea presented to you. Your best work will be highly opinionated. Believe it or not, the customer isn't always right. (An alternative version I've heard is "the customer is always right in matters of taste", which feels more like something I could get behind.) Often their ideas don't account for the full context at play, and likely aren't the best possible solution to the underlying problem. If someone approaches you with a specific solution, take the time to dig into what the actual problem is, and then try to solve that instead of simply saying "Yes" to whatever they've proposed. This is hard sometimes, and impossible in others, but is necessary if you want to have good results. Do everything with kindness</h2> Finally, in everything you do, do it with kindness. I can think of no worse outcome than becoming known as the person who doesn't. At the end of the day, our jobs are not so important that it's worth being unkind to someone else. Always assume that others are acting with positive intent, and ensure that your words and actions build others up instead of tearing them down. Focus on critiquing ideas, not people, and make sure your critique is constructive and actionable; there's no value in simply saying an idea is "bad". When others present ideas, even if you disagree, offer to work together to build towards something that you can agree with. Do this with eagerness and excitement. Working with others is a joy if you let it be, and even more so if you're a joy to work with yourself.
Pushing Buttons with Rust; or, How I Spent Four Hours Automating a One Second Task Mon, 23 Jan 2023 12:00:00 -0800 I've worked hard to set up my desk with the devices I enjoy using. This includes a nice monitor, a clicky mechanical keyboard with nice-looking keycaps, a webcam made out of a mirrorless camera, and a high-quality microphone. These are mostly used for my job but given how much money and effort I've spent on them, it'd be a waste not to use them for my personal use as well. My home office isn't spacious enough to support a work and personal desk, so I went with the next best thing: a KVM switch from Level 1 Techs</a> which allows me to easily switch all my peripherals from my personal computer to my work computer. I simply press a button (or a keyboard shortcut) and the KVM disconnects the peripherals from one computer and connects them to the other. There's just one problem. The KVM does this in a way that prevents the reset of most of the peripherals. While that's sufficient most of the time, it causes issues with devices that need a full power-cycle to reinitialize. In my case, it's my Elgato CamLink</a>, which I use to connect my Sony a5000 as a webcam. Without a power-cycle, the CamLink fails to present itself to the host computer. Once I figured out what the issue was, I came up with a hacky solution: put the powered USB hub on a smart outlet, and then use a button on my Elgato Stream Deck to power-cycle the hub, which in turn would power-cycle the CamLink. This has worked wonderfully for the past year, but there's still a problem. I keep forgetting to press the dang button. But what if I didn't need to remember? Why can't my computer just reset the hub for me when I switch from one computer to another? At first, I had some cursed ideas, like connecting an ESP32 to the Human Interface Device port on my KVM and using that to trigger both the device switching and the hub reset. But then I realized that if my computer knew what devices were connected to it at a given moment, surely there's a way to also know when a device gets removed! And sure enough, there is: enter the DeviceWatcher</a> API, which allows just that: receive notifications when a device is added or removed. To use the API, I turned to the windows</code></a> Rust crate, which is a projection of the Windows API for Rust. This allows me to access all of the Windows APIs using idiomatic Rust code, and felt like a perfect small project to do on a Sunday morning. So, without further ado, I'd like to introduce you to monitor-monitor</code></a>, a small Windows utility to send an off/on command to a TP-Link Kasa smart plug when a monitor is added and removed! I decided that the most important device to monitor was my monitor, since it's the cleanest indicator that the KVM has switched to my other computer. Using the keyboard wouldn't have worked since I regularly unplug it for cleaning, and my microphone requires that it be unplugged during software updates. I probably could have used the mouse or the CamLink, but the monitor seemed the best choice. Now, when I press the button to switch computers, monitor-monitor</code> receives an event when the monitor is removed and sends the off/on command to the smart plug. By the time the monitor has reacquired the display signal, all my peripherals are already reset and ready to go. Now I'm free to replace that muscle memory with this gif of two very good dogs: </video> Running Zola on WebAssembly Mon, 09 Jan 2023 12:00:00 -0800 The site you're reading this on is built using Zola</a> (unless of course you're reading this from some future date where I've decided to rebuild the site using something other than Zola, in which case how's the future?) and hosted on Vercel</a> (again, unless you're reading this after that's no longer the case). One of the neat features of Vercel is first-party support for various static-site generators, including the ability to control which version is used to render your site. When I was moving this site from Netlify to Vercel, I set the ZOLA_VERSION</code> environment variable to the latest available version, 0.16.1</code>, and was greeted with the following build log: [16:16:06.599] Cloning github.com/dstaley/dstaley.com (Branch: master, Commit: fcf0f09) [16:16:07.092] Cloning completed: 493.016ms [16:16:07.480] Looking up build cache... [16:16:07.773] Build Cache not found [16:16:07.807] Running "vercel build" [16:16:08.310] Vercel CLI 28.2.5 [16:16:08.522] Installing Zola version 0.16.1 [16:16:08.953] zola: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by zola) [16:16:08.953] zola: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by zola) [16:16:08.953] zola: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by zola) [16:16:08.954] zola: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by zola) [16:16:08.954] zola: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by zola) [16:16:08.954] zola: /lib64/libc.so.6: version `GLIBC_2.29' not found (required by zola) [16:16:08.954] Error: Command "zola build" exited with 1 </code></pre> Even though Zola is written in Rust, it still relies on glibc</a>, the GNU C Library. The update to v15 changed how the Zola binary for Linux was built, causing it to rely on newer versions of glibc. After a few emails with Vercel's support team, I confirmed that the build environment used by Vercel only had access to glibc 2.26, hence the errors when attempting to use the latest version of Zola. Now, at this point, I had a few options if I wanted to use the latest version of Zola to build my site, but the easiest was probably setting up my Vercel project to download a custom-built version of Zola that was built against a lower version of glibc. While it certainly would have worked, and wouldn't have been too much effort, it also wasn't a fun or interesting solution. </video> Instead, I decided to see if I could compile Zola to WASM targeting the WebAssembly System Interface (WASI) and run it as a standard npm package. Spoiler: I could! With most Rust projects, compiling for WASI is relatively simple. You can run cargo build --target wasm32-wasi</code> and get a neat .wasm</code> file that will then run using WASI runtimes like node</a>, Wasmtime</a>, WasmEdge</a>, and more. That is, unless the Rust project you're compiling uses features that aren't available in Rust's WASI implementation (such as networking, which has some support, but not enough for large libraries like hyper</a>). Zola, being a static site generator, heavily relies on networking support to provide the zola serve</code> command, which allows you to preview your static site using a local web server. If I wanted to build a WASM version of Zola that could be used to build my site, I was going to need to remove all of the networking code. One really neat aspect of Rust is its support for conditional compilation</a>, which allows you to exclude code from being compiled based on a number of different conditions. One of those conditions is called "features", which are basically what it says on the tin: optional features of your application. This meant that I could mark complete sections of code as relying on the serve</code> feature using the #[cfg(feature = "serve")]</code> attribute. By making the serve</code> feature a default feature, and compiling with the --no-default-features</code> flag, I could make sure that any code that relied on networking was completely disabled. However, networking isn't the only feature of Rust that isn't available in WASI. WebAssembly is single-threaded (although support for threads has been proposed</a>), so code that relied on spawning threads was also not going to work in my WASM port. The main instance of this lack of support came from rayon</code></a>, which is a data parallelism library that provides parallel loops that function the same way as sequential loops in the standard library. That parity is important, as I was able to basically provide an alternate implementation of rayon</code></a> that simply used the sequential version of the method provided by rayon</code>. This meant that even though the code was calling a .par_iter_mut()</code> method, it was actually invoking the built-in .iter_mut()</code> method. Most features that don't work on WASI will trigger compile-time errors, making it simple to make the necessary changes to get things compiling. However, there's unfortunately some issues that are only exposed at runtime. One such issue came from a fairly innocuous looking piece of code: pub fn is_path_in_directory(parent: &Path, path: &Path) -> Result<bool> { let canonical_path = path .canonicalize() .with_context(|| format!("Failed to canonicalize {}", path.display()))?; let canonical_parent = parent .canonicalize() .with_context(|| format!("Failed to canonicalize {}", parent.display()))?; Ok(canonical_path.starts_with(canonical_parent)) } </code></pre> This piece of code checks to see if the provided path</code> is contained by the provided parent</code>. This is achieved by canonicalizing each path and comparing the prefixes. While the .canonicalize()</code> method is provided when compiling to WASI, it will always error out since WASI doesn't really have the concept of paths</a> (at least not in the same way as most other operating systems think of them). Thankfully, the solution was to simply leave the path as is when running on WASI. So far, most of the issues I ran into were relatively easy to fix; it simply took a moment to figure out what was causing the issue, and then tweaking the code to act a bit differently when it was running on WASI. Unfortunately, there was a big roadblock ahead. Zola used libsass</code></a>. For those of you who are unaware, I envy you. LibSass, like Nokogiri</a>, is one of those dependencies that elicits long sighs from developers, primarily due to the fact that it's a C/C++ library that honestly has no business being integrated into non-C/C++ projects. It's even caused headaches for the Zola maintainers</a>, outside the scope of my WASM port. I did make a solid effort to get it working; I played around with virtually every aspect of WASI SDK</a> in an attempt to get it to compile. The main issue I was running into was that the Rust crate sass-rs</a> needed to link to the C++ standard library (sound familiar?). On Linux, this is usually provided by libstdc++</code>. However, in WASI SDK, this is provided by libc++</code>. I had to manually patch the build.rs</code></a> file to always return cargo:rustc-link-lib=dylib=c++</code> in an attempt to ensure it was linked correctly. Even though I was able to get things linking correctly on the Rust side, it would fail when compiling to WASM. As I've mentioned on this blog before I have limited patience when it comes to C code, so I eventually gave up and switched Zola's SASS implementation to grass</code></a>, a Sass compiler written purely in Rust. It worked wonderfully, and only required the smallest of changes</a>. Once I had a WASM version of Zola, I then needed to wrap the module in a bit of setup code so that the node runtime could execute the module. This was (thankfully!) trivially easy. Here's the complete implementation: "use strict"; const { readFile } = require("node:fs/promises"); const { WASI } = require("wasi"); const { env } = require("node:process"); const { join } = require("node:path"); /** * * @param {string} siteDir Path to Zola site, relative to the current working directory * @param {string} [baseUrl] */ module.exports = async function build(siteDir = ".", baseUrl) { let args = ["zola", "--root", "/", "build"]; if (baseUrl) { args = [...args, "--base-url", baseUrl]; } const wasi = new WASI({ args, env, preopens: { "/": join(process.cwd(), siteDir), }, }); const importObject = { wasi_snapshot_preview1: wasi.wasiImport }; const wasm = await WebAssembly.compile( await readFile(join(__dirname, "zola.wasm")) ); const instance = await WebAssembly.instantiate(wasm, importObject); wasi.start(instance); }; </code></pre> Once this was published to npm, all I needed to do to run this on Vercel was to point a build</code> script in my package.json</code> to the following file: import build from "@dstaley/zola-wasm"; const baseUrl = process.env.VERCEL_ENV === "production" ? "https://dstaley.com" : `https://${process.env.VERCEL_URL}`; await build(".", baseUrl); </code></pre> I dropped that into the repo for this site, created a pull request, and was greeted with the following output in the build log on Vercel: Cloning github.com/dstaley/dstaley.com (Branch: master, Commit: 78e53bf) Cloning completed: 569.844ms Restored build cache Running "vercel build" Vercel CLI 28.10.0 Installing dependencies... up to date in 172ms > dstaley.com@0.0.0 build > node --experimental-wasi-unstable-preview1 build.mjs (node:328) ExperimentalWarning: WASI is an experimental feature. This feature could change at any time (Use `node --trace-warnings ...` to show where the warning was created) Building site... Checking all internal links with anchors. > Successfully checked 0 internal link(s) with anchors. -> Creating 6 pages (0 orphan) and 1 sections Done in 6.2s. Build Completed in /vercel/output [14s] Generated build outputs: - Static files: 40 - Serverless Functions: 0 - Edge Functions: 0 Deployed outputs in 1s Build completed. Populating build cache... Uploading build cache [4.35 MB]... Build cache uploaded: 887.541ms Done with "." </code></pre> And with that I was able to build my site using the latest version of Zola on Vercel, despite the fact that Vercel didn't have the latest version of glibc. The code for this is, of course, available on GitHub</a> and on npm</a>. Was this a good idea? No, probably not. But it was a hell of a lot of fun. Hello iPod, Welcome to 2023 Tue, 03 Jan 2023 12:00:00 -0800 My first iPod was a silver iPod mini, given to me by my mom when I was around twelve years old. It wasn't my first portable audio player (that title going to a cheap flash-based MP3 player that, to this day, I've been unable to find out the model number), but it was undoubtedly the device that sparked an infatuation that remains to this day, over seventeen years later. I've owned a few different iPods since then, ending in a PRODUCT(RED) third generation iPod nano I purchased in my freshman year of high school, which would be the last iPod I owned before the smartphone became a thing. That is, until the summer of 2017. At the time, I was driving a car that, despite being made in 2014, still came with a USB port that you could plug an iPod into. I had grown tired of the instability of Bluetooth, and desperately missed having album art show up on my car's display. Armed with an annoyance and a modicum of programming knowledge (a terrible combination I'm afraid), I bought a fifth generation iPod and set out to see if I could figure out how to synchronize my meticulously curated Plex music library. When you consider the fact that the iPod has sold over 450 million devices since its introduction in 2001, you would think that there would be an abundance of ways to programmatically add music to an iPod. Unfortunately, that's not the case. Given that each subsequent generation of iPods modified the internal database structure, few projects managed to keep up with the changes enough to support later models of iPods. Even if you managed to keep up, Apple apparently sued developers</a> for figuring out how to correctly sign the iPod's database file. That being said, there are a few options when it comes to open-source software to interface with an iPod. libgpod libgpod is a C library used in software such as gtkpod</a>, Rhythmbox</a>, and Amarok</a>. Like most software written in C, this was an absolute nightmare to get working (for some definition of working that is). At the time I was using a macOS device, which wasn't technically supported by libgod, so I guess I was setting myself up for failure there. I tried various Python bindings, and even wrote some C code at one point. Desperate, I even tried getting it to run in Docker on my MacBook, but ultimately was unsuccessful in getting everything working; if I recall correctly, the main issue I was facing was a lack of album art, which was one of the primary motivators for even undertaking this adventure in the first place. ipod-sharp ipod-sharp</a> is a C# library originally used for Banshee</a> (before it switched to libgpod). Being written in C# and targeting Mono, the original cross-platform implementation of .NET, it was my hope that it'd be relatively easy to get running on macOS. Alas, the odds were not in my favor, and I was never able to get it running in a project, largely due to its use of Linux's HAL</a>, which has been deprecated. AppleScript Desperately grasping at straws, I eventually turned to AppleScript, the built-in scripting language that's a part of macOS. AppleScript allows for automating a wide variety of applications, including iTunes. While I was ultimately successful at copying a song to my iPod, I was never able to get album art to transfer. Plus, AppleScript was a macOS-only solution, and I really wanted something that would work on Windows, which I was planning to switch to at the time. gnupod Finally, I came across gnupod</a>, a tool written in Perl. Getting this running on macOS was a chore, so I eventually decided to take the easy route and use Docker to run the scripts provided by gnupod. With a smidge of Python, I was finally able to download my music from my Plex server, convert it to an iPod-compatible format using ffmpeg, and transfer it to the iPod via gnupod. Album art, one of the main reasons I wanted to use an iPod in the first place, was finally something I could handle! gnupod</code> worked wonderfully for over a year. I ended up using it right until I got a new car, which sadly didn't support iPod connections anymore. During my desperate search of the internet, I came across a mention on StackOverflow of an application called SharePod</a>. This Windows application allowed users to manage their iPods without using iTunes, and the developer had kindly provided the core management functions in an open-source library called sharepod-lib</code>. The only problem was that, contrary to the popular saying, it seemed the internet did forget about sharepod-lib. Originally uploaded to Google Code in 2010</a>, sharepod-lib</code> never found its way into any of the archives created of Google Code when it shut down in 2016. I combed through every possible location that might have retained a copy, even going so far as to run a paid Twitter campaign</a> in the hopes of tracking down someone who happened to have the repo. Every couple of months, I'd get interested in trying to recover a copy of sharepod-lib but would never make any progress. That would change on March 22, 2022. I can't remember what specifically made me change my search tactics, but I somehow came across the website for SuperSync</a>, an application that allowed you to synchronize multiple iTunes libraries. At one point in time, SuperSync relied on sharepod-lib</code> to power its iPod functionality, and given that this was a third-party application, there was a small chance that the application used the source code of sharepod-lib</code> rather than the compiled DLL. So, with my fingers crossed, I emailed the creator, Brad: Hi there! I noticed on your license page that you mention the usage of SharePodLib, a library that required a paid license for use in commercial products. I'm curious if when you acquired a license to use SharePodLib in a commercial product, you also got access to the source code for SharePodLib? While it was open sourced at one point, I'm afraid it's been lost to time, so I've been reaching out to people known to have used it in the past in the hopes that someone has a copy of the source. </blockquote> The next day, I received the following response from Brad: Hi Dylan, Here is a link to the two copies of the source code I have. One marked "newer" whatever that means. I think for a while I might have had a windows command line tool to connect to ipods. But gave up when Apple locked things down. https://drive.google.com/file/d/... I hope that helps! - Brad </blockquote> I can still remember my heart racing reading this email. I quickly downloaded the linked file and was greeted with the full directory structure for sharepod-lib</code>, a directory structure I must have looked at hundreds of times using Wayback Machine's capture of Google Code. I immediately made several backups, just in case. I spent the following few weeks updating sharepod-lib</code> to run on the modern, cross-platform version of .NET. Honestly, I was surprised at just how little I had to change to get things up and running. The primary changes were replacing old platform-specific libraries like SQLite and System.Drawing with modern, cross-platform versions. To ensure things worked across a variety of iPods, I amassed a small collection of nearly every model</a>. Knowing that there was finally going to be a modern, easily accessible way to manage iPods was something that really drove me over those few weeks. As a test run for the library, I built a prototype battery benchmark</a> that would copy a specific copyleft album to your connected iPod, create a playlist, and allow you to repeat the playlist until the battery died. The benchmark would then calculate the runtime based on the number of plays for each song, giving you an estimate of the battery life for your iPod. Once I finished updating the library, I had planned on recreating my original Plex sync application, allowing anyone to easily sync their iPod with their Plex music library, as I had done back in 2017. Sometimes, however, motivation is a fickle thing. As life got in the way, I found myself less and less enthused about spending hours writing unit tests and modernizing parts of the library. There was also the issue of accessing the iPod's extended SysInfo, a collection of data attributes necessary for album artwork support and creating valid database hashes (without which the iPod wouldn't parse the database). On Windows, it was relatively easy to issue the appropriate SCSI INQUIRY commands using an Administrator account (the same being true for Linux), but on modern versions of macOS that capability was reserved for kernel drivers. I spent many nights fighting with libusb, a userspace USB API in an attempt to recreate enough of the USB Mass Storage protocol to obtain the information necessary but was never able to get anything working reliably despite trying to write the application in C, Go, and Rust. This roadblock considerably dampened my motivations, and eventually I stopped working on the project completely. Looking at the files on my computer, I haven't updated anything in nearly six months. So, as they say, if you love something, set it free. That's why I'm super excited to announce that I've made sharepod-lib</code></a> and its modern successor Clickwheel</code></a> available on GitHub under the Lesser GNU Public License. It's my hope that someone finds one (or both!) of them useful and as inspiring to them as it has been to me over the last several years. I do have to give a note of warning though: it's in fairly rough shape. While there are some unit tests, most of my tests were written using disk images of my iPods, complete with copyrighted music. Because I have no desire whatsoever to have the repo removed because of a Digital Millenium Copyright Act violation, I've removed the tests that relied on copyrighted music. There are still some tests left, but I know they're not exhaustive. Clickwheel is in no way a production library, and it might never be, but I figured it'll do more good out there in the public than it would do sitting on my hard drive. It's my hope that by releasing this to the public, there might one day be an easy-to-use library to programmatically manage your iPod. And it won't have anything to do with C. Tailscale on Kobo Sage Mon, 22 Nov 2021 12:00:00 -0800 Last month, Rakuten finally released a reasonably-sized Kobo with USB C, the Kobo Sage. One of the most fascinating things about the Kobo eReaders (compared to the Kindle eReaders) is that they run a user-modifiable version of Linux, meaning you're free to do all sorts of weird and wild things with them. So I decided to put Tailscale</a> on my Kobo Sage. Tailscale is an easy-to-use VPN(ish) service that allows you to connect multiple devices over the internet to a secure, shared network. Since it's is distributed as statically-compiled binaries (thanks Go!), getting Tailscale downloaded and running on Kobo Sage was as easy as downloading the binaries, copying them to the onboard storage (since the root filesystem is only about 300 MB), symlinking them into a $PATH</code> friendly folder, and running tailscaled</code>. wget https://pkgs.tailscale.com/stable/tailscale_1.18.0_arm.tgz tar -xzf tailscale_1.18.0_arm.tgz mv tailscale_1.18.0_arm/tailscaled /mnt/onboard mv tailscale_1.18.0_arm/tailscale /mnt/onboard ln -s /mnt/onboard/tailscale /usr/bin/tailscale ln -s /mnt/onboard/tailscaled /usr/bin/tailscaled rm -rf tailscale_1.18.0_arm rm tailscale_1.18.0_arm.tgz tailscaled </code></pre> However, if you were to do this, you'll be greeted with this: wgengine.NewUserspaceEngine(tun "tailscale0") error: exec: "iptables": executable file not found in $PATH wgengine.New: exec: "iptables": executable file not found in $PATH </code></pre> While the Linux 4.9 kernel used by Kobo Sage supports iptables</code>, the actual binary is nowhere to be found on the device. Thankfully, acquiring a suitable binary isn't too terribly difficult. Getting iptables</code></h2> Unfortunately, there's no officially distributed pre-compiled binaries of iptables</code>. So, we can either compile from source (either on a 32-bit ARM device</a> or via cross-compilation), or we can yank the binary out of an image of a 32-bit ARM Linux distro. We need a distro that's old enough to be using glibc</code> version less than 2.19, so the best choice I came up with was the July 5, 2017 release of Raspbian, the last to be based on Debian 8 (codename jessie</code>). Using 7zip for Windows, I extracted the following files from the disk image: /sbin/xtables-multi</code></li> /lib/libip4tc.so.0.1.0</code></li> /lib/libip6tc.so.0.1.0</code></li> /lib/libxtables.so.10.0.0</code></li> </ul> Note: You can also extract these files on Linux, but it's a bit more complicated</a>. Now that we had a binary, it was time to put it on the device. Installing iptables</code></h2> Out of the box, Kobo Sage doesn't support ssh</code>, but it does support telnet</code> after enabliing dev mode</a>. Since I couldn't use scp</code> to copy files to the device, I ran a web server on my computer, and downloaded the files with wget</code>: wget http://callisto.localdomain:8000/xtables-multi wget http://callisto.localdomain:8000/libip4tc.so.0.1.0 wget http://callisto.localdomain:8000/libip6tc.so.0.1.0 wget http://callisto.localdomain:8000/libxtables.so.10.0.0 </code></pre> Once I had the files, I was able to move them into the appropriate locations: mv xtables-multi /sbin mv libip4tc.so.0.1.0 /lib mv libip6tc.so.0.1.0 /lib mv libxtables.so.10.0.0 /lib </code></pre> Since we need iptables</code> in $PATH</code>, I created a symlink in /sbin</code>: cd /sbin ln -s xtables-multi iptables </code></pre> We also need symlinks for the libraries, so I ran the following in /lib</code>: ln -s libxtables.so.10.0.0 libxtables.so.10 ln -s libip4tc.so.0.1.0 libip4tc.so.0 ln -s libip6tc.so.0.1.0 libip6tc.so.0 chmod 755 libxtables.so.10.0.0 chmod 755 libip4tc.so.0 chmod 755 libip6tc.so.0 </code></pre> Once I had the binary and libraries in the right spots, I was able to confirm that iptables</code> worked: iptables --version </code></pre> Testing Tailscale</h2> With the iptables</code> binary working, confirming Tailscale works was the next step. By default tailscaled</code> stores some state in /var/lib</code>, which the Kobo doesn't like. So, for the purposes of testing, I just manually told tailscaled</code> to store its state in a file at the root directory. tailscaled --state=tailscaled.state </code></pre> In another telnet session, I was able to bring Tailscale up with tailscale up </code></pre> After authenticating, the final step was to open a Tailscale-only URL in the Kobo's experimental web browser to confirm that I was, indeed, connected to my Tailscale network! </a> Staying Connected to Tailscale</h2> Getting Tailscale up and running wasn't as smooth as on a typical Linux device, but staying connected to Tailscale is an even trickier thing to do. Kobo Sage doesn't have systemd</code>, so we can't use the included service definition provided by Tailscale. To further complicate matters, if we try to hook into the lower levels of the OS and make a mistake, we can end up with a bricked device that's unable to be recovered. Thankfully, there's been a ton of exploration in this area by other developers (shoutout to the Kobo Developer's Corner on the MobileRead Forums</a>), and the generally agreed upon way of running things at boot is by using the udev system</a>. This allows us not only to hook into a startup event, but also into events that are triggered when Kobo Sage connects to and disconnects from WiFi. This allows us to run tailscale up</code> when we have an internet connection, and tailscale down</code> when we lose the connection. To start, I defined the udev rules in a file named 98-tailscale.rules</code> in /etc/udev/rules.d</code> KERNEL=="loop0", RUN+="/usr/local/dstaley/boot.sh" KERNEL=="wlan*", ACTION=="add", RUN+="/usr/local/dstaley/on-wlan-up.sh" KERNEL=="wlan*", ACTION=="remove", RUN+="/usr/local/dstaley/on-wlan-down.sh" </code></pre> The KERNEL=="loop0"</code> rule runs roughly at boot, so that's where I start tailscaled</code>. The next two rules are for when the WiFi is brought up and torn down respectively. The script for starting tailscaled</code> is straightforward. The catch, though (and with Kobo there's always a catch), is that the OS will normally kill long-running tasks spawned by udev. To circumvent that, I used a combination of renice</code> and setsid</code> to ensure that tailscaled</code> isn't killed off. The boot.sh</code> script contains the following: #!/bin/sh # Start by renicing ourselves to a neutral value, to avoid any mishap... renice 0 -p $$ # Launch in the background, with a clean env, after a setsid call to make very very sure udev won't kill us ;). env -i -- setsid /usr/local/dstaley/on-boot.sh & # Done :) exit 0 </code></pre> This script runs on-boot.sh</code> in the background. That script contains: #!/bin/sh # make absolutely sure that iptables is in the PATH export PATH=/usr/sbin:/usr/bin:$PATH # make sure /mnt/onboard is mounted timeout 5 sh -c "while ! grep /mnt/onboard /proc/mounts; do sleep 0.1; done" if [[ $? -eq 143 ]]; then exit 1 fi case "$(pidof tailscaled | wc -w)" in 0) tailscaled --state=/tailscaled.state &> /tailscaled.log & ;; esac exit 0 </code></pre> Since we're not starting a long running process on connecting and disconnecting to WiFi, the scripts on-wlan-up.sh</code> and on-wlan-down.sh</code> only contain a single command: tailscale up</code> and tailscale down</code> respectively. After placing these files in their new homes and rebooting, I can now access my Kobo Sage via my Tailscale network! This also means that I can access other devices on my network, which will come in handy in the future when I want to do things like backup my notebooks to my NAS at home. Other Kobo Devices</h2> Since Rakuten reuses a lot of the same software for all of their Kobo devices, there's a decent chance that these steps will work on any Kobo. If you do decide to go that route, you'll need to verify that your Kobo's kernel was compiled with iptables</code> support, and that the included glibc version is at least equal to the glibc version used to compile the iptables</code> binary. Christmas with Dart Tue, 08 Jan 2019 12:00:00 -0800 Over the Christmas holiday, I sat down to work on a project I've been attempting to get going for several months now. I'm a light Pinboard</a> user, and I'd love a minimal Pinboard client for Android that makes it easy to bookmark sites I find for safe keeping. Recently, Google's Flutter</a> released their first 1.0 release, so I decided to take a look in lieu of reaching for my typical choice of React Native. First off, I didn't really understand what made Flutter impressive until I learned more about what Flutter actually is. Essentially, Flutter is a natively-compiled version of Skia</a>, a 2D graphics library used in Chrome, Firefox, Android, and more. Paired with Skia is an ahead-of-time compiled version of a Dart application that communicates with Skia to draw your applications UI. Flutter also handles integrating things like native platform widgets that would be cumbersome to recreate in Skia (like Google Maps embeds or a WebView). When you stop and think about it, Flutter is almost like a web browser in that it executes arbitrary UI code and renders it, while also connecting to the underlying platform. It's basically a native version of Electron. Basically. If you squint enough. What makes me most excited about Flutter is the Flutter Desktop Embedding Engine</a>, which provides a way to run Skia and Flutter on virtually any platform. I don't think it's quite ready for people to start building with it yet, but pretty soon you'll be able to run Flutter apps on Android, iOS, Windows, macOS, and Linux using 100% native code. While I don't think this will completely displace Electron, I can easily see it becoming a better choice for smaller, desktop-focused applications. But what about Dart?</h2> Flutter uses Dart</a>, a programming language with a, well, complicated past. In its most recent incarnation, however, there's quite a bit to like about Dart! Coming from a web development background, Dart is probably the most JavaScript-like typed language outside of TypeScript. I felt productive right away, and didn't feel the need to read a bunch of tutorials or watch a bunch of videos before building things I was proud of. Things I liked</h3> It's like JavaScript, but with types!</h4> First and foremost, Dart is a typed language. But like many newer typed languages, it feels much more like a dynamic language. For example, the following is totally valid Dart code: var users = [ {'id': 1, 'name': 'Dylan'}, {'id': 2, 'name': 'Josh'}, ]; </code></pre> users</code> is inferred to be List<Map<String, Object>></code>, without having to declare that beforehand. One interesting aspect of this is Dart's dynamic</code> type, which is essentially TypeScript's any</code>. It allows you to pass any type you want around, and is basically an escape hatch from the type system. However, it's important to minimize the use of dynamic</code> in your code since you really want to benefit from Dart's type system. An incredible out-of-the-box experience</h4> Another aspect of Dart that really blew me away was the development experience that the community has rallied around. Dart has a neat package called stagehand</code></a> that handles project scaffolding, which provides nice things like package management with pub</code>, testing with Dart's test</code> package, and a package structure that makes it easy to create quality package pages for Pub</a>, Dart's package repository. Compared to JavaScript, creating and publishing a well-tested Dart package was a breeze. Code Generation</h4> Dart also has some excellent packages for code generation, including the really nifty code_builder</code></a>, which provides a typed API for building Dart code. /// Outputs: /// /// ```dart /// class Animal extends Organism { /// void eat() => print('Yum!'); /// } /// ``` String animalClass() { final animal = Class((b) => b ..name = 'Animal' ..extend = refer('Organism') ..methods.add(Method.returnsVoid((b) => b ..name = 'eat' ..body = refer('print').call([literalString('Yum!')]).code))); return _dartfmt.format('${animal.accept(DartEmitter())}'); } </code></pre> Dart compiles to JavaScript</h4> I will admit that Dart's ability to compile to JavaScript is really cool, but in the current environment, I don't really think it's that compelling of a feature. TypeScript provides all the same type guarantees, and manages to have a much more robust and expressive type system as well. However, this will definitely come in handy when Hummingbird</a>, a port of Flutter to the web, launches. Things I wish were better</h3> JSON support</h4> One of the first snags I ran into with Dart was its JSON support. On Dart's Language Tour</a>, the following sample is provided: var jsonString = ''' [ {"score": 40}, {"score": 80} ] '''; var scores = jsonDecode(jsonString); </code></pre> Looks easy enough, right? Well, the type of scores</code> is actually dynamic</code> as opposed to the expected List<Map<String, int>></code>, and needs to be casted to List</code>. Even after casting, however, the type is still List<dynamic></code>, which means that calling int</code> methods requires an additional cast: var jsonString = ''' [ {"score": 40}, {"score": 80} ] '''; var scores = jsonDecode(jsonString) as List; scores.forEach((s) { print((s['score'] as int).abs()); }); </code></pre> Well, what if instead of casting to List, I cast to List<Map<String, int>></code>? var jsonString = ''' [ {"score": 40}, {"score": 80} ] '''; var scores = jsonDecode(jsonString) as List<Map<String, int>>; scores.forEach((s) { print(s['score'].abs()); }); </code></pre> The above code compiles, but throws a runtime error: type 'List<dynamic>' is not a subtype of type 'List<Map<String, int>>' in type cast </code></pre> For comparison, here's how Go handles a similar situation: var jsonString = []byte(`[ {"score": 40}, {"score": 80} ]`) var scores []map[string]int _ = json.Unmarshal(jsonString, &scores) for _, s := range scores { fmt.Println(s["score"]) } </code></pre> Go's excellent JSON library allows decoding JSON into arbitrarily complex objects. Its superiority is even more obvious when you consider how much ceremony is required from Dart in order to decode more complex objects. Here's how Go handles decoding a suggested tag response from Pinboard: var jsonString = []byte(` [ {"popular": ["tech", "programming"]}, {"recommended": ["compsci"]} ] `) var response []map[string][]string _ = json.Unmarshal(jsonString, &response) for _, s := range response { fmt.Println(s) } </code></pre> And here's the Dart version: var jsonString = ''' [ {"popular": ["tech", "programming"]}, {"recommended": ["compsci"]} ] '''; var response = jsonDecode(jsonString) as List; var result = response .map((i) => (i as Map).map((dynamic key, dynamic value) => MapEntry<String, List<String>>(key, List<String>.from(value)))) .toList(); print(result); </code></pre> And finally, here's a complete Go program to decode a Pinboard Post: package main import ( "encoding/json" "fmt" "log" ) type Post struct { Href, Description, Extended, Meta, Hash, Time, Shared, Toread, Tags string } func main() { var jsonString = []byte(` { "href": "https://date-fns.org/", "description": "date-fns - modern JavaScript date utility library", "extended": "", "meta": "0c4f66fb2dd90d6feeab250a9640d8f4", "hash": "680e1c195c9be62896b1bc5875f89453", "time": "2018-10-19T03:36:58Z", "shared": "yes", "toread": "no", "tags": "javascript" } `) var post Post err := json.Unmarshal(jsonString, &post) if err != nil { log.Fatal(err) } fmt.Printf("%+v\n", post) } </code></pre> And here's the Dart version: import 'dart:convert'; class Post { String href, description, extended, meta, hash, time, shared, toread, tags; Post({ this.href, this.description, this.extended, this.meta, this.hash, this.time, this.shared, this.toread, this.tags, }); factory Post.fromJson(Map<String, Object> json) { return Post( href: json['href'], description: json['description'], extended: json['extended'], meta: json['meta'], hash: json['hash'], time: json['time'], shared: json['shared'], toread: json['toread'], tags: json['tags'], ); } } void main() { var jsonString = ''' { "href": "https://date-fns.org/", "description": "date-fns - modern JavaScript date utility library", "extended": "", "meta": "0c4f66fb2dd90d6feeab250a9640d8f4", "hash": "680e1c195c9be62896b1bc5875f89453", "time": "2018-10-19T03:36:58Z", "shared": "yes", "toread": "no", "tags": "javascript" } '''; var decodedJson = jsonDecode(jsonString); var post = Post.fromJson(decodedJson); print(post); } </code></pre> While the Dart version seems not too bad aside from the verbosity, it really gets out of hand once you're decoding complex, deeply nested structures. It'd be great if converting JSON to a class in Dart was as simple as it is in Go. Non-nullable types</h4> Believe it or not, the following Dart program compiles just fine: String mockingCase(String message) { return message .split('') .asMap() .map((i, letter) { return MapEntry<int, String>( i, i % 2 == 0 ? letter : letter.toUpperCase(), ); }) .values .join(''); } void main() { print(mockingCase(null)); } </code></pre> This is because the String</code> type is nullable, meaning it can be String</code> or null</code>. With Dart, the compiler doesn't warn you when you pass null</code> into a function that requires a valid type, nor does it require you to check that your input isn't null. Thankfully there's a proposal</a> in the works to add non-nullable types to Dart. Building with Dart</h2> Remember a few minutes ago when I mentioned building a Pinboard app with Flutter? Well, it started with building a Pinboard API client in Dart. When I got done building the client and writing the tests, I wanted to check if my tests appropriately covered all the code I had written. Unfortunately, there isn't a great library like nyc</code></a> in JavaScript to generate coverage reports. So, I wrote one myself. duvet</code></a> is a test coverage report library that uses the test</code> package to run your tests, collect coverage with coverage</code>, and generate a nice-looking coverage report. However, while building duvet</code>, I realized I needed something to help me generate HTML documents. There didn't seem to be anything out there that I really liked, so I again took inspiration from the JavaScript ecosystem and built hyper</code></a>, a simple way to build HTML documents with Dart. Here's what it looks like: var fullHtmlDoc = hyper( 'html', children: [ hyper('head', children: [ hyper('title', children: [ t('Hello, world!'), ]) ]), hyper('body', children: [ hyper( 'h1', attrs: {'class': "greeting"}, children: [ t('Hello from Hyper!'), ], ) ]), ], ); </code></pre> Using the previously mentioned code_builder</code></a> package, I was also able to generate an API that matches the HTML specification to give a little more safety to building HTML: var a = h.a( href: 'https://google.com', children: [ t('Click me!'), ], ); // <a href="https://google.com">Click me!</a> </code></pre> hyper</code></a> was heavily inspired by the excellent HyperScript</code></a> library for JavaScript. With duvet</code> and hyper</code>, I was then able to calculate coverage for my Pinboard API client</a> (the third and final library in this grand adventure into Dart). It was nice to see so many green bars! So, what's next?</h2> After my experience with Dart, I came away impressed and excited for what it means for the future: an expressive, accessible, ahead-of-time compiled, type-safe language that runs on virtually every platform. Combined with Flutter's web-like approach to cross-platform support, I honestly think that both Dart and Flutter have an exciting amount of potential going forward. Now maybe I can finally get around to writing that Pinboard client. Gigabit in Baton Rouge Sun, 30 Apr 2017 12:00:00 -0800 A few days ago, I asked my ISP what I thought was a simple question: @CoxHelp</a> I’m trying to find out where in my city Cox is offering Gigablast internet. Who can I reach out to that would be able to help me? — Dylan Staley (@dstaley) April 25, 2017</a> </blockquote> The response I received pointed me towards a page on Cox's website that allows you to enter an address to see if it had access to Gigablast, Cox's gigabit internet service. Since I was more curious about the areas in which gigabit was offered (as opposed to a single address), I tried several more times to get in touch with someone at Cox who might be able to provide a list of neighborhoods. According to the five separate Cox representatives I spoke with, the only way to figure out where Cox is offering gigabit internet is to put addresses into their website, one by one. So, I did. I checked whether Cox was offering gigabit internet for all 168,000 addresses listed in the city's Open Data portal</a>. After checking every address in the parish, I was incredibly surprised by how difficult it apparently was for Cox to give me a list of neighborhoods considering that such a list would have only been two neighborhoods long. In the almost two years since Cox announced the availability of their gigabit service</a> at a ceremony in a neighborhood where the cheapest home starts at $288,000</a>, they've managed to roll the service out to a single additional neighborhood. The cheapest home in that neighborhood starts at $390,000. In November of last year, AT&T announced</a> that their fiber network was available in Baton Rouge, so I used the same method to check where they were offering service. Whereas only 200 addresses have access to Cox's gigabit service, almost 10,000 have access to AT&T's. That means that while Cox's gigabit network is available to only 0.12% of addresses in the parish, AT&T's gigabit service is available to 6%, fifty times more. What's even more surprising is that AT&T's network isn't limited to recently built homes. I was pleasantly surprised to find several homes built in the 70s that have access to gigabit from AT&T. However, what I didn't find were addresses in North Baton Rouge, despite the fact that the area was explicitly called out as having fiber in AT&T's November press release. In fact, if you map out the areas where AT&T is offering gigabit and compare it to median household income, a (sadly unsurprising) trend emerges. </iframe> Income data from the U.S. Census Bureau’s 2015 American Community Survey. View map in new tab</a></figcaption> </figure> A few weeks ago, Ars Technica published an article with the headline "AT&T allegedly "discriminated" against poor people in broadband upgrades"</a>, which highlighted a report that demonstrated there was virtually no overlap in areas that had fiber-to-the-home and areas that had a poverty rate of 35% or more. It's unsurprising that lower-income areas in Baton Rouge are being left out as well. Stencil and Webpack 2 Sun, 12 Feb 2017 12:00:00 -0800 Over the past few weeks I've read a lot about code splitting, or the idea of delivering just the JavaScript needed to run the current page/route. Code splitting is an attempt to balance the benefits and drawbacks to large client-side applications. Instead of downloading and parsing all of the JavaScript for your entire application, you can deliver just a portion. As users navigate to new routes within your application, the code necessary for them is downloaded on the fly. Initial page load times are quicker thanks to less JavaScript being downloaded, but subsequent page loads of new routes have to download a small amount of JavaScript. It's all about finding a balance that's right for your application and users. Since code splitting works by mapping modules to routes, I wanted to see if I could apply the strategy to Stencil, which already has a good one-module-per-route setup. The first step though was to upgrade to Webpack 2, which I knew had quite a few breaking changes. Upgrading to Webpack 2</h2> Annotated Commit</a> Since some of the default Stencil dependencies rely on Webpack 1, I had to upgrade them along with Webpack. npm install babel-loader babel-core babel-preset-es2015 webpack karma-webpack --save-dev</code> After updating dependencies, I modified the Webpack config to accommodate for breaking changes. Thankfully, the only configuration change needed was a tweak to the configuration for babel-loader</code>. In Webpack 1, you could simply define the loader as babel</code>, and Webpack would automatically add the -loader</code> prefix. In Webpack 2, that's no longer the case, so you need to specify it manually. With that change made, I ran stencil start</code> and was greeted with a fairly cryptic stack trace: /Users/dstaley/.nvm/versions/node/v4.3.2/lib/node_modules/@bigcommerce/stencil-cli/node_modules/jspm/node_modules/systemjs/node_modules/es6-module-loader/dist/es6-module-loader.src.js:2712 throw new TypeError('Illegal module name "' + name + '"'); ^ TypeError: Illegal module name "/Users/dstaley/Desktop/stencil-error/node_modules/babel-loader/lib/index.js" at Loader.$__Object$defineProperty.value (/Users/dstaley/.nvm/versions/node/v4.3.2/lib/node_modules/@bigcommerce/stencil-cli/node_modules/jspm/node_modules/systemjs/node_modules/es6-module-loader/dist/es6-module-loader.src.js:2712:19) at Loader.loader.normalize (/Users/dstaley/.nvm/versions/node/v4.3.2/lib/node_modules/@bigcommerce/stencil-cli/node_modules/jspm/node_modules/systemjs/dist/system.src.js:1672:44) at Loader.loader.normalize (/Users/dstaley/.nvm/versions/node/v4.3.2/lib/node_modules/@bigcommerce/stencil-cli/node_modules/jspm/node_modules/systemjs/dist/system.src.js:1713:44) at Loader.loader.normalize (/Users/dstaley/.nvm/versions/node/v4.3.2/lib/node_modules/@bigcommerce/stencil-cli/node_modules/jspm/node_modules/systemjs/dist/system.src.js:2182:44) at Loader.import (/Users/dstaley/.nvm/versions/node/v4.3.2/lib/node_modules/@bigcommerce/stencil-cli/node_modules/jspm/node_modules/systemjs/node_modules/es6-module-loader/dist/es6-module-loader.src.js:2280:40) at Loader.loader.import (/Users/dstaley/.nvm/versions/node/v4.3.2/lib/node_modules/@bigcommerce/stencil-cli/node_modules/jspm/node_modules/systemjs/dist/system.src.js:103:25) at loadLoader (/Users/dstaley/Desktop/stencil-error/node_modules/webpack/node_modules/loader-runner/lib/loadLoader.js:3:16) at iteratePitchingLoaders (/Users/dstaley/Desktop/stencil-error/node_modules/webpack/node_modules/loader-runner/lib/LoaderRunner.js:169:2) at runLoaders (/Users/dstaley/Desktop/stencil-error/node_modules/webpack/node_modules/loader-runner/lib/LoaderRunner.js:362:2) at NormalModule.doBuild (/Users/dstaley/Desktop/stencil-error/node_modules/webpack/lib/NormalModule.js:129:2) </code></pre> The thing that made this error really strange is that manually running webpack</code> wouldn't trigger it; only when running stencil start</code> would I encounter the error. Knowing this, I set out to figure out what Stencil was doing that would interfere with Webpack's ability to load babel-loader</code>. Long story short, Stencil is using JSPM, which polyfills System.import</code>. Webpack's loader-runner</code> defaults to using System.import</code> if it's defined, otherwise it will load things itself. The issue arose from the fact that JSPM's polyfill wasn't compatible with Webpack. Since transitioning Stencil away from JSPM wasn't what I wanted to spend my Sunday on, I decided to use a workaround that I'm honestly not very proud of: I straight up removed the System</code> object from the global scope. The good news though is that it worked, and it didn't seem to break anything obvious! After disabling the polyfill, stencil start</code> was able to properly call Webpack and build the bundle! So now it was time to move on to the difficult part. Code Splitting</h2> Annotated Commit</a> Since Stencil already ships with a very nicely modularized structure, adding code splitting was surprisingly easy. Every page on a Stencil-based site has page type, and each page type has a module containing the functionality for that page type. The first step was to remove all the imports, and redefine them as calls to System.import</code>. So this: 'pages/account/orders/all': account, </code></pre> became 'pages/account/orders/all': () => System.import('./theme/account'), </code></pre> Since System.import</code> uses a network request to load the module, it returns a Promise. I had to make some slight adjustments to how Stencil invokes the module to accommodate for this: // Comment const pageTypePromise = pages.get(templateFile); if (pageTypePromise !== false) { pageTypePromise().then((PageTypeFn) => { const pageType = new PageTypeFn.default(context); pageType.context = context; return loader(pageType, pages); }); } </code></pre> After setting up all the routes and attempting to run Stencil, I was incredibly excited to see my main.js</code> bundle loaded, and then a network call for the homepage's module. Unfortunately, it was requesting the module from the root of the web server, instead of relative to the location of the entry point. After some quick Googling, I discovered that you can manually specify the directory that contains your bundles</a>, and Webpack will use that when requesting them. However, since Stencil uses dynamic paths for each build of your theme, I knew that manually choosing the directory wouldn't work. Thankfully, there's a little known browser API</a> that will report the script element that the current script is invoking from. Like most good things in this world, it's not available on Internet Explorer</a>, but is available in all evergreen browsers. const scriptURL = document.currentScript.src; __webpack_public_path__ = scriptURL.slice(0, scriptURL.lastIndexOf("/") + 1); </code></pre> After figuring that out, I was able to successfully load modules dynamically! To confirm that code splitting was working, I used the excellent Webpack Visualizer</a> by Chris Bateman</a>. When I loaded my main</code> bundle, I was a bit taken aback by the fact that Lodash represented a whopping 32% of the bundle size! One of the really neat things about Lodash is that it's incredibly modular, and also happens to have a very nice implementation of ES modules through lodash-es</code></a>, where each function is exported as an ES module. There's currently an issue with tree shaking for lodash-es</a>, but I worked around it by importing modifying my imports to point directly to the specific file, instead of using module resolution. You can see the changes I made to several of Stencil's JavaScript files in this annotated commit</a>. Results</h2> Going into this, I wasn't really expecting massive performance improvements from code splitting. I'm mainly drawn to the idea of not loading code I'm not going to use, so I didn't perform any benchmarks. (Also the Stencil development server is quite slow because of all the extra things it does behind the scene, so I wouldn't have gotten accurate numbers anyhow.) What I did examine, however, was JavaScript bundle size. I was quite impressed with the difference! Without code splitting, my bundle was 1,941KB (397KB gzip). With code splitting, the two files necessary for the homepage totaled 851KB (202KB gzip). That's an over 49% reduction in bytes transferred, and a 56% reduction in parsed size! The best part of code splitting is that as pages grow in functionality, you need not burden every route with the added cost of downloading and parsing a large amount of JavaScript that will never be executed. Combined with BigCommerce's aggressive fingerprinting and caching, the additional network request on page load isn't that big of a deal. If you're curious about how to implement any of this, you can checkout the code on GitHub</a>, where I've also annotated my commits. It's really important to note that this is highly experimental, and I'm not currently using it in production. In fact, I haven't even attempted to upload a theme built with code splitting, so I'm not entirely sure it'd even work. It's more of a thought experiment in how Stencil can be optimized, rather than a specific recommendation. If you have any questions or comments about Webpack, code splitting, or Stencil, please feel free to tweet me</a>! Migrating to Stencil Mon, 31 Oct 2016 12:00:00 -0800 On June 4, 2015, I managed to convince my current employer Marucci Sports</a> to take on the audacious task of redesigning our website. Exactly four weeks later, I interviewed the amazing designer who would eventually become our Lead Web Designer, Diana Perkins. Over the next 20 weeks Diana and our marketing team created a design that would go on to win our ecommerce platform's first ever Design Awards</a>. We launched late November of that year, just five days before Black Friday. And this year, we raised the stakes. We rebuilt and redesigned our site in just six weeks. Right when we began work on our site in 2015, Bigcommerce</a> (the ecommerce platform maruccisports.com is powered by) launched their new theme development framework Stencil</a>. However, at launch Stencil was missing some of the functionality we had used to create our existing site, namely custom templates for products, categories, and pages. Essentially, if we wanted to use Stencil from the beginning, we would have had to redesign everything to use the same template. This wasn't something we were okay with, so we held out on transitioning to Stencil until custom template support was added. Thankfully, on August 31 of this year, custom template support was added to Stencil. What follows is how we took our Blueprint site and rebuilt it using Stencil. Getting Started</h2> Before we could even begin with Stencil, we realized that there was one restriction we'd quickly run into. Our theme relies heavily on high-resolution imagery, stored in our codebase as massive files that are then downscaled by our image CDN imgix</a>. Unfortunately, Stencil limits your theme to 50MB total, and requires you to host any files that put you above that threshold somewhere else. It would have been an absolute nightmare to keep our image files outside of our Stencil theme; it would have essentially negated all benefits we would have received from being able to run our site locally. This is where one of my favorite aspects of Stencil comes into play: it's completely open source. The code powering the local development environment and the server-side template rendering is available on GitHub</a>, and they even left pull requests enabled. So, with just a few short lines of code I sent in a pull request</a> that enables theme developers to keep large files locally in their project, but reference them in a way that when in production, a remote resource is served. For instance {% raw %}{{cdn "imgix:really-large-image.png"}}{% endraw %} </code></pre> would become /assets/cdn/imgix/really-large-image.png </code></pre> when developing locally, and https://source-name.imgix.net/really-large-image.png </code></pre> when the theme is deployed to a store. This does mean we need to remember to deploy the cdn/imgix</code> folder when we deploy our site, but it's still better than needing to deploy images just to test locally! The other small problem we had to overcome was that when we bundled our theme with stencil bundle</code>, our CDN folder would get included. We submitted a PR to fix that as well</a>, but as of writing it hasn't been merged. Thankfully there's no harm in running a fork of stencil-cli</code>, so that's how we're working around that for the time being. Getting Started (For Real This Time)</h2> Once we figured out our image hosting strategy, we started working on migrating our theme from Blueprint to Stencil. We began by forking Cornerstore</a>, Bigcommerce's showcase Stencil theme. From there, we slowly moved over each custom layout we'd designed, focusing first on pages that didn't have dynamic content. Moving the pages over was relatively simple, complicated only by the fact we were also moving from a Bootstrap-based theme to a Foundation-based one. Our previous theme used Bootstrap's grid classes, so rewriting templates and stylesheets to be more semantic was probably the most complex change we had to make. After our static content pages were added, we focused on sprucing up dynamically generated pages like our category and product pages. However, there's one page in particular that I'm incredibly proud of, and that's the layout for our custom wood bats. Our designer, Diana, really knocked it out of the park with this layout. </picture> In the process of rewriting the custom bat layout, we also took the opportunity to upgrade our bat preview to take advantage of Stencil's new event system. In our Blueprint theme, we were listening for clicks on the individual swatches that represented the colors available. $(".productAttributeList").on("click", "li.swatch", function (e) { if (e.target.tagName === "SPAN") { var children = e.currentTarget.children[0].childNodes; var piece = children[3].dataset.option; var color = children[5].textContent.replace(/\s+/g, "_").toLowerCase(); var state = {}; state[piece] = color; el.setState(state); } }); </code></pre> This…worked. However if you look closely you can see we're relying on a very specific order of childNodes</code>. Thankfully I don't think we ever touched them, but it was still less than ideal. In Stencil, this is a bit more elegant: utils.hooks.on("product-option-change", (event, changedOption) => { // this.options is a mapping of unfriendly Bigcommerce IDs to friendly values. if (this.options.hasOwnProperty(changedOption.name)) { const option = this.options[changedOption.name]; const piece = option.name; const color = option.values[changedOption.value]; canvas.set(piece, color); } }); </code></pre> While the event system is a great addition, the most powerful addition to Stencil is definitely the ability to inject context data (such as the current product, category, customer, etc.) into JavaScript. That's what this.options</code> is in the previous example; it's a collection of the current product's options. By giving developers access to the raw product data, you can build great experiences like our bat preview entirely on the Bigcommerce platform, no add-on needed. One caveat to this, though, is that you don't get a lot of flexibility in what data you receive. For instance, we wanted to add previews of available colors for our products to our category pages. On a product page, you have access to the product's options, but you don't have access to this information from a category page. Because of this limitation (most likely made to reduce backend processing time), we had to build a simple off-platform API integration to get color values for a given product, and render them on a category page. This is actually the only feature on our entire site that requires something other than what you get out of the box with Bigcommerce! </picture> Speaking of things you get out of the box, one of the big things we tried to focus on with our transition to Stencil was using native functionality instead of trying to go at it on our own. This enables us to rely more heavily on the control panel, and enable more people in the company to make minor changes like editing the order of products on our category pages. One of the largest downsides of using static category layouts was that it forced us to make and deploy even small changes like updating a price. Looking back, we should have sucked it up and figured out how to properly use Blueprint to render a list of products. If any theme developer is reading this and thinking of doing what we did and treating their category pages like a static site: don't. In Blueprint we felt we would have been restricted in how we presented our products, but we're incredibly thankful not to feel that way with Stencil. By working with the theme framework instead of against it, we were able to add things like product filters and sorting to our category pages, which in turn provides a much more engaging shopping experience to our users. Things I like about Stencil</h2> There's a ton to love about Stencil, from both the store owner perspective and that of a theme developer. I think I'm a bit lucky in that I can speak from both points of view. As a developer, the single biggest benefit that Stencil brings is a local development environment. As far as I know, Bigcommerce is the only SaaS eCommerce platform that is able to provide an environment like this to their developers, and it's a Really Big Deal. (Although all rendering happens locally, Stencil still requires an internet connection to retrieve live-store information. Thankfully it wouldn't be too terribly difficult to create a mock server that returns static data.) One of the other big things that I really love about Stencil is that it feels like a tool designed for how the web is built in 2016. Stylesheets are written in SCSS; JavaScript is transpiled with Babel (and you can even add your own plugins and transforms!) and bundled with webpack (although you're free to use whatever you'd like), and templates are written in Handlebars. With our Blueprint theme, we basically designed our own build system based on Gulp, SCSS, Babel, and Nunjucks, so we felt right at home with Stencil. The last feature of Stencil I want to mention is it's new optimized single-page checkout. On our previous site, our checkout process was part of our theme, meaning we were the only ones able to make changes to it. On Stencil, the entire checkout experience is controlled by Bigcommerce. While we lose out on the ability to control the overall design, we basically gain an entire team of product designers and developers who are dedicated to making our checkout experience the best it can be. The optimized single page checkout experience is still in beta, but once it rolls out to all our customers we're anticipating a nice decrease in our abandonment rate. One other neat thing about letting Bigcommerce control our checkout is that we were able to being accepting Apple Pay with a few button clicks! Things I'd like to see improved</h2> As of writing, Stencil has only been tested on Node 4 and npm 2. I'd love to see official support for the latest LTS version of Node (v6.9.1) and npm 3 (and yarn). I tried running Stencil on Node 6 way back when it was first released, so I'm not sure if support has been improved since then. For a locally running site, Stencil's performance doesn't seem all that great. I have a MacBook Pro with a 3 GHz Core i7 and 16GB of RAM, and my page load times are still pretty bad. The biggest bottleneck is that Stencil has to make multiple API requests to get the information it needs to render the page. There's a caching mechanism in place, but it doesn't seem to help all that much. Towards the end of our Stencil migration, we setup a Heroku app running Stencil in a Docker container. Sadly the free dyno wasn't powerful enough to run the app without timing out, so we had to move to a standard-2x</code> dyno to keep response times reasonable. Even then, I had to give our team warning that the live site wouldn't be as slow. Another aspect I think Stencil could greatly benefit from is a more robust JavaScript API client. For instance, the included stencil-utils</code> library has a function api.product.getById</code>. At first glance, you might think this allows you to retrieve a product as a JSON object, but by default it will return the rendered HTML of the product page. You actually need to manually add a Handlebars component to your theme</a> that emits the JSON representation of the page's context, and then pass that template to the function as a parameter. I'm glad there's at least a workaround, but it feels silly that it was necessary in the first place. Other quibbles with the JavaScript API include the inability to request product categories by ID (you have to use their URLs like /apparel/adult</code>) and the opaqueness of available configuration options (like including related products in a product response). One of the things we spent a lot of time building for our previous theme was an automated build-and-deploy system. We could run gulp deploy --environment production</code> and have our latest changes live on the website in about two minutes. With Stencil, however, there's no developer-friendly deployment method. Deploying a Stencil site consists of running stencil bundle</code> to create a ZIP archive containing your build artifacts, logging into your site, navigating to your store's design settings, uploading the ZIP file, waiting for it to process, selecting it, clicking apply, and then confirming your change. It's tedious to say the least, but at least there's now a way to atomically deploy a theme. Previously, we were syncing a WebDAV directory with our local repo, which resulted in about two minutes where anyone who visited the site could potentially receive a non-functional page. We mitigated this as much as possible with asset fingerprinting, but it couldn't be entirely prevented. That being said, I did spend some time reverse-engineering the Stencil theme upload process, and so far I'm able to bundle a theme and submit it for processing all from the command line. Pretty soon I think I'll be able to run stencil bundle && gulp deploy</code> to deploy the site, which is an important first step to building a continuous delivery pipeline for the site. My final suggestion is more of a "here's something that would be awesome" as opposed to "here's something that needs to be improved". I'd love to see a Stencil theme that relies heavily on client-side rendering. The beginnings are definitely there, since the stencil-utils</code> package can return rendered HTML, but being able to run your entire store as an Ember or React app would be a really awesome feature. Closing Thoughts</h2> Over the last several weeks, there have been a ton of moments where I really felt the benefits Stencil has brought to Marucci. I will admit, however, that a lot of our previous pain was self-inflicted because of our insistence on using static category pages. Recently I was in a meeting where we wanted to make an older product available for sale again. With a few clicks we were able to edit the product's name and description, change the category it was in, and make it available for sale. For anyone who runs a store, I'm sure that sounds like the most basic thing, but previously that change would have required opening our local repo, copy/pasting the product's nunjucks object from one category file to another, changing the information, and then deploying the site (which took a solid two minutes by the way. Two minutes every time I wanted to preview a change or debug some JavaScript.). Then, we'd still have to go into the admin panel and change the product's name so that it was the same as in the template file. To be honest, writing that all out really drives home how poor of a decision it was on my part, but it does make me that much more excited to be using Stencil! For theme developers who are thinking about moving from Blueprint to Stencil, I say definitely get started. I wouldn't try to rush and get it done before the holiday shopping season, but it should definitely be a priority for you. The improvements it brings on just a style level alone are worth it, but the developer ergonomics really seal the deal. If you're a merchant on Bigcommerce, especially one who is using a stock theme, I'd highly recommend taking a look at the gorgeous Stencil themes available on the Bigcommerce Theme Store</a>. I spoke about a lot of the development benefits to Stencil, but I strongly feel that a good development experience leads to better themes with more features for merchants! If you have any questions about our experiences with Bigcommerce or with Stencil, please feel free to tweet me</a>!

Dylan Staley

What's On My Desk: 2024 Edition

Dynamic Open Graph Images with @vercel/og and Zola

Do It With Kindness; or, Why a Skateboard Can (Sometimes) Be Faster Than a Ferrari

Pushing Buttons with Rust; or, How I Spent Four Hours Automating a One Second Task

Running Zola on WebAssembly

Hello iPod, Welcome to 2023

Tailscale on Kobo Sage

Christmas with Dart

Things I liked</h3> It's like JavaScript, but with types!</h4> First and foremost, Dart is a typed language. But like many newer typed languages, it feels</em> much more like a dynamic language. For example, the following is totally valid Dart code:</p>

It's like JavaScript, but with types!</h4> First and foremost, Dart is a typed language. But like many newer typed languages, it feels</em> much more like a dynamic language. For example, the following is totally valid Dart code:</p>

Gigabit in Baton Rouge

Stencil and Webpack 2

Migrating to Stencil