WordPress.com in Detroit

Automattic (my employer) has been partnering with Rebrand Detroit to get 100 local, small businesses online.

I’m thrilled that our international company is focusing on Detroit. If you are too (and you are reading this on May 17, 2017) you should come to the Rebrand Cities Salon this afternoon in Midtown.

Matt Mullenweg, the CEO, just posted these new ads showing some of the work that has been done so far:

Workaround for IKEA Bekant Sit/Stand Desks

When I set up my home office in 2015, I knew I wanted a motorized standing desk. The easier it is to raise and lower, the less commitment there is in standing up. The less commitment, the easier it is to stand up. At the time, IKEA had just released their Bekant Sit/Stand desk for about 2/3 the cost of the competition. I wish I had paid more to get a better desk.

The problem: It’s well documented that the Bekant has a power supply issue. The symptom is that it stops after raising just a few millimeters, then later refuses to move at all. This is caused by the power supply not putting out enough juice, according to some reports. My desk got stuck in a standing position for a week, maybe the longest work week I’ve had.

The workaround: Unplug the power supply when you are not actively raising or lowering the desk. Leave it unplugged until the moment you feel like switching between sitting and standing. I’ve been doing this for the past week and have had extremely consistent success. My power supply is hanging in the netting under the desk so I just need to pop the cord into that. No need to mess with the outlet.

The real fix: Keep returning your power supply to IKEA until you get one that works. Some people report never having an issue, and maybe you will get a power supply as glorious as theirs.

The good news is that the desk has a 10 year warranty – if you have your receipt. And your return will be much smoother if you get an employee who doesn’t demand that you disassemble your entire desk just for a detachable power supply.

I was lucky to have an employee tell me that they accept scanned copies of receipts, so mine lives in Evernote and comes out once a year when I want to try to fix my desk yet again. It takes about an hour to do the exchange in my experience, and I’m on my 3rd power supply. I also happen to live 10 minutes from an IKEA; I can’t imagine driving a couple of hours to do the exchange.

You may have some luck in contacting Rol Ergo directly for a replacement, according to this Facebook comment.

[Photo credit: mastermaq under CC 2.0 BY-SA license. Because my desk is in no condition be seen by anyone.]

Coworking at Workantile

One of the challenges of distributed work is isolation. Our third places are already disappearing, losing the office as a second place can be tough. So, about once a week, I drive over half an hour to Workantile, a coworking space in Ann Arbor.

It started with the Ann Arbor Software Co-Workers. I attended a few of those meetups at a coffee shop and was reminded that there is something energizing about being around people. At the same time, I wanted somewhere that I could hop on a video call. I wanted somewhere I didn’t have to make sure that someone watched my bag when I went to the bathroom. Somewhere I didn’t have to OD on caffeine to make sure I’m paying my rent. So I wound up at Workantile.

[As an aside, my employer has a coworking stipend that covers traditional coworking spaces as well as coffee rent at coffee shops. Companies that want to support distributed workers, take note.]

Workantile is part of a vanishing breed. There are plenty of coworking spaces that are “you pay money, get desk, end of story.” There are also plenty of incubators that want to host your startup. What Workantile focuses on are the independent contractors and the remote workers that want a second place. Here’s how they put it:

At Workantile our goal is to support the work and social needs of independent workers typically already established in their job or business, whether they’re self-employed or remote employees. Workantile creates a diverse workplace environment without the office politics. Our social network isn’t for smarmy business-oriented “networking,” but for real social interaction and camaraderie.

Quartz has a great write-up on how coworking has moved from the Workantile model to the other models. Companies like WeWork have been focused on flexible office space, ignoring any sort of community aspect. That probably makes financial sense, but I’m not really interested in “Office Hotelling.” It’s kind of disheartening that as more workers are becoming remote and distributed, social coworking is on its way out.

There’s a new coworking space “Pastel” opening much closer to me – a 5 minute commute instead of a 35 minute commute. Their vision is “a place for independent business women to find support, balance and connection.” That’s not quite me, but I’m glad to see other coworking spaces opening to focus on something other than real estate. I will be keeping an eye on them but for now it sounds like Workantile is a better fit.

Don’t get me wrong: I still loathe commutes but once a week is the right balance for me right now.

[Photo credit Chris Salzman]

My Net Neutrality Letter

[Below is the letter I sent to my representative, my senators, and the FCC. Well, I will send it to the FCC when their site stops being overloaded. It seems that John Oliver told the entire internet to Go FCC Yourself and they did. And then someone also spammed the site with purloined identities and anti-Net-Neutrality comments. So the FCC site is a little slow right now. But once it gets back on its feet, this baby is headed right there too.]

I want to voice my support for Net Neutrality. I am a software engineer in your district, and make my livelihood on the open internet.

I can appreciate the concerns of people opposed to Net Neutrality. I agree with them that efficient markets generally provide better outcomes than centralized control for most people. I can sympathize with the attraction of “zero rating” – letting people use some services without counting against their bandwidth. Ideally we should be able to continue to benefit from the phenomenal economic growth that the internet has provided.

Internet service is not an efficient market today. Until it is, we need Net Neutrality. I am fortunate to have 2 high-speed ISPs available to me, which makes me part of the only 10% of Americans with that choice. Markets don’t work when 90% of the consumers don’t have a choice. This Econ 101 principle is clear in an industry that is both highly profitable and consistently hated by their customers.

Net Neutrality also ensures that the internet can stay fertile ground for new businesses. If ISPs are allowed to provide incumbents with exclusive traffic, the cost to disrupt goes up. Would Google or Facebook have been able to grow if they had been crushed in their infancy by Alta Vista and MySpace? I want to ensure that the services I love 10 years from now are able to start today.

Net Neutrality is a key piece to the internet’s economic engine. Please keep the internet a Title II service.

[The John Oliver piece is below, I imagine the FCC is already aware of it so I chose not to include it.]

iOS Widgets for Home Assistant with Workflow

One of the key principles of home automation is being able to control things quickly and easily. I don’t want to have to unlock my phone, open an app, wait for it to load, and then finally find the device I want to control.

I like that SmartThings supports the Today screen widgets in iOS. They let me swipe dow on my locked phone and trigger a routine. For example, I have a button set up to run “Good Night” when I get to bed, which turns off lights, locks the door, and sets the thermostat.

I wanted the same in Home Assistant. It turns out that Workflow is perfect for this.

If you have the Workflow app installed, here’s an example workflow you can customize.

To make this, I created a new “Today Widget” workflow in the app. Basically we are going to create a call to the Home Assistant RESTful API, so to start you need to add a URL action. This should be in the form of https://your-home-assistant-site/api/services/domain/service. For example, if we want to toggle the bedroom lights of the Home Assistant demo site we would put in the URL https://home-assistant.io/demo/api/services/light/toggle

Next, add the “Get Contents of URL” action. Set the Method to POST, under headers add “x-ha-access” as the key and your Home Assistant password as the value. Set the Request Body to JSON and add any service parameters in the Request body. In our example we send a text value for “entity_id” as “group.bedroom”.

At this point you should be all ready to try it out! Here’s what it looks like:


(note that is not my real passphrase)

Workflow is a great way to build easy-access buttons to run things in Home Assistant. Along with the Home Assistant native app, I think Workflow is a must-have for any iOS user.

Speaking at Self.conference 2017

I will be speaking about home automation with Home Assistant at Self.conference in May! One of my goals this year is to do more public speaking, so I’m thrilled to get to be part of this event. There is a pretty amazing lineup of speakers so you should probably go ahead and register already.

Here’s the write-up of my talk:

A Smart Home To Call Your Own

The stuff in your home is getting smarter every day. It’s time to take control of it with Home Assistant, a Python open source app.

This talk will provide an overview of how Home Assistant can be set up and configured to make your home smarter. You’ll be able to control your smart home devices so that your home starts working for you. With presence detection, logging, and mobile access, you’ll be the master of your domicile.

Once we see what Home Assistant can do we will explore how to write a custom components to start automating our own devices. Welcome to the world of tomorrow!

My Favorite Bug

Years ago, at a financial services company…

It started with some mild grumbling. Some people were complaining about getting signed out on our customer portal. But it was probably just user error, right? The reports seemed to happen more and more though. Then, employees who worked on the product started to get signed out.

At this point we have to admit that there is a problem. Every so often someone is using the site and they are suddenly booted out.

It happens seemingly at random. If you have a bug that you cannot reliably reproduce, your goal is to gather enough information until you can. “Seemingly at random” is a euphemism for “caused by some piece that we aren’t considering.”

Being able to reproduce a bug is important. First, you want to be confident that you have fixed the issue. If you can’t be sure you’ll trigger the bug, you can’t be sure your fix worked. Second, the factors that go into triggering the bug give you insight into where the problem is. Working toward a consistent reproduction is helping you triangulate the problem. So we started gathering information in hopes that we could reproduce it.

One thing you can do is ask yourself “what changed?” One of the things that changed was that we were rolling out a new version of our in-house PHP framework. This felt important, but also not, since the affected site was on the old framework. What’s more, both frameworks used the same way to delete sessions. Something like:

sprintf('DELETE FROM `sessions` WHERE `expires_at` < %d', now());

Which is a pretty reasonable way to expire sessions. The session creation code was similarly reasonable

sprintf('INSERT INTO `sessions` (`cookie`, …, `expires_at`)  VALUES (:cookie, …, :expires )', $cookie, …, now() + SESSION_LENGTH);

Jason, one of our analytics folks, was able to add HTTP monitoring. We figured out what the problem looked like from the browser side (a request with a session id setting a different session id in less than n seconds) and started logging how often it happened.

What we learned from that the frequency was accelerating. The longer it took us to fix, the worse it was going to get.

We still cannot reliably reproduce this at this point, so we have to keep gathering information. I was tempted to deploy changes out to see if they worked. This is dangerous. If you don’t understand the problem, you don’t know if a fix is going to inadvertently make things worse. Remember: if it seems random, you don’t understand it.

Adam, one of the devs on the team, added logging to our session delete code. Before it ran the DELETE statement, it did something like

debug_log(query(sprintf('SELECT cookie, expires_at FROM `sessions` WHERE `expires_at` < %d', now())));

so we could see exactly which sessions were being deleted. That’s when things started to get weird.

We would compare that to our HTTP monitoring and find the session and see that it definitely should not have expired based on the timing. But the session logging says that it definitely should be deleted based on the time it was logged.

At this point we reach the “Eureka” moment. For context, “Eureka” is what old Archimedes shouted as he jumped out of the bathtub. While the bathtub is clearly useful in realizing that water displacement can measure volume, I don’t think enough credit is given to the bathtub as a place of thought in that story. I consider showering (along with dog walking, driving, and sleeping) to be an important part of debugging.

And so I was, when I was trying to figure out how to reconcile the two facts:

  1. Sessions were expiring before they should be
  2. Sessions were expiring on time

I realized the disconnect, hopped out of the shower and sent an email to our Unix admins asking them to check the clocks on our production servers. Sure enough, one of the load-balanced servers wasn’t running ntp and it’s clock had drifted out of sync.

What was happening was if the session code was called on the rogue server then now() was wrong. That meant that sessions created on it would live extra long, but sessions deleted on it would die young.

Why was it accelerating? It turns out that in V1 of our framework the session-delete code was mistakenly not hooked up, but that was fixed in V2. The way PHP works is that every n requests it calls the cleanup function. We were moving more and more traffic to V2, which meant that the cleanup function was being called more often.

I like debugging. Well, I like having debugged. It can be maddening in the middle of it but there is a lot of satisfaction in solving a puzzle. A bug is a puzzle with stakes. If you are spending your time on a bug, that means its important. There’s also satisfaction in finally arriving at a simple, testable, theory of a bug.

I like this bug because it required us to understand so many pieces that make up the system: HTTP requests, session persistence, load balancing, and server clocks. It’s also nice when the fix is to adjust the clock instead of recode your entire app.

Image credit: Public Domain image NH 96566-KN The First “Computer Bug”

Well… how did I get here?

Johns Malkovich

Employee referral programs are killing diversity in the tech industry

Roughly 70% of tech companies have programs to encourage referrals and most importantly, referrals account for up to 50% of new hires in the US.

Unfortunately, an employee being asked to refer someone for a given position will almost always go for somebody who shares the same social capital: skills, tastes, posture, clothing, mannerisms, material belongings, credentials, degree etc.

I’ve had seven tech jobs. I got three of them as a referral from an existing employee. Importantly, those three jobs also accounted for the bulk of my professional growth. Furthermore, for the second and third jobs in that list I was referred by my network from my first referred job.

I got the best jobs of my career because I knew someone who worked at the company. Someone who was like me.

That doesn’t mean I didn’t deserve the job, wasn’t qualified for it or anything. But it’s useful to remember that someone without the same networks as me would find it difficult to compete with me for those jobs. At a macro level it means access consolidates around similar people. That’s a problem for companies, since diversity is good for business.

Referrals are cheaper and more successful than less targeted funnels. Referral bonuses seem like a decent thing too. A referral bonus is also a good answer to the question “if you’re paying that recruiter to hire people, why should I do their job for free?” I’ve earned a few referral bonuses over the years and I’ve known a couple of people who turned referral bonuses into a second income stream.

That post I linked glosses over another piece of referrals. When you are interviewing for a job, you are also interviewing the company to see if you want to work there. Tech workers are in a tight labor market (for now). Employers have a tough time finding talent. A large part of the reason I switched to the jobs at which I knew people is because I had social proof that they were good jobs.

I only switched jobs because people I trust told me it was a good place to work.

If referral programs are successful, if they make hiring more efficient, and if they benefit workers, then what is the problem? This could be a reminder that “what you measure, you improve” is both a promise and a warning. Changing the metrics of success changes all of that. However, employee retention and success are easier to track than the innovation benefits of diversity. One of my takeaways from Deep Work was “The Principle of Least Resistance:”

In a business setting, without clear feedback on the impact of various behaviors to the bottom line, we will tend toward behaviors that are easiest in the moment.

So what is the answer for companies?  How do they fill that funnel while still convincing experienced talent to work there? I don’t really know, sorry. I know there are people who think about this deeply. If you know who they are please point me in their direction.

I do know that diverse workforces attract diversity. Knowing that there are people like you is reassuring for people, including me. Getting to a workforce where there are “people like you” (along whatever axis you’re considering) for candidates is the sticky wicket.

It’s not like there were no hurdles in getting those jobs, but it’s valuable to remember that a hurdle doesn’t prove an even playing field. What I can say is that when you are unpacking things like privilege and diversity, I think it’s important to ask “Well… how did I get here?

Deep Work

I recently finished Deep Work by Cal Newport.

To build your working life around the experience of flow produced by deep work is a proven path to deep satisfaction.

Deep Work feels like a continuation of Newport’s previous book So Good They Can’t Ignore You. That book argues that your mastery of your craft is a larger factor in your fulfillment than the choice of work. It then goes on to prescribe mastery through “deliberate practice” to improve. Once you have leveled up, Deep Work offers a next step: a path to productivity.

The ability to concentrate intensely is a skill that must be trained.

Newport spends a good amount of pages making the case that blocks of focus and solitude are vital for a knowledge worker to produce the best output. His strategies should not be surprising. “Don’t goof off on the internet” seems like “eat right and exercise” for productivity – advice that’s easy to say but hard to live. Hopefully, by arguing the value of deep work and the need to train your brain, readers will have success with his methods.
When I get the urge to check Twitter during a build/deploy, it’s helpful to have a reminder that small choices matter. Reducing my time on social media – the internet’s Skinner box – has helped me focus. It’s also been helpful to also consider the schedules he describes. Blocking off time to be unreachable has been a boon. So much so, that I’m revisiting RescueTime to try creating focus blocks.

The book offers working templates for different personalities. While I am productive in the solitude of my office, I feel that I am most productive when I am pairing with someone (this may be some selection bias – only certain work lends itself to pairing). In his section on work styles, I’m glad that he steps back from the isolation he describes elsewhere and offers support for working in partnership.

I enjoyed this book and am glad to have found it. The most valuable piece for me has been the reminder that my minute-by-minute decisions become powerful habits. I recommend this to any knowledge worker. I’d also be extremely interested to find out if it benefits other professions.

See also: There’s a good hacker news discussion

Brewing on a school night

One thing I tell friends about parenthood is that the important stuff sticks around. You may not be able to spend a weekend binge-watching shows anymore, but you’ll find ways to keep the hobbies that are important to you.

Jenny got me Adventures In Homebrewing’s Vanilla Porter recipe for my birthday. I’ve made this kit before and it’s tasty. So, last night after we got the girls in bed, I was able to brew a batch of beer.

I don’t have 6 hours to spare for an all-grain batch. It was satisfying to knock out an extract beer in 3 hours (including clean up). Sure it was a bit cold in the garage, but the nice thing about brewing in the winter is that the wort chills much faster.

Recipe Details

Batch Size Boil Time IBU SRM Est. OG Est. FG ABV
5 gal 60 min 25.0 IBUs 22.8 SRM 1.056 1.015 5.4 %

Style Details

Name Cat. OG Range FG Range IBU SRM Carb ABV
American Porter 20 A 1.05 - 1.07 1.012 - 1.018 25 - 50 22 - 40 2.3 - 2.9 4.8 - 6.5 %


Name Amount %
Caramel/Crystal Malt - 60L 8 oz 5.56
Black (Patent) Malt 4 oz 2.78
Chocolate Malt 4 oz 2.78
Pale Liquid Extract 8 lbs 88.89


Name Amount Time Use Form Alpha %
Cluster 1 oz 60 min Boil Pellet 7
Willamette 1 oz 5 min Boil Pellet 5.5


Name Amount Time Use Type
Vanilla Bean 2.00 Items 0 min Secondary Spice


Name Lab Attenuation Temperature
SafAle English Ale (S-04) DCL/Fermentis 73% 59°F - 75.2°F