People seem to get stuck in support work. That would be my aversion to doing too much of it. At some of my prior workplaces, there have been people who have been so good at support that they never got assigned to do new development work. Engineers far more experienced and senior than I was (or currently am) were dealing with trivial issues as they were good at it while I got the nice greenfield project.
They had to quit to get out of support.
I work in finance, where we have clearly defined processes for production support, simply because developers don’t have access to production environments.
However, production support teams don’t have a real understanding of our application and how it’s build. So most of the times you have engineers on call with production support, telling them how to debug the problem and come up with relevant logs.
It’s incredibly infuriating and time consuming, and I absolutely hate doing it this way.
90% of the time you also get incredibly vague bug reports with irrelevant logs, and a description of what they think the problem is. Most of the time you need to spend another day finding correct logs and somehow debugging it. Most teams log every single request with all parameters and payloads because they can just replicate the problem locally instead of relying on production support.
We’ve long advocated for either having dedicated support or have engineers on some sort of schedule that can do support.
I like to write software but I don't want to be on-call if the software I wrote breaks at 3am in the morning. I do take my job with professionalism, I do write tests for most of it (not 100% coverage, but 100% coverage of the critical parts), I do monitoring (and answer and fix alerts if they happen during working hours) and I don't deploy on Fridays (and don't allow people to deploy on Fridays).
My code will crash sooner or later. I already know that. I don't write 100% bug-free code. But I cannot accept to give 100% of my time one week per month or so to a company in exchange for money. I just don't understand why people can't understand that I can be a professional only during 8 hours per day, but not more.
If you aren't doing production support, you don't actually know your product. You aren't connected to the pain points your users experience and you miss what is, to your support team and your customers, the glaringly obvious.
I would argue all developers should be required to do some support work.
Another pro for consultancy work: you can chose contracts that that don't require you to be on call for production problems so this isn't forced upon you.
I'm not saying you wouldn't learn from working on production, but whether it's worth the stress is another question. In terms of software development, it's hard to think of a worse feeling than when you do a production deploy, you hit refresh on the website or whatever it is, and it shows a fatal error, then there's a mad scramble to roll back the change and figure out quickly what went wrong before the consequences grow too great. Most of the time bosses + coworkers aren't that understanding about it either and get into finger-pointing.
I really like the author's first point, that one of his learnings was empathy for customers. Being on support for a product you didn't develop really calibrates you to understand what a product needs in order to be supportable, and where customers have problems with products. My time years ago in support was invaluable in shaping me as an engineer; I regularly push back against features that I know will be difficult to support or difficult for customers to understand.
It sounds very much like the poster is describing an on-call rotation rather than "production support", which is a very different thing altogether.
Production support is customer support: responding to chat messages or communications from users.
An on-call rotation, on the other hand, involves responding to production incidents and mounting a proper incident response.
The Google SRE workbook has a great chapter on the subject: https://landing.google.com/sre/workbook/chapters/on-call/
I feel like the author is referring more specifically to troubleshooting issues when production goes down, which I'm fine with if I have no other people asking me for updates. But, I burnt out at my lost job trying to do support, because there was individual support baked into every contract for our SDK and not remotely enough people to handle it all. I was hired as a software developer, not a customer support person, and they are not the same thing. It's unfortunate, because it was my first and highest paying gig after realizing that I also have ADHD, and it was a good company. Thing is, if I have a problem to solve or task to complete, I'm not going to think about how long it's been since I replied to whoever about their pet issue. I'm just going to zone in on my thing, and if the guy next to me doesn't break me out of it by cracking eggs on the desk and burping, then I'll stay on that thread till it's done. That's how my brain works, and expecting otherwise is naive. Anyway, this constant context switching and battling my apparent insufficiency killed my spirit for the work and I turned into a blob of productivity. It's as stupid as expecting me to cook while programming, because either the food, myself, or the code will get burnt.
At some large tech companies, production support as a software engineer does not seem to be a path to promotion, unless you are a junior level engineer. And yet, solving some of the bugs encountered in production environments, especially with a heterogeneous set of users, requires expert level knowledge of a particular software library. It is probably a great way to coast, so I hear.
Teams I've worked on have not had dedicated support staff, but rather engineers rotate 24 hour support duties on a weekly basis.
I have always had mixed feelings about "on call". I dread my turn on the rotation because the imminent threat of a prod issue has a psychological impact on my entire week, even off hours, and usually for a day or two after.
If everybody on the team feels that way, maybe it can act as a forcing function for product quality. I've seen this work on teams that already cultivate a strong sense of ownership.
On the flip side, it really stresses me out, and I sometimes resent that I'm not getting paid overtime for 24hr on call days. Maybe that's just baked into an engineer's salary these days, though...
99% of the companies don't pay for On call rotation / Production Support. Exploiting H1Bs for On call support is very common practice in IT industry.
A lot of the pain around production support is easily solved by having staff in multiple time zones.
A good structure is to have first line support be relatively generic ops people. They can handle problems related to infrastructure, e.g. hardware failures, network problems, or issues that can be handled by adding resources. The deployment process should be consistent enough across applications that they can e.g. roll back to a previous release.
This covers the majority of production problems. After that, it's time to bring in someone who understands the details of how the application works. If the dev team is geographically distributed, then someone is available during working hours. Otherwise, we have to get someone out of bed.
If the dev team has done their job right, this should be a rare occasion. Making the dev team fully responsible for the reliability of the application means that they are motivated to make it reliable. Otherwise there is a tendency to have an underclass of ops people who get abused.
A fundamental mindset here is taking responsibility for the user experience, including reliability. If this is not owned by the product development team, then who?
Our company is trying to move to "devops" and having dev teams on pager duty, doing manual reporting, etc. They seem surprised at the amount of pushback from the devs.
I am an independent developer living off 3 software products I created and sell. I have done all my own support over the last 15 years. While it can be frustrating (especially for B2C) it is also my superpower, as it gives me some much more insight into how I can improve my products. I only do support via email (not phone or chat).
It depends on what and how much one is learning from it. I once interviewed a guy for an ETL developer role. He was supporting an ETL application. In four years, all he had done was restart the application in case of any issues and it somehow worked for him. I've seen my fair share of such cases.
This seems very telling:
> I no longer work at Gojek
...because your company wants you to work two jobs for the price of one?
Engineers absolutely should do prod support. But... There should be layers below. It should only come to engineering when nobody else can figure it out.
You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking.
Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem.
The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your sleeping hours as well; as if that's less important and the company can avoid spending money to hire dedicated support for those hours and instead make you suffer (yes, it's just that - there's no other name for it like "satisfaction", "learning", "growing" or any of those buzzwords).
You want engineers to do production support? Well, let them do it during normal office hours and only few times a month. Or heck, let them do it for weeks but let them punch in and punch out normal office hours. Let them choose to do only one half of the day and have someone else willing to do the another half.
There's no excuse for burning out engineers (esp. unsuspecting youngsters) by pushing them into ungodly hours of work ruining their health among other things while trying to constantly tell them - "do you even realise what a service to humanity you are doing!".
It's just exploitation.