Tech Support. The epitome of frustration and apathy. Phrases like “Did you try turning it off and on?” and “Did you try unplugging it?” are expected when we call our cable company’s tech support. XKCD truly captures our feelings in this famous comic:

All support jobs are not equal though….

How I ended up on Support #

Rewind to 2009. My career was stalled out after college, working on an internal tool for managing the parts on a helicopter. While it’s a critical to ensure your helicopter doesn’t fall apart while in flight, I didn’t feel like I was growing as a developer.

I interviewed with a few companies and I didn’t do well in the technical interviews. I did manage though to get a job as a Developer Support Engineer at a startup in Palo Alto. The role would support the technical business team on how to build plugins using our API. When I wasn’t supporting the API, I would be helping out the rest of the Support team to answer questions and troubleshoot issues.

Communication Lag #

One of the first things I learned was that talking on the phone was the last thing we, as a support team, wanted to do. Instead, we focused on using email and our ticketing system (JIRA) to communicate with the customer. This helped us understand where issues stood as well as keep a track record of what we had done thus far.

An additional wrinkle was some of our clients didn’t allow phones or outside communications devices on site. What would end up happening is a business development person would have to come out of the office, email what they saw, and wait for a response. Sometimes, the communication lag between a question from Support and an answer was on the order of hours.

I quickly learned that I had to master communicating clearly over email. By getting my point across clearly, I could avoid back-and-forth emails with followup questions. I even created screenshots for less technical people, saying “Click here, write down what you saw, click this, write down what this dialog says, etc”. Even though this seems overkill now, it helped get the answers the first time, without the back-and-forth.

What? I can’t use a debugger? #

One of my most memorable support issues was a tech representative who couldn’t get the application to run on a laptop. He wasn’t very technical and was in a different country with a really bad internet connection. This ruled out having him reinstall the software. He was able to send me the logs though, which was better than nothing.

The logs were standard log4j output and gave some details. The exception was a ClassNotFoundException, and the class was in a standard jar we shipped as part of the application. I had him check if the jar was there and sure enough it wasn’t. Okay, I thought, this should be a simple fix. After the fix was applied, there was a new ClassNotFoundException. At this point, I had him run a quick script to list out all the jars that should be deployed for the application. Half were missing, and it was completely random which ones were missing.

I asked him if he had accidentally deleted them, but he said that he didn’t even know where the jars were, let alone how to delete them. After a long series of emails, we came to the conclusion that the “free” anti-virus software he installed identified the jars as viruses and removed them.

Logging is typically low on the priority list of things to do when developing an application; but, it makes your future-self more sane when trying to reproduce a weird, hard-to-reproduce bug.

Troubleshooting without all the information is challenging. The best advice I ever got was from a coworker who encouraged me to troubleshoot a different problem through the binary search method: try something and see if it works. If it doesn’t, take a little off and try again. If it works, the thing you removed is probably causing the issue. Obviously YMMV but it’s been a great method.

Working with Developers #

I was fortunate to work with some amazing developers who reviewed my code and provided “friendly” feedback. For example:

Me: “Hey, I’m having issues getting these threads to join once they finish. Any ideas?”

Dev: “OMGWTF!?!? Why are you using java.lang.Thread!?!?! Here, use this code as a template. I don’t have time to fix this.”

Me: “Wow, thanks. Didn’t realize that was a thing.”

Dev: “No worries. You should check out Java Concurrency in Practice too, it’s a good reference.”

That conversation seriously helped me understand threading beyond my one class in college. In time, I found a few opportunities to commit changes to the application. I quietly became an unofficial developer, making fixes to issues the devs didn’t have time to fix. By learning from others, take ownership for my growth and finding opportunities to fix issues, I became an actual developer.

Legacy Code #

Whether I was writing documentation on an API or troubleshooting an API issue, I always ended up having to ready someone else’s code. Sometimes I was greeted with one character variables with no meaning; sometimes I was greeted with a complex chain of methods through some proto-builder pattern. Whatever the code, the person who wrote it was either busy or long gone. It became an exercise, almost a game, to understand how the code worked.

Reading and understand others code is key to working on any project. Most of your career will be spent writing code with others. You’ll undoubtedly will look at someone else’s code and have to understand what they were thinking 3, 5, 10 years ago.

Faster #

Performance was always the main complaint from clients. Why was my search taking so long? Why did these calculations take forever? Why am I having OutOfMemoryError when I have a 1 TB RAM server? Usually we were able to get access to these servers to diagnose them; however, we had to get creative sometimes and emulate the data they had. Through various questions, we typically were able to create a similar environment in our office.

This was where I got exposed to tools like YourKit and Eclipse’s Memory Analyzer Tool.  I dug around in memory dumps, CPU profiles, and other data points to draw conclusions on what was slowing the application down. Sometimes it was a HashMap that never really got released, sometimes it was a query using a terrible execution plan, sometimes it was simply our code couldn’t scale. Understanding how to find and address performance issues helped me not only fix these kind of issues, but also better appreciate scaling an application.

Conclusion #

I spent 3 years on Support working through API issues, performance issues, and even the standard “Did you try restarting your computer” issues. Those three years allowed me to learn, grow, and find opportunities to move within the company to become a dedicated developer.

I was very lucky with the leaders, coworkers, and opportunities I had at my support job. They are not common in the tech world, but finding those opportunities, no matter how small or insignificant, can make a major difference in one’s career.