The tenth circle of (sysadmin) hell

• date2021-12-27
• lang

  en fr

---

• read time 3 min

One of my weirdest missions as an IT contractor. I was send to another contracting company as a helping hand because they were short on staff. It was a mixed bag of horrible HR management and absurd IT ecosystem. I was supposed to be the "transverse team" of one of their development sites with ~50 people, split into various teams for a month or two so they could find a replacement for the previous guy. The mission included managing their ISP contract, their on-site datacenter, their Active Directory and the whole IT environment, their deployement process, and even helping with the Q&A of their software and some development. A real Jack-of-all-trades position, which was already a bad sign.

The following shit circus ensued:

• after a single week of skill transfer and one week of collaboration together, the person that I was supposed to replace left, leaving me as the sole captain of this (sinking) boat;
• one of their apps somehow used the PL/SQL engine of their database as a web server which built the pages dynamically from the resources (html, css, images, data) which were obviously stored in the database itself;
• we had to recreate a Java environment for an app due to technical constraints but were stuck on a dependency which was unavailable on Tomcat. We didn't had the source code because the development had team left some time ago, so we had to analyse the available bytecode to replace said dependency. Failing to do so, we tried to replace the whole app as it didn't seemed to do that much (basically connecting file pipelines). We replaced the 80Mo .jar file with five lines of bash script;
• a guy originally hired as a Linux administrator accepted to work on a Java project despite no prior knowledge of Java. All of his time was now spent supervising himself working on three Java projects against his own will;
• another guy was working on five radically different projects, supposedly giving 100% of his time to all projects. He worked 7/7 and was totally alienated;
• an "X resource" (meaning, yet to be hired human resource) was already lead-developer on five projects;
• a database contained a table named "ONEROW", which effectively contained only one row, being the number "1": it was somehow important not to tamper with it (I wish I was joking);
• the register page of one of the apps required to provide a color in the sign-on form as a mandatory information (despite no known use), with hundreds of possiblities. The default value was bisque (because why not);
• the biggest app of the project was single-user: if someone tried to connect while someone else was logged in, they would receive a "sorry, x is already using the app, try again later" pop-up. The app was processing data into PDF files, which took five minutes per file: scaled up to the frequent hundreds, the app would take hours to run. But as there was no feedback during the processing, people would think that no action was taken and would spam the "start processing" button, which would effectively cue even more data. Best part being, if the person disconnected while the app was processing, the server would completely crash when trying to deliver the finished result. To prevent them from logging out, the developers added three (3) warning pop-ups to "solve" the issue.

I ended up being exfiltrated by my manager after a month and a half, and with no -victim- replacement available the site Lead-manager took over the transverse mission by default.