Why do I still have a Job?
Because everything is still done in Email and Excel. A long somewhat coherent rant. On AI, LLMs, and Structured Data.
In around 2023 I first started working on a project utilizing MegaDetector (now Pytorch-Wildlife) that came about when I was trying to automate my animal classification and labeling job using image detection models as a research assistant. In that time I came across a book simply titled âReproducible Analysis with Râ. In section 6.4 there is simple an image titled âMultiple Tablesâ .
It pictures one of the easiest ways that data can become unorganized - that there are different observations or pieces of data within the same row. This problem of course is relatively easy to fix if you put them into separate tables. However, this âfixâ compounds over time as more imperfections take place, making it difficult to analyze or visualize the data.
Before I go in further, Iâm going to make conflicting opinions because I hold the two foundational beliefs in regard to AI, LLMs, and Automation and why people will still have jobs in the coming future.
- When data is organized, there is little need for human labor. I fully believe most business task can be automated when data is structured.
- Current LLM (called âAIâ, even when itâs not) systems, even with error correction practices and continuous improvement have to interface with the imperfect data formats that currently are solved in closed environments. Muddy data can never be cleaned without proper context.
Right now, the current work infrastructure is based in some conjunction of a messaging platform (email, teams) + software specific task management (Reconciliation application, tax application) + a project management platform (such as Jira, Ticket System). There have been attempts to broach this of course with the advent of Enterprise-Resource-Planning (ERP) systems. ERP systems reduce the problem of data silos by combining multiple business operations pipelines into a structured format. However, if there is ever a user error in either using the system or trying to not use the system (such as downloading the data to excel) then the purpose is lost.
When I read about the concept of âmanagement execution appsâ from Cal Newportâs book âA World Without Emailâ it made complete sense to me why currently a majority of hand-off task are still inefficient. In my current company we call this an âend-to-end business processâ and in the broader broad of computer science a âdata pipeline. They are inefficient because there is a lack of clarity and structure. Much of my current jobs, and many out there, are making sure that youâre pinging people enough times through email until they respond with the email that you need.
In Derek Thompsonâs podcast âPlain Englishâ there was an idea that came up of âEmptying email inbox as a competency of AGIâ. I think this is an extremely inefficient and incorrect way to think about the future of AGI and job replacement with the context of âLive in the future, then build whatâs missingâ from Paul Graham. By using a metric of email - We are solving todayâs problems, not building tomorrowâs solution.
To give an example of this - the RSS/Atom Feed. There is a plethora of reasons why it was not widely adopted. (such as not being able to collect information about who subscribes to a feed, aka data collection instead of privacy). But it is a dramatically more energy friendly way to post content to followers than newsletters, that require you to send an email individually to each subscriber.
The Newsletter solved todayâs problems, RSS built tomorrowâs solution.
LLMs/AI solve todayâs problem. Structured data builds tomorrowâs solutions
For the area I live in of Hampton roads there are collection of RSS feed and Newsletterâs to know whatâs going to happen in the area. You can use an LLM to organize this data. [For example - the project Iâm currently working on at-least has 20+ sources that Iâm pulling from.]. However, it would have been much easier for me if everything was an RSS feed so I wouldnât have to HTTPS scrape a website individually for their blog post and updates.
Then I came across ai-2027 that âpredict[s] that the impact of superhuman AI over the next decade will be enormous, exceeding that of the Industrial Revolution. This raised an eyebrow for me.
For those that have any experience with power automate, n8n, Zapier, IFTTT, Decisions.ai there are many finite and specific pieces at play that make a workflow either succeed or fail. A huge portion of that is error correction and decisions trees. When I was recently trying to make my own process, I came upon a very particular problem. When I was parsing an HTML document to get a string of characters for my URL link. Every single time through an LLM model it would mess up the link and change it up even with variability set to zero. I ultimately gave up because the marginal benefit was not there when a model was able to efficiently pull all other information and links.
To say again: LLMs/AI solve todayâs problem. Structured data builds tomorrowâs solutions
- Why donât we have a fully auditable government? Because of Data Silos
- Why do I still have a job? Because of data silos, unorganized data, and limited business structurization relative to what is needed for automation
- Why donât we have a unified healthcare information system? Because of Data Silos and Political Factors
- In essence, why donât we have XYZ when it comes to unification of a system? Because there is a massive infrastructure, cultural, and monetary cost associated with change.
So, what exactly do I mean by structured data? Going back to the beginning of the article there is a different between data being structured and data being organized. The easiest way to think about it is bring the mindset of âcan I make this into a digital form that people can enter information intoâ.
- Grouping the M&M candy by color is organized
- Labeling each M&M candy by Unique ID, assigning them a color, giving a date and time labeled, unique position in color category, person assigned to eat candy, person candy was bought from, candy expiration date, list of ingredients in candy, is structured
I found a quote on reddit which adds to this sentiment of why âEverything is a Graphâ:
Everything is literally a graph.
When you dereference a pointer, the instruction doing the dereference is your source node. The address is your edge. The memory itâs pointing to is the destination node.
Whether or not the memory is cached and retrievable without going into main memory is determined by, you guessed it, a graph.
Everything is a graph.
Your process is connected to a process tree - a graph. It gets allocated a time slice, and is then scheduled by your OS. The scheduler stores information as a graph.
Everything is a graph.
When you go to Google, you perform a DNS lookup first. The closest DNS server thatâs available is a node adjacent to your network whose edge contains the least weight among all the other nodes.
Everything is a graph. When you parse user input, youâre creating an FSM to tokenize the information. Each state is a node. Each transition is an edge.
Everything is a graph.
The world in which my job is replace is that in which data is structured not organized. My email can already be organized by rules or an LLM system, but neither is going to solve the work tasks that needs to be done because there is an underlying structure that is not contained within the email file. I have a bias towards information systems because an introduction of an unstructured form like a person or LLM model can introduce untended consequences within that defined system. And even then, there are technologies coming out that can interact with your operating system and applications. So, itâs not that hard to imagine an autonomous agent replacing that work task. But again - itâs solving todayâs problems, not building tomorrowâs solutions. If there is ever a change in the business process, there will have to be a reprogramming in how an agent will handle the input data. By driving change at the input level, you completely eliminate the need to an agent to exist.
Sometime this isnât the case. We have technology such as self-driving cars that are increasing in capability with Waymo cars now being in 10 cities itâs a future into a world of less accident-prone commutes. So, all we have to do is wait if Iâm wrong. And wellâŚget paid to do some work.