One Year Working With Data
People always say that time flies when you’re having fun, and I can definitely see why. I started my data career in 2021 and I can’t believe how quickly a year has gone by! After graduating last year, I went through an engaging and hands-on training program at work to learn about the energy industry and relevant tools and software. As a newcomer to the industry and as a data professional, our Ignite training was an optimal avenue for me to gain functional knowledge and understand the business of different types of clients that I could be working with. I always stress on the importance of data understanding as it’s necessary to have at least a foundational understanding of what you’re building and why you’re building it if you want to be able to explain its significance and add relevant features. Our 8-week training provided a good head start for me in that regard. Since then, I’ve been working as a consultant, primarily engaged on a project requiring skills in data engineering and data visualization and I’ve learned a lot through practical experience. In one year, I’ve accomplished many professional milestones, and I’ve gained a valuable set of fundamental skills that I am already trying to take to the next level.
Here’s a quick summary of what I’ve learned:
RightAngle: During my 8-week training, I was introduced to RightAngle software and I learned how to implement it from end to end for a client. RightAngle is a software specifically built for Commodities Trading and Risk Management (CTRM), and is used to support the entire workflow of companies that operate in the CTRM space. I learned how to gather requirements from clients and model their business in RightAngle to accomplish their specific goals. RightAngle is also a system that can display reports based on all the underlying data stored in the database, and I learned how to extract data from the software to build more sophisticated solutions for clients.
Data Engineering: I learned about cloud-based solutions, such as Google Cloud Platform, Microsoft Azure and AWS, which were all new to me. Although I learned AWS outside of work, it was part of my year’s learning journey and helped me with making connections and comparisons between my experiences on each platform. I learned about building data lakes and data warehouses and with the help of these cloud platforms, I learned about using ETL/ELT techniques to implement a risk management solution. Additionally, when I was a student, all of my data processing was done locally (using IDES or Notepad++ on my device), but in the last year, I learned about processing data in the cloud which required (by my standards) at least a minimum of intermediate skills in SQL and Python, and beginner skills in cloud technologies. I became familiar with Google products (BigQuery), Azure products (Blob Storage, Databricks, Synapse), and AWS products (EC2, S3, Redshift) for data storage, transformation and processing. I was also able to translate business requirements into coding logic in order to add specific functionality to our solutions.
Business Intelligence/Reporting Software: As a student, I had a lot of exposure to Tableau for data visualization and minimal exposure to Power BI. In the last year, I became competent in Power BI as I built reports, connected them to the data and added various functionalities. I learned about creating new columns and measures in Power BI, and I learned how to use DAX to achieve more advanced or dynamic reporting features. DAX Functions are not always fun to use, but I grew to be more comfortable with them as more complex requirements arose. DAX stands for Data Analysis Expressions, and it is typically used to manipulate or interact with data in Microsoft platforms such as Power BI. When you think of DAX, you can think about Excel formulas to get an idea of what a few of the basic functions look like, but note that they are fundamentally very different from each other. Overall, what used to take me at least 15-20 minutes in the beginning probably takes me about 5 minutes to complete now that I’m more proficient with DAX and aware of some Power BI limitations. I don’t know if I would call myself an expert yet because, as always, there is so much more to learn and Microsoft is always making updates, but I feel confident in the Power BI skills I’ve acquired - confident enough to teach others.
Debugging: Troubleshooting is actually my favorite thing to do, and debugging is just a subset! Debugging is the step-by-step process of fixing a problem and I usually feel energized when there are problems to solve. Obviously, you never want to have too many problems, but I personally liked having challenges or seeing errors that I had never seen before because that meant that I was going to have to figure it out and possibly learn something new in the process. In my first year, I learned a lot about thorough debugging strategies, especially when it comes to errors that may have been caused by code. Debugging can be tedious, but the biggest lesson I learned is that I should always use my resources when things don’t work as they should. Google and YouTube have been superb teachers for me, as well as official documentation pages for whichever tools I’m using (e.g. Power BI). However, no man (or woman) is an island and sometimes we all need to ask for help. With their wealth of experience, my coworkers were excellent resources for all of the things that the internet couldn’t explain very well. I quickly learned that even if I want to try to solve a problem on my own, debugging doesn’t always have to be a solo exercise.
Testing and Validation: Before deploying, I’ve realized that it’s important to thoroughly test different elements of the solutions to ensure that they work as expected. I learned about validating my BI reports against the source system and the importance of ensuring that the data “makes sense.” This is where a bit of domain knowledge or data understanding comes in handy. Other important considerations were testing for performance and fine tuning solutions to work as efficiently as they can. I learned about different methods of testing, especially when it comes to ensuring that edge cases are accounted for or making sure that new code changes didn’t affect prior functionality that they weren’t supposed to affect.
One year can make such a big difference! There were things that I had no knowledge of that I have now created tutorials for, and there are things that I only had beginner knowledge of that I have become more advanced in. When I was job hunting, I knew that I wanted to work with data, but I wasn’t sure if I would end up being an analyst, a scientist, an engineer or any specific data job title to be honest. In my role, I’ve had the pleasure of getting comfortable with data engineering and visualization, and although there is room for me to grow, I’m so much further than where I started. I’ve always wanted to be a problem solver and I get to do just that! I’m still working on my professional goals and making progress each day, hoping that one day I will use data to design new solutions and solve the biggest, most complicated problems. I’ve only just begun.
